Skip to main content

Ralsina.Me — Roberto Alsina's website

Making Linux systems that don't suck. Part II

Introduction

Wel­come to part II of this se­ries. Here is Part I and here is Part 0.

To­day we will talk about sched­ul­ing unat­tend­ed tasks.

The clas­si­cal way to do this in Unix is us­ing the at and cron sys­tem­s.

I say sys­tems be­cause they are not sim­ple com­mand­s, they are a whole set of com­mands and dae­mon­s. Or rather two of them.

On one hand, you have at, atq, atrm, batch and atd.

On the oth­er, there's crond and crontab.

So, all things con­sid­ered, there are 7 com­mands you need to con­sid­er to man­age unat­tend­ed tasks [1].

While one of the pil­lars of Unix is that each com­mand should do one thing and do it well, we have here one of the most pe­cu­liar re­sult­s. You have sev­en com­mands to do one or two things, and they do them rather bad­ly.

What's wrong with them?

The artificial division of work.

The usu­al rea­son for hav­ing two com­mand sets is:

  • At is meant for one-shot, non-re­cur­ring tasks.

  • Cron is meant for repet­i­­tive tasks.

Let's see that again.

  • Do you use any kind of soft­­ware to sched­ule your ap­­point­­ments?

  • Do you use two pro­­gram­s, one for things that re­­peat, one for those that don't?

  • Would­n't that us­age strike you as slight­­ly nut­s?

There is ab­so­lute­ly no rea­son why you should have two sets of pro­grams with com­plete­ly dif­fer­ent work­flows, syn­tax­es, CLIs, spools and what­ev­er else for this. It's triv­ial to ex­tend ei­ther one to do the job of the oth­er.

But it's not done be­cause...

This is old unmaintained code

Look at the De­bian pack­age page for cron and for at. These are the most com­mon (but not the on­ly) im­ple­men­ta­tions used on Lin­ux dis­tros.

First un­usu­al thing: they have no web­page. You know why? Be­cause their last re­lease is from when web­pages were not all that com­mon.

In the cron sources, the new­est file is dat­ed 08/06/96.

In at, it's dat­ed from last year, but that's a bit de­ceiv­ing: look at the at changel­og:

  • A mi­nor up­­­date in 2006.

  • A larg­er patch in 2005.

  • A crit­i­­cal patch in 2002.

Yes, this piece of soft­ware sees main­te­nance ev­ery two years or so. And it's not be­cause it's fin­ished and per­fec­t, it's fix­ing things like "al­low user­names longer than 8 char­ac­ter­s" in 2005!

Did I men­tion that...

  • at/atq/a­trm is SUID root?

  • crond runs as root?

Yup. 1996 code.

Which is why these pro­gram­s....

Suck really, really bad for many uses.

First, cron (keep in mind that this is just an ex­am­ple, there are dozens of things cron can't do right).

Here's how you run a cron job on the last day of each month (a com­mon busi­ness use case):

0 3 * * * if [ "`date -d tomorrow +%d`" = 1 ]; then run_the_script; fi

Of course that may work on­ly on Lin­ux, check this dis­cus­sion for more fun.

That's al­so the an­swer to "How do you run a cron job ev­ery X min­utes" when X is not a di­vider of 60. Or ev­ery Y hours, when Y is not a di­vider of 24: run it ev­ery day/hour/minute and make it fail when it should­n't run.

Now think about how to make it run on the last mon­day of each month. Come on, I give you ten min­utes to fig­ure it out.

Now, let's con­side at.

Here are some ex­am­ples of time spec­i­fi­ca­tion syn­tax that work with at:

teatime + 4 hours

tomorrow

now + 6 hours

now + 23 minutes

10am Jul 31

And here are some that don't work:

now + 6 hours 23 minutes

now + 6 hours + 23 minutes

Jul 31 10am

What on earth is that syn­tax? What's wrong with sim­ply spec­i­fy­ing a date and time / a time from now?

By the way, here's what the at man page tells you to read to know the way to spec­i­fy a time. Please, please check it out. It's a yacc gram­mar. Yes. You are sup­posed to un­der­stand yacc gram­mars to fig­ure out at.

Now, as­sume I sched­uled a job (just "ls /tm­p"), and want to change it. Here's what I get to work with (not for the weary):

#!/bin/sh
# atrun uid=500 gid=100
# mail ralsina 0
umask 22
MANPATH=/usr/man:/usr/X11R6/man:/opt/java/man:/opt/java/jre/man:/opt/kde/man:/opt/plan9/man:/opt/qt/man:/opt/qt4/man; export MANPATH
:
:
ANOTHER 50 lines of environment variables
:
:
cd /mnt/centos/home/ralsina || {
        echo 'Execution directory inaccessible' >&2
        exit 1
}
ls /tmp

The idea seems to be run­ning the job in con­di­tions as close as pos­si­ble to the mo­ment you sched­uled it. Al­though that's al­ready ar­guably wrong (I pre­fer my sched­uled tasks to run in a con­trolled en­vi­ron­men­t, not in what­ev­er mess my xterm is!), I am pret­ty sure there is a way to do this less messy. Like a "ed­it en­vi­ron­men­t" switch sep­a­rate from "ed­it the job".

So, if cron and at suck, what should you use? Well there are some...

Alternative implementations

If you are go­ing to keep on us­ing the cron/at sys­tem, please don't use those. In­ves­ti­gate al­ter­na­tives, here are a few point­er­s.

One con­ser­va­tive al­ter­na­tive for cron is bcron [2]:

  • It has been worked on this cen­­tu­ry.

  • It's de­signed to be se­cure (no suid bi­­na­ries).

  • Has a main­­tain­er.

  • Seems to be ful­­ly com­­pat­i­ble with Vix­ie Cron (what you prob­a­bly use now)

On the oth­er hand, it adds no new fea­tures, and cron re­al­ly needs some.

There is al­so fcron:

  • Ac­­tive­­ly main­­tained

  • Works well if your sys­tem is not up 24/7 (no need for anacron or oth­­er ug­­ly­­ness)

  • Many nice and use­­ful new fea­­tures.

Bruce Guenter (au­thor of bcron) claims fcron has some is­sues like lack of /etc/cron.d and /etc/crontab and some pars­ing in­com­pat­i­bil­i­ties with Vix­ie cron, but I have not ver­i­fied it, and they look like you can work around them.

There are oth­er cron­s:

  • dcron

    Dil­lon's cron, last up­­­dat­ed in 2005.

  • hc-cron

    Based on Vix­ie cron, last up­­­dat­ed in 2002.

  • Mi­cron

    An in­­ter­est­ing cron aim­ing to be smal­l, se­cure and not re­­ly on mail for no­ti­­fi­­ca­­tion­s.

    Last up­­­dat­ed in 2005.

  • Mcron

    Writ­ten in Guile, al­lows for al­ter­­na­­tive con­­fig files in scheme. In­­ter­est­ing but a bit ex­ot­ic for my taste. Last up­­­dat­ed in 2006.

I am sure I am miss­ing a few more.

At re­place­ments:

  • mini-at

    Reg­u­lar at, with­­out the too-s­­mart syn­­tax. There seems to be no sup­­port for run­n­ing com­­mands as dif­fer­­ent users un­­less each runs a copy of the dae­­mon, so I don't rec­om­­mend this.

That's it. I can't find a de­cent im­ple­men­ta­tion of at [3]. Suck­s, does­n't it?

But maybe you don't need to use these be­cause there are...

Incompatible Alternatives

If you are will­ing to be more dar­ing, you can con­sid­er al­ter­na­tive job sched­ul­ing sys­tem­s.

Here are some I have found:

  • usched­ule

    Ad­­van­­tages:

    • You can sched­ule repet­i­­­tive tasks start­ing af­ter a cer­­­tain date/­­­time

    • You can sched­ule non-repet­i­­­tive tasks as well as repet­i­­­tive.

    • Says in the docs "U­nix cron of­ten needs a sep­a­rat­ed at dae­­­mon to ex­e­­­cute one-­­­time-job­s. This is noth­ing more than a de­sign prob­lem in cron." which means the au­thor is not brain­dead.

    • Se­cure: no SUID crap.

    Dis­­ad­­van­­tages:

    • Can't ful­­­ly em­u­late cron, even ig­nor­ing syn­­­tax

    • Runs a copy of the dae­­­mon for each us­er who needs it (but it's smal­l­).

    • Last up­­­­­dat­ed in 2004. Could mean it's sta­ble, could mean it's aban­­­doned.

  • up­­s­tart

    • Maybe some­­­day, the cron/at re­­­place­­­ment fea­­­ture is planned, at least.

  • launchd

    • In­­­trigu­ing but I can't find enough in­­­­­for­­­ma­­­tion (or a Lin­ux port)

  • Sev­er­al batch sched­ul­ing sys­tem­s.

    • Ori­en­t­ed gen­er­al­­­ly to­wards re­­­plac­ing at and run­n­ing tasks with con­trolled en­vi­ron­­­ments.

This will not be an easy ride. Since your dis­tro is cur­rent­ly planned around at and cron, when you in­stall sot­ware they will al­so in­stall cron job­s. If you switch com­plete­ly to a non-­com­pat­i­ble sys­tem, then many tasks will sim­ply not be ex­e­cut­ed, and your sys­tem will rot.

And re­al­ly, there are a bazil­lion pack­agers and soft­ware writ­ers and dis­tro man­agers and get­ting them all to aban­don cron/at is not go­ing to hap­pen soon.

So, maybe we could have...

A reasonable replacement (a proposal)

Do a rea­son­able sched­uler (some­thing like usched­ule) and add com­pat­i­bil­i­ty lay­er­s.

  • Cre­ate a dae­­mon. One that does­n't run as root (like bcron's or usched­ule's).

    That dae­­mon will take care of run­n­ing the tasks.

  • Add an in­­ter­­face, one that makes sense, like a DB of sched­uled tasks, with a 21st cen­­tu­ry syn­­tax and ca­­pa­­bil­i­ties (ie: not cron's). The in­­ter­­faces men­­tioned be­low are sim­­ply tran­s­lat­ed in­­­to this.

    This is the pre­­ferred in­­ter­­face. The oth­­ers are just com­­pat­i­­bil­i­­ty lay­er­s. Say that a lot. Give the ad­mins nice we­b-based GUI-based and CLI-based in­­ter­­faces to this.

  • Add to that dae­­mon a Vix­ie-cron com­­pat­i­ble in­­ter­­face (ie, crontab com­­mand, /etc/crontab, /etc/cron.d). Again, bcron al­ready has this, usched­ule does­n't (sad­­ly). Maybe hack some­thing out of both?

  • Im­­ple­­ment an at com­­mand that sched­ules tasks against that dae­­mon. Do­ing this for usched­ule should be pret­­ty sim­­ple, spe­­cial­­ly if you ig­nore most of the crazy at syn­­tax.

It should not be ter­ri­bly hard to do. But we won't see it any­time soon.

Pablo Antonio / 2007-07-29 17:49:

Roberto: Muy buen artículo. Sigue imprimiéndome la idea de "no uses lo que usan todos sólo porque sí; preguntate si son las mejores herramientas" que ya me impusiste cuando presencié tu charla en la CaFeConf, y eso es bueno.

Algo para criticar. Quizás le resulta ćomodo a los demás, pero a mí, como fiel lector de tu blog, me molesta de sobremanera la ventana de "snapshots" que aparece al pasar el mouse por los links.

Saludos, y gracias por seguir compartiendo tus conocimientos.

Roberto Alsina / 2007-07-29 21:49:

Me parece que la voy a sacar. Gracias por el comentario!

Brammeleman / 2007-07-30 14:50:

Thanks for your great article!
I am working on internet cafe managing software for Linux. At first I tried to use the standard tools like cron and at as much as possible to perform scheduled logouts, messaging etc. I've encountered a lot of problems using cron and at. Especially the 'feature' that grabs the environmental settings and try to mimic the sircumstances when the scheduled commands actually execute.
I've ended up with a home brewed solution where all the events are stored in a mysql database. There's one script that (initiated by cron) executes every minute to query the database for any scheduled tasks were execution time < now(). With that system I've crontrol over the environment variables no matter who puts in the new tasks in the database.

Christopher Browne / 2007-07-31 23:26:

There is a need for a "free software" alternative to the bigger guys' systems; anyone accustomed to mainframe systems will have *wildly* higher expectations than the very primitive things that cron offers.

Notably, things like
- Logging results ("in email" doesn't count)
- Ability to manage jobs across multiple hosts so that I don't have to log onto 50 servers to look at the crontab on each one.
- Some notion of dependancies; after completing job A, run job B, then job C, and so forth
- Some degree of logic; if job A runs OK, run B next, otherwise run C next
- Some awareness of system state so that if I'm doing maintenance on a host, jobs will get deferred
- Ability to write scripts that manipulate the queues

For doing cross-host work, it only makes sense to use a proper database system, as opposed to hacking something up using hash tables or such. The OSS database offering proper temporal types and such is PostgreSQL, of course.

Pete Cervasio / 2007-08-01 01:21:

Doing something on the last Monday of the month isn't too hard (though it did take me about 10 minutes to get to this version of it). Check today's month against a week from today. When they're different, it's time to run.

0 3 * * * [ $(date +%m) -ne $(date -d '+7 days' +%m) ] && run_the_script

Roberto Alsina / 2007-08-01 03:30:

Pete: almost there!

It should be 0 3 * * 1

Pete Cervasio / 2007-08-01 04:42:

DOH! You are completely right, although I really had meant to put that in. I sometimes type three splats without even thinking about it... Which reminds me what a good thing peer review is. :)

Thanks for catching that. It's what I get for being so smug after asking a co-worker how he'd do the same thing. :) :) :)

Best regards!

WindRider / 2008-06-09 14:19:

The reason why I hate cron, is because when a cron job starts up, the cron job's environment is NOT the same as your own.

In particular, your startup file isn't source'd, so the PATH isn't the same. The reason why this is so terrible, is that "non-standard" programs like ruby and perl get put into sometimes /usr/bin or /usr/local/bin or ...

which means if you ever move your ruby or perl scripts, you're constantly tweaking the paths.

Another thing to hate:
For reasons I don't understand, SOMETIMES, errors get sent to the email of the person running the code and sometimes it doesn't.

The third thing I hate, is if the command line is TOO long, cron silently fails.


Contents © 2000-2024 Roberto Alsina