It should not hap­pen but it does: Your qmail serv­er is over­load­ed. Maybe you are un­der a DOS at­tack, or there is a rea­son why you are get­ting 10x your usu­al amount of mail.

But then you start see­ing how your "not pre­pro­cessed" queue starts grow­ing, and grow­ing...

This can al­so mean things like cla­mav or spa­mas­sas­s­in, which need to check the mail be­fore it gets queued are not keep­ing up with the mail flow, or maybe some IO per­for­ma­ce is­sue.

But what can you do righ now to fix it?

Well, you can dis­able spa­mas­sas­s­in, or, in ex­treme cas­es, shut­down SMTP so the sys­tem has a chance to catch its breath so to speak.

Of course, clos­ing SMTP means your own users can't send email ei­ther, which suck­s.

Now there is a lighter al­ter­na­tive: shut­down SMTP for those who are not your user­s.

Here's the triv­ial code, im­ple­ment­ed as a SPP plug­in, fit to be used in the [mail] sec­tion:

#!/bin/dash

if [ -f /var/qmail/control/overloaded ]
then
      if [ -z "$SMTPAUTHUSER" ]
      then
              echo R451 Temporary Failure: Server overload
              echo overload: $PPID Temporary Failure: Server overload >&2
      fi
fi

And if you are dar­ing and want to make your sys­tem self­-­cor­rect­ing, maybe you should cron some­thing like this:

* * * * * if [ `qmail-qstat  | tail -1 | cut -d: -f2` -gt 100 ];\
then touch /var/qmail/control/overloaded ;\
else rm -f /var/qmail/control/overloaded; fi

I will prob­a­bly code it again in C and make it part of ra/­plu­g­in­s.

• If you don't know what qmail-spp is, please check it out. It makes qmail much much bet­ter.

• If you know qmail-sp­p, then maybe my plug­in col­lec­­tion will be handy for you.

• My most use­­ful plug­in is prob­a­bly ipthrot­tle, which you can use to make overea­ger IPs con­nect less of­ten.

• The ver­­sion cur­ren­t­­ly in SVN will au­­to­block those IPs for a con­­fig­urable amount of time if you are us­ing ipsvd which is like tcpserver, on­­ly much bet­ter.

• I re­al­­ly need some­one to help me test the SVN ver­­sion, which should be way, way bet­ter than the re­leas­es on the page.

• The SVN re­po is at google­­code

I am hack­ing a bit on rater my dae­mon/­client to see if things are hap­pen­ing more of­ten than they should (in oth­er word­s, gener­ic rate lim­it­ing).

I had to take a few days of­f, since my broth­er got mar­ried and we all went back to San­ta Fe for that and a week­end, and then ev­ery­one else has sore throats and I am the on­ly one healthy.

But hey, it works well enough al­ready:

• The sim­­plis­tic pro­­to­­col is done

• The serv­er works

  • It can take hours of gib­ber­ish with­­­out prob­lem­s.

  • It can take hours of valid in­­­put with­­­out prob­lem­s.

  • It does what it's sup­­­posed to do.

• It's stay­ing be­low 300S­LOC, which was my goal.

Miss­ing stuff:

• Val­­grind it.

• Client li­brary.

• Gen­er­ic CLI clien­t.

• A qmail-spp plug­in that us­es it.

And then, I can for­get all about it.

Mon­day was a very spe­cial day:

• Hol­i­­day (In­de­pen­­dence day)

• An­niver­sary (3 years as Rosar­i­o's boyfriend)

• The first snow­­fall in Buenos Aires in 89 years.

Be­sides that, this week my broth­er is get­ting mar­ried so the whole fam­i­ly (in­clud­ing 2.5 mon­th-old JF) is leav­ing for my an­ces­tral lands to­mor­row.

And I start­ed a new small pro­jec­t, wh­cih should be fin­ished soon.

This is some­thing that seems use­ful to me in the con­text of mail server­s, but maybe it will al­so find its us­es else­where.

I call it rater, and it tells you if things are hap­pen­ing faster than a spe­cif­ic rate.

For ex­am­ple, I in­tend to use it to fig­ure out if a spe­cif­ic IP is con­nect­ing to a serv­er more than X times ev­ery Y sec­ond­s, or if a us­er is send­ing more than Z emails ev­ery T min­utes.

The on­ly thing I found for this is re­lay­d, which is old, un­main­tained and whose site has van­ished.

The con­fig file is some­thing like this (thanks to lib­con­fig):

limits : {
      user: (
                    ("rosario",90,20),
                    ("ralsina",90,10),
              ("*",2,10)
              );
      ip:   (
                    ("10.0.0.*",90 , 20),
                    ("10.0.1.*",90 , 20),
                    ("*",2 , 10)
              );

};

You can de­fine as many class­es of lim­its as you want (that would be ip and us­er in this ex­am­ple) and as many lim­it keys as you wan­t, that will be matched us­ing some­thing like fn­match.

I am us­ing an in­-mem­o­ry SQLite DB for the ac­count­ing, and an in­ter­est­ing li­brary called libut for the sock­et­s, log­ging, and event loop.

This li­brary has a very in­ter­est­ing fea­ture: your app gets an ad­min­is­tra­tive in­ter­face for free!

[ralsina@monty rater]$ telnet localhost 4445
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.
libut control port interpreter
Type 'help' for command list.

help
command           description
----------------- -------------------------------------
* mem             - memory pool usage summary
* var             - Display or set config variables
* log             - Change log file or verbosity
  fds             - list selected file descriptors
  tmr             - show pending timers
  uptime          - show uptime
* prf             - Performance/profiling stats
* cops            - List coprocesses
  help            - view command help
  exit            - close connection

Commands preceded by * have detailed help. Use help <command>.

Ok
var
 name                 description                    value
--------------------- ------------------------------ --------------------
*ut_log_level         log level                      Debugk
*ut_log_file          log file                       /dev/stdout
*ut_jobname           job name                       job1
*ut_control_port      control port IP/port           127.0.0.1:4445
*ut_basedir           shl base directory             /mnt/centos/home/ralsina/Desktop/proyectos/rater

Variables prefixed with '*' can be changed.

Ok
var ut_log_level Debug

Ok
var
 name                 description                    value
--------------------- ------------------------------ --------------------
*ut_log_level         log level                      Debug
*ut_log_file          log file                       /dev/stdout
*ut_jobname           job name                       job1
*ut_control_port      control port IP/port           127.0.0.1:4445
*ut_basedir           shl base directory             /mnt/centos/home/ralsina/Desktop/proyectos/rater

Variables prefixed with '*' can be changed.

Ok

Pret­ty neat.

Be­yond this, there will be a small clien­t-­side li­brary that hides all the net­work stuff be­hind a cou­ple of block­ing calls (or you can do your own be­cause the pro­to­col is sil­ly sim­ple).

I am us­ing ra-­plu­g­ins as a toy to do things I nev­er both­ered in oth­er project­s.

I am do­ing unit-test­ing. And now... some lit­er­ate pro­gram­ming!

Ok, not much, and not very well, but at least I am play­ing with Lp4all which is a nice, sim­ple tool to gen­er­ate nice HTML from slight­ly wik­i-­marked sources.

You can see some lit­tle things in my code here. My vere­dict so far? A nice way to keep the code doc­u­ment­ed in a fash­ion that ocasi­nal browsers can fol­low.

The main thing miss­ing is au­to­mat­ic cross-ref­er­enc­ing.

In gen­er­al, I am find­ing that this (and unit test­ing) helps me ex­press ex­plic­it­ly to my­self what the heck I am try­ing to do, and see if the code ac­tu­al­ly does it. Which is a re­al­ly good thing.