[Alpine-info] Very slow IMAP CHECK

[Mark Crispin]

On Mon, 16 Mar 2009, Chris Wilson wrote:

> I believe that this is due at least in part to the IMAP CHECK command,

> which Alpine seems to run quite regularly. My IMAP server, Dovecot, takes

> 40 seconds to checkpoint my mailbox.


The purpose of IMAP checkpoints is to allow a sever to defer necessary, 
but potentially time-consuming, maintenance tasks to a checkpoint, instead 
of per command.  The SELECT, EXAMINE, EXPUNGE, LOGOUT, CLOSE, and CHECK 
commands all do a checkpoint.  In the case of CHECK, that is all that is 
done.

Note that word "necessary".  Checkpoint activity includes such "optional" 
behavior as saving flag changes in traditional UNIX format.  The VERY SAME 
people who complain about checkpoints in traditional UNIX format (and 
demand that checkpoints be removed) are the ones who scream bloody murder 
when their work isn't saved.

The whole point of the IMAP CHECK command is to give the client some say 
as to when in-session checkpoints happen.  Otherwise, one of two bad 
things will happen with a long-lived session:
  [1] checkpoints never happen, and at some point a LOT of work is lost.
  [2] the server unilaterally decides to checkpoint, regardless of whether
      or not it is a good time for the client or user.

In the case of maildir, an endemic weakness in maildir (especially when 
the mailbox becomes very large) is that messages can go missing. 
Dovecot's rescan finds the lost messages and brings them back into the 
fold.  If you don't mind messages vanishing, perhaps forever, then you may 
want to disable checkpoints.  Otherwise, it's a BAD idea.

The church of maildir called me numerous vicious names when I said that 
maildir had scalability problems.  Supposedly, I was ignorant and stupid 
and evil and etc.  Your problem reminds me of H.G. Wells' epitaph:
 	"I told you so.  You DAMNED fools."

In this case, it applies both to using maildir and to disabling 
checkpoints!

If you modify Alpine in this area at all, I recommend that your 
modification take the form of making the decision to checkpoint be 
smarter.  That is, instead of removing checkpoints or making them happen 
less often, Alpine should look for better opportunities to get in a 
checkpoint.  Extended user inactivity points to a very obvious time to do 
a checkpoint.

Also, now that Alpine has threading, it would be an excellent idea if the 
composer would do the various mailbox background tasks for the mailbox in 
a separate thread rather than blocking the user interface in the composer, 
and then just have a mutex over any non-composer command.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.