[Imap-uw] Simultaneous access to imap mailbox?

[Mark Crispin]

On Sun, 16 Sep 2007, John Kelly wrote:
> Can you speculate on how mix compares to mbx, in terms of I/O load?
> Seems like I remember you saying mix does not need atime.

Correct.

In terms of data volume, mix's I/O load is comparable to mbx.

mix wins big in reduced seek times when reading message metadata, 
including synchronizing flag updates may by another simultaneous session.

mbx needs to do a "seek to message, read 64 bytes" for each message in the 
mailbox.  For new messages, mbx will start at the previous end of file; 
but flag updates requires doing all those seeks and reads.

mix needs only to read an index file and a status file.  If the first 
record of either/both file(s) indicates no change since the last read, it 
doesn't bother reading any more.

mix caches message data needed for sort, so it is possible to sort/thread 
a mailbox without having to parse messages which are in the sortcache 
(those messages which aren't in the sort cache and have to be parsed have 
their data added to the sortcache).

mix currently rewrites the entire index file when adding new messages or 
expunging.  Adding new messages is slower in mix.  Under certain 
circumstances, expunge in mix may be slower than mbx; it depends upon how 
many messages are slid in the mix data file vs. the flat mbx file.

In my opinion: The add-new-messages slowdown is not significant.  The 
cases where mbx is faster in expunge are the cases in which expunge time 
is not significant either way.  For example, given a 10,000 message 
mailbox, mbx will expunge message 9999 faster than mix; but either way it 
isn't a big deal.  Conversely, mbx will expunge message 1 out of 10,000 
FAR slower (it's the Expunge From Hell) than mix.

mix currently rewrites the entire status file when changing flags, and 
furthermore the status file update is under a mutex (unlike mbx where flag 
update timing races can occur).  This is quite a bit slower than mbx for a 
single message.  You won't notice it for just one message, but you will 
notice 10,000 single message updates in mix far more than in mbx.

On the other hand, in mix an aggregate flags update has exactly the same 
cost as a single message update.  Aggregate updates win in mbx too, 
because it doesn't do an fsync() until all the updates are applied.  But 
mbx still has to do a seek()/read()/write() per message.

Put another way, mix updates flags in 10,000 messages individually far 
slower than mbx, but mix updates flags in 10,000 messages as an aggregate 
far faster than mbx.

mix never writes data files except to append new messages to the current 
data file, and to expunge mesages.  Flags changes do NOT write into any 
data file.


 	Bottom line:

Given a buffer cache large enough to accomodate the entire mbx file, mbx 
works quite well.  However, in this day and age of 1GB mailboxes, it is 
rare that an IMAP server has the luxury of keeping all currently-selected 
mailboxes in the buffer.  mbx falters when it is not in the buffer cache, 
and falters big time when it loses its space in the buffer cache.

mix has far lower demands of the buffer cache.  It also mutexes flag 
updates, providing more consistent results, at the cost of a trap if the 
application does many successive single-message flag updates (note the 
recent discussion about the POP3 server).

It is conceivable that a 64-bit system with multiple terabytes of RAM 
allocated to the buffer cache could perhaps support huge mbx mailboxes 
faster than the equivalent mix mailbox.  Try it and tell me how it works 
out... ;-)

Unlike mbx, mix doesn't tweak file times and doesn't have large files that 
change all the time.  Your backup system maintainer will thank you for 
choosing mix for that reason alone.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.