[Imap-uw] Simultaneous access to imap mailbox?
mrc at CAC.Washington.EDU
Sun Sep 16 14:57:58 PDT 2007
On Sun, 16 Sep 2007, John Kelly wrote:
> Can you speculate on how mix compares to mbx, in terms of I/O load?
> Seems like I remember you saying mix does not need atime.
In terms of data volume, mix's I/O load is comparable to mbx.
mix wins big in reduced seek times when reading message metadata,
including synchronizing flag updates may by another simultaneous session.
mbx needs to do a "seek to message, read 64 bytes" for each message in the
mailbox. For new messages, mbx will start at the previous end of file;
but flag updates requires doing all those seeks and reads.
mix needs only to read an index file and a status file. If the first
record of either/both file(s) indicates no change since the last read, it
doesn't bother reading any more.
mix caches message data needed for sort, so it is possible to sort/thread
a mailbox without having to parse messages which are in the sortcache
(those messages which aren't in the sort cache and have to be parsed have
their data added to the sortcache).
mix currently rewrites the entire index file when adding new messages or
expunging. Adding new messages is slower in mix. Under certain
circumstances, expunge in mix may be slower than mbx; it depends upon how
many messages are slid in the mix data file vs. the flat mbx file.
In my opinion: The add-new-messages slowdown is not significant. The
cases where mbx is faster in expunge are the cases in which expunge time
is not significant either way. For example, given a 10,000 message
mailbox, mbx will expunge message 9999 faster than mix; but either way it
isn't a big deal. Conversely, mbx will expunge message 1 out of 10,000
FAR slower (it's the Expunge From Hell) than mix.
mix currently rewrites the entire status file when changing flags, and
furthermore the status file update is under a mutex (unlike mbx where flag
update timing races can occur). This is quite a bit slower than mbx for a
single message. You won't notice it for just one message, but you will
notice 10,000 single message updates in mix far more than in mbx.
On the other hand, in mix an aggregate flags update has exactly the same
cost as a single message update. Aggregate updates win in mbx too,
because it doesn't do an fsync() until all the updates are applied. But
mbx still has to do a seek()/read()/write() per message.
Put another way, mix updates flags in 10,000 messages individually far
slower than mbx, but mix updates flags in 10,000 messages as an aggregate
far faster than mbx.
mix never writes data files except to append new messages to the current
data file, and to expunge mesages. Flags changes do NOT write into any
Given a buffer cache large enough to accomodate the entire mbx file, mbx
works quite well. However, in this day and age of 1GB mailboxes, it is
rare that an IMAP server has the luxury of keeping all currently-selected
mailboxes in the buffer. mbx falters when it is not in the buffer cache,
and falters big time when it loses its space in the buffer cache.
mix has far lower demands of the buffer cache. It also mutexes flag
updates, providing more consistent results, at the cost of a trap if the
application does many successive single-message flag updates (note the
recent discussion about the POP3 server).
It is conceivable that a 64-bit system with multiple terabytes of RAM
allocated to the buffer cache could perhaps support huge mbx mailboxes
faster than the equivalent mix mailbox. Try it and tell me how it works
Unlike mbx, mix doesn't tweak file times and doesn't have large files that
change all the time. Your backup system maintainer will thank you for
choosing mix for that reason alone.
-- Mark --
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
More information about the Imap-uw