[Imap-use] replication

[Mark Crispin]

On Fri, 28 Sep 2007, Jeroen van Aart wrote:
>> Any potential failure for an IMAP server is a potential failure for
>> an NFS server, so there is no point trying to cover the former by using
>> the latter. If anything, it makes the problem much worse.
> Yes I agree, but also looking at recent discussion on one of the imap lists 
> and judging from the documentation, NFS doesn't work well with imap. Due to 
> it's lack of certain features imap needs.

There are two independent issues here.

The first is that, as Joel Reicher points out, NFS doesn't really add any 
benefit.  It just moves the single point of failure from the IMAP server 
to the NFS server, at the cost of considerable additional complexity.

The attractive idea is that you set up a cluster of IMAP server machines 
talking to the NFS Server From Hell (NSFH), and if one IMAP server machine 
crashes then there's fall through to another IMAP server machine.  The 
problem is that there is NO REASON why an IMAP server machine should be 
any more likely to crash than the NSFH.

The next attractive notion is to have a single IMAP Server From Hell 
(ISFH); but then you discover that the user load scales to a greater point 
than the ISFH can handle.  So you think back about an NSFH, until it 
finally dawns on you that since IMAP load is primarily I/O load, the NSFH 
is going to fail to scale for that load just as badly as the ISFH fails.

It saddens me that, after 20 years, I still have to explain the same thing 
over and over again.  I have little doubt that certain salecritters push 
the idea of NFS as Your One And Only Solution To Your Scaling And 
Reliability Needs.  Boys and girls, repeat after me: "salescritters lie."


The second issue is that most mail stores do NOT work well with NFS 
anyway.

Of the formats supported by UW imapd, only traditional UNIX format and 
MMDF (a rarely-used variation of traditional UNIX format) stand a chance 
of working with NFS, and only in a limited mode.

Traditional UNIX format only allows one session to have the mailbox open 
at a time.  With local file access, a "kiss-of-death" protection is in 
place that will cause a new session to kill a previous session that has 
the mailbox open.

When NFS is used, there is no kiss-of-death.  What presently happens is 
that one session will make a change without any synchronization with the 
other sessions.  Assuming that all goes well, the other sessions will see 
that this has happened and crash.  If all doesn't go well, the other 
sessions will be unaware of the change, and may start returning random 
data for message texts and/or corrupt the mailbox.

The other formats, most notably mbx and mix, allow multiple session access 
and synchronize their operations across those sessions.  This, in turn, 
requires that the filesystem do certain operations in an atomic fashion. 
This is the case with local files, but is not the case with NFS.  Read all 
the old papers about NFS statelessness, and how the designers decided that 
it wouldn't be a problem since nobody would ever try using NFS with 
applications which need atomic filesystem operations...

My understanding is that the mailbox format used by Cyrus imapd from CMU 
has the same restriction.

The maildir format used by Dovecot (and the broken Courier server) tries 
very hard to work over NFS, but it doesn't quite solve all the NFS 
problems (there's a writeup on Wikipedia on that).  Furthermore, Maildir 
was never designed to work with IMAP, and consequently has severe 
performance issues if you try to do IMAP with "vanilla" maidir.  Both 
Dovecot and Courier have an add-on metadata file to make maildir work 
better with IMAP, at the cost of being non-standard with DJB-defined qmail 
and some (if not complete) loss of compatibility with NFS.

Hence, this is why we all say "don't use NFS with IMAP".

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.