MRC at CAC.Washington.EDU
Fri Sep 28 14:01:14 PDT 2007
On Fri, 28 Sep 2007, Jeroen van Aart wrote:
>> Any potential failure for an IMAP server is a potential failure for
>> an NFS server, so there is no point trying to cover the former by using
>> the latter. If anything, it makes the problem much worse.
> Yes I agree, but also looking at recent discussion on one of the imap lists
> and judging from the documentation, NFS doesn't work well with imap. Due to
> it's lack of certain features imap needs.
There are two independent issues here.
The first is that, as Joel Reicher points out, NFS doesn't really add any
benefit. It just moves the single point of failure from the IMAP server
to the NFS server, at the cost of considerable additional complexity.
The attractive idea is that you set up a cluster of IMAP server machines
talking to the NFS Server From Hell (NSFH), and if one IMAP server machine
crashes then there's fall through to another IMAP server machine. The
problem is that there is NO REASON why an IMAP server machine should be
any more likely to crash than the NSFH.
The next attractive notion is to have a single IMAP Server From Hell
(ISFH); but then you discover that the user load scales to a greater point
than the ISFH can handle. So you think back about an NSFH, until it
finally dawns on you that since IMAP load is primarily I/O load, the NSFH
is going to fail to scale for that load just as badly as the ISFH fails.
It saddens me that, after 20 years, I still have to explain the same thing
over and over again. I have little doubt that certain salecritters push
the idea of NFS as Your One And Only Solution To Your Scaling And
Reliability Needs. Boys and girls, repeat after me: "salescritters lie."
The second issue is that most mail stores do NOT work well with NFS
Of the formats supported by UW imapd, only traditional UNIX format and
MMDF (a rarely-used variation of traditional UNIX format) stand a chance
of working with NFS, and only in a limited mode.
Traditional UNIX format only allows one session to have the mailbox open
at a time. With local file access, a "kiss-of-death" protection is in
place that will cause a new session to kill a previous session that has
the mailbox open.
When NFS is used, there is no kiss-of-death. What presently happens is
that one session will make a change without any synchronization with the
other sessions. Assuming that all goes well, the other sessions will see
that this has happened and crash. If all doesn't go well, the other
sessions will be unaware of the change, and may start returning random
data for message texts and/or corrupt the mailbox.
The other formats, most notably mbx and mix, allow multiple session access
and synchronize their operations across those sessions. This, in turn,
requires that the filesystem do certain operations in an atomic fashion.
This is the case with local files, but is not the case with NFS. Read all
the old papers about NFS statelessness, and how the designers decided that
it wouldn't be a problem since nobody would ever try using NFS with
applications which need atomic filesystem operations...
My understanding is that the mailbox format used by Cyrus imapd from CMU
has the same restriction.
The maildir format used by Dovecot (and the broken Courier server) tries
very hard to work over NFS, but it doesn't quite solve all the NFS
problems (there's a writeup on Wikipedia on that). Furthermore, Maildir
was never designed to work with IMAP, and consequently has severe
performance issues if you try to do IMAP with "vanilla" maidir. Both
Dovecot and Courier have an add-on metadata file to make maildir work
better with IMAP, at the cost of being non-standard with DJB-defined qmail
and some (if not complete) loss of compatibility with NFS.
Hence, this is why we all say "don't use NFS with IMAP".
-- Mark --
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
More information about the Imap-use