[Imap-uw] Re: moving mail between folders is intermittently
tss at iki.fi
Mon May 3 09:09:20 PDT 2010
On Mon, 2010-05-03 at 08:32 -0700, Mark Crispin wrote:
> For what it's worth: Timo is the author of Dovecot. His comments about
> its implementation should be considered authoritative. Mine are based
> upon memory and second-hand/third-hand information.
> What is important - and what I will/do comment upon - is whether or not
> another server is compliant with the specification. Dovecot is compliant.
> I guess that the threaded semaphores stuff was in Communigate Pro. Linda
> comment about threading obviously confused me.
> So, if I read you correctly, shared mbox access doesn't communicate flag
> changes? You don't use an external index file to avoid having to re-read
> the entire file? Do you allow shared expunge?
I use an external index file to avoid re-reading entire file again and
allow shared expunge. But there are all kinds of tricks to get better
performance and ability to use non-Dovecot software to access the
1) If mbox file's mtime and size match what is stored in index, index is
assumed to be up to date and mbox file isn't even opened until
2) If mtime changes but size doesn't, assume that someone else wrote
flag changes to messages -> re-read the entire mbox file.
3) If file size decreases, assume expunged messages -> re-read entire
4) If file size increases, assume a new message was appended -> try to
read it. If the reading fails (no valid From_-line at expected offset),
re-read the entire mbox file. If reading succeeds, enable "dirty flag",
because it's not known if there could have been also other changes.
Whenever entire mbox file is re-read, the dirty flag is cleared.
5) Whenever reading a message from cached offset, verify that there's a
valid From_-line. If dirty flag is set, verify also that X-UID: header
is for expected message. If either fails, re-read the mbox file.
6) Whenever SELECTing mbox file and dirty flag is set, optionally either
re-read mbox file (default) or just open it and keep the dirty flag.
7) Writing flag changes (and other header updates) to mbox file are
optionally delayed (default), until mailbox is closed or messages are
expunged or CHECK is run. This is same as with UW-IMAP I think. If
non-Dovecot MUA changes flags during this session and Dovecot also
notices those changes (due to above checks), the changes that don't
conflict with internal unwritten flag changes are applied to index.
Also as long as Dovecot is the only thing modifying the mbox file, state
is shared via index files, so 1) check always succeeds and the
performance stays good.
As long as the only changes are appends by (non-Dovecot) MDA, only 1)
and 4) can happen and there are again no problems. If non-Dovecot MUA
does other changes, Dovecot might not always notice the changes
immediately, but it never causes corruption.
Some small details above are probably incomplete.
Filesystems are also beginning to support micro/nanosecond mtime
resolution (well, I guess everything except ext2/ext3 does nowadays), so
saving the timestamp in nanosecond resolution could also help notice
external changes more reliably. But I haven't bothered to add support
> Did you ever test it over NFS and SMB (and NFS and SMB simultaneously)?
> That's the kind of crap that I had to support when I did the code in UW
> IMAP. I hope that nobody has to support such nonsense ever again.
I don't know about SMB, but some people are using it over NFS and I
haven't heard complaints for a while. It should work pretty well as long
as fcntl locking is used, because it reliably also clears NFS caches
(hoping of course that nfs.lockd itself doesn't break).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 205 bytes
Desc: This is a digitally signed message part
Url : http://mailman2.u.washington.edu/pipermail/imap-uw/attachments/20100503/0a75bfd2/attachment.bin
More information about the Imap-uw