[Imap-protocol] IMAP MOVE extension

[Witold Kręcicki]

On Saturday 12 of June 2010 16:58:17 Timo Sirainen wrote:

> On 12.6.2010, at 8.58, Witold Kręcicki wrote:

> >> So let's say that we punt and say that an error is an error and we'll

> >> break the all-of-nothing guarantee of IMAP.

> > 

> > This guarantee is a myth that is not working in eg. Maildirs.

> > The same problems that appear with COPY or EXPUNGE in those critical

> > situations but nobody is thinking about removing COPY or EXPUNGE from the

> > protocol.

> 

> Since you keep talking about maildirs..:

> 

> Assumptions: Maildir can be accessed mostly without locks. Reading mails

> should never get any locks, and that's also how Dovecot works. Changing

> flags or expunging messages would be possible to do without locks,

> although currently Dovecot doesn't. Assigning UIDs to new messages (i.e.

> making new messages visible) pretty much requires some kind of locking.

> 

> The way I've implemented COPY is that it first link()s files to the

> destination Maildir/tmp/ directory. Then it locks the maildir, assigns

> UIDs to messages, rename()s files from tmp/ to new/ and unlocks. Messages

> won't be visible to other connections until unlock is complete. This means

> that as long as the system doesn't crash, there is a single atomic event

> (unlock) when the messages either become visible or the copy operation

> gets rollbacked. This can fail only if there are non-Dovecot MUAs

> accessing the maildir that aren't using Dovecot's locking, but in normal

> Dovecot-only situations this works perfectly.

> 

> Now, if you wanted to implement MOVE operation using rename(), there is a

> problem: Immediately after you do a rename() the message is gone from the

> source mailbox. If another session tries to open the mail (remember, no

> locks while reading), or for some other reason scans the maildir for what

> messages it contains, it sees that the message is gone and issues EXPUNGE

> to client. There is no way to revert that! And why would the MOVE

> operation fail? 1) COPY fails if one of the messages has already been

> expunged, so MOVE should probably too, and this won't be known beforehand.

> 2) Filesystem quota can cause rename() to fail if it would increase the

> destination directory's size.

> 

> So the difference is that COPY can under normal operating conditions work

> as intended, while MOVE implemented with rename() can't.

Maybe MOVE implemented with rename() can't, but move implemented with link & 
delete (so 'internally' as copy + expunge with a single lock) can.


> > What if the server deletes 3 out of 10 messages and then is struck by a

> > filesystem failure (cannot delete remaining 7, cannot undelete of the

> > first 3)? This situation is (formally) not covered in the protocol,

> > server cannot send untagged EXPUNGE responses for those 3 messages

> > (because then it would have to send OK) and it cannot send OK because

> > EXPUNGE after a success removes ALL messages with \Deleted flags...

> 

> I don't think that's a correct assumption.

>

> > It should send NO but then the client would

> > think that no messages were deleted permanently...

> 

> I don't think that's a correct assumption either.

> 

> My server replies to EXPUNGE always with OK, even when there are \Deleted

> messages that weren't expunged because ACLs didn't allow that (it used to

> send NO, but some clients tried to send EXPUNGE internally and kept giving

> popup windows saying the EXPUNGE failed). Clients should use the untagged

> EXPUNGE replies to figure out what happened, the OK reply from EXPUNGE

> doesn't tell anything.

"The EXPUNGE command permanently removes *all* messages that have the \Deleted 
flag set from the currently selected mailbox.". It is not my assumption, it's 
stated in the protocol.

-- 
Witold Kręcicki

Grupa o2 Spółka z o.o., ul. Jutrzenki 177, 02-231 Warszawa,
KRS 0000140518, Sąd Rejonowy dla m.st. Warszawy, 
Kapitał zakładowy 377.298,00 zł., NIP 521-31-11-513