[Imap-protocol] Should IMAP APPEND create a sent or unsent message?

[Mark Crispin]

On Thu, 31 Jan 2008, John DeMillion wrote:

> The question that I'm dealing with is how the mobile vendor's

> already-sent-by-SMTP message is represented in the user's groupware

> system.  The mobile vendor wants to create the illusion of an

> already-sent message via an IMAP APPEND, but the way that my groupware

> vendor is implementing APPEND breaks the illusion.


There is no IMAP-assigned representation of "already-sent-by-SMTP".  It is 
a concept solely within the client software.

IMAP provides a mechanism, called message flags, by which status can be 
set (or cleared) on a per-message basis.

Flags which start with "\" are called "system flags" and are defined by 
the IMAP specification; clients can not add system flags that are not 
defined in the specification.  These flags are: \Deleted, \Flagged, 
\Answered, \Seen, \Recent, and \Draft.  Of these, \Recent is a read-only 
flag to clients; and \Seen and \Deleted have interactions with other IMAP 
commands.  System flags are mandatory-to-implement in all IMAP servers.

Flags which do not start with "\" are called "keywords" and can be created 
and defined by clients.  A convention has arisen over the years in which 
keywords that start with "$" are defined and used by software and other 
keywords are defined by the human user.  Keywords are 
optional-to-implement in IMAP servers, but good quality IMAP servers 
implement them.

Thus, it would be reasonable for a client to define a keyword called $Sent 
to make "already-sent-by-SMTP".  But nothing (other than common sense and 
courtesy, which sadly tends to be lacking in today's world) prevents some 
other other from using that same name to mean "stinking excrement, not 
tasty".


> It seems that the \Draft flag, or at least my understanding of it,

> might have some relevance here.


\Draft is a wart in the IMAP specification; it never should have happened 
and it wouldn't happen today.  It came about only because a couple of 
client implementors wanted a flag for draft messages and were afraid to 
use $Draft because keywords are not mandatory-to-implement.

Most clients today use a Drafts mailbox instead of the \Draft flag; and 
there is considerable use of other $ keywords so servers that do not 
implement keywords can not be taken seriously in the IMAP world.


> The mobile software vendor is *not*

> using the \Draft flag with the APPEND command, but it seems that the

> groupware vendor is treating the APPENDed message as such:  the

> resulting message in the groupware system is user-changeable and

> appears essentially in "saved draft" form.  The user could modify any

> part of it and then hit "Send" and it would go out via local SMTP

> again.  In the words of the RFC, the appearance of the APPENDed message

> to my groupware user is that the "message has not completed

> composition".


Although IMAP defines all system flags, there is no enforcement on the 
part of IMAP to any special semantics to \Draft (nor, for that matter, to 
\Flagged or \Answered).  Since I do not know what the clients in question 
are doing (or expecting), I can't go into specifics; but the bottom line 
is that this is a client software issue and not an IMAP issue.

I should point out that most mobile client implementors have demonstrated 
remarkable cluelessness about IMAP; you need to take what they say with a 
rock of salt.  There is a good deal of "I don't understand any of this, so 
I'll just copy the examples and do what I think sounds good" within that 
cadre of developers.  Similarly, groupware has historically not been 
particularly well-regarded among IMAP servers.


> There seems to be an inherent conflict there:  the very existence of

> the \Draft flag defined by the IMAP protocol seems to differentiate

> "sent mail" (no \Draft) from "drafts" (using \Draft)....the IMAP

> protocol does indeed seem to say something on the topic.  Is it logical

> that the lack of \Draft mean "not Draft", therefore meaning "completed

> composition" and therefore not changeable?


It means no such thing.  \Draft simply means that some client, for 
whatever reason, chose to apply that flag.  There is no magic property of 
\Draft.

Let me emphasize that ALL messages in IMAP are immutable.  Any message can 
be downloaded to the client, modified on the client, and the modified 
version uploaded via APPEND -- but that uploaded version is a *new* copy 
of that message (and similarly immutable).


> With this in mind, would a reasonable/correct interpretation be:

>     *   an "APPEND \Draft" creates a message that has the *appearance*

> to the user of being an unsent, saved draft message, allowing user

> changes (as my groupware vendor is doing with all APPENDs, regardless

> of the flags)

>     *   an unflagged "APPEND" creates a message with the *appearance*

> to the user as having already been sent, just as if they sent it using

> their groupware client, totally unchangeable (as my mobile vendor

> intends)


These are semantics that may be meaningful in the groupware context, but 
have no meaning elsewhere.  You can't expect other software to obey (or 
even know) about groupware's sematnics.


> Getting past my n00b interpretations, I'm curious what the production

> behavior of APPEND is (with and without the \Draft flag) in other

> groupware/mailsystems that the obviously experienced list members are

> knowledgeable about.


APPEND (and the more advanced CATENATE in Lemonade) is simply a mechanism 
to upload messages.  There is nothing about a message being APPENDed that 
is in any way special from one that was delivered via SMTP.

The most common use of APPEND is as part of an operation to move messages 
to an IMAP server from a local disk or other IMAP/POP/NNTP server.  Some 
clients also use APPEND to save message drafts being composed; but as 
noted above these messages are usually put into a special named mailbox by 
themselves rather than using the \Draft draft in a mailbox used by other 
messages.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.