[Alpine-info] bug reports

[Mark Crispin]

On Wed, 7 May 2008, Mike Miller wrote:

> Is mailutil available on an open-source license?  It is surprisingly hard to 

> find the mailutil source code given how easily I can find a mailutil man 

> page.


mailutil is distributed as part of the UW IMAP toolkit.  The official 
distribution including all sources is (and has been for the past 18+ or so 
years):
 	ftp://ftp.cac.washington.edu/mail/imap.tar.Z

I assume, if you have a mailutil binary and man page but not the source, 
that you got this software from a third party distribution that split the 
package up.

Please understand that any third party distribution is unsupported by me; 
I don't know how any third party distribution was packaged, nor how it may 
different from the official distribution.  If you have any issues that I 
can not reproduce in the official distribution, my standard answer will be 
for you to try the official distribution.


> I guess I don't knwo what metadata is.  The "From " line contains very little 

> information:  It looks like just "From:" and "Date:".


Although it does not use the term "metadata", the IMAP specification 
defines various data items associated with each message, and with the 
mailbox as a whole.  None of this data is actually part of any message. 
There are quite a few of these:

per-message	INTERNALDATE (date/time message was delivered)
 		FLAGS (status flags associated with the mailbox:
 		 pre-defined "system flags" such as "message has been
 		 read", "message is deleted", "message is answered",
 		 etc.; and user-defined "keyword flags")
 		RFC822.SIZE (size of the message in octets using CR/LF
 		 newlines)
 		UID (unique identifier for this message)

mailbox		FLAGS (list of all system and keyword flags defined in
 		 this mailbox)
 		UIDVALIDITY (validity value for the UID regime; must be
 		 changed if some issue causes the mailbox to reassign
 		 UIDs)
 		UIDNEXT (next UID to be assigned)

There are other data items which are arguably metadata (such as the number 
of messages in the mailbox, or the results of parsing headers and MIME 
structure).  Various extensions to the IMAP protocol also define 
additional metadata.

The traditional UNIX mailbox format's "From " line contains only the 
INTERNALDATE.  It also contains the SMTP return-path (not the actual From 
address).  What this means is that the mailbox metadata has to be stored 
in a hidden message (that funny "DO NOT DELETE" message at the start of 
the mailbox) and/or a hidden header in the first message; and the 
per-message metadata has to be stored in various hidden headers in the 
messages.  It's a mess.


> So why can't I 

> recreate a reasonable facsimile of the original mbox by dumping metadata 

> lines and inserting a "From " line?


You can indeed do that if you insist.

The procedure is to take each data file (".mix????????" where "?????????" 
are 8 hexadecimal digits) and changing the metadata lines (which start 
with ":msg:") to "From " lines.  You need the third field to construct the 
"From " line date/time and the fourth field to know how large the message 
is and thus where the next ":msg:" line is.  That is, given a ":msg:" line 
of:
 	:msg:00014fc2:20070516181739-0700:00000e7e:
The "From " line to construct is something like:
 	From converted Wed May 16 18:17:39 -0700
(note that the ":msg:" line doesn't say what day of week May 16, 2007 was 
so you have to calculate that yourself).  Then copy the next 3710 bytes 
from the file, converting all CR/LF to LF only, and making sure that you 
insert a ">" in front of any line that starts with "From " and looks like 
a traditional UNIX "From " line.

Easy as pie.  ;-)

Or you can use my mailutil program, which does all of this work for you 
and does it right.  Why make more work for yourself.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.