[Alpine-info] saving to large folders

[Kai Petzke]

Hello,


first of all: thank you very much for the work on (al)pine! A tty based 
mailer might look backwards to many, but in these times of virusses and 
other threats in e-mail, it is definitely very usefull. We even use it on 
a ssh clients on smartphones like the Nokia E90 :-)


But now to my issue: saving new e-mails to large existing folders with 
alpine. pine used to be very fast on saving messages. According to a 
respsonse from Mark Crispin to a previous post, adding UIDPLUS support to 
the general folder handler code slowed down things with mbox, and one 
should move to mbx or mix format instead.

To test things, I created a folder with approximately 200000 messages and 
1 GB size in each of the formats "mbox", "mbx" and "mix", and benchmarked 
with "alpine":
                          mbox        mbx       mix
appending a message       12s      << 1s        2s
opening folder            17s        16s        1s

The folder file(s) in question was/were in memory cache for each of these 
operations.

Most notably:
- "mbx" is MUCH faster than mbox, when appending a new e-mail at the end
   of a folder. "mbx" requires appriximately 1/10 of a second, mbox over
   10 seconds!
- When opening, "mbx" has hardly any advantages over "mbox", though.
- "mix" opens 10-fold faster than "mbx", but is 10-fold slower, when
   appending a message.

Therefore, there is currently no "best bet" for large folders. When they 
are opened often, "mix" is the winner. When they are mostly appended to 
and only opened once in a while for an old mail, "mbx" is best.


What puzzles me, is the relative slowness of "mix" in appending a new 
message. It seems, that the "index" and/or "status" file of the mix dir 
are read and/or even overwritten in their entire length, just because a 
message is added. However, it should suffice to:
- update the serial number at the beginning of the index and status file
- append the new message at the end of both
- update the meta file, but that's only a few bytes
- and of course write the message to the data file

When processing a couple hundred e-mail messages, having to wait one or 
two seconds for each of them, causes a lot of obstruction! Therefore, 
speeding up append to mix folders from 2 seconds to 0.2 seconds would be 
very helpfull. Not only for alpine users, but also for mail servers, that 
save to "mix" files.


Actually, not only saving to mix, even saving to mbox could be sped up:
- Open mbox file
- If the "FOLDER INTERNAL DATA" is found at the beginning of the file,
   the UID value found there is updated in place and used to assign an
   X-UID to the message and append it at the end of the file.
- If no "FOLDER INTERNAL DATA" is found, the message is just added, or
   the whole folder file gets UIDs assigned.
This concept has its problem, if another program appends messages to the 
mbox file without setting the X-UID header. A variant could be, that also 
the length of the total mbox file is recorded in the first header message 
(in exchange, the description text could be shortened, so that total size 
remains the same). If the length of the actual file mismatches at the 
beginning of the append, alpine or imapd would switch from fast to slow 
write.


Second issue, why I write this e-mail, is the default format when creating 
folders. I understand, it can be set by changing a variable in a Makefile. 
But that requires people to fetch and modify the source instead of using 
one of the many binary versions of alpine.

Of course, it is possible to prefix a driver spec like #driver.mbx/ to the 
folder directory. However, the folder collection specified with that spec 
can then only be used to create new folders. Opening folders for reading 
takes the driver spec as a literal path name and fails, unless one sets 
some weird symbolic links.

Therefore, it would be REALLY helpful, if in a future version of alpine, 
it would be possible to specify the default format for new mboxes along 
the other parameters of a collectionList!


Greetings


Kai Petzke
www.teltarif.de - Kommunikation ganz einfach
-- 


teltarif.de Onlineverlag GmbH
Alt-Moabit 96c, 10559 Berlin
Tel:  +49 (0)30 453 081-0
Fax:  +49 (0)30 453 081-11
Mail: mailto:info at teltarif.de
WWW:  http://www.teltarif.de

!!! Achtung wir sind umgezogen !!!
Beachten Sie bitte die neue Adresse
und die neuen Durchwahlen.

Geschäftsführer: Kai Petzke, Martin Müller
eingetragen beim Amtsgericht Berlin-Charlottenburg, HRB 70507
Umsatzsteuer-ID: DE201038407

Attraktives Werbeumfeld gefällig?
Vielfältige Targetingoptionen, eine E-Commerce-affine Leserschaft und
passgenaue Platzierungen sorgen für Ihren Werbeerfolg bei teltarif.de.
Weitere Informationen: http://www.teltarif.de/mediadaten