[Alpine-info] saving to large folders
petzke at teltarif.de
Fri Jul 11 04:43:38 PDT 2008
first of all: thank you very much for the work on (al)pine! A tty based
mailer might look backwards to many, but in these times of virusses and
other threats in e-mail, it is definitely very usefull. We even use it on
a ssh clients on smartphones like the Nokia E90 :-)
But now to my issue: saving new e-mails to large existing folders with
alpine. pine used to be very fast on saving messages. According to a
respsonse from Mark Crispin to a previous post, adding UIDPLUS support to
the general folder handler code slowed down things with mbox, and one
should move to mbx or mix format instead.
To test things, I created a folder with approximately 200000 messages and
1 GB size in each of the formats "mbox", "mbx" and "mix", and benchmarked
mbox mbx mix
appending a message 12s << 1s 2s
opening folder 17s 16s 1s
The folder file(s) in question was/were in memory cache for each of these
- "mbx" is MUCH faster than mbox, when appending a new e-mail at the end
of a folder. "mbx" requires appriximately 1/10 of a second, mbox over
- When opening, "mbx" has hardly any advantages over "mbox", though.
- "mix" opens 10-fold faster than "mbx", but is 10-fold slower, when
appending a message.
Therefore, there is currently no "best bet" for large folders. When they
are opened often, "mix" is the winner. When they are mostly appended to
and only opened once in a while for an old mail, "mbx" is best.
What puzzles me, is the relative slowness of "mix" in appending a new
message. It seems, that the "index" and/or "status" file of the mix dir
are read and/or even overwritten in their entire length, just because a
message is added. However, it should suffice to:
- update the serial number at the beginning of the index and status file
- append the new message at the end of both
- update the meta file, but that's only a few bytes
- and of course write the message to the data file
When processing a couple hundred e-mail messages, having to wait one or
two seconds for each of them, causes a lot of obstruction! Therefore,
speeding up append to mix folders from 2 seconds to 0.2 seconds would be
very helpfull. Not only for alpine users, but also for mail servers, that
save to "mix" files.
Actually, not only saving to mix, even saving to mbox could be sped up:
- Open mbox file
- If the "FOLDER INTERNAL DATA" is found at the beginning of the file,
the UID value found there is updated in place and used to assign an
X-UID to the message and append it at the end of the file.
- If no "FOLDER INTERNAL DATA" is found, the message is just added, or
the whole folder file gets UIDs assigned.
This concept has its problem, if another program appends messages to the
mbox file without setting the X-UID header. A variant could be, that also
the length of the total mbox file is recorded in the first header message
(in exchange, the description text could be shortened, so that total size
remains the same). If the length of the actual file mismatches at the
beginning of the append, alpine or imapd would switch from fast to slow
Second issue, why I write this e-mail, is the default format when creating
folders. I understand, it can be set by changing a variable in a Makefile.
But that requires people to fetch and modify the source instead of using
one of the many binary versions of alpine.
Of course, it is possible to prefix a driver spec like #driver.mbx/ to the
folder directory. However, the folder collection specified with that spec
can then only be used to create new folders. Opening folders for reading
takes the driver spec as a literal path name and fails, unless one sets
some weird symbolic links.
Therefore, it would be REALLY helpful, if in a future version of alpine,
it would be possible to specify the default format for new mboxes along
the other parameters of a collectionList!
www.teltarif.de - Kommunikation ganz einfach
teltarif.de Onlineverlag GmbH
Alt-Moabit 96c, 10559 Berlin
Tel: +49 (0)30 453 081-0
Fax: +49 (0)30 453 081-11
Mail: mailto:info at teltarif.de
!!! Achtung wir sind umgezogen !!!
Beachten Sie bitte die neue Adresse
und die neuen Durchwahlen.
Geschäftsführer: Kai Petzke, Martin Müller
eingetragen beim Amtsgericht Berlin-Charlottenburg, HRB 70507
Attraktives Werbeumfeld gefällig?
Vielfältige Targetingoptionen, eine E-Commerce-affine Leserschaft und
passgenaue Platzierungen sorgen für Ihren Werbeerfolg bei teltarif.de.
Weitere Informationen: http://www.teltarif.de/mediadaten
More information about the Alpine-info