[Alpine-info] extreme slowness working with mbox files

[Mark Crispin]

I'm sorry that you are having this problem.  Unfortunately, it is unavoidable
unless you change the underlying way that you do things.

First: you have disabled the "Save combined copies" option.  By doing so,
you effectively demanded that Alpine do an individual message save for
each selected message instead of an aggregate save.

The fix is to enable this option, so that Alpine does an aggregate save and
incurs the save setup overhead only once instead of per message.

That way, you will only have to wait 22 seconds instead of
<# messages> * 22 seconds.

Second: you use the traditional UNIX mbox format for mailbox files greatly
larger than that format was ever designed to support.  That format was
designed in the 1970s when a "large" mailbox was over 100K.  That format
has been obsolete for at least 20 years.

The fix is to use a more modern mailbox format, preferably mix, but at
least mbx.  A more modern mailbox format does not have the save setup
overhead that exists in traditional UNIX mbox format.

That way, you won't have to wait at all.

Third: you use the SVR4 (e.g., Solaris, AIX, HP-UX) operation system.
These systems are inferior garbage sold by sleazy vendors such as SUN,
who knowingly sell this garbage to trick you into paying more for
hardware than you need.

The fix is to get rid of your SUN crap and replace it with a system running
Linux and/or BSD.  The next time the SUN salesman shows up, throw him
out of your office.

That way, you don't have to go to the extra overhead of having to spawn
an inefficient slave process in order to work around the defective SVR4
operating system.

I recommend that you do all of the above, and certainly the first of these.
You will never be fully happy with performance until you do all of these.  I
know that changing mailbox formats and trashing your expensive SUN
equipment in favor of cheap Linux equipment sounds crazy and radical,
but I assure you that when you eventually do it, the different will be so
substantial that you will wonder why you didn't listen to me and do it much
earlier.

Last, but not least -- yes, there is a larger save overhead in the traditional
UNIX mailbox format.  The 22 seconds sounds about right for your huge
mailboxes.  Unless you disable Save Combined Copies, that cost is incurred
only once in a single save commend.  If you disable Save Combined Copies,
that cost is paid per message.

That cost must be paid.  Certain other sleazy vendors (large companies)
unilaterally decided that an OPTIONAL facility of IMAP was now mandatory,
and began spreading statements that Pine/UW IMAP was defective because
it did not implement this optional facility.  That facility was known to be a
problem with mbox format, which is why it had not been implemented
before.

If you feel that this is wrong, you should have backed me up years
ago when I was being called 69 flavors of "idiot" for standing up for people
who still use legacy mailbox formats, and being called another 105 flavors
of "idiot" for thinking that it was important to maintain halfway decent
performance for mbox format.  But nobody did, and I lost that fight.

It's long too late to undo this now.  That train left the station years ago.

As I said above, the simple step of making sure that Save Combined Copies
is enabled will cause you to have just one 22 second delay instead of the
extended delay that you experienced.  The additional step of using a better
mailbox format -- one designed to support the above-named "optional" (now
mandatory) IMAP facility -- will remove the delay.  And the other additional
step of using a good operating system (Linux, BSD) instead of SVR4 will
remove the overhead of the additional process.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

----------------------------------------

> Date: Tue, 14 Oct 2008 19:24:10 -0500

> From: mbmiller at taxa.epi.umn.edu

> To: alpine-info at u.washington.edu

> Subject: [Alpine-info] extreme slowness working with mbox files

> 

> I understand that some changes were made in the way Alpine handles mbox 

> files compared to the way that Pine handles mbox files.  The difference in 

> speed is so massive that I am hoping something can be done to fix Alpine 

> so that it works more like Pine.  Here is what happened to me:

> 

> I had two mbox files.  The first was 456 messages long and about 30 MB. 

> The second was 3,788 messages long and about 100 MB.  I wanted to add all 

> of the mesages from the first mbox to the end of the second mbox.  I 

> highlighted the messages in Alpine and used Apply, Save.  It sat there for 

> hours.  I looked at processes and this is what I saw:  For every one of 

> the 456 messages to be moved, a new Alpine process was spawned.  That 

> process ran for about 22 seconds and the message then appeared in the 

> target mbox file.

> 

> I left it running overnight but I assume that it took about three hours to 

> move the files.  I'm sure Pine would have done it in a few seconds.

> 

> Mike

> _______________________________________________

> Alpine-info mailing list

> Alpine-info at u.washington.edu

> http://mailman2.u.washington.edu/mailman/listinfo/alpine-info


_________________________________________________________________
See how Windows Mobile brings your life together—at home, work, or on the go.
http://clk.atdmt.com/MRT/go/msnnkwxp1020093182mrt/direct/01/