A bit unrelated to: [PATCH] Convert LocalStatus to sqlite

[David Favro <offlineimap@xxxxxxxxxxxxxxxx>]

Stewart Smith wrote:
> Server is courier-imap serving out of Maildir. [...]  Disk is RAID1
> running XFS.
> [...] about 153 mailboxes. Largest is about 170,000 messages... at least
> another a bit over 100,000. 5-10k messages fairly common. 30k messages
> in a few folders.
>
> The typical test was deleting  about 20,000 messages from a 120,000
> message folder.
>   

Not regarding the patch, nor offlineimap performance, but your numbers
make me think about something that's bothered me about Maildir for a
long time... I don't know much (anything) about XFS, but in general it
strikes me as unlikely that most filesystems can provide decent
performance when accessing, or opening a file from, a single directory
that contains, e.g., 120k or 170k files -- certainly older OS's will
perform poorly.  It seems like a glaring defect in the whole Maildir
format (as well as modern data storage in general, which I have thought
for some time is ill-suited to the hierarchical directory structure of
most filesystems).  I have wanted to write an SQL-based mailbox for a
long time but haven't gotten around to it, and recently noticed that
these guys have done it already, though I haven't tried it yet:
http://dbmail.org/

Anyhow, just a comment, but I was also wondering what kind of
performance you get, Stewart, on those mailboxes.  Did you ever consider
a different storage format?  Does anyone know if linear search is still
typical of filesystem directory traversal or are people using hashes or
btrees these days?  That would also seem odd because it's overkill for
the more typical FS directory that contains <100 files.