[PATCH] Convert LocalStatus to sqlite

[Stewart Smith <stewart@xxxxxxxxxxxxxxxx>]

Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable
Forgive the potentially bad python, not my native tongue :)

This patch is motivated by three things:
- offlineimap is extremely slow at syncing lots of locally deleted
messages
- offlineimap uses lots of memory
- LocalStatus files aren't written safely (a hard crash can cause
corruption)
        - I've been bitten by this in the past, causing a complete resync of
the folder... so I get duplicate messages.

I am currently using 4.0.14 (from Debian) with this patch. I used it to
convert the files and everything. Seems quite reliable and quick.

In my tests, execution time for a normal sync is relatively the same.

Execution time for when lots of messages have been deleted in a
reasonably sized folder (e.g. during re-organisation of mail folders) is
as much as 10x faster.

In my tests, running with 1 thread uses as much as 20% less memory with
this patch (i.e. about 160MB instead of 200MB+ for my maildir)

Disk space used by the LocalStatus files isn't much more... for me it
looks like it's 6.5MB now versus 4.5MB then. We get the added benefit of
indexes for all our queries... nice :)

I had disable the threading for copying messages as this means that
LocalStatus objects are shared between threads, which pysqlite doesn't
like (it asserts).

I think the part of this patch that implements the uidexists does
actually slow things down compared with having the messagelist.... a
more optimal implementation may be possible, but I think the other speed
improvements (and memory savings) are worth it.

A future patch may convert other storage types to sqlite (or similar) to
further reduce memory consumption (and hopefully runtime).

This does add a dependency on pysqlite... which is packaged in debian
(and ubuntu) - and i'm using the stock packages for these.

Comments very much appreciated.
--=20
Stewart Smith (stewart@xxxxxxxxxxxxxxxx)
http://www.flamingspork.com/



-- Binary/unsupported file stripped by Ecartis --
-- Type: text/x-patch
-- File: offlineimap_sqlite_localstatus.patch



-- Attached file included as plaintext by Ecartis --
-- File: signature.asc
-- Desc: This is a digitally signed message part

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQBGClmaKglWCUL+FDoRAiseAJ9M37d+aaacs4f3D6ycClOkvoPbzACg0ndA
wdKmyqKOhRjFFpQTcDvfDNU=
=kZly
-----END PGP SIGNATURE-----