Re: Assorted patches

[Stewart Smith <stewart@xxxxxxxxxxxxxxxx>]

On Sun, 2007-09-30 at 13:54 -0400, Daniel Jacobowitz wrote:
> Here's the result of a lazy Sunday hacking on offlineimap.  Sorry for
> not breaking this into multiple patches.  They're mostly logically
> independent so just ask if that would make a difference.
> First, a new -q (quick) option.  The quick option means to only update
> folders that seem to have had significant changes.  For Maildir, any
> change to any message UID or flags is significant, because checking
> the flags doesn't add a significant cost.  For IMAP, only a change to
> the total number of messages or a change in the UID of the most recent
> message is significant.  This should catch everything except for
> flags changes.
> 
> The difference in bandwidth is astonishing: a quick sync takes 80K
> instead of 5.3MB, and 28 seconds instead of 90.
BEFORE:
rcvd=4863, sent=25921292, time=511
rcvd=5672, sent=15135719, time=509
rcvd=11413, sent=13180623, time=509


AFTER:
rcvd=3405, sent=26544, time=72
rcvd=4175, sent=27820, time=66
rcvd=3877, sent=781402, time=66


NEAT!

(both runs with my localstatus in sqlite patch)

> Third, some speedups in Maildir checking.  It's still pretty slow
> due to a combination of poor performance in os.listdir (never reads
> more than 4K of directory entries at a time) and some semaphore that
> leads to lots of futex wake operations, but at least this saves
> 20% or so of the CPU time running offlineimap on a single folder:

For another project I've written some neat code that optimises reading
of Maildirs... perhaps I should port some of that code over... (it
avoids a lot of disk seeks).

I should also work on getting the memory consumption down more... for
me, offlineimap still uses a few hundred MB RAM... although it's a lot
better with my "localstatus in sqlite" patch.

-- 
Stewart Smith (stewart@xxxxxxxxxxxxxxxx)
http://www.flamingspork.com/


-- Attached file included as plaintext by Ecartis --
-- File: signature.asc
-- Desc: This is a digitally signed message part

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iQIVAwUARwMbOb3yNwHyU3DLAQK+dg//dXzavfIR0GM8yJR47t0BUMDERTQRA0S9
XI2OdZwTyUeW/7Ugn/tiqqpi2fzYy41A+fhcqfyzB4Qmi6eKNdsc9+sI+0AJ5/B4
igeKdVA41VmubbMcqXKYv6AxNzMI7oohcO0W6q+xEv632vx3sDn9dfVK/yALrAva
8yMNEtuA5d6Ov/7Pegxi5RM517HU98DAw5kHHdpnW+EeNa0S+9lKJofddCP3tB7r
OMxYqTDvbYQzIhXAFsz940nFJF/VjK7K7GCoJJqJp5h0w/O+rrFCDHxVVQlkSjw3
dg73gz371I4yg58fjwwdb59N0RH6BkZQ0Eab125Mf1H2dKrCzoyWflggaPNu++3Z
kHaZAjrk2n1bMdy8OilSbuOZWFsuYE8Vd4jPUVnxTZCuIvgMfYYynlCQdTydetjF
MA4oIfPexS+L4yD/nDb5Db/YqZDecyU2o8Q147B3tPDpE6TqylMJ4RSVY5SuhEgp
2x2iNTyLMef/raZoUMDWMctafc7XsxTLbLsJCZBIschlGm+DSYyHqMueOKijygSP
2XK6YF89y9s3B6v3EaqnSzqe0I2W+fYUK86hJaQSRnTaZreoSoUgfFPXIHGa3STC
nnwPe94R80jmK7QV29XdsuyghKoxaNqPoO7ZhNEdjZ07JeAe4jXegHB6mi7DLerv
3XfUYJoaIRM=
=zYEk
-----END PGP SIGNATURE-----