offlineimap and traffic usage.

[Bruce Campbell <bru-offlineimap@xxxxxxxxxxxxxxxxxxxxxx>]

[rambling rant, don't expect anything useful in here beyond a basic
  calculation of traffic incurred per sync]

On Fri, 15 Dec 2006, Martin Pohlack wrote:

> The initial copy of my whole mail-account took quiet some time (~ 1.3
> GB).  This was to be expected :-).
>
> However, even empty synchronization runs (with 0 mails being
> transferred) use about 5 MB of traffic, which is a lot every, say, 5
> minutes.

I had cause to go through this at the time and more recently, and have 
noticed the same; a huge amount of traffic each time the synchronisation 
runs on my mailboxes.

Looking through the debug output, offlineimap tends to do on each sync:

        for each mailbox-in-original-list:
                > SELECT mailbox
                        < FLAGS blah blah
                        < EXISTS blah
                        < UIDVALIDITY blah
                > SELECT mailbox
                        < FLAGS blah blah
                        < EXISTS blah
                        < UIDVALIDITY blah
                > FETCH 1:EXISTS-VALUE (FLAGS UID)
                        < 1 FETCH (UID Y FLAGS blah)
                        < 2 FETCH (UID Y FLAGS blah)
                        < 3 FETCH (UID Y FLAGS blah)
                        ( so forth up to EXISTS-VALUE )
                send any updates if differences found.
        done
        snooze till next run.

With even a small number of mailboxes (say, 20) and 100 messages in each, 
each sync run will have 20 * 100 FETCH lines (ignoring the output from 
multiple SELECT commands), or at least 2000 lines of ~30 characters each; 
60k right there.

As you get more messages, you add another ~30 bytes to each sync that you 
are doing, leading to situations where 5 meg per transfer, simply because 
you have ~175000 messages, is the normal case.  I'd be setting your 
refresh time to somethign pretty high in that case.

In the language of George Orwell, this is double-plus ungood, particularly 
if you are using offlineimap to avoid leaving your mail client logged into 
your main mail server.

Unfortunately, the nature of IMAP means that to do an accurate sync, you 
do need to do this type of check, but you don't need to do it to every 
mailbox every time.  As a starter, you can easily split your 
.offlineimaprc file into 'small mailboxes, changes frequently' and 'large 
mailboxes, changes infrequently', and run multiple instances of 
offlineimap over these two config files accordingly.

It would be pretty neat to have this split within offlineimap itself, with 
the mailboxes tagged as 'quiet' being synced at longer intervals than the 
'noisy' mailboxes.  This would require offlineimap keeping more state data 
than it already does.

If you have control over both ends of your connection, you could introduce 
a proxy between offlineimap and your IMAP server, the server side of which 
does all the magic of (ab)using your IMAP server for the current settings 
of all message flags, and only passes over the link the flags that have 
changed on the server side (the local side faking a full list to 
offlineimap as required).

If patience is on your side, you could wait for both offlineimap and your 
favourite IMAP server to support the CONDSTORE extension (rfc4551), which 
would allow offlineimap to instantly detect (via HIGHESTMODSEQ) changes to 
a given mailbox which would require a more thorough sync to be performed 
for that mailbox.  Sending patches in would make this wait go faster.

Until then, take the number of messages you have across all of your 
mailboxes, multiple by 30, and thats about how many bytes your traffic 
bill will increase by each time you run your sync, whether using 
offlineimap or your IMAP mail client directly to your server.

-- 
   Bruce Campbell.