Re: offlineimap imap<->imap reliability?

[John Goerzen <jgoerzen@xxxxxxxxxxxx>]

On Tue, Oct 21, 2003 at 04:58:00PM +0200, James Leifer wrote:
> First of all, thanks for your detailed reply.
> Let me just clarify a few points.
> 
> > Even if a state file is corrupted (and likely this would just result in it
> > being truncated), the worst that will happen is your mail will be
> > duplicated.
> 
> Are state files overwritten as the sync goes or are they written to a
> temporary file and then atomically moved into place a the end of the
> sync?

Well, both :-)

The files of primary concern are the "local status cache" files, which
record the state of each folder as of the last sync.  These are used to
determine what changed locally (new messages, deleted messages, flag
changes, etc).  There is one file per message and that file may be updated
zero or more times per sync.  Presently, OfflineIMAP defaults to a paranoid
mode where it writes the file after each change.  Each time the file is
saved, OfflineIMAP:

 * Acquires a thread lock to protect the critical section of code.
   Only one thread is allowed to save a given file at a time.
 * Writes the data to filename + ".tmp"
 * Calls os.rename() to rename it atomically to the real name
 * Releases the lock

OfflineIMAP v5 does not use threads (it uses Twisted, which internally uses
no-blocking I/O and a callback system), so the locking code becomes
superfluous and goes away.  This file is implemented in
offlineimap/folder/LocalStatus.py.

Other state files include:
 * uidvalidity files, used to insure that the server UIDs have not changed
   since the last run.  Contain a simple integer, nothing else.  Usually
   written only on the first run on OfflineIMAP.  Same
   lock/save/rename/unlock algorithm as above.  Implemented in
   offlineimap/folder/Base.py.

 * UID mapping files, used for local side of IMAP<->IMAP syncing.
   Same lock/save/rename/unlock algorithm as above.  Implemented in
   offlineimap/folder/UIDMaps.py.

Other items:
 * Messages in Maildirs.  Protected by strict implementation of standard
   Maildirs.  No locking necessarey.

 * Local storage of remote UIDs with Maildirs -- part of the filename
   of the Maildir message.  No mapping or locking necessary.

When necessary, reading proccesses use exclusive locks as well.

> Are the internal integrity of state files verified when loaded by
> offlineimap?

There's not much to check because there's not much structure.  Nothing
complex like XML is used.  Status cache files have a header line identifying
them, which is verified on read.  Other than that, if lines are missing from
either file, there's no way to tell, but if a given line is corrupted
(particularly by having the wrong number of fields, or by having something
wacky in them), this will be noticed.  The check is not explicit, but rather
a side-effect of writing tight parsing code -- an exception will be raised.

UID maps do not have the leading line but otherwise the rest of this applies
to them.

UID validity files have only a single int.  If OfflineIMAP can't get an int
from them, it dies.

> > There is one small potential race condition that I hope to fix for
> > OfflineIMAP 5.0:  If a new message arrives on the remote end, and the remote
> > server tells OfflineIMAP about it when it asks for a list of messages in the
> > folder, but it gets deleted from the remote before OfflineIMAP can download
> > it, OfflineIMAP will exit with an error.
> 
> OK, good to know.  Here's another scenario where this might occur:
> there's a spam filter running concurrently with offlineimap that opens
> a connection to the remote server and walks through the files, moving
> suspected spams to another folder.
> 
> When you say ``deleted'' does that include ``moved to another folder''?

Yes.  OfflineIMAP handles a move as a delete and an add, and incidentally
can handle this on both ends.  You might want to run your spam filter over
your local tree, BTW.  OfflineIMAP will still propogate the changes.

> >> * What happens if ``conflicts'' arise, i.e. each replica has different
> >>   flags modified for a particular message since the last sync?
> >
> > Flags are handled on an as-changed basis; so if the local system added the
> > seen flag and the remote added replied, the message should end up with 
> > both. 
> > (I'll have to verify this is the case, but I think it is.)
> 
> And if on one replica a message is moved to another folder and on the
> other replica, the message is not moved but receives new flags, what
> does the synced state look like? Is it moved+flags changed, or just
> one or just the other, or something else?  Does it matter whether the
> move is local and the flag change remote or vice versa?

Since a move is a delete and an add, it will basically take precedence over
everything else.  The other end will end up looking like the end that moved,
with the flags set however they were set on the message in the new folder.

> > Otherwise, local changes override remote ones.
> 
> So if flag foo is added on the local side and removed on the remote
> then in the newly synced state foo is present on both?

Well, this is not really a logical situation.  If the message started out
already having flag foo after the last sync, then you could remove it on the
remote, but adding it locally doesn't make sense.  So I'm not sure I really
know how to answer.

I suppose, though, that if everything I said before about this was correct,
then there is not really a situation in which there is a need for one side
to override the other.  To be quite honest, I have not spent a great deal of
time thinking about the details in this situation since it has never struck
me as very critical.

-- John