Re: [PATCH] improve savemessage_searchforheader - take 2

[Adam Spiers <offlineimap@xxxxxxxxxxxxxx>]

Adam Spiers (offlineimap@xxxxxxxxxxxxxx) wrote:
> OK, I branched your darcs repository, made savemessage_searchforheader
> more robust, updated the .spec file, and added a couple of targets to
> the Makefile to make it easy to build rpms.

Argh, this "robustness" actually ends up being too paranoid.  

Firstly, after coding that patch, I found offlineimap/folder/Base.py
which nicely documents the expected return codes from savemessage().
Therefore I needed to revise my handling of 'return 0' vs 'raise an
exception' in the case where the IMAP server does not implement UID
SEARCH at all.

Secondly, I found the below edge case with the Groupwise IMAP server
which breaks it :-(

My Groupwise account has rules set up to move newly arrived mail with
certain patterns in the Subject: header to designated folders.
Unfortunately these rules not only trigger with new mail arriving in
the main "Mailbox" folder ("INBOX" in IMAP parlance), but also when
offlineimap does an APPEND to place a mail in an auxiliary folder.
That means that offlineimap execution looks like this:

- ...
C: APPEND mail to folder with X-OfflineIMAP-... header
S: OK APPEND completed
C: Issue a CHECK command to let the server do internal flushing etc.
S: OK CHECK completed
C: UID SEARCH for X-OfflineIMAP-... header
S: * SEARCH (i.e. header not found)
S: OK UID SEARCH completed
C: NOOP
S: OK NOOP completed
C: UID SEARCH for X-OfflineIMAP-... header (again, hoping the NOOP will help)
S: * SEARCH (i.e. header not found)
S: OK UID SEARCH completed

And here the header is *still* not found, because somewhere in between
completion of the APPEND and here, the rule triggered and moved the
newly-appended mail to another folder!

It's my belief that the rule should not trigger because the mail isn't
really a new mail unless it's being delivered straight to the main
"Mailbox" folder by the MTA.  I've submitted a bug for this:

  https://bugzilla.novell.com/show_bug.cgi?id=175830

However, I'm struggling to find a workaround :-(  Imagine this
scenario:

- New mail arrives
- Mail matches a rule and automatically gets moved to an auxiliary inbox
- offlineimap connects and happily retrieves it into aux. inbox on the
  local filesystem
- User reads mail locally with MTA and decides to file it in an
  archive folder for future reference
- offlineimap runs again, deletes the message from aux. inbox and APPENDs
  it to archive folder
- Rule immediately triggers AGAIN and moves it back to the aux. inbox!
- offlineimap's UID SEARCH fails twice.

Now what do we do?  We can't raise an exception because that stops the
user (me!) receiving any other mail.  But if we return 0 to indicate
that the mail was successfully saved on the IMAP server with an
unknown uid, then it will get deleted from the local folder and
redownloaded back into the aux. inbox when the next sync occurs.
In other words, no matter how many times the user files the mail in
the archive, it keeps reappearing in the aux. inbox!

Suggestions VERY welcome at this point!  I'm getting a bit desperate
for a stable email solution here :-(

Having said that, http://www.adamspiers.org/darcs/offlineimap/ now
reflects the latest code, which I believe is the least prone to
crashing, and also does the most *accurate* synchronisation so far,
even if it's not helpful to the user for the reasons above ...

Adam

P.S. I never said before, but congratulations John for turning a
complex task into such legible code :-)