Problem with one mailbox in a large mailstore

[Scott Lambert <lambert@xxxxxxxxxxxxxx>]

I just started using OfflineIMAP in January when I bought my new laptop.
At that time my IMAP spool was somewhat small since I had lost my entire
mail spool to the harddrive that died with my old laptop.  OfflineIMAP
is great because it lets me have a local copy of my mail and a backup
on the server.  After loosing my mailspool once, I'm not interested
in it happening again.  Also, interactive IMAP is slowwwwwww.  It's
much faster to read from files on the local machine than to read from
IMAP even on localhost.  The various small interactive IMAP delays were
probably adding up to more than 5 - 10 minutes per day.

Before February, I started having a problem with a particular mailbox,
the one that holds the SpamAssassin users mailing list.  Every time
OfflineIMAP gets to that particular mailbox it blows up with the
following traceback.

My mailspool on the CourierIMAP 2.2.1 server is somewhat large, 734MB.
Of that only 659MB has made it to my laptop.

Thread 'Folder sync LambertFam[Lists.Archives.SAtalk]' terminated with 
exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.3/site-packages/offlineimap/threadutil.py", line 
153, in run
    Thread.run(self)
  File "/usr/local/lib/python2.3/threading.py", line 416, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.3/site-packages/offlineimap/accounts.py", line 
183, in syncfolder
    statusfolder.cachemessagelist()
  File 
"/usr/local/lib/python2.3/site-packages/offlineimap/folder/LocalStatus.py", 
line 73, in cachemessagelist
    uid, flags = line.split(':')
ValueError: unpack list of wrong size


No debug messages were logged for Folder sync LambertFam[Lists.Archives.SAtalk].


My laptop and server are running FreeBSD-CURRENT.  I'm using Python
2.3.3 on the laptop and OfflineIMAP 4.0.0.  I've thought of deleteing
that mailbox locally, but I'm not sure how to do that without
OfflineIMAP trying to delete all those messages from the server.
Deleteing the entire OfflineIMAP local spool and metadata represents
a significant time investment even with my really fast cable internet
provider.  It takes on the order of 15 minutes to run normally.


.offlineimaprc
[general]
accounts = LambertFam
[Account LambertFam]
localrepository = Local-LambertFam
remoterepository = Remote-LambertFam
autorefresh = 5
[Repository Local-LambertFam]
type = Maildir
localfolders = ~/OfflineIMAP/LambertFam
[Repository Remote-LambertFam]
type = IMAP
ssl = yes
remotehost = mail.lambertfam.org
remoteuser = lambert
holdconnectionopen = no
nametrans = lambda foldername: re.sub('^INBOX.', '', foldername)
[Account INCH]
localrepository = Local-INCH
remoterepository = Remote-INCH
[Repository Local-INCH]
type = Maildir
localfolders = ~/OfflineIMAP/INCH
[Repository Remote-INCH]
type = IMAP
ssl = yes
remotehost = dellbsd.inch.com
remoteuser = lambert
[mbnames]
enabled = yes
filename = ~/.muttrc.offlineimap.mailboxes
header = "mailboxes "
peritem = "+%(accountname)s/%(foldername)s"
sep = " "
footer = "\n"

DEBUG run with "offlineimap -1 -u TTY.TTYUI -d imap,maildir,thread -o":
<Big SNIP>
Folder sync LambertFam[Lists.Archives.Postfix_TLS]: <Condition(<thread.lock 
object at 0x8367d70>, 0)>.notify(): no waiters
Account sync LambertFam: <Condition(<thread.lock object at 0x8152180>, 
0)>.wait(): got it
Account sync LambertFam: <threading._BoundedSemaphore object at 
0x835ea6c>.acquire: success, value=0
Account sync LambertFam: <InstanceLimitedThread(Folder sync 
LambertFam[Lists.Archives.SAtalk], initial daemon)>.start(): starting thread
Folder sync LambertFam[Lists.Archives.SAtalk]: <InstanceLimitedThread(Folder 
sync LambertFam[Lists.Archives.SAtalk], started daemon)>.__bootstrap(): thread 
started
Folder sync LambertFam[Lists.Archives.SAtalk]:
   Syncing INBOX.Lists.Archives.SAtalk: IMAP -> Maildir
Folder sync LambertFam[Lists.Archives.SAtalk]: <threading._BoundedSemaphore 
object at 0x835ea6c>.release: success, value=1
Folder sync LambertFam[Lists.Archives.SAtalk]: <Condition(<thread.lock object 
at 0x8152180>, 0)>.notify(): no waiters
Folder sync LambertFam[Lists.Archives.SAtalk]: <InstanceLimitedThread(Folder 
sync LambertFam[Lists.Archives.SAtalk], started daemon)>.__bootstrap(): normal 
return
Folder sync LambertFam[Lists.Archives.SAtalk]: <Condition(<thread.lock object 
at 0x8590fe0>, 0)>.notify(): no waiters
Account sync LambertFam: <threading._BoundedSemaphore object at 
0x835ea6c>.acquire: success, value=0
Account sync LambertFam: <InstanceLimitedThread(Folder sync 
LambertFam[Lists.Archives.VLC], initial daemon)>.start(): starting thread
Thread 'Folder sync LambertFam[Lists.Archives.SAtalk]' terminated with 
exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.3/site-packages/offlineimap/threadutil.py", line 
153, in run
    Thread.run(self)
  File "/usr/local/lib/python2.3/threading.py", line 416, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.3/site-packages/offlineimap/accounts.py", line 
183, in syncfolder
    statusfolder.cachemessagelist()
  File 
"/usr/local/lib/python2.3/site-packages/offlineimap/folder/LocalStatus.py", 
line 73, in cachemessagelist
    uid, flags = line.split(':')
ValueError: unpack list of wrong size


No debug messages were logged for Folder sync LambertFam[Lists.Archives.SAtalk].
Unhandled exception in thread started by <bound method 
InstanceLimitedThread.__bootstrap of <InstanceLimitedThread(Folder sync 
LambertFam[Lists.Archives.VLC], stopped daemon)>>
Traceback (most recent call last):
  File "/usr/local/lib/python2.3/threading.py", line 451, in __bootstrap
    self.__stop()
  File "/usr/local/lib/python2.3/threading.py", line 460, in __stop
    self.__block.notifyAll()
  File "/usr/local/lib/python2.3/threading.py", line 256, in notifyAll
    self.notify(len(self.__waiters))
  File "/usr/local/lib/python2.3/threading.py", line 238, in notify
    currentThread() # for side-effect
TypeError: 'NoneType' object is not callable

There are 10257 messages in Lists.Archives.SAtalk in the local repository.

There are 14078 messages in Lists.Archives.SAtalk on the IMAP server.

There are folders with more messages which are synced before it gets to
Lists.Archives.SAtalk.

-- 
Scott Lambert                    KC5MLE                       Unix SysAdmin
lambert@xxxxxxxxxxxxxx       http://www.lambertfam.org/~lambert/resume.html