Re: Errors synching large dirs

[Sweth Chandramouli <offlineimap@xxxxxxxxxxxxxxx>]

On Tuesday, 09 August 2005 at 09:37:48 EDT,
   John Goerzen (<jgoerzen@xxxxxxxxxxxx>) wrote:
> On Mon, Aug 08, 2005 at 06:51:24PM -0400, Sweth Chandramouli wrote:
> > I'm trying to synch some very large IMAP dirs, and a few of them
> > in particular are failing every few messages with the following error:
> > 
> > ValueError: invalid literal for long(): 
> 
> To troubleshoot that, I'll need a debug log.  Run offlineimap -d imap -1
> and save the result to a file.  Scrub any passwords or sensitive data
> from that, and send it along.

Sorry about the huge delay in following up on this; Real Life
suddenly got very busy.  A debug log of one of the messages that
caused the problem is attached.

> > Any suggestions about what the problem might be?  Each time I
> > restart the process, it continues fine for a few messages before
> > the next error; when offlineimap fails during a transfer, does it
> > somehow remember the "bad" message, and skip it during the next
> > attempt, or is it actually successfully downloading the problematic
> > messages on subsequent runs?
> 
> Nope, it doesn't remember the bad message.  This is interesting, and may
> indicate a server issue.  What server are you running?

Dovecot.  It occurred to me later that the problem would only happen
when I was trying to synch these large dirs from the local machine
to the server, by the way--never from the server to the local box.
 
> > (Also, is there any way to manually "seed" a local repository for
> No.  The reason is that OfflineIMAP must match the local messages with
> the UID assigned by the server.  Hopefully if the error gets fixed, then
> everything would be OK.

How exactly does the matching work, though?  I've got some other
archives on my local box that I'd like to put up on the IMAP server,
but we're talking gigs of mail here; the ideal solution from my POV,
then, would be to just tar everything up and drop it into the FS
datastore for the IMAP server, then run an app on the server to
query the IMAP server and map the filenames to UIDs, and then
take the results of that mapping and rename the local mail files
and update any local indices, so then when I finally run offlineimap
for the first time, no actual messages need to be transferred.
If the rename/reindex piece is feasible (and there's an API into
the index if it isn't all FS-based), then I should be able to write
some quick perl that will take care of the filename-to-UID mapping
and then the local rename/reindex stuff as well.

TIA,

Sweth.

-- 
Sweth Chandramouli
sweth@xxxxxxxxxxxxxxxx - http://EthicalHomes.com/
Planning on selling your house? Ask me about how thinking like a buyer can
help you gets thousands more for your house AND put you in the best position
to buy your next home.