Patch: signalling offlineimap to synch immediately and to terminate

[Jim Pryor <lists+offlineimap@xxxxxxxxxxxx>]

Here's the way I'd like to use offlineimap on my laptop:
    1. Have a regular cron job running infrequently. The cron job checks to see
if I'm online, plugged in, and that no other copy of offlineimap is running. If
all of these conditions are satisfied, it runs offlineimap just once:
"offlineimap -o -u Noninteractive.Quiet"

    2. When I start up mutt, I do it by calling a wrapper script that delays
until cron-started copies of offlineimap have finished, then starts offlineimap
on its regular, stay-alive and keep checking schedule. When I quit mutt, the
wrapper script tells offlineimap to stop.

This way I get frequent regular checks while I have mutt running, but I don't
waste my battery/cpu checking frequently for mail when I'm not interested in
it.

To make this work, though, it'd be nicer if it were easier to tell offlineimap,
from the outside, things like "terminate cleanly now" and "when you've finished
synching, then terminate instead of sleeping and synching again."

OK, to put my money where my mouth is, I attach two patches against offlineimap
6.0.3.

The first, "cleanup.patch", cleans up a few spots that tend to throw exceptions
for me as offlineimap is exiting from a KeyboardInterrupt.

The second adds signaling capabilities to offlineimap.


* sending a SIGTERM tells offlineimap to terminate immediately but cleanly,
  just as if "q" had been pressed in the GUI interface

* sending a SIGUSR1 tells every account to do a full sync asap: if it's
  sleeping, then wake up and do the sync now. If it's mid-sync, then re-synch
  any folders whose syncing has already been started or completed, and continue
  to synch the other, queued but not-yet-synched folders.

* sending a SIGHUP tells every account to die as soon as it can (but not
  immediately: only after finishing any synch it's now engaged in)

* sending a SIGUSR2 tells every account to do a full sync asap (as with
  SIGUSR1), then die


It's tricky to mix signals with threads, but I think I've done this correctly.
I've been using it now for a few weeks without any obvious problems. But I'm 
passing it
on so that others can review the code and test it out on their systems. I 
developed the
patch when I was running Python 2.5.2, but to my knowledge I don't use any 
Python 2.5-specific
code. Now I'm using the patch with Python 2.6.

Although I said "without any obvious problems," let me confess that I'm
seeing offlineimap regularly choke when I do things like this: start up
my offlineimap-wrapped copy of mutt, wait a while, put the machine to
sleep (not sure if offlineimap is active in the background or idling),
move to a different spot, wake the machine up again and it acquires a
new network, sometimes a wired network instead of wifi. Offlineimap
doesn't like that so much. I don't yet have any reason to think the
problems here come from my patches. But I'm just acknowledging them, so
that if others are able to use offlineimap without any difficulty in
situations like I described, then maybe the fault is with my patches. 



With offlineimap patched in the ways I'm attaching, here's my mutt-wrapper 
script, described
in item 2 at the top. I've edited it here to remove stuff specific to my
system; I think I didn't leave anything out but let me know if you try it out
and it doesn't work.



SERVER="your.server.here"
MUTTCMD=/usr/bin/mutt

function promptchars {
        # Usage: promptchars PROMPT chars [timeout=1 sec]
        local prompt="$1" timeout="${3:-1}"
        set "${2:0:1}" "${2:1:1}" "${2:2:1}" "${2:3:1}" "${2:4:1}"
        while true; do
                [[ -n "$prompt" ]] && echo -ne "${prompt% } "
                if read -n1 -t "$timeout"; then
                        # got input
                        case $(echo "$REPLY"| tr [A-Z] [a-z]) in
                                "") [[ -z "$prompt" ]] && return 11;;
                                "$1") [[ -n "$prompt" ]] && echo
                                        return 1;;
                                "$2") [[ -n "$prompt" ]] && echo
                                        return 2;;
                                "$3") [[ -n "$prompt" ]] && echo
                                        return 3;;
                                "$4") [[ -n "$prompt" ]] && echo
                                        return 4;;
                                "$5") [[ -n "$prompt" ]] && echo
                                        return 5;;
                                *)      echo
                                        [[ -z "$prompt" ]] && return 11;;
                        esac
                else # timed out
                        [[ -n "$prompt" ]] && echo
                        return 0
                fi
        done
}


function server_avail {
    ping -W 1 -c 1 "$1" 2>/dev/null | grep -q "1 received"
}


function mutt-with-offlineimap {

        local pid
        while true; do
                pid=$(/bin/ps -opid= -C offlineimap | head -n1)
                pid="${pid## *}"
                if [[ -z "$pid" ]]; then break; fi
                echo "Offlineimap running (pid=$pid). Waiting for it to stop..."
                echo -n "  (k)ill it  (q)uit  (s)tart mutt without it? "
                while [[ -d "/proc/$pid" ]]; do
                        promptchars "" kqs
                        case $? in
                                0)  #timedout
                                        ;;
                                11) # refresh prompt
                                        echo -n "  (k)ill it  (q)uit  (s)tart 
mutt without it? "
                                        ;;
                                1)  kill "$pid" # send TERM signal to 
offlineimap; send HUP 
                                # instead if you only want it to die after 
                                # current synch is finished
                                        pid=
                                        break;;
                                2)  echo; return;;
                                3)  echo
                                        exec "$MUTTCMD"
                        esac
                done
                echo
                if [[ -z "$PID" ]]; then sleep 1; fi    # if killed and broke 
out of 
                                                # /proc monitoring loop, give 
                                                # process chance to die
        done

        echo -n "Offlineimap starting..."
        /usr/bin/offlineimap -u Noninteractive.Quiet &
        pid="$!"
        echo
        "$MUTTCMD"
    # when mutt has finished...
        if [[ -d "/proc/$pid" ]]; then
                kill -USR2 "$pid"       # tell offlineimap to do a complete 
synch then die
                if false; then
                        echo "Offlineimap will terminate when it finishes 
synching"
                else
                        echo "Waiting for offlineimap to finish synching..."    
                        wait "$pid"
                fi
        else
                echo "Offlineimap already terminated"
        fi
}


/bin/pidof -sx $(abspath "$0") >/dev/null       # is another mutt-wrapper 
running?
OFFLINEIMAP=$?

# if script was called with cmd-line options, 
# or another mutt-wrapper is running, then
# don't try to start offlineimap, just exec mutt directly
if [[ "$#" -gt 0 || "$OFFLINEIMAP" != 1 ]]; then
        exec "$MUTTCMD" "$@"
fi

echo -n "Checking server..."
if ! server_avail "$SERVER"; then
        # server not available; don't start offlineimap
        echo "not available; working offline..."
        exec "$MUTTCMD"
fi

# server available, try to start offlineimap then mutt
echo
mutt-with-offlineimap


-- 
Jim Pryor
jim@xxxxxxxxxxxx


-- Attached file included as plaintext by Ecartis --

diff -Naur offlineimap.orig/accounts.py offlineimap.cleaned/accounts.py
--- offlineimap.orig/offlineimap/accounts.py    2008-08-22 19:29:10.000000000 
-0400
+++ offlineimap.cleaned/offlineimap/accounts.py 2008-08-23 08:28:19.000000000 
-0400
@@ -208,11 +208,11 @@
     if len(localfolder.getmessagelist()) or len(statusfolder.getmessagelist()):
         if not localfolder.isuidvalidityok():
             ui.validityproblem(localfolder)
-           localrepos.restore_atime()
+            localrepos.restore_atime()
             return
         if not remotefolder.isuidvalidityok():
             ui.validityproblem(remotefolder)
-           localrepos.restore_atime()
+            localrepos.restore_atime()
             return
     else:
         localfolder.saveuidvalidity()
diff -Naur offlineimap.orig/threadutil.py offlineimap.cleaned/threadutil.py
--- offlineimap.orig/offlineimap/threadutil.py  2008-08-22 19:29:10.000000000 
-0400
+++ offlineimap.cleaned/offlineimap/threadutil.py       2008-08-23 
08:28:19.000000000 -0400
@@ -159,16 +159,18 @@
                             self.getName() + ".prof")
         except:
             self.setExitCause('EXCEPTION')
-            self.setExitException(sys.exc_info()[1])
-            sbuf = StringIO()
-            traceback.print_exc(file = sbuf)
-            self.setExitStackTrace(sbuf.getvalue())
+            if sys:
+                self.setExitException(sys.exc_info()[1])
+                sbuf = StringIO()
+                traceback.print_exc(file = sbuf)
+                self.setExitStackTrace(sbuf.getvalue())
         else:
             self.setExitCause('NORMAL')
         if not hasattr(self, 'exitmessage'):
             self.setExitMessage(None)
 
-        exitthreads.put(self, True)
+        if exitthreads:
+            exitthreads.put(self, True)
 
     def setExitCause(self, cause):
         self.exitcause = cause
@@ -230,7 +232,8 @@
         try:
             ExitNotifyThread.run(self)
         finally:
-            instancelimitedsems[self.instancename].release()
+            if instancelimitedsems and instancelimitedsems[self.instancename]:
+                instancelimitedsems[self.instancename].release()
         
     
 ######################################################################


-- Attached file included as plaintext by Ecartis --

diff -Naur offlineimap.cleaned/accounts.py offlineimap/accounts.py
--- offlineimap.cleaned/offlineimap/accounts.py 2008-08-23 08:28:19.000000000 
-0400
+++ offlineimap/offlineimap/accounts.py 2008-08-23 10:52:35.000000000 -0400
@@ -19,8 +19,71 @@
 import offlineimap.repository.Base, offlineimap.repository.LocalStatus
 from offlineimap.ui import UIBase
 from offlineimap.threadutil import InstanceLimitedThread, ExitNotifyThread
-from threading import Event
+from threading import Event, Lock
 import os
+from Queue import Queue, Empty
+
+class SigListener(Queue):
+    def __init__(self):
+        self.folderlock = Lock()
+        self.folders = None
+        Queue.__init__(self, 20)
+    def put_nowait(self, sig):
+        self.folderlock.acquire()
+        try:
+            if sig == 1:
+                if self.folders is None or not self.autorefreshes:
+                    # folders haven't yet been added, or this account is 
once-only; drop signal
+                    return
+                elif self.folders:
+                    for folder in self.folders:
+                        # requeue folder
+                        self.folders[folder] = True
+                    self.quick = False
+                    return
+                # else folders have already been cleared, put signal...
+        finally:
+            self.folderlock.release()
+        Queue.put_nowait(self, sig)
+    def addfolders(self, remotefolders, autorefreshes, quick):
+        self.folderlock.acquire()
+        try:
+            self.folders = {}
+            self.quick = quick
+            self.autorefreshes = autorefreshes
+            for folder in remotefolders:
+                # new folders are queued
+                self.folders[folder] = True
+        finally:
+            self.folderlock.release()
+    def clearfolders(self):
+        self.folderlock.acquire()
+        try:
+            for folder in self.folders:
+                if self.folders[folder]:
+                    # some folders still in queue
+                    return False
+            self.folders.clear()
+            return True
+        finally:
+            self.folderlock.release()
+    def queuedfolders(self):
+        self.folderlock.acquire()
+        try:
+            dirty = True
+            while dirty:
+                dirty = False
+                for folder in self.folders:
+                    if self.folders[folder]:
+                        # mark folder as no longer queued
+                        self.folders[folder] = False
+                        dirty = True
+                        quick = self.quick
+                        self.folderlock.release()
+                        yield (folder, quick)
+                        self.folderlock.acquire()
+        finally:
+            self.folderlock.release()
 
 def getaccountlist(customconfig):
     return customconfig.getsectionlist('Account')
@@ -61,7 +124,7 @@
     def getsection(self):
         return 'Account ' + self.getname()
 
-    def sleeper(self):
+    def sleeper(self, siglistener):
         """Sleep handler.  Returns same value as UIBase.sleep:
         0 if timeout expired, 1 if there was a request to cancel the timer,
         and 2 if there is a request to abort the program.
@@ -82,14 +145,25 @@
             item.startkeepalive()
         
         refreshperiod = int(self.refreshperiod * 60)
-        sleepresult = self.ui.sleep(refreshperiod)
+#         try:
+#             sleepresult = siglistener.get_nowait()
+#             # retrieved signal before sleep started
+#             if sleepresult == 1:
+#                 # catching signal 1 here means folders were cleared before 
signal was posted
+#                 pass
+#         except Empty:
+#             sleepresult = self.ui.sleep(refreshperiod, siglistener)
+        sleepresult = self.ui.sleep(refreshperiod, siglistener)
+        if sleepresult == 1:
+            self.quicknum = 0
+
         # Cancel keepalive
         for item in kaobjs:
             item.stopkeepalive()
         return sleepresult
             
 class AccountSynchronizationMixin:
-    def syncrunner(self):
+    def syncrunner(self, siglistener):
         self.ui.registerthread(self.name)
         self.ui.acct(self.name)
         accountmetadata = self.getaccountmeta()
@@ -105,19 +179,19 @@
         self.statusrepos = 
offlineimap.repository.LocalStatus.LocalStatusRepository(self.getconf('localrepository'),
 self)
             
         if not self.refreshperiod:
-            self.sync()
+            self.sync(siglistener)
             self.ui.acctdone(self.name)
             return
         looping = 1
         while looping:
-            self.sync()
-            looping = self.sleeper() != 2
+            self.sync(siglistener)
+            looping = self.sleeper(siglistener) != 2
         self.ui.acctdone(self.name)
 
     def getaccountmeta(self):
         return os.path.join(self.metadatadir, 'Account-' + self.name)
 
-    def sync(self):
+    def sync(self, siglistener):
         # We don't need an account lock because syncitall() goes through
         # each account once, then waits for all to finish.
 
@@ -141,19 +215,24 @@
             self.ui.syncfolders(remoterepos, localrepos)
             remoterepos.syncfoldersto(localrepos, [statusrepos])
 
-            folderthreads = []
-            for remotefolder in remoterepos.getfolders():
-                thread = InstanceLimitedThread(\
-                    instancename = 'FOLDER_' + self.remoterepos.getname(),
-                    target = syncfolder,
-                    name = "Folder sync %s[%s]" % \
-                    (self.name, remotefolder.getvisiblename()),
-                    args = (self.name, remoterepos, remotefolder, localrepos,
-                            statusrepos, quick))
-                thread.setDaemon(1)
-                thread.start()
-                folderthreads.append(thread)
-            threadutil.threadsreset(folderthreads)
+            siglistener.addfolders(remoterepos.getfolders(), 
bool(self.refreshperiod), quick)
+
+            while True:
+                folderthreads = []
+                for remotefolder, quick in siglistener.queuedfolders():
+                    thread = InstanceLimitedThread(\
+                        instancename = 'FOLDER_' + self.remoterepos.getname(),
+                        target = syncfolder,
+                        name = "Folder sync %s[%s]" % \
+                        (self.name, remotefolder.getvisiblename()),
+                        args = (self.name, remoterepos, remotefolder, 
localrepos,
+                                statusrepos, quick))
+                    thread.setDaemon(1)
+                    thread.start()
+                    folderthreads.append(thread)
+                threadutil.threadsreset(folderthreads)
+                if siglistener.clearfolders():
+                    break
             mbnames.write()
             localrepos.forgetfolders()
             remoterepos.forgetfolders()
diff -Naur offlineimap.cleaned/init.py offlineimap/init.py
--- offlineimap.cleaned/offlineimap/init.py     2008-08-23 08:28:19.000000000 
-0400
+++ offlineimap/offlineimap/init.py     2008-08-23 10:52:50.000000000 -0400
@@ -26,6 +26,7 @@
 from threading import *
 import threading, socket
 from getopt import getopt
+import signal
 
 try:
     import fcntl
@@ -131,6 +132,11 @@
 
     lock(config, ui)
 
+    def sigterm_handler(signum, frame):
+        # die immediately
+        ui.terminate(errormsg="terminating...")
+    signal.signal(signal.SIGTERM,sigterm_handler)
+
     try:
         pidfd = open(config.getmetadatadir() + "/pid", "w")
         pidfd.write(str(os.getpid()) + "\n")
@@ -183,12 +189,31 @@
                 else:
                     threadutil.initInstanceLimit(instancename,
                                                  
config.getdefaultint('Repository ' + reposname, "maxconnections", 1))
+        siglisteners = []
+        def sig_handler(signum, frame):
+            if signum == signal.SIGUSR1:
+                # tell each account to do a full sync asap
+                signum = (1,)
+            elif signum == signal.SIGHUP:
+                # tell each account to die asap
+                signum = (2,)
+            elif signum == signal.SIGUSR2:
+                # tell each account to do a full sync asap, then die
+                signum = (1, 2)
+            # one listener per account thread (up to maxsyncaccounts)
+            for listener in siglisteners:
+                for sig in signum:
+                    listener.put_nowait(sig)
+        signal.signal(signal.SIGHUP,sig_handler)
+        signal.signal(signal.SIGUSR1,sig_handler)
+        signal.signal(signal.SIGUSR2,sig_handler)
 
         threadutil.initexitnotify()
         t = ExitNotifyThread(target=syncmaster.syncitall,
                              name='Sync Runner',
                              kwargs = {'accounts': syncaccounts,
-                                       'config': config})
+                                       'config': config,
+                                       'siglisteners': siglisteners})
         t.setDaemon(1)
         t.start()
     except:
diff -Naur offlineimap.cleaned/syncmaster.py offlineimap/syncmaster.py
--- offlineimap.cleaned/offlineimap/syncmaster.py       2008-08-23 
08:28:19.000000000 -0400
+++ offlineimap/offlineimap/syncmaster.py       2008-08-23 08:31:27.000000000 
-0400
@@ -20,27 +20,31 @@
 from offlineimap import imapserver, repository, folder, mbnames, threadutil, 
version
 from offlineimap.threadutil import InstanceLimitedThread, ExitNotifyThread
 import offlineimap.accounts
-from offlineimap.accounts import SyncableAccount
+from offlineimap.accounts import SyncableAccount, SigListener
 from offlineimap.ui import UIBase
 import re, os, os.path, offlineimap, sys
 from ConfigParser import ConfigParser
 from threading import *
 
-def syncaccount(threads, config, accountname):
+def syncaccount(threads, config, accountname, siglisteners):
     account = SyncableAccount(config, accountname)
+    siglistener = SigListener()
     thread = InstanceLimitedThread(instancename = 'ACCOUNTLIMIT',
                                    target = account.syncrunner,
-                                   name = "Account sync %s" % accountname)
+                                   name = "Account sync %s" % accountname,
+                                   kwargs = {'siglistener': siglistener} )
+    # the Sync Runner thread is the only one that will mutate siglisteners
+    siglisteners.append(siglistener)
     thread.setDaemon(1)
     thread.start()
     threads.add(thread)
     
-def syncitall(accounts, config):
+def syncitall(accounts, config, siglisteners):
     currentThread().setExitMessage('SYNC_WITH_TIMER_TERMINATE')
     ui = UIBase.getglobalui()
     threads = threadutil.threadlist()
     mbnames.init(config, accounts)
     for accountname in accounts:
-        syncaccount(threads, config, accountname)
+        syncaccount(threads, config, accountname, siglisteners)
     # Wait for the threads to finish.
     threads.reset()
diff -Naur offlineimap.cleaned/ui/Blinkenlights.py 
offlineimap/ui/Blinkenlights.py
--- offlineimap.cleaned/offlineimap/ui/Blinkenlights.py 2008-08-23 
08:28:19.000000000 -0400
+++ offlineimap/offlineimap/ui/Blinkenlights.py 2008-08-23 08:31:44.000000000 
-0400
@@ -128,10 +128,10 @@
         finally:
             s.tflock.release()
             
-    def sleep(s, sleepsecs):
+    def sleep(s, sleepsecs, siglistener):
         s.gettf().setcolor('red')
         s.getaccountframe().startsleep(sleepsecs)
-        UIBase.sleep(s, sleepsecs)
+        return UIBase.sleep(s, sleepsecs, siglistener)
 
     def sleeping(s, sleepsecs, remainingsecs):
         if remainingsecs and s.gettf().getcolor() == 'black':
diff -Naur offlineimap.cleaned/ui/Curses.py offlineimap/ui/Curses.py
--- offlineimap.cleaned/offlineimap/ui/Curses.py        2008-08-23 
08:28:19.000000000 -0400
+++ offlineimap/offlineimap/ui/Curses.py        2008-08-23 08:31:57.000000000 
-0400
@@ -541,10 +541,10 @@
         s.c.stop()
         UIBase.mainException(s)
 
-    def sleep(s, sleepsecs):
+    def sleep(s, sleepsecs, siglistener):
         s.gettf().setcolor('red')
         s._msg("Next sync in %d:%02d" % (sleepsecs / 60, sleepsecs % 60))
-        BlinkenBase.sleep(s, sleepsecs)
+        return BlinkenBase.sleep(s, sleepsecs, siglistener)
             
 if __name__ == '__main__':
     x = Blinkenlights(None)
diff -Naur offlineimap.cleaned/ui/Noninteractive.py 
offlineimap/ui/Noninteractive.py
--- offlineimap.cleaned/offlineimap/ui/Noninteractive.py        2008-08-23 
08:28:19.000000000 -0400
+++ offlineimap/offlineimap/ui/Noninteractive.py        2008-08-23 
08:32:09.000000000 -0400
@@ -33,10 +33,10 @@
             warntxt = 'warning'
         sys.stderr.write(warntxt + ": " + str(msg) + "\n")
 
-    def sleep(s, sleepsecs):
+    def sleep(s, sleepsecs, siglistener):
         if s.verbose >= 0:
             s._msg("Sleeping for %d:%02d" % (sleepsecs / 60, sleepsecs % 60))
-        UIBase.sleep(s, sleepsecs)
+        return UIBase.sleep(s, sleepsecs, siglistener)
 
     def sleeping(s, sleepsecs, remainingsecs):
         if sleepsecs > 0:
diff -Naur offlineimap.cleaned/ui/UIBase.py offlineimap/ui/UIBase.py
--- offlineimap.cleaned/offlineimap/ui/UIBase.py        2008-08-23 
08:28:19.000000000 -0400
+++ offlineimap/offlineimap/ui/UIBase.py        2008-08-23 08:32:26.000000000 
-0400
@@ -19,6 +19,7 @@
 import offlineimap.version
 import re, time, sys, traceback, threading, thread
 from StringIO import StringIO
+from Queue import Empty
 
 debugtypes = {'imap': 'IMAP protocol debugging',
               'maildir': 'Maildir repository debugging',
@@ -324,7 +325,7 @@
 
     ################################################## Other
 
-    def sleep(s, sleepsecs):
+    def sleep(s, sleepsecs, siglistener):
         """This function does not actually output anything, but handles
         the overall sleep, dealing with updates as necessary.  It will,
         however, call sleeping() which DOES output something.
@@ -334,7 +335,12 @@
 
         abortsleep = 0
         while sleepsecs > 0 and not abortsleep:
-            abortsleep = s.sleeping(1, sleepsecs)
+            try:
+                abortsleep = siglistener.get_nowait()
+                # retrieved signal while sleeping: 1 means immediately 
resynch, 2 means immediately die
+            except Empty:
+                # no signal
+                abortsleep = s.sleeping(1, sleepsecs)
             sleepsecs -= 1
         s.sleeping(0, 0)               # Done sleeping.
         return abortsleep