Re: Darwin patches

[Michael Witten <mfwitten@xxxxxxx>]

On 28 Sep 2008, at 2:25 PM, John Goerzen wrote:

> Michael Witten wrote:
>> On 26 Sep 2008, at 10:43 AM, Michael Witten wrote:
>>
>>> Frankly, the original code doesn't even make much sense, and
>>> moreover it's not forward compatible with python 2.6, and
>>> furthermore:
>>>
>>>  DeprecationWarning: socket.ssl() is deprecated.
>>>
>>> So, this commit is just a temporary fix.
>>
>> I see that recently the following patch was included:
>
> Are you saying that when MacOS X switches to Python 2.6, the entire
> patch will be unnecessary?

Sorry, I should have been more clear.

I was having MemoryError exceptions whilst trying to download
"large" emails (because the emails probably have attachments).

After doing some research, I found that the problem lies with
Mac OS X's memory allocation and that the solution is available
in python 2.6 (rc2 in my case); it is roughly the same fix
that has been included at a higher level in OfflineIMAP's repo
(but I hadn't cloned the repo yet to find that out; it's in
commit 10c2b6fbaa333a7fe4c58e5ef6b1186fdb5f8d7e).

After installing python 2.6rc2 and then OfflineIMAP (from the
release tar ball), I got a runtime error caused by the else
block of the following code in OfflineIMAP:

> if sys.version_info[0] <= 2 and sys.version_info[1] <= 2:
>     self.sslobj = socket.ssl(self.sock, self.keyfile, self.certfile)
> else:
>     self.sslobj = socket.ssl(self.sock._sock, self.keyfile, self.certfile)

The last line makes a reference to "self.sock._sock", which has
been handled (generously) by the python internals so far, but
no longer in 2.6.

Take Python 2.5's ssl.py for instance:

> def ssl(sock, keyfile=None, certfile=None):
>     if hasattr(sock, "_sock"):
>         sock = sock._sock
>     return _realssl(sock, keyfile, certfile)

Yet, Python 2.6 tries to clean things up and do things right:

> def ssl(sock, keyfile=None, certfile=None):
>     # we do an internal import here because the ssl
>     # module imports the socket module
>     import ssl as _realssl
>     warnings.warn("socket.ssl() is deprecated.  Use ssl.wrap_socket() 
> instead.",
>                   DeprecationWarning, stacklevel=2)
>     return _realssl.sslwrap_simple(sock, keyfile, certfile)

As you can see, it no longer permits the assumption that it is
acceptable to use _sock directly (at least at this level---and
it turns out none of the lower levels allow it either).

I imagine python 2.5 is trying to wean people away from using
_sock explicitly, so they handle it just in case there are a few
stragglers out there, and it would appear that OfflineIMAP is going
further by trying to remain backward compatible with some version
of ssl() that perhaps *expects* _sock, but even so the OfflineIMAP
code would be wrong; the condition is flaky besides the fact that
the true and false clauses seem reversed.

Basically, the OfflineIMAP code should just be the first block:

> self.sslobj = socket.ssl(self.sock, self.keyfile, self.certfile)

and that is precisely what my patch makes it.

Unfortunately, socket.py's ssl() is deprecated, which is why I
say that it's a temporary fix. Eventually it won't just be
deprectated---it will be GONE!

>> Also, you guys seem to be wasting a lot of cycles every read():
>>
>>> if (system() == 'Darwin') and (size>0) :
>>
>> Shouldn't that information be cached somewhere? Why not bind 'read'  
>> to a particular implementation at program startup, so that there is
>> virtually no overhead.
>
> A patch would be welcome on that.  I have no MacOS box on which to  
> test.

Well, maybe I'll do it; unfortunately, there are a couple of other
projects for which I'm working on patches at the moment, and I've
only just begun shaping my email work flow around tools such as
OfflineIMAP.

P.S.

Speaking of Mac OS X, you list the following:

> An informal benchmark yields these results for my setup:
> 
>     10 minutes with MacOS X Mail.app "manual cache"
>     5 minutes with GNUS agent sync
>     20 seconds with OfflineIMAP 1.x
>     9 seconds with OfflineIMAP 2.x
>     3 seconds with OfflineIMAP 3.x "cold start"
>     2 seconds with OfflineIMAP 3.x "held connection"

However, Mail.app is probably indexing those emails, finding out if
senders are online, running filters, and maybe even getting the
attachments at the same time; moreover, Mail.app is probably doing
all of this grunt work with code written in C. Therefore, there is
just no way that this benchmark can be taken seriously, even as an
informal benchmark. As I'm sure you know, python is MUCH slower than
natively compiled code, so there is no way OfflineIMAP could possibly
be faster at anything similarly written in C, especially when all of
the python library code that OfflineIMAP uses is in itself written in
Python.

I don't no much about GNUS either, but I imagine it's written in
emacs lisp, which is supposedly incredibly inefficient.

Basically, my point is that OfflineIMAP may be speedy for its task,
but in all probabiliy, it's actually laughably inefficient (that's
not meant to incite anger, just thought).