IPC and the GIL

As recently described excelently, threads are ugly beasts, waiting to get you when you let your guard down (and eventually you will). Yes, that means that I should really get my head round the asynchat module and stop using ThreadingMixIn with the SocketServer, but that's not the point of this post.

Inter process communication, aka IPC, is way safer and scalable when you want to distribute the work between many processors. But what bothers me is the complete lack of a nice support for this in the stdlib, while there is an excellent threading module.

So what options are there?

You could stay inside the stdlib and construct something out of pickle and sockets, but that isn't amazing as you'll have to write a lot of boiler plate code that will have it's own bugs. Leaving the stdlib you could probably replace pickle with json, but appart from being able to talk with non-python code you're not really better off.

There is of course the excellent omniORBpy if you grok CORBA. But the python mapping of CORBA was written by C++ programmers, so despite being the best CORBA mapping available it is still not the most lovely thing to work with as python developer. And CORBA is a rather heavy weight approach that is far from applicable in all situations. XML-RPC is another internet-aware IPC option that even has support in the stdlib, but once more the overhead when you're just on a local machine and want to use 2 CPUs is silly. Twisted must surely have some solution too, but I don't really know that and again I think it will be rather heavy-weight. Lastly I feel obliged to mention SOAP in this paragraph too, but can't help to shrudder when thinking about it.

I guess what I'm looking for is some simple, light weight and scalable module that does something similar to how Erlang passes messages around. Maybe all I'm asking for is a module that ties up subprocess and pickle and lives under the IPC section in the stdlib. But it would be nice if it could also transparantly stream the data accross a socket so you can also run on multiple hosts if you want (maybe in a separate module though). Having a module like that around would obsolete the need for boilerplate code as well as establishing some sort of "best practice" IPC.

Anyways, I think this is more like me wondering what people use or would want to have for their IPC in python. Anyone?