Communicating between one Read and one Write process in python

I am looking for a way to pass information safely (!) between two processes of python scripts.

One process reads only, and the other writes only. The data I need to pass is a dictionary.

The best solution I found so far is a local SQL server (since the data itself is kind of a table) and I assume SQLite will handle the transaction safely.

Is there a better way, maybe a module to lock a file from being read while it is written to and vice versa?

I am using linux ubuntu 11.10, but a cross platform solution will be welcome.

Answers


For one-way communication you could use e.g. multiprocessing.Queue or multiprocessing.queues.SimpleQueue.

Shared memory is also an option using multiprocessing.Array. But you'd have to split up the dictionary in at least two arrays (keys and values). This will only work well if all the values are of the same basic type.

The good point about both multiprocessing.Queue and multiprocessing.Array is that they are both protected by locks internally, so you don't have to worry about that.


To transfer data between processes, you can use pipes and sockets. Python uses pickling to convert between objects and byte streams.

To transfer the data safely, you need to make sure that the data is transferred exactly once. Which means: The destination process needs to be able to say "I didn't get everything, send it again" while the sender needs some form of receipt.

To achieve this, you should add a "header" to the data which gives it a unique key (maybe a timestamp or a hash). Both receiver and sender can then keep a list of those they have sent/seen to avoid processing data twice.


Need Your Help

How to create typedef for ``boost::array_view`` in a templated class

c++ boost boost-multi-array

I want to have a dimensional agnostic template (usefull for both 3d and 4d), most of operations will be performed on submatrices with first dimension stripped.

Why can't I focus a control that I reveal with slideDown()?

jquery

I want to reveal a div with an input inside when you click a button, and set its focus.

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.