Some simple distributed computing components using only the python standard
library. The motivation for the project was to create a simple distributed
map-reduce style interface that can act as a drop-in replacement for python's
inbuilt map and the itertools.imap family.
myriad can be installed in the usual way with
python setup.py install
It should be installed on all computers that will do any processing.
myriad contains client and server components. The server manages input and
output queues, which clients can access. The basic pattern is to start a
server, push work to its input queues, launch clients, and finally collect
results from the server. An example is given in the myriad.components
module, which provides the myriad command-line entry point. Here's a
near-full example.
from myriad.components import MyriadServer
from random import randint
host = 'localhost'
port = '12345'
key = 'auth_key'
worker = ... # mysterious function that transforms inputs to outputs
# Create a server
server = MyriadServer(worker, port, key)
# Push some jobs to the server
for _ in xrange(5):
server.put(randint(1, 3))
# Start a client
subprocess.Popen(['myriad', '--client',
'--host', 'localhost', '--port', port, '--key', key])
# Collect results from server
for result in server:
print "Server got back '{}'".format(result)In the above we start a subprocess running myriad with the --client option.
Of course this line could be amended to, for example, submit clients to a
cluster. In common with similar packages, the function worker must be
pickleable.