Python Architecture – Designing Applications with Many Small Scripts

Architecturemessage-queuepython

I am building an application which, at the moment, consists of many small Python scripts.

Each Python script processes items from one Amazon SQS queue. Emails come into an initial queue and are processed by a script, and typically the script will do a small unit of processing (for example, parse email and store some database fields), then an item will be placed on the next queue for further processing, until eventually the email has finished going through the various scripts and queues.

What I like about this approach is that it is very loosely coupled.

However, I'm not sure how I should implement live. Should I make each script a daemon which is constantly polling its inbound queue for things to do? Or should there be some overarching orchestration program or process? Or maybe I should not have lots of small Python scripts but one large application?

Specific questions:
How should I run each of these scripts – as a daemon with some sort or restart monitor to restart them in case they stop for any reason? If yes, should I have some program which orchestrates this?

Or is the idea of many small script not a good one, would it make more sense to have a larger python program which contains all the functionality and does all the queue polling and execution of functionality for each queue?
What is the current preferred approach to daemonising Python scripts?

Broadly I would welcome any comments or opinions on any aspect of this.

thanks

Best Answer

If this application needs to stay running for a long time then build in resilience but kill it at random times (the monitor too - but less frequently), and have a monitor restart it. That way you are constantly exercising the monitor and restart capability of the system so when some true system failure shafts your system it will be more likely to recover smoothly.

Related Solutions

Python – One Repository or Multiple for Many Small Scripts?

Unless there exists a specific reason for you to think that each one of them deserves an invidual repo (Will they grow a lot? Probably not!) it seems more reasonable to put them all in one repo, and save yourself the trouble of cloning all of them from 20 repos.

Keeping each one in a separate repo seems like the path of creating a problem where a problem does not exist.

Don't create extra work for yourself (and others).

Python Bash – Encoding a Bash Script for Use in Python

I'd use the first option; embed the bash script in a multi-line string in the Python script, then write that out and use the subprocess module to run the script:

import tempfile
import subprocess

script_one = '''\
echo "Hello world!"
echo "This is a bash script. :-)"
'''

def run_script(script):
    with tempfile.NamedTemporaryFile() as scriptfile:
        scriptfile.write(script)
        scriptfile.flush()
        subprocess.call(['/bin/bash', scriptfile.name])

run_script(script_one)

The run_script() function takes a bash script source, writes it to a temporary file, executes it, and automatically cleans up the temporary file when done.

Demo:

$ python /tmp/scripttest.py 
Hello world!
This is a bash script. :-)

Python decorators are of no help here; there is no point in looking at those; decorators are simply Python functions that get to wrap other Python functions.

If the script is large, you could try compressing it with zlib, base64 encode it, then embed that into the Python script file; using .decode('base64') and passing the result to the zlib module first gives you a decompressed string again. The ZLIB compression reduces the size but results in binary data, always a little harder to embed without having to deal with proper escaping and line lengths. The Base64 encoding turns the binary data into multi-line ASCII-safe data again, albeit at the price of 25% more bytes:

>>> import zlib
>>> zlib.compress(script_one).encode('base64')
'eJxLTc7IV1DySM3JyVcozy/KSVFU4koFi4VkZBYrAFGiQlJicYZCcXJRZkGJnoKVrqYSFwDnexDe\n'
>>> print """script_one = '''\\\n{}'''""".format(_)
script_one = '''\
eJxLTc7IV1DySM3JyVcozy/KSVFU4koFi4VkZBYrAFGiQlJicYZCcXJRZkGJnoKVrqYSFwDnexDe
'''

Note that for this short script sample, this increased the size, from 54 characters to 77, but for longer scripts you should see a decent size reduction.

Decompression is then as simple as zlib.decompress(script_one.decode('base64'))

Best Answer

Related Solutions

Python – One Repository or Multiple for Many Small Scripts?

Python Bash – Encoding a Bash Script for Use in Python

Related Topic