Opening Thoughts

How did you come to the conclusion that some parts of the system would fare better in another language? Are you suffering performance problems? How severe are those problems? If it can be faster, is it essential that it is faster?

Single-Thread Asynchrony

There are several questions and other web resources that already deal with the differences, pros, and cons of single-thread asynchrony vs. multi-thread concurrency. It’s interesting to read about how Node.js's single-thread asynchronous model performs when I/O is the major bottleneck, and there are many many requests being serviced at once.

Twisted, Tornado, and other asynchronous models make excellent use of a single-thread. Since a lot of web programming has lots of I/O (network, database, etc.), the time spent waiting on remote calls adds up significantly. That is time that could be spent doing other things—like kicking off other database calls, rendering pages, and generating data. The utilisation of that single-thread is extremely high.

One of the greatest benefits of single-thread asynchrony is that it uses much less memory. In multi-thread execution, each thread requires a certain amount of reserved memory. As the number of threads increases, so does the amount of memory required just for the threads to exist. Since memory is finite, it means there are bounds on the number of threads that can be created at any one time.

---

Example

In the case of a web server, pretend each request is given its very own thread. Say 1MB of memory is required for each thread, and the web server has 2GB of RAM. This web server would be capable of processing (approximately) 2000 requests at any point in time before there just isn't enough memory to process any more.

If your load is significantly higher than this, requests are going to take a very long time (when waiting for older requests to complete), or you’re going to have to throw more servers into the cluster to expand the number of concurrent requests possible.

---

Multi-thread Concurrency

Multi-thread concurrency instead relies on executing several tasks at the same time. That means that if a thread is blocked waiting on a database call to return, other requests can be processed at the same time. Thread utilisation is lower, but the number of threads executing is much larger.

Multi-thread code is also much harder to reason about. There are issues with locking, synchronisation, and other fun concurrency problems. Single-thread asynchrony doesn't suffer from the same problems.

Multi-thread code is much more performant for CPU intensive tasks, however. If there exists no opportunities for a thread to “yield”—such as a network call that would normally block—a single-thread model just isn't going to have any concurrency whatsoever.

Both May Coexist

There's of course overlap between the two; they are not mutually exclusive. For instance, multi-thread code can be written in a non-blocking way, to better utilise each thread.

The Bottom Line

There are many other issues to consider, but I like to think about the two like this:

• If your program is I/O bound, then single-thread asynchrony is probably going to work quite well.
• If your program is CPU bound, then a multi-thread system will probably be best.

In your particular case, you need to determine what kind of asynchronous work is being completed, and how often those tasks arise.

• Do they occur on every request? If so, memory is probably going to become an issue as the number of requests rise.
• Are these tasks ordered? If so, you’re going to have to consider synchronisation if using multiple threads.
• Are these tasks CPU intensive? If so, is a single-thread able to keep up with the load?

There's no simple answer. You must consider what your use cases are, and design accordingly. Sometimes an asynchronous single-thread model is better. Other times, using a number of threads to achieve massive parallel processing is required.

Other Considerations

There are other issues you need to consider also, rather than just the concurrency model you choose. Do you know Erlang or Clojure? Do you think you'd be capable of writing safe multi-thread code in one of these languages such that you improve the performance of your application? Is it going to take a long time to get up to speed in one of these languages, and will the language you learn benefit you in the future?

How about the difficulties associated with communication between these two systems? Will it be overly complex maintaining two separate systems in parallel? How will the Erlang system receive tasks from Django? How will Erlang communicate those results back to Django? Is performance significant enough a problem that the added complexity is worth it?

Final Thoughts

I've always found Django to be quick enough, and it is used by some very heavily trafficked sites. There are several performance optimisations you can make to increase the number of concurrent requests and response time. Admittedly, I haven’t done anything with Celery thus far, so the usual performance optimisations are probably not going to solve any issues you may be having with these asynchronous tasks.

Of course, there’s always the suggestion of throwing more hardware at the problem. Is the cost of provisioning a new server cheaper than the development and maintenance cost of an entirely new subsystem?

I've asked far too many questions at this point, but that was my intention. The answer is not going to be easy without analysis and further detail. Being able to analyse the problems comes down to knowing the questions to ask, though…so hopefully I've helped on that front.

My gut feeling says that a rewrite in another language is unnecessary. The complexity and cost will probably be too great.

---

Edit

Response to Follow-Up

Your follow-up presents some very interesting use cases.

1. Django working outside HTTP requests

Your first example involved reading NFC tags, then querying the database. I don’t think that writing this part in another language will be that useful to you, simply because querying the database or an LDAP server is going to be bound by network I/O (and potentially database performance). On the other hand, the number of concurrent requests will be bound by the server itself, since each management command will be run as its own process. There will be setup and teardown time that impacts performance, since you aren't sending messages to an already running process. You will, however, be able to send multiple requests at the same time, since each will be an isolated process.

For this case, I see two avenues you can investigate:

1. Ensure that your database is capable of handling multiple queries at once with connection pooling. (Oracle, for example, requires that you configure Django accordingly 'OPTIONS': {'threaded':True}.) There may be similar configuration options at the database level or Django level that you can tweak for your own database. No matter what language you write your database queries in, you will have to wait for this data to return before you can light up the LEDs. The performance of the querying code can make a difference though, and the Django ORM isn't lightning fast (but, usually fast enough).
2. Minimise setup/teardown time. Have a constantly running process, and send messages to it. (Correct me if I’m wrong, but this is what your original question is actually focusing on.) Whether this process is written in Python/Django or another language/framework is covered above. I don’t like the idea of using management commands so frequently. Is it possible to have a small piece of code running constantly, that pushes messages from the NFC readers onto the message queue, which Celery then reads and forwards to Django? The setup and teardown of a small program, even if it’s written in Python (but not Django!), should be better than starting and stopping a Django program (with all of its subsystems).

I’m unsure what web server you’re using for Django. mod_wsgi for Apache lets you configure the number of processes and threads within the processes that service requests. Be sure to tweak the relevant configuration of your web server to optimise the number of serviceable requests.

2. “Message-passing” with Django signals

Your second use case is also fairly interesting; I’m not sure if I have the answers for that. If you’re deleting model instances, and wish to operate on them later, it might be possible to serialize them JSON.dumps and then deserialize JSON.loads. It’ll be impossible to fully recreate the object graph later on (querying related models), since related fields are lazy loaded from the database, and that link will no longer exist.

The other option would be to somehow mark an object for deletion, and only delete it at the end of the request/response cycle (after all signals have been serviced). It might require a custom signal to implement this, rather than relying on post_delete.

True multiprocessing Python programs (as opposed to multi-threaded Python programs which have to deal with the dreaded GIL) are no different from those in any other language. They all have the same basic challenges:

1. Task assignment and result reporting. Even if they are mostly working on independent datasets, they normally have to go back to the master thread to report results and get new data to work on. This can be a choke point.
2. Race conditions. Processes are trying to use a one-at-a-time resource and they need to use mutex (or something similar) to keep from stepping all over each other's data. Failure to protect these kinds of resources can lead to really, really painful debug sessions.
3. Sequentiality. Sometimes you are trying to make something parallel that isn't. The various processes end up waiting for each other to do something and the end result is that you have, for all intents and purposes, taken a sequential program, made it parallel, and it still ends up executing in linear time (or worse).

Although there are development methods that try to avoid each of these problems, at the end of the day you really need to think about what you are doing. I recommend heavy stress testing — far beyond anything you think might happen in real life — so that you have a good chance of hitting those Windows of Opportunity and blowing up in development as opposed to in the middle of a major demo or during production.

We used to use microsecond-timestamped log files and then created a color-coded log viewing app so we could try to visualize what was happening between N process running on M processors. We also tried (and mostly succeeded) in creating a system that would drive off the log files to recreate the context of the crash.

But the best tool is good design and really wicked, nasty people who try to blow your app out of the water. (Hi, dbell!)