I finally found the setting that was really limiting the number of connections: net.ipv4.netfilter.ip_conntrack_max. This was set to 11,776 and whatever I set it to is the number of requests I can serve in my test before having to wait tcp_fin_timeout seconds for more connections to become available. The conntrack table is what the kernel uses to track the state of connections so once it's full, the kernel starts dropping packets and printing this in the log:
Jun 2 20:39:14 XXXX-XXX kernel: ip_conntrack: table full, dropping packet.
The next step was getting the kernel to recycle all those connections in the TIME_WAIT state rather than dropping packets. I could get that to happen either by turning on tcp_tw_recycle or increasing ip_conntrack_max to be larger than the number of local ports made available for connections by ip_local_port_range. I guess once the kernel is out of local ports it starts recycling connections. This uses more memory tracking connections but it seems like the better solution than turning on tcp_tw_recycle since the docs imply that that is dangerous.
With this configuration I can run ab all day and never run out of connections:
net.ipv4.netfilter.ip_conntrack_max = 32768
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_max_orphans = 8192
net.ipv4.ip_local_port_range = 32768 61000
The tcp_max_orphans setting didn't have any effect on my tests and I don't know why. I would think it would close the connections in TIME_WAIT state once there were 8192 of them but it doesn't do that for me.
Your question is short on details and long on hand-waving, but it sounds like your initial thinking is a pretty sound start. Your app sounds pretty similar to the Zenoss monitoring suite, which uses essentially the same load-distribution architecture to scale up: Multiple monitoring hosts sharing the data collection workload, with a single admin interface, and a database on either the admin host or a separate system.
If your bottleneck is at point #1 (devices sending data to your server), splitting those tasks across a second machine should carve out some room for load growth. The biggest implementation obstacle is usually how to manage tasks across multiple Django servers. Celery, a distribued task queue engine, is probably the best option at the moment. It was originally designed around Django, which is good for you, and it has very active and helpful community of developers and users.
If points #2 and #4 are your current limitation, though, you're probably talking about database scalability. This is just a hard problem, in general: There is no code-transparent, load-neutral, and cheap way to scale up database capacity.
If you only need to get more database "read" IO capacity, replication will probably do the trick. Postgres supports replication using an external tool called Slony-I. The is single-master replication, with multiple read-only "slave" hosts that get fed copies of statements executed on the master. All of your app's writes (UPDATE, INSERT, DELETE...) go through the single master host, but you distribute your reads (SELECT...) across the master and all of the slaves.
The code modifications needed for distributed reads are usually pretty straightforward. Django recently added support for replicated databases, which I haven't used, but it's supposed to be pretty good.
If you need more database write IO capacity, sharding will probably work. Each host keeps a separate, unique chunk of each database table. The DB clients use a deterministic function decides where any given record should reside, so the load distribution is effectively stateless and can scale up to huge numbers of DB servers. Django's new multi-database support (same link as above) also supports sharding. You'll need some code changes, the pain should be limited.
Also, I want to mention Memcached, which seems to be part of just about every highly scalable web application on the Internet, today (Facebook, Google, Twitter...). A good caching implementation can cut your database requirements to a fraction of their original size, by converting expensive, slow DB lookups into cheap, fast cache lookups. Django has supported Memcached integration for quite a while, now.
I realize none of this is too specific, but it should give you a pretty good starting place for working out the details. Good luck with your project.
Best Answer
I don't think your 60k clients are the actual problem. You will more likely have problem with not enough file descriptors, but that should be easy to fix as part of OS configuration.
Here's why connections will not be your problem. Each connection is characterised by its source ip address, source port, destination ip address and destination port. Inside the network stack this quadruple is used to match packets to file descriptors(each file descriptor represents a connection). Your server has fixed destination ip address and destination port (your server is destination for their client) but source ip address and source port are variable. Port is a 16bit number therefore maximum number of connections from one client is 64K. IPv4 address is a 32 bit number which gives you 4,294,967,296 possible source addresses. Doing some basic maths, your server could have 64K * 4,294,967,296 connections mapped to a single source ip and port.
This is why you will more likely have problem with maximum number of open file descriptors then the number of clients.