Load Balancer – What Does It Return?

When you use a software load balancer or caching proxy, that load balancer or caching proxy is a bottleneck – but it's a much wider bottleneck than before.

Assume we have an application server that can only handle 50 requests per second, since it has to query the database and do all kinds of expensive stuff. How can we scale?

If we add a second application server, that won't help. We need a load balancer that distributes load between these application servers. Because the load balancer itself does nothing except passing through connections, let's say that it can handle 1000 requests per second – that's 20× as much, and my actual capacity is 2·50=100 requests per seconds since I have two application servers. I can now add further application servers if the need arises.

In most cases, this is absolutely sufficient, and it's unlikely that you'll push the load balancer to its limits. If you want to scale beyond this load balancer, you cannot add another software load balancer in front of the existing load balancer, and you will have to beef up the hardware (or switch to a balancer with better performance).

If you get beyond the capacity for a single load balancer (e.g. when you hit the millions of daily users range) and if you need to maintain servers in multiple geographic locations, then you can use a DNS server that resolves your domain name to different IP addresses. Content Delivery Networks use this technique to distribute load through a large number of distributed proxies. Instead of going that route yourself, it will likely be much easier to put your servers behind an existing commercial CDN that handles all that for you.

Why not just broadcasting the message to all the nodes, and then let the nodes chose the messages they want to forward through WebSockets to clients?

If you have a lot of messages (that much that the impact on the network becomes noticeable), using the user ID in the routing key may be the easy way to reduce the traffic between the MQS and the nodes.

Imagine you have a chat application with rooms and users. Using a topic exchange, you may use routing keys such as chat.room123.user456 when transmitting a message of user 456 posted in room 123. You can consume those messages by having a queue with a binding key chat.room123.*, which will listen to all messages in a specific room. You may also want to consume the messages posted by a specific user, independently of the room, using the binding key chat.*.user456. In order to scale it, you may have some servers handling some rooms, and other servers handling some users. For instance, a given server may have WebSockets connections to users 17, 41 and 48, and have the binding keys chat.*.user17, chat.*.user41 and chat.*.user48.

The tricky part here might be the handling of the case where the WebSockets connection is dropped, and the client reconnects to a different node. Sticky sessions may solve this issue. Aside that, handling errors and loss of WebSockets connection is not different from any other scenario where messages from a message queue service should be preprocessed and sent to a third party.