From a purely performance perspective, let benchmarking make these decisions for you rather than assuming -- using a tool like httperf is invaluable when making architecture changes.
From an architectural philosophy perspective, I'm a little curious why you have both nginx and apache on the application servers. Nginx blazes at static content and efficiently handles most backend frameworks/technologies (Rails, PHP via FastFCGI, etc), so I would drop the final Apache layer. Once again, this comes from a limited understanding of the technologies that you're using, so you may have a need for it that I'm not anticipating (but if that's the case, you could always drop nginx on the app servers and just use apache -- it's not THAT bad at static content when configured properly).
Currently, I use nginx -> haproxy on load balancing servers and nginx on the app servers with much success. As Willy Tarreau stated, nginx and haproxy are a very fast combination, so I wouldn't worry about the speed of having both on the front-end, but keep in mind that adding additional layers increases complexity as well as the number of points of failure.
Update (28 Aug 2012): I now tend to use haproxyctl nowadays, which utilizes the methods described below.
I've fixed it after a little more research, for anyone else with the same issue:-
You can add a unix socket in your config, then interact with that (here are the possible commands).
To set up:
sudo nano /etc/haproxy/haproxy.cfg
In your 'global' section add in:
stats socket /etc/haproxy/haproxysock level admin
Restart your haproxy daemon (e.g. sudo service haproxy restart)
Now you need socat (if you don't have it, just apt-get install socat on Ubuntu).
Now all you need to do is fire off this command to take down a node:
echo "disable server yourbackendname/yourservername" | socat stdio /etc/haproxy/haproxysock
To bring it back up replace disable with enable in the command above.
Best Answer
Yes you can.
If the health check file is missing. Check the disable-on-404 option.