I think that the bottleneck is the disc. Here are my reasons for this:
You have a pretty busy web server.
Zabbix is slow, I suspect to be reads from the disk (can be from the network too).
Run again the strace, and find the file descriptor in Zabbix
Then find if the file descriptor is a file or a socket:
ls -l /prod/<PID_of_straced_process>/fd/<FD_from_strace>
EDIT1:
You should not change the TIME_WAIT timeouts. The problem with small HTTP keep-alive, or with no HTTP keep-alive is that you increase the latency and bandwidth. Instead you should increase a little bit the HTTP keep-alive and install/enable SPDY.
EDIT2:
Use dstat -ta 10
and compare the first line with the rest. The first line is the average since boot. Next lines are 10 seconds average (the last parameter).
EDIT3:
Check if you do not have packets lost, use something like smokeping to monitor the server and the website from outside your network. You have a significant number of connections in CLOSING, FIN_WAIT1, FIN_WAIT2, SYN_RECV, LAST_ACK. I think your network is congested or you have a lot of short lived connections (confirmed by the high TIME_WAIT/ESTABILISHED ratio). See: http://en.wikipedia.org/wiki/Transmission_Control_Protocol#Protocol_operation
If the agent is deployed active, and the FW allows each agent to connect back to the server on the correct port, you shouldn't have a problem getting alerts.
But if you have several agents behind a FW you should deploy a Zabbix Proxy, as Jacob mentioned.
Best Answer
How do you install Zabbix? You could use some configuration management tool (Puppet, Chef, etc.) and create a template for the conf file. Then just pass in the Zabbix server. This gives you a consistent and flexible solution.