Nagios & Temperature

For DNS/DHCP, you can use the existing Nagios plugins from your Nagios server to check that they're DNS-ing and DHCP-ing properly. If your cluster's running on SLES, you should be able to install the Nagios plugins and NRPE on the SLES servers, then monitor that with check_nrpe on the Nagios box. That will let you check mount points, running processes, etc.

I've always believed the administration advantage of SSH (I use push_check) outweighs any additional load. Modern CPUs are so fast that the cost of encrypting a handful of bytes is pretty minimal, so it comes down to running two processes (SSH and the check script) vs one (check script fired off by NRPE).

For check scripts written in an interpreted language, I would expect the overhead of firing up the interpreter (Perl, Python, Bash) to exceed the CPU cost of starting an SSH session. Given modern CPUs, your machines are more likely to be disk or memory limited rather than CPU limited.

Provided your Nagios machine is coping -- it has to set up 20 SSH connections every second -- I would err on the side of convenience.

Not really an answer to your question, more of an argument that life is too short to worry about it :)