Zabbix – Some of the monitored items don’t refreh

monitoringperformance-monitoringzabbix

I'm experiencing a strange issue with Zabbix monitoring a MySQL server.

Most of the data from the server such as MySQL queries per second and MySQL uptime , Buffers memory etc. update nicely while some data like CPU iowait time (avg1) , Host local time ,MySQL number of threads and other items which were monitored in the past has last check time of about a week ago.

I can't find any logic in this, for example Mysql number of threads and Mysql queries per second are obtained in a similar way so it does not make sense one of them is monitored and one is not.

Please help- how can I fix this?

Update – I used zabbix_get from the zabbix server to check one of the items on the zabbix client and it works so the problem must be on the zabbix server side

Best Answer

Here are some more places to look for clues...

Item List

Go to Configure -> Hosts -> (the host or template) -> Items to get a list of items that the host is checking. The right-most column will either be a green checkmark or a red x. If the item in question is a red x, you can hover your mouse over it to see the error.

Queue

Go to Administration -> Queue to see how many items are "stuck" and are not updating. Is it just a couple or do you have a systemic problem with lots of items?

Server Log

As someone else said, check your zabbix_server.log file. You may also want to temporarily crank up the debug output by editing the zabbix_server.conf file.

Check those things and update your question with the findings so we can help you further.

Related Solutions

Linux – What Do the Colors in htop Status Bars Mean

Hitting F1 or h will show you the key. But for reference, the default colors are:

CPU:

Blue = Low priority threads
Green = Normal priority threads
Red = Kernel threads

Memory:

Green = Used memory
Blue = Buffers
Yellow/Orange = Cache

There are a couple of different color-schemes available, you can see them through hitting F2.

Zabbix agent – high CPU usage

I think that the bottleneck is the disc. Here are my reasons for this:

You have a pretty busy web server. Zabbix is slow, I suspect to be reads from the disk (can be from the network too).

Run again the strace, and find the file descriptor in Zabbix

Then find if the file descriptor is a file or a socket:

ls -l /prod/<PID_of_straced_process>/fd/<FD_from_strace>

EDIT1:
You should not change the TIME_WAIT timeouts. The problem with small HTTP keep-alive, or with no HTTP keep-alive is that you increase the latency and bandwidth. Instead you should increase a little bit the HTTP keep-alive and install/enable SPDY.

EDIT2: Use dstat -ta 10 and compare the first line with the rest. The first line is the average since boot. Next lines are 10 seconds average (the last parameter).

EDIT3: Check if you do not have packets lost, use something like smokeping to monitor the server and the website from outside your network. You have a significant number of connections in CLOSING, FIN_WAIT1, FIN_WAIT2, SYN_RECV, LAST_ACK. I think your network is congested or you have a lot of short lived connections (confirmed by the high TIME_WAIT/ESTABILISHED ratio). See: http://en.wikipedia.org/wiki/Transmission_Control_Protocol#Protocol_operation