Mysql – Lots of NULL processes in MySQL

MySQLprocess

Since last week without a particular reason my website started to drastically slow down. I logged into MySQL and run show processlist which resulted in something like this:

mysql> show full processlist;
+------+------+-----------+---------+---------+------+-------+------------------+
| Id   | User | Host      | db      | Command | Time | State | Info             |
+------+------+-----------+---------+---------+------+-------+------------------+
| 2047 | web  | localhost | myDB    | Sleep   |   81 |       | NULL             |
| 2049 | web  | localhost | myDB    | Sleep   |   81 |       | NULL             |
| 2050 | web  | localhost | myDB    | Sleep   |   81 |       | NULL             |
| 2052 | web  | localhost | myDB    | Sleep   |   81 |       | NULL             |
| 2053 | web  | localhost | myDB    | Sleep   |   81 |       | NULL             |
| 2054 | web  | localhost | myDB    | Sleep   |   81 |       | NULL             |
| 2055 | web  | localhost | myDB    | Sleep   |   81 |       | NULL             |
...
| 2066 | web  | localhost | myDB    | Sleep   |   80 |       | NULL             |
| 2067 | web  | localhost | myDB    | Sleep   |   80 |       | NULL             |
| 2178 | web  | localhost | myDB    | Sleep   |   76 |       | NULL             |
+------+------+-----------+---------+---------+------+-------+------------------+
131 rows in set (0.00 sec)

When I restart MySQL I have no processes which is to be expected. When I refresh my website and quickly check processes I can see some queries which disappear after a moment. Everything is perfect but then after some time out of the blue again I get lots of processes like shown above. This of course freezes my website.

They seem to disappear after some time but before the value of wait_timeout and then appear again after some time with different Id's.

Does anyone have a clue what could be causing this?

My server's specs:
CentOS 5
MySQL version: 5.1.33-log Source distribution
RAM 8GB

part of my.cnf

[mysqld]
default-character-set   = utf8
character_set_server    = utf8
default-collation       = utf8_unicode_ci
collation_server        = utf8_unicode_ci
ft_min_word_len         = 3
port                    = 5689
socket                  = /tmp/mysql.sock
skip-locking
skip-name-resolve
key_buffer_size         = 256M
group_concat_max_len    = 2048
max_allowed_packet      = 10M
table_cache             = 512
table_definition_cache  = 512
sort_buffer_size        = 10M
net_buffer_length       = 8K
read_buffer_size        = 40M
read_rnd_buffer_size    = 10M
myisam_sort_buffer_size = 8M
long_query_time         = 1
net_read_timeout        = 30
net_write_timeout       = 60
log_slow_queries        = /var/log/slowqueries
query_cache_type        = 1
query_cache_limit       = 5MB
query_cache_size        = 134217728
max_heap_table_size     = 24MB
tmp_table_size          = 24MB
expire_logs_days        = 99
max_connections         = 500
init-connect            = 'SET NAMES latin1'
thread_cache_size       = 4
wait_timeout            = 600

I hope that will help one of you to point me in the right direction. I'm out of ideas.

EDIT
The whole thing seems to be relevant to those MySQL bugs:
http://bugs.mysql.com/bug.php?id=8945
http://bugs.mysql.com/bug.php?id=6070
Unfortunately they're not solved 🙁

Best Answer

Check the values of your mysql server. Are you using default users in mysql? I'd also take a look at two things:

Be sure that only specific servers can connect to the mysql database. If you have a user that is connecting from % be sure that you have a good password (and good reason).
Be sure that someone isn't sending queries to your application outside of the code requirements that you have in place to run the application. Someone may be injecting data into your database without even visiting your site.

Keep track of you active connections for a while by paying close attention to netstat. The wait_timeout is an indication that someone/something has queried your server. Keeping close track will help you discover who they are. Once you do that if you need to quickly block them without making drastic changes, use iptables.

One last thing... enable mysql logging briefly. Your server will take a hit but you'll see exactly what's going on with mysql.

edit the my.cnf file and add (or update) this:

log             = /var/log/mysql/mysql.log

EDIT: I made the preceding suggestions because the NULL value in the info field indicates that no statements are being executed.

Related Solutions

MySQL Configuration – How to Bind MySQL Server to Multiple IP Addresses

No, there isn't (I just checked 1 hour ago). You can comment the bind-address in my.cnf:

Note: « 1 hour ago » is now more than 10 years ago.

#skip-networking
#bind-address                   = 127.0.0.1

If you want only 2 IPs, you will then have to use a firewall.

For MySql version 8.0.13 and above, you can specify a list of comma-separated IP addresses.

bind-address = 10.0.0.1,10.0.1.1,10.0.2.1

Relevant MySql documentation.

Remember to restart your MySQL instance after changing the config file.

Mysql – Too many MySQL processes

mysql     9804  0.0  0.6 58556 22960 pts/0   S    12:43   0:00  \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/myhostname.pid --skip-external-locking --port=3306 --socket=/var/lib/mysql/mysql.sock
mysql     9807  0.0  0.6 58556 22960 pts/0   S    12:43   0:00      \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/myhostname.pid --skip-external-locking --port=3306 --socket=/var/lib/mysql/mysql.sock
mysql     9808  0.0  0.6 58556 22960 pts/0   S    12:43   0:00          \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/myhostname.pid --skip-external-locking --port=3306 --socket=/var/lib/mysql/mysql.sock
mysql     9809  0.0  0.6 58556 22960 pts/0   S    12:43   0:00          \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/myhostname.pid --skip-external-locking --port=3306 --socket=/var/lib/mysql/mysql.sock
mysql     9810  0.0  0.6 58556 22960 pts/0   S    12:43   0:00          \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/myhostname.pid --skip-external-locking --port=3306 --socket=/var/lib/mysql/mysql.sock
[ ... repeated output truncated ... ]

They're not using 13 * 50mb of ram -- They're probably using something like 70mb total. Remember that that linux will share unmodified memory pages between processes, so if your server is freshly started most of that memory would all be shared. In fact, since mysql is threaded there will only probably be even less memory allocated per thread. If you're concerned about the memory usage of each mysql process, look at /etc/mysql/my.cnf, and look at the variables in the mysqld section:

key_buffers
thread_stack
thread_cache_size
max_connections
query_cache_limit
query_cache_size

Be warned though, that those are very powerful variables to tune, and you can easily kill your mysql performance setting them too low, or waste memory that could be used elsewhere by setting them too high.

One easy starting point to figure out the best way to tune your mysql instance is to run your app for a little bit, then run the mysqltune script, which will analyze your performance counters, then produce a recommendation on what you should change in your server config.

Best Answer

Related Solutions

MySQL Configuration – How to Bind MySQL Server to Multiple IP Addresses

Mysql – Too many MySQL processes

Related Topic