Ubuntu – Can’t connect to MySQL from another server

I have two servers on the same subnet.

• 192.168.100.1 (mysql)
• 192.168.100.2 (php)

MySQL seems to be running fine on it's box but I can't seem to connect to it. These are both ubuntu boxes and iptables and ufw are not running. The following commands are all from the MySQL box itself. None of these work from the .100.2 (php) box.

my.cnf

#bind-address           = 127.0.0.1
bind-address = 192.168.100.1

netstat says this is true:

root@192.168.100.1 /etc/mysql $ netstat -plutn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      8583/sshd       
tcp        0      0 192.168.100.1:3306      0.0.0.0:*               LISTEN      30654/mysqld    
tcp        0      0 0.0.0.0:2828            0.0.0.0:*               LISTEN      29946/monit     
tcp        0      0 0.0.0.0:1167            0.0.0.0:*               LISTEN      23175/cdp       
tcp6       0      0 :::22                   :::*                    LISTEN      8583/sshd  

telnet says it's good

root@192.168.100.1 /etc/mysql $ telnet 192.168.100.1 3306
Trying 192.168.100.1...
Connected to 192.168.100.1.
Escape character is '^]'.
[
5.5.40-0ubuntu0.12.04.1&KM~Se$!u??L\hw=gp{<Uvrmysql_native_password

!#08S01Got packets out of orderConnection closed by foreign host.

mysql client says it's good:

root@192.168.100.1 /etc/mysql $ mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.

When I try to connect from .100.2 I get nothing back from the connection until it timeouts:

root@192.168.100.2 /etc/php5/fpm $ mysql -uroot -p -h 192.168.100.1 --connect-timeout 4
Enter password: 
ERROR 2003 (HY000): Can't connect to MySQL server on '192.168.100.1' (4)

I can ping .100.1 from the .100.2 box though with sub ms response times.

Here is the MySQL Grant settings:

mysql> SELECT user,host from mysql.user;
+------------------+--------------+
| user             | host         |
+------------------+--------------+
| root             | 127.0.0.1    |
| root             | 192.168.1.%  |
| root             | 192.168.100.%|
| root             | 192.168.200.%|
| root             | ::1          |
| debian-sys-maint | localhost    |
| example          | localhost    |
| o3backup         | localhost    |
| root             | localhost    |
| root             | voice-mysql  |
+------------------+--------------+
10 rows in set (0.00 sec)

I added a new grant for everywhere and the problem persists.

mysql> show grants for 'root'@'%';
+--------------------------------------------------------------------------------------------------------------+
| Grants for root@%                                                                                            |
+--------------------------------------------------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY PASSWORD '...' |
+--------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Update

Turns out the system was somehow locked-down by changes made to system files I was unaware of. Apparently it was blocking certain network operations that aren't standard. This is honestly frustrating that the same people that gave me this box forgot all about this. I honestly have no idea what they did to it and even they cannot remember. Nevertheless, the answers listed here are the most thorough trouble-shooting guide for normal people though. It should have been one of these problems.