Ubuntu – I just removed a folder in /tmp/ and now apache/thesql is bugged. What just happened

MySQLUbuntu

So I just deleted a folder and a file in /tmp/ on my ubuntu server (12.10). Now everything related to MySQL is bugged. It also seems like the root mysql account I created has reset. When trying to log in using PHPMyadmin:

Connection for controluser as defined in your configuration failed.

And when typing in "MySQL" using SSH (putty)

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

What just happened and How do I fix it?

Also, I just checked the status and it's offline. Will not start back up. Job failed.

Just restarted the server as well as mysql.

stop: Unknown instance: start: Job failed to start


130219 22:17:42 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130219 22:18:01 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130219 22:18:01 [Note] Plugin 'FEDERATED' is disabled.
130219 22:18:01 InnoDB: The InnoDB memory heap is disabled
130219 22:18:01 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130219 22:18:01 InnoDB: Compressed tables use zlib 1.2.7
130219 22:18:01 InnoDB: Using Linux native AIO
130219 22:18:01 InnoDB: Initializing buffer pool, size = 128.0M
130219 22:18:01 InnoDB: Completed initialization of buffer pool
130219 22:18:01 InnoDB: highest supported file format is Barracuda.
130219 22:18:01 InnoDB: Waiting for the background threads to start
130219 22:18:02 InnoDB: 1.1.8 started; log sequence number 69234959 /usr/sbin/mysqld: Too many arguments (first extra is 'restart'). Use --verbose --help to get a list of available options
130219 22:18:02 [ERROR] Aborting

I am using Ubuntu Server.

When trying to restart mysql using sudo /etc/init.d/mysql restart

Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the stop(8) and then start(8) utilities,
e.g. stop mysql ; start mysql. The restart(8) utility is also available.
start: Job failed to start

MySQL error log:

130220  8:15:54 [Note] Plugin 'FEDERATED' is disabled.
130220  8:15:54 InnoDB: The InnoDB memory heap is disabled
130220  8:15:54 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130220  8:15:54 InnoDB: Compressed tables use zlib 1.2.7
130220  8:15:54 InnoDB: Using Linux native AIO
130220  8:15:54 InnoDB: Initializing buffer pool, size = 128.0M
130220  8:15:54 InnoDB: Completed initialization of buffer pool
130220  8:15:54 InnoDB: highest supported file format is Barracuda.
130220  8:15:54  InnoDB: Waiting for the background threads to start
130220  8:15:55 InnoDB: 1.1.8 started; log sequence number 69234959
130220  8:15:55 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3305
130220  8:15:55 [Note]   - '127.0.0.1' resolves to '127.0.0.1';
130220  8:15:55 [Note] Server socket created on IP: '127.0.0.1'.
130220  8:15:55 [Note] Event Scheduler: Loaded 0 events
130220  8:15:55 [Note] mysqld: ready for connections.
Version: '5.5.29-0ubuntu0.12.10.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3305  (Ubuntu)
130220  8:16:16 [ERROR] mysqld: Table './site_forum/mybb_datacache' is marked as crashed and should be repaired
130220  8:16:16 [Warning] Checking table:   './site_forum/mybb_datacache'
130220  8:16:16 [ERROR] mysqld: Table './site_forum/mybb_themes' is marked as crashed and should be repaired
130220  8:16:16 [Warning] Checking table:   './site_forum/mybb_themes'
130220  8:16:16 [ERROR] mysqld: Table './site_forum/mybb_templates' is marked as crashed and should be repaired
130220  8:16:16 [Warning] Checking table:   './site_forum/mybb_templates'
130220  8:16:16 [ERROR] mysqld: Table './site_forum/mybb_forums' is marked as crashed and should be repaired
130220  8:16:16 [Warning] Checking table:   './site_forum/mybb_forums'
130220  8:16:16 [ERROR] mysqld: Table './site_forum/mybb_threads' is marked as crashed and should be repaired
130220  8:16:16 [Warning] Checking table:   './site_forum/mybb_threads'
130220  8:16:16 [ERROR] mysqld: Table './site_forum/mybb_posts' is marked as crashed and should be repaired
130220  8:16:16 [Warning] Checking table:   './site_forum/mybb_posts'
130220  8:16:16 [ERROR] Got an error from unknown thread, /build/buildd/mysql-5.5-5.5.29/storage/myisam/ha_myisam.cc:912
130220  8:16:16 [Warning] Recovering table: './site_forum/mybb_posts'
130220  8:16:16 [Note] Retrying repair of: './site_forum/mybb_posts' with keycache
130220  8:16:16 [ERROR] mysqld: Disk is full writing './site_forum/mybb_posts.TMD' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
130220  8:16:16 [ERROR] mysqld: Retry in 60 secs. Message reprinted in 600 secs
130220  8:16:51 [ERROR] mysqld: Table './site_forum/mybb_spiders' is marked as crashed and should be repaired
130220  8:16:51 [Warning] Checking table:   './site_forum/mybb_spiders'
130220  8:16:51 [ERROR] mysqld: Table './site_forum/mybb_sessions' is marked as crashed and should be repaired
130220  8:16:51 [Warning] Checking table:   './site_forum/mybb_sessions'
130220  8:18:40 [ERROR] mysqld: Table './site_forum/mybb_users' is marked as crashed and should be repaired
130220  8:18:40 [Warning] Checking table:   './site_forum/mybb_users'
130220  8:18:40 [ERROR] mysqld: Table './site_forum/mybb_userfields' is marked as crashed and should be repaired
130220  8:18:40 [Warning] Checking table:   './site_forum/mybb_userfields'
130220  8:18:40 [ERROR] mysqld: Table './site_forum/mybb_privatemessages' is marked as crashed and should be repaired
130220  8:18:40 [Warning] Checking table:   './site_forum/mybb_privatemessages'

Status information:

Current dir: /var/lib/mysql/
Running threads: 21  Stack size: 196608
Current locks:
lock: 0x7f50c80229c0:

lock: 0x7f50c801fd20:

lock: 0x7f50c80183a0:

lock: 0x7f50e0009c10:

lock: 0x7f50e4010160:

lock: 0x7f50e400b490:

lock: 0x7f50f0028930:

lock: 0x7f50f00235e0:

lock: 0x7f50f0037320:

lock: 0x7f50f001dcc0:

lock: 0x7f50f0002de0:

lock: 0x7f50f0015270:

lock: 0x7f50ec0866b0:

lock: 0x7f50ec02d690:

lock: 0x7f50ec02bb90:

lock: 0x7f50ec0105a0:

df -h

Filesystem      Size  Used Avail Use% Mounted on
rootfs           20G   19G     0 100% /
/dev/root        20G   19G     0 100% /
none            3.2G  280K  3.2G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none             16G     0   16G   0% /run/shm
none            100M     0  100M   0% /run/user
overflow        1.0M  8.0K 1016K   1% /tmp
/dev/md3         92G   20G   68G  23% /home

Best Answer

The answer is in the error log itself:

130220 8:16:16 [ERROR] mysqld: Disk is full writing './site_forum/mybb_posts.TMD' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)

$ perror 28
OS error code  28:  No space left on device

I would suggest you to install ncdu, then run ncdu / to find large files/folders and delete them.


df -h

Filesystem      Size  Used Avail Use% Mounted on
rootfs           20G   19G     0 100% /
/dev/root        20G   19G     0 100% /
none            3.2G  280K  3.2G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none             16G     0   16G   0% /run/shm
none            100M     0  100M   0% /run/user
overflow        1.0M  8.0K 1016K   1% /tmp
/dev/md3         92G   20G   68G  23% /home

OK. Move the datadir to /home by running:

sudo mv /var/lib/mysql /home

Don't forget to change the owner of the new folder:

sudo chown -R mysql:mysql /home/mysql

then create a symbolic link in order not to have to change the datadir in my.cnf:

sudo ln -s /home/mysql/ /var/lib/mysql

Now try to start the MySQL to see it works.