MySQL 500% CPU Usage and Crashes Server

cpu-usageMySQLperformance

My dedicated server is a XEON E3 1240 V3 8GB DDR3 1TB HDD SATA (4 cores @ 3.4 GHz), which I've had for a couple of days only and just finished migrating about 20 wordpress sites to it (each database around 50MB to 150MB in size). The server has consistently had 500%+ CPU usage from mysqld, which has caused the httpd service to crash and websites unable to load.

I have not yet changed the default my.cnf file, which is as follows:

#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

[mysqld]
log-error=/var/lib/mysql/server1.nc-ph-0741-26.web-hosting.com.err
performance-schema=0
default-storage-engine=MyISAM
innodb_file_per_table=1
max_allowed_packet=268435456
open_files_limit=10000

I have downloaded MySQLTuner – running perl mysqltuner.pl --host 127.0.0.1 returns the following:

[--] Skipped version check for MySQLTuner script
[--] Performing tests on 127.0.0.1:3306
[OK] Currently running supported MySQL version 10.3.18-MariaDB
[OK] Operating on 64-bit architecture

-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/lib/mysql/server1.nc-ph-0741-26.web-hosting.com.err exists
[--] Log file: /var/lib/mysql/server1.nc-ph-0741-26.web-hosting.com.err(32K)
[OK] Log file /var/lib/mysql/server1.nc-ph-0741-26.web-hosting.com.err is readable.
[OK] Log file /var/lib/mysql/server1.nc-ph-0741-26.web-hosting.com.err is not empty
[OK] Log file /var/lib/mysql/server1.nc-ph-0741-26.web-hosting.com.err is smaller than 32 Mb
[!!] /var/lib/mysql/server1.nc-ph-0741-26.web-hosting.com.err contains 57 warning(s).
[!!] /var/lib/mysql/server1.nc-ph-0741-26.web-hosting.com.err contains 12 error(s).
[--] 6 start(s) detected in /var/lib/mysql/server1.nc-ph-0741-26.web-hosting.com.err
[--] 1) 2019-10-11  1:13:04 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 2) 2019-10-09  6:41:42 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 3) 2019-10-09  6:41:39 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 4) 2019-10-09  6:41:34 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 5) 2019-10-09  6:40:48 140404223039680 [Note] /usr/sbin/mysqld: ready for connections.
[--] 6) 2019-10-09  6:40:39 140607405234368 [Note] /usr/sbin/mysqld: ready for connections.
[--] 5 shutdown(s) detected in /var/lib/mysql/server1.nc-ph-0741-26.web-hosting.com.err
[--] 1) 2019-10-11  1:09:43 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 2) 2019-10-09  6:41:41 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 3) 2019-10-09  6:41:39 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 4) 2019-10-09  6:40:59 140404100704000 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 5) 2019-10-09  6:40:48 140607307523840 [Note] /usr/sbin/mysqld: Shutdown complete

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +Aria +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE
[--] Data in MyISAM tables: 791.8M (Tables: 738)
[--] Data in InnoDB tables: 672.0K (Tables: 36)
[OK] Total fragmented tables: 0

-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.

-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[--] There are 620 basic passwords in the list.

-------- CVE Security Recommendations --------------------------------------------------------------
[OK] NO SECURITY CVE FOUND FOR YOUR VERSION

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 34m 29s (169K q [82.103 qps], 7K conn, TX: 1G, RX: 20M)
[--] Reads / Writes: 99% / 1%
[--] Binary logging is disabled
[--] Physical Memory     : 7.6G
[--] Max MySQL memory    : 856.4M
[--] Other process memory: 0B
[--] Total buffers: 417.0M global + 2.9M per thread (151 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 472.3M (6.07% of installed RAM)
[OK] Maximum possible memory usage: 856.4M (11.01% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (14/169K)
[OK] Highest usage of available connections: 12% (19/151)
[OK] Aborted connections: 0.11%  (8/7083)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache efficiency: 0.0% (0 cached / 139K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 34K sorts)
[!!] Joins performed without indexes: 4942
[!!] Temporary tables created on disk: 64% (6K on disk / 10K total)
[OK] Thread cache hit rate: 99% (19 created / 7K connections)
[OK] Table cache hit rate: 99% (920 open / 926 opened)
[!!] table_definition_cache(400) is lower than number of tables(1039)
[OK] Open file limit used: 15% (1K/10K)
[OK] Table locks acquired immediately: 100% (156K immediate / 156K locks)

-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema is installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 8 thread(s).
[--] Using default value is good enough for your version (10.3.18-MariaDB)

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 21.1% (28M used / 134M cache)
[OK] Key buffer size / total MyISAM indexes: 128.0M/34.2M
[OK] Read Key buffer hit rate: 100.0% (31M cached / 3K reads)
[!!] Write Key buffer hit rate: 85.9% (71 cached / 61 writes)

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 128.0M/672.0K
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (75 %): 48.0M * 2/128.0M should be equal to 25%
[OK] InnoDB buffer pool instances: 1
[--] Number of InnoDB Buffer Pool Chunk : 1 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[!!] InnoDB Read buffer efficiency: 80.51% (2569 hits/ 3191 total)
[!!] InnoDB Write Log efficiency: 0% (1 hits/ 0 total)
[OK] InnoDB log waits: 0.00% (0 waits / 1 writes)

-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/1B
[!!] Aria pagecache hit rate: 87.1% (50K cached / 6K reads)

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: MIXED
[--] XA support enabled: ON
[--] Semi synchronous replication Master: OFF
[--] Semi synchronous replication Slave: OFF
[--] This is a standalone server

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Control warning line(s) into /var/lib/mysql/server1.nc-ph-0741-26.web-hosting.com.err file
    Control error line(s) into /var/lib/mysql/server1.nc-ph-0741-26.web-hosting.com.err file
    MySQL was started within the last 24 hours - recommendations may be inaccurate
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
             See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
             (specially the conclusions at the bottom of the page).
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries which have no LIMIT clause
    Performance schema should be activated for better diagnostics
Variables to adjust:
    query_cache_size (=0)
    query_cache_type (=0)
    query_cache_limit (> 1M, or use smaller result sets)
    join_buffer_size (> 256.0K, or always use indexes with JOINs)
    tmp_table_size (> 16M)
    max_heap_table_size (> 16M)
    table_definition_cache(400) > 1039 or -1 (autosizing if supported)
    performance_schema = ON enable PFS
    innodb_log_file_size should be (=16M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.

What can I do to fix this?

Best Answer

Running SHOW PROCESSLIST; in phpmyadmin showed a strange query stuck on a database that had close to 1 million rows, killing that solved the issue.