Quick overview of the company product I'm working on before I ask my question:
We have a server running a pretty much stock install of Ubuntu 18.04 and a vision app. The main purpose of the server is to run the vision app. The vision app logs data to a MySQL database (the MySQL install is pretty much stock as well). We also have a website that is essentially a front end for the database. If the mysql systemd service stops running and does not automatically restart then the vision app will crash and not be able to restart until MySQL is started again manually, which is absolutely unacceptable (the vision app has to run 24 hours a day with not more than an occasional few seconds of downtime).
To protect out client's networks, our servers are on isolated local networks that do NOT have internet access.
What has been happening lately is the mysql systemd service has been crashing and not restarting. I made this post:
mysql.service location on Ubuntu 18.04 server?
which resulted in doing the following:
cd /etc/systemd/system
sudo mkdir mysql.service.d
cd mysql.service.d
sudo nano override.conf
Then I entered the following for override.conf:
[Service]
Restart=always
This did not solve the problem, that is to say MySQL crashes and does not restart after anywhere from 5 to 20 hours of running. I've check the RAM and hard disk usage during the MySQL crash and both are substantially below 75% usage so that does not seem to be the problem either.
Some Googling and browsing /var/log
led me to /var/log/unattended-upgrades
. The most recent crash occurred at about 6:28 am on January 3rd, 2019.
cd /var/log/unattended-upgrades
cat unattended-upgrades.log
output:
(some stuff omitted)
2019-01-02 15:43:33,230 INFO Starting unattended upgrades script
2019-01-02 15:43:33,230 INFO Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
2019-01-03 06:28:00,499 INFO Initial blacklisted packages:
2019-01-03 06:28:00,534 INFO Initial whitelisted packages:
2019-01-03 06:28:00,534 INFO Starting unattended upgrades script
2019-01-03 06:28:00,534 INFO Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
2019-01-03 06:28:02,248 INFO Removing unused kernel packages: linux-modules-extra-4.15.0-29-generic linux-image-4.15.0-29-generic linux-headers-4.15.0-29 linux-headers-4.15.0-29-generic linux-modules-4.15.0-29-generic
2019-01-03 06:28:36,159 ERROR Auto-removing the packages failed!
2019-01-03 06:28:36,159 ERROR Error message: installArchives() failed
2019-01-03 06:28:36,159 ERROR dpkg returned an error! See /var/log/unattended-upgrades/unattended-upgrades-dpkg.log for details
2019-01-03 06:28:36,561 INFO No packages found that can be upgraded unattended and no pending auto-removals
2019-01-03 07:13:21,059 INFO Initial blacklisted packages:
2019-01-03 07:13:21,137 INFO Initial whitelisted packages:
(some stuff omitted)
next command:
cat unattended-upgrades-shutdown.log
output:
(some stuff omitted)
Log started: 2019-01-03 06:28:02
(Reading database ... 232457 files and directories currently installed.)
Removing linux-modules-extra-4.15.0-29-generic (4.15.0-29.31) ...
Setting up mysql-server-5.7 (5.7.24-0ubuntu0.18.04.1) ...
Checking if update is needed.
Checking server version.
Error occurred: The mysql.session exists but is not correctly configured. The mysql.session needs SELECT privileges in the performance_schema database and the mysql.db table and also SUPER privileges.
mysql_upgrade failed with exit status 5
dpkg: error processing package mysql-server-5.7 (--configure):
installed mysql-server-5.7 package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
mysql-server-5.7
Log ended: 2019-01-03 06:28:35
I'm a vision person, not a server/networking expert, so I would not profess to understand these logs entirely, but it seems that unattended-upgrades
is attempting to update MySQL and failing due to not having root permissions, which makes absolutely no sense since unattended-upgrades
would need root permissions to do almost everything it does. Anyhow, after more Googling I found this:
cd /etc/apt/apt.conf.d
cat 20auto-upgrades
output:
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
So at this point here are my questions:
1) Based on the logs above, does seems likely that unattended-upgrades
is causing the MySQL service to crash? If so, why is this the case?
2) Being that our server does not have internet connectivity, would it be a good general practice to disable automatic updates anyway?
3) Is there anything else I should check that could cause the MySQL service to crash and not auto-restart?
4) Why did adding override.conf
with Restart=always
not make MySQL automatically restart? When I type sudo systemctl start mysql.service
MySQL does successfully restart. Isn't that what the Restart=always
line is supposed to do automatically? Is there another way I can enter this line so it will actually work?
5) Has anybody else experienced unattended-upgrades
causing MySQL to crash? Is this a known problem? I couldn't find anybody else mentioning this upon Googling which seems odd.
6) I'm under the impression that the next step is to disable auto-updates. If this is the case, in 20auto-upgrades
should I set Update-Package-Lists
to 0
, Unattended-Upgrade
to 0
, or both? I don't see any reason to leave updating package lists on if unattended upgrade is turned off so my inclination is to set both to 0 unless somebody suggests otherwise.
Best Answer
After unattended upgrade of mysql server I had no option but remove and reinstall mysql-server and restore databases.
The samples of log files: