16.04 upgrade broke mysql-server

Question

My overall upgrade went well, but I am left with a critically lingering issue of mysql-server not being able to install itself, and nothing I am trying gets it to work.

This is the error I see when trying to install/reinstall it:

Setting up mysql-server-5.7 (5.7.11-0ubuntu6) ...
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
invoke-rc.d: initscript mysql, action "start" failed.
dpkg: error processing package mysql-server-5.7 (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of mysql-server:
 mysql-server depends on mysql-server-5.7; however:
  Package mysql-server-5.7 is not configured yet.

dpkg: error processing package mysql-server (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
                   Processing triggers for systemd (229-4ubuntu4) ...
Processing triggers for ureadahead (0.100.0-19) ...
Errors were encountered while processing:
 mysql-server-5.7
 mysql-server
E: Sub-process /usr/bin/dpkg returned an error code (1)

I've tried completely removing it, though doing so attempts to install MariaDB because of dependencies (?). Any suggestions on what I can do to fix this would be most welcome.

EDIT: Looks like I'm not the only one: https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1573279

Solution 1

The instructions @andrew-beerman posted are on the right track, though they aren't quite clear to me and seem to recommend more than is necessary. I pieced together the answer from the above and a helpful post in the bug thread.

These are the steps I took to correct this:

1. Back up your my.cnf file in /etc/mysql and remove or rename it

   sudo mv /etc/mysql/my.cnf /etc/mysql/my.cnf.bak
   

2. Remove the folder /etc/mysql/mysql.conf.d/ using

   sudo rm -r /etc/mysql/mysql.conf.d/
   

3. Verify you don't have a my.cnf file stashed somewhere else (I did in my home dir!) or in /etc/alternatives/my.cnf use

   sudo find / -name my.cnf
   

4. Backup and remove /etc/mysql/debian.cnf files (not sure if needed, but just in case)

   sudo mv /etc/mysql/debian.cnf /etc/mysql/debian.cnf.bak
   sudo apt purge mysql-server mysql-server-5.7 mysql-server-core-5.7
   sudo apt install mysql-server
   

5. In case your syslog shows an error like "mysqld: Can't read dir of '/etc/mysql/conf.d/'" create a symbolic link:

   sudo ln -s /etc/mysql/mysql.conf.d /etc/mysql/conf.d
   

   Then the service should be able to start with sudo service mysql start.

That got it working!

Solution 2

Today I got the same problem, after trying many solutions i found that the problem was the command sudo systemctl disable mysql.service that i used to disable MySQL auto starting, so to get it working i re-enabled again MySQL server using the command sudo systemctl enable mysql.service and run again the upgrade process and it terminated perfectly.

Solution 3

Your error message contains this line:

subprocess installed post-installation script returned error exit status 1

However, this installed post-installation script is not mentioned by name. After much tinkering, I found out that its name is (in my case) /var/lib/dpkg/info/mysql-server-5.7.postinst.

Open this file with sudo vi /var/lib/dpkg/info/mysql-server-5.7.postinst, or your preferred editor.

At the top, change line 3 (or so): set -e to set -x, save the file. (option -e is "exit on errors", -x means "explicitly show command executed", presumably)

Run sudo dpkg --configure -a --log /tmp/dpkg.log (the --log option is optional). You can also simply run apt upgrade if you know it'll be the only package that will be upgraded.

Now you get verbose output of the mysql-server-5.7.postinst bash script, and you can figure out what's wrong.

In my case it unsuccessfully tried to (re-)run mysql_upgrade, but that was not needed for my customized mysql installation. I was sure I've run it manually before, successfully, and all was well.

So I commmented out line 321 (for older mysqld releases try line 281),

#mysql_upgrade --defaults-file=/etc/mysql/debian.cnf || result=$?

and the command that has failed before, sudo apt upgrade (run it again), finished successfully, and dpkg removed the error status for this package.

Now you can set back the set -x to set -e (mentioned above). And optionally uncomment the mysql-upgrade line.

Extra work might be required if you have moved your mysql data partition to a nonstandard location. I moved mine from /var/lib/mysql/data to a different drive via symlink. Then you might have to remove the symbolic link temporarily, before the postinst script manipulation. Then re-create it after running the package upgrade.

After the next minor version upgrade of the mysqld debian package, this problem with the /var/lib/dpkg/info/mysql-server-5.7.postinst script can show up again.

Solution 4

The instructions here fixed it on my server: https://bugs.mysql.com/bug.php?id=72722

I can understand the pain of having your system in inconsistent state but lets not worry about the whole situation and take it step by step to get the system clean.

First lets see the current state of all the mysql packages on machine using: dpkg -l | grep mysql (Please paste the output excluding last column)

The first column denotes the current status of the package. Here are the possible options:

ii) Installed rc) Removed config-files kept (This should be the state of all the packages you have removed with 'apt-get remove' that does not remove config-files under /etc)

For this to work, you will need to run 'apt-get purge <pkg-name>' till you do not see any packages in the above list.

Please remember that some non-mysql-server packages like python-mysql.connector and python-mysqldb, if installed, need not be removed as they do not have any affect on this situation but if removed might cause trouble to applications using them.

We will definitely try to re-visit our docs to see how can we safeguard users from getting into this trouble. Thanks for sharing your feedback in detail with us.

Solution 5

In my case, with strace, I saw that /var/run/mysqld/ didn't exist and mysqld can't create the file mysqld.sock.

These commands solved my problem:

mkdir /var/run/mysqld
chown mysql.mysql /var/run/mysqld
chmod 700 /var/run/mysqld

Now:

systemctl start mysql

And mysql works again :)

View more solutions

Comments

[Hinrich]

Had to uninstall phpmyadmin, which was causing the dependency problems somehow. After reinstalling, everything is working fine again.

[Marco Marsala]

do-release-upgrade from Ubuntu 14 to 16 performs an unsupported upgrade from MySQL 5.5 to 5.7, so it is expected MySQL is broken after, as reported here: bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+b‌​ug/…

[TheGremlyn]

@MarcoMarsala I don't know if we can say it is expected for MySQL to be broken after the Ubuntu 14 to 16 upgrade, though I suppose it depends on your perspective. To the average user, it is certainly not expected to have an LTS upgrade break something like MySQL. It is surprising that was not caught during testing, though.

[Andy]

apt install phpmyadmin --reinstall fixed the above with no need to touch mysql after... go figure

[Allen]

I ran in to this same problem and the same fix worked for me.

[Serge P]

after all these operations (I did it before) I still have same error issues...

[Błażej Michalik]

Worked for me. Did every solution here, mysql worked after this one.

[TheGremlyn]

Are you sure you got rid of ALL config files? I had a my.cnf hanging out in a user folder that seemed to be preventing a complete purge.

[Serge P]

yeah, I'm sure and btw I had updated version of my.cnf without old syntax. Anyway I resolved it by installation MariaDB server.

[hitzi]

You have to remove every my.cnf.* from the /etc/mysql directory. Look for my.cnf.backup, my.cnf.fallback and my.cnf.migrated - all of these files have to be removed too (backup first!)

[David Tay]

mysql-server-core- should be mysql-server-core-5.7. Otherwise worked like a charm. Thank you!

[Linux1Man]

+1. You've made my day! Also morning and evening! I was stuck.

[Linux1Man]

Thanks for dpkg -l | grep mysql . It helped to understand the direction.

[Thomas Carlisle]

Thank you for this! These steps were able to fix my broken mysql. @David Tay for me, using "mysql-server-core", apt automatically processed it as mysql-server-core-5.7.

[scoobydoo]

I suggest you add these lines to /usr/share/mysql/mysql-systemd-start starting at line 25, then you won't have to recreate this directory after every reboot (sorry the line returns don't work in this comment): if [ ! -d /run/msyqld ]; then mkdir -p -m0755 /run/mysqld || { echo "Unable to create /run/mysqld"; exit 1; } chown mysql:mysql /run/mysqld || { echo "Unable to chown /run/mysqld"; exit 1; } fi

[skonsoft]

After trying many times this solution, i made a decision to backup the /etc/mysql folder and then remove it. I reinstalled mysql and i followed steps described here and everything works !

[Jonathan]

I ran into mysqld: Can't read dir of '/etc/mysql/conf.d/' (Errcode: 2 - No such file or directory) and retried the purge, install step after: sudo mkdir /etc/mysql/conf.d/; sudo chown -R mysql:mysql /etc/mysql/conf.d successfully

[RyanNerd]

I ended up purging everything and then downloading the community mysql-server version and manually installing from here: dev.mysql.com/downloads/mysql

[a1phanumeric]

Same here - this should be top post.

[emiliopedrollo]

Exactly the same thing happened to me, and your steps solved it to. Bur for me my mysql_upgrade call was on line 320. Can you explain why it return a non-zero value when called from postinst script?

[knb]

@emiliopedrollo No I can't explain here. But I think the line number is now 320 because recently the package maintainers have augmented the postinstall script, I have observed the same thing the other day, during the last run of the software-updater (which included a new mysql-deb-package).

[Spl2nky]

sudo apt-get remove --purge mysql-server mysql-client mysql-common sudo rm -rf /var/lib/mysql

[pwned]

all I needed was to remove mysql-common (also be careful not to have mariadb-common) which got rid of the invalid my.cnf

[Fernando Paladini]

Worked for me too! Thank you so much. Just to be explicit, run: sudo systemctl enable mysql.service and after that run sudo apt install -f.

[Purvez]

I know this is supposed to be an old bug but I got this problem today (2 Apr 2017). So I tried the accepted answer above. It seemed to be working well until right at the end I got: ERROR: Unable to start MySQL server: mysqld: Can't read dir of '/etc/mysql/mysql.conf.d' (Errcode: 13 Permission denied) [ERROR] Fatal error in defaults handling. ANY ADVICE PLEASE?

[Purvez]

Sorry should have said that my error seems to be slightly different to the : can't read conf.d directory mentioned in a previous comment.

[Purvez]

I tried the whole process again and this time it gave me the Errcode 2 for /etc/mysql/conf.d. So I did the symlink and tried to restart the mysql service but the Job failed. I'm getting exasperated now sadly.

[Thai Tran]

When installing mysql-server (last step), it complains i don't have the my.cnf.fallback, so i created one based on this askubuntu.com/a/865931/77700 and trigger the command again and It started to work

[frandevel]

Is this removing the DB schemas that one might have?

[TheGremlyn]

Your databases should remain in tact, but it's always a good idea to back them up.

[Allen Hamilton]

Thanks! With set -e I was able to work out the exact problem - MySQL didn't have access for the ubuntu system user specified in /etc/mysql/debian.cnf . So I added this user to MySQL and granted privileges, ran dpkg again and it worked!

[Jack Holt]

This is all I had to do. I had disabled autostart like described by @naruto. I enabled autostart and run "sudo apt upgrade". Now fixed. Should be top answer. This is a much simpler, more standard and less trouble prone solution.

[Jenson Jose]

After trying numerous solutions across so many sites, this finally fixed my issue.

[oooo ooo]

This solution worked perfectly :-)

[Fran Marzoa]

This should be the selected answer.

[Csaba Toth]

isntall -> install typo. Stupid SO does not allow me to correct one character.

[PlasmaBinturong]

It solves the OP bug, but the problem is that then you have a mysqld server always running in the background. Maybe you don't want that. I'm gonna try to disable it again...

[Toskan]

command was wrong for purge it must be sudo apt-get purge <pkg-name> I fixed it with an edit but: I used unicode characters for the < because otherwise it wouldnt display. Be warned copy pasting this command

[Sinscary]

Why did you go to the oracle to download mysql

[ostrokach]

@Sinscary This answer is from awhile back, but it was probably because they had a newer version than the system package manager?

[reignsly]

This fix my problem, all the database is still there, thanks.

[Stack Underflow]

After doing all this, I was getting a MySQL "Access Denied" error. I solved it with this answer. Now everything's back to normal!! :-D

[Angsuman Chakraborty]

Typically this happens when you disable mysql as you stopped using it on that VM/Server. When you upgrade and MySQL is also set to be upgraded, it tries to connect to the MySQL server to execute the upgrade instructions. It fails because it cannot find and running instance and it is not intelligent enough to enable mysql - run it - execute the upgrades - disable it again or give a meaningful message.