MariaDB won't start after server crash. No error logs

Question

My CentOS 7 server recently crashed by having a hard drive unplugged while it was running and now MariaDB won't start and I'm not getting any error logs.

Everything else on the server seems to be running fine and running xfs_repair (similar to fsck for xfs) on the disk didn't help. Here's specifically what errors I'm getting:

$ sudo systemctl start mariadb
Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.

$ sudo systemctl -l status mariadb
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2018-07-29 19:34:03 MDT; 1min 22s ago
  Process: 21579 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
  Process: 21577 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 27150 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=1/FAILURE)
 Main PID: 21577 (code=exited, status=0/SUCCESS)

Jul 29 19:34:03 giivaserver systemd[1]: Starting MariaDB database server...
Jul 29 19:34:03 giivaserver mariadb-prepare-db-dir[27150]: Database MariaDB is not initialized, but the directory /var/lib/mysql is not empty, so initialization cannot be done.
Jul 29 19:34:03 giivaserver mariadb-prepare-db-dir[27150]: Make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.
Jul 29 19:34:03 giivaserver systemd[1]: mariadb.service: control process exited, code=exited status=1
Jul 29 19:34:03 giivaserver systemd[1]: Failed to start MariaDB database server.
Jul 29 19:34:03 giivaserver systemd[1]: Unit mariadb.service entered failed state.
Jul 29 19:34:03 giivaserver systemd[1]: mariadb.service failed.

As far as I can tell, I'm getting no information back about what is happening or why it can't start. There are also no error log files anywhere that I could find.

Here's my my.cnf:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

There are no files at all in my /var/log/mariadb/ or /var/run/mariadb/ folders. I also don't have a host-name.err (giivaserver.err in my case) in either /var/log/ or /var/log/mysql as described on the official article What to Do if MariaDB Doesn't Start.

I tried putting innodb_force_recovery = 1 incrementing from 1 to 5 under [mysqld] in my my.cnf and it didn't affect anything.

I also tried backing up and renaming my /var/lib/mysql folder and rerunning mysql_install_db as some online said deleting it could help, but that made no difference.

I have tried everything I can think of to this point and can't find any feedback from my system as to why MariaDB won't start.

Edit:

The contents of my /var/lib/mysql folder after renaming the original:

drwx------.  6 mysql mysql   122 Jul 29 19:03 .
drwxr-xr-x. 61 root  root   4096 Jul 29 19:03 ..
-rw-rw----.  1 mysql mysql 16384 Jul 29 19:02 aria_log.00000001
-rw-rw----.  1 mysql mysql    52 Jul 29 19:02 aria_log_control
drwx------.  2 mysql mysql  4096 Jul 29 19:02 mysql
drwxr-xr-x.  8 mysql mysql   232 Jul 29 18:58 mysqlBAK
drwx------.  2 mysql mysql  4096 Jul 29 19:02 performance_schema
drwx------.  2 mysql mysql     6 Jul 29 19:02 test

Solution 1

After several days I was able to fix it without any data loss. As I suspected, there was no data loss from the outage. I did many things to try to fix the issue but here is what I believe fixed it:

I renamed my /var/lib/mysql folder to make sure I had a backup, then I deleted my /etc/my.cnf file. I reinstalled mariadb and mariadb-server then started it up. From there I copied my backed-up /var/lib/mysql folder back in and in my /etc/my.cnf added the following lines (gotten from here):

port = 8881
innodb_force_recovery=3
innodb_purge_threads=0

(I think I actually used recovery mode 4 but 3 would have worked for me too)

From there I was able to start up MariaDB with all of my databases. I then ran mysqlcheck --all-databases -p to find that none of my databases had been corrupted. From here, I was unable to get MariaDB to successfully stop, and then I had issues starting it up again, so I repeated the process of deleting the /var/lib/mysql/ folder and running the database installation so that I could copy my backup back in.

From there I started MariaDB with InnoDB forced recovery and once it was running I dumped everything using the command mysqldump -u root -p --all-databases > alldb.sql. I then force stopped MariaDB as it wouldn't stop properly, then rebooted. I uninstalled MariaDB, deleted my /var/lib/mysql/, and renamed my /etc/my.cnf, then reinstalled MariaDB. Once I got it running fresh, I imported my dumped databases using mysql -u root -p < alldb.sql and flushed privileges in the MySQL shell using FLUSH PRIVILEGES; and everything is working great now with no data loss.

As I am running CentOS, I had to do some SELinux stuff along the way. I mostly just followed what the pop-up diagnostics dialogue said to do, then reran the SELinux commands I did originally when installing my Nextcloud server in the first place to get all of the permissions right.

Solution 2

Looks like the database storage has become corrupted. Restore from backup, and put a big sign on your hard drives, "do not unplug whilst running".

Comments

[Altimus Prime]

Did you have any mounting options or symlinks set up for /var/lib/mysql? By that I mean is your MariaDB and your data on the same partition of the same hard drive?

[Altimus Prime]

Something like ls -l /var/lib/mysql could help us see which files are there vs which might be missing.

[Isvvc]

@AuntJemima updated the post with contents of /var/lib/mysql

[Altimus Prime]

I don't think that you have all the files that should be there. Shouldn't there be a sys file and an auto.cnf? I think womble is probably right, that you hopefully have a backup that you can restore. Have you tried changing the data directory to something else and restarting mariadb?

[Isvvc]

I don't know that it was corrupted though. Nothing else on the server seems to be having issues. When I reinstalled mariadb and deleted my /var/lib/mysql folder it still couldn't start. And running xfs_repair didn't seem to find unreparable errors.

[Tero Kilkanen]

MariaDB not starting is a strong indicator that the database is corrupt.