How to fix MariaDB when it gets stuck during shutdown (Waiting for page_cleaner)?

Question

I am running MariaDB (v 10.3.13) on Arch Linux (Manjaro). It worked normally for the last months but now there are some issues. When I try to reboot the computer, it gets stuck at

A stop job is running for MariaDB 10.3.13 database server

The time it should take gets increased every minute till it finally times out (after approx 20 min).

Stopping MariaDB manually also gets stuck when I try to restart/stop it with

sudo systemctl stop/restart mariadb

Then running sudo systemctl status mariadb shows

/usr/bin/mysqld (initiated by: unknown): Normal shutdown
Event Scheduler: Purging the queue. 0 events
InnoDB: FTS optimize thread exiting.
InnoDB: Starting shutdown...
InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
InnoDB: Buffer pool(s) dump completed at 190402 20:31:01
InnoDB: Waiting for page_cleaner to finish flushing of buffer pool 

with every minute another line with InnoDB: Waiting for page_cleaner to finish flushing of buffer pool being added.

Is there a way to reset MariaDB? Or a way to get more information what the problem is?

PS: I installed some updates through the Package Manager lately, after which I noticed the issue for the first time.

Update:

When I uninstall, reboot and reinstall MariaDB, I can stop/restart MariaDB without issues and shutdown goes without issues. But after booting again, I am back to the above described problems.

Solution 1

After a multiple hour long struggle, I could finally resolve the issue.

The culprit was a time setting (timedatectl set-local-rtc 1) for dual booting with Win10.

Due to a recent time change the clock was messed up. (The status log entries in sudo systemctl status mariadb were 1h in the future)

Setting

timedatectl set-local-rtc 0

and rebooting resolved it.

Solution 2

Also you could change and reduce from TimeoutSec=300(5 minutes) to a lower timeout value from mysql start service config file:

vi /usr/lib/systemd/system/mariadb.service

### after editing timout value in above config file you can apply it by using:
systemctl daemon-reload

Note: This is not a fix or solution for this problem. It just reduce service termination timeout value. You should consider finding a correct solution based on the situation or bug causing this happened on your OS and Service and reverting timeout to its default recommended value after issue is solved.

Comments

[Eek]

Wow, thanks for sharing. Apparently I had the exact same issue. I just moved the clock a bit forward in time to resume :)) I'll move it back after it's done.

[Eric Sebasta]

I don't have systemd ... so this does not work. :/

[Admin]

Which just kills the mysql server process, preventing a clean shutdown.

[Admin]

@GáborHéja true, but if no working solution, its a fast escape while stack is overflowing.

[Admin]

Yes it is, but it sounds important and I think should be in the answer. I edited it and removed my downvote. Thanks for the response.

[Admin]

@GáborHéja respectfully I will edit my answer according to your suggestion but unfortunatly I disagree with the exact note you suggest because the 5 minutes value was default value provided by mysql-service to terminate forcefully and my suggestion was to reduce it for shorter service timeouts. Here the situation is showing that there is a bug between OS and the Mysql-Service and that long time of termination is not doing any positiove actions (no difference if reduced) and could be just annoying on every restart. Still I agree this is not a solution but just an escape to be fixed later.

[Admin]

I have already edited but probably was discarded for some reason and my downvote was not removed, I fixed that now. In my case the InnoDB was doing a recovery that took way more than even 5 minutes so I think in some corner cases the note might be useful. Thanks for the responsiveness and being open to comments!