Error 2013: Lost connection to MySQL server during query when executing CHECK TABLE FOR UPGRADE

Question

I just upgraded Ubuntu from 11.10 to 12.04. My rails app now returns the (passenger) error "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) (Mysql2::Error)". I get a similar error when I try to access mysql at the command line on my Ubuntu server using mysql -u root -p.

I have mysql-server 5.5 installed. I've checked and mysql is not running. When I try to restart it, it fails.

Here are some key lines from the tail of /var/log/syslog after an attempted restart:

dean@dgwjasonfried:/etc/mysql$ tail -f /var/log/syslog
Mar  7 08:55:27 dgwjasonfried /etc/mysql/debian-start[5107]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Mar  7 08:55:27 dgwjasonfried /etc/mysql/debian-start[5107]: Running 'mysqlcheck' with connection arguments: '--port=3306' '--socket=/var/run/mysqld/mysqld.sock' '--host=localhost' '--socket=/var/run/mysqld/mysqld.sock' '--host=localhost' '--socket=/var/run/mysqld/mysqld.sock' 
Mar  7 08:55:27 dgwjasonfried /etc/mysql/debian-start[5107]: Running 'mysqlcheck' with connection arguments: '--port=3306' '--socket=/var/run/mysqld/mysqld.sock' '--host=localhost' '--socket=/var/run/mysqld/mysqld.sock' '--host=localhost' '--socket=/var/run/mysqld/mysqld.sock' 
Mar  7 08:55:27 dgwjasonfried /etc/mysql/debian-start[5107]: /usr/bin/mysqlcheck: Got error: 2013: Lost connection to MySQL server during query when executing 'CHECK TABLE ...  FOR UPGRADE'
Mar  7 08:55:27 dgwjasonfried /etc/mysql/debian-start[5107]: FATAL ERROR: Upgrade failed
Mar  7 08:55:27 dgwjasonfried /etc/mysql/debian-start[5107]: molex_app_development.assets                       OK
Mar  7 08:55:27 dgwjasonfried /etc/mysql/debian-start[5107]: molex_app_development.ecd_types                    OK
Mar  7 08:55:27 dgwjasonfried /etc/mysql/debian-start[5124]: Checking for insecure root accounts.
Mar  7 08:55:27 dgwjasonfried kernel: [ 7551.769657] init: mysql main process (5064) terminated with status 1
Mar  7 08:55:27 dgwjasonfried kernel: [ 7551.769697] init: mysql respawning too fast, stopped

Here is most of /etc/mysql/my.cnf:

Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock

Here is entries for some specific programs
The following values assume you have at least 32M ram

This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket          = /var/run/mysqld/mysqld.sock
nice            = 0

[mysqld]

  Basic Settings

user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking

 Instead of skip-networking the default is now to listen only on
 localhost which is more compatible and is not less secure.
bind-address            = 127.0.0.1

And here are permissions for var/run/mysqld/mysqld.sock:

srwxrwxrwx 1 mysql mysql 0 Mar 7 09:18 mysqld.sock

I'd be grateful for any suggestions the community might have. I reviewed the related questions here and attempted some of the fixes offered but to no avail.

Thanks!

Dean Richardson

Update:

Thanks to quanta's suggestion, I looked at the /var/log/mysql/error.log file. I found error messages relating to pointers, fatal signals, and more stuff that I really couldn't make much sense of. I also found mysql man page references, however. One suggested that I try starting mysqld with the --innodb_force_recovery=# option, then attempt to dump (or drop) the offending/corrupted database or table.

I worked through the escalating option levels one-by-one (innodb_force_recovery=1, innodb_force_recovery=2, etc.) This allowed me to successfully run mysql -u root -p from the command line and execute several commands. I was able to run queries on my production database, but any attempt to query, dump, or even drop my development database raised an error and led to me losing the connection to mysql.

So I've made progress, but until I'm somehow able to drop or repair my development db I'm still unable to get my app to load.

Any further advice or suggestions?

Thanks!

Dean

Update:

Right after running sudo mysqld --innodb_force_recover=1 from the command line, the error.log contains this:

Right after retrying sudo mysqld --innodb_force_recover=1, The error.log file shows this:

130308  4:55:39 [Note] Plugin 'FEDERATED' is disabled.
130308  4:55:39 InnoDB: The InnoDB memory heap is disabled
130308  4:55:39 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130308  4:55:39 InnoDB: Compressed tables use zlib 1.2.3.4
130308  4:55:39 InnoDB: Initializing buffer pool, size = 128.0M
130308  4:55:39 InnoDB: Completed initialization of buffer pool
130308  4:55:39 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
130308  4:55:39  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
130308  4:55:40  InnoDB: Waiting for the background threads to start
130308  4:55:41 InnoDB: 1.1.8 started; log sequence number 10259220
130308  4:55:41 InnoDB: !!! innodb_force_recovery is set to 1 !!!
130308  4:55:41 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
130308  4:55:41 [Note]   - '127.0.0.1' resolves to '127.0.0.1';
130308  4:55:41 [Note] Server socket created on IP: '127.0.0.1'.
130308  4:55:41 [Note] Event Scheduler: Loaded 0 events
130308  4:55:41 [Note] mysqld: ready for connections.
Version: '5.5.29-0ubuntu0.12.04.2'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)

Then after mysql -u root -p and

mysql> drop database molex_app_development;
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql> 

the error.log contains:

dean@dgwjasonfried:/var/log/mysql$ tail -f error.log
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f6a3ff9ecbd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f6a1c004bd8): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
130308  4:55:39 [Note] Plugin 'FEDERATED' is disabled.
130308  4:55:39 InnoDB: The InnoDB memory heap is disabled
130308  4:55:39 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130308  4:55:39 InnoDB: Compressed tables use zlib 1.2.3.4
130308  4:55:39 InnoDB: Initializing buffer pool, size = 128.0M
130308  4:55:39 InnoDB: Completed initialization of buffer pool
130308  4:55:39 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
130308  4:55:39  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
130308  4:55:40  InnoDB: Waiting for the background threads to start
130308  4:55:41 InnoDB: 1.1.8 started; log sequence number 10259220
130308  4:55:41 InnoDB: !!! innodb_force_recovery is set to 1 !!!
130308  4:55:41 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
130308  4:55:41 [Note]   - '127.0.0.1' resolves to '127.0.0.1';
130308  4:55:41 [Note] Server socket created on IP: '127.0.0.1'.
130308  4:55:41 [Note] Event Scheduler: Loaded 0 events
130308  4:55:41 [Note] mysqld: ready for connections.
Version: '5.5.29-0ubuntu0.12.04.2'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
130308  4:58:23 [ERROR] Incorrect definition of table mysql.proc: expected column 'comment' at position 15 to have type text, found type char(64).
130308  4:58:23  InnoDB: Assertion failure in thread 140168992810752 in file fsp0fsp.c line 3639
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
10:58:23 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=1
max_threads=151
thread_count=1
connection_count=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346681 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f7ba4f6c2f0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f7ba3065e60 thread_stack 0x30000
mysqld(my_print_stacktrace+0x29)[0x7f7ba3609039]
mysqld(handle_fatal_signal+0x483)[0x7f7ba34cf9c3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f7ba2220cb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f7ba188c425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f7ba188fb8b]
mysqld(+0x65e0fc)[0x7f7ba37160fc]
mysqld(+0x602be6)[0x7f7ba36babe6]
mysqld(+0x635006)[0x7f7ba36ed006]
mysqld(+0x5d7072)[0x7f7ba368f072]
mysqld(+0x5d7b9c)[0x7f7ba368fb9c]
mysqld(+0x6a3348)[0x7f7ba375b348]
mysqld(+0x6a3887)[0x7f7ba375b887]
mysqld(+0x5c6a86)[0x7f7ba367ea86]
mysqld(+0x5ae3a7)[0x7f7ba36663a7]
mysqld(_Z15ha_delete_tableP3THDP10handlertonPKcS4_S4_b+0x16d)[0x7f7ba34d3ffd]
mysqld(_Z23mysql_rm_table_no_locksP3THDP10TABLE_LISTbbbb+0x568)[0x7f7ba3417f78]
mysqld(_Z11mysql_rm_dbP3THDPcbb+0x8aa)[0x7f7ba339780a]
mysqld(_Z21mysql_execute_commandP3THD+0x394c)[0x7f7ba33b886c]
mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x10f)[0x7f7ba33bb28f]
mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1380)[0x7f7ba33bc6e0]
mysqld(_Z24do_handle_one_connectionP3THD+0x1bd)[0x7f7ba346119d]
mysqld(handle_one_connection+0x50)[0x7f7ba3461200]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f7ba2218e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f7ba1949cbd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f7b7c004b60): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

--Dean

Solution 1

Let us take the first error that appears as a starting point.. Seems it's closely related to the upgrade, according to this: http://forums.mysql.com/read.php?10,503634,503634#msg-503634 Try running mysql_upgrade

And you can also check all mysql database tables using:

myisamchk /var/lib/mysql/mysql/*.MYI

Please also post the exact versions of mysql, before and after the upgrade

Solution 2

i would recommend that you switch von Oracle mysql to Percona Mysql Server.

i had often problems with mysql and innodb problems. The percona Guys do a lot of developing patching and performance tuning on their mysql server.

http://www.percona.com/software/percona-server

you dont need to touch the datadir of mysql. It will work without touching/modifiying the data in transition.