Cannot Start MySql

Question

Our server has been running mysql just fine for over a year. I ran a set of sql script to build a rather large database and in the middle of those scripts, I started getting errors that I had lost connection. Nobody did anything else happened as far as we know. When I tried to log in to mysql, I got:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

When I try to restart mysql, I get:

# sudo service mysql restart
stop: Unknown instance:
start: Job failed to start

The error.log shows:

130212  9:37:51 [Note] Plugin 'FEDERATED' is disabled.
130212  9:37:51 InnoDB: The InnoDB memory heap is disabled
130212  9:37:51 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130212  9:37:51 InnoDB: Compressed tables use zlib 1.2.7
130212  9:37:51 InnoDB: Using Linux native AIO
130212  9:37:51 InnoDB: Initializing buffer pool, size = 10.0G
130212  9:37:51 InnoDB: Completed initialization of buffer pool
130212  9:37:51 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 2186809272046
130212  9:37:51  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 2186814514688
InnoDB: Doing recovery: scanned up to log sequence number 2186816162838
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 27964 row operations to undo
InnoDB: Trx id counter is 18834200
130212  9:37:51  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 4424818.
InnoDB: You may have to recover from a backup.
130212  9:37:51  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 058425e20043847200356a3f003e3720000001fd2807769645bf00000000000000000000000000ef31a083ca0000000031980005000003c8000$
InnoDB: End of page dump
18 130212  9:37:51  InnoDB: Page checksum 1501194131, prior-to-4.0.14-form checksum 441953139
InnoDB: stored checksum 92546530, prior-to-4.0.14-form stored checksum 1240647222
InnoDB: Page lsn 509 671577750, low 4 bytes of lsn at page end 441447404
InnoDB: Page number (if stored to page already) 4424818,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 1096815
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 4424818.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
130212  9:37:51  InnoDB: Assertion failure in thread 140114781574912 in file buf0buf.c line 3603
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
16:37:51 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346681 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7f72aa2435b9]
/usr/sbin/mysqld(handle_fatal_signal+0x3d8)[0x7f72aa12c548]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f72a8c8dcb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f72a82f6425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f72a82f9b8b]
/usr/sbin/mysqld(+0x605429)[0x7f72aa32d429]
/usr/sbin/mysqld(+0x631b69)[0x7f72aa359b69]
/usr/sbin/mysqld(+0x5c20a8)[0x7f72aa2ea0a8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f72a8c85e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f72a83b3cbd]

I cannot find anything running for mysql including any sockets. I am taking over for a previous SysAdmin and am fairly new to linux and MySql. We've got to get this system back online soon. Please help.

Solution 1

Have you tried adding the following line to your /etc/mysql/my.cnf and then restarting the server?

[mysqld]
innodb_force_recovery = 4

Solution 2

To add to @elico3000 you will now need to dump your corrupt table(s) and Data to repair the innodb fs. There are a number of ways to do this. You can read through the logs to determine the point of failure and possible tablenames, then dump and recreate those specific tables. Or you can dump the entire MySQL DB and all schemas using a single command, but that will take some time depending on how big your DB. Either way once you have addressed the corrupt table(s) you can set the innodb_force option to 0 and restart mysqld_safe.

Here is a good tutorial on recovery options for both MyISAM and InnoDB MySQL instances and covers a few options. It is far easier to point you here, than regurgitate the commands and concepts again in this answer.

Good luck and come back to ask more pointed questions once you have tried one of the options. There are probably more tutorials out there, but I have used this in Development to rebuild my Dev DB and it has plenty of information.

Look here