Server crashed, some MySQL databases are broken. Trying to recover with InnoDB recovery, but ibdata1 seems corrupt

Question

I've a problem. Some days ago my root server crashes after a power outage. I realised that the MySQL server won't start. In the logs I find out that an error occured with InnoDB's ibdata1.

I've tried the automatic InnoDB recovery program (innodb_force_recovery). I reinstalled the MySQL server. All these ways didn't help. I've tried out some more which I get from forums and other community platforms (other variables in the my.cnf etc.). This ways didn't help me too. So I think that I need personal help. The problem is that I don't find any "real" resolution for this problem in the internet.

At the moment I have a MariaDB server (version 5.5) on my root.

The error message when I start the server directly (mysqld --console) is:

root@GPR0420:/var/lib# mysqld
131125 17:06:17 [Warning] An old style --language or -lc-message-dir value with language specific part detected: /usr/share/mysql/english/
131125 17:06:17 [Warning] Use --lc-messages-dir without language specific part instead.
131125 17:06:17 InnoDB: The InnoDB memory heap is disabled
131125 17:06:17 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131125 17:06:17 InnoDB: Compressed tables use zlib 1.2.3.4
131125 17:06:17 InnoDB: Using Linux native AIO
131125 17:06:17 InnoDB: Initializing buffer pool, size = 128.0M
131125 17:06:17 InnoDB: Completed initialization of buffer pool
131125 17:06:17 InnoDB: highest supported file format is Barracuda.
InnoDB: The user has set SRV_FORCE_NO_LOG_REDO on
InnoDB: Skipping log redo
InnoDB: Error: trying to access page number 4294899583 in space 0,
InnoDB: space name ./ibdata1,
InnoDB: which is outside the tablespace bounds.
InnoDB: Byte offset 0, len 16384, i/o type 10.
InnoDB: If you get this error at mysqld startup, please check that
InnoDB: your my.cnf matches the ibdata files that you have in the
InnoDB: MySQL server.
131125 17:06:17  InnoDB: Assertion failure in thread 140125885400864 in file fil0fil.c line 5451
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
131125 17:06:17 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

 We will try our best to scrape up some info that will hopefully help
 diagnose the problem, but since we have already crashed,
 something is definitely wrong and this may fail.

 Server version: 5.5.34-MariaDB-1~squeeze
 key_buffer_size=167772160
 read_buffer_size=131072
 max_used_connections=0
 max_threads=153
 thread_count=0
 It is possible that mysqld could use up to
 key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 499482 K  bytes of memory
 Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x30000
addr2line: 'mysqld': No such file
mysqld(my_print_stacktrace+0x2e)[0x7f719a2e621e]
mysqld(handle_fatal_signal+0x4ac)[0x7f7199ee641c]
/lib/libpthread.so.0(+0xeff0)[0x7f71995deff0]
/lib/libc.so.6(gsignal+0x35)[0x7f7197eb81b5]
/lib/libc.so.6(abort+0x180)[0x7f7197ebafc0]
mysqld(+0x8304e9)[0x7f719a23c4e9]
mysqld(+0x7ff71d)[0x7f719a20b71d]
mysqld(+0x80025f)[0x7f719a20c25f]
mysqld(+0x7ef407)[0x7f719a1fb407]
mysqld(+0x835422)[0x7f719a241422]
mysqld(+0x8380fc)[0x7f719a2440fc]
mysqld(+0x7ce0f2)[0x7f719a1da0f2]
mysqld(+0x802fe1)[0x7f719a20efe1]
mysqld(+0x80392c)[0x7f719a20f92c]
mysqld(+0x7a4af5)[0x7f719a1b0af5]
mysqld(+0x760e9e)[0x7f719a16ce9e]
mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x7f7199ee8808]
mysqld(+0x38bd16)[0x7f7199d97d16]
mysqld(_Z11plugin_initPiPPci+0x785)[0x7f7199d9ad15]
mysqld(+0x2f3cb7)[0x7f7199cffcb7]
mysqld(_Z11mysqld_mainiPPc+0x5cd)[0x7f7199d00b7d]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7f7197ea4c8d]
mysqld(+0x2e90b5)[0x7f7199cf50b5]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

This error says me that there is an error in the ibdata1 or that this file is corrupt. Is it possible to repair the ibdata1-file? And when not, can we extract some data which aren't corrupt from this file?

I hope you can help me. It were nice to have the "new" (not in any backup) data back because the databases contains statistics.

Best regards, Jonniboy

Solution

Looks like innodb_force_recovery is set to 6 in your my.cnf, so it will not try to recover based on binary logs, try setting it to 0 instead.

http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html

check also that you had binary logs activated at the time of crash: http://dev.mysql.com/doc/refman/5.5/en/binary-log.html

Comments

[Cedric Simon]

This one might interest you if setting to zero does not sove your problem: dba.stackexchange.com/questions/152866/…