Apache2 configtest segmentation fault

Question

For whatever reason I managed to mess up Apache quite bad (Ubuntu 12.04). The output below complains about configtest failing and a segmentation fault. However, I can't find anything in /var/log/apache2/error.log. How can I debug this "configtest"?

jeroen@jeroen-ubuntu:~$ sudo service apache2 stop
 * Stopping web server apache2                                                  
 * The apache2 configtest failed, so we are trying to kill it manually. This is almost certainly suboptimal, so please make sure your system is working as you'd expect now!
 ... waiting                                                             [ OK ]
jeroen@jeroen-ubuntu:~$ sudo service apache2 start
 * Starting web server apache2
Segmentation fault
Action 'start' failed.
The Apache error log may have more information.
                                                                     [fail]

Edit: the problem is related to the loading of an apache2 module. I found some traces of the problem in/var/log/kern.log:

Mar 14 20:44:27 jeroen-ubuntu kernel: [13267.367260] show_signal_msg: 9 callbacks suppressed
Mar 14 20:44:27 jeroen-ubuntu kernel: [13267.367264] apache2[20784]: segfault at b638f7b5 ip b638f7b5 sp b5ed42e0 error 14
Mar 14 20:44:27 jeroen-ubuntu kernel: [13267.367268] apache2[20786]: segfault at b638f7b5 ip b638f7b5 sp b383f2e0 error 14

I have been using this module for years and it has always been stable. I have recently been installing some backports from PPA repositories, I suspect that one of these packages has broken my system...

Solution

If configtest failed that is not anything to panic about. It basically means something is screwy in one of your Apache config files. So this is my punch down list of what you should do:

1. Were you editing an Apache config file before this happened? Which one? Where? Can you restore a backup or a known good config for it? Basically, I would go over the config with a fine-tooth comb. Something as simple as misspelling in a config option can be the cause. Or maybe the file is damaged?
2. Was this on a main config like /etc/apache2/apache2.conf or something in /etc/apache2/sites-enabled? If it was in the main config file, it would show up in the error.log more often than not? If not, maybe in a separate log connected to your virtual hosts?
3. What is your virtual host config like when you run sudo apachectl -S? That might give another clue as to what configs Apache is using.
4. Past any of that, did you add—or attempt to add—any modules to Apache? Can you disable them? You indicate in some of your comment answers that the module might be the cause. Don’t be married to a piece of replaceable software. Just delete it & reinstall the module. Could be corrupt.
5. Absolute worst case scenario, unless you have an incredibly idiosyncratic install of Apache—I would recommend backing up your configs in /etc/apache2—removing Apache from your system and reinstall it clean. Then see if that clean install works & if it does, then redo your configuration based on the backup files.

Comments

[ravi yarlagadda]

Is the log completely empty?

[Jeroen Ooms]

Yes, nothing gets written to the log. One of my modules fails to load.

[ravi yarlagadda]

What's telling you that the issue is with a module?

[Jeroen Ooms]

When I disable the module, the problem disappears. But I have been running this module (mod_R) for years, so I don't understand where the problem comes from all of a sudden. But the configtest doesn't really tell me anything.

[ravi yarlagadda]

What's changed? Upgrade of any components? Configuration changes?

[Giacomo1968]

I gave a detailed answer below, but I want to specifically mention that in this thread: If you know it’s a module, just remove it from the system & do a clean install of it from scratch. Unless it is a config connected to the module & in that case, maybe that config file is botched.

[Marin Bînzari]

I had this problem after compiling PHP 7. It enabled the php7 module while php5 was also enabled. And the apache has started normaly after disabling of php7 module.

[Jeroen Ooms]

Actually I had tried most of this. I have completely purged and removed apache2, the module, and related software. After reinstalling it the problem was still there.

[allo]

@Jeroen: You accepted the answer, but your comment says it's still there. What did actually solve your problem?

[Vladimir Ch]

also you can use to build in debug sudo apache2 -X