Issue with starting apache on CentOS 6.3

Question

I have some box that was handed to me and I can't even start the basic apache server. It's installed via yum (I've uninstalled and installed it) and when I start the service (service httpd start) its says "OK", but there's no pid. Status shows not running.

In the error_log there is the following:

[Wed Feb 13 16:16:36 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Feb 13 16:16:36 2013] [notice] Digest: generating secret for digest authentication ...
[Wed Feb 13 16:16:36 2013] [notice] Digest: done
/usr/sbin/httpd: symbol lookup error: /usr/lib64/libaprutil-1.so.0: undefined symbol: apr_os_uuid_get

I assume it's not starting because of the last line. Suggestions?

ldd $(which httpd)
    linux-vdso.so.1 =>  (0x00007fff9edff000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f4c8815c000)
    libpcre.so.0 => /lib64/libpcre.so.0 (0x00007f4c87f30000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f4c87d10000)
    libaprutil-1.so.0 => /usr/lib64/libaprutil-1.so.0 (0x00007f4c87aec000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f4c878b5000)
    libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f4c8768c000)
    libdb-4.7.so => /lib64/libdb-4.7.so (0x00007f4c87318000)
    libapr-1.so.0 => /usr/lib64/libapr-1.so.0 (0x00007f4c870e9000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4c86ecb000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f4c86b38000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f4c86934000)
    /lib64/ld-linux-x86-64.so.2 (0x000000369be00000)
    libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f4c8672f000)
    libfreebl3.so => /lib64/libfreebl3.so (0x00007f4c864cd000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f4c862c5000)

Solution 1

After taking a look at the ldd information, it looks like your httpd binary is compiled against different libraries than are currently installed. I would purge as many of the apache dependencies as is safe to do so and reinstall them. Before you do that, though, I would also check to see if you have any non-standard yum repositories installed. Certain of the third-party yum repositories are famous for causing some terrible conflicts that look somewhat similar to this.

For starters:

yum remove apr-devel apr-util-devel httpd

Unfortunately, even if this does solve the problem, you may already be neck deep in system incompatibility. It looks like whoever administered this system before you either compiled things from source, installed RPMs that were incompatible, or misused third-party repos. You may spend less time reinstalling the system than trying to clean it up.

Solution 2

a) Update all os provided packages:

yum clean all   # Clean yum cache so all fresh data will be downloaded
yum --disablerepo='*' --enablerepo=base --enablerepo=updates update

b) Check if there are installed any additional packages besides available from default CentOS repositories — base and updates:

yum clean all   # Clean yum cache so all fresh data will be downloaded
yum --disablerepo='*' --enablerepo=base --enablerepo=updates list extras

If there are any, which look relevant, then disable all repositories other than these two and downgrade/upgrade these packages to distribution provided versions.

c) Verify files of all packages:

rpm -Va

Only modified configuration files should be printed as modified (look for "5" at 3rd character of each line printed). Certainly nothing in /lib, /lib64, /usr/lib, /usr/lib64, /bin or /usr/bin. If anything there is modified then it looks like somebody has overwritten distribution-provided binaries — either administrator with no clue or a black hat hacker with a rootkit. In that case you can reinstall packages that contain these modified binaries, but a full reinstall is recommended, as it's very hard to find out what else was modified (maybe installing the same set of packages on another computer or virtual machine and comparing).

d) If nothing above helps then I'd suspect a sophisticated rootkit. A full reinstall is strongly recommended then.

Solution 3

I ran into this issue on centos 5.8. Before removing packages, check your /usr/local/lib - someone installed a later version of apr onto this server and caused this problem (subversion is run on this server with dav, so I needed the httpd to start up).

You can check this by: /usr/sbin/httpd -V

output Server loaded: APR 1.3.3, APR-Util 1.2.7 Compiled using: APR 1.2.7, APR-Util 1.2.7

APR 1.3.3 was in /usr/local/lib. I moved the library of /usr/local/lib (I know this is bad) and then httpd started up.

Comments

[Scrivener]

It looks like your httpd binary is compiled against libraries which no longer exist, or it was compiled against a version different than the one currently on the disk. What's the output of ldd $(which httpd)

[Arthur Frankel]

Updated in original.

[Arthur Frankel]

That didn't get it working, but I understand your point and will probably have to reinstall the system. I even tried a few times to uninstall apache, but no success.

[Tometzky]

I don't think this is the case. ldd is almost the same as in my recently installed CentOS 6.3. Only librt reference is absent in my system, but this is a special kind of library.

[Arthur Frankel]

any other suggestions to try?

[Scrivener]

I'd do a careful vetting of the yum.repos.d and check the versions to make sure the installed versions of the packages are the same as the latest available in the main CentOS repos... and if they're newer, you can attempt a downgrade.

[Arthur Frankel]

Thank you for your comment. We decided to do a full reinstall.