apt-get upgrade openssl won't bring Ubuntu 12.04 to latest version

Question

I've tried the following, but I can't get a build date later than:

Tue, Aug 21 05:18:46 UTC 2012

I have done the following:

apt-get dist-upgrade
apt-get update
apt-get upgrade openssl

and

apt-get purge openssl
apt-get install openssl

and

apt-get purge libcrypto1.0.0
apt-get install libcrypto1.0.0

Everything seems to work fine, but the build date remains as above. And the http://filippo.io/Heartbleed/ test is still failing.

I know I'm not crazy, because I was able to update my identical server Wednesday. (Identical in EVERY way except hardware).

EDIT:

I compared the /etc/apt/sources.list files on both machines and they appear to be identical. How did one server update and the other won't?

EDIT:

Did as suggested:

apt-get purge openssl
reboot
apt-get install openssl
reboot

to no avail.

Tried same commands on libssl1.0.0, still same version as listed above.

This one's got me stumped.

Any suggestions?

EDIT

As soon as I get enough street cred (15), I will +1 the viable workarounds

EDIT

As suggested, I ran apt-get with --reinstall --print-uris and got back:

http://us.archive.ubuntu.com/ubuntu/pool/main/o/openssl/openssl_1.0.1-4ubuntu5.12_amd64.deb

Then rebooted, same version as listed above. Still failing heartbleed.

Solution 1

Forcing a package re-installation

Apt thinks that the packages are installed and upgraded. Manual verification and testing of libssl suggests otherwise, so the package database is inconsistent with the installed files (maybe files were previously upgraded previously without package manager involvement). For whatever reason this happened, the package is not being correctly upgraded or reinstalled. This assumes that it has been established that the system is reporting fixed versions, but is still showing as vulnerable.

First, attempt to forcibly reinstall the affected packages:

apt-get install --reinstall libssl1.0.0

If that fails, try forcing the complete removal of the package without involving any of Apt's dependency management and sanity checks:

dpkg --force-all --remove libssl1.0.0

At this point, the system is effectively 'broken' because libssl is missing and many packages are still installed that depend on it (this is what Apt tries so hard to prevent, and the reason we are going behind Apt's back), so reinstall libssl1.0.0, re-downloading the latest package from the repository:

apt-get clean && apt-get install libssl1.0.0

Alternatively, if you have downloaded the known good deb package, you can use dpkg to install and force overwrite of any existing files:

dpkg --force-overwrite -i libssl1.0.0_1.0.1-4ubuntu5.12_amd64.deb

Re-test and check (debsums, sha1sum) the installed files against known good configuration.

Solution 2

If your apt-get repositories don't contains any precompiled 1.0.1g OpenSSL version, so just download sources from official website and compile it.

Below the single command line to compiling and install the last openssl version.

curl https://www.openssl.org/source/openssl-1.0.1g.tar.gz | tar xz && cd openssl-1.0.1g && sudo ./config && sudo make && sudo make install

Replace old openssl binary file by the new one via a symlink.

sudo ln -sf /usr/local/ssl/bin/openssl `which openssl`

You are all good !

# openssl version should return
openssl version
OpenSSL 1.0.1g 7 Apr 2014

Cf this blog post.

NB: As stated in the blog post, this workaround will not fix "Nginx and Apache server who have to be recompile with 1.0.1g openSSL sources."

Comments

[Maxx Daymon]

Did you reboot? Old/vulnerable copies will remain in use/memory so long as the processes using them do.

[Maxx Daymon]

What is reported for dpkg-query --list libssl1.0.0 next, what is "Filename:" from apt-cache show libssl1.0.0 next, does that match the file you get when you apt-get download libssl1.0.0?

[Maxx Daymon]

Also helpful, apt-get clean to clear your download cache, then apt-get install openssl libssl1.0.0 --reinstall --print-uris and check the download URIs. Download the packages manually to inspect.

[user1182988]

Thanks Maxx, I ran the dpkg-query and apt-get download, and the versions match. I then ran apt-get clean, downloaded the packages and inspected their content, which matched the dpkg-query versions. Does that mean the repository is broken?

[Maxx Daymon]

@user1182988 The repository looks good from here. I downloaded and verified the packages and they were correct. You can manually download the libssl and openssl deb packages and install them, but it looks like apt is selecting and installing the patch. Does your sha1sum for /usr/bin/openssl match 22297ff89bbdfd4befb6f66f79e41210768aa81a and /lib/x86_64-linux-gnu/libssl.so.1.0.0 match 4fbb25a3e82d3d835c68421a6c4647afe534b393? What server(s) are you using, and is there a chance you have a statically linked libssl in it/them?

[Maxx Daymon]

@user1182988 Another test you can do is to download the .deb package and run debsums libssl1.0.0_1.0.1-4ubuntu5.12_amd64.deb to verify that the files you have installed match the sums from the known good package. Do a dpkg -i filename.deb to install manually. OpenSSL should return OpenSSL 1.0.1 14 Mar 2012, built on: Mon Apr 7 20:33:29 UTC 2014 Your build date implies that the package is not getting installed. Check dpkg and apt logs for errors?

[user1182988]

@MaxxDaymon, Thanks for helping. I ran sha1sum on openssl, and the checksum matches the one you provided, however the libssl.so.1.1.0 did not match. I compared to my "identical" server, and both checksums matched. I searched my dpkg and apt logs, but saw nothing that looked like an error. The servers are typical LAMP with postfix and dovecot, and associated spam/security apps. I was about to remove libssl1.0.0, but it told me that there were too many things relying on it, and that I should reconsider unless I "Really know what I'm doing", so I reconsidered. will a dpkg-reconfigure fix it?

[Maxx Daymon]

@user1182988 Can you do a apt-cache policy libssl1.0.0 and tell me what your Installed: and Candidate: lines are?

[Maxx Daymon]

@user1182988 I think I would recommend a wget http://us.archive.ubuntu.com/ubuntu/pool/main/o/openssl/libs‌​sl1.0.0_1.0.1-4ubunt‌​u5.12_amd64.deb followed by a dpkg --force-all --remove libssl1.0.0 and then a dpkg --force-overwrite -i libssl1.0.0_1.0.1-4ubuntu5.12_amd64.deb (You could also do a plain apt-get install libssl1.0.0 instead, eliminating the need to download the deb) I just tried both approaches on a 12.04 test server and they both worked.

[user1182988]

@MaxxDaymon, you nailed it. I ran dpkg --force-all --remove libssl1.0.0 followed by apt-get install libssl1.0.0, and now my build is 4/7/14. I learned that dpkg will remove a package without removing all of the stuff that depends on it (right?). Thanks Maxx. Make an answer out of this and I will check it.

[Maxx Daymon]

@user1182988 Fantastic! I'm stepping out for a few hours, I'll write it up in the form of an answer when I return. Glad to hear you're back in business!

[Maxx Daymon]

@user1182988 I posted an answer. It was a bit wall-of-text, so I reduced it to the essentials since we've got a lot of the diagnostics steps in the question now.

[user1182988]

Yes that's a great and simple workaround. The thing is, I know the binaries are available since the upgrade worked on Wednesday for my identical server. I thought perhaps that Ubuntu pulled it from the repository.

[user1182988]

Thanks for the reply. I did what you suggested (with -a) an got the same build date response. I have avoided compiling my own version of anything, as I need these servers to be easily-reproducible.

[Ramhound]

@user1182988 - I assume transferring the files from the working sever isn't an option?

[user1182988]

Thanks for the reply! I tried apt-get purge libssl1.0.0, reboot, apt-get install libssl1.0.0, reboot - same version as above. Tried the same thing using openssl, still same version and build mentioned above. This is weird!

[user1182988]

Transferring the files is a viable option, but I'm concerned that the apt-get won't work.

[user1182988]

Exactly right, and now it passes the Heartbleed test. Thanks so much!