Heartbleed: how to reliably and portably check the OpenSSL version?

Question

I was looking at a reliable and portable way to check the OpenSSL version on GNU/Linux and other systems, so users can easily discover if they should upgrade their SSL because of the Heartbleed bug.

I thought it would be easy, but I quickly ran into a problem on Ubuntu 12.04 LTS with the latest OpenSSL 1.0.1g:

openssl version -a

I was expecting to see a full version, but instead I got this:

OpenSSL 1.0.1 14 Mar 2012
built on: Tue Jun  4 07:26:06 UTC 2013
platform: [...]

To my unpleasant surprise, the version letter doesn't show. No f, no g there, just "1.0.1" and that's it. The listed dates do not assist in discovering a (non-)vulnerable version either.

The difference between 1.0.1 (a-f) and 1.0.1g is crucial.

Questions:

• What is a reliable way to check the version, preferably cross distro?
• Why isn't the version letter showing in the first place? I was unable to test this on anything else but Ubuntu 12.04 LTS.

Others are reporting this behaviour as well. A few examples:

• https://twitter.com/orblivion/status/453323034955223040
• https://twitter.com/axiomsofchoice/status/453309436816535554

Some (distro-specific) suggestions rolling in:

• Ubuntu and Debian: apt-cache policy openssl and apt-cache policy libssl1.0.0. Compare the version numbers to the packages here: http://www.ubuntu.com/usn/usn-2165-1/
• Fedora 20: yum info openssl (thanks @znmeb on twitter) and yum info openssl-libs

Checking if a older version of OpenSSL is still resident:

• It's not completely reliable, but you can try lsof -n | grep ssl | grep DEL. See Heartbleed: how to reliably and portably check the OpenSSL version? on why this may not work for you.

It turns out that updating the OpenSSL package on Ubuntu and Debian isn't always enough. You should also update the libssl1.0.0 package, and -then- check if openssl version -a indicates built on: Mon Apr 7 20:33:29 UTC 2014.

Solution 1

Based on the date displayed by your version of OpenSSL, it seems you are seeing the full version displayed there.

Open SSL 1.0.1 was released on March 14th, 2012. 1.0.1a was released on April 19th of 2012.

So, I'm going to go ahead and assert that openssl version -a is the proper, cross-distro way to display the full version of OpenSSL that's installed on the system. It seems to work for all the Linux distros I have access to, and is the method suggested in the help.ubuntu.com OpenSSL documentation, as well. Ubuntu LTS 12.04 shipped with vanilla OpenSSL v1.0.1, which is the version that looks like an abbreviated version, on account of not having a letter following it.

Having said that, it appears that there is a major bug in Ubuntu (or how they package OpenSSL), in that openssl version -a continues to return the original 1.0.1 version from March 14, 2012, regardless of whether or not OpenSSL has been upgraded to any of the newer versions. And, as with most things when it rains, it pours.

Ubuntu is not the only major distro in the habit of backporting updates into OpenSSL (or other packages), rater than relying on the upstream updates and version numbering that everyone recognizes. In the case of OpenSSL, where the letter version numbers represent only bug fix and security updates, this seems nearly incomprehensible, but I have been informed that this may be because of the FIPS-validated plugin major Linux distros ship packaged with OpenSSL. Because of requirements around revalidation that trigger due to any change, even changes that plug security holes, it is version-locked.

For example, on Debian, the fixed version displays a version number of 1.0.1e-2+deb7u5 instead of the upstream version of 1.0.1g.

As a result, at this time, there is no reliable, portable way to check SSL versions across Linux distributions, because they all use their own backported patches and updates with different version numbering schemes. You will have to look up the fixed version number for each different distribution of Linux you run, and check the installed OpenSSL version against that distribution's specific version numbering to determine if your servers are running a vulnerable version or not.

Solution 2

If you want something truly cross-platform, check for the vulnerability itself rather than relying on version numbers.

You might have code that reports a version number that is known to be vulnerable, but the actual code is not vulnerable. And the reverse -- silently vulnerable code -- could be even worse!

Many vendors who bundle open-source products like OpenSSL and OpenSSH will selectively retrofit urgent fixes to an older version of code, in order to maintain API stability and predictability. This is especially true for "long-term release" and appliance platforms.

But vendors who do this silently (without adding their own version string suffix) run the risk of triggering false positives in vulnerability scanners (and confusing users). So to make this transparent and verifiable, some vendors append their own strings to the major package version. Both Debian (OpenSSL) and FreeBSD (in OpenSSH, via the VersionAddendum sshd_config directive) sometimes do this.

Vendors who don't do this are probably doing so to minimize the chance of breakage due to the many direct and indirect ways that other programs check version numbers.

So it can look like this:

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.4 LTS"

$ openssl version
OpenSSL 1.0.1 14 Mar 2012

... even though it's been patched:

$ dpkg -l openssl | grep openssl
ii  openssl  1.0.1-4ubuntu5.12  [truncated]

$ ls -la `which openssl`
-rwxr-xr-x 1 root root 513208 Apr  7 12:37 /usr/bin/openssl

$ md5sum /usr/bin/openssl
ea2a858ab594905beb8088c7c2b84748  /usr/bin/openssl

With things like this in play, you're better off if you don't trust the version number.

Solution 3

Unfortunately, I'm not sure there is a cross-platform way of doing this. As I discuss in a blog posting, the version of OpenSSL displayed on Ubuntu 12.04 REMAINS 1.0.1 after upgrading to a fixed version.

For Ubuntu 12.04 ONLY, you can tell if you've been updated if all of the below are true:

1. dpkg -s openssl | grep Version shows version 1.0.1-4ubuntu5.12 or later.
2. dpkg -s libssl1.0.0 | grep Version shows version 1.0.1-4ubuntu5.12 or later.
3. openssl version -a shows a "built on" date of April 7, 2014 or later.

Thanks to @danny for the additional info.

Solution 4

Give the following a try. It will extract all the strings from the crypto library that ssh is linked against. It produces more than one line of output, but if necessary could be converted to 1 line.

ldd `which ssh` | awk '/\// { print $3 }' | grep crypto | xargs strings  | grep OpenSSL

produces

OpenSSLDie
DSA_OpenSSL
...
MD4 part of OpenSSL 1.0.1f 6 Jan 2014 
MD5 part of OpenSSL 1.0.1f 6 Jan 2014
... 
etc

e.g. on Gentoo before emerge

[ebuild     U  ] dev-libs/openssl-1.0.1f [1.0.1c] USE="bindist (sse2) tls-heartbeat%* zlib -gmp -kerberos -rfc3779 -static-libs {-test} -vanilla" 4,404 kB

the above command results in

...
OpenSSL 1.0.1c 10 May 2012

after

...
OpenSSL 1.0.1f 6 Jan 2014

Ouch, still no g.

Solution 5

Do any of these scripts test all services, or do they only test HTTPS? AFAIK, PostgreSQL is vulnerable, but that's only a rumor until an in-the-wild attack surfaces.

There is a metasploit script available for use.

https://github.com/rapid7/metasploit-framework/commit/dd69a9e5dd321915e07d8e3dc8fe60d3c54f551a

You can type this (tested with GnuWin32 OpenSSL binary version 1.0.1.6, dated 2014-01-14), or just use the script in the comment below this one. It's more accurate and simpler!

s_client -connect a23-75-248-141.deploy.static.akamaitechnologies.com:443 -debug -state

Once connected type B and you'll see on a vulnerable host and you won't be disconnected:

B

HEARTBEATING
write to 0x801c17160 [0x801cbc003] (66 bytes => 66 (0x42))
0000 - 18 03 03 00 3d 8f 6f 3c-52 11 83 20 9c a2 c0 49   ....=.o 5 (0x5))
0000 - 18 03 03 00 3d                                    ....=
read from 0x801c17160 [0x801cb7008] (61 bytes => 61 (0x3D))
0000 - 05 4d f5 c0 db 96 d1 f5-c7 07 e5 17 1f 3b 48 34   .M...........;H4
0010 - 6e 11 9d ba 10 0c 3a 34-eb 7b a5 7c c4 b6 c0 c0   n.....:4.{.|....
0020 - b0 75 0e fe b7 fa 9e 04-e9 4e 4a 7d 51 d3 11 1f   .u.......NJ}Q...
0030 - e2 23 16 77 cb a6 e1 8e-77 84 2b f8 7f            .#.w....w.+..
read R BLOCK

You will get a heartbeat response that looks similar to this one.

On a patched host, you will see a response similar to below and you'll be disconnected:

Enter B

HEARTBEATING
write to 0x801818160 [0x8019d5803] (101 bytes => 101 (0x65))
0000 - 18 03 03 00 60 9c a3 1e-fc 3b 3f 1f 0e 3a fe 4c   ....`....;?..:.L
0010 - a9 33 08 cc 3d 43 54 75-44 7d 2c 7b f3 47 b9 56   .3..=CTuD},{.G.V
0020 - 89 37 c1 43 1c 80 7b 87-66 ff cb 55 5f 8d 1a 95   .7.C..{.f..U_...
0030 - 1b 4c 65 14 21 a1 95 ac-7a 70 79 fc cc a0 cf 51   .Le.!...zpy....Q
0040 - 0f 7e c5 56 14 c8 37 c1-40 0b b8 cb 43 96 8a e6   [email protected]...
0050 - 21 42 64 58 62 15 fb 51-82 e6 7f ef 21 1b 6f 87   !BdXb..Q....!.o.
0060 - b9 c2 04 c8 47                                    ....G

Source:

• Blog post How to test if your OpenSSL heartbleeds

There's also these tools:

• https://github.com/titanous/heartbleeder

• http://filippo.io/Heartbleed/

• https://github.com/musalbas/heartbleed-masstest

View more solutions

Comments

[ppp]

at least be sure that the OpenSSL version you have isn't g because of the date it shows

[Snowlockk]

This works on CentOS [root@null~]# openssl version -a OpenSSL 1.0.1e-fips 11 Feb 2013

[Martijn]

@PatoSáinz I've checked apt-cache policy openssl and it responded with: Installed: 1.0.1-4ubuntu5.12 which is the 1.0.1g just released by Ubuntu for 12.04 LTS. I logged out and back in. Is there anything else I could do to verify?

[HopelessN00b]

I'll point out, for that that don't know, in case it's helpful... Ubuntu 12.04 LTS shipped with OpenSSL 1.0.1 (vanilla).

[Anti-weakpasswords]

If that build date is accurate, you cannot have "release versioned" code more recent than 1.0.1e, since 1.0.1f came out in 2014 per the OpenSSL 1.0.1 release notes. Individual lines or sections may have been backported to your Ubuntu version prior to the official OpenSSL 1.0.1f release, of course. And the build date may be less that completely helpful.

[VL-80]

Gentoo users could just run emerge -av openssl. Considering you have fresh portage tree it will offer you to update openssl and your current version with letter will be in output.

[Camille Le Mouëllic]

The apt-cache policy openssl worked for me on Debian and Ubuntu, you might want to add it in your question under "distro-specific".

[Martijn]

My install is a simple Ubuntu 12.04 LTS without anything I've compiled myself or downloaded from other sources than the Ubuntu repositories. If Ubuntu is distributing OpenSSL with abbreviated version numbers, then openssl version -a isn't a portable method (at least not portable to Ubuntu). I've checked apt-cache policy openssl and it responded with: Installed: 1.0.1-4ubuntu5.12 which is the 1.0.1g just released by Ubuntu for 12.04 LTS. I logged out and back in before checking.

[HopelessN00b]

@Martijn ...and what does openssl version -a report now that you've relogged? The March 14th, 2012 date is a dead give-away to the vanilla 1.0.1 version.

[Martijn]

It's still giving me OpenSSL 1.0.1 14 Mar 2012 and built on: Tue Jun 4 07:26:06 UTC 2013. This is reported by the binary in /usr/bin/openssl and I can't find any other openssl binaries on the system.

[HopelessN00b]

Oh, Ubuntu... of course it's too much to ask that they'd do anything that might be similar to the way every other Linux distro in the world does things. I'm getting dangerously close to spinning up a test Ubuntu VM. <sigh>

[EEAA]

Cripes, this ubuntu bug just caused several minutes of confusion and mild panic on my part. How stupid is that.

[Martijn]

Ok, in that case I must add that package version 1.0.1-4ubuntu5.12 is ONLY for Ubuntu 12.04 LTS. If you're on Ubuntu 12.10 you should see at least version 1.0.1c-3ubuntu2.7 and if you're on 13.10 then it should be at least version 1.0.1e-3ubuntu1.2, according to source: ubuntu.com/usn/usn-2165-1

[mattdm]

Useful, but doesn't tell you if you have a version with the extension and the fix.

[MadHatter]

HopelessN00b, there is nothing dubious about the policy of backporting fixes instead of bumping versions; it's a very good way of ensuring platform stability, which is highly-desirable in a server environment. Like any decision, it has consequences, which users need to be aware of; but just because it breaks the "I'm running foo x.y.z therefore I am/amn't vulnerable to the latest exploit" line of reasoning, that doesn't make it a bad thing.

[danny]

This is unfortunately insufficient. You must also upgrade libssl1.0.0 explicitly on ubuntu. If you're seeing a built on date before April 7 2014 even if openssl is version looks correct (1.0.1-4ubuntu5.12 for Ubuntu 12.04) you are probably still vulnerable.

[Adam]

Exactly what @MadHatter says. Ensure you're updated, and you're good to go. Just because it breaks a "copy and paste this command" scheme of "ensuring" you're running a safe version, it isn't stupid. [If you're about to tell me "nagios check", I'll respond with "check for vulnerability instead of version number".]

[HopelessN00b]

@towo Version numbers exist for a reason. If we're just gonna chuck the upstream version numbers out the window because "enterprisey," or whatever, why bother with version numbers at all? May as well just start naming all our stuff with alliterations. We can call the vulnerable OpenSSL versions Holy Heartbleed and the fixed ones Cunning Coagulant.

[NickW]

@HopelessN00b I think you're getting caught up on the "this was fixed in version X.Y.Z" trap, they don't follow the version numbers because all that's getting imported into the latest version are bug and security fixes. If they bumped the version number, you'd also expect the additional functionality.. "I have OpenSSL v X.Y.Z, why don't I have ECDHA???? ..etc". It makes sense when you understand that it's only bugfixes.

[Martijn]

I thought you were very close to providing a good solution, but unfortunately this doesn't work for the crypto library on Ubuntu 12.04 LTS. It displays all the strings with version [...] part of OpenSSL 1.0.1 14 Mar 2012, the same way as openssl version -a does. This is a trick that may work in other cases though!

[Schof]

@danny You just saved me SO much work. I was trying to figure out why the build date was right on some 12.04 systems and wrong on others. You're a lifesaver!

[Martijn]

I was unable to find any other openssl binaries or crypto libraries. It is suggested by others that the difference is that, on 12.04 LTS, Ubuntu is backporting the changes to 1.0.1 without upping the version. While 12.10 isn't an LTS and so Ubuntu uses the latest version there instead of a backport.

[Jubal]

@HopelessN00b: it's called ‘backporting security fixes’; there are no functionality updates. So Debian Wheezy's version is what it says, 1.0.1e + the security fixes, not the latest.

[HopelessN00b]

@NickW @Jubal @MadHatter the thing with OpenSSL, though, is that: After the release of OpenSSL 1.0.0 the versioning scheme changed. Letter releases (e.g. 1.0.1a) can only contain bug and security fixes and no new features. So, nothing is gained abandoning the upstream versioning scheme; backporting the updates is essentially the same as using the updated version, since the update only includes security and bug fixes anyway. What it does do is confuse things and leave us with no way to portably check OpenSSL version across Linux distros.

[Stefan Lasiewski]

This is indeed a good way to check for the vulnerability and is what some scripts do. It does not actually require SSH access.

[voretaq7]

BIG SCARY IMPORTANT WARNING - The machine you are running openssl s_client on MUST be using OpenSSL 1.0.1 or later in order for this to work. If you run this command on a machine with 0.9.8 or 1.0.0 IT WILL ALWAYS REPORT "Safe", even for vulnerable servers.

[Martijn]

It's clear that checking versions isn't as easy and transparant as I hoped it would be. Checking for the vulnerability is cross-platform, but also more difficult to do: you must have a reliable PoC or test handy for the particular vulnerable software service you're running. In this case it all started out with a PoC for Apache and nginx. What if I was only using SMTP with SSL at that moment, and I wanted to check if I'm vulnerable? Eventually we'll have tests for most services, but it may take a while.

[Royce Williams]

Martijn, that's a fair point. When a test is not available, secondary methods (like tracking down the checksum for the affected binaries on your target systems) are less optimal, but might be good enough to get the job done ... and then move on to the next fire. :-)

[Michael]

Odd. I'm running a version of OpenSSL that is supposedly affected by this bug, yet that string doesn't appear in the output...

[egarcia]

@StefanLasiewski I updated my answer and removed the "need ssh" part

[egarcia]

@voretaq7 Thanks for your comment. I added it at the end of my answer

[egarcia]

@Michael That probably means that even if the ssl version is "insecure", it doesn't have the module that manifests the vulnerability active.

[Stefan Lasiewski]

@voretaq7 : But according to CVE-2014-0160, OpenSSL 0.9.8 and 1.0.0 are not subject to the Heartbleed attack. So in that case, this is a valid test. OpenSSL 0.9.8 and 1.0.0 are 'safe' from Heartbleed.

[Stefan Lasiewski]

But overall, this answer doesn't really answer the question "how to reliably and portably check the OpenSSL version?".

[Patrick James McDougle]

openssl version -a might not need the build date of April 7, because the fix is being backported to the older releases.

[Steve Townsend]

@HopelessN00bGeniusofnetwork We can easily check versions which is a different matter than checking fixes (or backports). As for easily checking on Debian/Ubuntu: aptitude changelog libssl1.0.0 | grep -B10 CVE-2014-0160