How to patch the Heartbleed bug (CVE-2014-0160) in OpenSSL?

Question

As of today, a bug in OpenSSL has been found affecting versions 1.0.1 through 1.0.1f (inclusive) and 1.0.2-beta.

Since Ubuntu 12.04, we are all vulnerable to this bug. In order to patch this vulnerability, affected users should update to OpenSSL 1.0.1g.

How can every affected user apply this update now?

Solution 1

Security updates are available for 12.04, 12.10, 13.10 and 14.04 see Ubuntu Security Notice USN-2165-1.

So first you need to apply the available security updates, for example by running

sudo apt-get update
sudo apt-get upgrade

from the command line.

Do not forget to restart the services (HTTP, SMTP, etc.) that use the affected OpenSSL version, otherwise you are still vulnerable. See also Heartbleed: What is it and what are options to mitigate it? on Serverfault.com.

The following command shows (after an upgrade) all services that need to be restarted:

sudo find /proc -maxdepth 2 -name maps -exec grep -HE '/libssl\.so.* \(deleted\)' {} \; | cut -d/ -f3 | sort -u | xargs --no-run-if-empty ps uwwp

After that, you need to regenerate all server SSL keys, then evaluate whether your keys may have leaked, in which case attackers may have retrieved confidential information from your servers.

Solution 2

The bug is known as Heartbleed.

Am I vulnerable?

Generally, you're affected if you run some server that you generated an SSL key for at some point. Most end-users are not (directly) affected; at least Firefox and Chrome don't use OpenSSL. SSH is not affected. The distribution of Ubuntu packages isn't affected (it relies on GPG signatures).

You are vulnerable if you run any kind of server that uses OpenSSL versions 1.0–1.0.1f (except of course versions that were patched since the bug was discovered). The affected Ubuntu versions are 11.10 oneiric through 14.04 trusty pre-releases. It's an implementation bug, not a flaw in the protocol, so only programs that use the OpenSSL library are affected. If you have a program linked against the old 0.9.x version of OpenSSL, it isn't affected. Only programs that use the OpenSSL library to implement the SSL protocol are affected; programs that use OpenSSL for other things are not affected.

If you ran a vulnerable server exposed to the Internet, consider it compromised unless your logs show no connection since the announcement on 2014-04-07. (This assumes that the vulnerability wasn't exploited before its announcement.) If your server was only exposed internally, whether you need to change the keys will depend on what other security measures are in place.

What is the impact?

The bug allows any client who can connect to your SSL server to retrieve about 64kB of memory from the server. The client doesn't need to be authenticated in any way. By repeating the attack, the client can dump different parts of the memory in successive attempts.

One of the critical pieces of data that the attacker may be able to retrieve is the server's SSL private key. With this data, the attacker can impersonate your server.

How do I recover on a server?

1. Take all affected servers offline. As long as they're running, they're potentially leaking critical data.

2. Upgrade the libssl1.0.0 package, and make sure that all affected servers are restarted.
   You can check if affected processes are still running with ``grep 'libssl.(deleted)' /proc//maps`

3. Generate new keys. This is necessary because the bug might have allowed an attacker to obtain the old private key. Follow the same procedure you used initially.

   • If you use certificates signed by a certification authority, submit your new public keys to your CA. When you get the new certificate, install it on your server.
   • If you use self-signed certificates, install it on your server.
   • Either way, move the old keys and certificates out of the way (but don't delete them, just ensure they aren't getting used any more).

4. Now that you have new uncompromised keys, you can bring your server back online.

5. Revoke the old certificates.

6. Damage assessment: any data that has been in the memory of a process serving SSL connections may potentially have been leaked. This can include user passwords and other confidential data. You need to evaluate what this data may have been.

   • If you're running a service that allows password authentication, then the passwords of users who connected since a little before the vulnerability was announced should be considered compromised. (A little before, because the password may have remained unused in memory for a while.) Check your logs and change the passwords of any affected user.
   • Also invalidate all session cookies, as they may have been compromised.
   • Client certificates are not compromised.
   • Any data that was exchanged since a little before the vulnerability may have remained in the memory of the server and so may have been leaked to an attacker.
   • If someone has recorded an old SSL connection and retrieved your server's keys, they can now decrypt their transcript. (Unless PFS was ensured — if you don't know, it wasn't.)

How do I recover on a client?

There are only few situations in which client applications are affected. The problem on the server side is that anyone can connect to a server and exploit the bug. In order to exploit a client, three conditions must be met:

• The client program used a buggy version the OpenSSL library to implement the SSL protocol.
• The client connected to a malicious server. (So for example, if you connected to an email provider, this isn't a concern.) This had to happen after the server owner became aware of the vulnerability, so presumably after 2014-04-07.
• The client process had confidential data in memory that wasn't shared with the server. (So if you just ran wget to download a file, there was no data to leak.)

If you did that between 2014-04-07 evening UTC and upgrading your OpenSSL library, consider any data that was in the client process's memory to be compromised.

References

• The Heartbleed Bug (by one of the two teams who independently discovered the bug)
• How exactly does the OpenSSL TLS heartbeat (Heartbleed) exploit work?
• Does Heartbleed mean new certificates for every SSL server?
• Heartbleed: What is it and what are options to mitigate it?

Solution 3

To see which OpenSSL version is installed on Ubuntu run:

dpkg -l | grep openssl

If you see the following version output, patch for CVE-2014-0160 should be included.

ii  openssl      1.0.1-4ubuntu5.12      Secure Socket Layer (SSL)...

Looking at https://launchpad.net/ubuntu/+source/openssl/1.0.1-4ubuntu5.12, it shows which kind of bugs are fixed:

...
 SECURITY UPDATE: memory disclosure in TLS heartbeat extension
    - debian/patches/CVE-2014-0160.patch: use correct lengths in
      ssl/d1_both.c, ssl/t1_lib.c.
    - CVE-2014-0160
 -- Marc Deslauriers <email address hidden>   Mon, 07 Apr 2014 15:45:14 -0400
...

Solution 4

If your apt-get repositories don't contains any precompiled 1.0.1g OpenSSL version, so just download sources from official website and compile it.

Below the single command line to compiling and install the last openssl version.

curl https://www.openssl.org/source/openssl-1.0.1g.tar.gz | tar xz && cd openssl-1.0.1g && sudo ./config && sudo make && sudo make install

Replace old openssl binary file by the new one via a symlink.

sudo ln -sf /usr/local/ssl/bin/openssl `which openssl`

You are all good !

# openssl version should return
openssl version
OpenSSL 1.0.1g 7 Apr 2014

Cf this blog post.

NB: As stated in the blog post, this workaround will not fix "Nginx and Apache server who have to be recompile with 1.0.1g openSSL sources."

Solution 5

For those who do not want to do a serverwide package upgrade. I read a bunch of these guides today and apt-get upgrade openssl === apt-get upgrade this will apply all security fixes required by your machine. Wonderful, unless you are explicitly leaning on an old package version somewhere.

This is the minimal action required on Ubuntu 12.04 LTS running Apache 2:

• Go to this address and prove you have the vulnerability. You should use the DIRECT EXTERNAL ADDRESS OF YOUR WEB SERVER. If you use a loadbalancer (for example ELB) you might not be contacting your web server directly.

• Run the following 1 liner to upgrade packages and restart. Yes I saw all the guides saying that you should have a timestamp later than April 4th 2014, this doesn't seem to be the case to me.

  apt-get update && apt-get install openssl libssl1.0.0 && /etc/init.d/apache2 restart

• Ensure you have appropriate package versions installed and check your webserver for the vulnerability once more.

The key packages are as follows, I determined this information using the command below then edited away the cruft (you don't need to know that much about the state of my machines).

$ dpkg -l | grep ssl

ii  libssl-dev                       1.0.1-4ubuntu5.12          SSL development libraries, header files and documentation
ii  libssl1.0.0                      1.0.1-4ubuntu5.12          SSL shared libraries
ii  openssl                          1.0.1-4ubuntu5.12          Secure Socket Layer (SSL)* binary and related cryptographic tools

1.0.1-4ubuntu5.12 should NOT contain the vulnerability. Ensure this is the case by again going to the website below, and testing your web server.

http://filippo.io/Heartbleed/

View more solutions

Comments

[Admin]

Do you have an affected version of openssl?

[Admin]

I've got the patched version 1.0.1-4ubuntu5.12 and I've restarted apache service but filippo.io/Heartbleed testing on my site still says I'm vulnerable Any idea why ?

[Admin]

@Mat I don't know what that site tests, but maybe it detects that you're using an old key. Since the key may have leaked, you need to regenerate it.

[Admin]

You really don't want to update OpenSSL to wholesale new versions, that's an unbelievable pain. Far easier is to just install the updated package that patches the issue: ubuntu.com/usn/usn-2165-1

[Admin]

have you restarted your services after upgrading?

[Christopher B. Adkins]

As usually Ubuntu doesn't provide the new upstream version but patched the versions for all the supported releases to keep the changes at a minimum.

[Paul Cantrell]

Not sure this works on Ubuntu 12.04.4 LTS. After full update, openssl version gives OpenSSL 1.0.1 14 Mar 2012. That’s not the patched version, right? Or am I misreading it?

[Frodik]

What to do with Ubuntu 13.04 ? No upgraded openssl available :-(

[Riking]

Make sure to REBOOT your server processes! Otherwise the update won't actually take effect.

[Reza Mohammadi]

Is it tested on ubuntu 12.04? I upgraded and restated but apt-cache show libssl1.0.0 gives Version: 1.0.1-4ubuntu3. And openssl version gives OpenSSL 1.0.1 14 Mar 2012.

[dan]

On Ubuntu 12.04 even the fixed OpenSSL displays version 1.0.1 14 Mar 2012. Read crimi's answer to find out whether your installation is including the fix.

[Paul Cantrell]

Thanks, @dan! Resummarizing @crimi's answer here: if you run dpkg -l | grep ' openssl ' and you get 1.0.1-4ubuntu5.12 then you’re good to go.

[Alexander M. Turek]

If you're on 13.04, you won't receive updated packages. However, you can download and install the saucy packages for openssl and libssl1.0.0 – they will work. Anyway, you should consider moving on to 13.10.

[h3.]

Patching and restarting is not enough. You need to regenerate keys and assess whether your keys have been leaked as well as other confidential material. See e.g. Does Heartbleed mean new certificates for every SSL server?

[h3.]

@NewWorld The bug may have allowed an attacker to retrieve the keys (see my answer for a short explanation and follow the references if you want more). The only way you can be sure is if no one who isn't you has made any connection to your server since the bug was announced (assuming the bug wasn't exploited before its public announcement).

[toxaq]

I've upgraded and get version 5.12 but this tool still tells me I'm vulnerable filippo.io/Heartbleed Thoughts?

[crimi]

I have tested our updated servers over this side and it told me that I'm not affected. Did you reboot your system, or at least are you sure that all necessary processes have been restarted?

[armb]

I don't believe "Only the server side of SSL/TLS connections is affected" is true. openssl.org/news/secadv_20140407.txt says it can reveal secrets from either client or server. ubuntu.com/usn/usn-2165-1 agrees. The chances that you are using client certificates while connecting to a malicious server are small, but the possibility exists.

[h3.]

@armb You make a good point. It doesn't even matter whether client certificates are used, the data leakage is unrelated to the use of certificates. I've enlisted the help of professionals.

[Scott Arciszewski]

If you're on 13.04, pastebin.com/SF7StZBX this bash script will upgrade opnessl to 1.0.1g

[Fabian]

You can use checkrestart (should come with Debian, is inside the debian-goodies package in Ubuntu) for determing which processes need restarting.

[rurban]

restarting apache-ssl is not done by just apachectl graceful (the typical restart). you need to stop it and start-ssl it to get the patched version in all sessions.

[Lucio]

This answer and @Scott script is missing to verify the integrity of the files!

[Tom Hert]

After updating the OPENSSL, all I had to do was to restart the apache service, but graceful did not helped. I had to go and restart by using sudo service apache2 restart

[dAngelov]

Note: Make sure you restart your server after updating OpenSSL. Apache and Nginx picked up the new lib and the vulnerability was closed.

[dan]

I can confirme that the test reports no vulnerability for our precise maschine after the update to version 1.0.1-4ubuntu5.12.

[toxaq]

@crimi yes, I unfortunately rebooted several times to see if that made a difference. Seems upgrading just openssl was insufficient and that I had to do an apt-get upgrade on everything. I'm guessing the ssl libraries were the problem...

[Rinzwind]

Using an external site to prove a vulnerability in a server seems to be the wrong approach to me.

[Adrian]

External vulnerability testing scripts are becoming more and more commonplace these days. It does exactly what an internal script does, the connection is just initiated from an external webserver. You can look to sites like WhiteHatSecurity.com for an example of a program that initiates all connections remotely. There are instances where this wouldn't fly, for example network vulnerability testing but for testing a forward facing webserver (which in general an SSL server will be) this is almost ideal.

[hsmiths]

Thank you! Seems odd that Ubuntu 13.04 isn't getting this update. Your recipe plugs the hole.

[sarnold]

Heh, now that I take the time to read the details of this posting, I'm even more aghast -- downloading a tarball from some random place off the Internet, unpacking, and executing parts of it as root is just reckless behaviour. It'd be slightly better if the tarball signatures were downloaded and checked, but making sure you validate the signatures were signed by the right key is itself a difficult question. Distributions have already gone to the effort to ensuring safe provenance of tarballs and patches. Thanks.

[Braiam]

Why install the package if it's being upgraded?

[Andreas Roth]

I just found the cause of my vulnerability: i had mod-spdy-beta installed. After removing it and restarting apache all tests are green now.

[Andreas Roth]

or just install version 0.9.4.2 of mod-spdy-beta which fixes the heartbleed issue.

[topher]

apt-get install openssl libssl1.0.0 did it for me. Running openssl version -a now shows: built on: Mon Apr 7 20:33:29 UTC 2014

[armb]

Client certificates are the case where you would leak private keys, but yes, passwords, authorization cookies etc. could leak anyway. However, with an OpenSSL based client like curl or wget in typical usage, you wouldn't have secrets for other sites in memory while connecting to a malicious server, so in that case I think the only leakage would be if you gave the client secrets anticipating giving them to a legitimate site, and Heartbleed leaked them during handshake before certificate verification reveals you aren't connected to the right site.

[armb]

(RFC 6520 says heartbeat messages received during handshake SHOULD be silently dropped, but heartbleed.com says they are replied to.) AIUI Chrome on Android uses OpenSSL, but mainstream browsers on Ubuntu don't.

[armb]

Note that "I doubt that clients have any use for the Heartbeat Extension" does not mean that clients using OpenSSL won't reply to useless Heartbeat messages anyway.

[Sylwester]

@Scott Ubuntu has already released patches. Replacing it will make it less secure as it then falls off future updates by Ubuntu security team.

[Ethan Leroy]

ubuntu.com/usn/usn-2165-1 says that I have to install openssl version 1.0.1-4ubuntu5.12, but when I run apt-get update && aptitude show openssl it says that the newest version is 1.0.1-4ubuntu5.10. My mirror is cz.archive.ubuntu.com/ubuntu

[Lekensteyn]

Gilles, the /proc/*/fd trick does not work as processes do not keep fds open to libraries. Check /proc/*/maps instead.

[David]

I posted a solution for 13.04 here: askubuntu.com/questions/445710/…

[h3.]

@Lekensteyn Indeed, I don't know why I wrote that. Thanks for the tip

[h3.]

@armb I've updated my analysis of the client side. If you have analysis of programs that may be concretely exploitable (p2p, maybe?), I'm all ears.

[Lekensteyn]

@Gilles You might be interested in the answers to What clients are proven to be vulnerable to Heartbleed?. I managed to gain "interesting" memory on nginx (proxy mode), wget, links and others.

[armb]

There is another case where clients are vulnerable - because heartbeat requests in some cases can be accepted before certificate verification (thanks to @Lekensteyn for actual testing), and aren't included in the handshake HMAC, then a man in the middle can send them when you connect to a legitimate server. So if you are a possible NSA target (or Chinese Ministry of State Security, or other similar agency), you should consider the possibility that they knew of the vulnerability long before it was announced. But then client-side Heartbleed is unlikely to be your highest priority problem.

[tnj]

Updating openssl doesn't fix applications such as Apache, Nginx or postfix. You have to update libssl1.0.0 and restart them as explained in other posts.

[Rinzwind]

"External vulnerability testing scripts are becoming more and more commonplace these days. " that opens the possibility of that external site abusing my system: all they need to know it fails and hack my system before I patch it. No this is not the correct way. (and yes I host my own sites with apache and openssl).

[nwgat]

it might be a good idea to compile from source NOW, and install a newer one later on from apt, that way your more secure than without expectly on older versions of ubuntu anyhow just my two cents

[GTodorov]

Incomplete guide! Most likely it won't work!

[GTodorov]

Very accurate guide. Congrats! Works on the fly with Debian 7. Linux Machine 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 GNU/Linux

[Srikanth Jeeva]

stackoverflow.com/a/22976413/453486 this helped me to fix the openssl version.

[Rustavore]

@sarnold openssl.org doesn't seem like a random place to download the source for openssl. Canonical should make this unnecessary, but openssl.org should be the authoritative upstream to work from.

[sarnold]

@Rustavore, if you build your system from scratch, yes. But dropping in new versions of OpenSSL to an existing system with existing programs compiled against an older version is bound to create trouble. (In fact, we see broken ABIs from OpenSSL on a regular basis.) If you're going to replace your system OpenSSL then it'd be a good idea to rebuild all the software that uses it, too.

[Muhamed Huseinbašić]

I do not want to upgrade packages that I do not need to. Is there some specific command than apt-get upgrade?

[Muhamed Huseinbašić]

I do not need to update openssl package? Only libssl1.0.0?

[h3.]

@MuhamedHuseinbašić The package openssl contains command line tools. It is not used by applications that use the OpenSSL library to implement the SSL protocol (such as Apache). But you should just apply the distribution's security updates.