Encryption/decryption doesn't work well between two different openssl versions

Question

I've downloaded and compiled openssl-1.1.0.

I can encrypt and decrypt using the same exe of openssl (as is here)

me@ubuntu:~/openssl-1.1.0$ LD_LIBRARY_PATH=. ./apps/openssl aes-256-cbc -a -salt -in file.txt -out file.txt.enc
enter aes-256-cbc encryption password: 123
Verifying - enter aes-256-cbc encryption password:
me@ubuntu:~/openssl-1.1.0$ LD_LIBRARY_PATH=. apps/openssl aes-256-cbc -a -d -in file.txt.enc -out file.txt.dec
enter aes-256-cbc decryption password: 123

This openssl uses: libcrypto.so.1.1, libssl.so.1.1

When I try to decrypt with the openssl installed on my ubuntu, which uses: /lib/x86_64-linux-gnu/libssl.so.1.0.0, /lib/x86_64-linux-gnu/libcrypto.so.1.0.0

I get an error:

me@ubuntu:~/openssl-1.1.0$ openssl aes-256-cbc -a -d -in file.txt.enc -out file.txt.dec2
enter aes-256-cbc decryption password: 123
bad decrypt
140456117421728:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:539:

What may cause this? Thanks

Solution 1

The default digest was changed from MD5 to SHA256 in Openssl 1.1

Try using -md md5

cgs@ubuntu:~$ echo "it-works!" > file.txt
cgs@ubuntu:~$ LD_LIBRARY_PATH=~/openssl-1.1.0/ openssl-1.1.0/apps/openssl aes-256-cbc -a -salt -in ~/file.txt -out ~/file.txt.enc -md md5
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
cgs@ubuntu:~$ LD_LIBRARY_PATH=~/openssl-1.0.1f/ openssl-1.0.1f/apps/openssl aes-256-cbc -a -in ~/file.txt.enc -d
enter aes-256-cbc decryption password:
it-works!

The ugly details:

The entered password is not used as is by aes (or other encryption) but the command implicitly derives a key from it. The key derivation uses message digest that was changed in openssl 1.1 Use SHA256 not MD5 as default digest.

In case you want to keep it simple password, and not start messing with the keying martial (-K,-iv) just force the same digest with -md

Solution 2

I tested the AES encryption and decryption with version 1.1.0a (downloaded from openssl.org) and the version 1.0.2g-fips (from my ubuntu 16.04)

When using the -p option on with 2 different versions of openssl, the IV and key are different:

$ LD_LIBRARY_PATH=~/openssl-1.1.0a/ ~/openssl-1.1.0a/apps/openssl aes-256-cbc -a -p -salt -in file -out file.enc
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
salt=6A80B2A3B4CFE048
key=637E17094DF7892A7AFC14957EAA13991DFFD3273A2459EDA613F3AD8A406C38
iv =6AC7CE5C9AADC6C46C633BF5124DAFBF

$ openssl aes-256-cbc -a -d -p -in file.enc -out file.dec
enter aes-256-cbc decryption password:
salt=6A80B2A3B4CFE048
key=6220AF2E25CB0B5D9994A0A1B05503D82AC5B0B4C9015E241CACBF8BF62DAC77
iv =2DC04EF29AA57478EBE606DF87277EA6
bad decrypt
140557073118872:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:592:

I suspect a different derivation of key and IV based on the salt with the 2 versions.

If you want to get rid of this decryption error, you may remove the -salt option and use the options -K for the key and -iv in your openssl command.

Solution 3

There are various error strings that are thrown from openssl, depending on respective versions, and scenarios. Below is the checklist I use in case of openssl related issues:

1. Ideally, openssl is able to encrypt/decrypt using same key (+ salt) & enc algo only.

2. Ensure that openssl versions (used to encrypt/decrypt), are compatible. For eg. the hash used in openssl changed at version 1.1.0 from MD5 to SHA256. This produces a different key from the same password. Fix: add "-md md5" in 1.1.0 to decrypt data from lower versions, and add "-md sha256 in lower versions to decrypt data from 1.1.0

3. Ensure that there is a single openssl version installed in your machine. In case there are multiple versions installed simultaneously (in my machine, these were installed :- 'LibreSSL 2.6.5' and 'openssl 1.1.1d'), make the sure that only the desired one appears in your PATH variable.

Solution 4

This issue can also occur between OpenSSL 1.1 and LibreSSL. In this case, and in other cases where more secure message digests are available, you should avoid using -md md5 to encrypt new files since the MD5 algorithm has extensive vulnerabilities.

You should instead use -md sha256 or some other more secure message digest supported by all versions. -md md5 should only be used for decrypting old files, and they should ideally be re-encrypted using sha256. This is also mentioned in the OpenSSL FAQ:

A message digest is used to create the encrypt/decrypt key from a human-entered passphrase. In OpenSSL 1.1.0 we changed from MD5 to SHA-256. We did this as part of an overall change to move away from the now-insecure and broken MD5 algorithm. If you have old files, use the "-md md5" flag to decrypt them.

---

To check which message digests are supported by the different versions you have in play, run openssl help:

LibreSSL 2.2.7 (included with macOS 10.13 High Sierra):

$ openssl help
…
Message Digest commands (see the `dgst' command for more details)
gost-mac          md4               md5               md_gost94
ripemd160         sha               sha1              sha224
sha256            sha384            sha512            streebog256
streebog512       whirlpool
…

OpenSSL 1.1f:

$ openssl help
…
Message Digest commands (see the `dgst' command for more details)
blake2b512        blake2s256        gost              md4
md5               rmd160            sha1              sha224
sha256            sha384            sha512
…

View more solutions

Comments

[starfry]

Spent the best part of a day thinking our code had broken. Stumbled on your answer. Saved the day!

[Adnan]

how to force md5 in C++ program?

[Scott Stensland]

Yes specifying the digest -md md5 works however better solution is to re-encrypt using -md sha256 (and same for decrypt) which is more secure than md5 hence the change in the default digest on new versions of openssl

[Aaron Brager]

Don’t use md5. It's insecure and broken.

[Broper]

+1, and yes, the newer default is obviously better (md5 suffers from collision issues as well), but in my case I needed to use it to decrypt older files that were encrypted using the older standard

[anthony]

This is why I am now using a wrapper around openssl to save the options used to encrypt files. That way the file can be decrypted even when the normal options used for encrypting changes. Basically it saves the encryption metadata needed to decrypt, Especially the new v1.1.1 -pbkdf2 iteration count that should increase with time. It basically future proofs the encryption. See "keepout" antofthy.gitlab.io/software/#keepout