How to encrypt data using RSA, with SHA-256 as hash function and MGF1 as mask generating function?

Question

I was doing some experiments with cryptography. Now I have the public key of receiver and i want to encrypt some data and pass to the receiver.

I want to use RSAES-OAEP algorithm. with SHA-256 as hash function and MGF1 as mask generation function.

I want do this using openssl. I found a function RSA_public_encrypt() with this function we can specify the padding. One of the padding option available was

RSA_PKCS1_OAEP_PADDING
EME-OAEP as defined in PKCS #1 v2.0 with SHA-1 , MGF1 .

they are using sha-1.

I want to reconfigure the function to use SHA256 as hash function ans MGF1 as hash function. How can I do it ?

Solution 1

The following excerpt allows using OAEP with SHA256 for both the MGF and hash function. Tested with OpenSSL 1.0.2L

int flags = CMS_BINARY | CMS_PARTIAL | CMS_KEY_PARAM;
cms = CMS_encrypt(NULL, in, cipher, flags)
ri = CMS_add1_recipient_cert(cms, cert, flags);
pctx = CMS_RecipientInfo_get0_pkey_ctx(ri);
EVP_PKEY_CTX_set_rsa_padding(pctx, RSA_PKCS1_OAEP_PADDING);
EVP_PKEY_CTX_set_rsa_oaep_md(pctx, EVP_sha256());
EVP_PKEY_CTX_set_rsa_mgf1_md(pctx, EVP_sha256());

Solution 2

With a newer OpenSSL 1.0.2+ you can do it using the command:

openssl pkeyutl -in PlaintextKeyMaterial.bin -out EncryptedKeyMaterial.bin -inkey PublicKey.bin -keyform DER -pubin -encrypt -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256

This is taken from AWS KMS doc here: https://aws.amazon.com/es/premiumsupport/knowledge-center/invalidciphertext-kms/

Solution 3

OpenSSL uses definitions from PKCS #1 v2.0 and so the default for EME-OAEP is SHA-1 and MGF1. If you need to use SHA-256, you'll need to do the encoding yourself. This isn't terribly difficult however, see PKCS #1 v2.2 PDF Page 18 for details.

Solution 4

The PKCS#1 doc referred to above is more mathematical and doesn't give coding examples, and the CMS answer is for ASN.1/SMIME data and not really relevant for the question asked, which was how to replace RSA_public_encrypt() which deals with simple binary/text data. I spent a whole day with trial and error and online searching to find the answer to this, and eventually got the answer (which was to use OpenSSL's EVP API) from looking at the source code of "openssl pkeyutl" - once I discovered that it was not difficult.

In my case I was looking to replace RSA_private_decrypt() for decryption using the private key and this is how to do that - it should be pretty easy to put to together an RSA_public_encrypt() replacement based on this:

EVP_PKEY    *privKey = NULL;
BIO         *bioPrivKey;
int         outLen = 0, ret;

if ((bioPrivKey = BIO_new(BIO_s_mem())))
{
    // Read the private key from the RSA context into the memory BIO,
    // then convert it to an EVP_PKEY:
    if ((ret = PEM_write_bio_RSAPrivateKey(bioPrivKey, rsaCtxt, NULL, NULL, 0, NULL, NULL)) &&
        (privKey = PEM_read_bio_PrivateKey(bioPrivKey, NULL, NULL, NULL)))
    {
        EVP_PKEY_CTX *ctx = EVP_PKEY_CTX_new(privKey, NULL);

        EVP_PKEY_free(privKey);

        if (ctx)
        {
            if (EVP_PKEY_decrypt_init(ctx) > 0)
            {
                EVP_PKEY_CTX_ctrl_str(ctx, "rsa_padding_mode", "oaep");
                EVP_PKEY_CTX_ctrl_str(ctx, "rsa_oaep_md", "sha256");
                EVP_PKEY_CTX_ctrl_str(ctx, "rsa_mgf1_md", "sha256");

                outLen = dataOutMax;
                ret = EVP_PKEY_decrypt(ctx, dataOut, &outLen, dataIn, inDataLen);
                if (ret > 0 && outLen > 0 && outLen <= dataOutMax)
                {
                    // Success :-)
                }
            }
            EVP_PKEY_CTX_free(ctx);
        }
    }

    BIO_free_all(bioPrivKey);
}

You can add error handling for the failure cases using ERR_get_error().

Solution 5

In the latest version of Openssl(1.0.2k) the signature of the API is changed which gives us more flexibility. Please find the details below,

int RSA_padding_check_PKCS1_OAEP_mgf1(unsigned char *to, int tlen, const unsigned char *from, int flen, int num, const unsigned char *param, int plen, const EVP_MD *md, const EVP_MD *mgf1md)

You can pass the EVP_MD structure to invoke the SHA-256 hashing using this.

View more solutions