Windows CryptoAPI: CryptSignHash with CALG_SHA_256 and private key from MY keystore

Question

I am trying to generate digital signatures on Windows (from XP SP3, but currently testing with Windows 7) with CryptoAPI that will be compatible with the following openssl commands:

openssl dgst -sha256 -sign <parameters> (for signing)
openssl dgst -sha256 -verify <parameters> (for validation)

I want to use a private key from the Windows "MY" keystore for signing.

I managed to sign files using the SHA1 digest algorithm by using the following CryptoAPI functions (omitting parameters for brevity):

CertOpenStore
CertFindCertificateInStore
CryptAcquireCertificatePrivateKey
CryptCreateHash (with CALG_SHA1)
CryptHashData
CryptSignHash

The generated signature is compatible with "openssl dgst -sha1 -verify" (once the byte order is reversed).

My problem is: when I try to use CALG_SHA_256 with CryptCreateHash, it fails with error 80090008 (NTE_BAD_ALGID). By googling around, I found that I needed to use a specific provider (PROV_RSA_AES) instead of the default one. Since I would have a provider handle, I would also need to replace CryptAcquireCertificatePrivateKey by CryptGetUserKey. So I modified my program to look like:

CryptAcquireContext (with PROV_RSA_AES)
CertOpenStore
CertFindCertificateInStore
CryptGetUserKey
CryptCreateHash (with CALG_SHA256)
CryptHashData
CryptSignHash

Unfortunately, this didn't work as expected: CryptGetUserKey failed with error 8009000D (NTE_NO_KEY). If I remove the CryptGetUserKey call, the program runs until CryptSignHash, which fails with error 80090016 (NTE_BAD_KEYSET). I know the keyset does exist and works fine, since I was able to use it to sign the SHA1 digest.

I tried acquiring the context again with information from the certificate context I got from CertFindCertificateInStore: the best I could do was a successful CryptGetUserKey call, but CryptSignHash would always fail with the same error.

The private key I am trying to use is 2048 bits long, but I don't expect it to be a problem since it works with the SHA1 digest. I am at a loss, so any suggestion would be very welcome!

Solution 1

The problem is most likely to be that certificates on Windows "know" in which provider their private keys are stored. When you import your cert it would put the key into a certain provider type (probably PROV_RSA_FULL), when you then later try to access the key via the certificate it will probably end up in the same provider type.

You probably need to open the associated context for the certificate (have a look at CertGetCertificateContextProperty with the CERT_KEY_PROV_HANDLE_PROP_ID option). With that handle yyou could try exporting the key from the original provider context and reimporting into a new PROV_RSA_AES one (assuming the key is exportable).

Solution 2

80090008 is caused because of Base provider does not support SHA256, SHA384 and SHA512, you got to use CryptAcquireContext(hProv, NULL, MS_ENH_RSA_AES_PROV, PROV_RSA_AES, 0);

Solution 3

Most of the previous answers have parts of the real answer. The way to do this is to get a HCRYPTPROV that uses the key container and the "Microsoft Enhanced RSA and AES Cryptographic Provider" by calling

CryptAcquireContext(&hCryptProv, <keyContainerName>, MS_ENH_RSA_AES_PROV, PROV_RSA_AES, CRYPT_SILENT)

The resulting hCryptProv can be used to sign hashes created with all the supported SHA2: 256, 384, 512.

The Key Container Name can be obtained either with CertGetCertificateContextProperty() with argument CERT_KEY_PROV_INFO_PROP_ID if the key is obtained through the certificate, or by CryptGetProvParam() with argument PP_CONTAINER if there's already a HCRPYPTPROV for that key ( for example, obtained with CryptAcquireCertificatePrivateKey).

This technique works even if the private key is not exportable.

Comments

[Dominique Eav]

Thanks for your quick answer! This is very likely my problem. Unfortunately, I cannot assume a key is exportable, so I need to tackle my problem differently.

[Dominique Eav]

I'd love to try your suggestion, but I cannot find a way to pass the hash byte array directly to CryptSignHash: this function requires a hash handle (HCRYPTHASH). I guess I can use CryptCreateHash and initialize the hash using the byte array, but if the hash algorithm is not supported by the provider I won't get very far.

[Rasmus Faber]

Sorry, you are right. I don't think you can do it that way. I will write up another suggestion, you may try.

[Dominique Eav]

I had tried this, and I tried again after you updated your answer, but it doesn't seem to work in my case (still NTE_BAD_KEYSET).

[Cobaia]

@Dominique Eav Have you found the solution? "if key is not exportable" I have the same problem...

[Dominique Eav]

@Cobaia: if key is not exportable... then it is not possible, I'm afraid.

[vond]

Thanks, this worked. I believe this one shall be marked as answer!