"unknown ca" with self-generated CA, certificates and client/server

Question

I'm writing a custom client & server that I want to communicate securely over the public Internet, therefore I want to use OpenSSL and have both ends do peer verification to ensure that my client isn't mis-directed by a MITM, and likewise that an unauthorized client isn't able to connect to the server.

This is the error received from the server during the SSL_connect / SSL_accept phase:

15620:error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:ssl\record\rec_layer_s3.c:1528:SSL alert number 48

I'm running under Windows 10, using OpenSSL 1.1.1. I'm using the following batch file to create them. I enter the ca private key passphrase by hand for obvious reasons.

openssl genrsa -out -des3 ca.key.pem 2048
openssl genrsa -out server.key.pem 2048
openssl genrsa -out client.key.pem 2048

openssl req -x509 -new -nodes -key ca.key.pem -sha256 -days 365 -out ca.cert.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=Foobar

openssl req -new -sha256 -key server.key.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=Foobar -out server.csr
openssl x509 -req -in server.csr -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out server.cert.pem -days 365 -sha256

openssl req -new -sha256 -key client.key.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=Foobar -out client.csr
openssl x509 -req -in client.csr -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out client.cert.pem -days 365 -sha256

The intent here is to create a self-signed CA, and then have that directly sign both the client and server keys.

ca.key.pem will be stored in a secure place: on an encrypted veracrypt volume.

Both client and server use the following call to enable peer verification:

    SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT, nullptr);

I'm fairly certain this is a certificate issue because the errors go away if I remove that line.

Solution 1

Answering this myself so that it can help anyone else that might arrive here looking for solutions to this problem. The answer was found in another SO question, but is worth repeating here: The Common Name for the CA cannot be the same as the Common Name for the client and server certificates.

So changing the fourth line of the batch file to this:

openssl req -x509 -new -nodes -key ca.key.pem -sha256 -days 365 -out ca.cert.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=FoobarCA

fixed the problem.

Solution 2

$ openssl req -x509 -new ... -addext basicConstraints=critical,CA:TRUE 

This essentially creates a certificate which has 2 basic contrains CA:TRUE extensions:

$ openssl x509 -in ca.cert.pem -text
    X509v3 extensions:
        ...
        X509v3 Basic Constraints: critical
            CA:TRUE
        X509v3 Basic Constraints: critical
            CA:TRUE

Trying to use the CA to verify the server certificate will not work:

$ openssl verify -CAfile ca.cert.pem server.cert.pem 
C = XX, ST = XX, L = XX, O = XX, CN = CA
error 24 at 1 depth lookup: invalid CA certificate
error server.cert.pem: verification failed

Given that this simple check does not work, the client will also not be able to validate the server certificate, resulting in an unknown ca alert:

...:tlsv1 alert unknown ca:...

When skipping the -addext it will create a self-signed certificate as documented, which already has CA:TRUE

$ openssl req -x509 -new ... 
...
$ openssl x509 -in ca.cert.pem -text
    X509v3 extensions:
        ...
        X509v3 Basic Constraints: critical
            CA:TRUE

And using this to verify the server certificate works:

$ openssl verify -CAfile ca.cert.pem server.cert.pem 
server.cert.pem: OK

This certificate should also be successfully validated by your client, thus no longer resulting in unknown ca.

Comments

[dgnuff]

I've edited the question to reflect what things look like in response to this, and the server is still generating the "unknown ca" error. So I've still got something wrong. Is it getting to the point that I should post the entire source rather than the snippets?

[dgnuff]

Found the problem, here on this SO question: stackoverflow.com/questions/19726138/…. I should perhaps have avoided replacing the parameters in the -subj option with XX, but did so because I didn't want to publish them. That said, I'd been using the same CN= value for both the CA and the two keys, which is apparently a complete no-no.

[Steffen Ullrich]

@dgnuff: it needs to find the issuer by subject or authority key identifier (which you don't have), so it must not have the same subject and issuer information unless subject and issuer are the same.

[Chavez]

Thank. God. Such an oversight on my end, but totally made this same mistake.