Calling a REST web service over SSL

Question

I'm struggling to connect to a REST web service that's working only over HTTPS / SSL from my .NET application.

I received the certificate and private key to use as two separate files - a certificate.pem file which contains the certificate, and the webservice.key file which contains the private key. Those are both text files with BASE64 encoded binary data contained in them.

The provider also sent me a PDF showing how to call that web service using CURL and those two files, and that works just fine:

curl.exe -k -v "https://(URL)" --cert certificate.pem --key webservice.key

I need to use the -k option since there seems to be a self-signed certificate somewhere in the hierarchy of certs. Without this option, the call fails.

In order to call this web service from a .NET application (a console app for now), I used OpenSSL (on Windows) to combine these two files into a *.pfx file using this command:

openssl pkcs12 -export -out webservice.pfx -in certificate.pem -inkey webservice.key 

This seems to have worked, too - no errors were reported, the file was created and is about 3K in size and it's a totally binary file.

Now, I tried to call that web service from my .NET code something like this:

try
{
    // use the SSL protocol (instead of TLS)
    ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;

    // ignore any certificate complaints
    ServicePointManager.ServerCertificateValidationCallback += (sender, certificate, chain, sslPolicyErrors) => { return true; };

    // create HTTP web request with proper content type
    HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
    request.ContentType = "application/xml;charset=UTF8";

    // grab the PFX as a X.509 certificate from disk
    string certFileName = Path.Combine(certPath, "webservice.pfx");

    // load the X.509 certificate and add to the web request
    X509Certificate cert = new X509Certificate(certFileName, "(top-secret password)");
    request.ClientCertificates.Add(cert);
    request.PreAuthenticate = true;

    // call the web service and get response
    WebResponse response = request.GetResponse();

    Stream responseStream = response.GetResponseStream();
}
catch (Exception exc)
{
    // log and print out error
}

However, I can try whatever I like (fiddling around with various settings, on the ServicePointManager and the HttpWebRequest, but I just keep getting these errors:

WebException: The underlying connection was closed: An unexpected error occurred on a send.

IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

SocketException: An existing connection was forcibly closed by the remote host

and no response - even though communicating with the service with CURL has worked just fine.....

What am I missing?? I'm a bit puzzled and mystified by all those certificates, private keys, service point manager options and so on - just waaaaay too many knob and switches to turn, set or turn off - what are the RIGHT settings here??

Update:

If I use

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;

then the error just simply is:

WebException: The request was aborted: Could not create SSL/TLS secure channel.

S O L U T I O N :

In the end, with looking at the output from curl and a lot of help from @Alexandru and @JurajMajer, I was able to get this to work with this code:

try
{
    // use the TLS protocol 
    ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;

    // create HTTP web request with proper content type
    HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
    request.ContentType = "application/xml;charset=UTF8";

    // grab the PFX as a X.509 certificate from disk
    string certFileName = Path.Combine(certPath, "webservice.pfx");

    // load the X.509 certificate and add to the web request
    X509Certificate2 cert = new X509Certificate2(certFileName, "(top-secret password)");
    request.ClientCertificates.Add(cert);
    request.PreAuthenticate = true;

    // call the web service and get response
    WebResponse response = request.GetResponse();

    Stream responseStream = response.GetResponseStream();

    string xmlContents = new StreamReader(responseStream).ReadToEnd();
}
catch (Exception exc)
{
    // log and print out error
}

Solution 1

You've used the X509Certificate(String, String) constructor with a PKCS#12 certificate, but that constructor only works for PKCS#7 certificates, as MSDN says it...

Initializes a new instance of the X509Certificate class using the name of a PKCS7 signed file and a password to access the certificate.

PKCS#7 does not include the private (key) part of a certificate/private-key pair, which you will need. This means you will need to use your PKCS#12 certificate given the nature of your certificate.

You may want to try the X509Certificate2(String, String) constructor with your existing PKCS#12 certificate, as this constructor is used with PKCS#12 (PFX) files that contain the certificate's private key, as MSDN says...

This constructor creates a new X509Certificate2 object using a certificate file name and a password needed to access the certificate. It is used with PKCS12 (PFX) files that contain the certificate's private key. Calling this constructor with the correct password decrypts the private key and saves it to a key container.

Solution 2

Try to enable Network Tracing in App.config on the client - instructions here. That should create network.log with more debug info. In my test environment I have one pfx which works and one which doesn't.

network.log for working pfx:

SecureChannel#9343812 - We have user-provided certificates. The server has specified 34 issuer(s). Looking for certificates that match any of the issuers. SecureChannel#9343812 - Left with 1 client certificates to choose from. SecureChannel#9343812 - Trying to find a matching certificate in the certificate store. SecureChannel#9343812 - Locating the private key for the certificate: SecureChannel#9343812 - Certificate is of type X509Certificate2 and contains the private key.

network log for non-working pfx:

SecureChannel#26756241 - We have user-provided certificates. The server has specified 34 issuer(s). Looking for certificates that match any of the issuers. SecureChannel#26756241 - Left with 0 client certificates to choose from.

So for me the problem is my non-working certificate was issued by CA not in list.

Interesting points (possible problems):

1.) Server sends the list of known issuers for client certificate.

2.) Client code is looking for certificate and private key in certificate store event though both are in pfx.

Comments

[marc_s]

Thanks - I'll try that and get back to you.

[marc_s]

OK, the result is: when trying to provide the certificate from a file on disk (without installing it in the cert store), that doesn't seem to work at all. The network.log shows that the certificate is searched for in the cert store (and not found) and therefore the whole thing doesn't work. Once I've installed the PFX certificate into the cert store, that step works - the cert is found in the "current user" store, sent, but then after that, the connection is still aborted....

[Juraj Majer]

Interesting. You get this error: WebException: The request was aborted: Could not create SSL/TLS secure channel.? Any inner exception? Anything suspicious in network log, mainly SecureChannel entries? Do you use the lambda expression for ServerCertificateValidationCallback or you successfully validate server certificate any other way? Maybe you can sniff SSL handshake with WireShark for faulty WCF case and working cURL case and compare both.

[marc_s]

Thanks, Alexandru. Unfortunately, even with using the X509Certificate2, I still have the same errors.....

[Alexandru]

Perhaps there is a way to set up a similar environment as your customer and trace to see what happens on the server side of things where the connection is being closed, because the client logs are not very meaningful.

[marc_s]

Good day today - with a bit of looking at the curl output, I was able to finally figure out how to get this to work. I needed to set the SecurityProtocol to TLS, Http Version to 1.1, and use the X509Certificate2 constructor as you had mentioned here. Taking all this together, it now actually W O R K S ! :-) :-) :-) Thanks for all your comments and all your help ! Highly appreciated !

[Alexandru]

It brings me great joy to hear that. I've learned a few new things from our journey together. :)