Client and Server communication using ssl c/c++ - SSL protocol don't works

c++ c sockets openssl ssl-certificate

96,582

Solution 1

With the above server and client programs, I was getting the following error:

140671281543104:error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned:s3_srvr.c:3292:

I had generated self-signed certificates using the procedure mentioned in https://help.ubuntu.com/community/OpenSSL.

After juggling with the error for one day, i found that the error was because the self-generated CA was not in the trust chain of the machine I was using.

To add the CA to the trust chain in RHEL-7, one can follow the below procedure:

        To add a certificate in the simple PEM or DER file formats to the
        list of CAs trusted on the system:

        Copy it to the
                /etc/pki/ca-trust/source/anchors/
        subdirectory, and run the
                update-ca-trust
        command.

        If your certificate is in the extended BEGIN TRUSTED file format,
        then place it into the main source/ directory instead.

I think the above procedure can be followed for fedora too. If this does not work, it might be useful to explore the commands like "update-ca-certificates". Hope this will be useful to someone.

Solution 2

you must to modify your code (server side): your code:

int main()
{   SSL_CTX *ctx;
    int server;
    **char portnum[]="5000";**

        char CertFile[] = "/home/myCA/mycert.pem";
        char KeyFile[] = "/home/myCA/mycert.pem";

    SSL_library_init();

    **portnum = strings[1];**

instead you should use this:

int main(int argc, char **argv)
{   SSL_CTX *ctx;
    int server;
    //char portnum[]="5000"; ---> You can pass it as an argument

        char CertFile[] = "/home/myCA/mycert.pem";
        char KeyFile[] = "/home/myCA/mycert.pem";

    SSL_library_init();

    //portnum = strings[1]; 
    portnum = argv[1];  // ---> You can pass port number here, instead of put it in the code

I got this output (client):

toc@UnixServer:~$ ./ssl_client 
Connected with AES256-SHA encryption
Server certificates:
Subject: /C=FR/ST=Some-State/L=PARIS/O=TOC/OU=TOC/CN=TOC/[email protected]
Issuer: /C=FR/ST=Some-State/L=PARIS/O=TOC/OU=TOC/CN=TOC/[email protected]
Received: "<html><body><pre>Hello???</pre></body></html>

"

And this output (server):

Connection: 127.0.0.1:59066
Server certificates:
Subject: /C=FR/ST=Some-State/L=PARIS/O=TOC/OU=TOC/CN=TOC/[email protected]
Issuer: /C=FR/ST=Some-State/L=PARIS/O=TOC/OU=TOC/CN=TOC/[email protected]
Client msg: "Hello???"

When you use a tool like ssldump (http://www.rtfm.com/ssldump/) (on unix box), you can see clearly what's going on:

toc@UnixServer:~$sudo ssldump -i lo port 5000
New TCP connection #1: localhost(59071) <-> localhost(5000)
1 1  0.0012 (0.0012)  C>S  Handshake
      ClientHello
        Version 3.0 
        cipher suites
        Unknown value 0xc014
        Unknown value 0xc00a
        SSL_DHE_RSA_WITH_AES_256_CBC_SHA
        SSL_DHE_DSS_WITH_AES_256_CBC_SHA
        Unknown value 0x88
        Unknown value 0x87
        Unknown value 0xc00f
        Unknown value 0xc005
        SSL_RSA_WITH_AES_256_CBC_SHA
        Unknown value 0x84
        Unknown value 0xc012
        Unknown value 0xc008
        SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        Unknown value 0xc00d
        Unknown value 0xc003
        SSL_RSA_WITH_3DES_EDE_CBC_SHA
        Unknown value 0xc013
        Unknown value 0xc009
        SSL_DHE_RSA_WITH_AES_128_CBC_SHA
        SSL_DHE_DSS_WITH_AES_128_CBC_SHA
        Unknown value 0x9a
        Unknown value 0x99
        Unknown value 0x45
        Unknown value 0x44
        Unknown value 0xc00e
        Unknown value 0xc004
        SSL_RSA_WITH_AES_128_CBC_SHA
        Unknown value 0x96
        Unknown value 0x41
        Unknown value 0xc011
        Unknown value 0xc007
        Unknown value 0xc00c
        Unknown value 0xc002
        SSL_RSA_WITH_RC4_128_SHA
        SSL_RSA_WITH_RC4_128_MD5
        SSL_DHE_RSA_WITH_DES_CBC_SHA
        SSL_DHE_DSS_WITH_DES_CBC_SHA
        SSL_RSA_WITH_DES_CBC_SHA
        SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
        SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
        SSL_RSA_EXPORT_WITH_DES40_CBC_SHA
        SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5
        SSL_RSA_EXPORT_WITH_RC4_40_MD5
        Unknown value 0xff
        compression methods
                unknown value
                  NULL
1 2  0.0019 (0.0006)  S>C  Handshake
      ServerHello
        Version 3.0 
        session_id[32]=
          13 e2 5a f0 10 93 18 56 c8 66 54 94 29 ab 8b 2d 
          7b c6 9c 3b 7b ea c7 54 e6 86 7d 3a 56 8c 96 14 
        cipherSuite         SSL_RSA_WITH_AES_256_CBC_SHA
        compressionMethod                 unknown value
1 3  0.0019 (0.0000)  S>C  Handshake
      Certificate
1 4  0.0019 (0.0000)  S>C  Handshake
      CertificateRequest
        certificate_types                   rsa_sign
        certificate_types                   dss_sign
      ServerHelloDone
1 5  0.0155 (0.0136)  C>S  Handshake
      Certificate
1 6  0.0155 (0.0000)  C>S  Handshake
      ClientKeyExchange
1 7  0.0155 (0.0000)  C>S  Handshake
      CertificateVerify
        Signature[128]=
          ac 94 31 89 64 75 20 5f 4f 00 73 4e e8 de 51 b7 
          f1 bb 16 da 63 b1 8d e9 15 9b af f8 32 d7 84 f5 
          b5 7d 4f 48 1c 2b 41 58 81 d3 a8 50 40 25 90 95 
          44 de 9d bb c4 79 5c 64 a8 a9 28 f4 16 7c 0e 17 
          b2 77 cf b0 8c a9 90 50 34 a5 76 a2 57 39 8d 37 
          12 d8 a5 8d f4 08 3a 1e 83 7e 6c 0a e9 75 ec 85 
          3d 56 f2 2e 4a 7d 71 88 29 26 99 40 43 4e f3 29 
          26 bf eb 15 be 36 22 72 f3 d9 be 4a e3 c9 0b cc 
1 8  0.0155 (0.0000)  C>S  ChangeCipherSpec
1 9  0.0155 (0.0000)  C>S  Handshake
1 10 0.0245 (0.0089)  S>C  ChangeCipherSpec
1 11 0.0245 (0.0000)  S>C  Handshake
1 12 0.0250 (0.0005)  C>S  application_data
1 13 0.0250 (0.0000)  C>S  application_data
1 14 0.0258 (0.0007)  S>C  application_data
1 15 0.0258 (0.0000)  S>C  application_data
1    0.0261 (0.0002)  C>S  TCP FIN
1    0.0275 (0.0013)  S>C  TCP FIN

Regards.

Solution 3

As I was saying in comments to one of your previous question, the fact that you get "Malformed Packet: GSM over IP" or something odd here is normal.

You're using port 5000, which is normally reserved for protocol commplex-main. Therefore, without any extra information, Wireshark tries to analyse the traffic it sees with the commplex-main decoders.

Of course, since the data you're exchanging on that port is in fact SSL/TLS (because you're using a port that's not normally used for that), decoding it as if it was commplex-main leads to a number of odd messages regarding malformed packets.

Wireshark only guesses the protocol using the port number it sees. You have to tell it to try a different decoder, if you're not using the standard port for that protocol.

More specifically, right click on a packet and choose Decode As... -> Transport -> SSL.

96,582

Author by

David Viana

Updated on July 09, 2022

Comments

David Viana almost 2 years

I'm trying use an SSL client/server example from:http://simplestcodings.blogspot.com.br/2010/08/secure-server-client-using-openssl-in-c.html to create a secure connection using SSLv3. I made some changes for request the certificate on server-side, the comunication works fine and is understood in both sides. Thus, my problem is that when the client connects to server, the protocol communication SSLv3 is not working, I verified using the wirkeshark and in the protocol field just shows TCP, or IPA(RSL Malformed Packet) somebody can help me? Thanks!

I've created my certificates following the tutorial https://help.ubuntu.com/community/OpenSSL.

Here is my client code:

//SSL-Client.c
#include <stdio.h>
#include <errno.h>
#include <unistd.h>
#include <malloc.h>
#include <string.h>
#include <sys/socket.h>
#include <resolv.h>
#include <netdb.h>
#include <openssl/ssl.h>
#include <openssl/err.h>

#define FAIL    -1

    //Added the LoadCertificates how in the server-side makes.    
void LoadCertificates(SSL_CTX* ctx, char* CertFile, char* KeyFile)
{
 /* set the local certificate from CertFile */
    if ( SSL_CTX_use_certificate_file(ctx, CertFile, SSL_FILETYPE_PEM) <= 0 )
    {
        ERR_print_errors_fp(stderr);
        abort();
    }
    /* set the private key from KeyFile (may be the same as CertFile) */
    if ( SSL_CTX_use_PrivateKey_file(ctx, KeyFile, SSL_FILETYPE_PEM) <= 0 )
    {
        ERR_print_errors_fp(stderr);
        abort();
    }
    /* verify private key */
    if ( !SSL_CTX_check_private_key(ctx) )
    {
        fprintf(stderr, "Private key does not match the public certificate\n");
        abort();
    }
}

int OpenConnection(const char *hostname, int port)
{   int sd;
    struct hostent *host;
    struct sockaddr_in addr;

    if ( (host = gethostbyname(hostname)) == NULL )
    {
        perror(hostname);
        abort();
    }
    sd = socket(PF_INET, SOCK_STREAM, 0);
    bzero(&addr, sizeof(addr));
    addr.sin_family = AF_INET;
    addr.sin_port = htons(port);
    addr.sin_addr.s_addr = *(long*)(host->h_addr);
    if ( connect(sd, (struct sockaddr*)&addr, sizeof(addr)) != 0 )
    {
        close(sd);
        perror(hostname);
        abort();
    }
    return sd;
}

SSL_CTX* InitCTX(void)
{   SSL_METHOD *method;
    SSL_CTX *ctx;

    OpenSSL_add_all_algorithms();  /* Load cryptos, et.al. */
    SSL_load_error_strings();   /* Bring in and register error messages */
    method = SSLv3_client_method();  /* Create new client-method instance */
    ctx = SSL_CTX_new(method);   /* Create new context */
    if ( ctx == NULL )
    {
        ERR_print_errors_fp(stderr);
        abort();
    }
    return ctx;
}

void ShowCerts(SSL* ssl)
{   X509 *cert;
    char *line;

    cert = SSL_get_peer_certificate(ssl); /* get the server's certificate */
    if ( cert != NULL )
    {
        printf("Server certificates:\n");
        line = X509_NAME_oneline(X509_get_subject_name(cert), 0, 0);
        printf("Subject: %s\n", line);
        free(line);       /* free the malloc'ed string */
        line = X509_NAME_oneline(X509_get_issuer_name(cert), 0, 0);
        printf("Issuer: %s\n", line);
        free(line);       /* free the malloc'ed string */
        X509_free(cert);     /* free the malloc'ed certificate copy */
    }
    else
        printf("No certificates.\n");
}

int main()
{   SSL_CTX *ctx;
    int server;
    SSL *ssl;
    char buf[1024];
    int bytes;
    char hostname[]="127.0.0.1";
    char portnum[]="5000";
    char CertFile[] = "/home/myCA/cacert.pem";
    char KeyFile[] = "/home/myCA/private/cakey.pem";

    SSL_library_init();

    ctx = InitCTX();
    LoadCertificates(ctx, CertFile, KeyFile);
    server = OpenConnection(hostname, atoi(portnum));
    ssl = SSL_new(ctx);      /* create new SSL connection state */
    SSL_set_fd(ssl, server);    /* attach the socket descriptor */
    if ( SSL_connect(ssl) == FAIL )   /* perform the connection */
        ERR_print_errors_fp(stderr);
    else
    {   char *msg = "Hello???";

        printf("Connected with %s encryption\n", SSL_get_cipher(ssl));
        ShowCerts(ssl);        /* get any certs */
        SSL_write(ssl, msg, strlen(msg));   /* encrypt & send message */
        bytes = SSL_read(ssl, buf, sizeof(buf)); /* get reply & decrypt */
        buf[bytes] = 0;
        printf("Received: \"%s\"\n", buf);
        SSL_free(ssl);        /* release connection state */
    }
    close(server);         /* close socket */
    SSL_CTX_free(ctx);        /* release context */
    return 0;
}

And the server:

//SSL-Server.c
#include <errno.h>
#include <unistd.h>
#include <malloc.h>
#include <string.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <netinet/in.h>
#include <resolv.h>
#include "openssl/ssl.h"
#include "openssl/err.h"

#define FAIL    -1

int OpenListener(int port)
{   int sd;
    struct sockaddr_in addr;

    sd = socket(PF_INET, SOCK_STREAM, 0);
    bzero(&addr, sizeof(addr));
    addr.sin_family = AF_INET;
    addr.sin_port = htons(port);
    addr.sin_addr.s_addr = INADDR_ANY;
    if ( bind(sd, (struct sockaddr*)&addr, sizeof(addr)) != 0 )
    {
        perror("can't bind port");
        abort();
    }
    if ( listen(sd, 10) != 0 )
    {
        perror("Can't configure listening port");
        abort();
    }
    return sd;
}

SSL_CTX* InitServerCTX(void)
{   SSL_METHOD *method;
    SSL_CTX *ctx;

    OpenSSL_add_all_algorithms();  /* load & register all cryptos, etc. */
    SSL_load_error_strings();   /* load all error messages */
    method = SSLv3_server_method();  /* create new server-method instance */
    ctx = SSL_CTX_new(method);   /* create new context from method */
    if ( ctx == NULL )
    {
        ERR_print_errors_fp(stderr);
        abort();
    }
    return ctx;
}

void LoadCertificates(SSL_CTX* ctx, char* CertFile, char* KeyFile)
{
    //New lines 
    if (SSL_CTX_load_verify_locations(ctx, CertFile, KeyFile) != 1)
        ERR_print_errors_fp(stderr);

    if (SSL_CTX_set_default_verify_paths(ctx) != 1)
        ERR_print_errors_fp(stderr);
    //End new lines

    /* set the local certificate from CertFile */
    if (SSL_CTX_use_certificate_file(ctx, CertFile, SSL_FILETYPE_PEM) <= 0)
    {
        ERR_print_errors_fp(stderr);
        abort();
    }
    /* set the private key from KeyFile (may be the same as CertFile) */
    if (SSL_CTX_use_PrivateKey_file(ctx, KeyFile, SSL_FILETYPE_PEM) <= 0)
    {
        ERR_print_errors_fp(stderr);
        abort();
    }
    /* verify private key */
    if (!SSL_CTX_check_private_key(ctx))
    {
        fprintf(stderr, "Private key does not match the public certificate\n");
        abort();
    }

    //New lines - Force the client-side have a certificate
    SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT, NULL);
    SSL_CTX_set_verify_depth(ctx, 4);
    //End new lines
}

void ShowCerts(SSL* ssl)
{   X509 *cert;
    char *line;

    cert = SSL_get_peer_certificate(ssl); /* Get certificates (if available) */
    if ( cert != NULL )
    {
        printf("Server certificates:\n");
        line = X509_NAME_oneline(X509_get_subject_name(cert), 0, 0);
        printf("Subject: %s\n", line);
        free(line);
        line = X509_NAME_oneline(X509_get_issuer_name(cert), 0, 0);
        printf("Issuer: %s\n", line);
        free(line);
        X509_free(cert);
    }
    else
        printf("No certificates.\n");
}

void Servlet(SSL* ssl) /* Serve the connection -- threadable */
{   char buf[1024];
    char reply[1024];
    int sd, bytes;
    const char* HTMLecho="<html><body><pre>%s</pre></body></html>\n\n";

    if ( SSL_accept(ssl) == FAIL )     /* do SSL-protocol accept */
        ERR_print_errors_fp(stderr);
    else
    {
        ShowCerts(ssl);        /* get any certificates */
        bytes = SSL_read(ssl, buf, sizeof(buf)); /* get request */
        if ( bytes > 0 )
        {
            buf[bytes] = 0;
            printf("Client msg: \"%s\"\n", buf);
            sprintf(reply, HTMLecho, buf);   /* construct reply */
            SSL_write(ssl, reply, strlen(reply)); /* send reply */
        }
        else
            ERR_print_errors_fp(stderr);
    }
    sd = SSL_get_fd(ssl);       /* get socket connection */
    SSL_free(ssl);         /* release SSL state */
    close(sd);          /* close connection */
}

int main()
{   SSL_CTX *ctx;
    int server;
    char portnum[]="5000";

        char CertFile[] = "/home/myCA/mycert.pem";
        char KeyFile[] = "/home/myCA/mycert.pem";

    SSL_library_init();

    ctx = InitServerCTX();        /* initialize SSL */
    LoadCertificates(ctx, CertFile, KeyFile); /* load certs */
    server = OpenListener(atoi(portnum));    /* create server socket */
    while (1)
    {   struct sockaddr_in addr;
        socklen_t len = sizeof(addr);
        SSL *ssl;

        int client = accept(server, (struct sockaddr*)&addr, &len);  /* accept connection as usual */
        printf("Connection: %s:%d\n",inet_ntoa(addr.sin_addr), ntohs(addr.sin_port));
        ssl = SSL_new(ctx);              /* get new SSL state with context */
        SSL_set_fd(ssl, client);      /* set connection socket to SSL state */
        Servlet(ssl);         /* service connection */
    }
    close(server);          /* close server socket */
    SSL_CTX_free(ctx);         /* release context */
}

David Viana almost 12 years

Sorry but the test is fixing the port 5000. I need comment the line when I change here forget the comment "//portnum = strings[1];" - Thank you, but the error is not about the port.
TOC almost 12 years

@DavidViana : Put your error here please, cause i tested the code and there's no pb!
David Viana almost 12 years

Yes.. the outputs are ok.. but the protocol, when a connection SSL/TLS is established, needs be a SSLvX, but when I verify the protocol, the filed protocol is TCP or IPA(RSL Malformed Packet). The message is encrypted but my doubt is about the protocol field don't be SSL. Thank you!
David Viana almost 12 years

You need use WireShark to see the protocol field. The error is in protocol field shows TCP or IPA(RSL Malformed Packet). I don't know why don't shows SSLv3.
TOC almost 12 years

@DavidViana : I added some traces again all seems to be ok, if you don't see the SSL traffic with wireshark, may be you have missed some configuration settings, look at this url (wiki.wireshark.org/SSL) for more details
David Viana almost 12 years

I see traffic SSL but not with this example client-server SSL. Any other application that use SSL I see all SSL traffic. Thanks!
TOC almost 12 years

@DavidViana : can you put please the output of wireshark, and edit your post to show the problem, cause i don't understand what's your problem!
David Viana over 11 years

I'm using the port 5555 or 7000, and the message "Malformed Packet: GSM over IP" is not showing. But the info protocol is TCP and not SSL or TLS. The information is crypt and I security, but I don't know why using the SSL/TLS protocol on socket the protocol info is not SSL/TCP. I apply the decode "Decode As... -> Transport -> SSL" and I don't have changes. But is ok, thanks!