How to list all openssl ciphers available in statically linked python releases?

Question

In the python 2.7.8 to 2.7.9 upgrade, the ssl module changed from using

_DEFAULT_CIPHERS = 'DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2'

to

_DEFAULT_CIPHERS = (
    'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
    'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:ECDH+RC4:'
    'DH+RC4:RSA+RC4:!aNULL:!eNULL:!MD5'
)

I'd like to know how this affects the actual "ordered SSL cipher preference list" that gets used when establishing SSL/TLS connections with my python installs on Windows.

For example, to figure out what "ordered SSL cipher preference list" a cipher list expands to, I'd normally use the openssl ciphers command line (see man page) e.g with openssl v1.0.1k I can see what that default python 2.7.8 cipher list expands to:

$ openssl ciphers -v 'DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2'
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA384
ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
ECDHE-RSA-AES256-SHA    SSLv3 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA1
ECDHE-ECDSA-AES256-SHA  SSLv3 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA1
SRP-DSS-AES-256-CBC-SHA SSLv3 Kx=SRP      Au=DSS  Enc=AES(256)  Mac=SHA1
SRP-RSA-AES-256-CBC-SHA SSLv3 Kx=SRP      Au=RSA  Enc=AES(256)  Mac=SHA1
...
snip!

That works great when on Linux where python is dynamically loading the same OpenSSL library that openssl ciphers uses:

$ ldd /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so | grep libssl
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007ff75d6bf000)
$ ldd /usr/bin/openssl | grep libssl
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fa48f0fe000)

However, on Windows the Python build appears to statically link the OpenSSL library. This means that the openssl ciphers command cannot help me, because it uses a different version of the library, which may have support for different ciphers than the library built into python.

I can find out what version of OpenSSL was used to build each of the two python releases easily enough:

$ python-2.7.8/python -c 'import ssl; print ssl.OPENSSL_VERSION'
OpenSSL 1.0.1h 5 Jun 2014

$ python-2.7.9/python -c 'import ssl; print ssl.OPENSSL_VERSION'
OpenSSL 1.0.1j 15 Oct 2014

But even if I could find and download a build of the openssl command line for both the 1.0.1h and 1.0.1j releases, I cannot be sure that they were compiled with the same options as the lib built into python, and from the man page we know that

Some compiled versions of OpenSSL may not include all the ciphers listed here because some ciphers were excluded at compile time.

So, is there a way to get python's ssl module to give me output similar to that from the openssl ciphers -v command?

Solution 1

You might want to have a look into openssl cipher's source code at https://github.com/openssl/openssl/blob/master/apps/ciphers.c

The crucial steps seem to be:

1. meth = SSLv23_server_method();
2. ctx = SSL_CTX_new(meth);
3. SSL_CTX_set_cipher_list(ctx, ciphers), whereas ciphers is your string
4. ssl = SSL_new(ctx);
5. sk = SSL_get1_supported_ciphers(ssl);
6. for (i = 0; i < sk_SSL_CIPHER_num(sk); i++) { print SSL_CIPHER_get_name(sk_SSL_CIPHER_value(sk, i)); }

The SSL_CTX_set_cipher_list function is called in Python 3.4 in _ssl's set_ciphers method for contexts. You can achieve the same using:

import socket
from ssl import SSLSocket
sslsock = SSLSocket(socket.socket(socket.AF_INET, socket.SOCK_STREAM))
sslsock.context.set_ciphers('DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2')

The next step would be calling SSL_get1_supported_ciphers() which, unfortunately, is not used in Python's _ssl.c. The closest you can get is the shared_ciphers() method of SSLSocket instances. The (current) implementation is

static PyObject *PySSL_shared_ciphers(PySSLSocket *self)
{
    [...]
    ciphers = sess->ciphers;
    res = PyList_New(sk_SSL_CIPHER_num(ciphers));
    for (i = 0; i < sk_SSL_CIPHER_num(ciphers); i++) {
        PyObject *tup = cipher_to_tuple(sk_SSL_CIPHER_value(ciphers, i));
        [...]
        PyList_SET_ITEM(res, i, tup);
    }
    return res;
}

That is, this loop is very similar as in the ciphers.c implementation above, and returns a Python list of ciphers, in the same order as the loop in ciphers.c would.

Continuing with the sslsock = SSLSocket(...) example from above, you cannot call sslsock.shared_ciphers() before the socket is connected. Otherwise, Python's _ssl module does not create a low-level OpenSSL SSL object, which is needed to read the ciphers. That is different from the implementation in ciphers.c, which creates a low level SSL object without requiring a connection.

That is how far I got, I hope that helps, and maybe you can figure out what you need based on these findings.

Solution 2

Jan-Philip Gehrcke's answer requires an as-yet-unreleased version of python to be useful (see the comments), that make it not practical for answering the question about older versions of python. But this paragraph inspired me:

...you cannot call sslsock.shared_ciphers() before the socket is connected. Otherwise, Python's _ssl module does not create a low-level OpenSSL SSL object, which is needed to read the ciphers.

This got me thinking about a possible solution. All in the same python program:

• Create a server socket that accepts any cipher (ciphers='ALL:aNULL:eNULL').
• Connect to the server socket with a client socket configured with the cipher list we want to check (say 'DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2' if we want to test the default from python 2.7.8)
• Once the connection is established, examine the cipher that actually got chosen by the client and print it e.g. 'AES256-GCM-SHA384'. The client will choose the highest priority cipher from its configured cipher list that matches one supplied by the server. The server accepts any cipher and is running in the same python program with the same OpenSSL lib so the server's list is guaranteed to be a superset of the client's list. So the cipher used must be the highest priority one from the expanded list supplied to the client socket. Hooray.
• Now repeat, connecting to the server socket again but this time exclude the cipher that was chosen in the previous round, by appending the negation of it to the client socket's cipher list e.g. 'DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2:!AES256-GCM-SHA384')
• Repeat until the SSL handshake fails, because we've run out of ciphers.

Here is the code (also available as a github gist):

"""An attempt to produce similar output to "openssl ciphers -v", but for
python's built-in ssl.

To answer https://stackoverflow.com/q/28332448/445073
"""
from __future__ import print_function

import argparse
import logging
import multiprocessing
import os
import socket
import ssl
import sys

def server(log_level, queue):
    logging.basicConfig(level=log_level)
    logger = logging.getLogger("server")

    logger.debug("Creating bind socket")
    bind_sock = socket.socket()
    bind_sock.bind(('127.0.0.1', 0))
    bind_sock.listen(5)

    bind_addr = bind_sock.getsockname()
    logger.debug("Listening on %r", bind_addr)
    queue.put(bind_addr)

    while True:
        logger.debug("Waiting for connection")
        conn_sock, fromaddr = bind_sock.accept()
        conn_sock = ssl.wrap_socket(conn_sock,
                                    ssl_version=ssl.PROTOCOL_SSLv23,
                                    server_side=True,
                                    certfile="server.crt",
                                    keyfile="server.key",
                                    ciphers="ALL:aNULL:eNULL")

        data = conn_sock.read()
        logger.debug("Read %r", data)
        conn_sock.close()
    logger.debug("Done")

def parse_args(argv):
    parser = argparse.ArgumentParser(
        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    parser.add_argument("--verbose", "-v", action="store_true",
                        help="Turn on debug logging")
    parser.add_argument("--ciphers", "-c",
                        default=ssl._DEFAULT_CIPHERS,
                        help="Cipher list to test. Defaults to this python's "
                        "default client list")
    args = parser.parse_args(argv[1:])
    return args

if __name__ == "__main__":
    args = parse_args(sys.argv)

    log_level = logging.DEBUG if args.verbose else logging.INFO

    logging.basicConfig(level=log_level)
    logger = logging.getLogger("client")

    if not os.path.isfile('server.crt') or not os.path.isfile('server.key'):
        print("Must generate server.crt and server.key before running")
        print("Try:")
        print("openssl req -x509 -newkey rsa:2048 -keyout server.key -out server.crt -nodes -days 365  -subj '/CN=127.0.0.1'")
        sys.exit(1)

    queue = multiprocessing.Queue()
    server_proc = multiprocessing.Process(target=server, args=(log_level, queue))
    server_proc.start()
    logger.debug("Waiting for server address")
    server_addr = queue.get()

    chosen_ciphers = []
    try:
        cipher_list = args.ciphers
        while True:
            client_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            client_sock = ssl.wrap_socket(client_sock,
                                          ssl_version=ssl.PROTOCOL_SSLv23,
                                          ciphers=cipher_list)
            logger.debug("Connecting to %r", server_addr)
            client_sock.connect(server_addr)
            logger.debug("Connected")

            chosen_cipher = client_sock.cipher()
            chosen_ciphers.append(chosen_cipher)

            client_sock.write("ping")
            client_sock.close()

            # Exclude the first choice cipher from the list, to see what we get
            # next time.
            cipher_list += ':!' + chosen_cipher[0]
    except ssl.SSLError as err:
        if 'handshake failure' in str(err):
            logger.debug("Handshake failed - no more ciphers to try")
        else:
            logger.exception("Something bad happened")
    except Exception:
        logger.exception("Something bad happened")
    else:
        server_proc.join()
    finally:
        server_proc.terminate()

    print("Python: {}".format(sys.version))
    print("OpenSSL: {}".format(ssl.OPENSSL_VERSION))
    print("Expanding cipher list: {}".format(args.ciphers))
    print("{} ciphers found:".format(len(chosen_ciphers)))
    print("\n".join(repr(cipher) for cipher in chosen_ciphers))

Note how it defaults to testing the default cipher list built-in to python:

day@laptop ~/test
$ python --version
Python 2.7.8

day@laptop ~/test
$ python ssltest.py -h
usage: ssltest.py [-h] [--verbose] [--ciphers CIPHERS]

optional arguments:
  -h, --help            show this help message and exit
  --verbose, -v         Turn on debug logging (default: False)
  --ciphers CIPHERS, -c CIPHERS
                        Cipher list to test. Defaults to this python's default
                        client list (default:
                        DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2)

so we can easily see what the default client cipher list expands to, and how this changed from python 2.7.8 to 2.7.9:

day@laptop ~/test
$ ~/dists/python-2.7.8-with-pywin32-218-x86/python ssltest.py
Python: 2.7.8 (default, Jun 30 2014, 16:03:49) [MSC v.1500 32 bit (Intel)]
OpenSSL: OpenSSL 1.0.1h 5 Jun 2014
Expanding cipher list: DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2
12 ciphers found:
('AES256-GCM-SHA384', 'TLSv1/SSLv3', 256)
('AES256-SHA256', 'TLSv1/SSLv3', 256)
('AES256-SHA', 'TLSv1/SSLv3', 256)
('CAMELLIA256-SHA', 'TLSv1/SSLv3', 256)
('DES-CBC3-SHA', 'TLSv1/SSLv3', 168)
('AES128-GCM-SHA256', 'TLSv1/SSLv3', 128)
('AES128-SHA256', 'TLSv1/SSLv3', 128)
('AES128-SHA', 'TLSv1/SSLv3', 128)
('SEED-SHA', 'TLSv1/SSLv3', 128)
('CAMELLIA128-SHA', 'TLSv1/SSLv3', 128)
('RC4-SHA', 'TLSv1/SSLv3', 128)
('RC4-MD5', 'TLSv1/SSLv3', 128)

day@laptop ~/test
$ ~/dists/python-2.7.9-with-pywin32-219-x86/python ssltest.py
Python: 2.7.9 (default, Dec 10 2014, 12:24:55) [MSC v.1500 32 bit (Intel)]
OpenSSL: OpenSSL 1.0.1j 15 Oct 2014
Expanding cipher list: ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:ECDH+RC4:DH+RC4:RSA+RC4:!aNULL:!eNULL:!MD5
18 ciphers found:
('ECDHE-RSA-AES256-GCM-SHA384', 'TLSv1/SSLv3', 256)
('ECDHE-RSA-AES128-GCM-SHA256', 'TLSv1/SSLv3', 128)
('ECDHE-RSA-AES256-SHA384', 'TLSv1/SSLv3', 256)
('ECDHE-RSA-AES256-SHA', 'TLSv1/SSLv3', 256)
('ECDHE-RSA-AES128-SHA256', 'TLSv1/SSLv3', 128)
('ECDHE-RSA-AES128-SHA', 'TLSv1/SSLv3', 128)
('ECDHE-RSA-DES-CBC3-SHA', 'TLSv1/SSLv3', 112)
('AES256-GCM-SHA384', 'TLSv1/SSLv3', 256)
('AES128-GCM-SHA256', 'TLSv1/SSLv3', 128)
('AES256-SHA256', 'TLSv1/SSLv3', 256)
('AES256-SHA', 'TLSv1/SSLv3', 256)
('AES128-SHA256', 'TLSv1/SSLv3', 128)
('AES128-SHA', 'TLSv1/SSLv3', 128)
('CAMELLIA256-SHA', 'TLSv1/SSLv3', 256)
('CAMELLIA128-SHA', 'TLSv1/SSLv3', 128)
('DES-CBC3-SHA', 'TLSv1/SSLv3', 112)
('ECDHE-RSA-RC4-SHA', 'TLSv1/SSLv3', 128)
('RC4-SHA', 'TLSv1/SSLv3', 128)

And I think this answers my question. Unless anyone can see a problem with this approach?

Comments

[Day]

Ah thank you. That was useful, and I may be able to build on this to get an answer. But SSLContext is new in python 2.7.9, so I'm not going to be able to use this approach to figure out what ciphers were available in the python 2.7.8 build, so I'm still not going to know what changed between those two builds :(

[Day]

Ah... and SSLSocket.shared_ciphers itself wasn't added until python 3.5 - a version so recent that it has not been released yet. In fact, the patch to add this only landed last month :(

[Dr. Jan-Philip Gehrcke]

I was afraid so, the whole SSL architecture in Python experienced a severe re-work in Python 2.7.9 and 3.4, leading to many further change requests from the community. Maybe you can use ctypes and access the static OpenSSL build shipped with Python for Windows?

[Dr. Jan-Philip Gehrcke]

Interesting approach! As of the level of complexity involved, you should make further tests to support the statement "Equivalent of "openssl ciphers -v" for python's built-in ssl". That would be great.

[Day]

Good point. Unfortunately I've tested this on linux where python is using the same lib as openssl ciphers -v and I get far fewer ciphers returned by my python than by the openssl command line. So looks like what I say is misleading at best. I'll perhaps have to change to "An attempt to produce similar output as "openssl ciphers -v" for python's built-in ssl"