Changing an OpenSSL BIO from blocking to non-blocking mode

Question

I have a multithreaded application that makes heavy use of OpenSSL in C. It is designed with the idea that all of its SSL connections are expected to block. Specifically, blocking BIOs. They are all allocated off a single incoming port like this:

ssl = SSL_new(ctx);
SSL_set_mode(ssl, SSL_MODE_AUTO_RETRY);
sock = BIO_new_socket(socket, BIO_CLOSE);
SSL_set_bio(ssl, sock, sock);

As it turns out though, there are a few small parts of the codebase where using non-blocking BIOs would be the best choice. The small parts that would benefit from the non-blocking BIOs have no way of knowing which SSL connections will belong to them. Thus, they always receive blocking BIOs.

The question is, can the blocking BIOs be changed to be non-blocking?

I know that BIO_set_nbio can be used to make a BIO non-blocking but the documentation says:

The call to BIO_set_nbio() should be made before the connection is established because non blocking I/O is set during the connect process.

Another possible option I have thought about would be to copy the BIO and recreate it, while somehow maintaining all of the state.

Solution

I did non-blocking SSL connections in my own "lion" code, but I did not use the BIO functionality in OpenSSL at all.

Rather, I went for the calls SSL_set_fd(ctx, fd ) and SSL_get_fd(ssl) to handle my own fdsets and calling select.

The biggest 'gotcha' that took a while to track down was to set SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER and SSL_MODE_ENABLE_PARTIAL_WRITE for it work the way I wanted.

If you want to read the SSL part of the code, it is here:

https://github.com/lundman/lion/blob/master/src/tls.c