Apache load balancer with https real servers and client certificates

Question

Our network requirements state that ALL network traffic must be encrypted.

The network configuration looks like this:

                                                              ------------
                                                /-- https --> | server 1 | 
                                               /              ------------
|------------|               |---------------|/               ------------
|   Client   | --- https --> | Load Balancer | ---- https --> | server 2 |
|------------|               |---------------|\               ------------
                                               \              ------------
                                                \-- https --> | server 3 |
                                                              ------------

And it has to pass client certificates.

I've got a config that can do load balancing with in-the-clear real servers:

<VirtualHost *:8666>

    DocumentRoot "/usr/local/apache/ssl_html"
    ServerName vmbigip1
    ServerAdmin [email protected]
    DirectoryIndex index.html


   <Proxy *>
        Order deny,allow
        Allow from all
   </Proxy>

    SSLEngine on
    SSLProxyEngine On
    SSLCertificateFile /usr/local/apache/conf/server.crt
    SSLCertificateKeyFile /usr/local/apache/conf/server.key


   <Proxy balancer://mycluster>
       BalancerMember http://1.2.3.1:80
       BalancerMember http://1.2.3.2:80
       # technically we aren't blocking anyone, but could here
       Order Deny,Allow
       Deny from none
       Allow from all
       # Load Balancer Settings
       # A simple Round Robin load balancer.
       ProxySet lbmethod=byrequests
   </Proxy>

   # balancer-manager
   # This tool is built into the mod_proxy_balancer module allows you
   # to do simple mods to the balanced group via a gui web interface.
   <Location /balancer-manager>
       SetHandler balancer-manager
       Order deny,allow
       Allow from all
   </Location>

    ProxyRequests Off
    ProxyPreserveHost On

    # Point of Balance
    # Allows you to explicitly name the location in the site to be
    # balanced, here we will balance "/" or everything in the site.
    ProxyPass /balancer-manager !
    ProxyPass / balancer://mycluster/ stickysession=JSESSIONID
</VirtualHost>

What I need is for the servers in my load balancer to be

       BalancerMember https://1.2.3.1:443
       BalancerMember https://1.2.3.2:443

But that does not work. I get SSL negotiation errors.

Even when I do get that to work, I will need to pass client certificates.

Any help would be appreciated.

Solution 1

You simply cannot use a client certificate directly with a back-end node that would request the user certificate and where the load-balancer would "terminate" the SSL/TLS connection from the end-user.

The SSL/TLS handshake with a client certificate requires that the client signs all the handshake messages exchanged between the client and the server at the end, which means that the client must be connected directly to the actually SSL/TLS server requesting the client certificate. If the browser's SSL/TLS connection only goes as far as the load balancer, it's the load balancer that is the client to the back-end node. The back-end node will see a different handshake, and this will fail.

There are two possible ways around this:

• Use a DNS or TCP-based load-balancer (e.g. something like ipchains): in this case the SSL/TLS connection from the browser will go directly to the back-end node. Direct client-certificate authentication will be possible.

• Have the load-balancer perform the client-certificate authentication, and simply convey that information to the back-end node. This requires the back-end node to trust the load-balancer to have made the verification properly, but if the back-end node can't trust the load-balancer, there's no point using one.

  mod_proxy_ajp (or mod_jk) can forward the client certificate as part of the AJP protocol, but that's mainly for Java containers and AJP traffic isn't encrypted anyway.

  If you're using mod_proxy_http, you can add an extra header with mod_header to pass the certificate via an HTTP header, using something like RequestHeader set X-ClientCert %{SSL_CLIENT_CERT}s. I can't remember the exact details, but it's important to make sure that this header is cleared so that it never comes from the client's browser (who could forge it otherwise). Whatever application you have running on the back-end node will need to be able to get its authentication information from that header (and trust it as having been verified already by the load-balancer).

  In addition to the SSLVerifyClient and SSLCACertificateFile/Path directives that you need to configure for normal client-certificate authentication on Apache Httpd with mod_ssl, you'll also need to configure SSLProxyCheckPeerCN on and SSLProxyCACertificateFile/Path to configure Apache Httpd as an SSL/TLS client to the back-end nodes (see introduction of the mod_proxy documentation, about the SSLProxy* directives).

  If you want the back-end servers to make sure that the request come from the load-balancer and not a different client (which could make a direct connection), you could make the load-balancer use a client-certificate too (using SSLProxyMachineCertificateFile), to authenticate to the back-end note. Note that this may get the authentication system on the back-end not a bit more complex: although the actual client-certificate authentication they get would be that of the proxy (which they'd need to verify as usual), the applications on those servers would need to be configured to use the header-based client-certificate as far as application users are concerned.

Solution 2

Your SSL configuration looks incomplete. For requiring client certificates, you need to add

SSLVerifyClient require
SSLVerifyDepth  1
SSLCACertificateFile <your CA cert file>

See How can I force clients to authenticate using certificates for details.

Not sure if it's mandatory, but you're also missing a SSLCipherSuite directive. If done like this, you can sort out weak ciphers and do what you can against the BEAST attack:

SSLHonorCipherOrder On
SSLCipherSuite RC4-SHA:ALL:!ADH:!EXPORT:!LOW:!MD5:!SSLV2:!NULL

Comments

[f_puras]

Looking at the OP's diagram, it seems to me that the load balancer does in fact not terminate the SSL connection but forwards it to one of the servers 1/2/3. IMHO this is the better solution anyway: Few load balancers are capable of handling client certs at all, and if so, they do not allow accessing details of the cert presented, as Apache does.

[Bruno]

@f_puras the fact that the sample Apache Httpd configuration uses BalancerMember and Proxy directives indicates that this is the configuration of the load-balancer, which is therefore terminating the SSL connection. (mod_proxy doesn't do simple TCP forwarding which would be required to forward the full SSL/TLS connection to the back-end.)

[f_puras]

Well, you're right! I'm used to thinking of load balancers as black boxes with funny company names on them. So configuration contradicts diagram here.

[Bruno]

@f_puras I think the diagram is OK in that it doesn't imply that the https connections to the back-ends are the same https connection as the one from the browser. However, the configuration is clearly missing all the SSLProxy* directives, if the connections from the load-balancer to the worker nodes should use https.