SSL Reverse Proxy not working after upgrade from Apache 2.2.14 to 2.2.22

Question

After upgrading my Apache to 2.2.22 I can no longer connect to my internal servers by https. Internal servers respond ok if I don't use HTTPS, otherwise I get this on the Apache log:

[Mon Jan 06 18:20:37 2014] [info] Init: Seeding PRNG with 648 bytes of entropy
[Mon Jan 06 18:20:37 2014] [info] Loading certificate & private key of SSL-aware server
[Mon Jan 06 18:20:37 2014] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Mon Jan 06 18:20:37 2014] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Mon Jan 06 18:20:37 2014] [info] Shared memory session cache initialised
[Mon Jan 06 18:20:37 2014] [info] Init: Initializing (virtual) servers for SSL
[Mon Jan 06 18:20:37 2014] [info] Configuring server for SSL protocol
[Mon Jan 06 18:20:37 2014] [info] mod_ssl/2.2.22 compiled against Server: Apache/2.2.22, Library: OpenSSL/1.0.1
[Mon Jan 06 18:20:37 2014] [notice] Apache/2.2.22 (Ubuntu) mod_ssl/2.2.22 OpenSSL/1.0.1 configured -- resuming normal operations
[Mon Jan 06 18:20:37 2014] [info] Server built: Jul 12 2013 13:38:27

[Mon Jan 06 18:22:37 2014] [info] [client 111.111.111.97] Connection to child 10 established (server name.server.com:443)
[Mon Jan 06 18:22:37 2014] [info] Seeding PRNG with 648 bytes of entropy
[Mon Jan 06 18:22:37 2014] [info] [client 111.111.111.97] (70014)End of file found: SSL input filter read failed.
[Mon Jan 06 18:22:37 2014] [info] [client 111.111.111.97] Connection closed to child 10 with standard shutdown (server name.server.com:443)
[Mon Jan 06 18:22:37 2014] [info] [client 111.111.111.97] Connection to child 65 established (server name.server.com:443)
[Mon Jan 06 18:22:37 2014] [info] Seeding PRNG with 648 bytes of entropy
[Mon Jan 06 18:22:37 2014] [info] Initial (No.1) HTTPS request received for child 65 (server name.server.com:443)
[Mon Jan 06 18:22:37 2014] [info] [client 172.111.111.47] Connection to child 0 established (server name.server.com:443)
[Mon Jan 06 18:22:37 2014] [info] Seeding PRNG with 648 bytes of entropy
[Mon Jan 06 18:22:37 2014] [info] [client 172.111.111.47] SSL Proxy connect failed
[Mon Jan 06 18:22:37 2014] [info] SSL Library Error: 336130329 error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
[Mon Jan 06 18:22:37 2014] [info] [client 172.111.111.47] Connection closed to child 0 with abortive shutdown (server name.server.com:443)
[Mon Jan 06 18:22:37 2014] [error] (502)Unknown error 502: proxy: pass request body failed to 172.111.111.47:443 (172.111.111.47)
[Mon Jan 06 18:22:37 2014] [error] [client 111.111.111.97] proxy: Error during SSL Handshake with remote server returned by /app/login.jsp, referer: https://name.server.com/app/login.jsp
[Mon Jan 06 18:22:37 2014] [error] proxy: pass request body failed to 172.111.111.47:443 (172.111.111.47) from 111.111.111.97 ()
[Mon Jan 06 18:22:37 2014] [info] [client 111.111.111.97] Connection closed to child 65 with standard shutdown (server name.server.com:443)

---

but, If I replace the current /usr/lib/apache2/modules/mod_ssl.so with the old Apache 2.2.14 mod_ssl.so, It will work perfectly (!):

[Mon Jan 06 18:29:24 2014] [notice] SIGUSR1 received.  Doing graceful restart
[Mon Jan 06 18:29:24 2014] [info] Init: Seeding PRNG with 648 bytes of entropy
[Mon Jan 06 18:29:24 2014] [info] Loading certificate & private key of SSL-aware server
[Mon Jan 06 18:29:24 2014] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Mon Jan 06 18:29:24 2014] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Mon Jan 06 18:29:24 2014] [info] Shared memory session cache initialised
[Mon Jan 06 18:29:24 2014] [info] Init: Initializing (virtual) servers for SSL
[Mon Jan 06 18:29:24 2014] [info] Configuring server for SSL protocol
[Mon Jan 06 18:29:24 2014] [info] mod_ssl/2.2.14 compiled against Server: Apache/2.2.14, Library: OpenSSL/0.9.8k
[Mon Jan 06 18:29:24 2014] [notice] Apache/2.2.22 (Ubuntu) mod_ssl/2.2.14 OpenSSL/0.9.8o configured -- resuming normal operations
[Mon Jan 06 18:29:24 2014] [info] Server built: Jul 12 2013 13:38:27


[Mon Jan 06 18:29:49 2014] [info] [client 111.111.111.97] Connection to child 197 established (server name.server.com:443)
[Mon Jan 06 18:29:49 2014] [info] Seeding PRNG with 648 bytes of entropy
[Mon Jan 06 18:29:49 2014] [info] [client 111.111.111.97] (70014)End of file found: SSL input filter read failed.
[Mon Jan 06 18:29:49 2014] [info] [client 111.111.111.97] Connection closed to child 197 with standard shutdown (server name.server.com:443)
[Mon Jan 06 18:29:49 2014] [info] [client 111.111.111.97] Connection to child 128 established (server name.server.com:443)
[Mon Jan 06 18:29:49 2014] [info] Seeding PRNG with 648 bytes of entropy
[Mon Jan 06 18:29:49 2014] [info] Initial (No.1) HTTPS request received for child 128 (server name.server.com:443)
[Mon Jan 06 18:29:49 2014] [info] [client 172.111.111.47] Connection to child 0 established (server name.server.com:443)
[Mon Jan 06 18:29:49 2014] [info] Seeding PRNG with 648 bytes of entropy
[Mon Jan 06 18:29:50 2014] [info] Subsequent (No.2) HTTPS request received for child 128 (server name.server.com:443)
[Mon Jan 06 18:29:50 2014] [info] [client 111.111.111.97] Connection to child 198 established (server name.server.com:443)
[Mon Jan 06 18:29:50 2014] [info] Seeding PRNG with 648 bytes of entropy
[Mon Jan 06 18:29:50 2014] [info] [client 111.111.111.97] (70014)End of file found: SSL input filter read failed.
[Mon Jan 06 18:29:50 2014] [info] Subsequent (No.3) HTTPS request received for child 128 (server name.server.com:443)
[Mon Jan 06 18:29:50 2014] [info] [client 111.111.111.97] Connection closed to child 198 with standard shutdown (server name.server.com:443)
[Mon Jan 06 18:29:50 2014] [info] Subsequent (No.4) HTTPS request received for child 128 (server name.server.com:443)
[Mon Jan 06 18:29:50 2014] [info] Subsequent (No.5) HTTPS request received for child 128 (server name.server.com:443)
[Mon Jan 06 18:29:51 2014] [info] [client 111.111.111.97] Connection to child 129 established (server name.server.com:443)
[Mon Jan 06 18:29:51 2014] [info] Seeding PRNG with 648 bytes of entropy
[Mon Jan 06 18:29:55 2014] [info] [client 111.111.111.97] (70007)The timeout specified has expired: SSL input filter read failed.
[Mon Jan 06 18:29:55 2014] [info] [client 111.111.111.97] Connection closed to child 128 with standard shutdown (server name.server.com:443)

apache 2.2.22 mod_ssl:

root@reverseserver:/etc# ldd /usr/lib/apache2/modules/mod_ssl.so
        linux-gate.so.1 =>  (0xb76f6000)
        libssl.so.1.0.0 => /lib/i386-linux-gnu/libssl.so.1.0.0 (0xb766a000)
        libcrypto.so.1.0.0 => /lib/i386-linux-gnu/libcrypto.so.1.0.0 (0xb74bf000)
        libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xb74a3000)
        libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb72f9000)
        libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xb72f4000)
        libz.so.1 => /lib/i386-linux-gnu/libz.so.1 (0xb72de000)
        /lib/ld-linux.so.2 (0xb76f7000)

apache 2.2.14 mod_ssl:

root@reverseserver:~# ldd /usr/lib/apache2/modules/mod_ssl.so
        linux-gate.so.1 =>  (0xb77d1000)
        libssl.so.0.9.8 => /lib/i386-linux-gnu/libssl.so.0.9.8 (0xb7750000)
        libcrypto.so.0.9.8 => /lib/i386-linux-gnu/libcrypto.so.0.9.8 (0xb75d7000)
        libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xb75bb000)
        libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb7411000)
        libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xb740c000)
        libz.so.1 => /lib/i386-linux-gnu/libz.so.1 (0xb73f6000)
        /lib/ld-linux.so.2 (0xb77d2000)

should I continue to use mod_ssl from version 2.2.14? Is there any workaround for this issue?

Any help would be greatly appreciated!

Solution

This may be the same problem we just resolved. We had front-end Apache using OpenSSL 0.9.8 and going to backend servers via HTTPS. We tried to upgrade to using OpenSSL 1.0.1 and we stated seeing the same problem. After the SSL Poodle issue, we were forced to disable SSLv3 on the front side.

We were determined to resolve the problem so I started playing with settings. I discovered if you disable SSLv2 and SSLv3 on the front side then disable SSLv2 and TLSv1 on the back side, the connection between your front side and back side machine will use SSLv3 and will connect!

The settings I used were:

SSLProtocol all -SSLv2 -SSLv3
SSLProxyProtocol all -SSLv2 -TLSv1

Now TLSv1 on the front side and SSLv3 on the back internal network.