Can't pass TLS-SNI-01 challenge with Let's Encrypt/Certbot - how to fix?

Question

I am running Ubuntu 14.04LTS. I have grabbed certbot-auto from the EFF website since certbot isn't in 14.04.

I've tried running it as

certbot-auto --apache -d my.domain

and it consistently times out during the TLS-SNI-01 challenge with the following message:

Failed authorization procedure. my.domain (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Failed to connect to ww.xx.yy.zz:443 for TLS-SNI-01 challenge

I have verified that inbound TLS is not being blocked by running openssl's s_server command and successfully connecting to the s_server "server" from external locations.

Looking at /var/log/apache2/error.log I do see a suspicious warning about (I assume) the docroot that certbot-auto is temporarily trying to set up not being found:

[Tue Dec 06 00:34:02.306032 2016] [mpm_prefork:notice] [pid 19521] AH00171: Graceful restart requested, doing restart
[Tue Dec 06 00:34:03.001014 2016] [mpm_prefork:notice] [pid 19521] AH00163: Apache/2.4.7 (Ubuntu) configured -- resuming normal operations
[Tue Dec 06 00:34:03.001094 2016] [core:notice] [pid 19521] AH00094: Command line: '/usr/sbin/apache2'
[Tue Dec 06 00:34:14.596461 2016] [mpm_prefork:notice] [pid 19521] AH00171: Graceful restart requested, doing restart
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
[Tue Dec 06 00:34:14.661372 2016] [ssl:warn] [pid 19521] AH01906: RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Tue Dec 06 00:34:15.000674 2016] [mpm_prefork:notice] [pid 19521] AH00163: Apache/2.4.7 (Ubuntu) OpenSSL/1.0.1f configured -- resuming normal operations
[Tue Dec 06 00:34:15.001293 2016] [core:notice] [pid 19521] AH00094: Command line: '/usr/sbin/apache2'

In /var/log/letsencrypt/letsencrypt.log I see the following, which looks totally reasonable:

2016-12-06 05:34:13,247:INFO:certbot.auth_handler:Performing the following challenges:
2016-12-06 05:34:13,268:INFO:certbot.auth_handler:tls-sni-01 challenge for mydomain.com
2016-12-06 05:34:13,359:INFO:certbot_apache.configurator:Enabled Apache socache_shmcb module
2016-12-06 05:34:13,520:INFO:certbot_apache.configurator:Enabled Apache ssl module
2016-12-06 05:34:14,345:DEBUG:certbot_apache.tls_sni_01:Adding Include /etc/apache2/le_tls_sni_01_cert_challenge.conf to /files/etc/apache2/apache2.conf
2016-12-06 05:34:14,351:DEBUG:certbot_apache.tls_sni_01:writing a config file with text:
 <IfModule mod_ssl.c>
<VirtualHost *>
    ServerName b103a17811594b35d3ade8eb763c8c42.74b423d383109b22eaecca3ea14cd1f7.acme.invalid
    UseCanonicalName on
    SSLStrictSNIVHostCheck on

    LimitRequestBody 1048576

    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /var/lib/letsencrypt/mRH5i2aIrWMqjsZn3jMXvdNnYV5Ejd0GBtcDoIJqQ4U.crt
    SSLCertificateKeyFile /var/lib/letsencrypt/mRH5i2aIrWMqjsZn3jMXvdNnYV5Ejd0GBtcDoIJqQ4U.pem

    DocumentRoot /var/lib/letsencrypt/tls_sni_01_page/
</VirtualHost>

</IfModule>

If I watch /var/lib/letsencrypt while certbot-auto runs I don't ever see a tls_sni_01_page being created. Here's the sequence I see -- a temp directory and some temp certs being made and then deleted when the challenge fails, but never tls_sni_01_page

/var/lib/letsencrypt$ ls
backups
/var/lib/letsencrypt$ ls
backups
/var/lib/letsencrypt$ ls
backups
/var/lib/letsencrypt$ ls
backups  temp_checkpoint
/var/lib/letsencrypt$ ls
backups  temp_checkpoint
/var/lib/letsencrypt$ ls
backups  e1pvy9TgUvOmBxC4dlkwG5Ly2mTY0DU63qhUFpBik2Y.crt  e1pvy9TgUvOmBxC4dlkwG5Ly2mTY0DU63qhUFpBik2Y.pem  temp_checkpoint

I've also changed /etc/letsencrypt/options-ssl-apache.conf to set the logging level to DEBUG and to have that temporary virtual host log to its own error and access logs. I see output but no problems in the error log, but don't see anything at all in the access log.

I've tried running tcpdump -n port 443 and I do see some sort of connection attempt from the remote machine:

09:14:44.388328 IP 66.133.109.36.48890 > w.x.y.z.443: Flags [S], seq 1627589669, win 29200, options [mss 1460,sackOK,TS val 2834688174 ecr 0,nop,wscale 7], length 0
09:14:44.388435 IP w.x.y.z.443 > 66.133.109.36.48890: Flags [S.], seq 773250860, ack 1627589670, win 28960, options [mss 1460,sackOK,TS val 424229197 ecr 2834688174,nop,wscale 7], length 0
09:14:44.451190 IP 66.133.109.36.48890 > w.x.y.z.443: Flags [.], ack 1, win 229, options [nop,nop,TS val 2834688237 ecr 424229197], length 0
09:14:44.451546 IP 66.133.109.36.48890 > w.x.y.z.443: Flags [P.], seq 1:220, ack 1, win 229, options [nop,nop,TS val 2834688237 ecr 424229197], length 219
09:14:44.451619 IP w.x.y.z.443 > 66.133.109.36.48890: Flags [.], ack 220, win 235, options [nop,nop,TS val 424229216 ecr 2834688237], length 0
09:14:44.454461 IP w.x.y.z.443 > 66.133.109.36.48890: Flags [P.], seq 1:393, ack 220, win 235, options [nop,nop,TS val 424229217 ecr 2834688237], length 392
09:14:44.454626 IP w.x.y.z.443 > 66.133.109.36.48890: Flags [F.], seq 393, ack 220, win 235, options [nop,nop,TS val 424229217 ecr 2834688237], length 0
09:14:44.517433 IP 66.133.109.36.48890 > w.x.y.z.443: Flags [.], ack 393, win 237, options [nop,nop,TS val 2834688303 ecr 424229217], length 0
09:14:44.517481 IP 66.133.109.36.48890 > w.x.y.z.443: Flags [P.], seq 220:227, ack 394, win 237, options [nop,nop,TS val 2834688303 ecr 424229217], length 7
09:14:44.517514 IP 66.133.109.36.48890 > w.x.y.z.443: Flags [F.], seq 227, ack 394, win 237, options [nop,nop,TS val 2834688303 ecr 424229217], length 0
09:14:44.517544 IP w.x.y.z.443 > 66.133.109.36.48890: Flags [.], ack 228, win 235, options [nop,nop,TS val 424229235 ecr 2834688303], length 0

Any idea what the heck is going on? Anyone ever get this working with Ubuntu 14.04LTS and apache?

Solution 1

I fought this for several hours, complete with the same output in my logs. I only stumbled across my answer. I had pasted some code from another webconfig and it already had a <virtual host _._._._:443> section in it. I altered it but left it in. After deleting that 443 section, the sudo certbot-auto --apache -d example.com ran without error and I had a working site.

I am drawing conclusions only from my experience: make sure you only have virtual hosts for port 80. Nowhere in the documentation that I read mentioned this issue, but it seems that certbot cannot alter a 443 virtual host section, it can only add one.

EDIT -- Dec 2017 I can now run certbot on sites-available.conf with <virtual host *:443> in them. All my custom modifications were kept intact, my only gripe is that the certificate path lines were not indented. <sigh>Oh, well!</sigh>

Solution 2

You could try standalone challenge validation. This bypasses apache and should potentially solve you problem with certbot not creating the right files in WWW root.

1. stop apache
2. run certbot-auto --standalone-supported-challenges tls-sni-01 -d my.domain
3. install the certificate you got in the right place (probably /etc/apache2/ssl)
4. start apache and test

Comments

[QuantumMechanic]

That did the trick! Thanks! Mine was slightly different. I had <VirtualHost *>. When I changed it to <VirtualHost *:80> then it worked.

[QuantumMechanic]

BTW - how did you figure out that having the :443 was causing the problem? Or did you just take it out in (quasi-)desperation and it just worked?

[QuantumMechanic]

The LE project has officially reproduced this and added a bug for it: github.com/certbot/certbot/issues/3981

[wruckie]

Desperation: I was trying everything.

[Axm]

God bless you, I was fighting this for hours.

[ivanivan]

This is how I've always done it - why trust a script to make the right changes to the right file(s)? Stand alone puts the cert files in /etc/letsencrypt/live/example.com/