Using Lets Encrypt certificates with openLDAP

Question

I've been running an openLDAP server for several months now and we use it to authenticate for a number of applications. A previous staff member set up the server and it doesn't seem to be a standard installation but it's pretty straightforward.

Recently one of our CA certificates expired and the decision was made to replace it with Let's Encrypt. My manager replaced the certificate on the server.

It works for the web application (LDAP Manager, self-service password changing), however no clients can authenticate against it. For example, if I try to test a Redmine LDAP configuration, I get a message saying "Unable to connect (SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A)"

Testing Nexus authentication against it it just doesn't connect.

Frustratingly there is nothing in the logs either on the LDAP server or those with the applications to indicate why this is failing. My investigations lead me to believe that it is something to do how the certificate/key are configured but I have tried everything I can think of and everything I can find online and nothing works.

Environment details are:

Debian 8 openLDAP openldap-2.4.40

My config is as below:

/etc/ldap/ldap.conf

# LDAP Defaults
#
# See ldap.conf(5) for details
# This file should be world readable but not world writable.
#BASE   dc=example,dc=com
#URI    ldap://ldap.example.com ldap://ldap-master.example.com:666
#SIZELIMIT      12
#TIMELIMIT      15
#DEREF          never
# TLS certificates (needed for GnuTLS)
TLS_CACERT      /etc/letsencrypt/live/myserver.com/fullchain.pem

/etc/ldap/slapd.d/cn=config.ldif

dn: cn=config
objectClass: olcGlobal
cn: config
olcArgsFile: /var/run/slapd/slapd.args
olcLogLevel: none
olcPidFile: /var/run/slapd/slapd.pid
olcToolThreads: 1
structuralObjectClass: olcGlobal
entryUUID: c6dd9e40-9dc2-1035-8c03-add74f928a5e
creatorsName: cn=config
createTimestamp: 20160423171552Z
entryCSN: 20160423171552.629347Z#000000#000#000000
modifiersName: cn=config
modifyTimestamp: 20160423171552Z

If I test the connection:

admin@ldap:~$ sudo openssl s_client -connect localhost:636 -showcerts -state -CAfile /etc/letsencrypt/live/myserver.com/fullchain.pem
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
140394818631312:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:184:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE

Does anyone had any idea what I am missing?

EDIT

As per suggestion from @84104 I have edited the tls.ldif file to read as follows:

dn: cn=config
changetype: modify
replace: olcTLSCACertificateFile
olcTLSCACertificateFile: /etc/letsencrypt/live/myserver/fullchain.pem
-
replace: olcTLSCertificateFile
olcTLSCertificateFile: /etc/letsencrypt/live/myserver/cert.pem
-
replace: olcTLSCertificateKeyFile
olcTLSCertificateKeyFile: /etc/letsencrypt/live/myserver/privkey.pem

Then run the command:

ldapmodify -Y EXTERNAL -H ldapi:/// -f tls.ldif

However the output I now get is:

SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
modifying entry "cn=config"
ldap_modify: Other (e.g., implementation specific) error (80)

I found suggestion this may be due to permissions on the certificate or key files but I changed these to match exactly with the ones on the previously used files and still got this message.

Again I apologise for my lack of general knowledge on the topic but can anyone suggest anything else?

EDIT

As per the suggestion I altered tls.ldif and changed all the commands from replace to delete, then ran the ldapmodify command again. There is another error.

    admin@ldap:/etc/ansible_ldif_work$ sudo ldapmodify -Y EXTERNAL -H   ldapi:/// -f tls.remove.ldif
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
modifying entry "cn=config"
ldap_modify: Inappropriate matching (18)
        additional info: modify/delete: olcTLSCACertificateFile: no equality matching rule

Solution 1

The fullchain.pem file is NOT a concatenation of the certificate chain above the cert.pem file, it is a concatenation of the chain.pem and cert.pem file.

The chain.pem file and the root authority file must be concatenated into the file you will present to slapd as olcTLSCACertificateFile

The privkey.pem file must be presented to slapd as olcTLSCertificateKeyFile.

The simple cert.pem file must be presented to slapd as olcTLSCertificateFile.

I am uncertain if the order of concatenation matters, but this is the order I used: cat chain.pem root.pem > ca.merged.crt

The openssl test you used shows everything is OK when set up like this.

The root authority file can be found here: https://www.identrust.com/certificates/trustid/root-download-x3.html

Test:

[root@█████ ssl]# openssl s_client -connect [REDACTED]:636 -showcerts -state -CAfile ca.merged.crt
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = [REDACTED]
verify return:1
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
---
Certificate chain
 0 s:/CN=[REDACTED]
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
[REDACTED]
-----END CERTIFICATE-----
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
-----BEGIN CERTIFICATE-----
[REDACTED]
-----END CERTIFICATE-----
 2 s:/O=Digital Signature Trust Co./CN=DST Root CA X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
-----BEGIN CERTIFICATE-----
[REDACTED]
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=[REDACTED]
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Server Temp Key: ECDH, secp384r1, 384 bits
---
SSL handshake has read 4417 bytes and written 405 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: [REDACTED]
    Session-ID-ctx:
    Master-Key: [REDACTED]
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1487882605
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Solution 2

Do you have any extra security measures enabled (like apparmor) which restrict read access to your certificates? I got the same error message ldap_modify: Other (e.g., implementation specific) error (80) because apparmor did not allow access for openldap to the let's encrypt certificates:

The following steps resolved the issue for me:

• Add line to /etc/apparmor.d/local/usr.sbin.slapd: /etc/letsencrypt/** r,

• service apparmor restart

Solution 3

Your OpenLDAP server doesn't appear to have TLS configured.

Your /etc/ldap/slapd.d/cn=config.ldif should have something like the following:

olcTLSCertificateKeyFile: /etc/ldap/ssl/ldap.key
olcTLSCACertificateFile: /etc/ldap/ssl/ldap_ca.cert
olcTLSCertificateFile: /etc/ldap/ssl/ldap.cert
olcTLSCipherSuite: HIGH:!aNull:!MD5:@STRENGTH
olcTLSProtocolMin: 3.1

You should add that in via ldapmodify.

Solution 4

There is a beautiful blog post about this topic. It works for me https://www.dahlem.uk/display/deb/Configure+and+enable+TLS+for+OpenLDAP

Update (blog backup on archive.org):https://web.archive.org/web/20161023210915/http://www.dahlem.uk:80/display/deb/Configure+and+enable+TLS+for+OpenLDAP

My system is this:

# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.8 (jessie)
Release:        8.8
Codename:       jessie

# slapd -V
@(#) $OpenLDAP: slapd  (Jul 16 2017 19:57:41) $
        Debian OpenLDAP Maintainers <[email protected]>

Here a quick run through. Handle file system access to letsencrypt ...

useradd letsencrypt
chown openldap:letsencrypt /etc/letsencrypt/ -R
usermod -a -G letsencrypt openldap

Activate services ...

# /etc/default/slapd
SLAPD_SERVICES="ldap:/// ldapi:/// ldaps:///"

And tell openldap about your certs ...

# /root/add_ssl.ldif
dn: cn=config
changetype: modify
add: olcTLSCipherSuite
olcTLSCipherSuite: NORMAL
-
add: olcTLSCRLCheck
olcTLSCRLCheck: none
-
add: olcTLSVerifyClient
olcTLSVerifyClient: never
-
add: olcTLSCACertificateFile
olcTLSCACertificateFile: /etc/letsencrypt/live/YOURDOMAIN/fullchain.pem
-
add: olcTLSCertificateFile
olcTLSCertificateFile: /etc/letsencrypt/live/YOURDOMAIN/cert.pem
-
add: olcTLSCertificateKeyFile
olcTLSCertificateKeyFile: /etc/letsencrypt/live/YOURDOMAIN/privkey.pem
-
add: olcTLSProtocolMin
olcTLSProtocolMin: 3.3

Read in the ldif file ...

ldapmodify -Y EXTERNAL -H ldapi:/// -f add_ssl.ldif

Finally restart and check slapd.

systemctl restart slapd.service
systemctl status slapd.service

Solution 5

I had the same problem setting up certifications from Lets Encrypt with OpenLDAP

The error:

~ # ldapmodify -Y EXTERNAL -H ldapi:/// -f add_ssl.ldif
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
modifying entry "cn=config"
ldap_modify: Other (e.g., implementation specific) error (80)

The log files contain:

... apparmor="DENIED" operation="open" profile="/usr/sbin/slapd" name="/etc/letsencrypt/archive/your.domain.tld/fullchain1.pem" ...

I have found the following solution:

1. Edit file /etc/apparmor.d/usr.sbin.slapd
2. Add line: /etc/letsencrypt/archive/your.domain.tld/* r,
3. Restart *apparmor*:service apparmor restart`
4. And now execute ldapmodify again

View more solutions

Comments

[gxx]

Which OS? Which software versions are you running?

[shaneoh]

Sorry - I've edited the original post. Debian 8 and openLDAP-2.4.40 to answer your questions,

[Jenny D]

What happens if you try to connect using openssl s_client to the LDAP server?

[shaneoh]

Im not entirely sure if I'm doing this correctly but so that I could include the code I have edited the original question.

[shaneoh]

I've edited my original post to show what happens when I tried this.

[84104]

@shaneoh While normally replace is the most expedient modify operation there are a few olc* attributes that don't do well with replace. Try add, or if you already have value present, delete followed by add.

[shaneoh]

changed to delete, but now get the error: ldap_modify: Inappropriate matching (18) additional info: modify/delete: olcTLSCACertificateFile: no equality matching rule

[shaneoh]

We don't have apparmor, no selinux, nothing else that I have been made aware of over and above the usual security

[Niols]

Thanks a lot! Note that, if you want to test it with ldapsearch (with -ZZ for StartTLS or through ldaps), you have to provide the ca-certificates file (/etc/ssl/certs/ca-certificates.crt on my computer) in the LDAPTLS_CACERT environment variable.

[StvnW]

This will break as soon as your certificates need to be renewed in 90 days.

[fri.K]

@StvnW why do you think so? It's working for a few months now without problem. New certificate is always placed in the same location, so ldap is using new signed cert without any issues.

[StvnW]

Renewals do not overwrite the same file. Certificates are stored in .../archive/<domain>/{fullchain,cert,privkey}n.pem where the version n is incremented with each renewal. If you point to a specific key pair in archive, that pair will be superseded by n+1 at renewal and you'll be left serving an expired certificate. The live folder solves this by providing symlinks to the latest version. It may not be obviously broken in the functional sense (clients can choose to accept expired certs), but it will be broken from a trust perspective and for clients that enforce checking.

[Max Muster]

the beautiful blog is now on archive.org web.archive.org/web/20161023210915/http://www.dahlem.uk:80/… hope someone ever republish it

[frank_108]

I've needed to add explicit all directories within apparmor, like: /etc/ssl/openldap/certs/ r, /etc/ssl/openldap/certs/* r, /etc/ssl/openldap/private/ r, /etc/ssl/openldap/private/* r, (each comma is end of the line) One simple wildcard wasn't enough.