Hashicorp Vault tls cert authentication does not read certificate

Question

I'm trying to setup certificate-based authentication in Vault. For tests I've just created a clean Vault setup. Vault configuration below:

listener "tcp" {
  address                  = "192.168.33.10:8200"
  tls_cert_file            = "/etc/vault/vault_cert.pem"
  tls_key_file             = "/etc/vault/vault_key.pem"
  tls_disable_client_certs = false
  tls_disable              = false
}

listener "tcp" {
  address                  = "127.0.0.1:8200"
  tls_disable              = true
}

storage "file" {
  path = "/etc/vault/data"
}

I've started and unsealed Vault and enabled cert authentication:

[vagrant@localhost ~]$ ./vault status
Key             Value
---             -----
Seal Type       shamir
Sealed          false
Total Shares    5
Threshold       3
Version         0.9.3
Cluster Name    vault-cluster-37dffb3b
Cluster ID      1ddd4712-99f6-3691-a066-d476fbc6d7c6
HA Enabled      false
[vagrant@localhost ~]$ ./vault auth list
Path      Type     Description
----      ----     -----------
cert/     cert     n/a
token/    token    token based credentials

Now I've generated ssl key/certificate pair which is not added yet to Vault, so I expect Vault to tell me that certificate is invalid (at least that's my understanding from reading sources here. Though the answer I'm getting says that the certificate was not supplied at all:

[vagrant@localhost ~]$ VAULT_ADDR='https://192.168.33.10:8200' ./vault login -method cert -tls-skip-verify -client-cert=./client_cert.pem -client-key=./client_key.pem
Error authenticating: Error making API request.

URL: PUT https://192.168.33.10:8200/v1/auth/cert/login
Code: 400. Errors:

* client certificate must be supplied

From what I was able to find in sources, this error message is only returned when there is no certificate supplied to Vault at all. Just to make sure that's not a problem in Vault cli client, I've tried to do the same with curl, but got the same result:

[vagrant@localhost ~]$ curl -iv -k -X POST --cert ./client_cert.pem --key ./client_key.pem https://192.168.33.10:8200/v1/auth/cert/login
* About to connect() to 192.168.33.10 port 8200 (#0)
*   Trying 192.168.33.10...
* Connected to 192.168.33.10 (192.168.33.10) port 8200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*   subject: CN=Vault,[email protected],O=Exmaple,L=Berlin,ST=BERLIN,C=DE
*   start date: Feb 14 15:59:37 2018 GMT
*   expire date: Feb 12 15:59:37 2028 GMT
*   common name: Vault
*   issuer: CN=Vault,[email protected],O=Exmaple,L=Berlin,ST=BERLIN,C=DE
> POST /v1/auth/cert/login HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.168.33.10:8200
> Accept: */*
>
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Cache-Control: no-store
Cache-Control: no-store
< Content-Type: application/json
Content-Type: application/json
< Date: Wed, 14 Feb 2018 16:20:57 GMT
Date: Wed, 14 Feb 2018 16:20:57 GMT
< Content-Length: 51
Content-Length: 51

<
{"errors":["client certificate must be supplied"]}
* Connection #0 to host 192.168.33.10 left intact

Vault server running in trace mode doesn't provide any logs during this interaction. Though, if I will use a key from the different certificate trying to deliberately break TLS, I see an error message in Vault indicating that.

Any idea what could be wrong with this setup?

Solution

After digging through commits regarding the TLS client verification for Vault, I finally have a working Vault instance.

For setting up client certificate verification for any request to the Vault TCP listener, you have to configure two values:

• tls_client_ca_file = "/my/intermediate/ca/intermediate.cert.pem
• tls_require_and_verify_client_cert = true

When these two settings are configured, all requests to the backend without a valid client certificate are blocked, all valid client requests are then forwarded to the Vault instance.

In my understanding this simply is a "filter" for unauthorised access.

So let's begin with the TLS cert authentication setup:

First of all, we have to set up the Vault CLI to provide a Client Certificate for each request it executes:

export VAULT_CLIENT_CERT="/my/cert/path/vault-client.cert.pem"
export VAULT_CLIENT_KEY="/my/cert/path/vault-client.key.pem"

export VAULT_ADDR="https://my.vault.app.com:8200"

With this command, we can get through the TCP listener's "filter".

And finally execute the command that enables the Client Certificate authentication

vault auth enable cert

For granting access to specific clients, the certificate has to be linked to a policy which itself describes access to a specific Secrets Engine:

Let's create a simple KV secrets engine for myorganization/dev:

vault secrets enable -path=myorganization/dev kv

Then we have to create a policy file (HCL syntax) for the Secrets Engine:

path "myorganization/dev/myapp" {
     capabilities = ["read"]
}

After creating the policy file, we can upload it to the vault's policy store:

vault policy write myapp-read-access myapp-read-access.hcl

And finally we have to assign an app or client certificate* to this policy by uploading the .cert.pem file to the certificate store and mapping it to this policy:

vault write auth/cert/certs/myapp display_name="My Vault Test App" policies=myapp-read-access certificate=myapp.cert.pem ttl=3600

* This can be the same as in the export directives or some other cert (same CA of course)

After this is configured, you can then use the CLI client:

vault login -method=cert

Alternatively, you can specify another certificate for the login by using:

vault login -method=cert -client-cert=myapp.cert.pem -client-key=myapp.key.pem

In my case I used a Java client, with the following Maven POM for dependencies:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-vault-dependencies</artifactId>
    <version>2.0.2.BUILD-SNAPSHOT</version>
    <scope>import</scope>
    <type>pom</type>
</dependency>

And the following Vault client configuration (bootstrap.yml):

spring.application.name: myapp
spring.cloud.vault:
  host: my.vault.app.com
  port: 8200
  scheme: https
  generic.backend: myorganization/dev
  authentication: CERT
  ssl:
    key-store: classpath:myapp.jks
    key-store-password: <MYKEYSTOREPW>
    cert-auth-path: cert

Et voilà:

2018-09-19 14:02:18.114 INFO 51832 --- [ main] b.c.PropertySourceBootstrapConfiguration : Located property source: CompositePropertySource {name='vault', propertySources=[LeaseAwareVaultPropertySource {name='myorganization/dev/myapp'}, LeaseAwareVaultPropertySource {name='myorganization/dev/application'}]}

Or in case a invalid certificate is provided:

org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://my.vault.app.com:8200/v1/auth/cert/login": Received fatal alert: bad_certificate; nested exception is javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

Comments

[alexK]

thanks for the answer. Unfortunately, that's not my case, I have tls authentication enabled.

[ixe013]

That's kind of strange though... I would expect the contrary: setting tls_disable_client_certs=true would, you know, disable client cert. As in disable=true...

[ixe013]

Thanks a lot @jAC! +For the record I would add tree things. 1) The certificate must have the extended key usage of client authentication (client_flag=true if you generate the certificate with Vault's PKI) and 2) Don't set tls_require_and_verify_client_cert=true in Vault's configuration file if you want "regular" vault calls to work. 3) A https proxy between the client and Vault could snafu the whole thing, test with a direct connection!