"x509: certificate signed by unknown authority" when running kubelet

10,656

Solution 1

in general the solution was to create another etcd2 port that attaches to loopback device of each machine and works on http instead of https. more information at calico-policy-controller requests etcd2 certificates of a different coreos server

Solution 2

The kubelet documentation says that the --tls-cert-file flag needs the CA be concatenated after the certificate. In you case it is the apiserver.pem:

--tls-cert-file File containing x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to --cert-dir.

If I read you certificate generation correctly, the apiserver.pem doesn't contain the root ca.

Share:
10,656

Related videos on Youtube

ufk
Author by

ufk

Programmer, Administrator, Gamer and Musician

Updated on June 04, 2022

Comments

  • ufk
    ufk over 1 year

    I'm trying to install kubernetes with kubelet 1.4.5 on CoreOS beta (1192.2.0).

    I'm using a slightly modified version of the controller and worker install scripts from https://github.com/coreos/coreos-kubernetes/tree/master/multi-node/generic

    so in general I created the licenses on Gentoo Linux using the following bash script:

    #!/bin/bash
    export MASTER_HOST=coreos-2.tux-in.com
    export K8S_SERVICE_IP=10.3.0.1
    export WORKER_IP=10.79.218.3
    export WORKER_FQDN=coreos-3.tux-in.com
    openssl genrsa -out ca-key.pem 2048
    openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=kube-ca"
    openssl genrsa -out apiserver-key.pem 2048
    openssl req -new -key apiserver-key.pem -out apiserver.csr -subj "/CN=kube-apiserver" -config openssl.cnf
    openssl x509 -req -in apiserver.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out apiserver.pem -days 365 -extensions v3_req -extfile openssl.cnf
    openssl genrsa -out ${WORKER_FQDN}-worker-key.pem 2048
    openssl req -new -key ${WORKER_FQDN}-worker-key.pem -out ${WORKER_FQDN}-worker.csr -subj "/CN=${WORKER_FQDN}" -config worker-openssl.cnf
    openssl x509 -req -in ${WORKER_FQDN}-worker.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out ${WORKER_FQDN}-worker.pem -days 365 -extensions v3_req -extfile worker-openssl.cnf
    openssl genrsa -out admin-key.pem 2048
    openssl req -new -key admin-key.pem -out admin.csr -subj "/CN=kube-admin"
    openssl x509 -req -in admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out admin.pem -days 365
    echo done
    

    and this is openssl.cnf

    [req]
    req_extensions = v3_req
    distinguished_name = req_distinguished_name
    [req_distinguished_name]
    [ v3_req ]
    basicConstraints = CA:FALSE
    keyUsage = nonRepudiation, digitalSignature, keyEncipherment
    subjectAltName = @alt_names
    [alt_names]
    DNS.1 = coreos-2.tux-in.com
    DNS.2 = coreos-3.tux-in.com
    IP.1 = 10.3.0.1
    IP.2 = 10.79.218.2
    IP.3 = 10.79.218.3
    

    and this is my worker-openssl.cnf

    [req]
    req_extensions = v3_req
    distinguished_name = req_distinguished_name
    [req_distinguished_name]
    [ v3_req ]
    basicConstraints = CA:FALSE
    keyUsage = nonRepudiation, digitalSignature, keyEncipherment
    subjectAltName = @alt_names
    [alt_names]
    IP.1 = 10.79.218.3
    DNS.1 = coreos-3.tux-in.com
    

    My controller machine is coreos-2.tux-in.com which resolves to the lan ip 10.79.218.2

    my worker machine is coreos-3.tux-in.com which resolves to lan ip 10.79.218.3

    it created the licenses just fine. but when I use them and install the controller script on the main machine, i see that when I run journalctl -xef -u kubelet and I noticed the following messages:

    Nov 08 21:24:06 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:06.805868    2018 event.go:208] Unable to write event: 'x509: certificate signed by unknown authority' (may retry after sleeping)
    Nov 08 21:24:06 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:06.950827    2018 reflector.go:203] pkg/kubelet/kubelet.go:384: Failed to list *api.Service: Get https://coreos-2.tux-in.com:443/api/v1/services?resourceVersion=0: x509: certificate signed by unknown authority
    Nov 08 21:24:07 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:07.461042    2018 reflector.go:203] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get https://coreos-2.tux-in.com:443/api/v1/pods?fieldSelector=spec.nodeName%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
    Nov 08 21:24:07 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:07.461340    2018 reflector.go:203] pkg/kubelet/kubelet.go:403: Failed to list *api.Node: Get https://coreos-2.tux-in.com:443/api/v1/nodes?fieldSelector=metadata.name%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
    Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.024366    2018 reflector.go:203] pkg/kubelet/kubelet.go:384: Failed to list *api.Service: Get https://coreos-2.tux-in.com:443/api/v1/services?resourceVersion=0: x509: certificate signed by unknown authority
    Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.171170    2018 eviction_manager.go:162] eviction manager: unexpected err: failed GetNode: node '10.79.218.2' not found
    Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.543619    2018 reflector.go:203] pkg/kubelet/kubelet.go:403: Failed to list *api.Node: Get https://coreos-2.tux-in.com:443/api/v1/nodes?fieldSelector=metadata.name%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
    Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.543926    2018 reflector.go:203] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get https://coreos-2.tux-in.com:443/api/v1/pods?fieldSelector=spec.nodeName%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
    
  • ufk
    ufk almost 7 years
    thanks for your answer. i appended the ca to the apiserver.pem file but the results are exactly the same.
  • ufk
    ufk almost 7 years
    hi. thanks for your answer. i closed the worker server and I still get the same error messages. it means that it doesn't really relate to the worker configuration no ?
  • Yves Blusseau
    Yves Blusseau almost 7 years
    I think you have invert the worker/master dns name and the worker/master ip. You have written that the worker is coreos-3.tux-in.com with ip 10.79.218.3 (in the bash script to generate the certificates) but after you have written: "my worker machine is coreos-2.tux-in.com which resolves to lan ip 10.79.218.3"
  • ufk
    ufk almost 7 years
    hi. thanks for your answer. so the problem is that they're self signed? if i want to keep using self signed certificates, what do i do? how do i get rid of this error ?
  • stephanlindauer
    stephanlindauer almost 7 years
    what do you mean by append?