Kubernetes ingress "an error on the server ("") has prevented the request from succeeding"

Question

I have a managed azure cluster (AKS) with nginx ingress in it. It was working fine but now nginx ingress stopped:

# kubectl -v=7 logs nginx-ingress-<pod-hash> -n nginx-ingress
GET https://<PRIVATE-IP-SVC-Kubernetes>:443/version?timeout=32s
I1205 16:59:31.791773       9 round_trippers.go:423] Request Headers:
I1205 16:59:31.791779       9 round_trippers.go:426]     Accept: application/json, */*
Unexpected error discovering Kubernetes version (attempt 2): an error on the server ("") has prevented the request from succeeding

# kubectl describe svc kubernetes
Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP:                <PRIVATE-IP-SVC-Kubernetes>
Port:              https  443/TCP
TargetPort:        443/TCP
Endpoints:         <PUBLIC-IP-SVC-Kubernetes>:443
Session Affinity:  None
Events:            <none>

When I tried to curl https://PRIVATE-IP-SVC-Kubernetes:443/version?timeout=32s, I've always seen the same output:

curl: (35) SSL connect error

Solution 1

I faced the same issue with three manager cluster and I was accessing it through ucp client bundle. I figured out 2 out of 3 manager nodes were in not ready state. On debugging further I found space issue on those not ready boxes. After cleaning a little (mainly /var folder) and restart docker, those nodes came to ready state and I'm not getting this error.

Solution 2

On my OCP 4.7 (OpenShift Container Registry) instances with 3 of master and 2 of worker nodes, the following log appears after kubelet and oc commands.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1-5-g76a04fc", GitCommit:"e29b355", GitTreeState:"clean", BuildDate:"2021-06-03T21:19:58Z", GoVersion:"go1.15.7", Compiler:"gc", Platform:"linux/amd64"}
Error from server (InternalError): an error on the server ("") has prevented the request from succeeding
$ oc get nodes
Error from server (InternalError): an error on the server ("") has prevented the request from succeeding

Also, when I wanted to login to the OCP dashboard, the following error occurred:

error_description": "The authorization server encountered an unexpected condition that prevented it from fulfilling the request

I restarted the whole master node machines then the problem solved.