kubectl: The connection to the server XXX.XXX.XXXXXX was refused

Question

I am trying to connect to kubernetes master (cluster) in Google Cloud Engine.

The error that always I get when the kubectl try to access to kubernetes master is:

The connection to the server XXX.XXX.XXX.XXX was refused - did you specify the right host or port?

For example:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server XXX.XXX.XXX.XXX was refused - did you specify the right host or port?

As far I check the client is using same version that the server (version 1.5.2). But for some weird reason, it is refusing to connect.

$ gcloud beta container get-server-config
Fetching server config for europe-west1-c
defaultClusterVersion: 1.5.2
defaultImageType: GCI
validImageTypes:
- CONTAINER_VM
- GCI
validMasterVersions:
- 1.5.2
- 1.4.8
validNodeVersions:
- 1.5.2
- 1.5.1
- 1.4.8
- 1.4.7
- 1.4.6
- 1.3.10
- 1.2.7

In kubernetes master cluster (server version) I get the following error:

# kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

I follow this steps for the kubernetes cluster master creation:

export APP_NAME=brand-project
export GOOGLE_CONTAINER_NAME=b.gcr.io/brand/project
gcloud container clusters create $APP_NAME --zone europe-west1-c --machine-type g1-small --num-nodes 1

I get and I set perfectly the credentials:

gcloud config set container/cluster $APP_NAME
gcloud container clusters get-credentials $APP_NAME
gcloud auth application-default login

The description is good:

gcloud container clusters describe $APP_NAME

The google config too:

gcloud config list

The context seem legit too in:

kubectl config get-contexts

Even I can ssh to kubernetes master cluster, but only SSH, no HTTP or HTTPS or for example run properly kubectl.

I read too in Kubernetes docs:

Google Container Engine uses SSH tunnels to protect the Master -> Cluster communication paths. In this configuration, the apiserver initiates an SSH tunnel to each node in the cluster (connecting to the ssh server listening on port 22) and passes all traffic destined for a kubelet, node, pod, or service through the tunnel. This tunnel ensures that the traffic is not exposed outside of the private GCE network in which the cluster is running.

So I don't know how to open the 8000 port in Kubernetes Cluster mastter for allow the connection (and opening all the ports in firewall in Google Cloud Engine seems not work too).

I am out of ideas, and I mostly search all google related entries. So I don't have idea how to solve to connect with the server or what I am doing wrong in the process. Any help is very appreciated!

EDIT:

After check "Container Registry Deprecation Notices" the container location was updated to eu.gcr.io instead b.gcr.io according to:

On February 28th, 2017, the use of “bring your own bucket” registries such as b.gcr.io and bucket.gcr.io is considered deprecated. After that date, Container Registry will no longer serve any container images that you had in those buckets.

But the issue still persist.

Solution

Resolving my own answer. It seems that the real problem was access and connecting to accounts.google.com via DNS. After I check that I have ping:

$ ping accounts.google.com
PING accounts.google.com (216.58.201.141) 56(84) bytes of data.
64 bytes from mad06s25-in-f13.1e100.net (216.58.201.141): icmp_seq=1 ttl=56 time=21.9 ms
64 bytes from mad06s25-in-f13.1e100.net (216.58.201.141): icmp_seq=2 ttl=56 time=19.0 ms
64 bytes from mad06s25-in-f13.1e100.net (216.58.201.141): icmp_seq=3 ttl=56 time=20.4 ms
^C
--- accounts.google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 19.070/20.468/21.914/1.173 ms

And stracing all the opened files during the command:

$ strace -eopenat kubectl version
openat(AT_FDCWD, "/proc/sys/net/core/somaxconn", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/proc/stat", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/proc/sys/kernel/hostname", O_RDONLY|O_CLOEXEC) = 3
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
openat(AT_FDCWD, "/home/shakaran/.kube/config", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/home/shakaran/.config/gcloud/application_default_credentials.json", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/proc/sys/kernel/hostname", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
The connection to the server 104.155.120.114 was refused - did you specify the right host or port?
+++ exited with 1 +++

I try to figure out the opened connections:

$ systemd-resolve --status | cat
Global
         DNS Servers: 127.0.1.1
                      8.8.8.8
                      8.8.4.4
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 10 (vboxnet3)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 9 (vboxnet2)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 8 (vboxnet1)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 7 (vboxnet0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 6 (docker0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 5 (tun0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 3 (wlan0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: no
         DNS Servers: 8.8.8.8
                      8.8.4.4

Link 2 (eth0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

I just discover that I have it the openvpn with tun0 enabled (blocking the connection to accounts.google.com), after I run the disable of the interface:

sudo ifconfig tun0 down

I get perfectly:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
So sorry for all the noise. But probably it is a good idea add this in FAQ's or so for warning the users about VPNs

So the issue was mostly a refused connection. It could be useful the issue #41975 in kubernetes project for debug with the -v=4 like:

$ kubectl version -v=4
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
I0224 11:32:36.914299   30751 helpers.go:221] Connection error: Get https://XXX.XXX.XXX.XXX/api: Post https://accounts.google.com/o/oauth2/token: dial tcp: lookup accounts.google.com on 127.0.1.1:53: read udp 127.0.0.1:46403->127.0.1.1:53: read: connection refused
F0224 11:32:36.914378   30751 helpers.go:116] The connection to the server XXX.XXX.XXX.XXX was refused - did you specify the right host or port?