How do I use kubectl via an ssh tunnel to access my internal ELB over my kubernetes API?

Question

I want to create a kubernetes cluster using KOPS that uses a private topology completely (all master/worker nodes are on private subnets, the API ELB is internal).

Once the cluster is created - how do I configure kubectl to use an ssh tunnel via my bastion server ?

Solution 1

I found a way to make kubectl to run through an SSH tunnel, it's not ideal, but until I find something better, I posted it now.

First create the tunnel:

ssh -f [email protected] -L 6443:localhost:6443 -N

Then copy the ~/.kube/config file on your local machine and change the cluster server in order to point to 127.0.0.1 instead of the server URL or IP address.

As the certificates are made for the server where the master node has been created, you'll get the following error:

Unable to connect to the server: x509: certificate is valid for 10.96.0.1, 10.0.0.1, not 127.0.0.1

You have to pass the --insecure-skip-tls-verify=true flag:

kubectl --insecure-skip-tls-verify=true version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:09:21Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

I hope this helps, and I hope to find a better way to avoid this --insecure-skip-tls-verify=true flag.

---

Update

Since my comment, I found the Teleport project from Gravitational, which was initially an SSH tool to authenticate without passwords (you login once, with an OTP, and a certificate with a validity limited in time for your user is delivered and used to authenticated to the allowed servers.), is also Kubernetes compatible.

Basically you have to :

1. deploy their binary and configure it (quite easy).
2. login using tsh login --proxy https://yourserveripaddress:3080
3. use kubectl to access your cluster.

The magic thing here is that Teleport will update your ~/.kube/config file in order to access your cluster.

It really works well and you should consider giving it a try.

In the case you're using Chef, I have made a cookbook for Teleport.

Solution 2

You can just use VPN over SSH what will be transparent for your kubectl, example tool: https://github.com/sshuttle/sshuttle which create VPN tunnel using SSH and iptables

Requirement is to have at least python 2.3 on bastion host.

Comments

[Andrew Shatnyy]

Adding yet another service on the box doesn't answer the question.

[weisjohn]

you can specify the insecure-skip-tls-verify: true option under the cluster settings in the ~/.kube/config