OpenVPN server on Kubernetes cluster / DNS and Service resolution

Question

I have a Kubernetes cluster running applications (currently on a set of Vagrant CoreOS VMs on a local server)

I want to be able to debug a particular application locally on my laptop, so I worked on setting up VPN into the cluster: a client/server VPN based on kylemanna/docker-openvpn, deployed as a regular Pod

I created the cert/key pairs, client certs etc... I can connect to the VPN fine.

Now, connecting to the VPN server doesn't get me much if I can't access the Services. I have the DNS addon running skyDNS in the cluster. I can nslookup my services from other pods in the cluster, so all that works fine, but I can't resolve Services by name on the VPN client. I can ping Pods by IP from the VPN client (in the subnet 10.2.0.0/16) but I can't resolve with DNS

a nslookup from the client returns:

nslookup myservice 10.3.0.10
Server:     10.3.0.10
Address:    10.3.0.10#53

** server can't find myservice: SERVFAIL

One of the problems of troubleshooting is that neither ping nor traceroute work on the DNS IP (from any pod), yet it resolves services, so nslookup is the way I know to check, but that is not very informative.

The VPN host IP the Pod binds to is 192.168.10.152 The Kubernetes subnet is 10.2.0.0/16 The SkyDNS server is at 10.3.0.10

The VPN server subnet is 10.8.0.0/24 On the VPN server ifconfig gives:

eth0      Link encap:Ethernet  HWaddr 02:42:0A:02:16:45
          inet addr:10.2.22.69  Bcast:0.0.0.0  Mask:255.255.255.0
tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:10.8.0.1  P-t-P:10.8.0.2  Mask:255.255.255.255

So 10.2.22.69 is the Pod IP and the VPN Server IP is 10.8.0.1 with the Gateway being 10.8.0.2 i guess.

On the VPN server pod the routign table looks like:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.2.22.1       0.0.0.0         UG    0      0        0 eth0
10.2.22.0       *               255.255.255.0   U     0      0        0 eth0
10.8.0.0        10.8.0.2        255.255.255.0   UG    0      0        0 tun0
10.8.0.2        *               255.255.255.255 UH    0      0        0 tun0
192.168.254.0   10.8.0.2        255.255.255.0   UG    0      0        0 tun0

I can reach my applications by IP (and get data) but couldn't when using the service IP (the proxy IP which is on the 10.3.0.0 subnet)

I added the route route add -net 10.3.0.0/16 gw 10.8.0.2 to the VPN Server and I can then use the Service IP to get data but the nslookup just times out then.

I guess the traffic may not be coming back from the DNS. DNS is itself a proxied service in Kubernetes, so that adds a level of complexity. Not sure how to fix this.

Solution 1

finally my config looks like this:

docker run -v /etc/openvpn:/etc/openvpn --rm kylemanna/openvpn ovpn_genconfig \
-u udp://192.168.10.152:1194 \
-n 10.3.0.10 \
-n 192.168.10.1 \
-n 8.8.8.8 \
-n 75.75.75.75 \
-n 75.75.75.76 \
-s 10.8.0.0/24 \
-N \
-p "route 10.2.0.0 255.255.0.0" \
-p "route 10.3.0.0 255.255.0.0" \
-p "dhcp-option DOMAIN-SEARCH cluster.local" \
-p "dhcp-option DOMAIN-SEARCH svc.cluster.local" \
-p "dhcp-option DOMAIN-SEARCH default.svc.cluster.local"

-u for the VPN server address and port

-n for all the DNS servers to use

-s to define the VPN subnet (as it defaults to 10.2.0.0 which is used by Kubernetes already)

-d to disable NAT

-p to push options to the client

-N to enable NAT: it seems critical for this setup on Kubernetes

the last part, pushing the search domains to the client, was the key to getting nslookup etc.. to work.

note that curl didn't work at first, but seems to start working after a few seconds. So it does work but it takes a bit of time for curl to be able to resolve.

Solution 2

Ran into the same issue a year ago and decided to make something specific for kubernetes: https://github.com/pieterlange/kube-openvpn

(based on kylemanna's openvpn docker image)

Comments

[phk]

Could you please add some additional information about it here, e.g. in case the link stops working.