Yet another routing issue in OpenVPN - Cannot access local machines while on VPN

Question

This is probably going to be quite a lengthy post because I want to make sure I don't forget anything. This is likely a very trivial issue but I can't figure out what's wrong with my setup.

Scenario

I recently configured a VPN server running OpenVPN, currently running inside a VM in ESXi. I have based the installation on this guide, and everything seems to be working fine: I can connect to my VPN server (through port forwarding on the router) and login with certificate authentication.

What is not working

I cannot access machines on my local network (192.168.0.0) when I am connected through the VPN.

At the moment, after successfully connecting via VPN to my home network, all the machines on my LAN are not accessible (I cannot even ping them).

Network information

The network configuration is very simple:

• My local network: 192.168.0.0
• My OpenVPN IP: 192.168.0.140
• My gateway: 192.168.0.1
• My VPN network: 10.8.0.0
• My OpenVPN VPN IP: 10.8.0.1
• The router has port forwarding configured on port 1194

What I have tried so far (high level)

I have followed three resources to try and fix this issue:

1. The OpenVPN routing guide, specifically the section Using routing and OpenVPN not running on the default gateway
2. This short article on how to set up routing with OpenVPN to connect to hosts on my LAN
3. Another article on how to configure ESXi for this to work, because apparently, in its default configuration, ESXi can cause some problems getting this to work
4. Enabled IP forwarding on the VPN server
5. Enabled all the traffic to go through OpenVPN's firewall

What I have tried so far (detailed)

This is the OpenVPN server.conf:

local 192.168.0.140
topology subnet
dev tun
proto udp
port 1194

ca /etc/openvpn/easy-rsa/keys/ca.crt
cert /etc/openvpn/easy-rsa/keys/vpnserver.crt
key /etc/openvpn/easy-rsa/keys/vpnserver.key
dh /etc/openvpn/easy-rsa/keys/dh1024.pem

server 10.8.0.0 255.255.255.0
# server and remote endpoints
ifconfig 10.8.0.1 10.8.0.2

# Add route to Client routing table for the OpenVPN Server
push "route 10.8.0.1 255.255.255.255"
# Add route to Client routing table for the OpenVPN Subnet
push "route 10.8.0.0 255.255.255.0"
# your local subnet
push "route 192.168.0.0 255.255.255.0"
# Set primary domain name server address to the SOHO Router
# If your router does not do DNS, you can use Google DNS 8.8.8.8
push "dhcp-option DNS 192.168.0.254"
# Override the Client default gateway by using 0.0.0.0/1 and
# 128.0.0.0/1 rather than 0.0.0.0/0. This has the benefit of
# overriding but not wiping out the original default gateway.
push "redirect-gateway def1"

client-to-client
duplicate-cn
keepalive 10 120
tls-auth /etc/openvpn/easy-rsa/keys/ta.key 0
cipher AES-128-CBC
comp-lzo
user nobody
group nogroup
persist-key
persist-tun
status /var/log/openvpn-status.log 20
log /var/log/openvpn.log
verb 1

An example client.ovpn:

client
dev tun
proto udp
remote <my_router_ip> 1194
resolv-retry infinite
nobind
persist-key
persist-tun
mute-replay-warnings
ns-cert-type server
key-direction 1
cipher AES-128-CBC
comp-lzo
verb 1
mute 20

Since my current router does not support static routes, unfortunately I cannot configure routing once and for all, but I have do so on each device I want to access while I am connected via VPN. So for example, on my Plex server (192.168.0.110), I have done this:

route add -net 10.8.0.0/24 gw 192.168.0.140

So that my OpenVPN machine (192.168.0.140) could connect the machines on the VPN network (10.8.0.0) with those on the local network (192.168.0.0).

I have also made sure that IP forwarding is enabled on the OpenVPN server, but uncommenting the following line from /etc/sysctl.conf:

net.ipv4.ip_forward=1

Finally, I have added the following iptables rules to allow all traffic from the VPN network to reach the local network:

# Allow traffic initiated from VPN to access LAN
iptables -I FORWARD -i tun0 -o eth0 -s 10.8.0.0/24 -m conntrack --ctstate NEW -j ACCEPT

# Allow established traffic to pass back and forth
iptables -I FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Finally, I have configured the virtual switch vSwitch on ESXi not to drop promiscuous packets, as the article linked above states that in its default configuration it would drop these packets, and this could cause issues when using OpenVPN while trying to access machines on the local network.

Unless I have missed something, this should really be all that needs to be done. Unfortunately, after successfully connecting to the VPN, I am still not able to ping the machines on the 192.168.0.0 network.

Again, this is likely something quite trivial, but I am willing to offer some bounty for the correct answer anyway, because one of the reasons I set up the VPN in the first place was to be able to access my other machines without the need to forward more ports on the router.

Client machine networking info

This is the full output of ipconfig of the machine I am using to connect (in order to "fake" connecting from a remote network, I am tethering my phone's 3G connection and connecting via Wi-Fi from Windows):

C:\Windows\System32>ipconfig

Windows IP Configuration


Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::2c0e:13f0:840c:37b4%15
   IPv4 Address. . . . . . . . . . . : 10.8.0.10
   Subnet Mask . . . . . . . . . . . : 255.255.255.252
   Default Gateway . . . . . . . . . :

Wireless LAN adapter Local Area Connection* 2:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

Wireless LAN adapter Wi-Fi:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::a5e8:546c:e046:a246%4
   IPv4 Address. . . . . . . . . . . : 192.168.43.220
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.43.1

Ethernet adapter Ethernet:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

Ethernet adapter VirtualBox Host-Only Network:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::1d14:52cd:fd6a:2395%10
   IPv4 Address. . . . . . . . . . . : 192.168.56.1
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :

Tunnel adapter isatap.{75888664-BED0-4908-8984-4DBCF9E9BDDC}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

Tunnel adapter isatap.{6F952140-AFCD-46E4-89E3-02CDEF869C50}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

Tunnel adapter isatap.{1F6CE10F-8498-4A7B-B647-FAE7422FF030}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

Output of route print:

C:\Windows\System32>route print
===========================================================================
Interface List
 15...00 ff 1f 6c e1 0f ......TAP-Windows Adapter V9
  7...12 56 f2 a5 d0 53 ......Microsoft Wi-Fi Direct Virtual Adapter
  4...80 56 f2 a5 d0 53 ......Killer Wireless-N 1202 Network Adapter
  3...80 fa 5b 00 d7 1f ......Realtek PCIe GBE Family Controller
 10...08 00 27 00 68 59 ......VirtualBox Host-Only Ethernet Adapter
  1...........................Software Loopback Interface 1
  8...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
  9...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #3
 11...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #4
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0     192.168.43.1   192.168.43.220     25
         10.8.0.4  255.255.255.252         On-link          10.8.0.6    276
         10.8.0.6  255.255.255.255         On-link          10.8.0.6    276
         10.8.0.7  255.255.255.255         On-link          10.8.0.6    276
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    306
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    306
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    306
     192.168.43.0    255.255.255.0         On-link    192.168.43.220    281
   192.168.43.220  255.255.255.255         On-link    192.168.43.220    281
   192.168.43.255  255.255.255.255         On-link    192.168.43.220    281
     192.168.56.0    255.255.255.0         On-link      192.168.56.1    276
     192.168.56.1  255.255.255.255         On-link      192.168.56.1    276
   192.168.56.255  255.255.255.255         On-link      192.168.56.1    276
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    306
        224.0.0.0        240.0.0.0         On-link      192.168.56.1    276
        224.0.0.0        240.0.0.0         On-link          10.8.0.6    276
        224.0.0.0        240.0.0.0         On-link    192.168.43.220    281
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    306
  255.255.255.255  255.255.255.255         On-link      192.168.56.1    276
  255.255.255.255  255.255.255.255         On-link          10.8.0.6    276
  255.255.255.255  255.255.255.255         On-link    192.168.43.220    281
===========================================================================
Persistent Routes:
  None

IPv6 Route Table
===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    306 ::1/128                  On-link
 10    276 fe80::/64                On-link
 15    276 fe80::/64                On-link
  4    281 fe80::/64                On-link
 10    276 fe80::1d14:52cd:fd6a:2395/128
                                    On-link
 15    276 fe80::2c0e:13f0:840c:37b4/128
                                    On-link
  4    281 fe80::a5e8:546c:e046:a246/128
                                    On-link
  1    306 ff00::/8                 On-link
 10    276 ff00::/8                 On-link
 15    276 ff00::/8                 On-link
  4    281 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

Solution

Ok, the problem was on the client side. I have no idea what the root cause was, but basically my client would not receive the static routes from the OpenVPN server for some reason. Uninstalling and reinstalling the OpenVPN client solved the issue :/

Therefore, the above configuration is perfectly correct.

Comments

[heavyd]

What is the IP of your client machine? If its on the same network (192.168.0.XXX), it won't forward the traffic to the VPN).

[user1301428]

@heavyd When I am connected to the VPN, I normally get IP 10.8.0.6. Let me update the question with the full output of ipconfig, it might be useful.

[user2313067]

The output of route print on the VPN client might also be useful. Also, what do you get when trying to ping a machine? And have you run sysctl -p or rebooted the server after modifying /etc/sysctl.conf?

[user1301428]

@user2313067 I have added the output of route print, and I confirm that I have also run sysctl -p. I have also double checked running cat /proc/sys/net/ipv4/ip_forward

[harrymc]

Is it possible that you redirected 192.168.0.0/24 into the tunnel with push "route 192.168.0.0 255.255.255.0" ?

[user1301428]

@harrymc This is my first OpenVPN server config so I must admit I might be mistaken in many things, but I thought that command pushed the route to the local network to the VPN clients.. In any case, even commenting it out I see the same behavior...

[harrymc]

Well, my experience is with OpenVPN servers is exactly zero, but I'm interested in the solution, so bear with me. My other theory is that when you talk to other computers on the LAN, the packets do reach them, but as the source address of these packets is from the OpenVPN network, the only thing they know to do is to uselessly send them to the default gateway which is probably the router. The solution might either be to add static routes to all LAN hosts, or use "bridging" instead of "routing". I have never done it, but read this for more info.

[user1301428]

@harrymc That is my understanding as well. The workaround for this is to add a static route (either on the router, which I cannot do unfortunately, or on each LAN machine that I want to access when connected to the VPN). This is what route add -net 10.8.0.0/24 gw 192.168.0.140 should do.

[harrymc]

It's clear that an outside connection thru the router using the VPN wouldn't be able to see the local network segment because of its IP address. I think you should either (1) Put VPN and local network on the same 192.168.43.xxx segment by carefully limiting the DHCP IP-ranges allocated by both the router and VPN server, or (2) Use OpenVPN bridging for both segments as in my above link, or (3) Use only VPN static IP addresses for all devices in the local network.

[user1301428]

@harrymc Correct, and I agree that these 3 methods will work. Also, putting the VPN server on the router would also solve this. The fact is, I want to solve this specific issue with routing because I want to learn what I have done wrong. I think I have read all online threads from people that had the same issue, tried all fixes but obviously there is still something that is missing and I can't for the life of me figure out what. I think I might just give up, forget about this and look for my next puzzle to solve. Or I could put the bounty to 500, maybe that would help too.

[harrymc]

See this article for using iptables for doing the NAT.

[user1301428]

@harrymc yep, that's one of the articles I linked in the "What I have tried so far section" :P

[harrymc]

It seems you have well-researched this problem. There might even not be any error in what you did, just some bug/gotcha in the software you use. I don't think that raising the bounty will make much of a difference. Waiting and later re-posting another bounty, if unsolved, might be more effective, or maybe on another StackExchange site. Or it might be practical to use another approach, maybe such as I have outlined above, just to make it work.

[user1301428]

@harrymc yeah, I will definitely look at bridging as an alternative. It would be a pity really, not to know what is wrong :) I appreciate your help though!

[heavyd]

@user1301428, your routing table on the client does not include any entries for the LAN subnet (in fact none of the 3 routes from the server config were actually pushed), so those packets are not going through the VPN. Perhaps check the openVPN logs on the client and sever to see if you can see why those routes were not configured.

[user1301428]

@heavyd hah, good catch, I will look into that. But does this mean that, when connected to the VPN, my packets are just going through my mobile gateway and are not being encrypted?

[heavyd]

Yes,but they're likely being dropped by your mobile provider since they're non-routable IPs.

[user1301428]

@heavyd what about browsing? I mean, browsing works just fine and I am getting my home public IP address, but is that traffic also not going through the VPN?

[heavyd]

Nope. Check your ip: checkip.net. If you're going through the VPN, it should be the same as a machine on your LAN. I'm betting it's not the same.

[user1301428]

@heavyd yes, when I am connected to the VPN I get my home router's public IP.

[heavyd]

And your routing table hasn't changed since you posted it? The posted table has your default gateway as the WiFi gateway, with no other special routes.

[user1301428]

@heavyd this is interesting: I am currently on another network and I can access my local machines. The routing table is indeed correct now (the routes are all being pushed - with the exception of the 192.168.0.0 route which is not there for some reason, but I do have both the 10.8.0.0 routes). At this point I am wondering whether my mobile operator does not allow using non-default routes?

[heavyd]

Those are local routes, you mobile operator should not even know you have them. It is interesting though that it is working through another network. Am assuming you're using the same client machine?

[user1301428]

@heavyd no, I am on another machine at the moment. Do you think it could be a problem with the machine?

[heavyd]

@user1301428, I do think it has something to do with your machine. I was able to recreate your scenario successfully (so none of the same problems you experienced) using near identical configuration files. Did you ever check the client log files? Perhaps setting the verb 3 instead of 1 might give more detail.

[user1301428]

@heavyd what are the permissions on your OpenVPN folder? I just realized that my client is not writing any logs, and in the connection phase I see a warning saying that it is not able to open my user.log file.

[user1301428]

@heavyd another interesting discovery: I am able to access all of the devices on the LAN (from the other machine, that is), even those on which I haven't manually added a static route. It was my understanding that this was necessary if OpenVPN wasn't running on the network gateway. Wizardry.

[heavyd]

@user1301428, yes, that is interesting. In my setup I had to add those routes manually in order to make it work. Did you ever get the original machine working?

[user1301428]

@heavyd nope. I should try using the same machine but on a different network. My gut is telling me that this might be an issue with the 3G connection. Sometimes (and only when I am on 3G), I successfully connect to the VPN but cannot even get the correct IP.

[heavyd]

@user1301428, that is strange. I actually tried the exact same setup, using my cell as a Wi-Fi hotspot for my laptop using OpenVPN. Everything worked great for me, so I'm kinda stumped.

[heavyd]

Let us continue this discussion in chat.

[user1301428]

@heavyd sorry, I thought I would get a notification when a message was added to the chat, but apparently this is not the case. Just so you know, I have resolved the issue; thank you very much for your help ;)

[heavyd]

@user1301428, welp, glad you got it working :)

[Mahdi]

Yes! But isnt that what you want (local traffic not going through VPN).

[user1301428]

I am sure split tunneling would make me reach my goal, but that is not how I would like to do it. I still want all of my traffic to go through my VPN server, but I also want to access a different network, and here's where routing is involved.

[Mahdi]

Split tunnling is basically tweaking your routing. It just does not make sense to send your lical traffic through VPN simply b/c the other end of the tunnel has no idea about you local network. You still send all outbounding traffic through VPN.

[user1301428]

"the other end of the tunnel has no idea about you local network", that's right, and that's once again where I need routing to work :)

[Mahdi]

Do you agree that your local traffic ever end up in the tunnel? Why you should congest your outgoing link with traffic?

[user1301428]

Please clarify: is "local traffic" that on the 192.168.0.0 network? What's the outgoing link you are referring to? Have a look at this article linked above to understand what I mean: blog.remibergsma.com/2013/01/13/…

[Ramhound]

So provide the detailed solution for those that don't know what it is