Openvpn, forward packets very slowly

Question

I rebooted my server, and an odd issue just came out. I am running on ArchLinux, the clients are Ubuntu, Android and Mac.

The problem is that accessing the internet via the clients is slow, about 2ko/s and slowly stop.

But downloading something from the server to the client directly is made at full speed. And, obviously, internet from the server is at his full speed (40mo/s).

I don't know what happened from the reboot, but this issue is here on all clients, and is only related to the traffic that openvpn forward to internet.

EDIT: Tried with tcp, did not solve. EDIT: Tested various fragment/mtu settings, no changes.

Here are all my confs:

╭─<root@Alduin>-</etc/openvpn>-<1:45:07>-◇
╰─➤ cat Alduin.conf ccd/Thunderaan
local 212.83.129.104
port 1194
proto udp
dev tun
ca keys/ca.crt
cert keys/Alduin.crt
key keys/Alduin.key
dh keys/dh1024.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "dhcp-option DNS 10.8.0.1"
client-to-client
keepalive 5 60
ping-timer-rem
comp-lzo
persist-key
persist-tun
status openvpn-status.log
verb 3
client-config-dir ccd
topology subnet

ccd from here +++++++++++++++


ifconfig-push 10.8.0.2 255.255.255.0
push "redirect-gateway def1"

Client conf:

client
dev tun
proto udp
remote 212.83.129.104 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert name.crt
key name.key
ns-cert-type server
comp-lzo
verb 3

and some output that might help you:

╭─<cubox@Alduin>-<~>-<1:49:43>-◇
╰─➤ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether b8:ac:6f:94:e2:4e brd ff:ff:ff:ff:ff:ff
    inet 88.190.15.135/24 scope global eno1
       valid_lft forever preferred_lft forever
    inet 212.83.129.104/32 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 2001:bc8:300a:dead::b12d/64 scope global
       valid_lft forever preferred_lft forever
    inet6 2a01:e0b:1000:15:baac:6fff:fe94:e24e/64 scope global dynamic
       valid_lft 2592000sec preferred_lft 604800sec
    inet6 fe80::baac:6fff:fe94:e24e/64 scope link
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether b8:ac:6f:94:e2:4f brd ff:ff:ff:ff:ff:ff
6: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
    link/none
    inet 10.8.0.1/24 brd 10.8.0.255 scope global tun0
       valid_lft forever preferred_lft forever
╭─<cubox@Alduin>-<~>-<1:49:47>-◇
╰─➤ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         88-190-15-1.rev 0.0.0.0         UG    0      0        0 eno1
10.8.0.0        *               255.255.255.0   U     0      0        0 tun0
88.190.15.0     *               255.255.255.0   U     0      0        0 eno1
╭─<cubox@Alduin>-<~>-<1:49:51>-◇
╰─➤ route -6
Kernel IPv6 routing table
Destination                    Next Hop                   Flag Met Ref Use If
::1/128                        ::                         U    256 0     0 lo
2001:bc8:300a:dead::/64        ::                         U    256 0     0 eno1
2a01:e0b:1000:15::/64          ::                         UAe  256 0     0 eno1
fe80::/64                      ::                         U    256 0     0 eno1
::/0                           fe80::225:45ff:fef6:947f   UGDAe 1024 2     0 eno1
::/0                           ::                         !n   -1  1  1891 lo
::1/128                        ::                         Un   0   2  5227 lo
2001:bc8:300a:dead::/128       ::                         Un   0   1     0 lo
2001:bc8:300a:dead::b12d/128   ::                         Un   0   1   131 lo
2a01:e0b:1000:15::/128         ::                         Un   0   1     0 lo
2a01:e0b:1000:15:baac:6fff:fe94:e24e/128 ::                         Un   0   3 29356 lo
fe80::/128                     ::                         Un   0   1     0 lo
fe80::baac:6fff:fe94:e24e/128  ::                         Un   0   1   311 lo
ff00::/8                       ::                         U    256 0     0 eno1
::/0                           ::                         !n   -1  1  1891 lo



-A POSTROUTING -s 10.8.0.0/24 -o eno1 -j MASQUERADE # The iptables rule

The iptables rule here is the only that is active on the server.

╰─➤ tc qd
qdisc mq 0: dev eno1 root
qdisc pfifo_fast 0: dev tun0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1

EDIT: Here is a log from the Archlinux client connecting.

Oct  2 16:54:17 Groat ovpn-openvpn[9216]: OpenVPN 2.2.1 x86_64-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [eurephia] [MH] [PF_INET6] [IPv6 payload 20110424-2 (2.2RC2)] built on Feb 13 2013
Oct  2 16:54:17 Groat ovpn-openvpn[9216]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Oct  2 16:54:17 Groat ovpn-openvpn[9216]: LZO compression initialized
Oct  2 16:54:17 Groat ovpn-openvpn[9216]: Control Channel MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:0 ]
Oct  2 16:54:17 Groat ovpn-openvpn[9216]: Socket Buffers: R=[212992->131072] S=[212992->131072]
Oct  2 16:54:17 Groat ovpn-openvpn[9216]: Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Oct  2 16:54:17 Groat ovpn-openvpn[9216]: Local Options hash (VER=V4): '41690919'
Oct  2 16:54:17 Groat ovpn-openvpn[9216]: Expected Remote Options hash (VER=V4): '530fdded'
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: UDPv4 link local: [undef]
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: UDPv4 link remote: [AF_INET]212.83.129.104:1194
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: TLS: Initial packet from [AF_INET]212.83.129.104:1194, sid=edfcb034 3452d72c
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: VERIFY OK: depth=1, /C=FR/ST=FR/L=Paris/O=Dragonborn/CN=Dragonborn_CA/[email protected]
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: VERIFY OK: nsCertType=SERVER
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: VERIFY OK: depth=0, /C=FR/ST=FR/L=Paris/O=Dragonborn/CN=Dragonborn/[email protected]
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: Data Channel Encrypt: Cipher 'BF-CBC' initialized with 128 bit key
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: Data Channel Decrypt: Cipher 'BF-CBC' initialized with 128 bit key
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 1024 bit RSA
Oct  2 16:54:17 Groat ovpn-openvpn[9217]: [Dragonborn] Peer Connection Initiated with [AF_INET]212.83.129.104:1194
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: SENT CONTROL [Dragonborn]: 'PUSH_REQUEST' (status=1)
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: PUSH: Received control message: 'PUSH_REPLY,dhcp-option DNS 10.8.0.1,route 212.83.129.0 255.255.255.0,route-gateway 10.8.0.1,topology subnet,ping 5,ping-restart 60,redirect-gateway def1,ifconfig 10.8.0.3 255.255.255.0'
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: OPTIONS IMPORT: timers and/or timeouts modified
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: OPTIONS IMPORT: --ifconfig/up options modified
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: OPTIONS IMPORT: route options modified
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: OPTIONS IMPORT: route-related options modified
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: ROUTE default_gateway=192.168.1.254
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: TUN/TAP device tun0 opened
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: TUN/TAP TX queue length set to 100
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: /sbin/ifconfig tun0 10.8.0.3 netmask 255.255.255.0 mtu 1500 broadcast 10.8.0.255
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: /sbin/route add -net 212.83.129.104 netmask 255.255.255.255 gw 192.168.1.254
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: /sbin/route add -net 0.0.0.0 netmask 128.0.0.0 gw 10.8.0.1
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: /sbin/route add -net 128.0.0.0 netmask 128.0.0.0 gw 10.8.0.1
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: /sbin/route add -net 212.83.129.0 netmask 255.255.255.0 gw 10.8.0.1
Oct  2 16:54:20 Groat ovpn-openvpn[9217]: Initialization Sequence Completed

EDIT: Here is a tcpdump of the server downloading directly a file: http://sprunge.us/aaJX Here is the client downloading this ressource: http://sprunge.us/WUCC and here is a normal client from another openvpn (working) server: http://www4.slashusr.com/57552.tcpdump

EDIT: As asked in comments, here are raw tcpdump captures. The tun0 capture from the server failed, I don't know why. Server showing outside here, client showing tun0 here, client showing outside here and server downloading directly the file here.

EDIT: The server is running an i3, which is not used at anytime (not even during openvpn use). Same for the client, i7 totaly idle.

EDIT: The issue is still here. Please, help :(

Solution

I am not sure if it is same cause but I think it worth to adjust about tun-mtu and mssfix as mentioned in openvpn-on-android-tcp-retransmissions-after-openvpn-server-reboot

EDIT: I found this one might be helpful too [RESOLVED] Inacceptable openVPN performance Changing a kernel parameter: net.inet.ip.fastforwarding = 1 (add in /etc/sysctl.conf on the your linux server)

Comments

[Zoredache]

I assume you have have looked at some captures with wireshark/tcpdump? The answer almost certainly can be found in a capture, if you capture in the right place.

[Cubox]

I have a tcpdump from the eno1 interface on a download from the client and one from the server (of the same file). And one from a working openvpn client too. I'll edit the question.

[YwH]

Can you add cpu information from the client and server while the traffic is being transferred?

[ott--]

In your tcpdump, I don't see slow traffic (it might be too short tho). Every client gets the same ip-address 10.8.0.2? Can you omit that and rather push a route to your network 212.83.129.0?

[Cubox]

Each client have his own ccd with his own ip address. I don't understand what do you mean by a route to the network.

[ott--]

In the server-config: push "route 212.83.129.0 255.255.255.0". Can you add a log from a client connecting?

[Cubox]

@ott-- Added the route, no changes. Added log.

[YwH]

Can you upload the tcpdump files in binary tcpdump format? This will make it easier to analyze in a tool such as wireshark (which makes this sort of debugging MUCH easier). Also, can you please tcpdump both the internal and external sides of the connection (simultaneously, if possible)? I.e. if you have packets to compare from tun0 and from eno1, it should be a piece of cake to figure out what is wrong.

[YwH]

When I'm troubleshooting vpn issues like this, I basically get 4 packet captures (after checking that CPU isn't bound up anywhere): 1) client inside, 2) client outside, 3) server inside, 4) server outside. On the client side, if you are pushing all traffic through the tunnel, then capture all the traffic. On the server side, filter to just that client's internal and external IPs, but don't filter any further.

[Cubox]

@JedDaniels Done! As you asked, I recorded everything. There might be garbage, especially on server showing eno1, which have a public ip.

[user1024]

Seems you have a loop in routes. What is the gateway for vpn clients: 88.190.15.1 or 212.83.129.104? Did you create separate table for 212.83.129.104 subnet routing (f.e. linux-ip.net/html/routing-tables.html)? Did you tried to use 88.190.15.135 address as server address? Also just in case: try to disable lso compression both server and client, in some cases it can become slow.

[Cubox]

Where do you see a push-gateway option?

[Cubox]

I have the masquerade rule now, it don't resolved the issue. I have two is on my server, with one of them being the failover and the other the main one. For having a server with four ips on the eth0 interface (on another Archlinux server) for over a year, I can tell you that multiples ips are not the issue here...

[bonsoy]

At CCD options there is redirect gateway. You need yo check if there is a static route on clients to theirs real GW.

[Cubox]

There is. Clients are able to talk to anything on the internet, they just do it slowly.

[Cubox]

Thanks for the answer. Changing the tun-mtu and mssfix option did not helped. The fastworwarding setting don't exist in Linux. Only BSD kernels.