How to Bond Two (Multiple) Internet Connections for Increased Speed and Failover

Question

We are located in a rural area with two slow ADSL connections (3.5/0.5 Mbps) and wanted to improve connection speed and reliability by somehow "combining" them. The following describes our solution which so far works very well.

This is therefore not a question we need resolved, but rather documentation of something that works because it was very difficult to find instructions and I hope this helps someone else in a similar situation. And maybe someone more advanced than me spots a bug that I'd be glad to come to know of and fix.

This documentation was compiled from the working system and had to be adjusted slightly to account for requirements specific to our location which are irrelevant for a more general description. So, while every effort has been made for reliable accuracy, I haven't redone the whole thing along this documentation, so there may be something missing or wrong. Post a comment if it doesn't work for you and I'll try to help.

The following two sources were very helpful in getting this to work, so hearty thanks to both authors!

• Bonding internet connections
• First attempts to use MLVPN (and shorewall)

Solution

This solution applies Linux Ethernet Bonding to two independent OpenVPN connections from a local gateway server to a server in the cloud that you have root access to, like a Linode or a DigitalOcean Droplet. The two Internet connections that the OpenVPN connections are based on use two ADSL routers in transparent bridge mode, i.e. we use their modem functionality only and create connections via PPP over Ethernet (PPPoE) driver. And the firewalls are implemented with FireHOL on both ends of the bonded connection.

The graph gives an overview. The LAN PCs are connected to eth0, 192.168.1.1/24, on the Gateway Server. The two routers are connected via individual, physical network interfaces eth1, 192.168.10.1/24, and eth2, 192.168.11.1/24. (Couldn't make PPPoE work with virtual interfaces on eth0.) The PPPoE driver creates interfaces ppp0 and ppp1 for individual connections to the ISP. OpenVPN binds to the Gateway Servers' 192.168.10.1/24 and 192.168.11.1/24 addresses on the LAN side, and to ports 1194 and 1195 of the Cloud Server's eth0 interface, 50.60.70.80/24. And these two OpenVPN connections are then bonded to create the virtual interfaces 10.80.0.2/30 on the LAN side and 10.80.0.1/30 on the Cloud Server side. Defining the Cloud Server's 10.80.0.1 address as default gateway on the LAN's Gateway Server allows all LAN computers to access the Internet at almost double the speed of one individual PPPoE connection.

The following configuration is based on Ubuntu Server (works here with 16.04 on the LAN side and 18.04 on the Cloud side). All commands assume root privileges.

Cloud Server Side

OpenVPN Tunnels

Install the latest OpenVPN version (replace bionic for 18.04 with xenial for 16.04)

cloud-server# wget -O - https://swupdate.openvpn.net/repos/repo-public.gpg|apt-key add -
cloud-server# echo "deb http://build.openvpn.net/debian/openvpn/stable bionic main" > /etc/apt/sources.list.d/openvpn-aptrepo.list
cloud-server# apt update && apt install openvpn

In /etc/default/openvpn make sure

AUTOSTART="none"

is active. Disable the service, tunnels will be managed from /etc/network/interfaces:

cloud-server# systemctl disable openvpn

Create /etc/openvpn/tap0.conf

# disable encryption, traffic continues unencrypted anyways
auth none
cipher none

dev tap0
mode p2p
port 1194
local 50.60.70.80
proto udp

log /var/log/tap0.log
verb 3

ping 2
ping-restart 10
persist-tun

compress lz4-v2

daemon

and /etc/openvpn/tap1.conf like /etc/openvpn/tap0.conf except

dev tap1
...
port 1195
...
log /var/log/tap1.log

Linux Ethernet Bonding

Using ifupdown for network management on the Cloud Server, modify your /etc/network/interfaces (adjust gateway for your environment):

auto eth0
iface eth0 inet static
  address 50.60.70.80
  netmask 255.255.255.0
  gateway 50.60.70.1
  post-up /usr/local/bin/vpn-start
  pre-down /usr/local/bin/vpn-stop

Options for the bonding device can be maintained in /etc/modprobe.d/bonding.conf:

options bonding mode=0 miimon=100

mode=0 means to use bonded lines in a round-robin fashion which should provide both, failover and speed enhancement.

The following two scripts create/destroy the bonding device. Create /usr/local/bin/vpn-start (and chmod +x):

#!/bin/bash
openvpn --config /etc/openvpn/tap0.conf
openvpn --config /etc/openvpn/tap1.conf

ip link add bond0 type bond
ip addr add 10.80.0.1/30 dev bond0

ip link set tap0 master bond0
ip link set tap1 master bond0

ip link set bond0 up mtu 1440
ip route add 192.168.1.0/24 via 10.80.0.2

You may need to adjust the mtu to your environment. Create /usr/local/bin/vpn-stop (and chmod +x):

#!/bin/bash
ip route del 192.168.1.0/24 via 10.80.0.2
ip link set bond0 down
ip link del bond0

pkill 'openvpn'

Firewall

For your firewalling needs you can install FireHOL:

cloud-server# apt install firehol

Leave START_FIREHOL=NO in /etc/default/firehol and instead create /etc/systemd/system/firehol.service

[Unit]
Description=FireHOL Stateful Packet Filtering Firewall
Documentation=man:firehol(1) man:firehol.conf(5)

DefaultDependencies=no

Before=network-pre.target
Wants=network-pre.target

Wants=systemd-modules-load.service local-fs.target
After=systemd-modules-load.service local-fs.target

Conflicts=shutdown.target
Before=shutdown.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/firehol start
ExecStop=/usr/sbin/firehol stop

[Install]
WantedBy=multi-user.target

and enable it

cloud-server# systemctl enable firehol

Create /etc/firehol/firehol.conf:

version 6

server_vpn_ports="udp/1194-1195"
client_vpn_ports="default"

snat4 to 50.60.70.80 outface eth0 dst not 50.60.70.80

interface eth0 web
  protection strong
  server ssh accept
  server vpn accept
  # more servers here as per your needs
  client all accept

interface bond0 vpn
  policy accept

router4 web2vpn inface eth0 outface bond0 dst 192.168.1.0/24,10.80.0.2
  client all accept

Activate and Check

Restart the Cloud Server. Check the bonding device:

cloud-server# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: tap0
MII Status: up
Speed: 10 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: aa:04:0b:ea:33:48
Slave queue ID: 0

Slave Interface: tap1
MII Status: up
Speed: 10 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 1e:70:4f:4b:2a:e8
Slave queue ID: 0

LAN Side

PPPoE Internet Connections

You would have to find out for your modems how to put them in transparent bridge mode and assign the LAN addresses. Using ifupdown for network management on the LAN Gateway Server, add the following to /etc/network/interfaces:

auto eth1
iface eth1 inet static
  address 192.168.10.1
  netmask 255.255.255.0
  network 192.168.10.0
  broadcast 192.168.10.255

auto eth2
iface eth2 inet static
  address 192.168.11.1
  netmask 255.255.255.0
  network 192.168.11.0
  broadcast 192.168.11.255

Install the PPPoE driver:

lan-server# apt update
lan-server# apt install pppoe pppoeconf

Create the two PPP configuration files, /etc/ppp/peers/dsl1:

plugin rp-pppoe.so eth1
unit 0
user "YourUsername1"
noauth
hide-password
persist
mtu 1492
noipdefault
defaultroute
replacedefaultroute

and /etc/ppp/peers/dsl2:

plugin rp-pppoe.so eth2
unit 1
user "YourUsername2"
noauth
hide-password
persist
mtu 1492
noipdefault
defaultroute

replacedefaultroute in /etc/ppp/peers/dsl1 makes this connection the default Internet connection before bonding.

Provide the passwords in /etc/ppp/chap-secrets and /etc/ppp/pap-secrets:

"YourUsername1" * "Password1"
"YourUsername2" * "Password2"

Make sure both files are owned by root and chmod 600.

Add the following to the end of /etc/network/interfaces:

auto dsl1
iface dsl1 inet ppp
  provider dsl1

auto dsl2
iface dsl2 inet ppp
  provider dsl2

This will automatically establish the PPPoE connections as defined in the configuration files their names given by the provider directives.

The VPN tunnels and the bonding device are managed by two scripts that run when the two PPP connections have come up/gone down. Create /etc/ppp/ip-up.d/bond0 (and chmod +x):

#!/bin/bash
nPpp=`ls -1 /etc/ppp/peers/* | wc -l`
if [[ `ip addr | grep -E 'ppp[0-9]:' | wc -l` -eq $nPpp ]] && \
    [[ `ip addr | grep -E 'tap[0-9]:' | wc -l` -eq 0 ]]; then
  /usr/local/bin/vpn-start
fi

and /etc/ppp/ip-down.d/bond0 (and chmod +x):

#!/bin/bash
if [[ `ip addr | grep -E 'ppp[0-9]:' | wc -l` -eq 0 ]]; then
  /usr/local/bin/vpn-stop
fi

See below for the implementations of the vpn-* scripts.

OpenVPN Tunnels

For the OpenVPN installation proceed like on the Cloud Server. OpenVPN will here be managed by scripts triggered when the PPPoE connections are established/destroyed.

Create /etc/openvpn/tap0.conf

remote 50.60.70.80

auth none
cipher none

dev tap0
mode p2p
port 1194
local 192.168.10.1
proto udp

log /var/log/tap0.log
verb 3

ping 2
ping-restart 10
persist-tun

compress lz4-v2

daemon

and /etc/openvpn/tap1.conf like /etc/openvpn/tap0.conf except

dev tap1
...
port 1195
local 192.168.11.1
...
log /var/log/tap1.log

Linux Ethernet Bonding

/usr/local/bin/vpn-start (chmod +x) creates the VPN tunnels, sets up the bonding device and creates routing table entries that ensure traffic goes through the right channels as the bonding driver requests:

#!/bin/bash
openvpn --config /etc/openvpn/tap0.conf
ip route add 192.168.10.0/24 dev eth1 scope link table dsl1
ip route add default dev ppp0 table dsl1
ip rule add pref 10 from 192.168.10.0/24 table dsl1

openvpn --config /etc/openvpn/tap1.conf
ip route add 192.168.11.0/24 dev eth2 scope link table dsl2
ip route add default dev ppp1 table dsl2
ip rule add pref 11 from 192.168.11.0/24 table dsl2

ip route flush cache

ip link add bond0 type bond
ip addr add 10.80.0.2/30 dev bond0

ip link set tap0 master bond0
ip link set tap1 master bond0

ip link set bond0 up mtu 1440

/usr/local/bin/gw bond0

The names for the routing tables need to be declared in /etc/iproute2/rt_tables:

...
10  dsl1
11  dsl2

Make sure the numbers are unique within this configuration file.

The mtu needs to correspond to the one configured on the Cloud Server.

/usr/local/bin/gw (chmod +x) allows to switch default gateway:

#!/bin/bash
newGw=$1
if [[ ! $newGw =~ ^ppp[0-9]$ && $newGw != bond0 ]]; then
  echo "$0 {ppp[0-9]|bond0}"
  exit -1
fi

ip addr show dev $newGw >/dev/null 2>&1
ret=$?
if [[ $ret -ne 0 ]]; then
  echo "$newGw is not available"
  exit -1
fi

via=0.0.0.0
if [[ $newGw == bond0 ]]; then
  via=10.80.0.1
fi
ip route repl default via $via dev $newGw
ip route show

Create /usr/local/bin/vpn-stop (chmod +x):

#!/bin/bash
/usr/local/bin/gw ppp0

ip link set bond0 down
ip link del bond0

pkill 'openvpn'

ip rule del pref 10 from 192.168.10.0/24 table dsl1
ip route del default dev ppp0 table dsl1
ip route del 192.168.10.0/24 dev eth1 scope link table dsl1

ip rule del pref 11 from 192.168.11.0/24 table dsl2
ip route del default dev ppp1 table dsl2
ip route del 192.168.11.0/24 dev eth2 scope link table dsl2

ip route flush cache

The two vpn-* and the gw scripts can of course be run manually also if need be.

Firewall

Install FireHOL like on the Cloud Server with the following /etc/firehol/firehol.conf configuration:

version 6

lan="eth0"
web="ppp+"
vpn="bond+"

tcpmss auto "${web}"
masquerade "${web}"

interface "${lan}" lan
  policy accept

interface "${web}" web
  protection bad-packets
  server ident reject with tcp-reset
  client all accept

interface "${vpn}" vpn
  policy accept

router web2lan inface "${web}" outface "${lan}"
  protection bad-packets
  server ident reject with tcp-reset
  client all accept

router vpn2lan inface "${vpn}" outface "${lan}"
  policy accept

Activate and Check

Restart the LAN Server and check the bonding device:

lan-server# cat /proc/net/bonding/bond0

Output should resemble Cloud Server.

If you now navigate e.g. to https://www.whatsmyip.org/ in your browser you should see your cloud server's IP address.

You can test your speed improvement e.g. by running

lan-server# wget -4 -O /dev/null http://proof.ovh.net/files/1Gio.dat

Here we see bonded speed only some 5% less than the sum of the individual lines' speeds.

Failover with Link Balancer

If one of the Internet connections goes down the bonding device doesn't continue on the remaining connection as one might expect. For this event one can prepare by setting up FireHOL's Link Balancer.

One way to do this is by creating a suitable /etc/firehol/link-balancer.conf and schedule /usr/sbin/link-balancer as a cron job to periodically (e.g. every 2 minutes) check the connections and failover to what is still available if need be. The following link-balancer.conf will make Internet access continue on the remaining good line if one goes bad:

check_bond0() {
  for dev in ppp0 ppp1 bond0; do
    /sbin/ip addr show dev $dev >/dev/null 2>&1
    ret=$?
    if [[ $ret -ne 0 ]]; then
      break
    fi
  done
  if [[ $ret -eq 0 ]]; then
    /bin/ping -w 8 -c 4 -I 10.80.0.2 10.80.0.1 >/dev/null 2>&1
    ret=$?
  fi
  return $ret
}

gateway bond dev bond0 gw 10.80.0.1 check bond0
gateway dsl1 dev ppp0
gateway dsl2 dev ppp1

table main
  default via bond weight 100
  fallback via dsl1
  fallback via dsl2

The default check for the bonded connection seems not to reliably detect its state, that's why the custom check_bond0.

When the bad connection gets back up again, Link Balancer will make both connections default gateways that will be balanced on a connection basis. Couldn't with reasonable effort revive the bonded connection, so in these rare cases one will have to manually restart potentially both VPN ends.

Comments

[Unknown123]

Do you have the solution for windows?

[legolas108]

Unfortunately, no, this is for Linux (Ubuntu) servers only.

[Unknown123]

Ah okay, I haven't understood what you're trying to accomplish. Are you building load balancing or channel bonding? The one that able to combine e.g. 1Mbps+1Mbps for 1 connection just like Speedify or just spread 1Mbps and the other 1Mbps for multiple connection like Connectify Dispatch?

[legolas108]

It's bonding connections like Speedify to achieve (almost) the sum of the individual lines' bandwidth.

[GDorn]

One thing missing from most of these tutorials is why, exactly, the server needs to have two IP addresses. Is this just a limitation of openvpn?

[legolas108]

@GDorn As far as I understand it, the bonding device needs to distribute traffic through both modems and for that needs to have individual addresses to send traffic to. OpenVPN is only running traffic through individual channels and is unaware of the bonding.

[GDorn]

@legolas108 additional research suggests this is indeed openvpn imposing this limitation, and that newer alternatives (such as MPTCP and Wireguard) do not require two IP addresses.

[Ray Foss]

The mode you're using is round-robin... anyway to tune the percentage of packets sent over each connection? say one connection is 25Mbps while the other is 75Mbps... You'd like to get somewhere around 100, but round-robin with the same weight seems inappropriate.

[legolas108]

@RayFoss Wouldn't know how to configure this system with connections of different speeds. You may want to take a look at GloryTun or ShadowSocks or OpenMPTCProuter.

[Ray Foss]

@legolas108 I ended up going with Speedify... they effectively add 100ms of latency to sort the packages and then steward a few IP addresses for Netflix support. Good stuff... but closed source