Load balancing with multiple gateways

Question

I have to different ISPs, each on each own network. The main connects via ethernet and the secondary via wifi. The two networks have no relation at all. I just connect to them simultaneously. The reason I want to load balance between them is to achieve higher Internet speeds. Note: I have no advanced network hardware. Just my pc and the two routers that I have no access... main network:

if: eth0
gw: 192.168.178.1
my ip: 192.168.178.95
speed: 400 kbit/s

secondary network:

if: wlan0
gw: 192.168.1.1
my ip: 192.168.1.95
speed: 300 kbit/s

A diagram to explain the situation:

I'm on Arch Linux x64. I use netcfg to configure the interfaces Configs:

# /etc/network.d/main
CONNECTION='ethernet'
DESCRIPTION='A basic static ethernet connection using iproute'
INTERFACE='eth0'
IP='static'
ADDR='192.168.178.95'

# /etc/network.d/second
CONNECTION='wireless'
DESCRIPTION='A simple WEP encrypted wireless connection'
INTERFACE='wlan0'
SECURITY='wep'
ESSID='wifi_essid'
KEY='the_password'
IP="static"
ADDR='192.168.1.95'

And I use iptables to load balance, rules:

#!/bin/bash
/usr/sbin/ip route flush table ISP1 2>/dev/null
/usr/sbin/ip rule del fwmark 101 table ISP1 2>/dev/null
/usr/sbin/ip route add table ISP1 192.168.178.0/24 dev eth0 proto kernel  scope link  src 192.168.178.95  metric 202
/usr/sbin/ip route add table ISP1 default via 192.168.178.1 dev eth0
/usr/sbin/ip rule add fwmark 101 table ISP1
/usr/sbin/ip route flush table ISP2 2>/dev/null
/usr/sbin/ip rule del fwmark 102 table ISP2 2>/dev/null
/usr/sbin/ip route add table ISP2 192.168.1.0/24 dev wlan0 proto kernel  scope link  src 192.168.1.95  metric 202
/usr/sbin/ip route add table ISP2 default via 192.168.1.1 dev wlan0
/usr/sbin/ip rule add fwmark 102 table ISP2
/usr/sbin/iptables -t mangle -F
/usr/sbin/iptables -t mangle -X
/usr/sbin/iptables -t mangle -N MARK-gw1
/usr/sbin/iptables -t mangle -A MARK-gw1 -m comment --comment 'send via 192.168.178.1' -j MARK --set-mark 101
/usr/sbin/iptables -t mangle -A MARK-gw1 -j CONNMARK --save-mark
/usr/sbin/iptables -t mangle -A MARK-gw1 -j RETURN
/usr/sbin/iptables -t mangle -N MARK-gw2
/usr/sbin/iptables -t mangle -A MARK-gw2 -m comment --comment 'send via 192.168.1.1' -j MARK --set-mark 102
/usr/sbin/iptables -t mangle -A MARK-gw2 -j CONNMARK --save-mark
/usr/sbin/iptables -t mangle -A MARK-gw2 -j RETURN
/usr/sbin/iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
/usr/sbin/iptables -t mangle -A PREROUTING -m comment --comment "this stream is already marked; escape early" -m mark ! --mark 0 -j ACCEPT
/usr/sbin/iptables -t mangle -A PREROUTING -m comment --comment 'prevent asynchronous routing' -i eth0 -m conntrack --ctstate NEW -j MARK-gw1
/usr/sbin/iptables -t mangle -A PREROUTING -m comment --comment 'prevent asynchronous routing' -i wlan0 -m conntrack --ctstate NEW -j MARK-gw2
/usr/sbin/iptables -t mangle -N DEF_POL
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'default balancing' -p tcp -m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'default balancing' -p udp -m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw1 tcp' -p tcp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 0 -j MARK-gw1
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw1 tcp' -p tcp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 0 -j ACCEPT
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw2 tcp' -p tcp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 1 -j MARK-gw2
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw2 tcp' -p tcp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 1 -j ACCEPT
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw1 udp' -p udp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 0 -j MARK-gw1
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw1 udp' -p udp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 0 -j ACCEPT
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw2 udp' -p udp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 1 -j MARK-gw2
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw2 udp' -p udp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 1 -j ACCEPT
/usr/sbin/iptables -t mangle -A PREROUTING -j DEF_POL
/usr/sbin/iptables -t nat -A POSTROUTING -m comment --comment 'snat outbound eth0' -o eth0 -s 192.168.0.0/16 -m mark --mark 101 -j SNAT --to-source 192.168.178.95
/usr/sbin/iptables -t nat -A POSTROUTING -m comment --comment 'snat outbound wlan0' -o wlan0 -s 192.168.0.0/16 -m mark --mark 102 -j SNAT --to-source 192.168.1.95
/usr/sbin/ip route flush cache

(this script was made by fukawi2, I don't know how to use iptables) but I have no Internet connection...

output of iptables -t mangle -nvL

Chain PREROUTING (policy ACCEPT 1254K packets, 1519M bytes)
 pkts bytes target     prot opt in     out     source               destination         
1278K 1535M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
21532   15M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* this stream is already marked; escape early */ mark match ! 0x0
  582 72579 MARK-gw1   all  --  eth0   *       0.0.0.0/0            0.0.0.0/0            /* prevent asynchronous routing */ ctstate NEW
 2376  696K MARK-gw2   all  --  wlan0  *       0.0.0.0/0            0.0.0.0/0            /* prevent asynchronous routing */ ctstate NEW
1257K 1520M DEF_POL    all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 1276K packets, 1535M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 870K packets, 97M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 870K packets, 97M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DEF_POL (1 references)
 pkts bytes target     prot opt in     out     source               destination         
1236K 1517M CONNMARK   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default balancing */ ctstate RELATED,ESTABLISHED CONNMARK restore
15163 2041K CONNMARK   udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default balancing */ ctstate RELATED,ESTABLISHED CONNMARK restore
  555 33176 MARK-gw1   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* balance gw1 tcp */ ctstate NEW statistic mode nth every 2
  555 33176 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* balance gw1 tcp */ ctstate NEW statistic mode nth every 2
  277 16516 MARK-gw2   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* balance gw2 tcp */ ctstate NEW statistic mode nth every 2 packet 1
  277 16516 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* balance gw2 tcp */ ctstate NEW statistic mode nth every 2 packet 1
 1442  384K MARK-gw1   udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* balance gw1 udp */ ctstate NEW statistic mode nth every 2
 1442  384K ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* balance gw1 udp */ ctstate NEW statistic mode nth every 2
  720  189K MARK-gw2   udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* balance gw2 udp */ ctstate NEW statistic mode nth every 2 packet 1
  720  189K ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* balance gw2 udp */ ctstate NEW statistic mode nth every 2 packet 1

Chain MARK-gw1 (3 references)
 pkts bytes target     prot opt in     out     source               destination         
 2579  490K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* send via 192.168.178.1 */ MARK set 0x65
 2579  490K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save
 2579  490K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain MARK-gw2 (3 references)
 pkts bytes target     prot opt in     out     source               destination         
 3373  901K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* send via 192.168.1.1 */ MARK set 0x66
 3373  901K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save
 3373  901K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Solution

Unless you can somehow split up the traffic based on their local parameters (like LAN IP), using this solution you will end up having a lot of random errors because a lot of sites don't allow you to use the same session from radically different IP addresses, not to mention other protocols like FTP, DNS, etc.

If you really want to do this, you will need to rent a (virtual) server with at least 2 IP addresses and build a VPN on each of your connections, then set up an OSPF load balancing to utilize both connections equally. Your server will need to have at least the double of your two connections' speed together.

In summary: without network support failover works well enough, load balancing however is going to be a constant pain.

Update: What you need to do:

1. Set up the server and make sure your local machine sees the two server IP's through different connections.
2. Start two OpenVPN servers on the server, listening on each of the addresses.
3. Start two OpenVPN clients on the client.
4. Install Quagga on both the server and the client.
5. Make sure the server and the client see each other through the OpenVPN without advertising any routes yet.
6. Set up routing advertisement on the server so the default route (0.0.0.0/0) gets advertised over OSPF one link to the client via redistributing static routes. (This is hard.)
7. Add the second link as a backup link over OSPF.
8. Change the configuration so the two links are used in load balancing modes.

OSPF is a whole new world, it's an integral part of how the internet out there works. If you really want to do this, I recommend you give yourself time and read some books as it is way out of scope for this description. I've done this once and I might get back to writing a tutorial on this, but it will take considerable time so I'll make no promises.

Comments

[teissler]

what is your exact question? A useful link to understand what you are trying to do is linux-ip.net/html/linux-ip.html#adv-multi-internet

[dzervas]

I have a VPS which does actually have the speed needed. Can you help me on this? Your idea was very good!

[Janos Pasztor]

Updated the answer, however, a complete tutorial is out of scope here.

[dzervas]

Yes, I agree. The 6th step actually sounds really hard... :P

[Janos Pasztor]

Actually that's the easy part. Setting up OSPF properly is the hard part.

[dzervas]

Is there any way to avoid the second IP? I have a domain name and I read in the arch wiki that I can use a fully qualified domain name (wiki.archlinux.org/index.php/…)

[dzervas]

ok, I'm stuck. I've set up the openvpn (only 1 server works right now as i will buy the second ip in the next few days) , but i'm not sure that it is right (tun or tap?). now, i have no idea how to set up quagga.

[Janos Pasztor]

You could avoid the second IP by doing policy routing by VPN port. What you need to achieve is your two OpenVPN connections going over separate ISP links and the only way to do that is to have a policy routing rule that one VPN (marked by something like destination IP, port, etc) goes over one connection, the other one goes over the other. Not having a second IP will make this whole thing a lot harder.

[Janos Pasztor]

As for OSPF: by doing this, you'll open up a whole new can of worms. Welcome to the world of network engineers. You must read up on how this stuff works and it's a whole hell of a lot of learning, not something that can be described in a Q&A fashion SU provides. If you want a tutorial, I'd be more than happy to oblige in a few weeks time.