Create a multi-homed linux load balancer with two internet connections and one LAN connection

Question

OK,... this has possibly been answered, unfortunately, I have not found anything that looks like it might be the resolution.

Ive looked at these pages and they have some useful information, but not complete resolutions:

1. Load balancing & NAT-ing multiple ISP connections on Linux
2. Two Internet Connections and 1 NIC, Possible?
3. Linux split access (multiple internet connections w/ load balancing)
4. linux firewall + load balance ISP connections

I will either be using Debian 8, Ubuntu 16.04 Desktop or Ubuntu 16.04 Server (to be honest, most likely Debian).

OK, so I will have three NICs installed two gigabit and the onboard 100 mbps LAN connection. The ISP connections will use the gigabit LAN cards via PPPoE authentication on both nics (if this is even possible otherwise, I'll settle for non bridging mode on the DSL modems). And both networks will have a static IP assigned to them. Currently (which will change) the primary connection has a group of five. I need to route traffic that comes in on one of those IP addresses to the proper server on the network.

Here are my questions:

1. How do I set this up? Do I use NATing, IP Chans, IP Masquerading, Routing, etc...?
2. How do I route the traffic coming into the system from the outside to a specific IP address on the network?

Please be gentle, this is the first time I have attempted something ANYTHING like this before :).

EDIT 1

Forgot to add the network topology:

EDIT 2

Just realized something... In order to do PPPoE authentication, I am going to have to force authentication of credentials on a specific interface. How is this done?

I posted that question here:

Create a multi-homed linux load balancer with two internet connections and one LAN connection

UPDATE 1

Still unable to get the multihop round robin to work. I tried to follow the steps outlined in the following sites with no luck:

1. Load balancing & NAT-ing multiple ISP connections on Linux
2. Linux - Dual Internet Connections / Load Balancing
3. HOWTO: Multirouting with Linux

I either keep getting a message stating that the file already exists or invalid device. Here is/are my settings/information

IFCONFIG

eth0      Link encap:Ethernet  HWaddr ec:08:6b:04:8e:ac  
          inet addr:172.16.0.2  Bcast:172.16.0.255  Mask:255.255.255.0
          inet6 addr: fe80::ee08:6bff:fe04:8eac/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:9525 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7722 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:9306973 (8.8 MiB)  TX bytes:949815 (927.5 KiB)

eth1      Link encap:Ethernet  HWaddr ec:08:6b:04:8c:95  
          inet addr:172.16.1.2  Bcast:172.16.1.255  Mask:255.255.255.0
          inet6 addr: fe80::ee08:6bff:fe04:8c95/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:9 errors:0 dropped:0 overruns:0 frame:0
          TX packets:42 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:558 (558.0 B)  TX bytes:6344 (6.1 KiB)

eth2      Link encap:Ethernet  HWaddr 00:16:76:90:49:b7  
          inet addr:172.16.2.1  Bcast:172.16.2.255  Mask:255.255.255.0
          inet6 addr: fe80::216:76ff:fe90:49b7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3793 errors:0 dropped:0 overruns:0 frame:0
          TX packets:79 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:283119 (276.4 KiB)  TX bytes:10338 (10.0 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:32 errors:0 dropped:0 overruns:0 frame:0
          TX packets:32 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:1858 (1.8 KiB)  TX bytes:1858 (1.8 KiB)

INTERFACES Configuration

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
allow-hotplug eth0
iface eth0 inet static
    address 172.16.0.2
    netmask 255.255.255.0
    gateway 172.16.0.1
    network 172.16.0.0
    broadcast 172.16.0.255

auto eth1
allow-hotplug eth1
iface eth1 inet static
    address 172.16.1.2
    netmask 255.255.255.0
    gateway 172.16.1.1
    network 172.16.1.0
    broadcast 172.16.1.255

auto eth2
allow-hotplug eth2
iface eth2 inet static
    address 172.16.2.1
    netmask 255.255.255.0
    network 172.16.2.0
    broadcast 172.16.2.255

ETH0

This is my primary WAN connection.

ETH1

This is my secondary WAN Connection.

ETH2

This is my Internal LAN connection.

I was able to get the routing to work on one WAN interface and linked to the internal LAN, however, I cannot replicate that and have no idea why.

Solution

With the assistance of a few friends helping me research this, I was FINALLY able to get it working.

Had to run this script in order to properly forward TCP/IP traffic:

#!/bin/sh

PATH=/usr/sbin:/sbin:/bin:/usr/bin

# Primary Connection - 2085426230
   IF1=eth0
   IP1=172.16.0.2
    P1=172.16.0.1
P1_NET=172.16.0.0

# Secondary Connection - 2085420213
   IF2=eth1
   IP2=172.16.1.2
    P2=172.16.1.1
P2_NET=172.16.1.0

# Local Network
   IF3=eth2
   IP3=172.16.2.2
    P3=172.16.2.1
P3_NET=172.16.2.0

T1=WAN1
T2=WAN2

# delete all existing rules.
iptables -F
iptables -t nat -F
iptables -t mangle -F
iptables -X

# Always accept loopback and WAN traffic
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i ${IF1} -j ACCEPT

# Allow established connections, and those not coming from the outside
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i ${IF1} -o ${IF3} -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow outgoing connections from the LAN side.
iptables -A FORWARD -i ${IF3} -o ${IF1} -j ACCEPT

# Masquerade.
iptables -t nat -A POSTROUTING -o ${IF1} -j MASQUERADE

# Enable routing.
echo 1 > /proc/sys/net/ipv4/ip_forward

Then I installed Net-ISP-Balance. Once the scripts ran and installed, I went to WhatsMyIP twice and sure enough, both interfaces are routing traffic as whatsmyip showed both IP addresses after I hit the refresh.

DISCLAIMER This worked for my particular network layout, configuration and settings. Your results may vary.

Comments

[Jure1873]

I think you have the answer in the first link you've posted. If the servers are on your LAN and have private addresses you definitely need NAT. Basically you need to split your problem into two problems: 1) Traffic coming IN, 2) Traffic coming out. Then ... do you need it only for load balancing or also for high availability? You need to have different routing tables for different providers (ip rule), then mark the incoming packets so they will go back out to the same provider. For the outgoing traffic you can balance it.

[John Schultz]

OK,... yes, come to think of it, I will need to be able to route the traffic accordingly (NATing) and I definately need to have the two incoming connections load balanced. So effectively having double the speed rather than single speed. Am I thinking about this correctly?

[Jure1873]

Do you have BGP on the outside? If you have separate internet links you can only "double the speed" on the outgoing side. The incoming side would only track incoming packets and make sure they go out on the same line as they came in.

[John Schultz]

I kinda figured that would be the case. No, I do not have BGP outside this location :( (Dont even know what it is).

[John Schultz]

Did a speed test.... I have a 24% increase in my dload traffic and 90% increase in my uload traffic.