Bonds vs. Aggregators

Question

I'm almost to the point where I can post the solution I wound up with for my complex port bonding question. However, in reading the bonding.txt file, I see this option text:

ad_select

Specifies the 802.3ad aggregation selection logic to use.  The possible values and their effects are:

  stable or 0

    The active aggregator is chosen by largest aggregate bandwidth.

    Reselection of the active aggregator occurs only when all slaves of the active aggregator are down or the active aggregator has no slaves.

    This is the default value.

  bandwidth or 1

    The active aggregator is chosen by largest aggregate bandwidth. Reselection occurs if: 

      - A slave is added to or removed from the bond

      - Any slave's link state changes

      - Any slave's 802.3ad association state changes

      - The bond's administrative state changes to up

  count or 2

    The active aggregator is chosen by the largest number of ports (slaves). Reselection occurs as described under the "bandwidth" setting, above.

The way this is written, I can't tell if a single bond can contain more than one aggregator, or not! If the bonding module is smart enough to sort out more than one aggregation within a bond, I'm golden!

Let me simplify my drawing from over there:

____________ eth1   ________   eth2 ____________
| switch 1 |========| host |--------| switch 2 |
------------ eth3   --------        ------------

These switches do not do 802.3ad across switches. So, if I put all three interfaces into a single 802.3ad bond, do I get two aggregators? One containing eth1 & eth3, the other just holding eth2? Conceivably, the LACP signals between the host and the switches would be enough to do that. I just don't know if that capability is actually built in.

Anyone? Anyone? Can I get two aggregators out of a single interface bond?

Solution

Yes, given the following config:

.-----------.   .-----------.
|  Switch1  |   |  Switch2  |
'-=-------=-'   '-=-------=-'
  |       |       |       |
  |       |       |       |
.-=----.--=---.---=--.----=-.
| eth0 | eth1 | eth2 | eth3 |
|---------------------------|
|           bond0           |
'---------------------------'

Where each switch has its two ports configured in a PortChannel, the Linux end with the LACP bond will negotiate two Aggregator IDs:

Aggregator ID 1
 - eth0 and eth1

Aggregator ID 2
 - eth2 and eth3

And the switches will have a view completely separate of each other.

Switch 1 will think:

Switch 1
 PortChannel 1
 - port X
 - port Y

Switch 2 will think:

Switch 2
 PortChannel 1
 - port X
 - port Y

From the Linux system with the bond, only one Aggregator will be used at a given time, and will fail over depending on ad_select.

So assuming Aggregator ID 1 is in use, and you pull eth0's cable out, the default behaviour is to stay on Aggregator ID 1.

However, Aggregator ID 1 only has 1 cable, and there's a spare Aggregator ID 2 with 2 cables - twice the bandwidth!

If you use ad_select=count or ad_select=bandwidth, the active Aggregator ID fails over to an Aggregator with the most cables or the most bandwidth.

Note that LACP mandates an Aggregator's ports must all be the same speed and duplex, so I believe you could configure one Aggregator with 1Gbps ports and one Aggregator with 10Gbps ports, and have intelligent selection depending on whether you have 20/10/2/1Gbps available.

If this doesn't make sense, let me know, I'd love to improve this answer. LACP is a fantastic protocol which can do a lot of things people don't know about, this is one of the common ones.

People always want to "bond bonds" which cannot be done, but LACP allows the same setup with even more advantages and smart link selection.

Note on VPC

Some switches can be configured to "logically join" an Aggregator, so then the two switches act as one Aggregator ID. This is commonly called "Virtual Port Channel" or "Multi Chassis Link Aggregation" (MLAG).

This is possible, but is not what we're talking about here. In this answer, we're talking about two discrete switches which have no knowledge of each other.

Comments

[batfastad]

I'm under the impression that this approach does not require any special switch support other than regular LACP. Is that correct? No multi-chassis LAG or switch stacking required?

[user3922906]

@bbradley Yes, that is correct. This is one Linux end-host negotiating two separate Aggregators to two separate switches. Each switch has no knowledge that there is another switch attached to the Linux partner. No MLAG or switch stacking required.

[batfastad]

Thanks for the reply. I was having problems with this though my setup was slightly different - I had an inter-switch link as well. The problem is bond0 takes the MAC address of the first link in the bond. And even though the bonding driver negotiates an active Aggregator ID (pair), the link status on the other pair is still up, which causes the passive switch to keep an entry for the bond's MAC. So the MAC is present on both switches and obviously causes... carnage!