How can I create a secondary lxd bridge?

Question

I want try to create a network infrastructure with containers, but how can I configure another lxd (eg.: lxcbr1) bridge for my lxc container?

on the host I type:

sudo brctl addbr lxcbr1

but when I launch a new container by typing:

lxc launch images:centos/7/amd64 MyCentos

this container takes network config by default lxdbr0 bridge.

Solution

NOTE: The LXC client on the computer is capable of using remote LXC instances. Anything between square brackets [ ] in the usage information here is to be ignored, as I'll explain what to do alongside it. I also use launch commands provided from the OP's question, but you can use any OS image for launching that you'd like.

Instead of using brctl directly, and then messing with individual container configurations, consider instead using lxc network create to do this, and let LXD manage those bridges. It makes creating the bridges really easy and doesn't give us headaches as much. (This is what I do on several LXD environments).

From the usage:

lxc network create [<remote>:]<network> [key=value...]
    Create a network.

Effectively, just do lxc network create lxcbr1 or whatever you want to name it.

The problem is, you have to specify this somewhere. I personally use configuration profiles for this, especially if I have to bind to multiple bridges.

Now, normally I have individual profiles for individual bridges, and don't put two on the same box, but in your case you'll have two LXD bridge definitions. So we'll need to make a configuraiton profile for these.

---

If you need multiple bridges on the container at the same time...

Start by copying the default config to a multibridge config:

lxc profile copy default multibridge

After that, we need to edit the new multibridge file, to make it have multiple network devices.

lxc profile edit multibridge

You'll see something like this in the file:

devices:
  eth0:
    nictype: bridged
    parent: lxdbr0
    type: nic

Make a new blank line after that type line, and add lines, to make it look like this:

devices:
  eth0:
    nictype: bridged
    parent: lxdbr0
    type: nic
  eth1:
    nictype: bridged
    parent: testing
    type: nic

Don't edit anything else in the file, and save it.

Now, we have to tell LXD to create a new container, but to use the multibridge profile:

lxc launch images:centos/7/amd64 MyCentos -p multibridge

This will tell LXD to use the multibridge profile we just set up as the configuration profile for the new container, and it should use both lxdbr0 and lxdbr1 on eth0 and eth1 respectively within the container.

One major caveat: The system isn't smart, and doesn't configure eth1 properly. As such, you need to go into your CentOS system, and set up the configuration for eth1 yourself, to set it to either DHCP or static IPs within the lxdbr1 config's IP ranges. Otherwise, that interface won't ever be brought up. As this configuration will vary from OS to OS, I can't give you a clear-cut answer here on how to configure each and every network interface on every image. There's other resources available for that.

---

If you only need the new bridge, and not multiple bridges on one container at a time...

Then we need to create a profile for that specifically.

lxc profile copy default lxdbr1

... And then edit that new profile.

lxc profile edit lxdbr1

... And once in there, find where lxdbr0 is specified and change it to lxdbr1.

And then like above, launch your LXD container and specify the lxdbr1 profile for it.

lxc launch images:centos/7/amd64 MyCentos -p lxdbr1

You won't need to do any specialized configuration for the network interface, as the default is to set the first interface to DHCP and that will autoconfigure properly.

---

Just some post-configuration notes now:

You also can revise all the individual network elements if you know the configurations to enter by editing the lxdbr1 network configuration without using brctl and just call lxc network edit lxdbr1 and add the individual configurations. This lets lxd actually handle everything, and doesn't make you have to set up the bridge manually yourself with brctl (and lets LXD manage the bridge).

You may want to set up the network config similar to the below, which disables automatic NAT and makes you have to configure how the data flows to the Internet manually for your second bridge (you can make changes to the bridge config itself via lxc network edit lxdbr1 if you wanted to use lxdbr1 as the name). Note the use of ipv4.nat: "false" here, which disables the iptables' MASQUERADE rule that makes it look like the system itself reaching outbound:

config:
  ipv4.address: 10.75.251.1/24
  ipv4.dhcp: "true"
  ipv4.dhcp.ranges: 10.75.251.200-10.75.251.250
  ipv4.nat: "false"

There's also IPv6 equivalent commands as well, but IPv4 is easier to work with so I omit that here. Once this is done, you need to reboot the containers. Especially since we set everything to dhcp.

Comments

[Thomas Ward]

Are you running lxd on Ubuntu and a CentOS container inside it, or are you using some other environment for running lxd? Your answer determines whether it is offtopic or not for Ask Ubuntu. Also, are you trying to get multiple bridges onto the one container, or just a different bridge (but still only one network interface for the container itself)?

[Thomas Ward]

CLOSE VOTERS: Please wait for a response from the user before marking this as offtopic