custom dnsmasq (or custom options) with libvrt?

Question

Currently only two options related to network boot are available in libvirt via bootp options:
server and file

So for config like this:

<bootp file='test.ipx' server='10.10.10.2'/>

this code gets generated:

dhcp-boot=test.ipx,,10.10.10.2

1) How can i add something like this:

dhcp-userclass=set:ipxe,iPXE
dhcp-boot=tag:ipxe,http://matchbox.foo:8080/boot.ipxe

2) Or can i change config file from var/lib/libvirt/dnsmasq/default.conf to something else

Option 1 seems to be impossible looking at libvirt code .
Option 2 also seems impossible

Solution 1

Libvirt v5.6.0 (2019-08-05) added support for passing custom options to dnsmasq.

From the documentation:

A special XML namespace is available for passing options directly to the underlying dnsmasq configuration file. Usage of XML namespaces comes with no support guarantees, so use at your own risk.

This example XML will pass the option strings foo=bar and cname=*.foo.example.com,master.example.com directly to the underlying dnsmasq instance.

<network xmlns:dnsmasq='http://libvirt.org/schemas/network/dnsmasq/1.0'>
  ...
  <dnsmasq:options>
    <dnsmasq:option value="foo=bar"/>
    <dnsmasq:option value="cname=*.foo.example.com,master.example.com"/>
  </dnsmasq:options>
</network>

Fedora 31 ships with libvirt v5.6.0-4.fc31.

In my case, I'm looking to use a custom DNS server with my libvirt network, rather than the one provided by dnsmasq. Thanks to the advice from this answer, I think this would be the XML (but I can't test until I update libvirt):

<network xmlns:dnsmasq='http://libvirt.org/schemas/network/dnsmasq/1.0'>
  ...
  <dnsmasq:options>
    <dnsmasq:option value="dhcp-option=6,192.168.0.90,192.168.0.98"/>
  </dnsmasq:options>
</network>

Solution 2

I've been in the exactly same situation, trying to configure libvirt dhcp for matchbox. For reference I was working on Fedora 25

first option is impossible due to limitations of xml parsing in libvirt. Second option won't work since config will be overwritten by libvirt. You can't also configure dnsmasq to behave as a dhcp proxy for the same reasons as point one. The only way that I found to get this working, was to disable dhcp for that network completely (using virsh net-edit) and run dhcp as a separate service.

The default libvirt network will start two instances of dnsmasq, one for dns, one for dhcp. In my case that was:

# netstat -tulpn | grep dnsmasq
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      2229/dnsmasq
udp        0      0 192.168.122.1:53        0.0.0.0:*                           2229/dnsmasq
udp        0      0 0.0.0.0:67              0.0.0.0:*                           2229/dnsmasq
# ps aux | grep [d]nsmasq
nobody    2229  0.0  0.0  49104   372 ?        S    19:45   0:00 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
root      2230  0.0  0.0  49076   372 ?        S    19:45   0:00 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

to disable dhcp open the network config:

virsh net-edit default

and remove the dhcp section

before:

<network>
  <name>default</name>
  <uuid>6fe7eafd-1925-4943-9596-2172bd55d1ac</uuid>
  <forward mode='route'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:08:ed:3b'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.99'/>
    </dhcp>
  </ip>
</network>

after:

<network>
  <name>default</name>
  <uuid>6fe7eafd-1925-4943-9596-2172bd55d1ac</uuid>
  <forward mode='route'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:08:ed:3b'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
  </ip>
</network>

restart network for changes to take effect:

virsh net-destroy default

virsh net-start default

and confirm there's only one dnsmasq instance running now:

# netstat -tulpn | grep dnsmasq
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      2431/dnsmasq
udp        0      0 192.168.122.1:53        0.0.0.0:*                           2431/dnsmasq
# ps aux | grep [d]nsmasq
nobody    2431  0.0  0.0  49104   368 ?        S    19:55   0:00 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

Now you want to start your own instance listening on 0.0.0.0:67

dnsmasq was already installed for libvirt (with disabled systemd service etc), so I just had to create the following config file (with some environment specific values, see http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html):

# cat /etc/dnsmasq.d/default_dhcp.conf
pid-file=/var/run/libvirt/network/default_dhcp.pid
bind-dynamic
port=0
except-interface=lo
interface=virbr0
dhcp-range=192.168.122.2,192.168.122.99
dhcp-no-override
enable-tftp
tftp-root=/var/lib/tftp
dhcp-lease-max=98
dhcp-hostsfile=/var/lib/libvirt/dnsmasq/default.hostsfile
dhcp-option=6,192.168.122.1

# if request comes from older PXE ROM, chainload to iPXE (via TFTP)
dhcp-boot=tag:!ipxe,undionly.kpxe
# if request comes from iPXE user class, set tag "ipxe"
dhcp-userclass=set:ipxe,iPXE
# point ipxe tagged requests to the matchbox iPXE boot script (via HTTP)
dhcp-boot=tag:ipxe,http://matchbox.foo:8080/boot.ipxe

# verbose
log-queries
log-dhcp

and start (and enable) the daemon with:

systemctl start dnsmasq
systemctl enable dnsmasq

which resulted in:

# netstat -tulpn | grep dnsmas
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      1642/dnsmasq
udp        0      0 192.168.122.1:53        0.0.0.0:*                           1642/dnsmasq
udp        0      0 0.0.0.0:67              0.0.0.0:*                           2048/dnsmasq

and I could iPXE boot kvm vms using matchbox

Solution 3

Libvirt explicitly avoids allows generic passthrough of options to dnsmasq, since we want to insulate the public config format / APIs from knowing about the specific choice of dnsmasq as the impl backend. If you were to try to change the dnsmasq config file that libvirt writes out, your changes would simply be overwritten by libvirt later.

If there are features missing in libvirt network XML that you need, I'd encourage you to file a bug report against libvirt requesting them to be added. Any information you can give to explain the rationale behind their use would be beneficial too.

Comments

[ndemou]

Very well documented answer Michal

[Jonathon Reinhart]

The current (2019 Nov) documentation says "A special XML namespace is available for passing options directly to the underlying dnsmasq configuration file." However, it doesn't mention in which version this was added. I can't seem to get it work for me in libvirt 5.1.0 -- the <dnsmasq:options> are just discarded.

[Jonathon Reinhart]

Looks like it was added in v5.6.0 -- see my answer.

[poleguy]

I used this approach to prevent virbr0 from reporting bad dns results. It seems to leave the original except-interface=lo in position, but fortunately a new except-interface=lo,virbr0 is now there, fixing my isssue.

[Lion]

This worked for me, at least for subdomains. But I had to call net-destroy and net-start. Otherwise virsh net-edit and virsh net-dumpxml returned me the old network (without the dnsmasq namespace and options entry) - this was a bit confusing for me. Strangely wildcards still doesn't work, only specific domains like cname=my.test.example.com,test.example.com