qemu/KVM/libvirt usermode networking: VM as a non-root user, what do I need for root-less networking?

Question

I have run several VMs in KVM/qemu with libvirt, and somehow most of the time networking works.

Now, networking has stopped working when VMs are run as a non-root user. I have found little helpful information in the libvirt and similar documentation pages - most seem to assume that I wish to run the VM as a system user which is not the case.

So: What exactly are the pre-requisites to run a VM with network (e.g. webbrowsing in guest) as non-root?

I have, in myvm.xml:

 94     <interface type='user'>
 95       <mac address='52:54:00:82:f1:27'/>
 96       <model type='virtio'/>
 97       <link state='up'/>
 98       <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
 99     </interface>

The interface is visible from inside the guest, and enabling DHCP client gives an address 10.0.2.5. The default route is 10.0.2.2 which I can ping from within the network. Anything beyond that fails.

My host does not have a network bridge (or any device for/by libvirtd) but I shouldn't need that for usermode NAT within user-libvirtd, right? Technically the network access comes from user-libvirtd?

The user running libvirtd is in groups kvm and libvirt. Which one do I need for what? The latter is not necessary for running the VM and makes no difference for my problem.

I have enabled and started libvirtd as root on the host. This provides a network bridge virbr0 but is seems my non-root user cannot access it so this makes no difference whatsoever. Do I need libvirtd at all?

Please note that this VM may not run as root so that would not be a solution for me. The user may not have elevated rights or access to other people's VMs on the same host.

Solution 1

Whew! Here we go for round three! Let's see if we can get this to work in the end. Here goes.

Firstly, my VM actually was in qemu:///system, NOT in qemu:///session. So, even though I wasn't having to enter my root password, the VM must have been running as root anyway (?! Why did they do that ?!). So, here goes trying a VM in qemu:///session. (I'm typing this as I do the steps to see if I can reproduce your problem and fix it, so if it seems a little unplanned as I go, that's because it is.)

So, first I went into virt-manager, and started to make a new connection to QEMU/KVM different than the default one - this time I'm using "QEMU/KVM user session". When I selected it in virt-manager, it told me that "networking options were very limited". So this looks like where the problem begins. Let's see if I can get around it.

Having established the connection, now I'm going to create a new KolibriOS VM in it and see what happens.

So, during VM creation, virt-manager no longer sees my ISO Files directory containing my VM installers. So I'm going to add a new storage pool pointing to my ISO files so that I can actually make the VM. (directory: /home/user/ISO Files)

OK, now I have access to my ISOs. Now, I'm going to make the new KolibriOS VM with my "kolibri.iso" file. (OS type: Generic default, CPU count: 1, Memory: 256 MB. Kolibri is a tiny OS.)

I'm not going to give the VM any disk storage since KolibriOS is designed to be used directly off of the ISO.

Now, finally I'm at the end, and I notice one interesting thing. I'm given the option of using either usermode networking or a shared device name. I'm going to start with usermode networking, and if that doesn't work, we're gonna try again with shared device "virbr0", and see what happens.

I hit the Finish button. Now my VM should boot up in short order.

OK, it booted up, and I got "You are now connected to the network". Seems promising.

Now I've opened WebView, and I'm going to go to "Kolibri Stuff" and see what happens. If that works, I'm going to see if I can get to Google.

The "Kolibri Stuff" button worked - now I see the page "http://store.kolibri-n.org/en.html". Now let's try Google.

Sure enough, there's Google, complete with it's Privacy Policy link.. Let's see what happens if I click that.

Well, quite obviously WebView doesn't understand what on earth this page says, but I did get a nice big bunch of convoluted JavaScript on my screen, so obviously it downloaded something. Let's try NSInstall.

OK, it's having to download the NetSurf application. If it can download that, I assume that networking is working.

Download complete. Now let's try Google again.

OK, NetSurf didn't like Google. Let's try Dedoimedo. This is basically a bunch of Linux reviews and stuff like that.

Final conclusion - NetSurf stinks! I'm going back to WebView. (http://www.dedoimedo.com/index.html). Finally! It opened!

So, since I can successfully browse around inside my usermode VM, I assume that this is working. A "virsh -c qemu:///session list" now shows my "UserKolibriOS" VM running. Here's what it shows:

 Id   Name            State
-------------------------------
 1    UserKolibriOS   running

And "virsh -c qemu:///system list" now shows this:

 Id   Name   State
--------------------

So, I have a usermode VM accessing the Internet just fine. Now, let's try again, doing the same thing, but this time with Lubuntu 18.04, so that we get the virtio network adapter. (I'm doing this battery of tests because I want to make absolutely sure that everything works before dumping loads of configuration files on you.)

Here's my Lubuntu 18.04 VM config: 2 CPUs, 1024 MB of RAM, usermode networking, no virtual hard drive.

OK, the VM is booting. Let's see what happens.

The VM is booted. It seems to think that it's connected to the network. I'm going to open up Google and do a search for "blue screen of death", and see what happens.

Wow! The Internet in my VM almost seems to be working faster than the Internet on my physical system. I was able to bring up "Blue Screen of Death" on Wikipedia from my search, and opened it up. I'm now staring at a relatively grim picture of a frowning Windows 10 on my VM window. So, I am concluding that usermode networking is working just fine for web browsing in the VM. Now, let's see what my configuration is doing.

Firstly, I noticed that no "connected to tun vnet0" appeared on my screen when I launched the KolibriOS VM, and when I launched the Lubuntu 18.04 VM.

Now, here's the network adapter configuration, first for KolibriOS:

<interface type="user">
  <mac address="52:54:00:6f:ab:33"/>
  <model type="e1000"/>
  <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
</interface>

Now, here's what Lubuntu 18.04 looks like:

<interface type="user">
  <mac address="52:54:00:7d:63:ba"/>
  <model type="virtio"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>

So now my configuration looks identical to yours, except for my configuration is missing the bit about "link state="up"". Yet, my networking works and yours doesn't. Hmm...

All I can think now is that the networking settings in your VMs OS must not be working, and that your VM itself must be configured perfectly.

Finally, I'm going to run one final test - Lubuntu 18.04 with the shared device "virbr0". Let's see if it works with the network bridge even though it's a usermode VM.

Total fail! I got this mess on my screen when I tried it:

Unable to complete install: 'internal error: /usr/lib/qemu/qemu-bridge/helper
--br=virbr0 --fd=29: failed to communicate with bridge helper: Transport
endpoint is not connectedH001F007F stderr=failed to parse default acl file`/
-etc/qemu/bridge.conf''

What?! Obviously it doesn't want to connect to my network bridge. I think that you're right that bridged networking won't work at all with a usermode VM. But usermode networking did work, so it shouldn't be necessary.

I did notice something on the link you gave me to the info about usermode networking. It had a link to a page about QEMU's networking, that ended in usermode networking. It said at the end that "this option does provide a very useful default in that the guest OS will have largely transparent network access almost like any other application running on the host." (This was at "https://people.gnome.org/~markmc/qemu-networking.html".) Is QEMU actually allowed to connect to the Internet? Or has it somehow been blocked? Not sure if it's even possible to block a single process from Internet access in Linux, but maybe. If QEMU can't connect, the VM won't be able to connect.

So, final conclusion - I think that this is a problem with the virtual OS, not with the configuration of your virtual machine. Try Lubuntu 18.04 - it worked immediately, right out of the box. You can download it from here: "https://lubuntu.me/downloads/". See if the networking works there. Other than that, it looks like you're doing everything right.

Edit - This problem was finally solved by editing some stuff in "/etc/resolv.conf" in the virtual OS. This worked on Ubuntu and Arch Linux. Usermode networking now works. Thanks Ned64! (See comment from Ned64 below for details.)

Hope this helps!

Solution 2

This is in answer to the two comments from Ned64 on my last answer. I am posting as a guest user, so I can't just reply to the comment - I have to write a whole new answer, so if you're wondering, "why not just reply to the comment?", now you know. Also, my answer turns out to be pretty beefy, so it wouldn't have fit in the comment section, anyway.

OK, here goes.

Some info that might be useful - I am using virt-manager to manage and use my VMs. I am using the Lubuntu 20.04 64-bit operating system on my computer. All my VMs (KolibriOS, PuppyLinux, and Lubuntu 18.04) all have Internet access.

I didn't have someone else create my network bridge for me. It was there from the time that I installed QEMU, libvirt, and virt-manager. However, when I installed all this stuff, a new user (called libvirt-qemu) appeared, as did three new groups (called libvirt, libvirt-qemu, and libvirt-dnsmasq), and I've noticed that it has access to some directories (like /var/lib/libvirt/images) that my standard user can't access, so I assume that libvirt-qemu is responsible for my network bridge. Also, my default user is in the libvirt group, something I did not have to do myself - again, the installation routine must have done it for me.

Here's the output of my "brctl show":

bridge name     bridge id               STP enabled     interfaces
virbr0          8000.5254006b64fb       yes             virbr0-nic

For what it's worth, if I click on my networking icon, I can see "virbr0" in the list of active connections, so my physical computer is connected to the "virbr0" network like it would be to a real ethernet network.

I noticed that "brctl show" looks a little bit different if I do it while a VM is launched and connected to the Internet; here's what happens if I do that:

bridge name     bridge id               STP enabled     interfaces
virbr0          8000.5254006b64fb       yes             virbr0-nic
                                                        vnet0

Also, "vnet0" appears in my active connections while the VM is on, and it disappears when I turn off my VM.

Here's what I got from "virsh net-dumpxml default" with no VMs running:

<network>
  <name>default</name>
  <uuid>940f02c2-f3ba-4f25-ad0f-5876a41b5d3b</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:6b:64:fb'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

I also did a "virsh net-dumpxml default" while a VM was running - it made no difference.

Another potentially useful note - my user is in the libvirt group, but it is NOT in the kvm group. This might be helpful, or it might simply be confusing.

One final note - I see that the XML code of my VM with proper networking is using the "e1000" model type, but that your VM is using "virtio". Here's the code of a VM with working Internet that uses a virtio network adapter:

<interface type="network">
  <mac address="52:54:00:97:df:ec"/>
  <source network="default" portid="59b9b7c2-9453-43b6-8420-99961b5065c7" bridge="virbr0"/>
  <target dev="vnet0"/>
  <model type="virtio"/>
  <alias name="net0"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>

This is a live Lubuntu 18.04 64-bit VM running off of the ISO file - I can access the Internet just fine inside of it.

Hope this helps!

Comments

[Roman Riabenko]

Hello. Did you figure out why the preconfigured DNS did not work? Was it not working on all networks or only a particular one? Did the primary hostname in the host machine resolve? To check: host [some domain name] [your first resolver IP from /etc/resolv.conf].

[Ned64]

@RomanRiabenko Sorry, now that I have solved the problem I am happy and cannot go back. Next time this happens I will test and add a note here.

[Ned64]

Thanks! Can you give details on your bridge please - that was apparently set up for you? (It would need root on the host.) Could you show necessary info please (brctl show and the like, and/or config files for that bridge)?

[Ned64]

Also, who has created the default network (see your code line 3)? I don't have one. Please show its configuration - virsh net-dumpxml default.

[Ned64]

Many thanks! When I try to virsh net-define yournetwork.xml with your dump I get error: internal error: bridge name 'virbr0' already in use. or (when I change the bridge name) error: error creating bridge interface virbr1: Operation not permitted. I suppose a normal user cannot define such a network? Perhaps your system has done that for you. Please confirm for me that you are running the VM in a user session: virsh -c qemu:///session list and virsh -c qemu:///system list - where does the VM appear? Thanks.

[Ned64]

According to wiki.libvirt.org/page/… "qemu:///session has a serious drawback: since the libvirtd instance does not have sufficient privileges, the only out of the box network option is qemu's usermode networking". So, usermode seems the only way - bridged networking will not work at all?!

[Ned64]

Many thanks for your continued support! OK, now I was able to do it! What does not work for me: 1 - DNS - I need to edit /etc/resolv.conf by hand both in an Arch Linux guest and in a Ubuntu guest. There were invalid nameserver entries (127.0.0.53 and 10.0.2.2). 2 - ping from within the (usermode/non-root) VM does not work. Both restrictions have perhaps lead to me thinking that there is no networking when at some point during my trials in fact there was. Anyway, it is important to know that no bridged network needs to be tried unless we can run as a system user in qemu://system.

[LaserLinux]

Congrats! Thanks for sticking with the project until it was finally worked out. You also helps me a lot. 1, I learned a great trick to keep malware from escaping a VM (usermode networking), and 2, I now have some encouragement to try Arch Linux! Now that this is sorted out, I've removed the questions part of my answer. (I only just figured out that you weren't supposed to ask questions in the middle of an answer!) Oh, and I figured out adding comments now. Thanks a ton!

[LaserLinux]

Also, all your reputation boosts on my account just earned me a ton of new Stack Exchange privileges. Thank you so much!

[Roman Riabenko]

@Ned64 Hello! What DNS IP did you use? The default DNS service should be 10.0.2.3. That's what I get and it works most of the time. Was yours different?