Restrict access to KVM virtual machines to specific users

Question

On my server I have a KVM virtual machine called "cards2". It was created by executing (as root):

# virt-install --connect qemu:///system --virt-type kvm --name cards2 --ram 2048 --disk /var/kvm/cards2.qcow,size=3 --vcpus=8 --cdrom /var/kvm/debian-8.5.0-amd64-netinst.iso --vnc --os-type linux --network network=default

The image has permissions:

# ls -l /var/kvm/cards2.qcow 
-rwxr-xr-x 1 libvirt-qemu libvirt-qemu 3221225472 Aug 17 18:49 /var/kvm/cards2.qcow

However I noticed that any user with SSH access is able to gain access to the VM by executing:

virt-viewer --connect qemu+ssh://[email protected]/system vmname

(Note, this command is executed remotely, not on the server. This connects to the hypervisor with connection URI qemu+ssh://[email protected] by tunneling over SSH)

The user username is only a member of the username group. When SSH'ing in with the username account the list of virtual machines appear empty:

$ virsh list --all
 Id    Name                           State
----------------------------------------------------

And I am also unable to connect using a socket when executing the following over SSH:

$ virsh --connect qemu:///system list --all
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied

I also tried removing permissions to all /usr/bin/vir* files to users not in the kvm group:

# chown root:kvm /usr/bin/vir*
# chmod o-rx /usr/bin/vir*
# ls /usr/bin/vir* -l
-rwxr-x--- 1 root kvm  321120 Jul  1 04:46 /usr/bin/virsh
-rwxr-x--- 1 root kvm   32184 Dec  7  2013 /usr/bin/virt-alignment-scan
-rwxr-x--- 1 root kvm   28128 Dec  7  2013 /usr/bin/virt-cat
-rwxr-x--- 1 root kvm    9774 Sep 29  2014 /usr/bin/virt-clone
-rwxr-x--- 1 root kvm   10277 Sep 29  2014 /usr/bin/virt-convert
-rwxr-x--- 1 root kvm     806 Dec  7  2013 /usr/bin/virt-copy-in
-rwxr-x--- 1 root kvm     808 Dec  7  2013 /usr/bin/virt-copy-out
-rwxr-x--- 1 root kvm   54584 Dec  7  2013 /usr/bin/virt-df
-rwxr-x--- 1 root kvm   33312 Dec  7  2013 /usr/bin/virt-edit
-rwxr-x--- 1 root kvm   54536 Dec  7  2013 /usr/bin/virt-filesystems
-rwxr-x--- 1 root kvm   30112 Dec  7  2013 /usr/bin/virt-format
-rwxr-x--- 1 root kvm   14656 Jul  1 04:46 /usr/bin/virt-host-validate
-rwxr-x--- 1 root kvm    7944 Sep 29  2014 /usr/bin/virt-image
-rwxr-x--- 1 root kvm   44696 Dec  7  2013 /usr/bin/virt-inspector
-rwxr-x--- 1 root kvm   36992 Sep 29  2014 /usr/bin/virt-install
-rwxr-x--- 1 root kvm    5338 Dec  7  2013 /usr/bin/virt-list-filesystems
-rwxr-x--- 1 root kvm    6686 Dec  7  2013 /usr/bin/virt-list-partitions
-rwxr-x--- 1 root kvm   53816 Dec  7  2013 /usr/bin/virt-ls
-rwxr-x--- 1 root kvm   18641 Dec  7  2013 /usr/bin/virt-make-fs
-rwxr-x--- 1 root kvm    9600 Jul  1 04:46 /usr/bin/virt-pki-validate
-rwxr-x--- 1 root kvm   36264 Dec  7  2013 /usr/bin/virt-rescue
-rwxr-x--- 1 root kvm 1322488 Dec  7  2013 /usr/bin/virt-resize
-rwxr-x--- 1 root kvm 1231256 Dec  7  2013 /usr/bin/virt-sparsify
-rwxr-x--- 1 root kvm 1289592 Dec  7  2013 /usr/bin/virt-sysprep
-rwxr-x--- 1 root kvm    8949 Dec  7  2013 /usr/bin/virt-tar
-rwxr-x--- 1 root kvm     804 Dec  7  2013 /usr/bin/virt-tar-in
-rwxr-x--- 1 root kvm     806 Dec  7  2013 /usr/bin/virt-tar-out
-rwxr-x--- 1 root kvm      55 Jul 12  2012 /usr/bin/virtualenv
-rwxr-x--- 1 root kvm  132400 May 28  2012 /usr/bin/virt-viewer
-rwxr-x--- 1 root kvm   23886 Dec  7  2013 /usr/bin/virt-win-reg
-rwxr-x--- 1 root kvm    3531 Jul  1 04:46 /usr/bin/virt-xml-validate

And even though now I can't access any of these commands over a regular SSH connection, I can still bring up the VM by executing virt-viewer remotely (like above) over the SSH tunnel.

So, how can I make it so that only specific user accounts have access to the VMs?

Edit:

Here's what appears in my /var/log/libvirt/qemu/cards2.log when the VM is started, in case that gives any indication:

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.1 -enable-kvm -m 2048 -smp 8,sockets=8,cores=1,threads=1 -name cards2 -uuid 70905b35-9df3-71c9-d5e9-f804a2826055 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/cards2.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/kvm/cards2.qcow,if=none,id=drive-ide0-0-0,format=raw -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:c6:14:68,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:3 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4

Edit 2:

On another note, this appears to be only a problem for virt-viewer, and not for virsh.

For example here are some commands being executed on the remote client:

$ virsh --connect qemu+ssh://[email protected]/system
error: failed to connect to the hypervisor
error: End of file while reading data: nc: unix connect failed: Permission denied: Input/output error

$ virsh --connect qemu+ssh://[email protected]/system
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

Solution 1

K I got it. virt-viewer does not interact with libvirtd - it connects over ssh to the host and sets up a tunnel to allow access the the VNC based virtual displays of your VMs (127.0.0.1:5903 in my case). This is difficult to fix short of firewalling VNC on 127.0.0.1. Otherwise regular users are normally free to make TCP connections to localhost. I have no idea how you could permit just root, perhaps there's a way.

Perhaps selinux does that?

Linux: Allow/restrict IP bind permissions by user

So the easiest thing to do would be to set a VNC password. You could do things like restrict forwarding and X11forwarding in SSH but that might not be 100% secure, and might cause a problem for root also. And users could still access 127.0.0.1/vnc once logged in.

Update

According to Mike (other answer) and Michael Hampton (comments), you can use TLS to communicate with VNC rather than a password, so you'll get some pretty decent security if passwords aren't good enough for you. Props to both of them, I had no idea.

http://wiki.libvirt.org/page/VNCTLSSetup

and

https://www.suse.com/documentation/sles11/book_kvm/data/sec_libvirt_connect_remote.html#sec_libvirt_connect_remote_tls

Solution 2

Did you try using polkit to restrict access for users? I also wanted to do this so I ended up with this result:

1. Name the vm as username*vmname

2. now in the polkit rule file edit the rule as below

   function myFunction(username, virtualmachine) {
       var arr = virtualmachine.split("*");
       if(arr[0]==username){
           return true;
       }
       else{
           return false;
       }
   }
   // Allow passwordless connection to qemu:///system
   polkit.addRule(function(action, subject) {
       if (action.id == "org.libvirt.unix.manage")
       {
           return polkit.Result.YES;
       }
   });
   // Give full access to 'vm'
   polkit.addRule(function(action, subject) {
       if (action.id.indexOf("org.libvirt.api.domain.") == 0 ) {
           if (action.lookup("connect_driver") == 'QEMU' &&  myFunction(subject.user, action.lookup("domain_name"))) {
               polkit.log("vm=" + action.lookup("domain_name") + "action =>"+myFunction(subject.user, action.lookup("domain_name")));
               polkit.log("subject=" + subject);
               polkit.log("ok");
               return polkit.Result.YES;
           } else {
               return polkit.Result.NO;
           }
       }
   });
   

3. now only the corresponding user would only be able to see the vm itself.

enjoy

Solution 3

As Ryan's answer says above, "virt-viewer does not interact with libvirtd". libvirtd and VNC (what virt-viewer connects to) have completely separate authentication methods. VNC can be secured with

• Single password authentication
• SASL authentication
• x509 certificates
• A combination certificates and one of the first two

Single passwords

If using a single password, it can be either a global password stored in your /etc/libvirt/qemu.conf file or you can add a VM-specific password in the VM configuration:

<graphics type='vnc' port='-1' autoport='yes' passwd='PASSWORD'/>

Note, these passwords must be stored in plain text.

SASL authentication

This allows for multiple username and password combinations for each VM. The passwords are also hashed, as opposed to single passwords, which are stored in plain text. For instructions see here.

x509 certificates

This is actually the part that I believe directly answers the question (however I haven't tested it). With this, you are able to limit which user can connect to the VNC instance by authenticating with certificates stored on the server. To revoke permissions for a specific user or group, simply set the permissions of the system wide client certificate file to not be readable by that user. You can also generate per user certificates and revoke access.

Unfortunately it is relatively difficult to set up. It involves first creating a root CA on the server, Generating x509 Client/Server Certificates, configuring the server, configuring the client and testing the setup, and then restricting access.

Comments

[rudimeier]

I'm not familar with "virt" but you could either use the --vnc option with specifying a vnc password. or you could use chmod/chown/setfacl on libvirt-sock to set permissions for certain users.

[Mike]

Using --vnc with a password would be a bit better than what it is now, however I would rather be able to deny access completely to all unauthorized users. Here are the permissions of my libvirt-sock: srwxrwx--- 1 root libvirt 0 Jul 6 02:36 /run/libvirt/libvirt-sock. This should be good enough to prevent users not in the libvirt group from connecting to a VM, but there's something different when tunneling the connection over SSH that appears to bypass any permissions I put on the necessary files; as if they are being accessed as root.

[Dravigon]

did you try polkit to restrict access?? <br>see this serverfault.com/questions/808172/…

[Mike]

That's exactly how I have my server set up. I checked around in the log files but didn't find much. I have added the startup command I found in my /var/log/libvirt/qemu/cards2.log file to my question. Other than that, I can see the SSH user being logged in in auth.log and syslog, but that's about it. It's kind of hard to debug when something is "working" when it shouldn't be.

[Ryan Babchishin]

@Mike I'm sorry. I wish my system was messed up too and I could try to debug with you.

[Ryan Babchishin]

@Mike You're in luck maybe. I was testing with virt-manager, not virt-viewer. When I tried it your way, I could access the consoles of my VMs with a regular user! Damn... I'm looking into it now.

[Michael Hampton]

VNC passwords can be set per-VM, or a global VNC password configured in /etc/libvirt/qemu.conf. Better lock down permissions on /etc/libvirt while you're at it.

[Ryan Babchishin]

I've been tweaking my answer. I think I'm done. Added an interesting link to an sf question.

[Mike]

Thanks for the answer. I don't understand everything about what's going on behind the scenes, but it almost seems like a bug to me that it's done this way without any sort of user/group restrictions. I guess libvirtd is more for editing the VM's configuration while virt-viewer is simply for connecting to an existing session. Anyway, it's a production server so I really don't want to start messing around with SELinux since I'm unfamiliar with it.

[Ryan Babchishin]

@Mike it does seem *almost like a bug that any user can connect to a VM console... you'd think the KVM guys would have thought about a way to deal with that. Or the OS distribution could implement something... I don't blame you about SELinux, especially if that server is production. Good luck.

[Ryan Babchishin]

@Mike FYI, I had to enable auditd to figure out what was happening. virt-viewer doesn't execute any commands when it connects to the server, it just sets up a tunnel. Then connects over the tunnel.

[rudimeier]

BTW in case your users have their ssh logins only to connect to certain VMs you could probably restrict them via sshd_conf and authorized_keys so that the only thing they are allowed to do is to connect to one single VM socket (no-port-forwarding, command="...")

[Michael Hampton]

In production we don't use ssh tunnels, but TLS server and client certificates. Nobody can connect - even to VNC - without the correct certificate.

[Ryan Babchishin]

@MichaelHampton Unless the user has access to the server, then they can access VNC via 127.0.0.1.

[Michael Hampton]

If you've set up certificates, then this also applies to VNC, and being on localhost won't help you.

[Michael Hampton]

wiki.libvirt.org/page/TLSSetup and wiki.libvirt.org/page/VNCTLSSetup

[Ryan Babchishin]

@MichaelHampton Got it here suse.com/documentation/sles11/book_kvm/data/… from the answer from Mike... Lots of reading :). Got it now. Thanks. So that should solve the OPs problem. Nice.

[Mike]

@RyanBabchishin Me too!

[Mike]

Thank you for your answer. The VMs are in a production environment, so I haven't been able to test this. Did you test it? Did it work for you?

[Dravigon]

Yes I tested it and actually it did work my friend

[Mike]

Sounds good. Upvote from me then.

[asoundmove]

What OS are you running that uses polkit v0.106 or later?