Ubuntu
libvirt package

libvirt inside lxd container cannot start virbr0 (Unable to set bridge virbr0 forward_delay: Permission denied)

Bug #1802906 reported by Brian Candler on 2018-11-12

This bug affects 1 person

	Status	Importance	Assigned to
libvirt (Ubuntu)	Fix Released	Undecided	Unassigned
Bionic	Won't Fix	Undecided	Unassigned
Cosmic	Won't Fix	Undecided	Unassigned

Bug Description

Ubuntu 18.04 lxd container, running on Ubuntu 18.04 host (kernel 4.15.0-38-generic)

Inside the container, I installed libvirt-bin. However it fails to start the predefined 'default' network:

root@bionic:/etc# virsh net-start default
error: Failed to start network default
error: Unable to set bridge virbr0 forward_delay: Permission denied

root@bionic:/etc# echo $?
1

root@bionic:/etc# virsh net-list
Name State Autostart Persistent
----------------------------------------------------------

root@bionic:/etc# virsh net-list --all
Name State Autostart Persistent
----------------------------------------------------------
default inactive yes yes

Here is the config:

root@bionic:/etc# cat /etc/libvirt/qemu/networks/default.xml

<network>
  <name>default</name>
  <uuid>0c431cb9-7348-48df-b692-8eece268b0a0</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:11:cc:e6'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

Using "virsh net-edit default" to remove "delay='0'" does not make a difference; it gets reinserted and the same error occurs.

/var/log/syslog shows:

Nov 12 11:59:11 bionic networkd-dispatcher[212]: WARNING:Unknown index 4 seen, reloading interface list
Nov 12 11:59:11 bionic systemd-networkd[178]: virbr0-nic: Gained carrier
Nov 12 11:59:11 bionic libvirtd[225]: 2018-11-12 11:59:11.257+0000: 251: error : virNetDevBridgeSet:140 : Unable to set bridge virbr0 forward_delay: Permission denied
Nov 12 11:59:11 bionic systemd-networkd[178]: virbr0-nic: Lost carrier
Nov 12 11:59:11 bionic networkd-dispatcher[212]: WARNING:Unknown index 5 seen, reloading interface list
Nov 12 11:59:11 bionic networkd-dispatcher[212]: ERROR:Unknown interface index 5 seen even after reload
Nov 12 11:59:11 bionic networkd-dispatcher[212]: WARNING:Unknown index 5 seen, reloading interface list
Nov 12 11:59:11 bionic networkd-dispatcher[212]: ERROR:Unknown interface index 5 seen even after reload
Nov 12 11:59:11 bionic networkd-dispatcher[212]: WARNING:Unknown index 5 seen, reloading interface list
Nov 12 11:59:11 bionic networkd-dispatcher[212]: ERROR:Unknown interface index 5 seen even after reload
Nov 12 11:59:11 bionic networkd-dispatcher[212]: WARNING:Unknown index 5 seen, reloading interface list
Nov 12 11:59:11 bionic networkd-dispatcher[212]: ERROR:Unknown interface index 5 seen even after reload

Attaching strace to libvirtd, this is what I see:

...
[pid 225] <... recvmsg resumed> {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000001}, msg_namelen=12, msg_iov=[{iov_base="add@/devices/virtual/net/virbr0-nic/queues/tx-0\0ACTION=add\0DEVPATH=/devices/virtual/net/virbr0-nic/queues/tx-0\0SUBSYSTEM=queues\0"..., iov_len=16384}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_PEEK|MSG_TRUNC) = 141
[pid 250] ioctl(23, SIOCGIFINDEX, {ifr_name="virbr0-nic" <unfinished ...>
[pid 225] recvmsg(13, <unfinished ...>
[pid 250] <... ioctl resumed> , }) = 0
[pid 225] <... recvmsg resumed> {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000001}, msg_namelen=12, msg_iov=[{iov_base="add@/devices/virtual/net/virbr0-nic/queues/tx-0\0ACTION=add\0DEVPATH=/devices/virtual/net/virbr0-nic/queues/tx-0\0SUBSYSTEM=queues\0"..., iov_len=16384}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 141
[pid 250] close(23 <unfinished ...>
[pid 225] poll([{fd=4, events=POLLIN}, {fd=6, events=POLLIN}, {fd=9, events=POLLIN}, {fd=10, events=POLLIN}, {fd=11, events=POLLIN}, {fd=12, events=POLLIN}, {fd=13, events=POLLIN}, {fd=14, events=POLLIN}, {fd=17, events=POLLIN}, {fd=18, events=POLLIN}, {fd=20, events=POLLIN}], 11, 4996 <unfinished ...>
[pid 250] <... close resumed> ) = 0
[pid 250] ioctl(22, SIOCBRADDIF) = 0
[pid 250] close(22) = 0
[pid 250] socket(AF_UNIX, SOCK_DGRAM, 0) = 22
[pid 250] fcntl(22, F_GETFD) = 0
[pid 250] fcntl(22, F_SETFD, FD_CLOEXEC) = 0
[pid 250] ioctl(22, SIOCGIFFLAGS, {ifr_name="virbr0-nic", ifr_flags=IFF_BROADCAST|IFF_MULTICAST}) = 0
[pid 250] ioctl(22, SIOCSIFFLAGS, {ifr_name="virbr0-nic", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_MULTICAST}) = 0
[pid 250] close(22) = 0
[pid 250] access("/var/lib/libvirt/dnsmasq/virbr0.macs", F_OK) = -1 ENOENT (No such file or directory)
[pid 250] socket(AF_UNIX, SOCK_DGRAM, 0) = 22
[pid 250] fcntl(22, F_GETFD) = 0
[pid 250] fcntl(22, F_SETFD, FD_CLOEXEC) = 0
[pid 250] access("/sys/class/net/virbr0/bridge/forward_delay", F_OK) = 0
[pid 250] openat(AT_FDCWD, "/sys/class/net/virbr0/bridge/forward_delay", O_WRONLY|O_TRUNC) = -1 EACCES (Permission denied)
[pid 250] gettid() = 250
[pid 250] write(2, "2018-11-12 12:02:07.815+0000: 250: error : virNetDevBridgeSet:140 : Unable to set bridge virbr0 forward_delay: Permission denied"..., 129) = 129
...

WORKAROUND: "lxc config set bionic security.privileged yes && lxc restart bionic"

However, I don't think that privileged mode should be necessary. If I turn off privileged mode, I can still create and edit bridges by hand, including setting the forwarding delay:

root@bionic:~# brctl show
bridge name bridge id STP enabled interfaces
root@bionic:~# brctl addbr testbr0
root@bionic:~# brctl show
bridge name bridge id STP enabled interfaces
testbr0 8000.000000000000 no
root@bionic:~# brctl setfd testbr0 0
root@bionic:~# brctl showstp testbr0 | grep "forward delay"
forward delay 0.00 bridge forward delay 0.00
root@bionic:~# cat /sys/class/net/testbr0/bridge/forward_delay
0
root@bionic:~# brctl setfd testbr0 1
root@bionic:~# brctl showstp testbr0 | grep "forward delay"
forward delay 1.00 bridge forward delay 1.00
root@bionic:~# cat /sys/class/net/testbr0/bridge/forward_delay
100

However, writing to the /sys filesystem directly does not work:

root@bionic:~# echo 0 > /sys/class/net/testbr0/bridge/forward_delay
bash: /sys/class/net/testbr0/bridge/forward_delay: Permission denied
root@bionic:~#

In fact, it looks like "brctl setfd" is failing silently to access the /sys entry, as shown by strace, but is falling back to using an ioctl which succeeds.

root@bionic:~# strace -f brctl setfd testbr0 1
...
socket(AF_UNIX, SOCK_STREAM, 0) = 4
brk(NULL) = 0x55e926464000
brk(0x55e926485000) = 0x55e926485000
openat(AT_FDCWD, "/sys/class/net/testbr0/bridge/forward_delay", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EACCES (Permission denied)
ioctl(4, SIOCDEVPRIVATE, 0x7fff63a06da0) = 0
exit_group(0) = ?
+++ exited with 0 +++
root@bionic:~# echo $?
0

This suggests that the proper solution is for libvirt to do something similar.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt0:amd64 4.0.0-1ubuntu8.5
ProcVersionSignature: Ubuntu 4.15.0-38.41-generic 4.15.18
Uname: Linux 4.15.0-38-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.4
Architecture: amd64
Date: Mon Nov 12 11:44:59 2018
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=C.UTF-8
SourcePackage: libvirt
UpgradeStatus: No upgrade log present (probably fresh install)

Tags:

Revision history for this message

Brian Candler (b-candler) wrote on 2018-11-12:

NonfreeKernelModules.txt Edit (1.5 KiB, text/plain; charset="utf-8")
Dependencies.txt Edit (3.0 KiB, text/plain; charset="utf-8")
KernLog.txt Edit (12.6 KiB, text/plain; charset="utf-8")
ProcCpuinfoMinimal.txt Edit (1.2 KiB, text/plain; charset="utf-8")
RelatedPackageVersions.txt Edit (149 bytes, text/plain; charset="utf-8")

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-11-12:

Hi,
to me it seems this is not a bug, but an issue with the default config.
To run virtualization in a LXD container - which by default is unprivileged for security you have to make some changes.
We will not change LXD/Libvirt defaults for that afaik, but the following is my recommendation as a container profile addition to get KVM+Libvirt running fine in a container.

config:
  boot.autostart: "true"
  linux.kernel_modules: openvswitch,nbd,ip_tables,ip6_tables,kvm
  security.nesting: "true"
  security.privileged: "true"
description: ""
devices:
  eth0:
    mtu: "9000"
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  kvm:
    path: /dev/kvm
    type: unix-char
  mem:
    path: /dev/mem
    type: unix-char
  tun:
    path: /dev/net/tun
    type: unix-char
name: kvm
used_by: []

You can create that with "lxc profile new kvm" and then launch those guests that need it with default+kvm profile, while leaving the other secure and unprivileged.
$ lxc launch ubuntu-daily:c/amd64 c --profile default --profile kvm

I hope that helps to understand, but IMHO it is not a bug.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-11-12:

I'm marking it incomplete assuming it is not a bug as explained and hope my explanations helped you.
Let me know if you think it is actually a bug and set it back to new then.

Changed in libvirt (Ubuntu):
status:	New → Incomplete

Revision history for this message

Brian Candler (b-candler) wrote on 2018-11-12:

Sorry, but I still think it's a bug.

The reason is that "brctl setfd ..." works fine in an unprivileged container, but libvirt-daemon fails. In other words: brctl shows that it *is* possible to create and manage bridges in an unprivileged container, but libvirt-daemon isn't doing it correctly.

I am currently only using libvirt to create virbr0 (for GNS3) - I'm not running kvm.

It's quite possible that anyone who wants to run kvm would require privileged mode for other reasons, but that would be a separate point.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-11-13:

Thanks for the clarification Brian.
I'll take a deeper look then, but FYI I'm on a trip this week - so it might take a few days more - sorry.

Changed in libvirt (Ubuntu):
status:	Incomplete → New

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-11-19:

Download full text (4.8 KiB)

Hi Brian, I now had more time to look at it - thanks for your patience and your detailed report!
The around that currently is this in libvirt doing that is like this:

Hi Brian, I now had more time to look at it - thanks for your patience and your detailed report!
The around that currently is this in libvirt doing that is like this:

110 #if defined(HAVE_STRUCT_IFREQ) && defined(__linux__)                             
 111 /*                                                                               
 112  * Bridge parameters can be set via sysfs on newish kernels,                     
 113  * or by  ioctl on older kernels. Perhaps we could just use                      
 114  * ioctl for every kernel, but its not clear what the long                       
 115  * term lifespan of the ioctl interface is...                                    
 116  */                                                                              
 117 static int virNetDevBridgeSet(const char *brname,                                
 118                               const char *paramname,  /* sysfs param name */     
 119                               unsigned long value,    /* new value */            
 120                               int fd,                 /* control socket */       
 121                               struct ifreq *ifr)      /* pre-filled bridge name */
 122 {                                                                                
 123     VIR_AUTOFREE(char *) path = NULL;                                            
 124                                                                                  
 125     if (virAsprintf(&path, SYSFS_NET_DIR "%s/bridge/%s", brname, paramname) < 0) 
 126         return -1;                                                               
 127                                                                                  
 128     if (virFileExists(path)) {                                                   
 129         char valuestr[INT_BUFSIZE_BOUND(value)];                                 
 130         snprintf(valuestr, sizeof(valuestr), "%lu", value);                      
 131         if (virFileWriteStr(path, valuestr, 0) < 0) {                            
 132             virReportSystemError(errno,                                          
 133                                  _("Unable to set bridge %s %s"), brname, paramname);
 134             return -1;                                                           
 135         }                                                                        
 136     } else {                                                                     
 137         unsigned long paramid;                                                   
 138         if (STREQ(paramname, "stp_state")) {                                     
 139             paramid = BRCTL_SET_BRIDGE_STP_STATE;                                
 140         } else if (STREQ(paramname, "forward_delay")) {                          
 141             paramid = BRCTL_SET_BRIDGE_FORWARD_DELAY;                            
 142         } else {                                                                 
 143             virReportSystemError(EINVAL,                                         
 144                                  _("Unable to set bridge %s %s"), brname, paramname);
 145             return -1;                                                           
 146         }                                                                        
 147         unsigned long args[] = { paramid, value, 0, 0 };                         
 148         ifr->ifr_data = (char*)&args;                                            
 149         if (ioctl(fd, SIOCDEVPRIVATE, ifr) < 0) {                                
 150             virReportSystemError(errno,                                          
 151                                  _("Unable to set bridge %s %s"), brname, paramname);
 152             return -1;                                                           
 153         }                                                                        
 154     }                                                                            
 155                                                                                  
 156     return 0;                                                                    
 157 }

Which means if the sysfs path exists it tries through sysfs (only) and only otherwise with the older ioctl interface.
The code is stable from libvirt 4.0 as in Ubuntu 18.04 until todays 4.9 just released upstream.

I think it would not be unreasonable to modify that be a fallback instead of an if/else.

I have coded something up (experimentally) that I'd want you to test from [1].
If that is working for you as expected than I'd suggest the same upstream in a polished way.
Please let me know what happens if you use this libvirt 4.0.0-1ubuntu8.7~ppa1 (in your container).

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3525/+packages

Revision history for this message

Brian Candler (b-candler) wrote on 2018-11-19:

Yes, it worked - as soon as the new packages were installed. Thank you!

root@bionic:~# ls
libvirt-bin_4.0.0-1ubuntu8.7~ppa2_amd64.deb
libvirt-clients_4.0.0-1ubuntu8.7~ppa2_amd64.deb
libvirt-daemon-driver-storage-rbd_4.0.0-1ubuntu8.7~ppa2_amd64.deb
libvirt-daemon-system_4.0.0-1ubuntu8.7~ppa2_amd64.deb
libvirt-daemon_4.0.0-1ubuntu8.7~ppa2_amd64.deb
libvirt0_4.0.0-1ubuntu8.7~ppa2_amd64.deb
root@bionic:~# dpkg -i *.deb
(Reading database ... 30717 files and directories currently installed.)
...
root@bionic:~# ifconfig virbr0
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
        inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
        ether 52:54:00:11:cc:e6 txqueuelen 1000 (Ethernet)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

root@bionic:~# virsh net-destroy default
Network default destroyed

root@bionic:~# virsh net-start default
Network default started

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-11-20:

I submitted it upstream for review there, if you want pass me your full email so I can add you as "Reported-by" on the patch. You can find my email here on launchpad on my users page in case you don't want to add it to the bug :-)

Ref: https://www.redhat.com/archives/libvir-list/2018-November/msg00747.html

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-11-20:

Note: upstream had made slight changes to the File in August, but the logical approach still was the same. Lets see what the feedback will be there.

Changed in libvirt (Ubuntu):
status:	New → Confirmed

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-11-26:

#10

Upstreaming complete:
https://libvirt.org/git/?p=libvirt.git;a=commit;h=6aa75b94627c9ad3bf6a836cc821750979a2fe05

I already have a working backport version from our PPA Tests on Bionic before.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-11-26:

#11

To hit this with virsh you'd see in an unprivileged contianer after install that it failed to start

$ apt install uvtool-libvirt
$ virsh net-list --all
Name State Autostart Persistent
----------------------------------------------------------
default inactive yes yes

If you try to do so you'll see your error:
$ virsh net-start default
error: Failed to start network default
error: Unable to set bridge virbr0 forward_delay: Permission denied

Right after the upgrade to the new version (or if you would have installed the new version to begin with) the fix works:

$ virsh net-list --all
Name State Autostart Persistent
----------------------------------------------------------
default active yes yes

If one wants he can now stop/start networks in unprivileged guest
root@b:~# virsh net-destroy default
Network default destroyed
root@b:~# virsh net-start default
Network default started

Of course without some privileges you will not be able to run full-virt KVM.
But qemu emulation based VMs work fine.

# Get an ISO
$ wget http://releases.ubuntu.com/18.04.1/ubuntu-18.04.1-live-server-amd64.iso
# Use virt manager to start a guest with that ISO
# It will auto-select Qemu-TCG mode as it can't run KVM without some privileges on the container

=> Works fine with the PPA.

I still think while nice this isn't an SRU case IMHO.
People that really need to run KVM in containers back in existing releases can just tweak the container privileges a bit - and they most likely want to to get full virtualization instead of "just" TCG-emulation.

But for upcoming releases this surely is a nice little feature to grow.

Changed in libvirt (Ubuntu Bionic):
status:	New → Won't Fix
Changed in libvirt (Ubuntu Cosmic):
status:	New → Won't Fix
Changed in libvirt (Ubuntu):
status:	Confirmed → In Progress

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-11-26:

#12

To get it ahead of the merge of a newer libvirt which will take a while I'll push the change to the current Ubuntu Development release (19.04 Disco) soon.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2018-11-26:

#13

This bug was fixed in the package libvirt - 4.6.0-2ubuntu4

---------------
libvirt (4.6.0-2ubuntu4) disco; urgency=medium

  * debian/patches/ubuntu/lp1787405-*: Support guest dedicated Crypto
    Adapters on s390x (LP: #1787405)
  * d/p/ubuntu/lp-1802727-netdevbridge-fall-back-to-ioctl-from-sysfs.patch:
    fix libvirt bridge handling in unprivileged containers (LP: #1802906)

-- Christian Ehrhardt <email address hidden> Fri, 09 Nov 2018 07:42:01 +0100

Changed in libvirt (Ubuntu):
status:	In Progress → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntulibvirt package

libvirt inside lxd container cannot start virbr0 (Unable to set bridge virbr0 forward_delay: Permission denied)

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
libvirt package