segfault when attaching disk with same physical device

Bug #455832 reported by Jamie Strandboge
22
This bug affects 2 people
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Invalid
High
Serge Hallyn
Karmic
Won't Fix
High
Unassigned
Lucid
Fix Released
High
Dustin Kirkland 
Maverick
Invalid
High
Serge Hallyn

Bug Description

I was testing attaching and detaching an AoE block device and all was going fine until I tried to attach a device twice in a row without changing the target device. Doing so resulted in a segfault. My example uses AoE but I bet any disk type='block' would work. This is easily a local DoS for libvirtd for anyone one in the libvirtd group or more than likely a remote user who has access to qemu+ssh://<vuln host>/system.

This happens with the apparmor security driver disabled too (ie, edit /etc/libvirt/qemu.conf to have 'security = "none"' and restart /etc/init.d/libvirt-bin).

Eg:
$ cat > /tmp/aoe.xml << EOM
<disk type='block'>
  <driver name='virtio'/>
  <source dev='/dev/etherd/e2.2'/>
  <target dev='vda' bus='virtio'/>
</disk>
EOM

$ virsh attach-device sec-karmic-amd64 /tmp/aoe.xml
Connecting to uri: qemu:///system
Device attached successfully

$ virsh detach-device sec-karmic-amd64 /tmp/aoe.xml
Connecting to uri: qemu:///system
Device detached successfully

$ virsh attach-device sec-karmic-amd64 /tmp/aoe.xml
Connecting to uri: qemu:///system
Device attached successfully

$ virsh attach-device sec-karmic-amd64 /tmp/aoe.xml
Connecting to uri: qemu:///system
error: Failed to attach device from /tmp/aoe.xml
error: server closed connection

$ dmesg| tail -1
[ 1006.485494] libvirtd[2909]: segfault at 70 ip 00000000004345f2 sp 00007f1f75c73b70 error 4 in libvirtd[400000+77000]

If you start libvirtd in another window under gdb, you can see the issue:

$ sudo gdb libvirtd
GNU gdb (GDB) 7.0-ubuntu
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/libvirtd...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/sbin/libvirtd
[Thread debugging using libthread_db enabled]
16:26:02.316: warning : qemudStartup:521 : Unable to create cgroup for driver: No such device or address
16:26:02.572: warning : lxcStartup:1460 : Unable to create cgroup for driver: No such device or address
[New Thread 0x7f8fb8346910 (LWP 4645)]
[New Thread 0x7f8fb7b45910 (LWP 4646)]
[New Thread 0x7f8fb7344910 (LWP 4647)]
[New Thread 0x7f8fb6b43910 (LWP 4648)]
[New Thread 0x7f8fb6342910 (LWP 4649)]
WARNING: Unhandled message: interface=org.freedesktop.DBus.Introspectable, path=/, member=Introspect
16:26:11.730: error : qemudDomainAttachPciDiskDevice:4857 : operation failed: target vda already exists
libvir: QEMU error : operation failed: target vda already exists

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f8fb7344910 (LWP 4647)]
0x00000000004345f2 in ?? ()
(gdb) bt
#0 0x00000000004345f2 in ?? ()
#1 0x000000000043489c in ?? ()
#2 0x0000000000434b94 in ?? ()
#3 0x0000000000434d91 in ?? ()
#4 0x000000000042cc2a in ?? ()
#5 0x00007f8fbcc53b01 in virDomainAttachDevice () from /usr/lib/libvirt.so.0
#6 0x000000000041dddf in ?? ()
#7 0x000000000041f5c6 in ?? ()
#8 0x000000000041f884 in ?? ()
#9 0x0000000000413a5c in ?? ()
#10 0x00007f8fbacfba04 in start_thread (arg=<value optimized out>)
    at pthread_create.c:300
#11 0x00007f8fbaa657bd in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#12 0x0000000000000000 in ?? ()

====
SRU:
 * IMPACT: If affected, libvirtd will crash unexpectedly when attempting to attach a disk device to a running Virtual Machine when it is already attached.
 * ADDRESSED: The patch prevents libvirt from entering a code path when cgroups are not used, thus preventing a NULL Pointer Exception/Dereference from occuring.
 * PATCH: Modification of my attached patch to match upstream patch.
 * TEST CASE:

1. Create/use any existing KVM virtual machine
2. Insert the contents below into a file called /tmp/455832-testcase.xml

---
<disk type='block'>
  <driver name='virtio'/>
  <source dev='/dev/sdd'/>
  <target dev='vdc' bus='virtio'/>
</disk>
---
   N.B. Change /dev/sdd to a device that exists, such as a blank USB Thumbdrive, ensure it is not mounted on the running system

3. Run "virsh attach-device <vmname> /tmp/455832-testcase.xml" twice
4. libvirt will crash unexpectedly w/o patch applied, will not crash w/ patch.
5. Refer to comment #8 for output details.

 * REGRESSION POTENTIAL: Patch is pretty simple, have been in upstream for 8 months+ and is in Maverick at the moment without complaint.

Note, also includes updated patch for Bug #571093.
====

security vulnerability: yes → no
visibility: private → public
Changed in libvirt (Ubuntu):
importance: Undecided → High
Changed in libvirt (Ubuntu Karmic):
milestone: none → ubuntu-9.10
Revision history for this message
Matt Zimmerman (mdz) wrote :

Has this worked for you previously? If so, with which Ubuntu release or package versions?

Revision history for this message
Thierry Carrez (ttx) wrote :

Per 20091020 meeting decision

Changed in libvirt (Ubuntu Karmic):
milestone: ubuntu-9.10 → none
status: New → Won't Fix
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

I tested this on 9.04 and got:
(jaunty)$ virsh attach-device testqemu /tmp/sdc.xml
Connecting to uri: qemu:///system
Device attached successfully

(jaunty)$ virsh attach-device testqemu /tmp/sdc.xml
Connecting to uri: qemu:///system
error: Failed to attach device from /tmp/sdc.xml
error: operation failed: target vda already exists

I also used the following xml, where /dev/sdc is a usb stick rather than an AoE device:
<disk type='block'>
  <driver name='virtio'/>
  <source dev='/dev/sdc'/>
  <target dev='vda' bus='virtio'/>
</disk>

I then tried this on karmic with the same results as before (ie, it segfaults):
$ virsh attach-device sec-karmic-amd64 /tmp/sdc.xml
Connecting to uri: qemu:///system
Device attached successfully

$ virsh attach-device sec-karmic-amd64 /tmp/sdc.xml
Connecting to uri: qemu:///system
error: Failed to attach device from /tmp/sdc.xml
error: server closed connection

$ dmesg|tail -1
[71676.786697] libvirtd[5018]: segfault at 70 ip 00000000004345f2 sp 00007f3d833ddb70 error 4 in libvirtd[400000+77000]

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Put simply, this is a regression over 9.04.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Adding 'regression-release' since this won't be fixed in 9.10.

tags: added: regression-release
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

I just noticed that if I have the AoE device partitioned (ie, there are /dev/etherd/e2.2 and /dev/etherd/e2.2p1) then try to attach either, libvirt segfaults, regardless of whether it was previously attached or not. Deleting the e2.2p1 partition and creating an ext2 filesystem directly on e2.2 allows the device to attach.

Changed in libvirt (Ubuntu Lucid):
status: New → Incomplete
assignee: nobody → Jamie Strandboge (jdstrand)
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Still an issue on Lucid:
[112639.277243] libvirtd[11372]: segfault at 70 ip 00007f1ee1bcb402 sp 00007f1edba7e8b0 error 4 in libvirt.so.0.7.5[7f1ee1baf000+113000]

Changed in libvirt (Ubuntu Lucid):
status: Incomplete → Confirmed
assignee: Jamie Strandboge (jdstrand) → nobody
Revision history for this message
Nigel Jones (dev-nigelj) wrote :
Download full text (3.4 KiB)

I had a look at this bug and I seem to have prevented it, i.e.:

njones@fantail:~$ cat /tmp/455832.xml
<disk type='block'>
  <driver name='virtio'/>
  <source dev='/dev/sdd'/>
  <target dev='vdc' bus='virtio'/>
</disk>

(sdd is a USB stick)

Before patch:

njones@fantail:~$ virsh list; virsh destroy lucid-test; virsh start lucid-test; virsh attach-device lucid-test /tmp/455832.xml; virsh attach-device lucid-test /tmp/455832.xml
 Id Name State
----------------------------------
  9 lucid-test running

Domain lucid-test destroyed

Domain lucid-test started

Device attached successfully

error: Failed to attach device from /tmp/455832.xml
error: server closed connection

After patch:

njones@fantail:~$ virsh list; virsh destroy lucid-test; virsh start lucid-test; virsh attach-device lucid-test /tmp/455832.xml; virsh attach-device lucid-test /tmp/455832.xml
 Id Name State
----------------------------------
 10 lucid-test running

Domain lucid-test destroyed

Domain lucid-test started

Device attached successfully

error: Failed to attach device from /tmp/455832.xml
error: operation failed: target vdc already exists

From GDB w/o patch:
Core was generated by `libvirtd -v'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f3cbe516402 in virCgroupPathOfController (group=0x0, controller=4, key=0x7f3cbe5d6d7c "devices.deny", path=0x7f3cb7fc88d8) at util/cgroup.c:246
246 if (group->controllers[controller].mountPoint == NULL)
(gdb) p
The history is empty.
(gdb) bt
#0 0x00007f3cbe516402 in virCgroupPathOfController (group=0x0, controller=4, key=0x7f3cbe5d6d7c "devices.deny", path=0x7f3cb7fc88d8) at util/cgroup.c:246
#1 0x00007f3cbe51661c in virCgroupSetValueStr (group=0x0, controller=4, key=0x7f3cbe5d6d7c "devices.deny", value=0x7f3cb0049b40 "b 8:16 rwm") at util/cgroup.c:271
#2 0x00007f3cbe516914 in virCgroupDenyDevice (group=0x0, type=<value optimized out>, major=<value optimized out>, minor=<value optimized out>) at util/cgroup.c:882
#3 0x00007f3cbe516bf4 in virCgroupDenyDevicePath (group=0x0, path=<value optimized out>) at util/cgroup.c:932
#4 0x000000000043ce2b in qemudDomainAttachDevice (dom=0x7f3cb0028920, xml=0x7f3cb006a9d8 "\340\244u") at qemu/qemu_driver.c:5505
#5 0x00007f3cbe551e81 in virDomainAttachDevice (domain=0x7f3cb0028920,
    xml=0x7f3cb000f080 "<disk type='block'>\n <driver name='virtio'/>\n <source dev='/dev/sdb'/>\n <target dev='vdc' bus='virtio'/>\n</disk>\n") at libvirt.c:5093
#6 0x000000000042296f in remoteDispatchDomainAttachDevice (server=<value optimized out>, client=<value optimized out>, conn=0x758bc0, hdr=<value optimized out>, rerr=0x7f3cb7fc8b80, args=0x7f3cb7fc8c10,
    ret=0x7f3cb7fc8c60) at remote.c:883
#7 0x0000000000424421 in remoteDispatchClientCall (server=<value optimized out>, client=0x762110, msg=0x7a30f0) at dispatch.c:506
#8 0x00000000004247d3 in remoteDispatchClientRequest (server=0x74f110, client=0x762110, msg=0x7a30f0) at dispatch.c:388
#9 0x000000000041768c in qemudWorker (data=<value optimized out>) at libvirtd.c:1522
#10 0x00007f3cbca0b9ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#11 0x0000...

Read more...

Revision history for this message
Nigel Jones (dev-nigelj) wrote :

Also, I forgot to mention in my previous comment, I'm yet to determine if this also affects the current release of libvirt.

Nigel Jones (dev-nigelj)
tags: added: patch
Revision history for this message
Thierry Carrez (ttx) wrote :

Serge, can you look into this one and give your opinion ?

Changed in libvirt (Ubuntu Maverick):
assignee: nobody → Serge Hallyn (serge-hallyn)
Revision history for this message
Nigel Jones (dev-nigelj) wrote :

Sorry, I forgot to mention that since my bug report, it has been fixed upstream, looks like:

commit eebf932e9418b8a626488ced40294e4f5d580e49
Author: Jim Meyering <email address hidden>
Date: Wed Dec 16 14:15:50 2009 +0100

    qemu_driver.c: avoid NULL dereference upon disk-op failure

    * src/qemu/qemu_driver.c (qemudDomainAttachDevice): Call
    virCgroupDenyDevicePath only if cgroup is non-NULL.

(when I tested the bug existed in the current release of libvirt that I had, but I've tested this week that Maverick is fixed and was i the process of getting a SRU debdiff ready)

Changed in libvirt (Ubuntu Maverick):
status: Confirmed → Invalid
Revision history for this message
Nigel Jones (dev-nigelj) wrote :

Attached is proposed debdiff for SRU.

Also includes updated patch for Bug #571093

description: updated
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Thanks, Nigel.

Uploading to lucid-proposed now.

Changed in libvirt (Ubuntu Lucid):
status: Confirmed → In Progress
assignee: nobody → Dustin Kirkland (kirkland)
Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Accepted libvirt into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in libvirt (Ubuntu Lucid):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Nigel Jones (dev-nigelj) wrote :

FYI: The uploaded package still works:

njones@fantail:~$ apt-cache policy libvirt-bin
libvirt-bin:
  Installed: 0.7.5-5ubuntu27.3
  Candidate: 0.7.5-5ubuntu27.3
  Version table:
 *** 0.7.5-5ubuntu27.3 0
        500 http://nz.archive.ubuntu.com/ubuntu/ lucid-proposed/main Packages
        100 /var/lib/dpkg/status
     0.7.5-5ubuntu27.2 0
        500 http://nz.archive.ubuntu.com/ubuntu/ lucid-updates/main Packages
     0.7.5-5ubuntu27 0
        500 http://nz.archive.ubuntu.com/ubuntu/ lucid/main Packages
njones@fantail:~$ virsh list; virsh destroy lucid-test; virsh start lucid-test; virsh attach-device lucid-test /tmp/455832.xml; virsh attach-device lucid-test /tmp/455832.xml
 Id Name State
----------------------------------

error: Failed to destroy domain lucid-test
error: Requested operation is not valid: domain is not running

Domain lucid-test started

Device attached successfully

error: Failed to attach device from /tmp/455832.xml
error: operation failed: target vdc already exists

Revision history for this message
Nigel Jones (dev-nigelj) wrote :

To be clear, when I said "FYI: The uploaded package still works:" I had meant:

The uploaded package contains the correct fix.

The 'target vdc already exists' error message is the correct error message for this patch.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 0.7.5-5ubuntu27.3

---------------
libvirt (0.7.5-5ubuntu27.3) lucid-proposed; urgency=low

  * debian/patches/9024-free-memory-for-invalid-devices.patch: updated
    to match upstream patch which includes a fix for an entry path
    not found originally, LP: #571093
  * debian/patches/9025-avoid-NULL-dereference-upon-disk-op-fail.patch:
    backport upstream patch to avoid failures when attempting to attach
    a disk or image twice. LP: #455832
 -- Nigel Jones <email address hidden> Fri, 27 Aug 2010 02:40:34 +1200

Changed in libvirt (Ubuntu Lucid):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.