virsh save and virsh managedsave failed with cpu mode = host-passthrough

Bug #1262641 reported by Swe W Aung
56
This bug affects 10 people
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Fix Released
High
Unassigned
Trusty
Fix Released
High
Unassigned

Bug Description

========================================================================
SRU Justification:
1. Impact: virsh save fails when cpu mode is passthrough
2. Test case: create a VM with <cpu mode='host-passthrough'> (see below for an example). Run
virsh managed save <vm domain> --bypass-cache --verbose
3. Regression potential: The patch is minimal and should not affect any VMs which are not cpud mode=passthrough
========================================================================

Environment:
libvirt_cpu = host-passthrough
libvirt0, libvirt-bin 1.1.1-0ubuntu8~cloud2
qemu-kvm, qemu-utils, qemu-common 1.0+noroms-0ubuntu14.12
kvm 1:84+dfsg-0ubuntu16+1.0+noroms+0ubuntu14.12
apparmor 2.7.102-0ubuntu3.9
Kernel 3.5.0-44-generic
lsb-release Ubuntu 12.04.3 LTS

Problem:

I have a VM created with cpu mode = host-passthrough.

When I do
virsh managed save <vm domain> --bypass-cache --verbose,
the vm was left in shutoff, instead of runing and not able to start up even I do virsh start <vm domain>

#virsh start instance-0000b600
error: Failed to start domain instance-0000b600
error: internal error: cannot load AppArmor profile 'libvirt-ba53a370-259f-4f47-b3e5-c76eddf4ec86'

in libvirtd.log
2013-12-19 12:08:49.169+0000: 27906: error : virCommandWait:2348 : internal error: Child process (/usr/lib/libvirt/virt-aa-helper -p 0 -r -u libvirt-ba53a370-259f-4f47-b3e5-c76eddf4ec86) unexpected exit status 1: virt-aa-helper: error: could not parse XML
virt-aa-helper: error: could not get VM definition

2013-12-19 12:08:49.169+0000: 27906: error : AppArmorGenSecurityLabel:451 : internal error: cannot load AppArmor profile 'libvirt-ba53a370-259f-4f47-b3e5-c76eddf4ec86'
2013-12-19 12:08:49.170+0000: 27906: warning : virAuditSend:135 : Failed to send audit message virt=kvm resrc=disk reason=start vm="instance-0000b600" uuid=ba53a370-259f-4f47-b3e5-c76eddf4ec86 old-disk="?" new-disk="/var/lib/nova/instances/ba53a370-259f-4f47-b3e5-c76eddf4ec86/disk": Operation not permitted
2013-12-19 12:08:49.170+0000: 27906: warning : virAuditSend:135 : Failed to send audit message virt=kvm resrc=disk reason=start vm="instance-0000b600" uuid=ba53a370-259f-4f47-b3e5-c76eddf4ec86 old-disk="?" new-disk="/var/lib/nova/instances/ba53a370-259f-4f47-b3e5-c76eddf4ec86/disk.local": Operation not permitted
2013-12-19 12:08:49.170+0000: 27906: warning : virAuditSend:135 : Failed to send audit message virt=kvm resrc=net reason=start vm="instance-0000b600" uuid=ba53a370-259f-4f47-b3e5-c76eddf4ec86 old-net=? new-net=fa:16:3e:0d:17:12: Operation not permitted
2013-12-19 12:08:49.170+0000: 27906: warning : virAuditSend:135 : Failed to send audit message virt=kvm resrc=mem reason=start vm="instance-0000b600" uuid=ba53a370-259f-4f47-b3e5-c76eddf4ec86 old-mem=0 new-mem=16777216: Operation not permitted
2013-12-19 12:08:49.170+0000: 27906: warning : virAuditSend:135 : Failed to send audit message virt=kvm resrc=vcpu reason=start vm="instance-0000b600" uuid=ba53a370-259f-4f47-b3e5-c76eddf4ec86 old-vcpu=0 new-vcpu=4: Operation not permitted
2013-12-19 12:08:49.170+0000: 27906: warning : virAuditSend:135 : Failed to send audit message virt=kvm op=start reason=restored vm="instance-0000b600" uuid=ba53a370-259f-4f47-b3e5-c76eddf4ec86 vm-pid=-1: Operation not permitted
2013-12-19 12:08:49.170+0000: 27906: warning : qemuDomainSaveImageStartVM:5452 : failed to restore save state label on /var/lib/libvirt/qemu/save/instance-0000b600.save

If I restart libvirt-bin at this point, the error complains about Non-empty feature list specified without CPU

2013-12-19 12:05:58.786+0000: 27913: info : libvirt version: 1.1.1
2013-12-19 12:05:58.786+0000: 27913: error : virCPUDefParseXML:367 : XML error: Non-empty feature list specified without CPU model
2013-12-19 12:05:58.787+0000: 27913: error : virCPUDefParseXML:367 : XML error: Non-empty feature list specified without CPU model
2013-12-19 12:05:58.788+0000: 27913: error : virCPUDefParseXML:367 : XML error: Non-empty feature list specified without CPU model
2013-12-19 12:05:58.789+0000: 27913: error : virCPUDefParseXML:367 : XML error: Non-empty feature list specified without CPU model
2013-12-19 12:05:58.790+0000: 27913: error : virCPUDefParseXML:367 : XML error: Non-empty feature list specified without CPU model

When I check xml of save file, the CPU model information are missing.

#virsh save-image-dumpxml instance-0000b600

 <cpu mode='host-passthrough'>
    <feature policy='require' name='perfctr_nb'/>
    <feature policy='require' name='perfctr_core'/>
    <feature policy='require' name='topoext'/>
    <feature policy='require' name='nodeid_msr'/>
    <feature policy='require' name='lwp'/>
    <feature policy='require' name='wdt'/>
    <feature policy='require' name='skinit'/>
    <feature policy='require' name='ibs'/>
    <feature policy='require' name='osvw'/>
    <feature policy='require' name='cr8legacy'/>
    <feature policy='require' name='extapic'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='fxsr_opt'/>
    <feature policy='require' name='mmxext'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='vme'/>
  </cpu>

xml of the vm

#virsh dumpxml <vm domain> --update-cpu

  <cpu mode='host-passthrough' match='minimum'>
    <model>Opteron_G4</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='perfctr_nb'/>
    <feature policy='require' name='perfctr_core'/>
    <feature policy='require' name='topoext'/>
    <feature policy='require' name='nodeid_msr'/>
    <feature policy='require' name='lwp'/>
    <feature policy='require' name='wdt'/>
    <feature policy='require' name='skinit'/>
    <feature policy='require' name='ibs'/>
    <feature policy='require' name='osvw'/>
    <feature policy='require' name='cr8legacy'/>
    <feature policy='require' name='extapic'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='fxsr_opt'/>
    <feature policy='require' name='mmxext'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='vme'/>
  </cpu>

I did not have the problem with libvirt 0.9.8 earlier, before upgrade libvirt to 1.1.1

PS. I found the similar report here also https://www.redhat.com/archives/libvirt-users/2013-November/msg00043.html

Thanks
regards,

Swe Aung

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in libvirt (Ubuntu):
status: New → Confirmed
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

In the libvirtd.log, all I see is

2013-12-24 17:05:23.406+0000: 1286: error : virSecurityDACRestoreSecurityFileLabel:307 : cannot resolve symlink /tmp/tmp1LJkjq/device_disk.img: No such file or directory

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Sorry, I attached that comment to the wrong bug.

Changed in libvirt (Ubuntu):
importance: Undecided → High
Revision history for this message
s10 (vlad-esten) wrote :

There is patch to workaround for this bug, that also fixes similar error after live migration of vm with cpu mode "host-passthrough".

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "dont-fail-without-cpu-model.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Revision history for this message
Serge Hallyn (serge-hallyn) wrote : Re: [Bug 1262641] Re: virsh save and virsh managedsave failed with cpu mode = host-passthrough

If I understand correctly, the new libvirt version I just pushed should
fix this bug. Please re-mark Confirmed if I am wrong and more patches
are needed.

 status: fixreleased

Changed in libvirt (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Blair Bethwaite (blair-bethwaite) wrote :

Serge, sorry if I've missed an obvious answer to this... but, where did you push the updated libvirt?

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

libvirt | 1.2.1-0ubuntu4 | trusty

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

I'm sorry, this was the wrong bug!

 status: triaged

Revision history for this message
Blair Bethwaite (blair-bethwaite) wrote :

Oh I see: https://launchpad.net/ubuntu/+source/libvirt/1.2.1-0ubuntu4

We really need a package compatible with Precise to fix this. We may be able to test the fix if we can cherry-pick the trusty version of libvirt without conflict, however we'll still require a backport to fix this in production. Will perhaps have to build our own package if a Precise backport is unavailable...

Changed in libvirt (Ubuntu):
status: Fix Released → Triaged
Revision history for this message
s10 (vlad-esten) wrote :

Unfortunately, in libvirt 1.2.1-0ubuntu4 this bug still exists. And I don't realize how the patch that enables host-passthrough for ARM/AArch64 (not amd64) can fix this bug for other architectures (i386, amd64, powerpc etc).

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Quoting s10 (<email address hidden>):
> Unfortunately, in libvirt 1.2.1-0ubuntu4 this bug still exists. And I
> don't realize how the patch that enables host-passthrough for
> ARM/AArch64 (not amd64) can fix this bug for other architectures (i386,
> amd64, powerpc etc).

Yes, I commented that in the wrong bug, sorry.

Revision history for this message
Blair Bethwaite (blair-bethwaite) wrote :

I am now thoroughly confused... so, we should try Vlad's patch and ignore Serge's updated package?

Revision history for this message
Simon Elmir (nerd65536) wrote :

Serge's package does not include Vlad's patch.
We need a package with Vlad's patch in it so it can be tested.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Has this patch been discussed upstream? I'm surprised it is not in 1.2.2.

Revision history for this message
s10 (vlad-esten) wrote :

This bug still present in libvirt 1.2.2. And patch has not been discussed upstream.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Quoting s10 (<email address hidden>):
> This bug still present in libvirt 1.2.2. And patch has not been
> discussed upstream.

Why has the patch not been discussed upstream? In general while we are
happy to take important patches which are not yet in a release, we
want them to be at least mentioned on the list, and preferably committed
to git. Three advantages there are that they have a Signed-off-by,
have a description, and have a chance for upstream to point out
potential problems. None of those have happened here. If we put this
into our package, were you expecting us to then push it upstream? We
again could not do so without a signed-off-by and a good description,
and since we didn't know that was your goal, it wouldn't have happened
and so we would be dropping the patch right now (before release).

Please discuss this patch upstream.

Revision history for this message
Swe W Aung (sirswa) wrote :

May I know if the patched has been discussed upstream and which version will include the patch?

for a workaround(from our colleague), we have included cpu model name in /usr/share/libvirt/cpu_map.xml base on the information we got from /proc/cpuinfo and virsh capabilities

regards,

Revision history for this message
Thomas Vachon (vachon) wrote :

Any update here? This is a massive blocker to nova-compute for us, it breaks all glance snapshots and even some reboots.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

This is fixed upstream by commit dd69a14f. It will be included in the next merge into vivid, and I'll SRU it into Trusty hopefully next week.

Revision history for this message
Thomas Vachon (vachon) wrote :

Were you able to SRU it? We are in desperate need here

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

I'm not sure what comment #20 was about - the fix analogous to the attached patch is adff345e1ec9a6f528731ae40168a76b8e7620e0 (which is upstream).

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@vachon,

the original bug report was about cloud archive versions. Which version are you looking for the fix to be in? I will push a test package for trusty so you can verify that the fix actually suffices, but I can't yet do the SRU as we're waiting for anohter version to clear from trusty-proposed.

no longer affects: libvirt (Ubuntu Precise)
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

I've uploaded a test package to ppa:serge-hallyn/virt. Could you please test whether that fixes the issue for you?

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in libvirt (Ubuntu Trusty):
status: New → Confirmed
Revision history for this message
Tony Link (tlink) wrote :

This bug is a show stopper for us as well. Juno/Utopic and we can't take snapshots.

Appears there's a fix here: http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=de0aeafe9ce3eb414c8b5d3aa8995d776a2952de

root@s# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.10
Release: 14.10
Codename: utopic
root@s# dpkg --list | grep libvirt
ii libvirt-bin 1.2.8-0ubuntu11.3~ppa2 amd64 programs for the libvirt library
ii libvirt0 1.2.8-0ubuntu11.3~ppa2 amd64 library for interfacing with different virtualization systems
ii nova-compute-libvirt 1:2014.2.1-0ubuntu1 all OpenStack Compute - compute node libvirt support
ii python-libvirt 1.2.8-0ubuntu2 amd64 libvirt Python bindings
root@s# apt-cache policy nova-compute-libvirt
nova-compute-libvirt:
  Installed: 1:2014.2.1-0ubuntu1
  Candidate: 1:2014.2.1-0ubuntu1
  Version table:
 *** 1:2014.2.1-0ubuntu1 0
        500 http://us.archive.ubuntu.com/ubuntu/ utopic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1:2014.2-0ubuntu1 0
        500 http://us.archive.ubuntu.com/ubuntu/ utopic/main amd64 Packages
root@s# apt-cache policy libvirtd0
N: Unable to locate package libvirtd0
root@c8d8-c0n1-c:~# apt-cache policy libvirt0
libvirt0:
  Installed: 1.2.8-0ubuntu11.3~ppa2
  Candidate: 1.2.8-0ubuntu11.3~ppa2
  Version table:
 *** 1.2.8-0ubuntu11.3~ppa2 0
        500 http://ppa.launchpad.net/serge-hallyn/virt/ubuntu/ utopic/main amd64 Packages
        100 /var/lib/dpkg/status
     1.2.8-0ubuntu11.2 0
        500 http://us.archive.ubuntu.com/ubuntu/ utopic-updates/main amd64 Packages
     1.2.8-0ubuntu11.1 0
        500 http://security.ubuntu.com/ubuntu/ utopic-security/main amd64 Packages
     1.2.8-0ubuntu11 0
        500 http://us.archive.ubuntu.com/ubuntu/ utopic/main amd64 Packages

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@tlink,

that patch looks nothing like the one posted in comment #4. That one is rather a subset of commit adff345e1ec9a6f528731ae40168a76b8e7620e0.

Could you please verify that you're talking about the same bug?

Could you check whether the package built for trusty in ppa:serge-hallyn-virt (mentioned in comment #24) fixes the issue?

Revision history for this message
Tony Link (tlink) wrote :

@serge,

I after some review, I don't think it may be the same bug . I saw the failure for 'virsh managedsave' and presumed it was for the same.

Apologies.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@tlink,

Would you mind opening a new high priority bug for your issue?

Revision history for this message
Thomas Vachon (vachon) wrote :

@serge,

Yes, I am on Trusty and the Cloud Archive version (openstack juno for us). I'll deploy it to one server and see how it looks

Revision history for this message
Thomas Vachon (vachon) wrote :

I was able to take the PPA libvirt (1.2.2-0ubuntu13.1.10~ppa1) package for Trusty and upgrade libvirt (and associated packages). I restarted nova-compute (since I'm on Openstack).

After checking both apparmor_status before and after (ensuring it was enforcing and loaded), it appears it was fixed.

I tested by doing a nova snapshot (which calls the underlying libvirt call) and I did not see any apparmor issue. The instance came back as expected.

Changed in libvirt (Ubuntu):
status: Triaged → Fix Released
Changed in libvirt (Ubuntu Trusty):
importance: Undecided → High
description: updated
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello sirswa, or anyone else affected,

Accepted libvirt into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/libvirt/1.2.2-0ubuntu13.1.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in libvirt (Ubuntu Trusty):
status: Confirmed → Fix Committed
tags: added: verification-needed
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@sirswa,

could you please verify this fix works for you?

Revision history for this message
Swe W Aung (sirswa) wrote :

HI Serge

I will test it out at one of the spare node.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Can someone please verify this fix soon? Otherwise we will abort this so we can proceed with other SRUs.

Revision history for this message
s10 (vlad-esten) wrote :

I confirm that fix (libvirt 1.2.2-0ubuntu13.1.10) works. The bug is fixed in libvirt from trusty-proposed. Thank you!

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 1.2.2-0ubuntu13.1.10

---------------
libvirt (1.2.2-0ubuntu13.1.10) trusty-proposed; urgency=medium

  * 9035-qemu-snapshot-save-persistent-domain-config: upstream fix for a
    regression where persistent domain config was not saved after an external
    snapshot. (LP: #1403841)
  * 9036-dont-fail-without-cpu-model.patch: fix virsh safe with cpu mode =
    host-passthrough (LP: #1262641)
 -- Serge Hallyn <email address hidden> Tue, 10 Feb 2015 14:34:16 -0600

Changed in libvirt (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Adam Conrad (adconrad) wrote : Update Released

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.