Comment 115 for bug 997978

Revision history for this message
kraig (kamador) wrote : Re: [Bug 997978] Re: KVM images lose connectivity with bridged network

Same here, thank you everyone!

--
Kraig Amador

On Friday, November 2, 2012 at 5:42 AM, Jonathan Tullett wrote:

> This bug is considered fixed for me. Not a single network glitch since
> installing the package from PPA. Many thanks to the development team!
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/997978
>
> Title:
> KVM images lose connectivity with bridged network
>
> Status in OpenStack Compute (Nova):
> Invalid
> Status in “qemu-kvm” package in Ubuntu:
> Fix Released
> Status in “qemu-kvm” source package in Precise:
> Fix Released
>
> Bug description:
> =========================================
> SRU Justification:
> 1. Impact: networking breaks after awhile in kvm guests using virtio networking
> 2. Development fix: The bug was fixed upstream and the fix picked up in a new
> merge.
> 3. Stable fix: 3 virtio patches are cherrypicked from upstream:
> a821ce5 virtio: order index/descriptor reads
> 92045d8 virtio: add missing mb() on enable notification
> a281ebc virtio: add missing mb() on notification
> 4. Test case: Create a bridge enslaving the real NIC, and use that as the bridge
> for a kvm instance with virtio networking. See comment #44 for specific test
> case.
> 5. Regression potential: Should be low as several people have tested the fixed
> package under heavy load.
> =========================================
>
> System:
> -----------
> Dell R410 Dual processor 2.4Ghz w/16G RAM
> Distributor ID: Ubuntu
> Description: Ubuntu 12.04 LTS
> Release: 12.04
> Codename: precise
>
> Setup:
> ---------
> We're running 3 KVM guests, all Ubuntu 12.04 LTS using bridged networking.
>
> From the host:
> # cat /etc/network/interfaces
> auto br0
> iface br0 inet static
> address 212.XX.239.98
> netmask 255.255.255.240
> gateway 212.XX.239.97
> bridge_ports eth0
> bridge_fd 9
> bridge_hello 2
> bridge_maxage 12
> bridge_stp off
>
> # ifconfig eth0
> eth0 Link encap:Ethernet HWaddr d4:ae:52:84:2d:5a
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:11278363 errors:0 dropped:3128 overruns:0 frame:0
> TX packets:14437384 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:4115980743 (4.1 GB) TX bytes:5451961979 (5.4 GB)
> Interrupt:36 Memory:da000000-da012800
>
> # ifconfig br0
> br0 Link encap:Ethernet HWaddr d4:ae:52:84:2d:5a
> inet addr:212.XX.239.98 Bcast:212.XX.239.111 Mask:255.255.255.240
> inet6 addr: fe80::d6ae:52ff:fe84:2d5a/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:1720861 errors:0 dropped:0 overruns:0 frame:0
> TX packets:1708622 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:210152198 (210.1 MB) TX bytes:300858508 (300.8 MB)
>
> # brctl show
> bridge name bridge id STP enabled interfaces
> br0 8000.d4ae52842d5a no eth0
>
> I have no default network configured to autostart in libvirt as we're using bridged networking:
> # virsh net-list --all
> Name State Autostart
> -----------------------------------------
> default inactive no
>
> # arp
> Address HWtype HWaddress Flags Mask Iface
> mailer03.xxxx.com (http://mailer03.xxxx.com) ether 52:54:00:82:5f:0f C br0
> mailer01.xxxx.com (http://mailer01.xxxx.com) ether 52:54:00:d2:f7:31 C br0
> mailer02.xxxx.com (http://mailer02.xxxx.com) ether 52:54:00:d3:8f:91 C br0
> dxi-gw2.xxxx.com (http://dxi-gw2.xxxx.com) ether 00:1a:30:2a:b1:c0 C br0
>
> From one of the guests:
> <domain type='kvm' id='4'>
> <name>mailer01</name>
> <uuid>d41d1355-84e8-ae23-e84e-227bc0231b97</uuid>
> <memory>2097152</memory>
> <currentMemory>2097152</currentMemory>
> <vcpu>1</vcpu>
> <os>
> <type arch='x86_64' machine='pc-1.0'>hvm</type>
> <boot dev='hd'/>
> </os>
> <features>
> <acpi/>
> </features>
> <clock offset='utc'/>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>destroy</on_crash>
> <devices>
> <emulator>/usr/bin/kvm</emulator>
> <disk type='file' device='disk'>
> <driver name='qemu' type='raw'/>
> <source file='/dev/mapper/vg_main-mailer01--root'/>
> <target dev='hda' bus='ide'/>
> <alias name='ide0-0-0'/>
> <address type='drive' controller='0' bus='0' unit='0'/>
> </disk>
> <disk type='file' device='disk'>
> <driver name='qemu' type='raw'/>
> <source file='/dev/mapper/vg_main-mailer01--swap'/>
> <target dev='hdb' bus='ide'/>
> <alias name='ide0-0-1'/>
> <address type='drive' controller='0' bus='0' unit='1'/>
> </disk>
> <controller type='ide' index='0'>
> <alias name='ide0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
> </controller>
> <interface type='bridge'>
> <mac address='52:54:00:d2:f7:31'/>
> <source bridge='br0'/>
> <target dev='vnet0'/>
> <model type='virtio'/>
> <alias name='net0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
> </interface>
> <serial type='pty'>
> <source path='/dev/pts/0'/>
> <target port='0'/>
> <alias name='serial0'/>
> </serial>
> <console type='pty' tty='/dev/pts/0'>
> <source path='/dev/pts/0'/>
> <target type='serial' port='0'/>
> <alias name='serial0'/>
> </console>
> <input type='mouse' bus='ps2'/>
> <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
> <listen type='address' address='127.0.0.1'/>
> </graphics>
> <video>
> <model type='cirrus' vram='9216' heads='1'/>
> <alias name='video0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
> </video>
> <memballoon model='virtio'>
> <alias name='balloon0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
> </memballoon>
> </devices>
> <seclabel type='dynamic' model='apparmor' relabel='yes'>
> <label>libvirt-d41d1355-84e8-ae23-e84e-227bc0231b97</label>
> <imagelabel>libvirt-d41d1355-84e8-ae23-e84e-227bc0231b97</imagelabel>
> </seclabel>
> </domain>
>
> From within the guest:
> # cat /etc/network/interfaces
> # The primary network interface
> auto eth0
> iface eth0 inet static
> address 212.XX.239.100
> netmask 255.255.255.240
> network 212.XX.239.96
> broadcast 212.XX.239.111
> gateway 212.XX.239.97
>
> # ifconfig
> eth0 Link encap:Ethernet HWaddr 52:54:00:d2:f7:31
> inet addr:212.XX.239.100 Bcast:212.XX.239.111 Mask:255.255.255.240
> inet6 addr: fe80::5054:ff:fed2:f731/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:5631830 errors:0 dropped:0 overruns:0 frame:0
> TX packets:6683416 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:2027322829 (2.0 GB) TX bytes:2076698690 (2.0 GB)
>
> A commandline which starts the KVM guest:
> /usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 2048 -smp 1,sockets=1,cores=1,threads=1 -name mailer01 -uuid d41d1355-84e8-ae23-e84e-227bc0231b97 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/mailer01.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -drive file=/dev/mapper/vg_main-mailer01--root,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive file=/dev/mapper/vg_main-mailer01--swap,if=none,id=drive-ide0-0-1,format=raw -device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev tap,fd=18,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d2:f7:31,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
>
> Problem:
> ------------
> Periodically (at least once a day), one or more of the guests lose network connectivity. Ping responds with 'host unreachable', even from the dom host. Logging in via the serial console shows no problems: eth0 is up, can ping the local host, but no outside connectivity. Restart the network (/etc/init.d/networking restart) does nothing. Reboot the machine and it comes alive again.
>
> I've verified there's no arp games going on on the primary host (the
> arp tables remain the same before - when it had connectivity - and
> after - when it doesn't.
>
> This is a critical issue affecting production services on the latest
> LTS release of Ubuntu. It's similar to an issue which was 'resolved'
> in 10.04 but appears to have risen its ugly head again.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/997978/+subscriptions
>
>