[Qemu-devel] q35 migration broken

[Qemu-devel] q35 migration broken

[Dr. David Alan Gilbert]

[-- Attachment #1: Type: text/plain, Size: 3458 bytes --]

Hi,
  I'm seeing a breakage on q35 migration on head (and possibly older
but certainly head; it's also on a 2.5.0 world I've got with a bunch
of patches but I've not tried a clean 2.5.0 yet).

It looks like some type of interrupt screwup; with a virtio-net device
I get a:
  BUG: soft lockup - CPU#0 stuck for 22s!
  ...  virtnet_config_changed_work 

but if I swap that out for an e1000 I get:
  Disabling IRQ #22

  and various timeouts on e1000 and cdrom (scsi).
The guest kind of limps along with an existing terminal scrolling dmesg -w output.

This is an f23 guest on a rhel7.2-ish host; with the guest sitting an idle
(MATE) Gui.

i440fx works.

qemu     30823     1 15 14:51 ?        00:00:07 /opt/qemu-head/bin/qemu-system-x86_64 -name f23-q35 -S -machine pc-q35-2.6,accel=kvm,usb=off,vmport=off -cpu SandyBridge -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 3cc93d9b-9b87-4472-847c-25cea2bfc51f -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-12-f23-q35/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x4 -drive file=/home/vms/f23-q35.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.2,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-scsi0-0-0-0,readonly=on -device scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 -netdev tap,fd=25,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:98:12:7d,bus=pci.2,addr=0x1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-12-f23-q35/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,bus=pcie.0,addr=0x1 -device intel-hda,id=sound0,bus=pci.2,addr=0x2 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x6 -msg timestamp=on

(Attaching libvirt xml that generated that lot).

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

[-- Attachment #2: f23-q35.xml --]
[-- Type: text/xml, Size: 4768 bytes --]

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit f23-q35
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>f23-q35</name>
  <uuid>3cc93d9b-9b87-4472-847c-25cea2bfc51f</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-2.6'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/opt/qemu-head/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/home/vms/f23-q35.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='sda' bus='scsi'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:98:12:7d'/>
      <source network='default'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <source mode='bind'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Re: [Qemu-devel] q35 migration broken

[Dr. David Alan Gilbert]

* Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
> * Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
> > Hi,
> >   I'm seeing a breakage on q35 migration on head (and possibly older
> > but certainly head; it's also on a 2.5.0 world I've got with a bunch
> > of patches but I've not tried a clean 2.5.0 yet).
> > 
> > It looks like some type of interrupt screwup; with a virtio-net device
> > I get a:
> >   BUG: soft lockup - CPU#0 stuck for 22s!
> >   ...  virtnet_config_changed_work 
> > 
> > but if I swap that out for an e1000 I get:
> >   Disabling IRQ #22
> > 
> >   and various timeouts on e1000 and cdrom (scsi).
> > The guest kind of limps along with an existing terminal scrolling dmesg -w output.
> > 
> > This is an f23 guest on a rhel7.2-ish host; with the guest sitting an idle
> > (MATE) Gui.
> 
> Also broken with 2.4.1 and 2.5.1 (with pc-q35-2.4 machine type);
> see a screen shot attached; note:
>    a) The large count on irq 22 (enp2s1) on cpu1
>    b) The large count on virtio2-config on cpu1
>    c) The count of 'Deferred Error APIC interrupts'.

OK, this seems to be the i82801b11-bridge; if I remove it from the config
it all works.

My minimum config that fails so far is:
/opt/qemu-head/bin/qemu-system-x86_64 -nographic -machine pc-q35-2.6,accel=kvm,usb=off,vmport=off -cpu SandyBridge -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 \
 -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e \
 -drive id=image,file=/home/vms/f23-serial.qcow2,if=none,cache=none \
 -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x5,drive=image,id=virtio-disk0,bootindex=1 \
 $*

if I flip the i82801b11-bridge for a pci-bridge  then it works.

Dave

--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] q35 migration broken

[Marcel Apfelbaum]

On 04/01/2016 06:54 PM, Dr. David Alan Gilbert wrote:
> * Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
>> * Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
>>> Hi,
>>>    I'm seeing a breakage on q35 migration on head (and possibly older
>>> but certainly head; it's also on a 2.5.0 world I've got with a bunch
>>> of patches but I've not tried a clean 2.5.0 yet).
>>>
>>> It looks like some type of interrupt screwup; with a virtio-net device
>>> I get a:
>>>    BUG: soft lockup - CPU#0 stuck for 22s!
>>>    ...  virtnet_config_changed_work
>>>
>>> but if I swap that out for an e1000 I get:
>>>    Disabling IRQ #22
>>>
>>>    and various timeouts on e1000 and cdrom (scsi).
>>> The guest kind of limps along with an existing terminal scrolling dmesg -w output.
>>>
>>> This is an f23 guest on a rhel7.2-ish host; with the guest sitting an idle
>>> (MATE) Gui.
>>
>> Also broken with 2.4.1 and 2.5.1 (with pc-q35-2.4 machine type);
>> see a screen shot attached; note:
>>     a) The large count on irq 22 (enp2s1) on cpu1
>>     b) The large count on virtio2-config on cpu1
>>     c) The count of 'Deferred Error APIC interrupts'.
>
> OK, this seems to be the i82801b11-bridge; if I remove it from the config
> it all works.
>
> My minimum config that fails so far is:
> /opt/qemu-head/bin/qemu-system-x86_64 -nographic -machine pc-q35-2.6,accel=kvm,usb=off,vmport=off -cpu SandyBridge -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 \
>   -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e \
>   -drive id=image,file=/home/vms/f23-serial.qcow2,if=none,cache=none \
>   -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x5,drive=image,id=virtio-disk0,bootindex=1 \
>   $*
>
> if I flip the i82801b11-bridge for a pci-bridge  then it works.

Hi Dave,
That's good news. I see we don't have a vmstate for this bridge, but we do have one for the regular pci-bridge.
Maybe this is the problem?

Thanks,
Marcel

>
> Dave
>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>

Re: [Qemu-devel] q35 migration broken

[Dr. David Alan Gilbert]

* Marcel Apfelbaum (marcel@redhat.com) wrote:
> On 04/01/2016 06:54 PM, Dr. David Alan Gilbert wrote:
> >* Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
> >>* Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
> >>>Hi,
> >>>   I'm seeing a breakage on q35 migration on head (and possibly older
> >>>but certainly head; it's also on a 2.5.0 world I've got with a bunch
> >>>of patches but I've not tried a clean 2.5.0 yet).
> >>>
> >>>It looks like some type of interrupt screwup; with a virtio-net device
> >>>I get a:
> >>>   BUG: soft lockup - CPU#0 stuck for 22s!
> >>>   ...  virtnet_config_changed_work
> >>>
> >>>but if I swap that out for an e1000 I get:
> >>>   Disabling IRQ #22
> >>>
> >>>   and various timeouts on e1000 and cdrom (scsi).
> >>>The guest kind of limps along with an existing terminal scrolling dmesg -w output.
> >>>
> >>>This is an f23 guest on a rhel7.2-ish host; with the guest sitting an idle
> >>>(MATE) Gui.
> >>
> >>Also broken with 2.4.1 and 2.5.1 (with pc-q35-2.4 machine type);
> >>see a screen shot attached; note:
> >>    a) The large count on irq 22 (enp2s1) on cpu1
> >>    b) The large count on virtio2-config on cpu1
> >>    c) The count of 'Deferred Error APIC interrupts'.
> >
> >OK, this seems to be the i82801b11-bridge; if I remove it from the config
> >it all works.
> >
> >My minimum config that fails so far is:
> >/opt/qemu-head/bin/qemu-system-x86_64 -nographic -machine pc-q35-2.6,accel=kvm,usb=off,vmport=off -cpu SandyBridge -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 \
> >  -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e \
> >  -drive id=image,file=/home/vms/f23-serial.qcow2,if=none,cache=none \
> >  -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x5,drive=image,id=virtio-disk0,bootindex=1 \
> >  $*
> >
> >if I flip the i82801b11-bridge for a pci-bridge  then it works.
> 
> Hi Dave,
> That's good news. I see we don't have a vmstate for this bridge, but we do have one for the regular pci-bridge.
> Maybe this is the problem?

Yeh that's one of my suspicions; but how do device classes work?
If a i82801b11-bridge is a subclass of pci-bridge should it just
pick up the vmstate of the subclass magically?

Dave
P.S. I've filed this as RH bz 1323273
> 
> Thanks,
> Marcel
> 
> >
> >Dave
> >
> >--
> >Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] q35 migration broken

[Marcel Apfelbaum]

On 04/01/2016 08:01 PM, Dr. David Alan Gilbert wrote:
> * Marcel Apfelbaum (marcel@redhat.com) wrote:
>> On 04/01/2016 06:54 PM, Dr. David Alan Gilbert wrote:
>>> * Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
>>>> * Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
>>>>> Hi,
>>>>>    I'm seeing a breakage on q35 migration on head (and possibly older
>>>>> but certainly head; it's also on a 2.5.0 world I've got with a bunch
>>>>> of patches but I've not tried a clean 2.5.0 yet).
>>>>>
>>>>> It looks like some type of interrupt screwup; with a virtio-net device
>>>>> I get a:
>>>>>    BUG: soft lockup - CPU#0 stuck for 22s!
>>>>>    ...  virtnet_config_changed_work
>>>>>
>>>>> but if I swap that out for an e1000 I get:
>>>>>    Disabling IRQ #22
>>>>>
>>>>>    and various timeouts on e1000 and cdrom (scsi).
>>>>> The guest kind of limps along with an existing terminal scrolling dmesg -w output.
>>>>>
>>>>> This is an f23 guest on a rhel7.2-ish host; with the guest sitting an idle
>>>>> (MATE) Gui.
>>>>
>>>> Also broken with 2.4.1 and 2.5.1 (with pc-q35-2.4 machine type);
>>>> see a screen shot attached; note:
>>>>     a) The large count on irq 22 (enp2s1) on cpu1
>>>>     b) The large count on virtio2-config on cpu1
>>>>     c) The count of 'Deferred Error APIC interrupts'.
>>>
>>> OK, this seems to be the i82801b11-bridge; if I remove it from the config
>>> it all works.
>>>
>>> My minimum config that fails so far is:
>>> /opt/qemu-head/bin/qemu-system-x86_64 -nographic -machine pc-q35-2.6,accel=kvm,usb=off,vmport=off -cpu SandyBridge -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 \
>>>   -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e \
>>>   -drive id=image,file=/home/vms/f23-serial.qcow2,if=none,cache=none \
>>>   -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x5,drive=image,id=virtio-disk0,bootindex=1 \
>>>   $*
>>>
>>> if I flip the i82801b11-bridge for a pci-bridge  then it works.
>>
>> Hi Dave,
>> That's good news. I see we don't have a vmstate for this bridge, but we do have one for the regular pci-bridge.
>> Maybe this is the problem?
>
> Yeh that's one of my suspicions; but how do device classes work?
> If a i82801b11-bridge is a subclass of pci-bridge should it just
> pick up the vmstate of the subclass magically?

I am not sure, I think you need something like:

static const VMStateDescription i82801b11-bridge_dev_vmstate = {
     .name = ""i82801b11_bridge,
     .fields = (VMStateField[]) {
         VMSTATE_PCI_DEVICE(parent_obj, PCIBridge),
         VMSTATE_END_OF_LIST()
     }
};



>
> Dave
> P.S. I've filed this as RH bz 1323273

good, we can track this now.

Thanks,
Marcel

>>
>> Thanks,
>> Marcel
>>
>>>
>>> Dave
>>>
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>