Re: [Qemu-devel] "iothread: release iothread around aio_poll" causes random hangs at startup

[Fam Zheng <famz@redhat.com>]

On Tue, 06/02 16:36, Christian Borntraeger wrote:
> Paolo,
> 
> I bisected 
> commit a0710f7995f914e3044e5899bd8ff6c43c62f916
> Author:     Paolo Bonzini <pbonzini@redhat.com>
> AuthorDate: Fri Feb 20 17:26:52 2015 +0100
> Commit:     Kevin Wolf <kwolf@redhat.com>
> CommitDate: Tue Apr 28 15:36:08 2015 +0200
> 
>     iothread: release iothread around aio_poll
> 
> to cause a problem with hanging guests.
> 
> Having many guests all with a kernel/ramdisk (via -kernel) and
> several null block devices will result in hangs. All hanging 
> guests are in partition detection code waiting for an I/O to return
> so very early maybe even the first I/O.
> 
> Reverting that commit "fixes" the hangs.
> Any ideas?

Christian, I can't reproduce this on my x86 box with virtio-blk-pci. Do you
have a reproducer for x86? Or could you collect backtraces for all the threads
in QEMU when it hangs?

My long shot is that the main loop is blocked at aio_context_acquire(ctx),
while the iothread of that ctx is blocked at aio_poll(ctx, blocking).

Thanks,
Fam

> 
> Christian
> 
> PS: A guest xml looks like
> 
> 
> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>   <name>test2</name>
>   <uuid>13bd8253-9abb-4be8-9399-73b899762aaa</uuid>
>   <memory unit='KiB'>286720</memory>
>   <currentMemory unit='KiB'>286720</currentMemory>
>   <vcpu placement='static'>10</vcpu>
>   <iothreads>8</iothreads>
>   <cputune>
>     <shares>12</shares>
>   </cputune>
>   <os>
>     <type arch='s390x' machine='s390-ccw-virtio'>hvm</type>
>     <kernel>/boot/vmlinux-4.0.0+</kernel>
>     <initrd>/boot/ramdisk.reboot</initrd>
>     <cmdline>root=/dev/ram0</cmdline>
>     <boot dev='hd'/>
>   </os>
>   <clock offset='utc'/>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>preserve</on_crash>
>   <devices>
>     <controller type='usb' index='0' model='none'/>
>     <console type='pty'>
>       <target type='sclp' port='0'/>
>     </console>
>     <memballoon model='none'/>
>   </devices>
>   <qemu:commandline>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null1,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null1,serial=null1,iothread=iothread1'/>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null2,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null2,serial=null2,iothread=iothread2'/>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null3,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null3,serial=null3,iothread=iothread3'/>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null4,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null4,serial=null4,iothread=iothread4'/>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null5,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null5,serial=null5,iothread=iothread5'/>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null6,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null6,serial=null6,iothread=iothread6'/>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null7,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null7,serial=null7,iothread=iothread7'/>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null8,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null8,serial=null8,iothread=iothread8'/>
>   </qemu:commandline>
> </domain>
> 
>