[Qemu-devel] Behavior of QMP "query-block"

[Qemu-devel] Behavior of QMP "query-block"

[Bruno Alvisio]

Hello all,

Summary

I am using XEN hypervisor to run a HVM with a QEMU backed disk. After I
start the HVM I use QMP "query-block" command to see the devices of the VM.
Initially the command returns the disk that I set as part of the
configuration. After a few seconds the a DEVICE_DELETE event occurs (for
device nic0) and after that the "query-block" command returns an empty
result.

QEMU Version: 4.6.5
Arch: x86-64
QMP : 2.2.1

Note:
1. I was not performing any activity on the VM when the event occurs.
2. After the event, the QEMU process is still alive and the VM runs
normally.
3. This behavior is consistently reproduced.


Any help on how can I debug this issue would be greatly appreciated .

Thanks,

Bruno



DETAILED LOGS

I am using XEN hypervisor to run a HVM with QEMU backed disk. When I start
the HVM I see the following QEMU process started:

root      2199     1  0 18:57 ?        00:00:02
/usr/local/lib/xen/bin/qemu-system-i386
-xen-domid 3 -chardev socket,id=libxl-cmd,path=/var/
run/xen/qmp-libxl-3,server,nowait -no-shutdown -mon
chardev=libxl-cmd,mode=control -chardev socket,id=libxenstat-cmd,path=
/var/run/xen/qmp-libxenstat-3,server,nowait -mon
chardev=libxenstat-cmd,mode=control -nodefaults -no-user-config -name
debianL2 -vnc :0,to=99 -display none -serial pty -device
cirrus-vga,vgamem_mb=8 -boot order=d -device
rtl8139,id=nic0,netdev=net0,mac=00:16:3e:1b:d0:7e
-netdev type=tap,id=net0,ifname=vif3.0-emu,script=no,downscript=no -machine
xenfv -m 1016 -drive file=/home/balvisio/debian-
disk.img,if=ide,index=0,media=disk,format=raw,cache=writeback -drive
if=ide,index=2,readonly=on,media=cdrom,id=ide-5632,file=/
home/balvisio/debian-live-8.7.1-amd64-gnome-desktop.iso,format=raw

After launching the VM, I connected to the QMP socket:

# rlwrap -C qmp socat STDIO UNIX:/var/run/xen/qmp-libxl-2

{"QMP": {"version": {"qemu": {"micro": 1, "minor": 2, "major": 2},
"package": ""}, "capabilities": []}}

{"execute":"qmp_capabilities"}

{"return": {}}

I issue the "query-block" command and I get:

{"execute":"query-block"}

{"return": [{"io-status": "ok", "device": "ide0-hd0", "locked": false,
"removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off",
"image": {"virtual-size": 10737418240, "filename":
"/home/balvisio/debian-disk.img", "format": "raw", "actual-size":
4940075008, "dirty-flag": false}, "iops_wr": 0, "ro": false,
"backing_file_depth": 0, "drv": "raw", "iops": 0, "bps_wr": 0, "encrypted":
false, "bps": 0, "bps_rd": 0, "file": "/home/balvisio/debian-disk.img",
"encryption_key_missing": false}, "type": "unknown"}]}



After a few seconds, the following event happens asynchronoausly:

{"timestamp": {"seconds": 1495590969, "microseconds": 898981}, "event":
"DEVICE_DELETED", "data": {"device": "nic0", "path":
"/machine/peripheral/nic0"}}

After that event, I perform a "query-block" command again but now the block
device is gone:

{"execute":"query-block"}

{"return": []}


Xen Config File Used

kernel="/usr/local/lib/xen/boot/hvmloader"

builder='hvm'

memory=1024

vcpus=1

name="debianL2"

vfb = ['type=vnc']

vif= ['bridge=xenbr0']

boot='b'

disk=['file:/home/balvisio/debian-disk.img,xvda,w']

acpi=1

device_model_version='qemu-xen'

serial='pty'

vnc=1

vnclisten=""

vncpasswd=""

Re: [Qemu-devel] Behavior of QMP "query-block"

[John Snow]

On 05/25/2017 12:27 AM, Bruno Alvisio wrote:
> Hello all,
> 
> Summary
> 
> I am using XEN hypervisor to run a HVM with a QEMU backed disk. After I
> start the HVM I use QMP "query-block" command to see the devices of the VM.
> Initially the command returns the disk that I set as part of the
> configuration. After a few seconds the a DEVICE_DELETE event occurs (for
> device nic0) and after that the "query-block" command returns an empty
> result.
> 
> QEMU Version: 4.6.5

I assume this is the KVM version.

> Arch: x86-64
> QMP : 2.2.1

And I assume this is the QEMU version.

> 
> Note:
> 1. I was not performing any activity on the VM when the event occurs.
> 2. After the event, the QEMU process is still alive and the VM runs
> normally.
> 3. This behavior is consistently reproduced.
> 
> 
> Any help on how can I debug this issue would be greatly appreciated .
> 
> Thanks,
> 
> Bruno
> 

I'm not very familiar with XEN, I do know that it likes to add more
devices than it needs and then deletes them before starting the VM
proper. The VM actually still runs normally despite the fact that query
block returns no information? ...

CCing qemu-block.

--js

> 
> 
> DETAILED LOGS
> 
> I am using XEN hypervisor to run a HVM with QEMU backed disk. When I start
> the HVM I see the following QEMU process started:
> 
> root      2199     1  0 18:57 ?        00:00:02
> /usr/local/lib/xen/bin/qemu-system-i386
> -xen-domid 3 -chardev socket,id=libxl-cmd,path=/var/
> run/xen/qmp-libxl-3,server,nowait -no-shutdown -mon
> chardev=libxl-cmd,mode=control -chardev socket,id=libxenstat-cmd,path=
> /var/run/xen/qmp-libxenstat-3,server,nowait -mon
> chardev=libxenstat-cmd,mode=control -nodefaults -no-user-config -name
> debianL2 -vnc :0,to=99 -display none -serial pty -device
> cirrus-vga,vgamem_mb=8 -boot order=d -device
> rtl8139,id=nic0,netdev=net0,mac=00:16:3e:1b:d0:7e
> -netdev type=tap,id=net0,ifname=vif3.0-emu,script=no,downscript=no -machine
> xenfv -m 1016 -drive file=/home/balvisio/debian-
> disk.img,if=ide,index=0,media=disk,format=raw,cache=writeback -drive
> if=ide,index=2,readonly=on,media=cdrom,id=ide-5632,file=/
> home/balvisio/debian-live-8.7.1-amd64-gnome-desktop.iso,format=raw
> 
> After launching the VM, I connected to the QMP socket:
> 
> # rlwrap -C qmp socat STDIO UNIX:/var/run/xen/qmp-libxl-2
> 
> {"QMP": {"version": {"qemu": {"micro": 1, "minor": 2, "major": 2},
> "package": ""}, "capabilities": []}}
> 
> {"execute":"qmp_capabilities"}
> 
> {"return": {}}
> 
> I issue the "query-block" command and I get:
> 
> {"execute":"query-block"}
> 
> {"return": [{"io-status": "ok", "device": "ide0-hd0", "locked": false,
> "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off",
> "image": {"virtual-size": 10737418240, "filename":
> "/home/balvisio/debian-disk.img", "format": "raw", "actual-size":
> 4940075008, "dirty-flag": false}, "iops_wr": 0, "ro": false,
> "backing_file_depth": 0, "drv": "raw", "iops": 0, "bps_wr": 0, "encrypted":
> false, "bps": 0, "bps_rd": 0, "file": "/home/balvisio/debian-disk.img",
> "encryption_key_missing": false}, "type": "unknown"}]}
> 
> 
> 
> After a few seconds, the following event happens asynchronoausly:
> 
> {"timestamp": {"seconds": 1495590969, "microseconds": 898981}, "event":
> "DEVICE_DELETED", "data": {"device": "nic0", "path":
> "/machine/peripheral/nic0"}}
> 
> After that event, I perform a "query-block" command again but now the block
> device is gone:
> 
> {"execute":"query-block"}
> 
> {"return": []}
> 
> 
> Xen Config File Used
> 
> kernel="/usr/local/lib/xen/boot/hvmloader"
> 
> builder='hvm'
> 
> memory=1024
> 
> vcpus=1
> 
> name="debianL2"
> 
> vfb = ['type=vnc']
> 
> vif= ['bridge=xenbr0']
> 
> boot='b'
> 
> disk=['file:/home/balvisio/debian-disk.img,xvda,w']
> 
> acpi=1
> 
> device_model_version='qemu-xen'
> 
> serial='pty'
> 
> vnc=1
> 
> vnclisten=""
> 
> vncpasswd=""
> 

Re: [Qemu-devel] Behavior of QMP "query-block"

[Bruno Alvisio]

Hello John,

Thanks. Yes, typo when I wrote the e-mail.

It might be possible that XEN does that (I will ask in the XEN forum).
However, I have noticed that it is not the case for all of the VMs I have
launched. In some of them I can query the block devices even after a long
time the VM has been running.

I was wondering if the device is removed in case the disk is corrupted or
something. When I mention it runs normally, I can log into ithe VM and
perform read/write operations such creating files.

Also, is there an easy way to log all qemu events so that I can see what
might be going on with the 'ide' device?

Thanks,

Bruno

On Thu, May 25, 2017 at 11:18 AM, John Snow <jsnow@redhat.com> wrote:

>
>
> On 05/25/2017 12:27 AM, Bruno Alvisio wrote:
> > Hello all,
> >
> > Summary
> >
> > I am using XEN hypervisor to run a HVM with a QEMU backed disk. After I
> > start the HVM I use QMP "query-block" command to see the devices of the
> VM.
> > Initially the command returns the disk that I set as part of the
> > configuration. After a few seconds the a DEVICE_DELETE event occurs (for
> > device nic0) and after that the "query-block" command returns an empty
> > result.
> >
> > QEMU Version: 4.6.5
>
> I assume this is the KVM version.
>
> > Arch: x86-64
> > QMP : 2.2.1
>
> And I assume this is the QEMU version.
>
> >
> > Note:
> > 1. I was not performing any activity on the VM when the event occurs.
> > 2. After the event, the QEMU process is still alive and the VM runs
> > normally.
> > 3. This behavior is consistently reproduced.
> >
> >
> > Any help on how can I debug this issue would be greatly appreciated .
> >
> > Thanks,
> >
> > Bruno
> >
>
> I'm not very familiar with XEN, I do know that it likes to add more
> devices than it needs and then deletes them before starting the VM
> proper. The VM actually still runs normally despite the fact that query
> block returns no information? ...
>
> CCing qemu-block.
>
> --js
>
> >
> >
> > DETAILED LOGS
> >
> > I am using XEN hypervisor to run a HVM with QEMU backed disk. When I
> start
> > the HVM I see the following QEMU process started:
> >
> > root      2199     1  0 18:57 ?        00:00:02
> > /usr/local/lib/xen/bin/qemu-system-i386
> > -xen-domid 3 -chardev socket,id=libxl-cmd,path=/var/
> > run/xen/qmp-libxl-3,server,nowait -no-shutdown -mon
> > chardev=libxl-cmd,mode=control -chardev socket,id=libxenstat-cmd,path=
> > /var/run/xen/qmp-libxenstat-3,server,nowait -mon
> > chardev=libxenstat-cmd,mode=control -nodefaults -no-user-config -name
> > debianL2 -vnc :0,to=99 -display none -serial pty -device
> > cirrus-vga,vgamem_mb=8 -boot order=d -device
> > rtl8139,id=nic0,netdev=net0,mac=00:16:3e:1b:d0:7e
> > -netdev type=tap,id=net0,ifname=vif3.0-emu,script=no,downscript=no
> -machine
> > xenfv -m 1016 -drive file=/home/balvisio/debian-
> > disk.img,if=ide,index=0,media=disk,format=raw,cache=writeback -drive
> > if=ide,index=2,readonly=on,media=cdrom,id=ide-5632,file=/
> > home/balvisio/debian-live-8.7.1-amd64-gnome-desktop.iso,format=raw
> >
> > After launching the VM, I connected to the QMP socket:
> >
> > # rlwrap -C qmp socat STDIO UNIX:/var/run/xen/qmp-libxl-2
> >
> > {"QMP": {"version": {"qemu": {"micro": 1, "minor": 2, "major": 2},
> > "package": ""}, "capabilities": []}}
> >
> > {"execute":"qmp_capabilities"}
> >
> > {"return": {}}
> >
> > I issue the "query-block" command and I get:
> >
> > {"execute":"query-block"}
> >
> > {"return": [{"io-status": "ok", "device": "ide0-hd0", "locked": false,
> > "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off",
> > "image": {"virtual-size": 10737418240, "filename":
> > "/home/balvisio/debian-disk.img", "format": "raw", "actual-size":
> > 4940075008, "dirty-flag": false}, "iops_wr": 0, "ro": false,
> > "backing_file_depth": 0, "drv": "raw", "iops": 0, "bps_wr": 0,
> "encrypted":
> > false, "bps": 0, "bps_rd": 0, "file": "/home/balvisio/debian-disk.img",
> > "encryption_key_missing": false}, "type": "unknown"}]}
> >
> >
> >
> > After a few seconds, the following event happens asynchronoausly:
> >
> > {"timestamp": {"seconds": 1495590969, "microseconds": 898981}, "event":
> > "DEVICE_DELETED", "data": {"device": "nic0", "path":
> > "/machine/peripheral/nic0"}}
> >
> > After that event, I perform a "query-block" command again but now the
> block
> > device is gone:
> >
> > {"execute":"query-block"}
> >
> > {"return": []}
> >
> >
> > Xen Config File Used
> >
> > kernel="/usr/local/lib/xen/boot/hvmloader"
> >
> > builder='hvm'
> >
> > memory=1024
> >
> > vcpus=1
> >
> > name="debianL2"
> >
> > vfb = ['type=vnc']
> >
> > vif= ['bridge=xenbr0']
> >
> > boot='b'
> >
> > disk=['file:/home/balvisio/debian-disk.img,xvda,w']
> >
> > acpi=1
> >
> > device_model_version='qemu-xen'
> >
> > serial='pty'
> >
> > vnc=1
> >
> > vnclisten=""
> >
> > vncpasswd=""
> >
>

Re: [Qemu-devel] [Qemu-block] Behavior of QMP "query-block"

[Kevin Wolf]

Am 25.05.2017 um 20:44 hat Bruno Alvisio geschrieben:
> Hello John,
> 
> Thanks. Yes, typo when I wrote the e-mail.
> 
> It might be possible that XEN does that (I will ask in the XEN forum). However,
> I have noticed that it is not the case for all of the VMs I have launched. In
> some of them I can query the block devices even after a long time the VM has
> been running.
> 
> I was wondering if the device is removed in case the disk is corrupted or
> something. When I mention it runs normally, I can log into ithe VM and perform
> read/write operations such creating files.
> 
> Also, is there an easy way to log all qemu events so that I can see what might
> be going on with the 'ide' device?

There is one completely crazy thing that Xen does with respect to disks.
Instead of having support for their PV disks (i.e. virtio-blk, just
different) in the BIOS, they add _both_ an IDE disk and a PV disk to the
VM, so that the bootloader or non-PV-aware guest OSes can access the IDE
disk, for which they most certainly do have drivers. As soon as a driver
for the PV disk is loaded, however, that driver calls a hypervisor
function that removes all the IDE disks from the VM and leaves only the
PV ones there, so that the PV-aware guest doesn't see two same disks.

I suspect that what you're seeing initially is the IDE disks, and when
the PV driver is loaded, they disappear.

Kevin

Re: [Qemu-devel] [Qemu-block] Behavior of QMP "query-block"

[John Snow]

On 05/26/2017 07:40 AM, Kevin Wolf wrote:
> Am 25.05.2017 um 20:44 hat Bruno Alvisio geschrieben:
>> Hello John,
>>
>> Thanks. Yes, typo when I wrote the e-mail.
>>
>> It might be possible that XEN does that (I will ask in the XEN forum). However,
>> I have noticed that it is not the case for all of the VMs I have launched. In
>> some of them I can query the block devices even after a long time the VM has
>> been running.
>>
>> I was wondering if the device is removed in case the disk is corrupted or
>> something. When I mention it runs normally, I can log into ithe VM and perform
>> read/write operations such creating files.
>>
>> Also, is there an easy way to log all qemu events so that I can see what might
>> be going on with the 'ide' device?
> 
> There is one completely crazy thing that Xen does with respect to disks.
> Instead of having support for their PV disks (i.e. virtio-blk, just
> different) in the BIOS, they add _both_ an IDE disk and a PV disk to the
> VM, so that the bootloader or non-PV-aware guest OSes can access the IDE
> disk, for which they most certainly do have drivers. As soon as a driver
> for the PV disk is loaded, however, that driver calls a hypervisor
> function that removes all the IDE disks from the VM and leaves only the
> PV ones there, so that the PV-aware guest doesn't see two same disks.
> 
> I suspect that what you're seeing initially is the IDE disks, and when
> the PV driver is loaded, they disappear.
> 
> Kevin
> 

Sure, but does that mean that you'd see *no* block devices via a query
afterwards?

Re: [Qemu-devel] [Qemu-block] Behavior of QMP "query-block"

[Kevin Wolf]

Am 26.05.2017 um 18:55 hat John Snow geschrieben:
> On 05/26/2017 07:40 AM, Kevin Wolf wrote:
> > There is one completely crazy thing that Xen does with respect to disks.
> > Instead of having support for their PV disks (i.e. virtio-blk, just
> > different) in the BIOS, they add _both_ an IDE disk and a PV disk to the
> > VM, so that the bootloader or non-PV-aware guest OSes can access the IDE
> > disk, for which they most certainly do have drivers. As soon as a driver
> > for the PV disk is loaded, however, that driver calls a hypervisor
> > function that removes all the IDE disks from the VM and leaves only the
> > PV ones there, so that the PV-aware guest doesn't see two same disks.
> > 
> > I suspect that what you're seeing initially is the IDE disks, and when
> > the PV driver is loaded, they disappear.
> 
> Sure, but does that mean that you'd see *no* block devices via a query
> afterwards?

I believe xen_disk creates an anonymous BlockBackend, so it wouldn't
appear in 'info block'. Though remembering that Xen does weird things is
enough for me, I don't really want to know all the details that I am
lucky enough to have forgotten. And trust me, neither do you.

Kevin

Re: [Qemu-devel] [Qemu-block] Behavior of QMP "query-block"

[Konrad Rzeszutek Wilk]

On Fri, May 26, 2017 at 10:02:32PM +0200, Kevin Wolf wrote:
> Am 26.05.2017 um 18:55 hat John Snow geschrieben:
> > On 05/26/2017 07:40 AM, Kevin Wolf wrote:
> > > There is one completely crazy thing that Xen does with respect to disks.
> > > Instead of having support for their PV disks (i.e. virtio-blk, just
> > > different) in the BIOS, they add _both_ an IDE disk and a PV disk to the
> > > VM, so that the bootloader or non-PV-aware guest OSes can access the IDE
> > > disk, for which they most certainly do have drivers. As soon as a driver
> > > for the PV disk is loaded, however, that driver calls a hypervisor
> > > function that removes all the IDE disks from the VM and leaves only the

Not hypervisor. It calls the PCI device which is emulated in QEMU. That
"unplugs" the IDE.

> > > PV ones there, so that the PV-aware guest doesn't see two same disks.
> > > 
> > > I suspect that what you're seeing initially is the IDE disks, and when
> > > the PV driver is loaded, they disappear.
> > 
> > Sure, but does that mean that you'd see *no* block devices via a query
> > afterwards?
> 
> I believe xen_disk creates an anonymous BlockBackend, so it wouldn't
> appear in 'info block'. Though remembering that Xen does weird things is
> enough for me, I don't really want to know all the details that I am
> lucky enough to have forgotten. And trust me, neither do you.

Keep in mind that PV drivers are kernel drivers. Unless the qdisk driver
is used - and then you should see that as well.
> 
> Kevin
>