Re: [Qemu-devel] How to emulate block I/O timeout on qemu side?

[Dongli Zhang <dongli.zhang@oracle.com>]

Hi all,

I tried with the patch at:

https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg00394.html

The patch is applied to qemu-3.0.0.


Below configuration is used to test the feature for guest VM nvme.

# qemu-system-x86_64 \
-smp 4 -m 2000M -enable-kvm -vnc :0 -monitor stdio \
-net nic -net user,hostfwd=tcp::5022-:22 \
-drive file=virtio-disk.img,format=raw,if=none,id=disk0 \
-device virtio-blk-pci,drive=disk0,id=disk0-dev,num-queues=2,iothread=io1 \
-object iothread,id=io1 \
-device nvme,drive=nvme1,serial=deadbeaf1 \
-drive file=blkdebug:blkdebug.config:nvme.img,if=none,id=nvme1

# cat blkdebug.config
[delay]
event = "write_aio"
latency = "9999999999"
sector = "40960"


The 'write' latency of sector=40960 is set to a very large value. When the I/O
is stalled in guest due to that sector=40960 is accessed, I do see below
messages in guest log:

[   80.807755] nvme nvme0: I/O 11 QID 2 timeout, aborting
[   80.808095] nvme nvme0: Abort status: 0x4001


However, then nothing happens further. nvme I/O hangs in guest. I am not able to
kill the qemu process with Ctrl+C. Both vnc and qemu user net do not work. I
need to kill qemu with "kill -9"


The same result for virtio-scsi and qemu is stuck as well.


About blkdebug, I can only trigger the error by the config file. Is there a way
to inject error or latency via qemu monior? For instance, I would like to inject
error not for a specific sector or state, but for the entire disk when I input
some command via qemu monitor.

Dongli Zhang


On 11/03/2018 02:17 AM, John Snow wrote:
> 
> 
> On 11/02/2018 01:55 PM, Marc Olson wrote:
>> On 11/2/18 10:49 AM, John Snow wrote:
>>> On 11/02/2018 04:11 AM, Dongli Zhang wrote:
>>>> Hi,
>>>>
>>>> Is there any way to emulate I/O timeout on qemu side (not fault
>>>> injection in VM
>>>> kernel) without modifying qemu source code?
>>>>
>>>> For instance, I would like to observe/study/debug the I/O timeout
>>>> handling of
>>>> nvme, scsi, virtio-blk (not supported) of VM kernel.
>>>>
>>>> Is there a way to trigger this on purpose on qemu side?
>>>>
>>>> Thank you very much!
>>>>
>>>> Dongli Zhang
>>>>
>>> I don't think the blkdebug driver supports arbitrary delays right now.
>>> Maybe we could augment it to do so?
>>>
>>> (I thought someone already had, but maybe it wasn't merged?)
>>>
>>> Aha, here:
>>>
>>> https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg05297.html
>>> V2: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg00394.html
>>>
>>> Let's work from there.
>>
>> I've got updates to that patch series that fell on the floor due to
>> other competing things. I'll get some screen time this weekend to work
>> on them and submit v3.
>>
>> /marc
>>
> 
> Great! Please CC the usual maintainers, but also include me.
> 
> In the meantime, Dongli Zhang, why don't you try the v2 patch and see if
> that helps you out for your use case? Report back if it works for you or
> not.
> 
> --js
>