From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36397) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gJiRT-0007pj-CQ for qemu-devel@nongnu.org; Mon, 05 Nov 2018 12:13:37 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gJiRP-00072p-Sb for qemu-devel@nongnu.org; Mon, 05 Nov 2018 12:13:35 -0500 References: <156d16ca-1e89-1b00-cba4-bfcfc100f9c4@redhat.com> <48544a04-a8f7-dec5-ecc0-b5c0ed5156a4@oracle.com> From: John Snow Message-ID: Date: Mon, 5 Nov 2018 12:13:04 -0500 MIME-Version: 1.0 In-Reply-To: <48544a04-a8f7-dec5-ecc0-b5c0ed5156a4@oracle.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] How to emulate block I/O timeout on qemu side? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Dongli Zhang , Marc Olson Cc: qemu-devel@nongnu.org, Qemu-block On 11/03/2018 01:24 PM, Dongli Zhang wrote: > Hi all, > Hi, please reply below the quoted text when writing to qemu-devel in the future; my reply is below. > I tried with the patch at: > > https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg00394.html > > The patch is applied to qemu-3.0.0. > > > Below configuration is used to test the feature for guest VM nvme. > > # qemu-system-x86_64 \ > -smp 4 -m 2000M -enable-kvm -vnc :0 -monitor stdio \ > -net nic -net user,hostfwd=tcp::5022-:22 \ > -drive file=virtio-disk.img,format=raw,if=none,id=disk0 \ > -device virtio-blk-pci,drive=disk0,id=disk0-dev,num-queues=2,iothread=io1 \ > -object iothread,id=io1 \ > -device nvme,drive=nvme1,serial=deadbeaf1 \ > -drive file=blkdebug:blkdebug.config:nvme.img,if=none,id=nvme1 > > # cat blkdebug.config > [delay] > event = "write_aio" > latency = "9999999999" > sector = "40960" > > > The 'write' latency of sector=40960 is set to a very large value. When the I/O > is stalled in guest due to that sector=40960 is accessed, I do see below > messages in guest log: > > [ 80.807755] nvme nvme0: I/O 11 QID 2 timeout, aborting > [ 80.808095] nvme nvme0: Abort status: 0x4001 > > > However, then nothing happens further. nvme I/O hangs in guest. I am not able to > kill the qemu process with Ctrl+C. Both vnc and qemu user net do not work. I > need to kill qemu with "kill -9" > > > The same result for virtio-scsi and qemu is stuck as well. > OK, sounds like a bug in the delay implementation here, then; or something I've not considered with the locking/drain specifics. Thanks for the report. > > About blkdebug, I can only trigger the error by the config file. Is there a way > to inject error or latency via qemu monior? For instance, I would like to inject > error not for a specific sector or state, but for the entire disk when I input > some command via qemu monitor. > I don't recall. There are some tricks you can play with set-state and rules that only apply when in a certain state. I don't remember if there are monitor or QMP commands to set the state explicitly. I'm looking at docs/devel/blkdebug.txt and don't see anything immediately. There's maybe a way you can use blockdev-add to create the blkdebug node and insert it live into the graph when you want it, and live-remove it when you don't, but I'm not sure of the syntax right away. (maybe that's not possible?) --js > Dongli Zhang > > > On 11/03/2018 02:17 AM, John Snow wrote: >> >> >> On 11/02/2018 01:55 PM, Marc Olson wrote: >>> On 11/2/18 10:49 AM, John Snow wrote: >>>> On 11/02/2018 04:11 AM, Dongli Zhang wrote: >>>>> Hi, >>>>> >>>>> Is there any way to emulate I/O timeout on qemu side (not fault >>>>> injection in VM >>>>> kernel) without modifying qemu source code? >>>>> >>>>> For instance, I would like to observe/study/debug the I/O timeout >>>>> handling of >>>>> nvme, scsi, virtio-blk (not supported) of VM kernel. >>>>> >>>>> Is there a way to trigger this on purpose on qemu side? >>>>> >>>>> Thank you very much! >>>>> >>>>> Dongli Zhang >>>>> >>>> I don't think the blkdebug driver supports arbitrary delays right now. >>>> Maybe we could augment it to do so? >>>> >>>> (I thought someone already had, but maybe it wasn't merged?) >>>> >>>> Aha, here: >>>> >>>> https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg05297.html >>>> V2: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg00394.html >>>> >>>> Let's work from there. >>> >>> I've got updates to that patch series that fell on the floor due to >>> other competing things. I'll get some screen time this weekend to work >>> on them and submit v3. >>> >>> /marc >>> >> >> Great! Please CC the usual maintainers, but also include me. >> >> In the meantime, Dongli Zhang, why don't you try the v2 patch and see if >> that helps you out for your use case? Report back if it works for you or >> not. >> >> --js >>