From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:44637) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Stmtg-00043u-S9 for qemu-devel@nongnu.org; Tue, 24 Jul 2012 17:44:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Stmtf-0001Ba-Bk for qemu-devel@nongnu.org; Tue, 24 Jul 2012 17:44:04 -0400 Received: from mail.linux-iscsi.org ([67.23.28.174]:54346 helo=linux-iscsi.org) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Stmtf-0001BT-4z for qemu-devel@nongnu.org; Tue, 24 Jul 2012 17:44:03 -0400 From: "Nicholas A. Bellinger" In-Reply-To: <1343161256.1813.87.camel@haakon2.linux-iscsi.org> References: <1343115724.1813.69.camel@haakon2.linux-iscsi.org> <500E5568.6030405@siemens.com> <1343161256.1813.87.camel@haakon2.linux-iscsi.org> Content-Type: text/plain; charset="UTF-8" Date: Tue, 24 Jul 2012 14:43:50 -0700 Message-ID: <1343166230.1813.137.camel@haakon2.linux-iscsi.org> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] vhost-scsi port to v1.1.0 + MSI-X performance regression List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Jan Kiszka Cc: Jens Axboe , kvm-devel , "Michael S. Tsirkin" , Stefan Hajnoczi , qemu-devel , target-devel , Hannes Reinecke , Anthony Liguori , Paolo Bonzini , Christoph Hellwig , Avi Kivity On Tue, 2012-07-24 at 13:20 -0700, Nicholas A. Bellinger wrote: > On Tue, 2012-07-24 at 09:57 +0200, Jan Kiszka wrote: > > On 2012-07-24 09:42, Nicholas A. Bellinger wrote: > > > Hi Anthony, Stefan & QEMU folks, > > > > > > > > > However, thus far I've not been able to get virtio-scsi <-> tcm_vhost > > > I/O to actually work against the latest qemu.git/master.. > > > > > > So while doing a (manual) bisection w/ this series to track down the > > > issue with qemu/master, I managed to run across something else.. With > > > the vhost-scsi series applied, everything is working as expected up > > > until the following commit: > > > > > > commit 1523ed9e1d46b0b54540049d491475ccac7e6421 > > > Author: Jan Kiszka > > > Date: Thu May 17 10:32:39 2012 -0300 > > > > > > virtio/vhost: Add support for KVM in-kernel MSI injection > > > > > > > > > This commit ends up triggering the following assert immediately after > > > starting qemu with virtio-scsi <-> tcm_vhost: > > > > > > qemu-system-x86_64: /usr/src/qemu.git/hw/msix.c:515: > > > msix_unset_vector_notifiers: Assertion `dev->msix_vector_use_notifier && > > > dev->msix_vector_release_notifier' failed. > > > > > > OK, so adding the following hack allows me to boot: > > Also, is there a git tree and a way to reproduce this without special > > hardware needs? > > > > I'll push this series + branches to demonstrate the issue into an public > tree this afternoon. > OK, the updated vhost-scsi QEMU working tree is now available here: http://git.kernel.org/?p=virt/kvm/nab/qemu-kvm.git;a=summary Please ignore the qemu-kvm.git name for the moment, as it really is based on upstream qemu. ;) The branch breakdown looks like: master 401a663 remove unused QemuOpts parameter from net init functions * vhost-scsi bdd00bd msix: Add msix_nr_vectors_allocated vhost-scsi-merge e4fdb8c [ahead 9] vhost-scsi: add -vhost-scsi host device vhost-scsi-workaround 32ed0db msix: Work-around for vhost-scsi with KVM in-kernel MSI injection - 'vhost-scsi' contains vhost-scsi patches + pre commit 1523ed9e1d logic (Running stable at ~67.5K IOPs, see below) - 'vhost-scsi-merge' contains a vhost-scsi series against yesterday's qemu.git HEAD (not working atm, but will be sending out for RFC shortly) - 'vhost-scsi-workaround' contains the msix.c workaround to get commit 1523ed9e1d to boot (reproducible performance regression, down to 6K IOPs) > Also, the particular backend is a Fusion-IO raw block flash device, but > I'm pretty sure that using a TCM RAMDISK into tcm_vhost would exhibit > the same type of behavior. (Will double check on that shortly..) > Interestingly enough, after running some test with the vhost-scsi branch above with the last good working commit (bdd00bd), the same fio benchmark inline below is now reaching ~67.5K IOPs using v3.5-rc2 guest + host.. rw=randrw rwmixwrite=25 rwmixread=75 size=32768m ioengine=libaio direct=1 iodepth=64 blocksize=4k numjobs=2 filename=/dev/sdb This ends up being within a few percentage of what the bare-metal HW is capable of.. (~70K IOPs per LUN for an ioDrive Duo, Jen's CC'ed). Note the older QEMU vhost-scsi tree based on pre 1.1.0 code was doing ~60K IOPs for the same test.. So this looks promising pre commit 1523ed9e1d vs. what we've been using with vhost-scsi so far.. ;) --nab