From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:41205) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1StZlI-0004xk-KG for qemu-devel@nongnu.org; Tue, 24 Jul 2012 03:42:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1StZlC-0007SR-KO for qemu-devel@nongnu.org; Tue, 24 Jul 2012 03:42:32 -0400 Received: from mail.linux-iscsi.org ([67.23.28.174]:58223 helo=linux-iscsi.org) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1StZlC-0007RA-Dm for qemu-devel@nongnu.org; Tue, 24 Jul 2012 03:42:26 -0400 From: "Nicholas A. Bellinger" Content-Type: text/plain; charset="UTF-8" Date: Tue, 24 Jul 2012 00:42:04 -0700 Message-ID: <1343115724.1813.69.camel@haakon2.linux-iscsi.org> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: [Qemu-devel] vhost-scsi port to v1.1.0 + MSI-X performance regression List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Anthony Liguori Cc: kvm-devel , "Michael S. Tsirkin" , Stefan Hajnoczi , qemu-devel , target-devel , Hannes Reinecke , Jan Kiszka , Paolo Bonzini , Christoph Hellwig , Avi Kivity Hi Anthony, Stefan & QEMU folks, So during the process of separating out the patches from Zhi's vhost-scsi tree this evening, I managed to squash everything down to nine nicely reviewable patches that apply against the current qemu.git/master: Nicholas Bellinger (1): virtio-scsi: Set max_target=0 during vhost-scsi operation Stefan Hajnoczi (8): notifier: add validity check and notify function virtio-pci: support host notifiers in TCG mode virtio-pci: check that event notification worked vhost: Pass device path to vhost_dev_init() virtio-scsi: Add wwpn and tgpt properties virtio-scsi: Open and initialize /dev/vhost-scsi virtio-scsi: Start/stop vhost vhost-scsi: add -vhost-scsi host device However, thus far I've not been able to get virtio-scsi <-> tcm_vhost I/O to actually work against the latest qemu.git/master.. So while doing a (manual) bisection w/ this series to track down the issue with qemu/master, I managed to run across something else.. With the vhost-scsi series applied, everything is working as expected up until the following commit: commit 1523ed9e1d46b0b54540049d491475ccac7e6421 Author: Jan Kiszka Date: Thu May 17 10:32:39 2012 -0300 virtio/vhost: Add support for KVM in-kernel MSI injection This commit ends up triggering the following assert immediately after starting qemu with virtio-scsi <-> tcm_vhost: qemu-system-x86_64: /usr/src/qemu.git/hw/msix.c:515: msix_unset_vector_notifiers: Assertion `dev->msix_vector_use_notifier && dev->msix_vector_release_notifier' failed. OK, so adding the following hack allows me to boot: diff --git a/hw/msix.c b/hw/msix.c index 59c7a83..6036909 100644 --- a/hw/msix.c +++ b/hw/msix.c @@ -511,6 +511,11 @@ void msix_unset_vector_notifiers(PCIDevice *dev) { int vector; + if (!dev->msix_vector_use_notifier && !dev->msix_vector_release_notifier) { + printf("Hit NULL msix_unset_vector_notifiers for: %s\n", dev->name); + return; + } + assert(dev->msix_vector_use_notifier && dev->msix_vector_release_notifier); -- and virtio-scsi is then able to load + detect tcm_vhost LUNs as expected. However the random I/O performance with commit 1523ed9e1d46b is off by a couple of orders of magnitude, ~6K IOPs compared to ~60K IOPs on raw block flash using just the previous commit bdd00bdc64ba in Jan's series. So AFAICT there appears to be a serious performance regression that is easily reproducible with that patch, which is about as far along as I've been able to diagnose yet. Interestingly enough, virtio-scsi-raw performance does not seem to be effected AFAICT by this regression, and is still able to go ~20K IOPs with the same workload using commit 1523ed9e1d46b. (Roughly the same as before) Does anyone have any idea why commit 1523ed9e1d46b would be killing vhost / tcm_vhost performance so terribly, or is there something else that vhost / vhost-scsi should be doing with new code..? Thanks! --nab