* [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver @ 2014-02-07 20:14 Nicholas A. Bellinger 2014-02-09 9:24 ` Yan Vugenfirer 0 siblings, 1 reply; 15+ messages in thread From: Nicholas A. Bellinger @ 2014-02-07 20:14 UTC (permalink / raw) To: Yan Vugenfirer Cc: kvm-devel, Michael S. Tsirkin, qemu-devel, target-devel, Paolo Bonzini, Asias He Hi Yan, So recently I've been doing some KVM guest performance comparisons between the scsi-mq prototype using virtio-scsi + vhost-scsi, and Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + vhost-scsi using PCIe flash backend devices. I've noticed that small block random performance for the MSFT guest is at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up being well below what the Linux guest with scsi-mq + virtio-scsi is capable of (~500K). After searching through the various vioscsi registry settings, it appears that MSIEnabled is being explicitly disabled (0x00000000), that is different from what vioscsi.inx is currently defining: [pnpsafe_pci_addreg_msix] HKR, "Interrupt Management",, 0x00000010 HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE here, regardless of MSI_SUPPORTED: https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 appears to be the default setting for the driver included in the offical virtio-win iso builds, right..? Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys build of my own, but before going down the WDK development rabbit whole, I'd like to better understand why you've explicitly disabled this logic within vioscsi.c code to start..? Is there anything that needs to be addressed / carried over from virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c miniport code..? TIA! --nab ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver 2014-02-07 20:14 [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver Nicholas A. Bellinger @ 2014-02-09 9:24 ` Yan Vugenfirer 2014-02-09 11:35 ` Vadim Rozenfeld 0 siblings, 1 reply; 15+ messages in thread From: Yan Vugenfirer @ 2014-02-09 9:24 UTC (permalink / raw) To: Nicholas A. Bellinger, Vadim Rozenfeld Cc: kvm-devel, Michael S. Tsirkin, qemu-devel, target-devel, Paolo Bonzini, Asias He Hi Nicholas, Adding Vadim Rozenfeld who wrote the virtio-scsi driver. Best regards, Yan. On Feb 7, 2014, at 10:14 PM, Nicholas A. Bellinger <nab@linux-iscsi.org> wrote: > Hi Yan, > > So recently I've been doing some KVM guest performance comparisons > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > vhost-scsi using PCIe flash backend devices. > > I've noticed that small block random performance for the MSFT guest is > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > being well below what the Linux guest with scsi-mq + virtio-scsi is > capable of (~500K). > > After searching through the various vioscsi registry settings, it > appears that MSIEnabled is being explicitly disabled (0x00000000), that > is different from what vioscsi.inx is currently defining: > > [pnpsafe_pci_addreg_msix] > HKR, "Interrupt Management",, 0x00000010 > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > here, regardless of MSI_SUPPORTED: > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > appears to be the default setting for the driver included in the offical > virtio-win iso builds, right..? > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > build of my own, but before going down the WDK development rabbit whole, > I'd like to better understand why you've explicitly disabled this logic > within vioscsi.c code to start..? > > Is there anything that needs to be addressed / carried over from > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > miniport code..? > > TIA! > > --nab > > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver 2014-02-09 9:24 ` Yan Vugenfirer @ 2014-02-09 11:35 ` Vadim Rozenfeld 2014-02-10 19:05 ` Nicholas A. Bellinger 0 siblings, 1 reply; 15+ messages in thread From: Vadim Rozenfeld @ 2014-02-09 11:35 UTC (permalink / raw) To: Yan Vugenfirer Cc: kvm-devel, Michael S. Tsirkin, qemu-devel, Nicholas A. Bellinger, target-devel, Paolo Bonzini, Asias He On Sun, 2014-02-09 at 11:24 +0200, Yan Vugenfirer wrote: > Hi Nicholas, > > Adding Vadim Rozenfeld who wrote the virtio-scsi driver. > > Best regards, > Yan. > > On Feb 7, 2014, at 10:14 PM, Nicholas A. Bellinger <nab@linux-iscsi.org> wrote: > > > Hi Yan, > > > > So recently I've been doing some KVM guest performance comparisons > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > > vhost-scsi using PCIe flash backend devices. > > > > I've noticed that small block random performance for the MSFT guest is > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > > being well below what the Linux guest with scsi-mq + virtio-scsi is > > capable of (~500K). > > > > After searching through the various vioscsi registry settings, it > > appears that MSIEnabled is being explicitly disabled (0x00000000), that > > is different from what vioscsi.inx is currently defining: > > > > [pnpsafe_pci_addreg_msix] > > HKR, "Interrupt Management",, 0x00000010 > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > > here, regardless of MSI_SUPPORTED: > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > > appears to be the default setting for the driver included in the offical > > virtio-win iso builds, right..? > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > > build of my own, but before going down the WDK development rabbit whole, > > I'd like to better understand why you've explicitly disabled this logic > > within vioscsi.c code to start..? > > > > Is there anything that needs to be addressed / carried over from > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > > miniport code..? Hi Nicholas, I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some reasons decided to keep it disabled until adding mq support. You definitely should be able to turn on MSI_SUPPORTED, rebuild the driver, and switch MSISupported to 1 to make vioscsi driver working in MSI mode. Cheers, Vadim. > > > > TIA! > > > > --nab > > > > > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver 2014-02-09 11:35 ` Vadim Rozenfeld @ 2014-02-10 19:05 ` Nicholas A. Bellinger 2014-02-18 21:00 ` Nicholas A. Bellinger 0 siblings, 1 reply; 15+ messages in thread From: Nicholas A. Bellinger @ 2014-02-10 19:05 UTC (permalink / raw) To: Vadim Rozenfeld Cc: kvm-devel, Michael S. Tsirkin, Yan Vugenfirer, qemu-devel, target-devel, dnk, Paolo Bonzini, Asias He Hi Vadim & Yan, On Sun, 2014-02-09 at 22:35 +1100, Vadim Rozenfeld wrote: > On Sun, 2014-02-09 at 11:24 +0200, Yan Vugenfirer wrote: > > Hi Nicholas, > > > > Adding Vadim Rozenfeld who wrote the virtio-scsi driver. > > > > Best regards, > > Yan. > > > > On Feb 7, 2014, at 10:14 PM, Nicholas A. Bellinger <nab@linux-iscsi.org> wrote: > > > > > Hi Yan, > > > > > > So recently I've been doing some KVM guest performance comparisons > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > > > vhost-scsi using PCIe flash backend devices. > > > > > > I've noticed that small block random performance for the MSFT guest is > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > > > being well below what the Linux guest with scsi-mq + virtio-scsi is > > > capable of (~500K). > > > > > > After searching through the various vioscsi registry settings, it > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that > > > is different from what vioscsi.inx is currently defining: > > > > > > [pnpsafe_pci_addreg_msix] > > > HKR, "Interrupt Management",, 0x00000010 > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > > > here, regardless of MSI_SUPPORTED: > > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > > > appears to be the default setting for the driver included in the offical > > > virtio-win iso builds, right..? > > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > > > build of my own, but before going down the WDK development rabbit whole, > > > I'd like to better understand why you've explicitly disabled this logic > > > within vioscsi.c code to start..? > > > > > > Is there anything that needs to be addressed / carried over from > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > > > miniport code..? > > Hi Nicholas, > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > reasons decided to keep it disabled until adding mq support. > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the > driver, and switch MSISupported to 1 to make vioscsi driver working in > MSI mode. > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over the next days with a test build on Server 2012 / Server 2008 R2 and see how things go.. Thanks again! --nab ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver 2014-02-10 19:05 ` Nicholas A. Bellinger @ 2014-02-18 21:00 ` Nicholas A. Bellinger 2014-02-18 21:11 ` Nicholas A. Bellinger 2014-02-19 8:03 ` Vadim Rozenfeld 0 siblings, 2 replies; 15+ messages in thread From: Nicholas A. Bellinger @ 2014-02-18 21:00 UTC (permalink / raw) To: Vadim Rozenfeld Cc: kvm-devel, Michael S. Tsirkin, Yan Vugenfirer, qemu-devel, target-devel, dnk, Paolo Bonzini On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: <SNIP> > > > > Hi Yan, > > > > > > > > So recently I've been doing some KVM guest performance comparisons > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > > > > vhost-scsi using PCIe flash backend devices. > > > > > > > > I've noticed that small block random performance for the MSFT guest is > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is > > > > capable of (~500K). > > > > > > > > After searching through the various vioscsi registry settings, it > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that > > > > is different from what vioscsi.inx is currently defining: > > > > > > > > [pnpsafe_pci_addreg_msix] > > > > HKR, "Interrupt Management",, 0x00000010 > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > > > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > > > > here, regardless of MSI_SUPPORTED: > > > > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > > > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > > > > appears to be the default setting for the driver included in the offical > > > > virtio-win iso builds, right..? > > > > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > > > > build of my own, but before going down the WDK development rabbit whole, > > > > I'd like to better understand why you've explicitly disabled this logic > > > > within vioscsi.c code to start..? > > > > > > > > Is there anything that needs to be addressed / carried over from > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > > > > miniport code..? > > > > Hi Nicholas, > > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > > reasons decided to keep it disabled until adding mq support. > > > > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the > > driver, and switch MSISupported to 1 to make vioscsi driver working in > > MSI mode. > > > > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over > the next days with a test build on Server 2012 / Server 2008 R2 and see > how things go.. > Just a quick update on progress. I've been able to successfully build + load a unsigned vioscsi.sys driver on Server 2012 with WDK 8.0. Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant performance and efficiency gain, on the order of 100K to 225K IOPs for 4K block random I/O workload, depending on read/write mix. Below is a simple patch to enable MSI operation by default. Any chance to apply this separate from future mq efforts..? Thanks, --nab >From 89adb6d5800386d44b36737d1587e0ffc09c4902 Mon Sep 17 00:00:00 2001 From: Nicholas Bellinger <nab@linux-iscsi.org> Date: Fri, 14 Feb 2014 10:26:04 -0800 Subject: [PATCH] vioscsi: Set MSI_SUPPORTED=1 by default Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org> --- vioscsi/SOURCES | 2 +- vioscsi/vioscsi.c | 2 -- vioscsi/vioscsi.inx | 2 +- vioscsi/vioscsi.vcxproj | 6 +++--- 4 files changed, 5 insertions(+), 7 deletions(-) diff --git a/vioscsi/SOURCES b/vioscsi/SOURCES index f2083de..f631bd2 100644 --- a/vioscsi/SOURCES +++ b/vioscsi/SOURCES @@ -6,7 +6,7 @@ C_DEFINES = -D_MINORVERSION_=$(_BUILD_MINOR_VERSION_) $(C_DEFINES) C_DEFINES = -D_NT_TARGET_MAJ=$(_NT_TARGET_MAJ) $(C_DEFINES) C_DEFINES = -D_NT_TARGET_MIN=$(_RHEL_RELEASE_VERSION_) $(C_DEFINES) -C_DEFINES = -DMSI_SUPPORTED=0 $(C_DEFINES) +C_DEFINES = -DMSI_SUPPORTED=1 $(C_DEFINES) C_DEFINES = -DINDIRECT_SUPPORTED=1 $(C_DEFINES) TARGETLIBS=$(SDK_LIB_PATH)\storport.lib ..\VirtIO\$(O)\virtiolib.lib diff --git a/vioscsi/vioscsi.c b/vioscsi/vioscsi.c index 77c0e46..70b9bb4 100644 --- a/vioscsi/vioscsi.c +++ b/vioscsi/vioscsi.c @@ -337,8 +337,6 @@ ENTER_FN(); adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1; } - adaptExt->msix_enabled = FALSE; - RhelDbgPrint(TRACE_LEVEL_ERROR, ("breaks_number = %x queue_depth = %x\n", ConfigInfo->NumberOfPhysicalBreaks, adaptExt->queue_depth)); diff --git a/vioscsi/vioscsi.inx b/vioscsi/vioscsi.inx index cc94b7c..ec717c6 100644 --- a/vioscsi/vioscsi.inx +++ b/vioscsi/vioscsi.inx @@ -79,7 +79,7 @@ HKR, "Parameters", "BusType", %REG_DWORD%, 0x00000001 [pnpsafe_pci_addreg_msix] HKR, "Interrupt Management",, 0x00000010 HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 -HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 +HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 1 HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 HKR, "Interrupt Management\Affinity Policy",, 0x00000010 HKR, "Interrupt Management\Affinity Policy", DevicePolicy, 0x00010001, 5 diff --git a/vioscsi/vioscsi.vcxproj b/vioscsi/vioscsi.vcxproj index 889e513..68d4e85 100644 --- a/vioscsi/vioscsi.vcxproj +++ b/vioscsi/vioscsi.vcxproj @@ -66,7 +66,7 @@ <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|Win32'"> <ClCompile> <WarningLevel>Level3</WarningLevel> - <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions> + <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions> </ClCompile> <Link> <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies> @@ -95,7 +95,7 @@ <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|x64'"> <ClCompile> <WarningLevel>Level3</WarningLevel> - <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions> + <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions> </ClCompile> <Link> <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies> @@ -140,4 +140,4 @@ <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" /> <ImportGroup Label="ExtensionTargets"> </ImportGroup> -</Project> \ No newline at end of file +</Project> -- 1.8.5.2.msysgit.0 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver 2014-02-18 21:00 ` Nicholas A. Bellinger @ 2014-02-18 21:11 ` Nicholas A. Bellinger 2014-02-19 7:47 ` Vadim Rozenfeld 2014-02-19 8:03 ` Vadim Rozenfeld 1 sibling, 1 reply; 15+ messages in thread From: Nicholas A. Bellinger @ 2014-02-18 21:11 UTC (permalink / raw) To: Vadim Rozenfeld Cc: kvm-devel, Michael S. Tsirkin, Yan Vugenfirer, qemu-devel, target-devel, dnk, Paolo Bonzini On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: > > <SNIP> > > > > > > Hi Yan, > > > > > > > > > > So recently I've been doing some KVM guest performance comparisons > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > > > > > vhost-scsi using PCIe flash backend devices. > > > > > > > > > > I've noticed that small block random performance for the MSFT guest is > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is > > > > > capable of (~500K). > > > > > > > > > > After searching through the various vioscsi registry settings, it > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that > > > > > is different from what vioscsi.inx is currently defining: > > > > > > > > > > [pnpsafe_pci_addreg_msix] > > > > > HKR, "Interrupt Management",, 0x00000010 > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > > > > > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > > > > > here, regardless of MSI_SUPPORTED: > > > > > > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > > > > > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > > > > > appears to be the default setting for the driver included in the offical > > > > > virtio-win iso builds, right..? > > > > > > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > > > > > build of my own, but before going down the WDK development rabbit whole, > > > > > I'd like to better understand why you've explicitly disabled this logic > > > > > within vioscsi.c code to start..? > > > > > > > > > > Is there anything that needs to be addressed / carried over from > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > > > > > miniport code..? > > > > > > Hi Nicholas, > > > > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > > > reasons decided to keep it disabled until adding mq support. > > > > > > > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the > > > driver, and switch MSISupported to 1 to make vioscsi driver working in > > > MSI mode. > > > > > > > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over > > the next days with a test build on Server 2012 / Server 2008 R2 and see > > how things go.. > > > > Just a quick update on progress. > > I've been able to successfully build + load a unsigned vioscsi.sys > driver on Server 2012 with WDK 8.0. > > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant > performance and efficiency gain, on the order of 100K to 225K IOPs for > 4K block random I/O workload, depending on read/write mix. > One other performance related question.. In vioscsi.c:VioScsiFindAdapter() code, the default setting for adaptExt->queue_depth ends up getting set to 32 (pageNum / 4) when indirect mode is enabled in the following bits: if(adaptExt->indirect) { adaptExt->queue_depth = max(2, (pageNum / 4)); } else { adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1; } Looking at viostor/virtio_stor.c:VirtIoFindAdapter() code, the default setting for ->queue_depth appears to be 128 (pageNum): #if (INDIRECT_SUPPORTED) if(!adaptExt->dump_mode) { adaptExt->indirect = CHECKBIT(adaptExt->features, VIRTIO_RING_F_INDIRECT_DESC); } if(adaptExt->indirect) { adaptExt->queue_depth = pageNum; } #else adaptExt->indirect = 0; #endif Is there a reason for the lower queue_depth for vioscsi vs. viostor..? How about using min(adaptExt->scsi_config.cmd_per_lun, pageNum) instead..? Thanks! -nab ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver 2014-02-18 21:11 ` Nicholas A. Bellinger @ 2014-02-19 7:47 ` Vadim Rozenfeld 0 siblings, 0 replies; 15+ messages in thread From: Vadim Rozenfeld @ 2014-02-19 7:47 UTC (permalink / raw) To: Nicholas A. Bellinger Cc: kvm-devel, Michael S. Tsirkin, Yan Vugenfirer, qemu-devel, target-devel, dnk, Paolo Bonzini On Tue, 2014-02-18 at 13:11 -0800, Nicholas A. Bellinger wrote: > On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: > > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: > > > > <SNIP> > > > > > > > > Hi Yan, > > > > > > > > > > > > So recently I've been doing some KVM guest performance comparisons > > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > > > > > > vhost-scsi using PCIe flash backend devices. > > > > > > > > > > > > I've noticed that small block random performance for the MSFT guest is > > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is > > > > > > capable of (~500K). > > > > > > > > > > > > After searching through the various vioscsi registry settings, it > > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that > > > > > > is different from what vioscsi.inx is currently defining: > > > > > > > > > > > > [pnpsafe_pci_addreg_msix] > > > > > > HKR, "Interrupt Management",, 0x00000010 > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > > > > > > > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > > > > > > here, regardless of MSI_SUPPORTED: > > > > > > > > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > > > > > > > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > > > > > > appears to be the default setting for the driver included in the offical > > > > > > virtio-win iso builds, right..? > > > > > > > > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > > > > > > build of my own, but before going down the WDK development rabbit whole, > > > > > > I'd like to better understand why you've explicitly disabled this logic > > > > > > within vioscsi.c code to start..? > > > > > > > > > > > > Is there anything that needs to be addressed / carried over from > > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > > > > > > miniport code..? > > > > > > > > Hi Nicholas, > > > > > > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > > > > reasons decided to keep it disabled until adding mq support. > > > > > > > > > > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the > > > > driver, and switch MSISupported to 1 to make vioscsi driver working in > > > > MSI mode. > > > > > > > > > > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over > > > the next days with a test build on Server 2012 / Server 2008 R2 and see > > > how things go.. > > > > > > > Just a quick update on progress. > > > > I've been able to successfully build + load a unsigned vioscsi.sys > > driver on Server 2012 with WDK 8.0. > > > > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant > > performance and efficiency gain, on the order of 100K to 225K IOPs for > > 4K block random I/O workload, depending on read/write mix. > > > > One other performance related question.. > > In vioscsi.c:VioScsiFindAdapter() code, the default setting for > adaptExt->queue_depth ends up getting set to 32 (pageNum / 4) when > indirect mode is enabled in the following bits: > > if(adaptExt->indirect) { > adaptExt->queue_depth = max(2, (pageNum / 4)); > } else { > adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1; > } > > Looking at viostor/virtio_stor.c:VirtIoFindAdapter() code, the default > setting for ->queue_depth appears to be 128 (pageNum): > > #if (INDIRECT_SUPPORTED) > if(!adaptExt->dump_mode) { > adaptExt->indirect = CHECKBIT(adaptExt->features, VIRTIO_RING_F_INDIRECT_DESC); > } > if(adaptExt->indirect) { > adaptExt->queue_depth = pageNum; > } > #else > adaptExt->indirect = 0; > #endif > > Is there a reason for the lower queue_depth for vioscsi vs. viostor..? It's a horrible work around for the following bug: https://bugzilla.redhat.com/show_bug.cgi?id=1013443 I'm going to remove it as soon as found better solution for it. Best regards, Vadim. > > How about using min(adaptExt->scsi_config.cmd_per_lun, pageNum) instead..? > > Thanks! > > -nab > > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver 2014-02-18 21:00 ` Nicholas A. Bellinger 2014-02-18 21:11 ` Nicholas A. Bellinger @ 2014-02-19 8:03 ` Vadim Rozenfeld 2014-02-19 23:25 ` Nicholas A. Bellinger 1 sibling, 1 reply; 15+ messages in thread From: Vadim Rozenfeld @ 2014-02-19 8:03 UTC (permalink / raw) To: Nicholas A. Bellinger Cc: kvm-devel, Michael S. Tsirkin, Yan Vugenfirer, qemu-devel, target-devel, dnk, Paolo Bonzini On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: > > <SNIP> > > > > > > Hi Yan, > > > > > > > > > > So recently I've been doing some KVM guest performance comparisons > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > > > > > vhost-scsi using PCIe flash backend devices. > > > > > > > > > > I've noticed that small block random performance for the MSFT guest is > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is > > > > > capable of (~500K). > > > > > > > > > > After searching through the various vioscsi registry settings, it > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that > > > > > is different from what vioscsi.inx is currently defining: > > > > > > > > > > [pnpsafe_pci_addreg_msix] > > > > > HKR, "Interrupt Management",, 0x00000010 > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > > > > > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > > > > > here, regardless of MSI_SUPPORTED: > > > > > > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > > > > > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > > > > > appears to be the default setting for the driver included in the offical > > > > > virtio-win iso builds, right..? > > > > > > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > > > > > build of my own, but before going down the WDK development rabbit whole, > > > > > I'd like to better understand why you've explicitly disabled this logic > > > > > within vioscsi.c code to start..? > > > > > > > > > > Is there anything that needs to be addressed / carried over from > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > > > > > miniport code..? > > > > > > Hi Nicholas, > > > > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > > > reasons decided to keep it disabled until adding mq support. > > > > > > > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the > > > driver, and switch MSISupported to 1 to make vioscsi driver working in > > > MSI mode. > > > > > > > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over > > the next days with a test build on Server 2012 / Server 2008 R2 and see > > how things go.. > > > > Just a quick update on progress. > > I've been able to successfully build + load a unsigned vioscsi.sys > driver on Server 2012 with WDK 8.0. > > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant > performance and efficiency gain, on the order of 100K to 225K IOPs for > 4K block random I/O workload, depending on read/write mix. > > Below is a simple patch to enable MSI operation by default. Any chance > to apply this separate from future mq efforts..? Yes, we differently can enable MSI and rebuild vioscsi. But then we need to re-spin WHQL testing for this particular driver. This process requires a lot of resources, and I doubt that it will be initiated soon, unless we have some significant amount of bug-fixes. Best regards, Vadim. > > Thanks, > > --nab > > From 89adb6d5800386d44b36737d1587e0ffc09c4902 Mon Sep 17 00:00:00 2001 > From: Nicholas Bellinger <nab@linux-iscsi.org> > Date: Fri, 14 Feb 2014 10:26:04 -0800 > Subject: [PATCH] vioscsi: Set MSI_SUPPORTED=1 by default > > Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org> > --- > vioscsi/SOURCES | 2 +- > vioscsi/vioscsi.c | 2 -- > vioscsi/vioscsi.inx | 2 +- > vioscsi/vioscsi.vcxproj | 6 +++--- > 4 files changed, 5 insertions(+), 7 deletions(-) > > diff --git a/vioscsi/SOURCES b/vioscsi/SOURCES > index f2083de..f631bd2 100644 > --- a/vioscsi/SOURCES > +++ b/vioscsi/SOURCES > @@ -6,7 +6,7 @@ C_DEFINES = -D_MINORVERSION_=$(_BUILD_MINOR_VERSION_) $(C_DEFINES) > C_DEFINES = -D_NT_TARGET_MAJ=$(_NT_TARGET_MAJ) $(C_DEFINES) > C_DEFINES = -D_NT_TARGET_MIN=$(_RHEL_RELEASE_VERSION_) $(C_DEFINES) > > -C_DEFINES = -DMSI_SUPPORTED=0 $(C_DEFINES) > +C_DEFINES = -DMSI_SUPPORTED=1 $(C_DEFINES) > C_DEFINES = -DINDIRECT_SUPPORTED=1 $(C_DEFINES) > TARGETLIBS=$(SDK_LIB_PATH)\storport.lib ..\VirtIO\$(O)\virtiolib.lib > > diff --git a/vioscsi/vioscsi.c b/vioscsi/vioscsi.c > index 77c0e46..70b9bb4 100644 > --- a/vioscsi/vioscsi.c > +++ b/vioscsi/vioscsi.c > @@ -337,8 +337,6 @@ ENTER_FN(); > adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1; > } > > - adaptExt->msix_enabled = FALSE; > - > RhelDbgPrint(TRACE_LEVEL_ERROR, ("breaks_number = %x queue_depth = %x\n", > ConfigInfo->NumberOfPhysicalBreaks, > adaptExt->queue_depth)); > diff --git a/vioscsi/vioscsi.inx b/vioscsi/vioscsi.inx > index cc94b7c..ec717c6 100644 > --- a/vioscsi/vioscsi.inx > +++ b/vioscsi/vioscsi.inx > @@ -79,7 +79,7 @@ HKR, "Parameters", "BusType", %REG_DWORD%, 0x00000001 > [pnpsafe_pci_addreg_msix] > HKR, "Interrupt Management",, 0x00000010 > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > -HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > +HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 1 > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > HKR, "Interrupt Management\Affinity Policy",, 0x00000010 > HKR, "Interrupt Management\Affinity Policy", DevicePolicy, 0x00010001, 5 > diff --git a/vioscsi/vioscsi.vcxproj b/vioscsi/vioscsi.vcxproj > index 889e513..68d4e85 100644 > --- a/vioscsi/vioscsi.vcxproj > +++ b/vioscsi/vioscsi.vcxproj > @@ -66,7 +66,7 @@ > <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|Win32'"> > <ClCompile> > <WarningLevel>Level3</WarningLevel> > - <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions> > + <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions> > </ClCompile> > <Link> > <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies> > @@ -95,7 +95,7 @@ > <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|x64'"> > <ClCompile> > <WarningLevel>Level3</WarningLevel> > - <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions> > + <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions> > </ClCompile> > <Link> > <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies> > @@ -140,4 +140,4 @@ > <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" /> > <ImportGroup Label="ExtensionTargets"> > </ImportGroup> > -</Project> > \ No newline at end of file > +</Project> ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver 2014-02-19 8:03 ` Vadim Rozenfeld @ 2014-02-19 23:25 ` Nicholas A. Bellinger 2014-02-21 2:14 ` Vadim Rozenfeld 0 siblings, 1 reply; 15+ messages in thread From: Nicholas A. Bellinger @ 2014-02-19 23:25 UTC (permalink / raw) To: Vadim Rozenfeld Cc: kvm-devel, Michael S. Tsirkin, Yan Vugenfirer, qemu-devel, target-devel, dnk, Paolo Bonzini On Wed, 2014-02-19 at 19:03 +1100, Vadim Rozenfeld wrote: > On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: > > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: > > > > <SNIP> > > > > > > > > Hi Yan, > > > > > > > > > > > > So recently I've been doing some KVM guest performance comparisons > > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > > > > > > vhost-scsi using PCIe flash backend devices. > > > > > > > > > > > > I've noticed that small block random performance for the MSFT guest is > > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is > > > > > > capable of (~500K). > > > > > > > > > > > > After searching through the various vioscsi registry settings, it > > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that > > > > > > is different from what vioscsi.inx is currently defining: > > > > > > > > > > > > [pnpsafe_pci_addreg_msix] > > > > > > HKR, "Interrupt Management",, 0x00000010 > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > > > > > > > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > > > > > > here, regardless of MSI_SUPPORTED: > > > > > > > > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > > > > > > > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > > > > > > appears to be the default setting for the driver included in the offical > > > > > > virtio-win iso builds, right..? > > > > > > > > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > > > > > > build of my own, but before going down the WDK development rabbit whole, > > > > > > I'd like to better understand why you've explicitly disabled this logic > > > > > > within vioscsi.c code to start..? > > > > > > > > > > > > Is there anything that needs to be addressed / carried over from > > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > > > > > > miniport code..? > > > > > > > > Hi Nicholas, > > > > > > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > > > > reasons decided to keep it disabled until adding mq support. > > > > > > > > > > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the > > > > driver, and switch MSISupported to 1 to make vioscsi driver working in > > > > MSI mode. > > > > > > > > > > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over > > > the next days with a test build on Server 2012 / Server 2008 R2 and see > > > how things go.. > > > > > > > Just a quick update on progress. > > > > I've been able to successfully build + load a unsigned vioscsi.sys > > driver on Server 2012 with WDK 8.0. > > > > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant > > performance and efficiency gain, on the order of 100K to 225K IOPs for > > 4K block random I/O workload, depending on read/write mix. > > > > Below is a simple patch to enable MSI operation by default. Any chance > > to apply this separate from future mq efforts..? > > Yes, we differently can enable MSI and rebuild vioscsi. > But then we need to re-spin WHQL testing for this particular > driver. This process requires a lot of resources, and I doubt that > it will be initiated soon, unless we have some significant amount of > bug-fixes. > Any idea on a rough time frame to expect an official WHQL build with MSI enabled..? Or, would it be possible to generate some -BETA builds that are at least signed and don't require extra hoops to jump through for testing..? Thanks again, --nab ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver 2014-02-19 23:25 ` Nicholas A. Bellinger @ 2014-02-21 2:14 ` Vadim Rozenfeld 0 siblings, 0 replies; 15+ messages in thread From: Vadim Rozenfeld @ 2014-02-21 2:14 UTC (permalink / raw) To: Nicholas A. Bellinger Cc: kvm-devel, Michael S. Tsirkin, Yan Vugenfirer, qemu-devel, target-devel, dnk, Paolo Bonzini On Wed, 2014-02-19 at 15:25 -0800, Nicholas A. Bellinger wrote: > On Wed, 2014-02-19 at 19:03 +1100, Vadim Rozenfeld wrote: > > On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: > > > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: > > > > > > <SNIP> > > > > > > > > > > Hi Yan, > > > > > > > > > > > > > > So recently I've been doing some KVM guest performance comparisons > > > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > > > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > > > > > > > vhost-scsi using PCIe flash backend devices. > > > > > > > > > > > > > > I've noticed that small block random performance for the MSFT guest is > > > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > > > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is > > > > > > > capable of (~500K). > > > > > > > > > > > > > > After searching through the various vioscsi registry settings, it > > > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that > > > > > > > is different from what vioscsi.inx is currently defining: > > > > > > > > > > > > > > [pnpsafe_pci_addreg_msix] > > > > > > > HKR, "Interrupt Management",, 0x00000010 > > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > > > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > > > > > > > > > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > > > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > > > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > > > > > > > here, regardless of MSI_SUPPORTED: > > > > > > > > > > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > > > > > > > > > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > > > > > > > appears to be the default setting for the driver included in the offical > > > > > > > virtio-win iso builds, right..? > > > > > > > > > > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > > > > > > > build of my own, but before going down the WDK development rabbit whole, > > > > > > > I'd like to better understand why you've explicitly disabled this logic > > > > > > > within vioscsi.c code to start..? > > > > > > > > > > > > > > Is there anything that needs to be addressed / carried over from > > > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > > > > > > > miniport code..? > > > > > > > > > > Hi Nicholas, > > > > > > > > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > > > > > reasons decided to keep it disabled until adding mq support. > > > > > > > > > > > > > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the > > > > > driver, and switch MSISupported to 1 to make vioscsi driver working in > > > > > MSI mode. > > > > > > > > > > > > > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over > > > > the next days with a test build on Server 2012 / Server 2008 R2 and see > > > > how things go.. > > > > > > > > > > Just a quick update on progress. > > > > > > I've been able to successfully build + load a unsigned vioscsi.sys > > > driver on Server 2012 with WDK 8.0. > > > > > > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant > > > performance and efficiency gain, on the order of 100K to 225K IOPs for > > > 4K block random I/O workload, depending on read/write mix. > > > > > > Below is a simple patch to enable MSI operation by default. Any chance > > > to apply this separate from future mq efforts..? > > > > Yes, we differently can enable MSI and rebuild vioscsi. > > But then we need to re-spin WHQL testing for this particular > > driver. This process requires a lot of resources, and I doubt that > > it will be initiated soon, unless we have some significant amount of > > bug-fixes. > > > > Any idea on a rough time frame to expect an official WHQL build with MSI > enabled..? In June for sure :) > Or, would it be possible to generate some -BETA builds that are at least > signed and don't require extra hoops to jump through for testing..? It's doable. I hope we can make a new build next week. Best regards, Vadim. > > Thanks again, > > --nab > > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver @ 2014-10-30 6:54 Wangting (Kathy) 2014-10-30 8:48 ` Vadim Rozenfeld 0 siblings, 1 reply; 15+ messages in thread From: Wangting (Kathy) @ 2014-10-30 6:54 UTC (permalink / raw) To: vrozenfe, qemu-devel > On Tue, 2014-02-18 at 13:11 -0800, Nicholas A. Bellinger wrote: >> On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: >> > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: >> > >> > <SNIP> >> > >> > > > > > Hi Yan, >> > > > > > >> > > > > > So recently I've been doing some KVM guest performance comparisons >> > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and >> > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + >> > > > > > vhost-scsi using PCIe flash backend devices. >> > > > > > >> > > > > > I've noticed that small block random performance for the MSFT guest >> > > > > > is >> > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which >> > > > > > ends up >> > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is >> > > > > > capable of (~500K). >> > > > > > >> > > > > > After searching through the various vioscsi registry settings, it >> > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), >> > > > > > that >> > > > > > is different from what vioscsi.inx is currently defining: >> > > > > > >> > > > > > [pnpsafe_pci_addreg_msix] >> > > > > > HKR, "Interrupt Management",, 0x00000010 >> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, >> > > > > > 0x00000010 >> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", >> > > > > > MSISupported, 0x00010001, 0 >> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", >> > > > > > MessageNumberLimit, 0x00010001, 4 >> > > > > > >> > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 >> > > > > > is >> > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as >> > > > > > well >> > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = >> > > > > > FALSE >> > > > > > here, regardless of MSI_SUPPORTED: >> > > > > > >> > > > > > >> > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 >> > > > > > >> > > > > > Also looking at virtio_stor.c for the raw block driver, >> > > > > > MSI_SUPPORTED=1 >> > > > > > appears to be the default setting for the driver included in the >> > > > > > offical >> > > > > > virtio-win iso builds, right..? >> > > > > > >> > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys >> > > > > > build of my own, but before going down the WDK development rabbit >> > > > > > whole, >> > > > > > I'd like to better understand why you've explicitly disabled this >> > > > > > logic >> > > > > > within vioscsi.c code to start..? >> > > > > > >> > > > > > Is there anything that needs to be addressed / carried over from >> > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c >> > > > > > miniport code..? >> > > > >> > > > Hi Nicholas, >> > > > >> > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some >> > > > reasons decided to keep it disabled until adding mq support. >> > > > >> > > > >> > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the >> > > > driver, and switch MSISupported to 1 to make vioscsi driver working in >> > > > MSI mode. >> > > > >> > > >> > > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over >> > > the next days with a test build on Server 2012 / Server 2008 R2 and see >> > > how things go.. >> > > >> > >> > Just a quick update on progress. >> > >> > I've been able to successfully build + load a unsigned vioscsi.sys >> > driver on Server 2012 with WDK 8.0. >> > >> > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant >> > performance and efficiency gain, on the order of 100K to 225K IOPs for >> > 4K block random I/O workload, depending on read/write mix. >> > >> >> One other performance related question.. >> >> In vioscsi.c:VioScsiFindAdapter() code, the default setting for >> adaptExt->queue_depth ends up getting set to 32 (pageNum / 4) when >> indirect mode is enabled in the following bits: >> >> if(adaptExt->indirect) { >> adaptExt->queue_depth = max(2, (pageNum / 4)); >> } else { >> adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks >> - 1; >> } >> >> Looking at viostor/virtio_stor.c:VirtIoFindAdapter() code, the default >> setting for ->queue_depth appears to be 128 (pageNum): >> >> #if (INDIRECT_SUPPORTED) >> if(!adaptExt->dump_mode) { >> adaptExt->indirect = CHECKBIT(adaptExt->features, >> VIRTIO_RING_F_INDIRECT_DESC); >> } >> if(adaptExt->indirect) { >> adaptExt->queue_depth = pageNum; >> } >> #else >> adaptExt->indirect = 0; >> #endif >> >> Is there a reason for the lower queue_depth for vioscsi vs. viostor..? > > It's a horrible work around for the following bug: > https://bugzilla.redhat.com/show_bug.cgi?id=1013443 > > I'm going to remove it as soon as found better solution for it. > > Best regards, > Vadim. > > Hi Vadim, I have found that Bug 1013443 has been closed with a resolution of ERRATA. The windows device queue must be between 20 and 254 for StorPortSetDeviceQueueDepth to succeed. So I have the question that why queue_depth can not be set to pageNum(128)? Best wishes, Ting Wang >> >> How about using min(adaptExt->scsi_config.cmd_per_lun, pageNum) instead..? >> >> Thanks! >> >> -nab >> >> ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver 2014-10-30 6:54 Wangting (Kathy) @ 2014-10-30 8:48 ` Vadim Rozenfeld 2014-10-30 10:28 ` Wangting (Kathy) 0 siblings, 1 reply; 15+ messages in thread From: Vadim Rozenfeld @ 2014-10-30 8:48 UTC (permalink / raw) To: Wangting (Kathy); +Cc: qemu-devel On Thu, 2014-10-30 at 14:54 +0800, Wangting (Kathy) wrote: > > On Tue, 2014-02-18 at 13:11 -0800, Nicholas A. Bellinger wrote: > >> On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: > >> > On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: > >> > > >> > <SNIP> > >> > > >> > > > > > Hi Yan, > >> > > > > > > >> > > > > > So recently I've been doing some KVM guest performance comparisons > >> > > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > >> > > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > >> > > > > > vhost-scsi using PCIe flash backend devices. > >> > > > > > > >> > > > > > I've noticed that small block random performance for the MSFT guest > >> > > > > > is > >> > > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which > >> > > > > > ends up > >> > > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is > >> > > > > > capable of (~500K). > >> > > > > > > >> > > > > > After searching through the various vioscsi registry settings, it > >> > > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), > >> > > > > > that > >> > > > > > is different from what vioscsi.inx is currently defining: > >> > > > > > > >> > > > > > [pnpsafe_pci_addreg_msix] > >> > > > > > HKR, "Interrupt Management",, 0x00000010 > >> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, > >> > > > > > 0x00000010 > >> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", > >> > > > > > MSISupported, 0x00010001, 0 > >> > > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", > >> > > > > > MessageNumberLimit, 0x00010001, 4 > >> > > > > > > >> > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 > >> > > > > > is > >> > > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as > >> > > > > > well > >> > > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = > >> > > > > > FALSE > >> > > > > > here, regardless of MSI_SUPPORTED: > >> > > > > > > >> > > > > > > >> > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > >> > > > > > > >> > > > > > Also looking at virtio_stor.c for the raw block driver, > >> > > > > > MSI_SUPPORTED=1 > >> > > > > > appears to be the default setting for the driver included in the > >> > > > > > offical > >> > > > > > virtio-win iso builds, right..? > >> > > > > > > >> > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > >> > > > > > build of my own, but before going down the WDK development rabbit > >> > > > > > whole, > >> > > > > > I'd like to better understand why you've explicitly disabled this > >> > > > > > logic > >> > > > > > within vioscsi.c code to start..? > >> > > > > > > >> > > > > > Is there anything that needs to be addressed / carried over from > >> > > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > >> > > > > > miniport code..? > >> > > > > >> > > > Hi Nicholas, > >> > > > > >> > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > >> > > > reasons decided to keep it disabled until adding mq support. > >> > > > > >> > > > > >> > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the > >> > > > driver, and switch MSISupported to 1 to make vioscsi driver working in > >> > > > MSI mode. > >> > > > > >> > > > >> > > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over > >> > > the next days with a test build on Server 2012 / Server 2008 R2 and see > >> > > how things go.. > >> > > > >> > > >> > Just a quick update on progress. > >> > > >> > I've been able to successfully build + load a unsigned vioscsi.sys > >> > driver on Server 2012 with WDK 8.0. > >> > > >> > Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant > >> > performance and efficiency gain, on the order of 100K to 225K IOPs for > >> > 4K block random I/O workload, depending on read/write mix. > >> > > >> > >> One other performance related question.. > >> > >> In vioscsi.c:VioScsiFindAdapter() code, the default setting for > >> adaptExt->queue_depth ends up getting set to 32 (pageNum / 4) when > >> indirect mode is enabled in the following bits: > >> > >> if(adaptExt->indirect) { > >> adaptExt->queue_depth = max(2, (pageNum / 4)); > >> } else { > >> adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks > >> - 1; > >> } > >> > >> Looking at viostor/virtio_stor.c:VirtIoFindAdapter() code, the default > >> setting for ->queue_depth appears to be 128 (pageNum): > >> > >> #if (INDIRECT_SUPPORTED) > >> if(!adaptExt->dump_mode) { > >> adaptExt->indirect = CHECKBIT(adaptExt->features, > >> VIRTIO_RING_F_INDIRECT_DESC); > >> } > >> if(adaptExt->indirect) { > >> adaptExt->queue_depth = pageNum; > >> } > >> #else > >> adaptExt->indirect = 0; > >> #endif > >> > >> Is there a reason for the lower queue_depth for vioscsi vs. viostor..? > > > > It's a horrible work around for the following bug: > > https://bugzilla.redhat.com/show_bug.cgi?id=1013443 > > > > I'm going to remove it as soon as found better solution for it. > > > > Best regards, > > Vadim. > > > > > Hi Vadim, > > I have found that Bug 1013443 has been closed with a > resolution of ERRATA. > > The windows device queue must be between 20 and 254 > for StorPortSetDeviceQueueDepth to succeed. > > So I have the question that why queue_depth can not be > set to pageNum(128)? It will create a problem on multi disk setup, when several disks are attached to the same virtio-scsi pci controller. Adding some sort of manually managed SRBs queue for storing and resubmitting pending requests can solve this problem. Cheers, Vadim. > > Best wishes, > Ting Wang > > >> > >> How about using min(adaptExt->scsi_config.cmd_per_lun, pageNum) instead..? > >> > >> Thanks! > >> > >> -nab > >> > >> > > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver 2014-10-30 8:48 ` Vadim Rozenfeld @ 2014-10-30 10:28 ` Wangting (Kathy) 2014-10-30 11:15 ` Vadim Rozenfeld 0 siblings, 1 reply; 15+ messages in thread From: Wangting (Kathy) @ 2014-10-30 10:28 UTC (permalink / raw) To: Vadim Rozenfeld; +Cc: qemu-devel On 2014-10-30 16:48, Vadim Rozenfeld wrote: > On Thu, 2014-10-30 at 14:54 +0800, Wangting (Kathy) wrote: >>> On Tue, 2014-02-18 at 13:11 -0800, Nicholas A. Bellinger wrote: >>>> On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: >>>>> On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: >>>>> >>>>> <SNIP> >>>>> >>>>>>>>> Hi Yan, >>>>>>>>> >>>>>>>>> So recently I've been doing some KVM guest performance comparisons >>>>>>>>> between the scsi-mq prototype using virtio-scsi + vhost-scsi, and >>>>>>>>> Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + >>>>>>>>> vhost-scsi using PCIe flash backend devices. >>>>>>>>> >>>>>>>>> I've noticed that small block random performance for the MSFT guest >>>>>>>>> is >>>>>>>>> at around ~80K IOPs with multiple vioscsi LUNs + adapters, which >>>>>>>>> ends up >>>>>>>>> being well below what the Linux guest with scsi-mq + virtio-scsi is >>>>>>>>> capable of (~500K). >>>>>>>>> >>>>>>>>> After searching through the various vioscsi registry settings, it >>>>>>>>> appears that MSIEnabled is being explicitly disabled (0x00000000), >>>>>>>>> that >>>>>>>>> is different from what vioscsi.inx is currently defining: >>>>>>>>> >>>>>>>>> [pnpsafe_pci_addreg_msix] >>>>>>>>> HKR, "Interrupt Management",, 0x00000010 >>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties",, >>>>>>>>> 0x00000010 >>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties", >>>>>>>>> MSISupported, 0x00010001, 0 >>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties", >>>>>>>>> MessageNumberLimit, 0x00010001, 4 >>>>>>>>> >>>>>>>>> Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 >>>>>>>>> is >>>>>>>>> explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as >>>>>>>>> well >>>>>>>>> as VioScsiFindAdapter() code always ends setting msix_enabled = >>>>>>>>> FALSE >>>>>>>>> here, regardless of MSI_SUPPORTED: >>>>>>>>> >>>>>>>>> >>>>>>>>> https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 >>>>>>>>> >>>>>>>>> Also looking at virtio_stor.c for the raw block driver, >>>>>>>>> MSI_SUPPORTED=1 >>>>>>>>> appears to be the default setting for the driver included in the >>>>>>>>> offical >>>>>>>>> virtio-win iso builds, right..? >>>>>>>>> >>>>>>>>> Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys >>>>>>>>> build of my own, but before going down the WDK development rabbit >>>>>>>>> whole, >>>>>>>>> I'd like to better understand why you've explicitly disabled this >>>>>>>>> logic >>>>>>>>> within vioscsi.c code to start..? >>>>>>>>> >>>>>>>>> Is there anything that needs to be addressed / carried over from >>>>>>>>> virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c >>>>>>>>> miniport code..? >>>>>>> >>>>>>> Hi Nicholas, >>>>>>> >>>>>>> I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some >>>>>>> reasons decided to keep it disabled until adding mq support. >>>>>>> >>>>>>> >>>>>>> You definitely should be able to turn on MSI_SUPPORTED, rebuild the >>>>>>> driver, and switch MSISupported to 1 to make vioscsi driver working in >>>>>>> MSI mode. >>>>>>> >>>>>> >>>>>> Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over >>>>>> the next days with a test build on Server 2012 / Server 2008 R2 and see >>>>>> how things go.. >>>>>> >>>>> >>>>> Just a quick update on progress. >>>>> >>>>> I've been able to successfully build + load a unsigned vioscsi.sys >>>>> driver on Server 2012 with WDK 8.0. >>>>> >>>>> Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant >>>>> performance and efficiency gain, on the order of 100K to 225K IOPs for >>>>> 4K block random I/O workload, depending on read/write mix. >>>>> >>>> >>>> One other performance related question.. >>>> >>>> In vioscsi.c:VioScsiFindAdapter() code, the default setting for >>>> adaptExt->queue_depth ends up getting set to 32 (pageNum / 4) when >>>> indirect mode is enabled in the following bits: >>>> >>>> if(adaptExt->indirect) { >>>> adaptExt->queue_depth = max(2, (pageNum / 4)); >>>> } else { >>>> adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks >>>> - 1; >>>> } >>>> >>>> Looking at viostor/virtio_stor.c:VirtIoFindAdapter() code, the default >>>> setting for ->queue_depth appears to be 128 (pageNum): >>>> >>>> #if (INDIRECT_SUPPORTED) >>>> if(!adaptExt->dump_mode) { >>>> adaptExt->indirect = CHECKBIT(adaptExt->features, >>>> VIRTIO_RING_F_INDIRECT_DESC); >>>> } >>>> if(adaptExt->indirect) { >>>> adaptExt->queue_depth = pageNum; >>>> } >>>> #else >>>> adaptExt->indirect = 0; >>>> #endif >>>> >>>> Is there a reason for the lower queue_depth for vioscsi vs. viostor..? >>> >>> It's a horrible work around for the following bug: >>> https://bugzilla.redhat.com/show_bug.cgi?id=1013443 >>> >>> I'm going to remove it as soon as found better solution for it. >>> >>> Best regards, >>> Vadim. >>> >>> >> Hi Vadim, >> >> I have found that Bug 1013443 has been closed with a >> resolution of ERRATA. >> >> The windows device queue must be between 20 and 254 >> for StorPortSetDeviceQueueDepth to succeed. >> >> So I have the question that why queue_depth can not be >> set to pageNum(128)? > > It will create a problem on multi disk setup, when several > disks are attached to the same virtio-scsi pci controller. > Adding some sort of manually managed SRBs queue for storing and > resubmitting pending requests can solve this problem. > > Cheers, > Vadim. > Is there a patch for it? >> >> Best wishes, >> Ting Wang >> >>>> >>>> How about using min(adaptExt->scsi_config.cmd_per_lun, pageNum) instead..? >>>> >>>> Thanks! >>>> >>>> -nab >>>> >>>> >> >> > > > > . > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver 2014-10-30 10:28 ` Wangting (Kathy) @ 2014-10-30 11:15 ` Vadim Rozenfeld 2014-10-31 1:55 ` Wangting (Kathy) 0 siblings, 1 reply; 15+ messages in thread From: Vadim Rozenfeld @ 2014-10-30 11:15 UTC (permalink / raw) To: Wangting (Kathy); +Cc: qemu-devel [-- Attachment #1: Type: text/plain, Size: 6325 bytes --] On Thu, 2014-10-30 at 18:28 +0800, Wangting (Kathy) wrote: > > On 2014-10-30 16:48, Vadim Rozenfeld wrote: > > On Thu, 2014-10-30 at 14:54 +0800, Wangting (Kathy) wrote: > >>> On Tue, 2014-02-18 at 13:11 -0800, Nicholas A. Bellinger wrote: > >>>> On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: > >>>>> On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: > >>>>> > >>>>> <SNIP> > >>>>> > >>>>>>>>> Hi Yan, > >>>>>>>>> > >>>>>>>>> So recently I've been doing some KVM guest performance comparisons > >>>>>>>>> between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > >>>>>>>>> Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > >>>>>>>>> vhost-scsi using PCIe flash backend devices. > >>>>>>>>> > >>>>>>>>> I've noticed that small block random performance for the MSFT guest > >>>>>>>>> is > >>>>>>>>> at around ~80K IOPs with multiple vioscsi LUNs + adapters, which > >>>>>>>>> ends up > >>>>>>>>> being well below what the Linux guest with scsi-mq + virtio-scsi is > >>>>>>>>> capable of (~500K). > >>>>>>>>> > >>>>>>>>> After searching through the various vioscsi registry settings, it > >>>>>>>>> appears that MSIEnabled is being explicitly disabled (0x00000000), > >>>>>>>>> that > >>>>>>>>> is different from what vioscsi.inx is currently defining: > >>>>>>>>> > >>>>>>>>> [pnpsafe_pci_addreg_msix] > >>>>>>>>> HKR, "Interrupt Management",, 0x00000010 > >>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties",, > >>>>>>>>> 0x00000010 > >>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties", > >>>>>>>>> MSISupported, 0x00010001, 0 > >>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties", > >>>>>>>>> MessageNumberLimit, 0x00010001, 4 > >>>>>>>>> > >>>>>>>>> Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 > >>>>>>>>> is > >>>>>>>>> explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as > >>>>>>>>> well > >>>>>>>>> as VioScsiFindAdapter() code always ends setting msix_enabled = > >>>>>>>>> FALSE > >>>>>>>>> here, regardless of MSI_SUPPORTED: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > >>>>>>>>> > >>>>>>>>> Also looking at virtio_stor.c for the raw block driver, > >>>>>>>>> MSI_SUPPORTED=1 > >>>>>>>>> appears to be the default setting for the driver included in the > >>>>>>>>> offical > >>>>>>>>> virtio-win iso builds, right..? > >>>>>>>>> > >>>>>>>>> Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > >>>>>>>>> build of my own, but before going down the WDK development rabbit > >>>>>>>>> whole, > >>>>>>>>> I'd like to better understand why you've explicitly disabled this > >>>>>>>>> logic > >>>>>>>>> within vioscsi.c code to start..? > >>>>>>>>> > >>>>>>>>> Is there anything that needs to be addressed / carried over from > >>>>>>>>> virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > >>>>>>>>> miniport code..? > >>>>>>> > >>>>>>> Hi Nicholas, > >>>>>>> > >>>>>>> I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > >>>>>>> reasons decided to keep it disabled until adding mq support. > >>>>>>> > >>>>>>> > >>>>>>> You definitely should be able to turn on MSI_SUPPORTED, rebuild the > >>>>>>> driver, and switch MSISupported to 1 to make vioscsi driver working in > >>>>>>> MSI mode. > >>>>>>> > >>>>>> > >>>>>> Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over > >>>>>> the next days with a test build on Server 2012 / Server 2008 R2 and see > >>>>>> how things go.. > >>>>>> > >>>>> > >>>>> Just a quick update on progress. > >>>>> > >>>>> I've been able to successfully build + load a unsigned vioscsi.sys > >>>>> driver on Server 2012 with WDK 8.0. > >>>>> > >>>>> Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant > >>>>> performance and efficiency gain, on the order of 100K to 225K IOPs for > >>>>> 4K block random I/O workload, depending on read/write mix. > >>>>> > >>>> > >>>> One other performance related question.. > >>>> > >>>> In vioscsi.c:VioScsiFindAdapter() code, the default setting for > >>>> adaptExt->queue_depth ends up getting set to 32 (pageNum / 4) when > >>>> indirect mode is enabled in the following bits: > >>>> > >>>> if(adaptExt->indirect) { > >>>> adaptExt->queue_depth = max(2, (pageNum / 4)); > >>>> } else { > >>>> adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks > >>>> - 1; > >>>> } > >>>> > >>>> Looking at viostor/virtio_stor.c:VirtIoFindAdapter() code, the default > >>>> setting for ->queue_depth appears to be 128 (pageNum): > >>>> > >>>> #if (INDIRECT_SUPPORTED) > >>>> if(!adaptExt->dump_mode) { > >>>> adaptExt->indirect = CHECKBIT(adaptExt->features, > >>>> VIRTIO_RING_F_INDIRECT_DESC); > >>>> } > >>>> if(adaptExt->indirect) { > >>>> adaptExt->queue_depth = pageNum; > >>>> } > >>>> #else > >>>> adaptExt->indirect = 0; > >>>> #endif > >>>> > >>>> Is there a reason for the lower queue_depth for vioscsi vs. viostor..? > >>> > >>> It's a horrible work around for the following bug: > >>> https://bugzilla.redhat.com/show_bug.cgi?id=1013443 > >>> > >>> I'm going to remove it as soon as found better solution for it. > >>> > >>> Best regards, > >>> Vadim. > >>> > >>> > >> Hi Vadim, > >> > >> I have found that Bug 1013443 has been closed with a > >> resolution of ERRATA. > >> > >> The windows device queue must be between 20 and 254 > >> for StorPortSetDeviceQueueDepth to succeed. > >> > >> So I have the question that why queue_depth can not be > >> set to pageNum(128)? > > > > It will create a problem on multi disk setup, when several > > disks are attached to the same virtio-scsi pci controller. > > Adding some sort of manually managed SRBs queue for storing and > > resubmitting pending requests can solve this problem. > > > > Cheers, > > Vadim. > > > > Is there a patch for it? Yes, not committed and not fully tested yet. Please, find it attached. Best, Vadim. > > >> > >> Best wishes, > >> Ting Wang > >> > >>>> > >>>> How about using min(adaptExt->scsi_config.cmd_per_lun, pageNum) instead..? > >>>> > >>>> Thanks! > >>>> > >>>> -nab > >>>> > >>>> > >> > >> > > > > > > > > . > > > > [-- Attachment #2: 0001-queue-depth254.patch --] [-- Type: text/x-patch, Size: 7902 bytes --] >From 3fbee0ab3165657626828f661244ad3145d8b7dc Mon Sep 17 00:00:00 2001 From: Vadim Rozenfeld <vrozenfe@redhat.com> Date: Sun, 14 Sep 2014 18:57:48 +1000 Subject: [PATCH 1/1] queue depth 254 (cherry picked from commit 540b638bf86bad908b4b7527fdfc634ddca3f0e8) --- vioscsi/helper.c | 52 ++++++++++++++++++++++++++++++++++++++++++++-------- vioscsi/helper.h | 7 ++++++- vioscsi/vioscsi.c | 40 ++++++++++++++++------------------------ vioscsi/vioscsi.h | 6 +++++- 4 files changed, 71 insertions(+), 34 deletions(-) mode change 100644 => 100755 vioscsi/helper.c mode change 100644 => 100755 vioscsi/helper.h mode change 100644 => 100755 vioscsi/vioscsi.c mode change 100644 => 100755 vioscsi/vioscsi.h diff --git a/vioscsi/helper.c b/vioscsi/helper.c old mode 100644 new mode 100755 index dcbb8d9..6cd6217 --- a/vioscsi/helper.c +++ b/vioscsi/helper.c @@ -37,28 +37,60 @@ SynchronizedSRBRoutine( ENTER_FN(); SET_VA_PA(); - if (virtqueue_add_buf(adaptExt->vq[2], + if (IsListEmpty(&adaptExt->list_head) && virtqueue_add_buf(adaptExt->vq[2], &srbExt->sg[0], srbExt->out, srbExt->in, &srbExt->cmd, va, pa) >= 0){ - virtqueue_kick(adaptExt->vq[2]); +// virtqueue_kick(adaptExt->vq[2]); return TRUE; } - Srb->SrbStatus = SRB_STATUS_BUSY; - StorPortBusy(DeviceExtension, 2); - virtqueue_kick(adaptExt->vq[2]); -EXIT_ERR(); + InsertTailList(&adaptExt->list_head, &srbExt->list_entry); return FALSE; } -BOOLEAN +VOID +ResendSRB( + IN PVOID DeviceExtension + ) +{ + PADAPTER_EXTENSION adaptExt = (PADAPTER_EXTENSION)DeviceExtension; + PVOID va; + ULONGLONG pa; + BOOLEAN kick = FALSE; + while (!IsListEmpty(&adaptExt->list_head)) { + PSCSI_REQUEST_BLOCK Srb; + PSRB_EXTENSION srbExt; + srbExt = (PSRB_EXTENSION)RemoveHeadList(&adaptExt->list_head); + Srb = (PSCSI_REQUEST_BLOCK)srbExt->Srb; + + SET_VA_PA(); + + if (virtqueue_add_buf(adaptExt->vq[2], + &srbExt->sg[0], + srbExt->out, srbExt->in, + &srbExt->cmd, va, pa) >= 0) { + kick = TRUE; + } else { + InsertHeadList(&adaptExt->list_head, &srbExt->list_entry); + return; + } + } + if (kick) { + virtqueue_kick(adaptExt->vq[2]); + } +} + +VOID SendSRB( IN PVOID DeviceExtension, IN PSCSI_REQUEST_BLOCK Srb ) { + PADAPTER_EXTENSION adaptExt = (PADAPTER_EXTENSION)DeviceExtension; ENTER_FN(); - return StorPortSynchronizeAccess(DeviceExtension, SynchronizedSRBRoutine, (PVOID)Srb); + if (StorPortSynchronizeAccess(DeviceExtension, SynchronizedSRBRoutine, (PVOID)Srb)) { + virtqueue_kick(adaptExt->vq[2]); + } EXIT_FN(); } @@ -152,6 +184,10 @@ ShutDown( { PADAPTER_EXTENSION adaptExt = (PADAPTER_EXTENSION)DeviceExtension; ENTER_FN(); + while (!IsListEmpty(&adaptExt->list_head)) { + StorPortStallExecution(999); + } + VirtIODeviceReset(&adaptExt->vdev); StorPortWritePortUshort(DeviceExtension, (PUSHORT)(adaptExt->device_base + VIRTIO_PCI_GUEST_FEATURES), 0); if (adaptExt->vq[0]) { diff --git a/vioscsi/helper.h b/vioscsi/helper.h old mode 100644 new mode 100755 index b107b42..17e62ef --- a/vioscsi/helper.h +++ b/vioscsi/helper.h @@ -24,12 +24,17 @@ #include "virtio_pci.h" #include "vioscsi.h" -BOOLEAN +VOID SendSRB( IN PVOID DeviceExtension, IN PSCSI_REQUEST_BLOCK Srb ); +VOID +ResendSRB( + IN PVOID DeviceExtension + ); + BOOLEAN SendTMF( IN PVOID DeviceExtension, diff --git a/vioscsi/vioscsi.c b/vioscsi/vioscsi.c old mode 100644 new mode 100755 index e2b2e0d..7e78c72 --- a/vioscsi/vioscsi.c +++ b/vioscsi/vioscsi.c @@ -351,14 +351,7 @@ ENTER_FN(); #else adaptExt->indirect = 0; #endif - - // The windows device queue must be between 20 and 254 for - // StorPortSetDeviceQueueDepth to succeed. - adaptExt->queue_depth = min(254, max(20, (pageNum / 4))); - RhelDbgPrint(TRACE_LEVEL_ERROR, ("breaks_number = %x queue_depth = %x\n", - ConfigInfo->NumberOfPhysicalBreaks, - adaptExt->queue_depth)); - + adaptExt->queue_depth = 254; adaptExt->uncachedExtensionVa = StorPortGetUncachedExtension(DeviceExtension, ConfigInfo, allocationSize); if (!adaptExt->uncachedExtensionVa) { LogError(DeviceExtension, @@ -373,7 +366,7 @@ ENTER_FN(); ASSERT(adaptExt->vq[1] != NULL); ASSERT(adaptExt->vq[2] != NULL); } - + InitializeListHead(&adaptExt->list_head); return SP_RETURN_FOUND; } @@ -520,15 +513,14 @@ VioScsiStartIo( ) { ENTER_FN(); - if (PreProcessRequest(DeviceExtension, Srb) || - !SendSRB(DeviceExtension, Srb)) + if (PreProcessRequest(DeviceExtension, Srb)) { - if(Srb->SrbStatus == SRB_STATUS_PENDING) - { - Srb->SrbStatus = SRB_STATUS_ERROR; - } CompleteRequest(DeviceExtension, Srb); } + else + { + SendSRB(DeviceExtension, Srb); + } EXIT_FN(); return TRUE; } @@ -623,9 +615,9 @@ VioScsiInterrupt( Srb->DataTransferLength = srbExt->Xfer; Srb->SrbStatus = SRB_STATUS_DATA_OVERRUN; } - --adaptExt->in_fly; CompleteRequest(DeviceExtension, Srb); } + ResendSRB(DeviceExtension); if (adaptExt->tmf_infly) { while((cmd = (PVirtIOSCSICmd)virtqueue_get_buf(adaptExt->vq[0], &len)) != NULL) { VirtIOSCSICtrlTMFResp *resp; @@ -814,9 +806,9 @@ VioScsiMSInterrupt ( Srb->DataTransferLength = srbExt->Xfer; Srb->SrbStatus = SRB_STATUS_DATA_OVERRUN; } - --adaptExt->in_fly; CompleteRequest(DeviceExtension, Srb); } + ResendSRB(DeviceExtension); return TRUE; } return FALSE; @@ -938,6 +930,7 @@ ENTER_FN(); RhelDbgPrint(TRACE_LEVEL_VERBOSE, ("<-->%s (%d::%d::%d)\n", DbgGetScsiOpStr(Srb), Srb->PathId, Srb->TargetId, Srb->Lun)); memset(srbExt, 0, sizeof(*srbExt)); + srbExt->Srb = Srb; cmd = &srbExt->cmd; cmd->sc = Srb; @@ -1035,13 +1028,12 @@ ENTER_FN(); { case SCSIOP_READ_CAPACITY: case SCSIOP_READ_CAPACITY16: - if (!StorPortSetDeviceQueueDepth( DeviceExtension, Srb->PathId, - Srb->TargetId, Srb->Lun, - adaptExt->queue_depth)) { - RhelDbgPrint(TRACE_LEVEL_ERROR, ("StorPortSetDeviceQueueDepth(%p, %x) failed.\n", - DeviceExtension, - adaptExt->queue_depth)); - } + if (!StorPortSetDeviceQueueDepth(DeviceExtension, Srb->PathId, + Srb->TargetId, Srb->Lun, + adaptExt->queue_depth)) + { + RhelDbgPrint(TRACE_LEVEL_ERROR, ("StorPortSetDeviceQueueDepth failed\n")); + } break; default: break; diff --git a/vioscsi/vioscsi.h b/vioscsi/vioscsi.h old mode 100644 new mode 100755 index 73869a7..45ef911 --- a/vioscsi/vioscsi.h +++ b/vioscsi/vioscsi.h @@ -208,6 +208,8 @@ typedef struct vring_desc_alias #pragma pack(1) typedef struct _SRB_EXTENSION { + LIST_ENTRY list_entry; + PSCSI_REQUEST_BLOCK Srb; ULONG out; ULONG in; ULONG Xfer; @@ -245,9 +247,11 @@ typedef struct _ADAPTER_EXTENSION { TMF_COMMAND tmf_cmd; BOOLEAN tmf_infly; - ULONG in_fly; PVirtIOSCSIEventNode events; + + LIST_ENTRY list_head; + }ADAPTER_EXTENSION, * PADAPTER_EXTENSION; #if (MSI_SUPPORTED == 1) -- 1.9.3 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver 2014-10-30 11:15 ` Vadim Rozenfeld @ 2014-10-31 1:55 ` Wangting (Kathy) 0 siblings, 0 replies; 15+ messages in thread From: Wangting (Kathy) @ 2014-10-31 1:55 UTC (permalink / raw) To: Vadim Rozenfeld; +Cc: qemu-devel Thank you, I will test it. Best regards, Ting On 2014-10-30 19:15, Vadim Rozenfeld wrote: > On Thu, 2014-10-30 at 18:28 +0800, Wangting (Kathy) wrote: >> >> On 2014-10-30 16:48, Vadim Rozenfeld wrote: >>> On Thu, 2014-10-30 at 14:54 +0800, Wangting (Kathy) wrote: >>>>> On Tue, 2014-02-18 at 13:11 -0800, Nicholas A. Bellinger wrote: >>>>>> On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: >>>>>>> On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: >>>>>>> >>>>>>> <SNIP> >>>>>>> >>>>>>>>>>> Hi Yan, >>>>>>>>>>> >>>>>>>>>>> So recently I've been doing some KVM guest performance comparisons >>>>>>>>>>> between the scsi-mq prototype using virtio-scsi + vhost-scsi, and >>>>>>>>>>> Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + >>>>>>>>>>> vhost-scsi using PCIe flash backend devices. >>>>>>>>>>> >>>>>>>>>>> I've noticed that small block random performance for the MSFT guest >>>>>>>>>>> is >>>>>>>>>>> at around ~80K IOPs with multiple vioscsi LUNs + adapters, which >>>>>>>>>>> ends up >>>>>>>>>>> being well below what the Linux guest with scsi-mq + virtio-scsi is >>>>>>>>>>> capable of (~500K). >>>>>>>>>>> >>>>>>>>>>> After searching through the various vioscsi registry settings, it >>>>>>>>>>> appears that MSIEnabled is being explicitly disabled (0x00000000), >>>>>>>>>>> that >>>>>>>>>>> is different from what vioscsi.inx is currently defining: >>>>>>>>>>> >>>>>>>>>>> [pnpsafe_pci_addreg_msix] >>>>>>>>>>> HKR, "Interrupt Management",, 0x00000010 >>>>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties",, >>>>>>>>>>> 0x00000010 >>>>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties", >>>>>>>>>>> MSISupported, 0x00010001, 0 >>>>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties", >>>>>>>>>>> MessageNumberLimit, 0x00010001, 4 >>>>>>>>>>> >>>>>>>>>>> Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 >>>>>>>>>>> is >>>>>>>>>>> explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as >>>>>>>>>>> well >>>>>>>>>>> as VioScsiFindAdapter() code always ends setting msix_enabled = >>>>>>>>>>> FALSE >>>>>>>>>>> here, regardless of MSI_SUPPORTED: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 >>>>>>>>>>> >>>>>>>>>>> Also looking at virtio_stor.c for the raw block driver, >>>>>>>>>>> MSI_SUPPORTED=1 >>>>>>>>>>> appears to be the default setting for the driver included in the >>>>>>>>>>> offical >>>>>>>>>>> virtio-win iso builds, right..? >>>>>>>>>>> >>>>>>>>>>> Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys >>>>>>>>>>> build of my own, but before going down the WDK development rabbit >>>>>>>>>>> whole, >>>>>>>>>>> I'd like to better understand why you've explicitly disabled this >>>>>>>>>>> logic >>>>>>>>>>> within vioscsi.c code to start..? >>>>>>>>>>> >>>>>>>>>>> Is there anything that needs to be addressed / carried over from >>>>>>>>>>> virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c >>>>>>>>>>> miniport code..? >>>>>>>>> >>>>>>>>> Hi Nicholas, >>>>>>>>> >>>>>>>>> I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some >>>>>>>>> reasons decided to keep it disabled until adding mq support. >>>>>>>>> >>>>>>>>> >>>>>>>>> You definitely should be able to turn on MSI_SUPPORTED, rebuild the >>>>>>>>> driver, and switch MSISupported to 1 to make vioscsi driver working in >>>>>>>>> MSI mode. >>>>>>>>> >>>>>>>> >>>>>>>> Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over >>>>>>>> the next days with a test build on Server 2012 / Server 2008 R2 and see >>>>>>>> how things go.. >>>>>>>> >>>>>>> >>>>>>> Just a quick update on progress. >>>>>>> >>>>>>> I've been able to successfully build + load a unsigned vioscsi.sys >>>>>>> driver on Server 2012 with WDK 8.0. >>>>>>> >>>>>>> Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant >>>>>>> performance and efficiency gain, on the order of 100K to 225K IOPs for >>>>>>> 4K block random I/O workload, depending on read/write mix. >>>>>>> >>>>>> >>>>>> One other performance related question.. >>>>>> >>>>>> In vioscsi.c:VioScsiFindAdapter() code, the default setting for >>>>>> adaptExt->queue_depth ends up getting set to 32 (pageNum / 4) when >>>>>> indirect mode is enabled in the following bits: >>>>>> >>>>>> if(adaptExt->indirect) { >>>>>> adaptExt->queue_depth = max(2, (pageNum / 4)); >>>>>> } else { >>>>>> adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks >>>>>> - 1; >>>>>> } >>>>>> >>>>>> Looking at viostor/virtio_stor.c:VirtIoFindAdapter() code, the default >>>>>> setting for ->queue_depth appears to be 128 (pageNum): >>>>>> >>>>>> #if (INDIRECT_SUPPORTED) >>>>>> if(!adaptExt->dump_mode) { >>>>>> adaptExt->indirect = CHECKBIT(adaptExt->features, >>>>>> VIRTIO_RING_F_INDIRECT_DESC); >>>>>> } >>>>>> if(adaptExt->indirect) { >>>>>> adaptExt->queue_depth = pageNum; >>>>>> } >>>>>> #else >>>>>> adaptExt->indirect = 0; >>>>>> #endif >>>>>> >>>>>> Is there a reason for the lower queue_depth for vioscsi vs. viostor..? >>>>> >>>>> It's a horrible work around for the following bug: >>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1013443 >>>>> >>>>> I'm going to remove it as soon as found better solution for it. >>>>> >>>>> Best regards, >>>>> Vadim. >>>>> >>>>> >>>> Hi Vadim, >>>> >>>> I have found that Bug 1013443 has been closed with a >>>> resolution of ERRATA. >>>> >>>> The windows device queue must be between 20 and 254 >>>> for StorPortSetDeviceQueueDepth to succeed. >>>> >>>> So I have the question that why queue_depth can not be >>>> set to pageNum(128)? >>> >>> It will create a problem on multi disk setup, when several >>> disks are attached to the same virtio-scsi pci controller. >>> Adding some sort of manually managed SRBs queue for storing and >>> resubmitting pending requests can solve this problem. >>> >>> Cheers, >>> Vadim. >>> >> >> Is there a patch for it? > > Yes, not committed and not fully tested yet. Please, find it attached. > > Best, > Vadim. > >> >>>> >>>> Best wishes, >>>> Ting Wang >>>> >>>>>> >>>>>> How about using min(adaptExt->scsi_config.cmd_per_lun, pageNum) instead..? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> -nab >>>>>> >>>>>> >>>> >>>> >>> >>> >>> >>> . >>> >> >> > ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2014-10-31 15:20 UTC | newest] Thread overview: 15+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-02-07 20:14 [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver Nicholas A. Bellinger 2014-02-09 9:24 ` Yan Vugenfirer 2014-02-09 11:35 ` Vadim Rozenfeld 2014-02-10 19:05 ` Nicholas A. Bellinger 2014-02-18 21:00 ` Nicholas A. Bellinger 2014-02-18 21:11 ` Nicholas A. Bellinger 2014-02-19 7:47 ` Vadim Rozenfeld 2014-02-19 8:03 ` Vadim Rozenfeld 2014-02-19 23:25 ` Nicholas A. Bellinger 2014-02-21 2:14 ` Vadim Rozenfeld -- strict thread matches above, loose matches on Subject: below -- 2014-10-30 6:54 Wangting (Kathy) 2014-10-30 8:48 ` Vadim Rozenfeld 2014-10-30 10:28 ` Wangting (Kathy) 2014-10-30 11:15 ` Vadim Rozenfeld 2014-10-31 1:55 ` Wangting (Kathy)
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).