Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver

[Vadim Rozenfeld <vrozenfe@redhat.com>]

On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote:
> On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote:
> 
> <SNIP>
> 
> > > > > Hi Yan,
> > > > > 
> > > > > So recently I've been doing some KVM guest performance comparisons
> > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
> > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
> > > > > vhost-scsi using PCIe flash backend devices.
> > > > > 
> > > > > I've noticed that small block random performance for the MSFT guest is
> > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up
> > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is
> > > > > capable of (~500K).
> > > > > 
> > > > > After searching through the various vioscsi registry settings, it
> > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that
> > > > > is different from what vioscsi.inx is currently defining:
> > > > > 
> > > > > [pnpsafe_pci_addreg_msix]
> > > > > HKR, "Interrupt Management",, 0x00000010
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
> > > > > 
> > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is
> > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well
> > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE
> > > > > here, regardless of MSI_SUPPORTED:
> > > > > 
> > > > >  https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
> > > > > 
> > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1
> > > > > appears to be the default setting for the driver included in the offical
> > > > > virtio-win iso builds, right..?
> > > > > 
> > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
> > > > > build of my own, but before going down the WDK development rabbit whole,
> > > > > I'd like to better understand why you've explicitly disabled this logic
> > > > > within vioscsi.c code to start..?
> > > > > 
> > > > > Is there anything that needs to be addressed / carried over from
> > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
> > > > > miniport code..?
> > > 
> > > Hi Nicholas,
> > > 
> > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
> > > reasons decided to keep it disabled until adding mq support.
> > > 
> > > 
> > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the
> > > driver, and switch MSISupported to 1 to make vioscsi driver working in
> > > MSI mode.
> > >    
> > 
> > Thanks for the quick response.  We'll give MSI_SUPPORTED=1 a shot over
> > the next days with a test build on Server 2012 / Server 2008 R2 and see
> > how things go..
> > 
> 
> Just a quick update on progress.
> 
> I've been able to successfully build + load a unsigned vioscsi.sys
> driver on Server 2012 with WDK 8.0.
> 
> Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant
> performance and efficiency gain, on the order of 100K to 225K IOPs for
> 4K block random I/O workload, depending on read/write mix.
> 
> Below is a simple patch to enable MSI operation by default.  Any chance
> to apply this separate from future mq efforts..?

Yes, we differently can enable MSI and rebuild vioscsi.
But then we need to re-spin WHQL testing for this particular
driver. This process requires a lot of resources, and I doubt that
it will be initiated soon, unless we have some significant amount of
bug-fixes.

Best regards,
Vadim.     

> 
> Thanks,
> 
> --nab
> 
> From 89adb6d5800386d44b36737d1587e0ffc09c4902 Mon Sep 17 00:00:00 2001
> From: Nicholas Bellinger <nab@linux-iscsi.org>
> Date: Fri, 14 Feb 2014 10:26:04 -0800
> Subject: [PATCH] vioscsi: Set MSI_SUPPORTED=1 by default
> 
> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
> ---
>  vioscsi/SOURCES         | 2 +-
>  vioscsi/vioscsi.c       | 2 --
>  vioscsi/vioscsi.inx     | 2 +-
>  vioscsi/vioscsi.vcxproj | 6 +++---
>  4 files changed, 5 insertions(+), 7 deletions(-)
> 
> diff --git a/vioscsi/SOURCES b/vioscsi/SOURCES
> index f2083de..f631bd2 100644
> --- a/vioscsi/SOURCES
> +++ b/vioscsi/SOURCES
> @@ -6,7 +6,7 @@ C_DEFINES = -D_MINORVERSION_=$(_BUILD_MINOR_VERSION_) $(C_DEFINES)
>  C_DEFINES = -D_NT_TARGET_MAJ=$(_NT_TARGET_MAJ) $(C_DEFINES)
>  C_DEFINES = -D_NT_TARGET_MIN=$(_RHEL_RELEASE_VERSION_) $(C_DEFINES)
>  
> -C_DEFINES = -DMSI_SUPPORTED=0 $(C_DEFINES)
> +C_DEFINES = -DMSI_SUPPORTED=1 $(C_DEFINES)
>  C_DEFINES = -DINDIRECT_SUPPORTED=1 $(C_DEFINES)
>  TARGETLIBS=$(SDK_LIB_PATH)\storport.lib ..\VirtIO\$(O)\virtiolib.lib
>  
> diff --git a/vioscsi/vioscsi.c b/vioscsi/vioscsi.c
> index 77c0e46..70b9bb4 100644
> --- a/vioscsi/vioscsi.c
> +++ b/vioscsi/vioscsi.c
> @@ -337,8 +337,6 @@ ENTER_FN();
>          adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1;
>      }
>  
> -    adaptExt->msix_enabled = FALSE;
> -
>      RhelDbgPrint(TRACE_LEVEL_ERROR, ("breaks_number = %x  queue_depth = %x\n",
>                  ConfigInfo->NumberOfPhysicalBreaks,
>                  adaptExt->queue_depth));
> diff --git a/vioscsi/vioscsi.inx b/vioscsi/vioscsi.inx
> index cc94b7c..ec717c6 100644
> --- a/vioscsi/vioscsi.inx
> +++ b/vioscsi/vioscsi.inx
> @@ -79,7 +79,7 @@ HKR, "Parameters", "BusType", %REG_DWORD%, 0x00000001
>  [pnpsafe_pci_addreg_msix]
>  HKR, "Interrupt Management",, 0x00000010
>  HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010
> -HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0
> +HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 1
>  HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4
>  HKR, "Interrupt Management\Affinity Policy",, 0x00000010
>  HKR, "Interrupt Management\Affinity Policy", DevicePolicy, 0x00010001, 5
> diff --git a/vioscsi/vioscsi.vcxproj b/vioscsi/vioscsi.vcxproj
> index 889e513..68d4e85 100644
> --- a/vioscsi/vioscsi.vcxproj
> +++ b/vioscsi/vioscsi.vcxproj
> @@ -66,7 +66,7 @@
>    <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|Win32'">
>      <ClCompile>
>        <WarningLevel>Level3</WarningLevel>
> -      <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
> +      <PreprocessorDefinitions>_X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions>
>      </ClCompile>
>      <Link>
>        <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies>
> @@ -95,7 +95,7 @@
>    <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Win8 Release|x64'">
>      <ClCompile>
>        <WarningLevel>Level3</WarningLevel>
> -      <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
> +      <PreprocessorDefinitions>_WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions)</PreprocessorDefinitions>
>      </ClCompile>
>      <Link>
>        <AdditionalDependencies>%(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib</AdditionalDependencies>
> @@ -140,4 +140,4 @@
>    <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
>    <ImportGroup Label="ExtensionTargets">
>    </ImportGroup>
> -</Project>
> \ No newline at end of file
> +</Project>