From: "Wangting (Kathy)" <kathy.wangting@huawei.com>
To: Vadim Rozenfeld <vrozenfe@redhat.com>
Cc: qemu-devel@nongnu.org
Subject: Re: [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver
Date: Thu, 30 Oct 2014 18:28:44 +0800 [thread overview]
Message-ID: <545212DC.2000503@huawei.com> (raw)
In-Reply-To: <1414658930.27378.24.camel@oscar>
On 2014-10-30 16:48, Vadim Rozenfeld wrote:
> On Thu, 2014-10-30 at 14:54 +0800, Wangting (Kathy) wrote:
>>> On Tue, 2014-02-18 at 13:11 -0800, Nicholas A. Bellinger wrote:
>>>> On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote:
>>>>> On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote:
>>>>>
>>>>> <SNIP>
>>>>>
>>>>>>>>> Hi Yan,
>>>>>>>>>
>>>>>>>>> So recently I've been doing some KVM guest performance comparisons
>>>>>>>>> between the scsi-mq prototype using virtio-scsi + vhost-scsi, and
>>>>>>>>> Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) +
>>>>>>>>> vhost-scsi using PCIe flash backend devices.
>>>>>>>>>
>>>>>>>>> I've noticed that small block random performance for the MSFT guest
>>>>>>>>> is
>>>>>>>>> at around ~80K IOPs with multiple vioscsi LUNs + adapters, which
>>>>>>>>> ends up
>>>>>>>>> being well below what the Linux guest with scsi-mq + virtio-scsi is
>>>>>>>>> capable of (~500K).
>>>>>>>>>
>>>>>>>>> After searching through the various vioscsi registry settings, it
>>>>>>>>> appears that MSIEnabled is being explicitly disabled (0x00000000),
>>>>>>>>> that
>>>>>>>>> is different from what vioscsi.inx is currently defining:
>>>>>>>>>
>>>>>>>>> [pnpsafe_pci_addreg_msix]
>>>>>>>>> HKR, "Interrupt Management",, 0x00000010
>>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties",,
>>>>>>>>> 0x00000010
>>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties",
>>>>>>>>> MSISupported, 0x00010001, 0
>>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties",
>>>>>>>>> MessageNumberLimit, 0x00010001, 4
>>>>>>>>>
>>>>>>>>> Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0
>>>>>>>>> is
>>>>>>>>> explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as
>>>>>>>>> well
>>>>>>>>> as VioScsiFindAdapter() code always ends setting msix_enabled =
>>>>>>>>> FALSE
>>>>>>>>> here, regardless of MSI_SUPPORTED:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340
>>>>>>>>>
>>>>>>>>> Also looking at virtio_stor.c for the raw block driver,
>>>>>>>>> MSI_SUPPORTED=1
>>>>>>>>> appears to be the default setting for the driver included in the
>>>>>>>>> offical
>>>>>>>>> virtio-win iso builds, right..?
>>>>>>>>>
>>>>>>>>> Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys
>>>>>>>>> build of my own, but before going down the WDK development rabbit
>>>>>>>>> whole,
>>>>>>>>> I'd like to better understand why you've explicitly disabled this
>>>>>>>>> logic
>>>>>>>>> within vioscsi.c code to start..?
>>>>>>>>>
>>>>>>>>> Is there anything that needs to be addressed / carried over from
>>>>>>>>> virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c
>>>>>>>>> miniport code..?
>>>>>>>
>>>>>>> Hi Nicholas,
>>>>>>>
>>>>>>> I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some
>>>>>>> reasons decided to keep it disabled until adding mq support.
>>>>>>>
>>>>>>>
>>>>>>> You definitely should be able to turn on MSI_SUPPORTED, rebuild the
>>>>>>> driver, and switch MSISupported to 1 to make vioscsi driver working in
>>>>>>> MSI mode.
>>>>>>>
>>>>>>
>>>>>> Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over
>>>>>> the next days with a test build on Server 2012 / Server 2008 R2 and see
>>>>>> how things go..
>>>>>>
>>>>>
>>>>> Just a quick update on progress.
>>>>>
>>>>> I've been able to successfully build + load a unsigned vioscsi.sys
>>>>> driver on Server 2012 with WDK 8.0.
>>>>>
>>>>> Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant
>>>>> performance and efficiency gain, on the order of 100K to 225K IOPs for
>>>>> 4K block random I/O workload, depending on read/write mix.
>>>>>
>>>>
>>>> One other performance related question..
>>>>
>>>> In vioscsi.c:VioScsiFindAdapter() code, the default setting for
>>>> adaptExt->queue_depth ends up getting set to 32 (pageNum / 4) when
>>>> indirect mode is enabled in the following bits:
>>>>
>>>> if(adaptExt->indirect) {
>>>> adaptExt->queue_depth = max(2, (pageNum / 4));
>>>> } else {
>>>> adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks
>>>> - 1;
>>>> }
>>>>
>>>> Looking at viostor/virtio_stor.c:VirtIoFindAdapter() code, the default
>>>> setting for ->queue_depth appears to be 128 (pageNum):
>>>>
>>>> #if (INDIRECT_SUPPORTED)
>>>> if(!adaptExt->dump_mode) {
>>>> adaptExt->indirect = CHECKBIT(adaptExt->features,
>>>> VIRTIO_RING_F_INDIRECT_DESC);
>>>> }
>>>> if(adaptExt->indirect) {
>>>> adaptExt->queue_depth = pageNum;
>>>> }
>>>> #else
>>>> adaptExt->indirect = 0;
>>>> #endif
>>>>
>>>> Is there a reason for the lower queue_depth for vioscsi vs. viostor..?
>>>
>>> It's a horrible work around for the following bug:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1013443
>>>
>>> I'm going to remove it as soon as found better solution for it.
>>>
>>> Best regards,
>>> Vadim.
>>>
>>>
>> Hi Vadim,
>>
>> I have found that Bug 1013443 has been closed with a
>> resolution of ERRATA.
>>
>> The windows device queue must be between 20 and 254
>> for StorPortSetDeviceQueueDepth to succeed.
>>
>> So I have the question that why queue_depth can not be
>> set to pageNum(128)?
>
> It will create a problem on multi disk setup, when several
> disks are attached to the same virtio-scsi pci controller.
> Adding some sort of manually managed SRBs queue for storing and
> resubmitting pending requests can solve this problem.
>
> Cheers,
> Vadim.
>
Is there a patch for it?
>>
>> Best wishes,
>> Ting Wang
>>
>>>>
>>>> How about using min(adaptExt->scsi_config.cmd_per_lun, pageNum) instead..?
>>>>
>>>> Thanks!
>>>>
>>>> -nab
>>>>
>>>>
>>
>>
>
>
>
> .
>
next prev parent reply other threads:[~2014-10-30 10:29 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-10-30 6:54 [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver Wangting (Kathy)
2014-10-30 8:48 ` Vadim Rozenfeld
2014-10-30 10:28 ` Wangting (Kathy) [this message]
2014-10-30 11:15 ` Vadim Rozenfeld
2014-10-31 1:55 ` Wangting (Kathy)
2014-12-09 8:20 ` [Qemu-devel] question about vioscsi queue depth patch Wangting (Kathy)
2014-12-09 9:07 ` Vadim Rozenfeld
-- strict thread matches above, loose matches on Subject: below --
2014-02-07 20:14 [Qemu-devel] MSI interrupt support with vioscsi.c miniport driver Nicholas A. Bellinger
2014-02-09 9:24 ` Yan Vugenfirer
2014-02-09 11:35 ` Vadim Rozenfeld
2014-02-10 19:05 ` Nicholas A. Bellinger
2014-02-18 21:00 ` Nicholas A. Bellinger
2014-02-18 21:11 ` Nicholas A. Bellinger
2014-02-19 7:47 ` Vadim Rozenfeld
2014-02-19 8:03 ` Vadim Rozenfeld
2014-02-19 23:25 ` Nicholas A. Bellinger
2014-02-21 2:14 ` Vadim Rozenfeld
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=545212DC.2000503@huawei.com \
--to=kathy.wangting@huawei.com \
--cc=qemu-devel@nongnu.org \
--cc=vrozenfe@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).