From: "Patel, Nirmal" <nirmal.patel@linux.intel.com>
To: Xinghui Li <korantwork@gmail.com>,
Jonathan Derrick <jonathan.derrick@linux.dev>
Cc: Keith Busch <kbusch@kernel.org>,
lpieralisi@kernel.org, linux-pci@vger.kernel.org,
linux-kernel@vger.kernel.org, Xinghui Li <korantli@tencent.com>
Subject: Re: [PATCH] PCI: vmd: Do not disable MSI-X remapping in VMD 28C0 controller
Date: Mon, 6 Feb 2023 11:11:36 -0700 [thread overview]
Message-ID: <9e6cfda1-4309-d1bb-a0cf-404a7fe7d3aa@linux.intel.com> (raw)
In-Reply-To: <CAEm4hYWeZFrYxSvAcBJ8iw=t507vZMqfBwiQXFSJd2Hcyfw7fA@mail.gmail.com>
On 2/6/2023 5:45 AM, Xinghui Li wrote:
> Friendly ping~
>
> Xinghui Li <korantwork@gmail.com> 于2023年1月10日周二 20:28写道：
>> Jonathan Derrick <jonathan.derrick@linux.dev> 于2023年1月10日周二 05:00写道：
>>> As the bypass mode seems to affect performance greatly depending on the specific configuration,
>>> it may make sense to use a moduleparam to control it
>>>
>> We found that each pcie port can mount four drives. If we only test 2
>> or 1 dirve of one pcie port,
>> the performance of the drive performance will be normal. Also, we
>> observed the interruptions in different modes.
>> bypass:
>> .....
>> 2022-12-28-11-39-14: 1224 181665 IR-PCI-MSI 201850948-edge nvme0q68
>> 2022-12-28-11-39-14: 1179 180115 IR-PCI-MSI 201850945-edge nvme0q65
>> 2022-12-28-11-39-14: RES 26743 Rescheduling interrupts
>> 2022-12-28-11-39-17: irqtop - IRQ : 3029, TOTAL : 2100315228, CPU :
>> 192, ACTIVE CPU : 192
>> disable:
>> ......
>> 2022-12-28-12-05-56: 1714 169797 IR-PCI-MSI 14155850-edge nvme1q74
>> 2022-12-28-12-05-56: 1701 168753 IR-PCI-MSI 14155849-edge nvme1q73
>> 2022-12-28-12-05-56: LOC 163697 Local timer interrupts
>> 2022-12-28-12-05-56: TLB 5465 TLB shootdowns
>> 2022-12-28-12-06-00: irqtop - IRQ : 3029, TOTAL : 2179022106, CPU :
>> 192, ACTIVE CPU : 192
>> remapping:
>> 022-12-28-11-25-38: 283 325568 IR-PCI-MSI 24651790-edge vmd3
>> 2022-12-28-11-25-38: 140 267899 IR-PCI-MSI 13117447-edge vmd1
>> 2022-12-28-11-25-38: 183 265978 IR-PCI-MSI 13117490-edge vmd1
>> ......
>> 2022-12-28-11-25-42: irqtop - IRQ : 2109, TOTAL : 2377172002, CPU :
>> 192, ACTIVE CPU : 192
>>
>> From the result it is not difficult to find, in remapping mode the
>> interruptions come from vmd.
>> While in other modes, interrupts come from nvme devices. Besides, we
>> found the port mounting
>> 4 dirves total interruptions is much fewer than the port mounting 2 or 1 drive.
>> NVME 8 and 9 mount in one port, other port mount 4 dirves.
>>
>> 2022-12-28-11-39-14: 2582 494635 IR-PCI-MSI 470810698-edge nvme9q74
>> 2022-12-28-11-39-14: 2579 489972 IR-PCI-MSI 470810697-edge nvme9q73
>> 2022-12-28-11-39-14: 2573 480024 IR-PCI-MSI 470810695-edge nvme9q71
>> 2022-12-28-11-39-14: 2544 312967 IR-PCI-MSI 470286401-edge nvme8q65
>> 2022-12-28-11-39-14: 2556 312229 IR-PCI-MSI 470286405-edge nvme8q69
>> 2022-12-28-11-39-14: 2547 310013 IR-PCI-MSI 470286402-edge nvme8q66
>> 2022-12-28-11-39-14: 2550 308993 IR-PCI-MSI 470286403-edge nvme8q67
>> 2022-12-28-11-39-14: 2559 308794 IR-PCI-MSI 470286406-edge nvme8q70
>> ......
>> 2022-12-28-11-39-14: 1296 185773 IR-PCI-MSI 202375243-edge nvme1q75
>> 2022-12-28-11-39-14: 1209 185646 IR-PCI-MSI 201850947-edge nvme0q67
>> 2022-12-28-11-39-14: 1831 184151 IR-PCI-MSI 203423828-edge nvme3q84
>> 2022-12-28-11-39-14: 1254 182313 IR-PCI-MSI 201850950-edge nvme0q70
>> 2022-12-28-11-39-14: 1224 181665 IR-PCI-MSI 201850948-edge nvme0q68
>> 2022-12-28-11-39-14: 1179 180115 IR-PCI-MSI 201850945-edge nvme0q65
>>> I'd vote for it being in VMD mode (non-bypass) by default.
>> I speculate that the vmd controller equalizes the interrupt load and
>> acts like a buffer,
>> which improves the performance of nvme. I am not sure about my
>> analysis. So, I'd like
>> to discuss it with the community.
I like the idea of module parameter to allow switching between the modes
but keep MSI remapping enabled (non-bypass) by default.
next prev parent reply other threads:[~2023-02-06 18:12 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-12-22 7:26 [PATCH] PCI: vmd: Do not disable MSI-X remapping in VMD 28C0 controller korantwork
2022-12-22 9:15 ` Jonathan Derrick
2022-12-22 21:56 ` Keith Busch
2022-12-23 8:02 ` Xinghui Li
2022-12-27 22:32 ` Jonathan Derrick
2022-12-28 2:19 ` Xinghui Li
2023-01-09 21:00 ` Jonathan Derrick
2023-01-10 12:28 ` Xinghui Li
2023-02-06 12:45 ` Xinghui Li
2023-02-06 18:11 ` Patel, Nirmal [this message]
2023-02-06 18:28 ` Keith Busch
2023-02-07 3:18 ` Xinghui Li
2023-02-07 20:32 ` Patel, Nirmal
2023-02-09 12:05 ` Xinghui Li
2023-02-09 23:05 ` Keith Busch
2023-02-09 23:57 ` Patel, Nirmal
2023-02-10 0:47 ` Keith Busch
2022-12-23 7:53 ` Xinghui Li
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9e6cfda1-4309-d1bb-a0cf-404a7fe7d3aa@linux.intel.com \
--to=nirmal.patel@linux.intel.com \
--cc=jonathan.derrick@linux.dev \
--cc=kbusch@kernel.org \
--cc=korantli@tencent.com \
--cc=korantwork@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=lpieralisi@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.