Re: [PATCH v4] PCI: vmd: Disable MSI remap only for low MSI count

[Bjorn Helgaas <helgaas@kernel.org>]

On Mon, May 06, 2024 at 05:39:01PM -0700, Nirmal Patel wrote:
> On Fri, 26 Apr 2024 17:39:57 -0500
> Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Thu, Apr 18, 2024 at 11:31:21AM -0400, Nirmal Patel wrote:
> ...

> > A reference to ee81ee84f873 ("PCI: vmd: Disable MSI-X remapping when
> > possible"), which added VMD_FEAT_CAN_BYPASS_MSI_REMAP, might be
> > useful because it has nice context.
> > 
> > IIUC this will keep MSI-X remapping enabled in more cases, e.g., on
> > new devices that support more vectors.  What is the benefit of keeping
> > it enabled?
> 
> VMD MSI-X remapping was a performance bottleneck in certain
> situations. Starting from 28c0, VMD has a capability to disable MSI-X
> remapping and improve the I/O performance. The first iteration of 28c0
> VMD HW had only 64 MSI-X vectors support while the newer iterations can
> support up to 128 and VMD is no longer a bottleneck. So I thought it
> would be a good idea to change it to MSI-X remapping default ON.
> 
> Also upon further testings, I noticed huge boost in performance because
> of this CID patch:
> https://lore.kernel.org/kvm/20240423174114.526704-5-jacob.jun.pan@linux.intel.com/T/ 
> 
> The performance boost we get from the CID patch as follow:
> Kernel 6.8.8 : 1Drive: 2000, 4Drives: 2300
> 6.9.0-rc6 + CID + MSI-X remap Disable: 1Drive: 2700, 4Drives: 6010
> 6.9.0-rc6 + CID + MSI-X remap Enabled: 1Drive: 2700, 4Drives: 6100
> 
> Since there is no significant performance difference between MSI-X
> enable and disable after addition of CID patch, I think we can drop this
> patch for now until we see significant change in I/O performance due to
> VMD's MSI-X remapping policy.

OK, great, thanks for the background and the performance testing!

Bjorn