Re: [PATCH v5 00/10] Support using MSI interrupts in ntb_transport

Jon Mason
2019-06-13T15:14:33Z

On Thu, May 23, 2019 at 04:30:50PM -0600, Logan Gunthorpe wrote:
> This is another resend as there has been no feedback since v4.
> Seems Jon has been MIA this past cycle so hopefully he appears on the
> list soon.
>
> I've addressed the feedback so far and rebased on the latest kernel
> and would like this to be considered for merging this cycle.
>
> The only outstanding issue I know of is that it still will not work
> with IDT hardware, but ntb_transport doesn't work with IDT hardware
> and there is still no sensible common infrastructure to support
> ntb_peer_mw_set_trans(). Thus, I decline to consider that complication
> in this patchset. However, I'll be happy to review work that adds this
> feature in the future.
>
> Also, as the port number and resource index stuff is a bit complicated,
> I made a quick out of tree test fixture to ensure it's correct[1]. As
> an excerise I also wrote some test code[2] using the upcomming KUnit
> feature.

Sorry for the delay. The patch is now in the ntb-next branch. We've
missed window for 5.2, but it will be in the 5.3 pull request (barring
last minute comments).

Thanks,
Jon

>
> Logan
>
> [1] https://repl.it/repls/ExcitingPresentFile
> [2] https://github.com/sbates130272/linux-p2pmem/commits/ntb_kunit
>
> --
>
> Changes in v5:
>
> * Rebased onto v5.2-rc1 (plus the patches in ntb-next)
>
> --
>
> Changes in v4:
>
> * Rebased onto v5.1-rc6 (No changes)
>
> * Numerous grammar and spelling mistakes spotted by Bjorn
>
> --
>
> Changes in v3:
>
> * Rebased onto v5.1-rc1 (Dropped the first two patches as they have
> been merged, and cleaned up some minor conflicts in the PCI tree)
>
> * Added a new patch (#3) to calculate logical port numbers that
> are port numbers from 0 to (number of ports - 1). This is
> then used in ntb_peer_resource_idx() to fix the issues brought
> up by Serge.
>
> * Fixed missing __iomem and iowrite calls (as noticed by Serge)
>
> * Added patch 10 which describes ntb_msi_test in the documentation
> file (as requested by Serge)
>
> * A couple other minor nits and documentation fixes
>
> --
>
> Changes in v2:
>
> * Cleaned up the changes in intel_irq_remapping.c to make them
> less confusing and add a comment. (Per discussion with Jacob and
> Joerg)
>
> * Fixed a nit from Bjorn and collected his Ack
>
> * Added a Kconfig dependancy on CONFIG_PCI_MSI for CONFIG_NTB_MSI
> as the Kbuild robot hit a random config that didn't build
> without it.
>
> * Worked in a callback for when the MSI descriptor changes so that
> the clients can resend the new address and data values to the peer.
> On my test system this was never necessary, but there may be
> other platforms where this can occur. I tested this by hacking
> in a path to rewrite the MSI descriptor when I change the cpu
> affinity of an IRQ. There's a bit of uncertainty over the latency
> of the change, but without hardware this can acctually occur on
> we can't test this. This was the result of a discussion with Dave.
>
> --
>
> This patch series adds optional support for using MSI interrupts instead
> of NTB doorbells in ntb_transport. This is desirable seeing doorbells on
> current hardware are quite slow and therefore switching to MSI interrupts
> provides a significant performance gain. On switchtec hardware, a simple
> apples-to-apples comparison shows ntb_netdev/iperf numbers going from
> 3.88Gb/s to 14.1Gb/s when switching to MSI interrupts.
>
> To do this, a couple changes are required outside of the NTB tree:
>
> 1) The IOMMU must know to accept MSI requests from aliased bused numbers
> seeing NTB hardware typically sends proxied request IDs through
> additional requester IDs. The first patch in this series adds support
> for the Intel IOMMU. A quirk to add these aliases for switchtec hardware
> was already accepted. See commit ad281ecf1c7d ("PCI: Add DMA alias quirk
> for Microsemi Switchtec NTB") for a description of NTB proxy IDs and why
> this is necessary.
>
> 2) NTB transport (and other clients) may often need more MSI interrupts
> than the NTB hardware actually advertises support for. However, seeing
> these interrupts will not be triggered by the hardware but through an
> NTB memory window, the hardware does not actually need support or need
> to know about them. Therefore we add the concept of Virtual MSI
> interrupts which are allocated just like any other MSI interrupt but
> are not programmed into the hardware's MSI table. This is done in
> Patch 2 and then made use of in Patch 3.
>
> The remaining patches in this series add a library for dealing with MSI
> interrupts, a test client and finally support in ntb_transport.
>
> The series is based off of v5.1-rc6 plus the patches in ntb-next.
> A git repo is available here:
>
> https://github.com/sbates130272/linux-p2pmem/ ntb_transport_msi_v4
>
> Thanks,
>
> Logan
>
> --
>
> Logan Gunthorpe (10):
> PCI/MSI: Support allocating virtual MSI interrupts
> PCI/switchtec: Add module parameter to request more interrupts
> NTB: Introduce helper functions to calculate logical port number
> NTB: Introduce functions to calculate multi-port resource index
> NTB: Rename ntb.c to support multiple source files in the module
> NTB: Introduce MSI library
> NTB: Introduce NTB MSI Test Client
> NTB: Add ntb_msi_test support to ntb_test
> NTB: Add MSI interrupt support to ntb_transport
> NTB: Describe the ntb_msi_test client in the documentation.
>
> Documentation/ntb.txt | 27 ++
> drivers/ntb/Kconfig | 11 +
> drivers/ntb/Makefile | 3 +
> drivers/ntb/{ntb.c => core.c} | 0
> drivers/ntb/msi.c | 415 +++++++++++++++++++++++
> drivers/ntb/ntb_transport.c | 169 ++++++++-
> drivers/ntb/test/Kconfig | 9 +
> drivers/ntb/test/Makefile | 1 +
> drivers/ntb/test/ntb_msi_test.c | 433 ++++++++++++++++++++++++
> drivers/pci/msi.c | 54 ++-
> drivers/pci/switch/switchtec.c | 12 +-
> include/linux/msi.h | 8 +
> include/linux/ntb.h | 196 ++++++++++-
> include/linux/pci.h | 9 +
> tools/testing/selftests/ntb/ntb_test.sh | 54 ++-
> 15 files changed, 1386 insertions(+), 15 deletions(-)
> rename drivers/ntb/{ntb.c => core.c} (100%)
> create mode 100644 drivers/ntb/msi.c
> create mode 100644 drivers/ntb/test/ntb_msi_test.c
>
> --
> 2.20.1