Re: [PATCH for-next v3] RDMA/efa: Add EFA query MR support

[Jason Gunthorpe <jgg@nvidia.com>]

On Wed, Dec 13, 2023 at 07:05:24PM +0200, Margolin, Michael wrote:
> > On Mon, Dec 11, 2023 at 05:47:15PM +0000, Michael Margolin wrote:
> >> diff --git a/drivers/infiniband/hw/efa/efa_admin_cmds_defs.h b/drivers/infiniband/hw/efa/efa_admin_cmds_defs.h
> >> index 9c65bd27bae0..597f7ca6f31d 100644
> >> --- a/drivers/infiniband/hw/efa/efa_admin_cmds_defs.h
> >> +++ b/drivers/infiniband/hw/efa/efa_admin_cmds_defs.h
> >> @@ -415,6 +415,32 @@ struct efa_admin_reg_mr_resp {
> >>        * memory region
> >>        */
> >>       u32 r_key;
> >> +
> >> +     /*
> >> +      * Mask indicating which fields have valid values
> >> +      * 0 : recv_pci_bus_id
> >> +      * 1 : rdma_read_pci_bus_id
> >> +      * 2 : rdma_recv_pci_bus_id
> >> +      */
> >> +     u8 validity;
> >> +
> >> +     /*
> >> +      * Physical PCIe bus used by the device to reach the MR for receive
> >> +      * operation
> >> +      */
> >> +     u8 recv_pci_bus_id;
> >> +
> >> +     /*
> >> +      * Physical PCIe bus used by the device to reach the MR for RDMA read
> >> +      * operation
> >> +      */
> >> +     u8 rdma_read_pci_bus_id;
> >> +
> >> +     /*
> >> +      * Physical PCIe bus used by the device to reach the MR for RDMA write
> >> +      * receive
> >> +      */
> >> +     u8 rdma_recv_pci_bus_id;
> > What driver is bound to this other PCIe bus and how did the iommu get
> > setup for it?
> 
> It's internal bus that is not directly exposed to the host. Addresses
> mapping is acquired from accelerator's driver as for any MR residing in
> accelerator memory, and the translation is owned by devices on that bus.

So if it isn't visible to the host, or connectable to anything Linux
would call a PCI RID, why are you giving it such specific names? Just
call it 'interconnect path id' or something and make it opaque?

Jason