Re: [PATCH RFC 0/9] A rendezvous module

[Jason Gunthorpe <jgg@nvidia.com>]

On Fri, Mar 19, 2021 at 02:49:29PM +0000, Wan, Kaike wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Friday, March 19, 2021 9:53 AM
> > To: Wan, Kaike <kaike.wan@intel.com>
> > Cc: dledford@redhat.com; linux-rdma@vger.kernel.org; Rimmer, Todd
> > <todd.rimmer@intel.com>
> > Subject: Re: [PATCH RFC 0/9] A rendezvous module
> > 
> > On Fri, Mar 19, 2021 at 08:56:26AM -0400, kaike.wan@intel.com wrote:
> > 
> > > - Basic mode of operations (PSM3 is used as an example for user
> > >   applications):
> > >   - A middleware (like MPI) has out-of-band communication channels
> > >     between any two nodes, which are used to establish high performance
> > >     communications for providers such as PSM3.
> > 
> > Huh? Doesn't PSM3 already use it's own special non-verbs char devices that
> > already have memory caches and other stuff? Now you want to throw that
> > all away and do yet another char dev just for HFI? Why?

> [Wan, Kaike] I think that you are referring to PSM2, which uses the
> OPA hfi1 driver that is specific to the OPA hardware.  PSM3 uses
> standard verbs drivers and supports standard RoCE.  

Uhhh.. "PSM" has always been about the ipath special char device, and
if I recall properly the library was semi-discontinued and merged into
libfabric.

So here you are talking about a libfabric verbs provider that doesn't
use the ipath style char interface but uses verbs and this rv thing so
we call it a libfabric PSM3 provider because thats not confusing to
anyone at all..

> A focus is the Intel RDMA Ethernet NICs. As such it cannot use the
> hfi1 driver through the special PSM2 interface. 

These are the drivers that aren't merged yet, I see. So why are you
sending this now? I'm not interested to look at even more Intel code
when their driver saga is still ongoing for years.

> Rather it works with the hfi1 driver through standard verbs
> interface.

But nobody would do that right? You'd get better results using the
hif1 native interfaces instead of their slow fake verbs stuff.

> > I also don't know why you picked the name rv, this looks like it has little to do
> > with the usual MPI rendezvous protocol. This is all about bulk transfers. It is
> > actually a lot like RDS. Maybe you should be using RDS?

> [Wan, Kaike] While there are similarities in concepts, details are
> different.  

You should list these differences.

> Quite frankly this could be viewed as an application accelerator
> much like RDS served that purpose for Oracle, which continues to be
> its main use case.

Obviously, except it seems to be doing the same basic acceleration
technique as RDS.

> The name "rv" is chosen simply because this module is designed to
> enable the rendezvous protocol of the MPI/OFI/PSM3 application stack
> for large messages. Short messages are handled by eager transfer
> through UDP in PSM3.

A bad name seems like it will further limit potential re-use of this
code.

Jason