Re: [RFC PATCH 00/13] Ultra Ethernet driver introduction

[Jason Gunthorpe <jgg@nvidia.com>]

On Mon, Mar 24, 2025 at 01:22:13PM -0700, Roland Dreier wrote:
> Hi Jason,
> 
> I think we were not clear on the overall discussion and so we are much
> closer to agreement than you might think, see below.

Certainly this was my impression after we talked, so is hard to
understand what this series is supposed to be. It doesn't seem to
advance things towards using the RDMA subsystem.
 
> First, want to clarify that this patchset is collaborative development
> within the overall Ultra Ethernet Consortium. 

Consortiums don't get to vote their way into Linux patch acceptance.

> UEC is definitely not trying to create anything new beyond adding
> support for Ultra Ethernet. By far the bulk of this patchset is adding
> a software model of the specific Ultra Ethernet transport's protocol /
> packet handling, and that code is currently in drivers/ultraeth. I
> don't feel that pathnames are particularly important, and we could
> move the code to something like drivers/infiniband/ultraeth, but that
> seems a bit silly. But certainly we are open to suggestions.

It is not the directory name that is at issue, it is completely
ignoring all the existing infrastructure and doing something entirely
new and entirely isolated.

I expect you will have uec named files under drivers/infiniband, and
they should be integrated within existing architecture, including
extensions.

It is a shame we won't rename the directory name, or rename alot of
other stuff, but I gather that is the community preference.

> To be clear - we are not trying to reinvent or bypass uverbs, and
> there is complete agreement within UEC that we should reuse the uverbs
> infrastructure so that we get the advantages of solid, mature
> mechanisms for memory pinning, resource tracking / cleanup ordering,
> etc.

My expectation is that the software interface path for a RDMA
transport exist mainly for testing and development purposes. It should
be deliberately designed to mimic a HW driver and exercise the same
interfaces. Even if that is more work or inconvenient.

> With that said, Ultra Ethernet devices likely will not have interfaces
> that map well onto QPs, MRs, etc. so we will be sending patches to
> drivers/infiniband/uverbs* that generalize things to allow "struct
> ib_device" objects that do not implement "required verbs."

That's fine, we can look at all those things as you go along.

Just be mindful that given UEC's lack of a HW standard it will be hard
to judge if things are HW specific or UEC general. Explain in the
patches why you think many HW devices will be using new general common
objects.

Job ID is a good example that is obviously required by spec to be
common.

My advice, and the same advice I gave to Habana, is to ignore the
spelling of things like PD, QP and MR and focus on the fundamental
purpose they represent. UEC has a "QP" in that all HW devices will
have some kind of queue structure to userspace. UEC has "PD" in that
it must have some kind of HW security boundary to keep one uverbs
context from touching another's resources (it may be that job is how
UEC spells PD), and so on.

Use driver specific calls when appropriate.

kernel-uet will be a different conversation, and I suspect kernel uet
will be very feature limited to focus just on something like storage.

> I think the netlink API and job handling overall is the area where the
> most discussion is probably required. UE is somewhat novel in

Yes, that is new, but also an idea that is being copied.

> elevating the concept of a "job" to a standard object with specific
> properties that determine the values in packet headers. But I'm open
> to making "job" a top-level RDMA object...

I think this is right

> I guess the idea would be
> to define an interface for creating a new type of "job FD" with a
> standard ABI for setting properties?

I suspect so? /dev/infiniband/job perhaps where opening the FD creates
a job container then some ioctls to realize it into a per-protocol job
description with per-protocol additional properties?

Present a job FD to a uverbs FD to join the job's security context

Another variation might be an entire jobfs but I would probably start
with job FD first and only do a jobfs later if people demand it..

I think CAP_SYS_NET_ADMIN is a bad security model for jobs.

Regards,
Jason