RE: [RFC PATCH 00/13] Ultra Ethernet driver introduction

Sean Hefty
2025-04-05T01:07:37Z

> On 4/1/2025 2:39 PM, Jason Gunthorpe wrote:
> >> I don't know that I can talk about the UEC spec,
> >
> > Right, it is too early to talk about UEC and Linux until people can
> > freely talk about what it actually needs.
>
> While the UE specs are not yet out, concepts can be discussed. I recognize this
> may be annoying without specs.
>
> The following is my understanding of the UE job model.
>
> A job ID identifies "who am I." There are two different operational
> modes:
>
> 1. Relative Addressing
>
> The endpoint address is only valid within the scope of the job ID.
> Endpoints can only transmit and receive on the associated job ID.
> Parallel applications will use this to restrict communication to
> only processes within the job.
>
> Processes must be granted access to the job ID. In addition,
> multiple processes may share a job ID. Some mechanism is required to
> restrict what job IDs an endpoint can be configured against. Having
> a device-level RDMA job object and a path to associate the job
> object with an endpoint seems reasonable.
>
> 2. Absolute Addressing
>
> The target endpoint address is outside the scope of the job ID. This
> behavior allows an endpoint to receive on all job IDs and transmit
> on only authorized job IDs. This mode enables server endpoints to
> support multiple clients with different job IDs.
>
> Since this mode impacts the job IDs transmitted in packets,
> processes must be granted access. A device-level RDMA job object
> seems reasonable for this as well.
>
> An optional mechanism to restrict a receive buffer and MR to a
> specific job ID is needed. This enables a server endpoint to have
> per client job ID resources. Job ID verification is unnecessary
> since the job ID associated with a receive buffer or MR does not
> impact the packet job ID.
>
> UE is going to need some object to restrict registered user-space memory.
> Having the PD as the object supporting local memory registration isolation
> seems ok. The UE object relationship could look like job <- 1 --- 0..* ->
> endpoint <- 0..* --- 1 -> PD.

There's also the MR relationship:

Job <- 1 --- 0..n -> MR <- 0..n --- 1 -> PD

There's discussion on defining this relationship:

Job <- 0..n --- 1 -> PD

I can't think of a technical reason why that's needed. Without it, the Job basically acts in the same role as a second PD, which feels off. Plus, some NIC implementations may need such a restriction.

- Sean