RE: [RFC PATCH 00/13] Ultra Ethernet driver introduction

Sean Hefty
2025-04-08T04:40:18Z

> > There's also the MR relationship:
> >
> > Job <- 1 --- 0..n -> MR <- 0..n --- 1 -> PD
>
> Current UE memory registration is centered around the libfabric
> FI_MR_ENDPOINT mode which states memory regions are associated with an
> endpoint instead of libfabric domain. For remotely accessible UE MRs, this
> translates to the following.
>
> - Relative Addressing: {job ID, endpoint, RKEY} identifies the MR.
>
> - Absolute Addressing: {endpoint, RKEY} identifies the MR with optional
> MR job-ID access control check.
>
> In addition, UE memory registration supports user-defined RKEYs. This enables
> programming model implementations the optimization to use well-known per
> process endpoint RKEYs. For relative addressing, this could result in the same
> RKEY value existing multiple times at the {job ID} level, but only once at the
> {job ID, endpoint} level.
>
> The UE remote MR relationship would look like:
>
> Job <- 1 --- 0..n -> EP <- 0..n ---------------------- 1 -> PD
> ^ ^
> |--- 1 --- 0..n -> MR <- 0..n --- 1 ---|

IMO, job support should be transport agnostic and include connection-oriented RDMA transports.

I might have ended up too concise. Transport header fields identify QP, MR, and job objects. I think in terms of these minimal mappings:

QPN -> QP
rkey -> MR
(also lkey -> MR)
Job ID -> job

It's an oversimplification. Obviously, QPN=1 uses other fields. Maybe Job ID is actually 'pkey'. Each vendor owns mapping transport fields to SW visible objects.

Libfabric supports both vendor and user selected rkeys. I don't think this matters to the model. If rkeys are device unique, rkey -> MR is trivial. If rkeys are not, that's a feature-complexity trade-off. Similarly, a vendor may support 1 job per QP or many. Regardless, I still view the security model as finding a valid tuple:

{PD, QP, MR, Job}

The MR and/or Job may be N/A for any given transfer or configuration.

> > There's discussion on defining this relationship:
> >
> > Job <- 0..n --- 1 -> PD
> >
> > I can't think of a technical reason why that's needed.
>
> From my UE perspective, I agree. UE needs to share job IDs across processes
> while still having inter-process isolation for things like local memory
> registrations.

I distinguish job instance running on the cluster from an instance of some job object. Two object instances, identified by some other scope, could result in the same job ID. (E.g. each process has its own object.)

I lean towards defining a device specific job object, assuming HW resources may be required. Job attributes include selected transport and address/ID.

Is this sufficient scope? If a job maps to HW resources and is used with transport processing, does it need process isolation that comes with pairing it to a PD? Or is it enough that the job is paired with the QP and, optionally, MR? Is the creation of a job object always a privileged operation, or is only the act of assigning the job ID privileged?

Thinking through implementation options, since job is specified per transfer, it seems easier to validate the 'jkey' written with a WR shares the same PD of the QP, similar to how an lkey check might work. One alternative is a jkey list off the QP. Other options could offload libfabric address vectors. It's possible vendors may implement this differently, with differences showing up in which MRs are reachable.

- Sean