RE: [RFC PATCH 00/13] Ultra Ethernet driver introduction

Sean Hefty
2025-04-16T23:58:48Z

> > There's discussion on defining this relationship:
> >
> > Job <- 0..n --- 1 -> PD
> >
> > I can't think of a technical reason why that's needed.
>
> From my UE perspective, I agree. UE needs to share job IDs across processes
> while still having inter-process isolation for things like local memory
> registrations.

We seem stuck on this. Here's a specific proposal that I'm considering:

1. Define a device level 'security key'. The skey encapsulates encryption attributes.
The skey may be shared between processes.
2. Define a device level 'job', or maybe more generic 'communication domain'*.
A job object is associated with a transport protocol and these optional attributes:
address, job id (required for UET), and security key.
The job object may be shared between processes.
3. Define a PD level 'job key'. The job key references a single job object.
Multiple job keys may be created under a single PD, if each references a separate job.
4. Support creating MRs that reference job keys.

We can share job IDs across processes with process-level isolation of MRs. The security model can be viewed as meeting these checks:

Endpoint ID (QPN) -> endpoint (QP) -> PD
job ID -> job key -> PD
rkey -> MR -> PD or rkey -> MR -> job key -> PD
lkey -> MR -> PD or lkey -> MR -> job key -> PD (?)

(Other fields carried in the headers are needed to make these mappings, but the concept is the same). Access is allowed if the PDs and job keys (if applicable) match. The endpoint can only send to jobs associated with the same PD. E.g. a jkey is specified in the WR. The endpoint can be configured to receive from any job or only those jobs associated with the same PD. E.g. On receives, enforce the second check or not. I am unsure of the lkey -> job key check.

If a NIC or endpoint only supports a single job, the job key is conceptually identical to the PD. (An endpoint can only receive from the assigned job).

* The job may also be used to store and peer addresses between processes. That is, it acts like a libfabric address vector restricted to a single authorization key or no key. (Conversely, a libfabric AV maps to multiple job objects, separated by auth_key). To reflect a more generic use, I would consider calling it a 'comm domain', rather than a job.

- Sean