RE: Further thoughts on uAPI

Liran Liss
2016-04-24T14:15:12Z

> From: ofvwg [mailto:ofvwg-bounces@lists.openfabrics.org] On Behalf Of Hefty,
> Sean

>
> > The kernel common code side is pretty straightforward, just a bunch of
> > tables of function pointers, templates and idrs for each object_type.
>
> This is the part where the intent and implementation are not clear to me. I see
> value in generic user space handle validation, but I prefer they map to driver
> specific resources. For example, I don't want a user space allocation of a QP to
> result in this:
>
> 1403 struct ib_qp {
> 1404 struct ib_device *device;
> 1405 struct ib_pd *pd;
> 1406 struct ib_cq *send_cq;
> 1407 struct ib_cq *recv_cq;
> 1408 struct ib_srq *srq;
> 1409 struct ib_xrcd *xrcd; /* XRC TGT QPs only */
> 1410 struct list_head xrcd_list;
> 1411 /* count times opened, mcast attaches, flow attaches */
> 1412 atomic_t usecnt;
> 1413 struct list_head open_list;
> 1414 struct ib_qp *real_qp;
> 1415 struct ib_uobject *uobject;
> 1416 void (*event_handler)(struct ib_event *, void *);
> 1417 void *qp_context;
> 1418 u32 qp_num;
> 1419 enum ib_qp_type qp_type;
> 1420 };
>
> monstrosity being allocated in the kernel. The size and layout of this sort of
> structure impacts scalability and performance. This is where I want to make
> sure there's alignment on the overall architecture.
>

How the kernel decides to represent objects is a kernel implementation issue, not an ABI issue.
As long as the ABI is expressive enough (e.g., pass only the relevant information for the typed object at hand), the kernel code could decide what to hold where.

> More specifically, I would switch the existing uABI commands to ioctl's, leaving
> the code paths mostly unchanged, but marking them as legacy. Then add the
> new ioctls that you suggested, with future drivers hooking directly into the ioctl
> framework. Honestly, I'm not sure the rdma core should define anything
> beyond the common ioctl structure. Anything else can just be placed into the
> driver specific portion, or maybe rdma core sub-subsystem (e.g. ib, iwarp, roce,
> opa, mlx, ...) Maybe we would define some common ioctl's across all devices,
> but I don't think they would look anything like the current commands. They
> would need to be much higher level.

That's where the provider channel comes in.
For provider stuff, uverbs won't do anything else other than a way to get a channel + some conventions (maybe).
There won't be any matching kAPI for it.

For generic interfaces (currently includes Verbs, Ethernet QPs, and IB management), the new scheme should map what we have today in a flexible manner.
This would enable us, for example, to pass only RoCE addressing attributes while modifying a RoCE QP (and optionally optimizing the kernel representation as well).
These interfaces have a matching kAPI.

New object types could be supported with additional/alternative attributes to existing objects, where it makes sense.
The flexibility of the ABI would allow for both optimal parameter passing and kernel representation.
For example, if I want to add a counter that counts completions, I would add it as a new type of CQ. This would allow it to interoperate with the existing object model.
The same would apply for new transport types.

If a completely new generic concept is introduced, we can add additional interfaces and extend the kAPI to support it.

>
> - Sean