Re: [PATCH RFC 04/20] Documentation: ctlv: Add userspace ABI

[Jiri Pirko <jiri@resnulli.us>]

Tue, Oct 20, 2026 at 01:29:45AM +0200, jgg@nvidia.com wrote:
>> Describe the CTLV transport, discovery, wire format and userspace
>> compatibility rules.
>> 
>> Assisted-by: Cursor: GPT-5.6 Sol
>> Assisted-by: Cursor: Claude Opus 5.5
>> Signed-off-by: Jiri Pirko <jiri@nvidia.com>
>> ---
>>  Documentation/userspace-api/ctlv.rst  | 507 ++++++++++++++++++++++++++
>>  Documentation/userspace-api/index.rst |   1 +
>>  MAINTAINERS                           |   1 +
>>  3 files changed, 509 insertions(+)
>>  create mode 100644 Documentation/userspace-api/ctlv.rst
>>
>> diff --git a/Documentation/userspace-api/ctlv.rst b/Documentation/userspace-api/ctlv.rst
>> new file mode 100644
>> index 00000000000000..5ebca3d4d94af8
>> --- /dev/null
>> +++ b/Documentation/userspace-api/ctlv.rst
>> @@ -0,0 +1,507 @@
>> +.. SPDX-License-Identifier: GPL-2.0
>> +
>> +==================
>> +CTLV userspace ABI
>> +==================
>> +
>> +:Author: Jiri Pirko
>> +
>
>This could use an introduction:
>
>Char-Device Tag Length Value (CTLV) is a flexible way to describe system calls
>delivered over ioctl for a kernel subsystem. It provides a 64 bit operation
>number with variable length input/output arguments described in a YAML schema.
>The arguments are serialized using a TLV scheme. Strong support for
>forwards/backward compatability is baked into the schema design.
>
>CTLV excels for uAPIs that are too unconstrained to fit in simple structs.
>Things that have arrays of arrays, variable size structs, variable sized data,
>strings and similar which are challenging to represent.
>
>As a transport CTLV uses a global number space. Inspection tools like
>strace/seccomp can always decode a CTLV ioctl without having to know what
>subsystem is using it. The global number space is organized into families, which
>roughly corresponds to a subsystem.
>
>A family describes what its devices accept in YAML, a generator turns that into
>kernel metadata and UAPI headers, and CTLV validates every message before a
>family callback sees it. Userspace sees one character device per instance, one
>ioctl, and ``read()`` for events.

okay, looks great.


>
>> +CTLV is a transport for device control interfaces. A family describes what its
>> +devices accept in YAML, a generator turns that into kernel metadata and UAPI
>> +headers, and CTLV validates every message before a family callback sees it.
>> +Userspace sees one character device per instance, one ioctl, and ``read()`` for
>> +events.
>> +
>> +This document is the ABI. What each family accepts is specified under
>> +``Documentation/ctlv/specs``; writing a family is :doc:`/driver-api/ctlv`.
>> +
>> +Why a TLV transport
>> +===================
>> +
>> +CTLV puts the rules of every operation in one description and validates them
>> +once:
>> +
>> +- One ioctl carries every operation of every family, so a new operation is a
>> +  new specification entry, not a new ioctl number.
>> +- Attributes are optional or required by declaration, so a message can grow
>> +  without breaking a caller that does not know the new attribute.
>> +- An unknown attribute is refused on a request and tolerated on a reply, so a
>> +  new kernel can say more to an old program. A request may mark an attribute an
>> +  old kernel is welcome to skip, so a new program can say more too.
>> +- Bounds, lengths, occurrence counts and nesting depth are checked before a
>> +  family callback runs.
>> +
>> +Devices
>> +=======
>> +
>> +Each registered instance appears as a character device in the ``ctlv`` class
>> +with a dynamic minor::
>> +
>> +    /dev/ctlv0
>> +    /sys/class/ctlv/ctlv0
>> +
>> +Sysfs shows class membership, the ``dev`` number and the device-model link to
>> +the parent device. There is no family sysfs attribute; ``families-dump`` on an
>> +open descriptor is the authoritative answer to what a device is.
>
>I've tried to like this but, I really am not a fan. I think it
>reflects too much of a netlink influence.

Well, yes and no. In contrast with Generic Netlink, CTLV builds on
inheritance. So *every* CTLV family is child of "root" CTLV family. So
if user opens any /dev/ctlvX, the "root" CTLV family ops work. Then the
families-dump op reveals the inheritance chain from root to the leaf.
The idea is to allow generic tool to work on upper family level and
allowing some driver-specific tool to work on the leaf.
Think of it as a part of introspection of a device.


>
>My two biggest problems:
>
>1) This prevents us from incrementally using CTLV as a more robust
>   uAPI. Like if I wanted to start using it in vfio for new uAPI I
>   cannot

I don't think that the existance of family-dump prevents you that. I
have idea of a split that would allow you to attach CTVL ioctl on
existing fd. Will try to sketch it and implement it.


>
>2) It destroys the kernel concept of a cdev subsystem, where we have a
>   distinct class, /dev/ directory and cdev nodes.
>
>   Lots of tooling and userspace is geared toward working this way. A
>   completely unstable /dev/ctlvX that could be *anything* will need a
>   meaningful new methodology in userspace. For instance anyone doing
>   containers that wants to pass these is going to need special
>   tooling.

The node is not opaque before open, though. It is a device-model child
of the device that registered it, so /sys/class/ctlv/ctlv0/device
points at that parent, and its uevents carry the family and its
ancestry (CTLV_FAMILY, CTLV_COMPATIBLE_*).

udev can name it from the same properties. ctlv0 is no more stable
than sda or event3, and it gets a stable name the same way: a rule
matches the family or the parent and adds a symlink, as /dev/disk/by-id
does for disks, e.g.:

    SUBSYSTEM=="ctlv", ENV{CTLV_FAMILY}=="ctlv-attestation-tdx", \
        SYMLINK+="attestation/tdx"

A container is then given /dev/attestation/tdx rather than whichever
ctlvN it happened to get; no special tooling is involved. The Discovery
section later in the doc has the details.


>
>> [ ... 143 lines skipped ... ]
>> +An attribute ID names the family that declared it, a group within that family
>> +and an attribute within that group::
>> +
>> +    63           48 47           32 31                             0
>> +    +--------------+--------------+--------------------------------+
>> +    |    family    |    group     |             attr               |
>> +    +--------------+--------------+--------------------------------+
>> +
>> +    CTLV_ATTR(family, group, attr)
>> +    CTLV_ATTR_FAMILY(attr) / CTLV_ATTR_GROUP(attr) / CTLV_ATTR_NUMBER(attr)
>> +
>> +An operation ID is the same shape with the group bits zero::
>> +
>> +    CTLV_OP(family, op)
>> +    CTLV_OP_FAMILY(op) / CTLV_OP_NUMBER(op)
>> +
>> +Use generated complete IDs. A family's header defines every ID it declares
>> +and includes its parent's.
>
>I'm OK with this, it seems like a fine design, very netlink inspired.

Netlink actually does not have it. Attr id is not global as it is here.


>
>One of the nice properties is an external tool, like strace or a
>seccomp does have enough information to capture the entire operation,
>including all parameters.
>
>Though I wonder how much trouble it will be for userspace to form
>these things.. It has to pre-determine how much memory it needs then
>fill in this structure. That's not so easy to use.
>
>How hard would it be to accept a linked list, or array, of buffer
>frags? With some strict rule that a leaf attribute cannot span frags?
>
>My thinking is userspace could allocate frag by frag as it builds the
>message instead of being forced to do a size, allocate, fill sequence?

There are no frags on the request side (reply has them).
The entire message is a single buf. For request, all arrays are bounded
and this is enforced for every family attribute. The introspection
provides kernel-computed max size of buffer for ever op request.
Then during processing, kernel copies the whole request buffer before it
parses it.

I deliberatelly made this limitation to keeps things simple. Makes
sense?


>
>> +
>> +Request occurrence order
>> +------------------------
>> +
>> +Occurrences of one attribute ID must be contiguous within each request root or
>> +nest. ``A A B`` is valid, while ``A B A`` is not. An ignored unknown attribute
>> +is not an occurrence of any known attribute, but it ends the preceding run, so
>> +``A ignored-unknown A`` is also invalid. Different IDs otherwise have no
>> +required numeric or schema order.
>
>I'm not entirely sure what this is trying to say. Is it just a single
>ID must be contiguous if it is repeated?

Yes it is a limitation of ordering of attributes that occur multiple
times in message/nest (arrays). Will reword.


>
>> +
>> +Attributes a kernel may skip
>> +----------------------------
>> +
>> +A request attribute, the root aside, may carry
>> +``CTLV_ATTR_F_IGNORE_IF_UNKNOWN``. If no family the device answers declares
>> +that complete ID, the kernel steps over the whole attribute without looking
>> +inside it; without the bit, an attribute the schema does not place where it
>> +appears fails the request as it always has.
>> +
>> +The bit forgives an ID this kernel has never heard of and nothing else. An ID
>> +the device's schema declares somewhere is held to that schema, so an attribute
>> +sent where it does not belong fails whether the bit is set or not, and one sent
>> +where it does belong is checked as usual. Every other rule applies too: the
>> +length has to fit its nest and the header may carry no bit the format does not
>> +define. Replies and events never carry it.
>
>This is pretty hard to read. AI?

Okay, will reword.


>
>> +
>> +This is how a program built against a newer version of a family talks to an
>> +older kernel. What that kernel knows is validated, what it does not is
>> +ignored, and a caller that has to know which is which asks ``op-schema-get``
>> +first.
>> +
>> +Executing an operation
>> +======================
>> +
>> +One ioctl executes every operation::
>> +
>> +    struct ctlv_op_exec {
>> +            __aligned_u64 op_id;
>> +            __aligned_u64 request;      /* struct ctlv_attr __user * */
>> +            __aligned_u64 reply;        /* struct ctlv_attr __user * */
>> +            __u32 request_len;
>> +            __u32 reply_len;
>> +            __aligned_u64 generation;   /* queries only */
>> +            __aligned_u64 cursor;       /* queries only */
>> +            __aligned_u64 reserved[2];  /* must be zero */
>> +    };
>> +
>> +    #define CTLV_IOC_OP_EXEC _IOWR(0xBB, 0x00, struct ctlv_op_exec)
>
>Generally I've ben frowning on using the size arugment for ioctl. This
>means you can never make that struct bigger, and that's historically
>been a bad choice...
>
>I've mostly been using this:
>
>  #define IOMMU_DESTROY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_DESTROY)
>
>And then put a length at the top of the struct and do the
>
>        ret = copy_struct_from_user(ucmd.cmd, op->size, ucmd.ubuffer,
>                                    ucmd.user_size);
>
>Kind of thing so there is a solid compatability story.

Okay, makes sense.


>
>> [ ... 81 lines skipped ... ]
>> +``ENOMEM``
>> +    The kernel could not allocate what the operation needed.
>> +
>
>EOVERFLOW for check_overflow failures

Okay.


Thanks!

>
>-- 
>Jason