Re: [PATCH v2 2/5] docs: media: add documentation for media client usage stats

[Detlev Casanova <detlev.casanova@collabora.com>]

Hi !

On 6/19/26 10:04, Nicolas Dufresne wrote:
> Hi,
>
> Le vendredi 19 juin 2026 à 14:58 +0200, Hans Verkuil a écrit :
>> Hi Detlev,
>>
>> Interesting, I had never heard of fdinfo, so if nothing else, I learned something new!
>>
>> On 17/06/2026 20:10, Detlev Casanova wrote:
>>> From: Christopher Healy <healych@amazon.com>
>>>
>>> Document the media fdinfo interface for per-file-descriptor usage
>>> statistics exposed by stateless V4L2 codec drivers via
>>> /proc/<pid>/fdinfo/<fd>.
>>>
>>> This interface is designed for stateless (request API based) codec
>>> devices where the kernel driver has per-job visibility into hardware
>>> execution. Stateful codecs cannot support all of this because their
>>> firmware manages job scheduling opaquely.
>>>
>>> The specification defines media- prefixed keys for engine utilization
>>> time, and operating frequency, following the same conventions as the DRM
>>> fdinfo mechanism documented in drm-usage-stats.rst.
>>>
>>> More fields can be added later.
>>>
>>> Signed-off-by: Christopher Healy <healych@amazon.com>
>>> Signed-off-by: Detlev Casanova <detlev.casanova@collabora.com>
>>> ---
>>>   .../userspace-api/media/drivers/index.rst          |  1 +
>>>   .../media/drivers/media-usage-stats.rst            | 85 ++++++++++++++++++++++
>>>   2 files changed, 86 insertions(+)
>>>
>>> diff --git a/Documentation/userspace-api/media/drivers/index.rst b/Documentation/userspace-api/media/drivers/index.rst
>>> index 02967c9b18d6..61879738836c 100644
>>> --- a/Documentation/userspace-api/media/drivers/index.rst
>>> +++ b/Documentation/userspace-api/media/drivers/index.rst
>>> @@ -34,6 +34,7 @@ For more details see the file COPYING in the source distribution of Linux.
>>>   	imx-uapi
>>>   	mali-c55
>>>   	max2175
>>> +	media-usage-stats
>>>   	npcm-video
>>>   	omap3isp-uapi
>>>   	thp7312
>>> diff --git a/Documentation/userspace-api/media/drivers/media-usage-stats.rst b/Documentation/userspace-api/media/drivers/media-usage-stats.rst
>>> new file mode 100644
>>> index 000000000000..d3dc07002f62
>>> --- /dev/null
>>> +++ b/Documentation/userspace-api/media/drivers/media-usage-stats.rst
>>> @@ -0,0 +1,85 @@
>>> +.. SPDX-License-Identifier: GPL-2.0
>>> +
>>> +.. _media-usage-stats:
>>> +
>>> +==========================
>>> +Media client usage stats
>> stats -> statistics
>>
>> But are these really statistics? Isn't it just the current status?
The term "statistics" was the first that came to my mind, but I agree 
it's not great here.
Thinking of it, the term "metrics" feels more adapted, what do you think ?

>> In many ways this feature looks to me similar to what VIDIOC_LOG_STATUS does,
>> except in a nicer format. When VIDIOC_LOG_STATUS was first added, fdinfo
>> didn't exist yet.
>>
>> And I wouldn't call it 'Media client': it's specific to stateless V4L2
>> codec drivers.
> While it is currently implemented for sateless m2m codec drivers as example
> (because we can't implement this everywhere at once really, its not practical),
> there is no reason why we cannot track to a process fdinfo memory usage
> associated with other video devices. In fact, I would pretty much want to see
> capture devices showing up in v4l2top, as these are using a lot of memory too,
> specially the new camera pipelines. And this is also perfectly suitable for
> stateful codec, converters, everything.
>
> So please, let's agree this is not "stateless V4L2 codec drivers" specific.
Indeed, but I get the confusion as the documentation suggests that 
stateless codec drivers fields are mandatory.
I'll rework it to have generic v4l2 fields that are mandatory for all 
driver types, and then fields per type (where it will focus on stateless 
codecs)
>
> About VIDIOC_LOG_STATUS, this is effectively broken for m2m, only the owning
> session process could call that. Its also pretty bad for tools to have to
> monitor the dmesg and filter it up. fdinfo on the other end, does not require
> opening the device and permissions are very clear, and bound to user process.
>
>>> +==========================
>>> +
>>> +Stateless V4L2 codec drivers can optionally expose per-file-descriptor usage
>>> +statistics via ``/proc/<pid>/fdinfo/<fd>``. This is analogous to the DRM fdinfo
>>> +mechanism documented in :ref:`drm-client-usage-stats`, but uses the ``media-``
>>> +key prefix for V4L2 media devices.
>>> +
>>> +This interface is specific to stateless (request API based) codec devices,
>>> +including both decoders and encoders. With stateless codecs, the kernel driver
>>> +explicitly submits each frame to the hardware and receives a completion
>>> +interrupt, providing a clean per-job boundary that can be attributed to the
>>> +submitting file descriptor.
>>> +
>>> +Stateful codec devices cannot support this interface because their firmware
>>> +manages job scheduling internally. The kernel driver submits bitstream data
>>> +but has no visibility into per-frame hardware execution timing.
>>> +
>>> +Implementation
>>> +==============
>>> +
>>> +The V4L2 core provides the plumbing: drivers implement the ``show_fdinfo``
>>> +callback in ``struct v4l2_file_operations``, and the core wires it into the
>>> +kernel ``struct file_operations`` so that ``/proc/<pid>/fdinfo/<fd>`` output
>>> +includes the driver-provided keys.
>>> +
>>> +File format specification
>>> +=========================
>>> +
>>> +- File shall contain one key value pair per one line of text.
>>> +- Colon character (``:``) must be used to delimit keys and values.
>>> +- All standardised keys shall be prefixed with ``media-``.
>>> +- Driver-specific keys shall be prefixed with ``driver_name-``.
>>> +
>>> +Mandatory keys
>>> +--------------
>>> +
>>> +- media-driver: <valstr>
>> I'd pick 'v4l2-driver'. Since 'media-driver' is too generic for this
>> since that encompasses also DVB/CEC/RC drivers.
v4l2-driver works for me.
>>
>>> +
>>> +  String shall contain the name of the media driver.
>> 'V4L2 stateless codec driver'
> With that said, we can refine, but not in that direction.
Then just 'V4L2 driver'
>>> +
>>> +- media-type: <valstr>
>> Poor name.
This fields will basically be the one telling userspace what fields to 
expect next.
"v4l2-type" sounds wrong as well. What about "v4l2-driver-type" ?
>>
>>> +
>>> +  String shall identify the type of media engine exposed through this file
>>> +  descriptor. Standard values are ``decoder`` and ``encoder``.
>> I think I would use 'stateless-decoder' and 'stateless-encoder'. It's more
>> specific than de/encoder since that can be stateful as well.
Yes, I totally agree.
>>
>> So I am missing the big picture here: right now this patch adds support for
>> this for stateless codecs, but what happens in the future if this is also
>> added for regular video capture devices, ISPs, etc.?
> We should work further on the type system, then we'd simply have to extend it.
> Different type of driver will provide different key/value pair. We should well
> document which key are mandatory for the type (or for all type, like giving a
> name for general identification purpose).
>
> Of course, the current proposal is very torward processing cores, hence the one
> to one match with what GPUs present in fdinfo today in mainline, such are
> frequency, core utilization time, etc.
>
> Though, DMA devices such as capture may expose more of a bandwidth kind of
> information. I really don't want to make up too may cases, so owners of these
> don't feel like this is mandated though. As we develop software using our
> drivers, we should be able to figure-out what kind of things are important to
> monitor. For CODECs, we have two majors things. a) core utilization (which can
> be with a timer like this one, but ideally using cycle counters, or firmware
> provided data). b) memory usage.
For core utilization, we can have both (HW cycles and time).
Time is less precise as it is computed by the driver, starting a bit 
before the hardware is run and a bit after the interrupt arrives.
HW cycles is very precise, but need the clock rate to be exposed and not 
all HW may support it.

We can say that time is mandatory (for stateless codec), as it can 
always be computed and HW cycles is optional.
For showing usage like in v4l2top, the userspace can compute it from HW 
cycles if available and fallback to less precise "Time" otherwise.

>
>> The naming here matters, it has to have some sort of scheme so it can be
>> extended to other types of drivers. So a 'media' prefix is too generic,
>> and it also looks like it refers to the /dev/mediaX device.
>>
>>> +
>>> +Utilization keys
>>> +----------------
>>> +
>>> +- media-engine-usage: <uint> ns
>> 'Media Engine': very vague.
Thinking of it, this one could be changed to 
"v4l2-core-usage-time-<core_id>".
That way we can have "v4l2-core-usage-cycles-<core_id>" too.

Separating by core id show more precise metrics.
>>
>>> +
>>> +  Time in nanoseconds that the hardware engine spent busy processing work
>> Cumulative since creating the file handle? Or since the last read?
I implemented cumulative since the file handle was created.
This is closer to what DRM does, so let's keep it. I'll specify it in 
the documentation.

>>
>>> +  belonging to this file descriptor. The engine being measured is identified
>>> +  by the ``media-type`` key.
>>> +
>>> +  Values are not required to be constantly monotonic if it makes the driver
>>> +  implementation easier, but are required to catch up with the previously
>>> +  reported larger value within a reasonable period.
>> Does this make sense for codecs?
>>
>> I can tell that this is heavily influenced by the drm documentation, but that
>> does not necessarily translate to V4L2.
> I would guess this is bound to usage of cycle counter, which we definitely have
> in some codec HW exposing. As the frequency fluctuate, I suppose there is ways
> you can get your calculation a bit off, as this is being sampled. Basically, the
> frequence change is not strictly tracked by our drivers, which is ok. But I also
> don't seen why we wouldn't keep the counters monotonic, its really easy to do.
Yeah, I think we can drop that comment. For each run, we just add a 
positive value,
so there is no reason not to be monotonic.
>>> +
>>> +Frequency keys
>>> +--------------
>>> +
>>> +- media-maxfreq: <uint> Hz
>>> +
>>> +  Maximum operating frequency of the main engine clock.
>>> +
>>> +- media-curfreq: <uint> Hz
>>> +
>>> +  Current operating frequency of the main engine clock.
>> 'Main engine clock'?
> The word "engine" seems indeed alien in this subsystem. It is referring to a
> system where you open one devices, so you have one fdinfo, but multiple cores.
> Each cores may be running at it owns frequency. But I think this is a little too
> vague to my taste.
Main doesn't refer to the main engine, but rather to the main clock of 
the engine.
e.g. rkvdec so far can have up to 5 clocks linked to it but only the one 
that actually clocks the HW should be exposed.

That being said, you are right that one file handle can have multiple 
cores, so the name should be changed to v4l2-maxfreq-<core_id> (and 
curfreq) and the text should use "main clock of the core <core_id>".

We also could just expose all clocks with their name, but userspace 
still needs to know which one can be used to compute usage %age from HW 
cycles.

Detlev.
>
> Hope this feedback does not cross to many wires, but tagging this as specific to
> stateless codec drivers did spark a little in my head. And we are really in need
> for a way to monitor our drivers.
>
> Nicolas
>
>>> +
>>> +Example output
>>> +==============
>>> +
>>> +::
>>> +
>>> +  media-driver:           hantro-vpu
>>> +  media-type:             decoder
>>> +  media-engine-usage:     123456789 ns
>>> +  media-maxfreq:          600000000 Hz
>>> +  media-curfreq:          600000000 Hz
>>>
>> Regards,
>>
>> 	Hans