[RFC] V4L2 & Metadata: switch to /dev/v4l-metaX instead of /dev/videoX

[RFC] V4L2 & Metadata: switch to /dev/v4l-metaX instead of /dev/videoX

[Hans Verkuil]

Hi all,

I am increasingly unhappy about the choice of /dev/videoX for metadata devices.

It is confusing for end-users (especially w.r.t. the common uvc driver) and
if we want to change this, then we need to do it soon.

This patch https://patchwork.linuxtv.org/patch/58693/ adds a new VFL_TYPE_METADATA
so at least drivers can now explicitly signal that they want to register a
metadata device.

This also makes it possible to add a kernel config option that allows you
to select whether you want metadata devices to appear as videoX or v4l-metaX.
I would prefer to set it to v4l-metaX by default.

We can also consider backporting this to the stable/long-term kernels.

Metadata capture was introduced in 4.12 for the vsp1 driver, in 4.16 for the
uvc driver and in 5.0 for the staging ipu3 driver.

Does someone remember the reason why we picked /dev/videoX for this in the
first place?

Regards,

	Hans

Re: [RFC] V4L2 & Metadata: switch to /dev/v4l-metaX instead of /dev/videoX

[Kieran Bingham]

Hi Hans,

On 12/09/2019 08:48, Hans Verkuil wrote:
> Hi all,
> 
> I am increasingly unhappy about the choice of /dev/videoX for metadata devices.
> 
> It is confusing for end-users (especially w.r.t. the common uvc driver) and
> if we want to change this, then we need to do it soon.
> 
> This patch https://patchwork.linuxtv.org/patch/58693/ adds a new VFL_TYPE_METADATA
> so at least drivers can now explicitly signal that they want to register a
> metadata device.
> 
> This also makes it possible to add a kernel config option that allows you
> to select whether you want metadata devices to appear as videoX or v4l-metaX.
> I would prefer to set it to v4l-metaX by default.

I think I prefer this separation (v4l-metaX).

Having metadata as a (separate) videoX seemed odd to me - but I only
saw/was affected by the metadata topics when it was too late it seemed ...


> We can also consider backporting this to the stable/long-term kernels.
> 
> Metadata capture was introduced in 4.12 for the vsp1 driver, in 4.16 for the
> uvc driver and in 5.0 for the staging ipu3 driver.
> 
> Does someone remember the reason why we picked /dev/videoX for this in the
> first place?

I've wondered why it's not a separate queue on the same video device -
much like we have multiple queues for V4L2-M2M devices ....

The data is relative to the same frames coming from the main queue right ?

That might have been awkward to express through our device type flags
though.

Anyway, I thought the horse had bolted on this topic ?

 :-D


> Regards,
> 
> 	Hans
> 


-- 
Regards
--
Kieran

Re: [RFC] V4L2 & Metadata: switch to /dev/v4l-metaX instead of /dev/videoX

[Hans Verkuil]

On 9/12/19 3:16 PM, Kieran Bingham wrote:
> Hi Hans,
> 
> On 12/09/2019 08:48, Hans Verkuil wrote:
>> Hi all,
>>
>> I am increasingly unhappy about the choice of /dev/videoX for metadata devices.
>>
>> It is confusing for end-users (especially w.r.t. the common uvc driver) and
>> if we want to change this, then we need to do it soon.
>>
>> This patch https://patchwork.linuxtv.org/patch/58693/ adds a new VFL_TYPE_METADATA
>> so at least drivers can now explicitly signal that they want to register a
>> metadata device.
>>
>> This also makes it possible to add a kernel config option that allows you
>> to select whether you want metadata devices to appear as videoX or v4l-metaX.
>> I would prefer to set it to v4l-metaX by default.
> 
> I think I prefer this separation (v4l-metaX).
> 
> Having metadata as a (separate) videoX seemed odd to me - but I only
> saw/was affected by the metadata topics when it was too late it seemed ...
> 
> 
>> We can also consider backporting this to the stable/long-term kernels.
>>
>> Metadata capture was introduced in 4.12 for the vsp1 driver, in 4.16 for the
>> uvc driver and in 5.0 for the staging ipu3 driver.
>>
>> Does someone remember the reason why we picked /dev/videoX for this in the
>> first place?
> 
> I've wondered why it's not a separate queue on the same video device -
> much like we have multiple queues for V4L2-M2M devices ....
> 
> The data is relative to the same frames coming from the main queue right ?
> 
> That might have been awkward to express through our device type flags
> though.

There can be only one receive queue and one transmit queue per device node.

Multiple receive queues (i.e. with different types) were allowed in the past,
but I don't think any driver ever implemented it as it was just too complicated.
Both in the driver and for the user.

> 
> Anyway, I thought the horse had bolted on this topic ?

Well, that's the question. For meta output it can certainly be changed since
only a staging driver uses it. For capture it is mainly the uvc driver that's
problematic, but I am not aware of any linux application in a distro that
uses metadata capture.

So with a new kernel option you can choose whether you want to keep the old
behavior or create v4l-meta device nodes instead.

I regularly get questions about what happened, and by no means all applications
properly check the capabilities of the video device nodes (yes, they should do
so, but in the past this never was an issue with your average desktop user).

Besides all that, it is just weird to have meta data arrive on a video node.
I expect video there.

Regards,

	Hans

> 
>  :-D
> 
> 
>> Regards,
>>
>> 	Hans
>>
> 
> 

Re: [RFC] V4L2 & Metadata: switch to /dev/v4l-metaX instead of /dev/videoX

[Mauro Carvalho Chehab]

Em Thu, 12 Sep 2019 14:16:11 +0100
Kieran Bingham <kieran.bingham@ideasonboard.com> escreveu:

> Hi Hans,
> 
> On 12/09/2019 08:48, Hans Verkuil wrote:
> > Hi all,
> > 
> > I am increasingly unhappy about the choice of /dev/videoX for metadata devices.
> > 
> > It is confusing for end-users (especially w.r.t. the common uvc driver) and
> > if we want to change this, then we need to do it soon.

Kernel has (about) nothing to do with how the userspace devnodes are
named, as the actual name is given by udev.

Anyway, from Kernel standpoint, it sounds too late to change the name
of the devices from "videoX" to something else, as a change like that
may break existing apps.

It could make sense to have something like that at udev rules.

Btw, at least at the apps I'm maintaining on userspace, I'm not using
/dev/foo to detect devices anymore. Instead, I'm relying on udev
in order to enum devices, checking if the devnode support video stream
capabilities before exposing them for userspace to select.

> > 
> > This patch https://patchwork.linuxtv.org/patch/58693/ adds a new VFL_TYPE_METADATA
> > so at least drivers can now explicitly signal that they want to register a
> > metadata device.
> > 
> > This also makes it possible to add a kernel config option that allows you
> > to select whether you want metadata devices to appear as videoX or v4l-metaX.
> > I would prefer to set it to v4l-metaX by default.  
> 
> I think I prefer this separation (v4l-metaX).
> 
> Having metadata as a (separate) videoX seemed odd to me - but I only
> saw/was affected by the metadata topics when it was too late it seemed ...
> 
> 
> > We can also consider backporting this to the stable/long-term kernels.
> > 
> > Metadata capture was introduced in 4.12 for the vsp1 driver, in 4.16 for the
> > uvc driver and in 5.0 for the staging ipu3 driver.
> > 
> > Does someone remember the reason why we picked /dev/videoX for this in the
> > first place?  
> 
> I've wondered why it's not a separate queue on the same video device -
> much like we have multiple queues for V4L2-M2M devices ....
> 
> The data is relative to the same frames coming from the main queue right ?
> 
> That might have been awkward to express through our device type flags
> though.
> 
> Anyway, I thought the horse had bolted on this topic ?
> 
>  :-D
> 
> 
> > Regards,
> > 
> > 	Hans
> >   
> 
> 



Thanks,
Mauro

Re: [RFC] V4L2 & Metadata: switch to /dev/v4l-metaX instead of /dev/videoX

[Hans Verkuil]

On 9/12/19 4:21 PM, Mauro Carvalho Chehab wrote:
> Em Thu, 12 Sep 2019 14:16:11 +0100
> Kieran Bingham <kieran.bingham@ideasonboard.com> escreveu:
> 
>> Hi Hans,
>>
>> On 12/09/2019 08:48, Hans Verkuil wrote:
>>> Hi all,
>>>
>>> I am increasingly unhappy about the choice of /dev/videoX for metadata devices.
>>>
>>> It is confusing for end-users (especially w.r.t. the common uvc driver) and
>>> if we want to change this, then we need to do it soon.
> 
> Kernel has (about) nothing to do with how the userspace devnodes are
> named, as the actual name is given by udev.

To my knowledge the standard udev rules do not rename anything for media
devices, so in reality it IS the kernel that decides this.

But this is why I suggested to put it under a kernel config option.

> 
> Anyway, from Kernel standpoint, it sounds too late to change the name
> of the devices from "videoX" to something else, as a change like that
> may break existing apps.
> 
> It could make sense to have something like that at udev rules.

But wouldn't that break existing apps as well?

A bigger problem is that it isn't easy to detect the difference between
a regular video device and a metadata device: you'd have to call QUERYCAP
to determine that.

Now that device_caps is always available, perhaps we should exports that
as an attribute? Regardless of this specific issue, it would make it a lot
easier to write udev rules.

Regards,

	Hans

> 
> Btw, at least at the apps I'm maintaining on userspace, I'm not using
> /dev/foo to detect devices anymore. Instead, I'm relying on udev
> in order to enum devices, checking if the devnode support video stream
> capabilities before exposing them for userspace to select.
> 
>>>
>>> This patch https://patchwork.linuxtv.org/patch/58693/ adds a new VFL_TYPE_METADATA
>>> so at least drivers can now explicitly signal that they want to register a
>>> metadata device.
>>>
>>> This also makes it possible to add a kernel config option that allows you
>>> to select whether you want metadata devices to appear as videoX or v4l-metaX.
>>> I would prefer to set it to v4l-metaX by default.  
>>
>> I think I prefer this separation (v4l-metaX).
>>
>> Having metadata as a (separate) videoX seemed odd to me - but I only
>> saw/was affected by the metadata topics when it was too late it seemed ...
>>
>>
>>> We can also consider backporting this to the stable/long-term kernels.
>>>
>>> Metadata capture was introduced in 4.12 for the vsp1 driver, in 4.16 for the
>>> uvc driver and in 5.0 for the staging ipu3 driver.
>>>
>>> Does someone remember the reason why we picked /dev/videoX for this in the
>>> first place?  
>>
>> I've wondered why it's not a separate queue on the same video device -
>> much like we have multiple queues for V4L2-M2M devices ....
>>
>> The data is relative to the same frames coming from the main queue right ?
>>
>> That might have been awkward to express through our device type flags
>> though.
>>
>> Anyway, I thought the horse had bolted on this topic ?
>>
>>  :-D
>>
>>
>>> Regards,
>>>
>>> 	Hans
>>>   
>>
>>
> 
> 
> 
> Thanks,
> Mauro
> 

Re: [RFC] V4L2 & Metadata: switch to /dev/v4l-metaX instead of /dev/videoX

[Philipp Zabel]

On Thu, 2019-09-12 at 16:49 +0200, Hans Verkuil wrote:
> On 9/12/19 4:21 PM, Mauro Carvalho Chehab wrote:
> > Em Thu, 12 Sep 2019 14:16:11 +0100
> > Kieran Bingham <kieran.bingham@ideasonboard.com> escreveu:
> > 
> > > Hi Hans,
> > > 
> > > On 12/09/2019 08:48, Hans Verkuil wrote:
> > > > Hi all,
> > > > 
> > > > I am increasingly unhappy about the choice of /dev/videoX for metadata devices.
> > > > 
> > > > It is confusing for end-users (especially w.r.t. the common uvc driver) and
> > > > if we want to change this, then we need to do it soon.
> > 
> > Kernel has (about) nothing to do with how the userspace devnodes are
> > named, as the actual name is given by udev.
> 
> To my knowledge the standard udev rules do not rename anything for media
> devices, so in reality it IS the kernel that decides this.
> 
> But this is why I suggested to put it under a kernel config option.
> 
> > 
> > Anyway, from Kernel standpoint, it sounds too late to change the name
> > of the devices from "videoX" to something else, as a change like that
> > may break existing apps.
> > 
> > It could make sense to have something like that at udev rules.
> 
> But wouldn't that break existing apps as well?
> 
> A bigger problem is that it isn't easy to detect the difference between
> a regular video device and a metadata device: you'd have to call QUERYCAP
> to determine that.

v4l_id does call VIDIOC_QUERYCAP [1] to fill the ID_V4L_CAPABILITIES
colon separated list.

[1] https://github.com/systemd/systemd/blob/master/src/udev/v4l_id/v4l_id.c#L66

regards
Philipp

Re: [RFC] V4L2 & Metadata: switch to /dev/v4l-metaX instead of /dev/videoX

[Hans Verkuil]

On 9/12/19 4:57 PM, Philipp Zabel wrote:
> On Thu, 2019-09-12 at 16:49 +0200, Hans Verkuil wrote:
>> On 9/12/19 4:21 PM, Mauro Carvalho Chehab wrote:
>>> Em Thu, 12 Sep 2019 14:16:11 +0100
>>> Kieran Bingham <kieran.bingham@ideasonboard.com> escreveu:
>>>
>>>> Hi Hans,
>>>>
>>>> On 12/09/2019 08:48, Hans Verkuil wrote:
>>>>> Hi all,
>>>>>
>>>>> I am increasingly unhappy about the choice of /dev/videoX for metadata devices.
>>>>>
>>>>> It is confusing for end-users (especially w.r.t. the common uvc driver) and
>>>>> if we want to change this, then we need to do it soon.
>>>
>>> Kernel has (about) nothing to do with how the userspace devnodes are
>>> named, as the actual name is given by udev.
>>
>> To my knowledge the standard udev rules do not rename anything for media
>> devices, so in reality it IS the kernel that decides this.
>>
>> But this is why I suggested to put it under a kernel config option.
>>
>>>
>>> Anyway, from Kernel standpoint, it sounds too late to change the name
>>> of the devices from "videoX" to something else, as a change like that
>>> may break existing apps.
>>>
>>> It could make sense to have something like that at udev rules.
>>
>> But wouldn't that break existing apps as well?
>>
>> A bigger problem is that it isn't easy to detect the difference between
>> a regular video device and a metadata device: you'd have to call QUERYCAP
>> to determine that.
> 
> v4l_id does call VIDIOC_QUERYCAP [1] to fill the ID_V4L_CAPABILITIES
> colon separated list.
> 
> [1] https://github.com/systemd/systemd/blob/master/src/udev/v4l_id/v4l_id.c#L66

Interesting. It's very outdated, though.

Regards,

	Hans

> 
> regards
> Philipp
> 

Re: [RFC] V4L2 & Metadata: switch to /dev/v4l-metaX instead of /dev/videoX

[Laurent Pinchart]

Hi Hans,

On Thu, Sep 12, 2019 at 09:48:11AM +0200, Hans Verkuil wrote:
> Hi all,
> 
> I am increasingly unhappy about the choice of /dev/videoX for metadata devices.
> 
> It is confusing for end-users (especially w.r.t. the common uvc driver) and
> if we want to change this, then we need to do it soon.
> 
> This patch https://patchwork.linuxtv.org/patch/58693/ adds a new VFL_TYPE_METADATA
> so at least drivers can now explicitly signal that they want to register a
> metadata device.
> 
> This also makes it possible to add a kernel config option that allows you
> to select whether you want metadata devices to appear as videoX or v4l-metaX.
> I would prefer to set it to v4l-metaX by default.
> 
> We can also consider backporting this to the stable/long-term kernels.
> 
> Metadata capture was introduced in 4.12 for the vsp1 driver, in 4.16 for the
> uvc driver and in 5.0 for the staging ipu3 driver.
> 
> Does someone remember the reason why we picked /dev/videoX for this in the
> first place?

One of the reason is CSI-2 sensors and CSI-2 receivers. A CSI-2 link
often carries both video and metadata using two different datatypes.
From the point of view of the CSI-2 receiver or the DMA engines, video
and metadata are not distinguishable, the CSI-2 receiver receives one
stream with two data types, demultiplexes them, and passes them to
different DMA engines. A sensor could send two video datatypes, or even
conceptually two metadata data types, and this would work the exact same
way, with each of the two DMA engines capturing data to buffers without
caring about the contents. Given that the datatype selection happens at
runtime, a given DMA engine is thus not dedicated to video or metadata,
any of the DMA engines (and there could also be more than two) could
handle any type of data. For this type of system we thus can't dedicate
device nodes to video or metadata, they need to support either.

-- 
Regards,

Laurent Pinchart