[RFC] V4L2 codecs in user space

[RFC] V4L2 codecs in user space

[Damian Hobson-Garcia]

Hello All,

I would like to ask for some comments about a plan to use user space 
video codecs through the V4L interface.  I am thinking of a situation 
similar to the one described on the linuxtv.org wiki at 
http://www.linuxtv.org/wiki/index.php/V4L2_Userspace_Library

The basic premise is to use a FUSE-like driver to connect the standard 
V4L2 api to a user space daemon that will work as an mem-to-mem driver 
for decoding/encoding, compression/decompression and the like.  This 
allows for codecs that are either partially or wholly implemented in 
user space to be exposed through the standard kernel interface.

Before I dive in to implementing this I was hoping to get some comments 
regarding the following:

1. I haven't been able to find any implementation of the design 
described in the wiki page.  Would anyone know if I have missed it?  
Does this exist somewhere, even in part? It seems like that might be a 
good place to start if possible.

2. I think that this could be implemented as either an extension to FUSE 
(like CUSE) or as a V4L2 device driver (that forwards requests through 
the FUSE API).  I think that the V4L2  device driver would be 
sufficient, but would the fact that there is no specific hardware tied 
to it be an issue?  Should it instead be presented as a more generic 
device?

3. And of course anything else that comes to mind.

Thank you,
Damian

Re: [RFC] V4L2 codecs in user space

[Damian Hobson-Garcia]

Hello again,

On 2015-06-02 10:40 AM, Damian Hobson-Garcia wrote:
> Hello All,
> 
> I would like to ask for some comments about a plan to use user space
> video codecs through the V4L interface.  I am thinking of a situation
> similar to the one described on the linuxtv.org wiki at
> http://www.linuxtv.org/wiki/index.php/V4L2_Userspace_Library
> 
> The basic premise is to use a FUSE-like driver to connect the standard
> V4L2 api to a user space daemon that will work as an mem-to-mem driver
> for decoding/encoding, compression/decompression and the like.  This
> allows for codecs that are either partially or wholly implemented in
> user space to be exposed through the standard kernel interface.
> 
> Before I dive in to implementing this I was hoping to get some comments
> regarding the following:
> 
> 1. I haven't been able to find any implementation of the design
> described in the wiki page.  Would anyone know if I have missed it? 
> Does this exist somewhere, even in part? It seems like that might be a
> good place to start if possible.
> 
> 2. I think that this could be implemented as either an extension to FUSE
> (like CUSE) or as a V4L2 device driver (that forwards requests through
> the FUSE API).  I think that the V4L2  device driver would be
> sufficient, but would the fact that there is no specific hardware tied
> to it be an issue?  Should it instead be presented as a more generic
> device?
> 
> 3. And of course anything else that comes to mind.

I've been advised that implementing kernel APIs is userspace is probably
not the most linux-friendly way to go about things and would most likely
not be accepted into the kernel.  I can see the logic of
that statement, but I was wondering if I could confirm that here. Is
this type of design a bad idea?

Also, if this method is not recommended, should there be a 1-2 line
disclaimer on the "V4L2_Userspace_Library" wiki page that mentions this?

Thank you,
Damian

Re: [RFC] V4L2 codecs in user space

[Hans Verkuil]

Hi Damian,

On 06/08/2015 06:42 AM, Damian Hobson-Garcia wrote:
> Hello again,
> 
> On 2015-06-02 10:40 AM, Damian Hobson-Garcia wrote:
>> Hello All,
>>
>> I would like to ask for some comments about a plan to use user space
>> video codecs through the V4L interface.  I am thinking of a situation
>> similar to the one described on the linuxtv.org wiki at
>> http://www.linuxtv.org/wiki/index.php/V4L2_Userspace_Library
>>
>> The basic premise is to use a FUSE-like driver to connect the standard
>> V4L2 api to a user space daemon that will work as an mem-to-mem driver
>> for decoding/encoding, compression/decompression and the like.  This
>> allows for codecs that are either partially or wholly implemented in
>> user space to be exposed through the standard kernel interface.
>>
>> Before I dive in to implementing this I was hoping to get some comments
>> regarding the following:
>>
>> 1. I haven't been able to find any implementation of the design
>> described in the wiki page.  Would anyone know if I have missed it? 
>> Does this exist somewhere, even in part? It seems like that might be a
>> good place to start if possible.
>>
>> 2. I think that this could be implemented as either an extension to FUSE
>> (like CUSE) or as a V4L2 device driver (that forwards requests through
>> the FUSE API).  I think that the V4L2  device driver would be
>> sufficient, but would the fact that there is no specific hardware tied
>> to it be an issue?  Should it instead be presented as a more generic
>> device?
>>
>> 3. And of course anything else that comes to mind.
> 
> I've been advised that implementing kernel APIs is userspace is probably
> not the most linux-friendly way to go about things and would most likely
> not be accepted into the kernel.  I can see the logic of
> that statement, but I was wondering if I could confirm that here. Is
> this type of design a bad idea?

Writing userspace drivers for hardware components is certainly not something
we want to see. The kernel is the gateway between userspace and hardware, so
the kernel is the one that controls the hardware. There are some exceptions
(printers and scanners come to mind), but by and large this rule holds.

But you want to do something different if I understand correctly: you want to
make a V4L2 API to interface to userspace codecs. That does not affect the kernel
at all, and I see no reason why this can't be done.

Basically libv4l2 allows the concept of plugins where all open/close/ioctl/etc.
operations go through the plugin. Today plugins interface with a kernel V4L2
driver, but it is also possible to make a plugin that is completely in userspace.

Nobody ever did it, but we discussed it in the past. The only problem is that
since there is no actual /dev/videoX device, what do you specify here? Perhaps
a predefined /dev/video-<codecname>X 'device name'?

There is currently only one plugin that is part of v4l-utils:
v4l-utils/lib/libv4l-mplane

That might be a good starting point.

Hope this helps,

Regards,

	Hans

> Also, if this method is not recommended, should there be a 1-2 line
> disclaimer on the "V4L2_Userspace_Library" wiki page that mentions this?
> 
> Thank you,
> Damian
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [RFC] V4L2 codecs in user space

[Damian Hobson-Garcia]

Hi Hans,

Thank you for your comments,
On 2015-06-08 5:25 PM, Hans Verkuil wrote:
> Hi Damian,
> 
> On 06/08/2015 06:42 AM, Damian Hobson-Garcia wrote:
>> Hello again,
>>
>> On 2015-06-02 10:40 AM, Damian Hobson-Garcia wrote:
>>> Hello All,
>>>
>>> I would like to ask for some comments about a plan to use user space
>>> video codecs through the V4L interface.  I am thinking of a situation
>>> similar to the one described on the linuxtv.org wiki at
>>> http://www.linuxtv.org/wiki/index.php/V4L2_Userspace_Library
>>>
>>> The basic premise is to use a FUSE-like driver to connect the standard
>>> V4L2 api to a user space daemon that will work as an mem-to-mem driver
>>> for decoding/encoding, compression/decompression and the like.  This
>>> allows for codecs that are either partially or wholly implemented in
>>> user space to be exposed through the standard kernel interface.
>>>
>>> Before I dive in to implementing this I was hoping to get some comments
>>> regarding the following:
>>>
>>> 1. I haven't been able to find any implementation of the design
>>> described in the wiki page.  Would anyone know if I have missed it? 
>>> Does this exist somewhere, even in part? It seems like that might be a
>>> good place to start if possible.
>>>
>>> 2. I think that this could be implemented as either an extension to FUSE
>>> (like CUSE) or as a V4L2 device driver (that forwards requests through
>>> the FUSE API).  I think that the V4L2  device driver would be
>>> sufficient, but would the fact that there is no specific hardware tied
>>> to it be an issue?  Should it instead be presented as a more generic
>>> device?
>>>
>>> 3. And of course anything else that comes to mind.
>>
>> I've been advised that implementing kernel APIs is userspace is probably
>> not the most linux-friendly way to go about things and would most likely
>> not be accepted into the kernel.  I can see the logic of
>> that statement, but I was wondering if I could confirm that here. Is
>> this type of design a bad idea?
> 
> Writing userspace drivers for hardware components is certainly not something
> we want to see. The kernel is the gateway between userspace and hardware, so
> the kernel is the one that controls the hardware. There are some exceptions
> (printers and scanners come to mind), but by and large this rule holds.
> 
I see, thank you.

> But you want to do something different if I understand correctly: you want to
> make a V4L2 API to interface to userspace codecs. That does not affect the kernel
> at all, and I see no reason why this can't be done.
> 
> Basically libv4l2 allows the concept of plugins where all open/close/ioctl/etc.
> operations go through the plugin. Today plugins interface with a kernel V4L2
> driver, but it is also possible to make a plugin that is completely in userspace.
> 
> Nobody ever did it, but we discussed it in the past. The only problem is that
> since there is no actual /dev/videoX device, what do you specify here? Perhaps
> a predefined /dev/video-<codecname>X 'device name'?

I suppose that I would also need to do something for mmap buffers as
well, since mmap() is not part of the plugin API.  I guess I could try
to abuse the fake mmap used by the libv4lconvert, but that might get
messy if an application requests a different pixel format from what the
codec delivers.

Thank you,
Damian

> 
> There is currently only one plugin that is part of v4l-utils:
> v4l-utils/lib/libv4l-mplane
> 
> That might be a good starting point.
> 
> Hope this helps,
> 
> Regards,
> 
> 	Hans
> 
>> Also, if this method is not recommended, should there be a 1-2 line
>> disclaimer on the "V4L2_Userspace_Library" wiki page that mentions this?
>>
>> Thank you,
>> Damian
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-media" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 

Re: [RFC] V4L2 codecs in user space

[Hans Verkuil]

On 06/08/2015 11:54 AM, Damian Hobson-Garcia wrote:
> Hi Hans,
> 
> Thank you for your comments,
> On 2015-06-08 5:25 PM, Hans Verkuil wrote:
>> Hi Damian,
>>
>> On 06/08/2015 06:42 AM, Damian Hobson-Garcia wrote:
>>> Hello again,
>>>
>>> On 2015-06-02 10:40 AM, Damian Hobson-Garcia wrote:
>>>> Hello All,
>>>>
>>>> I would like to ask for some comments about a plan to use user space
>>>> video codecs through the V4L interface.  I am thinking of a situation
>>>> similar to the one described on the linuxtv.org wiki at
>>>> http://www.linuxtv.org/wiki/index.php/V4L2_Userspace_Library
>>>>
>>>> The basic premise is to use a FUSE-like driver to connect the standard
>>>> V4L2 api to a user space daemon that will work as an mem-to-mem driver
>>>> for decoding/encoding, compression/decompression and the like.  This
>>>> allows for codecs that are either partially or wholly implemented in
>>>> user space to be exposed through the standard kernel interface.
>>>>
>>>> Before I dive in to implementing this I was hoping to get some comments
>>>> regarding the following:
>>>>
>>>> 1. I haven't been able to find any implementation of the design
>>>> described in the wiki page.  Would anyone know if I have missed it? 
>>>> Does this exist somewhere, even in part? It seems like that might be a
>>>> good place to start if possible.
>>>>
>>>> 2. I think that this could be implemented as either an extension to FUSE
>>>> (like CUSE) or as a V4L2 device driver (that forwards requests through
>>>> the FUSE API).  I think that the V4L2  device driver would be
>>>> sufficient, but would the fact that there is no specific hardware tied
>>>> to it be an issue?  Should it instead be presented as a more generic
>>>> device?
>>>>
>>>> 3. And of course anything else that comes to mind.
>>>
>>> I've been advised that implementing kernel APIs is userspace is probably
>>> not the most linux-friendly way to go about things and would most likely
>>> not be accepted into the kernel.  I can see the logic of
>>> that statement, but I was wondering if I could confirm that here. Is
>>> this type of design a bad idea?
>>
>> Writing userspace drivers for hardware components is certainly not something
>> we want to see. The kernel is the gateway between userspace and hardware, so
>> the kernel is the one that controls the hardware. There are some exceptions
>> (printers and scanners come to mind), but by and large this rule holds.
>>
> I see, thank you.
> 
>> But you want to do something different if I understand correctly: you want to
>> make a V4L2 API to interface to userspace codecs. That does not affect the kernel
>> at all, and I see no reason why this can't be done.
>>
>> Basically libv4l2 allows the concept of plugins where all open/close/ioctl/etc.
>> operations go through the plugin. Today plugins interface with a kernel V4L2
>> driver, but it is also possible to make a plugin that is completely in userspace.
>>
>> Nobody ever did it, but we discussed it in the past. The only problem is that
>> since there is no actual /dev/videoX device, what do you specify here? Perhaps
>> a predefined /dev/video-<codecname>X 'device name'?
> 
> I suppose that I would also need to do something for mmap buffers as
> well, since mmap() is not part of the plugin API.  I guess I could try
> to abuse the fake mmap used by the libv4lconvert, but that might get
> messy if an application requests a different pixel format from what the
> codec delivers.

Or just add mmap to the plugin API. It would make sense in this case.

Regards,

	Hans

Re: [RFC] V4L2 codecs in user space

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 618 bytes --]

Le lundi 08 juin 2015 à 13:42 +0900, Damian Hobson-Garcia a écrit :
> Also, if this method is not recommended, should there be a 1-2 line
> disclaimer on the "V4L2_Userspace_Library" wiki page that mentions 
> this?

I think you may have got that wrong. The V4L2 userspace library is not
implementing any device drivers. It allow older software to work with
latest V4L2 features by emulating what is possible. It also implement
platform specific setups (media controller) and eventually will contain
needed parsers that would otherwise represent a security threat if ran
inside the Linux Kernel.

Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [RFC] V4L2 codecs in user space

[Damian Hobson-Garcia]

Hi Nicolas,

On 2015-06-08 10:50 PM, Nicolas Dufresne wrote:
> Le lundi 08 juin 2015 à 13:42 +0900, Damian Hobson-Garcia a écrit
> :
>> Also, if this method is not recommended, should there be a 1-2
>> line disclaimer on the "V4L2_Userspace_Library" wiki page that
>> mentions this?
> 
> I think you may have got that wrong. The V4L2 userspace library is
> not implementing any device drivers. It allow older software to
> work with latest V4L2 features by emulating what is possible. It
> also implement platform specific setups (media controller) and
> eventually will contain needed parsers that would otherwise
> represent a security threat if ran inside the Linux Kernel.
> 
Just to verify that we're on the same page, but it sounds to me that
what you're describing is the libv4l library, whereas what I was
originally referring to was the section under "A solution" at
http://linuxtv.org/wiki/index.php/V4L2_Userspace_Library
that talks about a FUSE interface to a daemon running in a different
process.  Or perhaps we're both talking about the same thing.  That
page also seems to mention that something like libv4l would be the
ideal solution, but that it does not exist (it's an old page).

Thank you,
Damian


> Nicolas
> 

Re: [RFC] V4L2 codecs in user space

[Enrico Weigelt, metux IT consult]

Am 08.06.2015 um 12:04 schrieb Hans Verkuil:

<big_snip />

Just curious: as we're talking about userland libraries,
why not just using gstreamer ?


--mtx

--
Enrico Weigelt, metux IT consult
+49-151-27565287
MELAG Medizintechnik oHG Sitz Berlin Registergericht AG Charlottenburg HRA 21333 B

Wichtiger Hinweis: Diese Nachricht kann vertrauliche oder nur für einen begrenzten Personenkreis bestimmte Informationen enthalten. Sie ist ausschließlich für denjenigen bestimmt, an den sie gerichtet worden ist. Wenn Sie nicht der Adressat dieser E-Mail sind, dürfen Sie diese nicht kopieren, weiterleiten, weitergeben oder sie ganz oder teilweise in irgendeiner Weise nutzen. Sollten Sie diese E-Mail irrtümlich erhalten haben, so benachrichtigen Sie bitte den Absender, indem Sie auf diese Nachricht antworten. Bitte löschen Sie in diesem Fall diese Nachricht und alle Anhänge, ohne eine Kopie zu behalten.
Important Notice: This message may contain confidential or privileged information. It is intended only for the person it was addressed to. If you are not the intended recipient of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. If you received this email in error please notify the sender by replying and delete this message and any attachments without retaining a copy.

Re: [RFC] V4L2 codecs in user space

[Damian Hobson-Garcia]

Hi Enrico,

On 2015-06-10 5:17 PM, Enrico Weigelt, metux IT consult wrote:
> Am 08.06.2015 um 12:04 schrieb Hans Verkuil:
> 
> <big_snip />
> 
> Just curious: as we're talking about userland libraries,
> why not just using gstreamer ?
> 

I would prefer to use gstreamer if I could, but the application
(Chromium) doesn't support it, so maybe my best option (for now) is to
create a libv4l to gstreamer bridge using the plugin API.

Damian