Advice on extending libv4l for media controller support

Advice on extending libv4l for media controller support

[Sergio Aguirre]

Hi Hans,

I'm interested in using libv4l along with my omap4 camera project to
adapt it more easily
to Android CameraHAL, and other applications, to reduce complexity of
them mostly...

So, but the difference is that, this is a media controller device I'm
trying to add support for,
in which I want to create some sort of plugin with specific media
controller configurations,
to avoid userspace to worry about component names and specific
usecases (use sensor resizer, or SoC ISP resizer, etc.).

So, I just wanted to know your advice on some things before I start
hacking your library:

1. Should it be the right thing to add a new subfolder under "lib/",
named like "libomap4iss-mediactl" or something like that ?
2. Do you know if anyone is working on something similar for any other
Media Controller device ?

Thanks in advance for your inputs.

Regards,
Sergio

Re: Advice on extending libv4l for media controller support

[Sergio Aguirre]

+Atsuo

On Wed, May 9, 2012 at 7:08 PM, Sergio Aguirre
<sergio.a.aguirre@gmail.com> wrote:
> Hi Hans,
>
> I'm interested in using libv4l along with my omap4 camera project to
> adapt it more easily
> to Android CameraHAL, and other applications, to reduce complexity of
> them mostly...
>
> So, but the difference is that, this is a media controller device I'm
> trying to add support for,
> in which I want to create some sort of plugin with specific media
> controller configurations,
> to avoid userspace to worry about component names and specific
> usecases (use sensor resizer, or SoC ISP resizer, etc.).
>
> So, I just wanted to know your advice on some things before I start
> hacking your library:
>
> 1. Should it be the right thing to add a new subfolder under "lib/",
> named like "libomap4iss-mediactl" or something like that ?
> 2. Do you know if anyone is working on something similar for any other
> Media Controller device ?
>
> Thanks in advance for your inputs.
>
> Regards,
> Sergio

Re: Advice on extending libv4l for media controller support

[Hans de Goede]

Hi,

I somehow missed the original mail. This is in essence the same problem
as with the omap3 and Laurent and Sakari and I did a design for that
in Brussels in the last quarter of 2011, Laurent and Sakari would work
on fleshing that out, so it is probably best to talk to them about this.

Regards,

Hans


On 05/10/2012 03:54 PM, Sergio Aguirre wrote:
> +Atsuo
>
> On Wed, May 9, 2012 at 7:08 PM, Sergio Aguirre
> <sergio.a.aguirre@gmail.com>  wrote:
>> Hi Hans,
>>
>> I'm interested in using libv4l along with my omap4 camera project to
>> adapt it more easily
>> to Android CameraHAL, and other applications, to reduce complexity of
>> them mostly...
>>
>> So, but the difference is that, this is a media controller device I'm
>> trying to add support for,
>> in which I want to create some sort of plugin with specific media
>> controller configurations,
>> to avoid userspace to worry about component names and specific
>> usecases (use sensor resizer, or SoC ISP resizer, etc.).
>>
>> So, I just wanted to know your advice on some things before I start
>> hacking your library:
>>
>> 1. Should it be the right thing to add a new subfolder under "lib/",
>> named like "libomap4iss-mediactl" or something like that ?
>> 2. Do you know if anyone is working on something similar for any other
>> Media Controller device ?
>>
>> Thanks in advance for your inputs.
>>
>> Regards,
>> Sergio

Re: Advice on extending libv4l for media controller support

[Laurent Pinchart]

Hi Hans,

On Thursday 10 May 2012 16:20:42 Hans de Goede wrote:
> Hi,
> 
> I somehow missed the original mail. This is in essence the same problem
> as with the omap3 and Laurent and Sakari and I did a design for that
> in Brussels in the last quarter of 2011, Laurent and Sakari would work
> on fleshing that out, so it is probably best to talk to them about this.

Let's get Sakari into the loop then. I think he's the most knowledgeable about 
this, even though no implementation has been released (Nokia killing MeeGo 
obviously didn't help).

> On 05/10/2012 03:54 PM, Sergio Aguirre wrote:
> > +Atsuo
> > 
> > On Wed, May 9, 2012 at 7:08 PM, Sergio Aguirre wrote:
> >> Hi Hans,
> >> 
> >> I'm interested in using libv4l along with my omap4 camera project to
> >> adapt it more easily to Android CameraHAL, and other applications, to
> >> reduce complexity of them mostly...
> >> 
> >> So, but the difference is that, this is a media controller device I'm
> >> trying to add support for, in which I want to create some sort of plugin
> >> with specific media controller configurations, to avoid userspace to
> >> worry about component names and specific usecases (use sensor resizer, or
> >> SoC ISP resizer, etc.).
> >> 
> >> So, I just wanted to know your advice on some things before I start
> >> hacking your library:
> >> 
> >> 1. Should it be the right thing to add a new subfolder under "lib/",
> >> named like "libomap4iss-mediactl" or something like that ?
> >> 2. Do you know if anyone is working on something similar for any other
> >> Media Controller device ?
> >> 
> >> Thanks in advance for your inputs.
> >> 
> >> Regards,
> >> Sergio

-- 
Regards,

Laurent Pinchart

Re: Advice on extending libv4l for media controller support

[Ivan T. Ivanov]

Hi Sergio, 

On Thu, 2012-05-10 at 08:54 -0500, Sergio Aguirre wrote:
> +Atsuo
> 
> On Wed, May 9, 2012 at 7:08 PM, Sergio Aguirre
> <sergio.a.aguirre@gmail.com> wrote:
> > Hi Hans,
> >
> > I'm interested in using libv4l along with my omap4 camera project to
> > adapt it more easily
> > to Android CameraHAL, and other applications, to reduce complexity of
> > them mostly...
> >
> > So, but the difference is that, this is a media controller device I'm
> > trying to add support for,
> > in which I want to create some sort of plugin with specific media
> > controller configurations,
> > to avoid userspace to worry about component names and specific
> > usecases (use sensor resizer, or SoC ISP resizer, etc.).
> >
> > So, I just wanted to know your advice on some things before I start
> > hacking your library:
> >

Probably following links can help you. They have been tested
with the OMAP3 ISP.

Regards,
iivanov

[1] http://www.spinics.net/lists/linux-media/msg31901.html
[2]
http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/32704

Re: Advice on extending libv4l for media controller support

[Sergio Aguirre]

Hi Ivan,

Great to hear from you! Long time :)

On Thu, May 10, 2012 at 10:09 AM, Ivan T. Ivanov <iivanov@mm-sol.com> wrote:
>
> Hi Sergio,
>
> On Thu, 2012-05-10 at 08:54 -0500, Sergio Aguirre wrote:
>> +Atsuo
>>
>> On Wed, May 9, 2012 at 7:08 PM, Sergio Aguirre
>> <sergio.a.aguirre@gmail.com> wrote:
>> > Hi Hans,
>> >
>> > I'm interested in using libv4l along with my omap4 camera project to
>> > adapt it more easily
>> > to Android CameraHAL, and other applications, to reduce complexity of
>> > them mostly...
>> >
>> > So, but the difference is that, this is a media controller device I'm
>> > trying to add support for,
>> > in which I want to create some sort of plugin with specific media
>> > controller configurations,
>> > to avoid userspace to worry about component names and specific
>> > usecases (use sensor resizer, or SoC ISP resizer, etc.).
>> >
>> > So, I just wanted to know your advice on some things before I start
>> > hacking your library:
>> >
>
> Probably following links can help you. They have been tested
> with the OMAP3 ISP.

Awesome! This is certainly very useful. I'll look into adding OMAP4
ISS support too.

Thanks for sharing!

Regards,
Sergio

>
> Regards,
> iivanov
>
> [1] http://www.spinics.net/lists/linux-media/msg31901.html
> [2]
> http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/32704
>
>

Re: Advice on extending libv4l for media controller support

[Hans de Goede]

Hi,

On 05/10/2012 05:09 PM, Ivan T. Ivanov wrote:
>
> Hi Sergio,
>
> On Thu, 2012-05-10 at 08:54 -0500, Sergio Aguirre wrote:
>> +Atsuo
>>
>> On Wed, May 9, 2012 at 7:08 PM, Sergio Aguirre
>> <sergio.a.aguirre@gmail.com>  wrote:
>>> Hi Hans,
>>>
>>> I'm interested in using libv4l along with my omap4 camera project to
>>> adapt it more easily
>>> to Android CameraHAL, and other applications, to reduce complexity of
>>> them mostly...
>>>
>>> So, but the difference is that, this is a media controller device I'm
>>> trying to add support for,
>>> in which I want to create some sort of plugin with specific media
>>> controller configurations,
>>> to avoid userspace to worry about component names and specific
>>> usecases (use sensor resizer, or SoC ISP resizer, etc.).
>>>
>>> So, I just wanted to know your advice on some things before I start
>>> hacking your library:
>>>
>
> Probably following links can help you. They have been tested
> with the OMAP3 ISP.
>
> Regards,
> iivanov
>
> [1] http://www.spinics.net/lists/linux-media/msg31901.html
> [2]
> http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/32704

Ah, cool. But that is a bit old stuff. IIRC (pretty sure I do), then we
came to the conclusion that the following would be the best solution:

1) The existing mediactl lib would be extended with a libmediactlvideo lib, which
would be able to control media-ctrl video chains, ie it can:
-give a list of possibly supported formats / sizes / framerates
-setup the chain to deliver a requested format
Since the optimal setup will be hardware specific the idea was to give this
libs per soc plugins, and a generic plugin for simple socs / as fallback.

2) A cmdline utility to set up a chain using libmediactlvideo, so that things
can be tested using raw devices, ie without libv4l2 coming into play, just
like apps like v4l2-ctl allow low level control mostly for testing purposes

3) There would then be a libv4l2 plugin much like the above linked omap3 plugin,
but then generic for any mediactl using video devices, which would use
libmediactlvideo to do the work of setting up the chain (and which will fail to
init when the to be opened device is not part of a mediactl controlled chain).

And AFAIK some work was done in this direction. Sakari? Laurent?

Eitherway it is about time someone started working on this, and I would
greatly prefer the above plan to be implemented. Once we have this in place,
then we can do a new v4l-utils release which officially supports the plugin
API (which currently only lives in master, not in any releases).

Regards,

Hans

Re: Advice on extending libv4l for media controller support

[Robert Abel]

Hi,

Sorry to be late to the party... I wanted to follow up on this
discussion, but forgot and haven't read anything about it since...

On 10.05.2012 17:09, Ivan T. Ivanov wrote:
> On Wed, May 9, 2012 at 7:08 PM, Sergio Aguirre
> <sergio.a.aguirre@gmail.com> wrote:
>> I want to create some sort of plugin with specific media
>> controller configurations,
>> to avoid userspace to worry about component names and specific
>> usecases (use sensor resizer, or SoC ISP resizer, etc.).
> Probably following links can help you. They have been tested
> with the OMAP3 ISP.
>
> Regards,
> iivanov
>
> [1] http://www.spinics.net/lists/linux-media/msg31901.html
> [2]
> http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/32704

I recently extended Yordan Kamenov's libv4l-mcplugin to support multiple
trees per device with extended configurations (-stolen from- inspired by
media-ctl) not tied to specific device nodes (but to device names instead).

I uploaded the patches here
<https://sites.google.com/site/rawbdagslair/libv4l-mcplugin.7z?attredirects=0&d=1>(16kB).
Basically, I used Yordan's patches as a base and worked from there to
fix up his source code and Makefile for cross-compiling using
OpenEmbedded/Yocto.

There are a ton of minor issues with this, starting with the fact that I
did not put proper copyright notices in any of these files. Please
advise if this poses a problem.
Only integral frame size support and no support for native read() calls.
There's a dummy read() function, because for some reason this is
required in libv4l2 0.9.0-test though it's not mentioned anywhere. As
the original plug-in by Yordan, there is currently no cleaning-up of the
internal data structures.

I used this in conjunction with the Gumstix CASPA FS (MT9V032) camera
using some of Laurent's patches and some custom patches which add
ENUM_FMT support to the driver.

Basically, upon opening a given device, all trees are configured once to
load the respective end-point's formats for emulation of setting and
getting formats. Then regular format negotiation by the user application
takes place.

Regards,

Robert

Re: Advice on extending libv4l for media controller support

[Hans de Goede]

Hi,

On 07/26/2012 10:49 PM, Robert Abel wrote:
> Hi,
>
> Sorry to be late to the party... I wanted to follow up on this discussion, but forgot and haven't read anything about it since...
>
> On 10.05.2012 17:09, Ivan T. Ivanov wrote:
>> On Wed, May 9, 2012 at 7:08 PM, Sergio Aguirre
>> <sergio.a.aguirre@gmail.com>  <mailto:sergio.a.aguirre@gmail.com>  wrote:
>>> I want to create some sort of plugin with specific media
>>> controller configurations,
>>> to avoid userspace to worry about component names and specific
>>> usecases (use sensor resizer, or SoC ISP resizer, etc.).
>> Probably following links can help you. They have been tested
>> with the OMAP3 ISP.
>>
>> Regards,
>> iivanov
>>
>> [1]http://www.spinics.net/lists/linux-media/msg31901.html
>> [2]
>> http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/32704
>
> I recently extended Yordan Kamenov's libv4l-mcplugin to support multiple trees per device with extended configurations (-stolen from- inspired by media-ctl) not tied to specific device nodes (but to device names instead).
>
> I uploaded the patches here <https://sites.google.com/site/rawbdagslair/libv4l-mcplugin.7z?attredirects=0&d=1>(16kB). Basically, I used Yordan's patches as a base and worked from there to fix up his source code and Makefile for cross-compiling using OpenEmbedded/Yocto.
>
> There are a ton of minor issues with this, starting with the fact that I did not put proper copyright notices in any of these files. Please advise if this poses a problem.
> Only integral frame size support and no support for native read() calls. There's a dummy read() function, because for some reason this is required in libv4l2 0.9.0-test though it's not mentioned anywhere. As the original plug-in by Yordan, there is currently no cleaning-up of the internal data structures.
>
> I used this in conjunction with the Gumstix CASPA FS (MT9V032) camera using some of Laurent's patches and some custom patches which add ENUM_FMT support to the driver.
>
> Basically, upon opening a given device, all trees are configured once to load the respective end-point's formats for emulation of setting and getting formats. Then regular format negotiation by the user application takes place.

As discussed higher up in this thread, since the initial libv4l-mcplugin was done for
the omap3, we've had several meetings on the topic of libv4l and media-controller using
devices and we came to the following conclusions:

1) The existing mediactl lib would be extended with a libmediactlvideo lib, which
would be able to control media-ctrl video chains, ie it can:
-give a list of possibly supported formats / sizes / framerates
-setup the chain to deliver a requested format
Since the optimal setup will be hardware specific the idea was to give this
libs per soc plugins, and a generic plugin for simple socs / as fallback.

2) A cmdline utility to set up a chain using libmediactlvideo, so that things
can be tested using raw devices, ie without libv4l2 coming into play, just
like apps like v4l2-ctl allow low level control mostly for testing purposes

3) There would then be a libv4l2 plugin much like the above linked omap3 plugin,
but then generic for any mediactl using video devices, which would use
libmediactlvideo to do the work of setting up the chain (and which will fail to
init when the to be opened device is not part of a mediactl controlled chain).

And AFAIK some work was done in this direction. Sakari? Laurent?

Eitherway it is about time someone started working on this, and I would
greatly prefer the above plan to be implemented. Once we have this in place,
then we can do a new v4l-utils release which officially supports the plugin
API (which currently only lives in master, not in any releases).

Regards,

Hans

Re: Advice on extending libv4l for media controller support

[Laurent Pinchart]

Hi Hans,

On Saturday 28 July 2012 12:46:35 Hans de Goede wrote:
> On 07/26/2012 10:49 PM, Robert Abel wrote:
> > Sorry to be late to the party... I wanted to follow up on this discussion,
> > but forgot and haven't read anything about it since...> 
> > On 10.05.2012 17:09, Ivan T. Ivanov wrote:
> >> On Wed, May 9, 2012 at 7:08 PM, Sergio Aguirre
> >> 
> >> <sergio.a.aguirre@gmail.com>  <mailto:sergio.a.aguirre@gmail.com>  wrote:
> >>> I want to create some sort of plugin with specific media
> >>> controller configurations,
> >>> to avoid userspace to worry about component names and specific
> >>> usecases (use sensor resizer, or SoC ISP resizer, etc.).
> >> 
> >> Probably following links can help you. They have been tested
> >> with the OMAP3 ISP.
> >> 
> >> Regards,
> >> iivanov
> >> 
> >> [1]http://www.spinics.net/lists/linux-media/msg31901.html
> >> [2]
> >> http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/
> >> 32704> 
> > I recently extended Yordan Kamenov's libv4l-mcplugin to support multiple
> > trees per device with extended configurations (-stolen from- inspired by
> > media-ctl) not tied to specific device nodes (but to device names
> > instead).
> > 
> > I uploaded the patches here
> > <https://sites.google.com/site/rawbdagslair/libv4l-mcplugin.7z?attredirec
> > ts=0&d=1>(16kB). Basically, I used Yordan's patches as a base and worked
> > from there to fix up his source code and Makefile for cross-compiling
> > using OpenEmbedded/Yocto.
> > 
> > There are a ton of minor issues with this, starting with the fact that I
> > did not put proper copyright notices in any of these files. Please advise
> > if this poses a problem. Only integral frame size support and no support
> > for native read() calls. There's a dummy read() function, because for
> > some reason this is required in libv4l2 0.9.0-test though it's not
> > mentioned anywhere. As the original plug-in by Yordan, there is currently
> > no cleaning-up of the internal data structures.
> > 
> > I used this in conjunction with the Gumstix CASPA FS (MT9V032) camera
> > using some of Laurent's patches and some custom patches which add
> > ENUM_FMT support to the driver.
> > 
> > Basically, upon opening a given device, all trees are configured once to
> > load the respective end-point's formats for emulation of setting and
> > getting formats. Then regular format negotiation by the user application
> > takes place.
> As discussed higher up in this thread, since the initial libv4l-mcplugin was
> done for the omap3, we've had several meetings on the topic of libv4l and
> media-controller using devices and we came to the following conclusions:
> 
> 1) The existing mediactl lib would be extended with a libmediactlvideo lib,
> which would be able to control media-ctrl video chains, ie it can:
> -give a list of possibly supported formats / sizes / framerates
> -setup the chain to deliver a requested format
> Since the optimal setup will be hardware specific the idea was to give this
> libs per soc plugins, and a generic plugin for simple socs / as fallback.
> 
> 2) A cmdline utility to set up a chain using libmediactlvideo, so that
> things can be tested using raw devices, ie without libv4l2 coming into
> play, just like apps like v4l2-ctl allow low level control mostly for
> testing purposes
> 
> 3) There would then be a libv4l2 plugin much like the above linked omap3
> plugin, but then generic for any mediactl using video devices, which would
> use libmediactlvideo to do the work of setting up the chain (and which will
> fail to init when the to be opened device is not part of a mediactl
> controlled chain).
> 
> And AFAIK some work was done in this direction. Sakari? Laurent?

I'm currently in a plane to Finland to work on this with Sakari next week. You 
should hopefully hear from us soon :-)

> Eitherway it is about time someone started working on this, and I would
> greatly prefer the above plan to be implemented. Once we have this in place,
> then we can do a new v4l-utils release which officially supports the plugin
> API (which currently only lives in master, not in any releases).

-- 
Regards,

Laurent Pinchart