Configurable Video Controller Driver

Configurable Video Controller Driver

[Julien BERAUD]

Hi,

We are developing a driver for our video controller which has the 
particularity of being very reconfigurable.

We have reached a point at which the complexity and variety of the 
applications we need to implement forces us to
design an api/library that allows us to configure the interconnection of 
the different video processing units(Camera interfaces,
LCD interfaces, scalers, rotators, demosaicing, dead pixel correction, 
etc...) from userland.

The media controller api has the limitation of not being able to create 
links but just browsing and activating/deactivating them.
If we just allowed a user to activate/deactivate links, then we would 
have to declare all the possible connections between
the different blocks, which would make it very confusing from a userland 
point of view. Moreover, the interconnection constraints
would have to be dealt with very generically, which would make it very 
difficult in the kernel too.

The conclusion we have reached yet is that we have to design an API that 
allows us to create v4l2 subdevices that have certain 
capabilities(scaling,rotating, demosaicing, etc...) and then to create 
links between them from a userland library.
I would like to know if anything like this has been implemented yet or 
if someone has been thinking about such an
architecture before.

Thanks for your inputs,
Julien BERAUD

Re: Configurable Video Controller Driver

[Sakari Ailus]

Hi Julien,

On Thu, Jul 10, 2014 at 04:19:06PM +0200, Julien BERAUD wrote:
> We are developing a driver for our video controller which has the
> particularity of being very reconfigurable.
> 
> We have reached a point at which the complexity and variety of the
> applications we need to implement forces us to
> design an api/library that allows us to configure the
> interconnection of the different video processing units(Camera
> interfaces,
> LCD interfaces, scalers, rotators, demosaicing, dead pixel
> correction, etc...) from userland.
> 
> The media controller api has the limitation of not being able to
> create links but just browsing and activating/deactivating them.
> If we just allowed a user to activate/deactivate links, then we
> would have to declare all the possible connections between
> the different blocks, which would make it very confusing from a
> userland point of view. Moreover, the interconnection constraints
> would have to be dealt with very generically, which would make it
> very difficult in the kernel too.

How many different blocks do you have? Can they be connected in arbitrary
ways? If not, what kind of limitations do you have?

The Media controller is originally intended for modelling complex devices
with hardware data paths between the sub-blocks. The question is: does your
device fit into that group, even if could be a little more complex than the
devices that are currently supported?

> The conclusion we have reached yet is that we have to design an API
> that allows us to create v4l2 subdevices that have certain
> capabilities(scaling,rotating, demosaicing, etc...) and then to
> create links between them from a userland library.

Can you create arbitrary devices at will, or do these devices exist on
hardware all the time?

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: Configurable Video Controller Driver

[Julien BERAUD]

Hi Sakari,
Thank you for your answer.
Le 21/07/2014 13:14, Sakari Ailus a écrit :
> Hi Julien,
>
> On Thu, Jul 10, 2014 at 04:19:06PM +0200, Julien BERAUD wrote:
>> We are developing a driver for our video controller which has the
>> particularity of being very reconfigurable.
>>
>> We have reached a point at which the complexity and variety of the
>> applications we need to implement forces us to
>> design an api/library that allows us to configure the
>> interconnection of the different video processing units(Camera
>> interfaces,
>> LCD interfaces, scalers, rotators, demosaicing, dead pixel
>> correction, etc...) from userland.
>>
>> The media controller api has the limitation of not being able to
>> create links but just browsing and activating/deactivating them.
>> If we just allowed a user to activate/deactivate links, then we
>> would have to declare all the possible connections between
>> the different blocks, which would make it very confusing from a
>> userland point of view. Moreover, the interconnection constraints
>> would have to be dealt with very generically, which would make it
>> very difficult in the kernel too.
> How many different blocks do you have? Can they be connected in arbitrary
> ways? If not, what kind of limitations do you have?
>
> The Media controller is originally intended for modelling complex devices
> with hardware data paths between the sub-blocks. The question is: does your
> device fit into that group, even if could be a little more complex than the
> devices that are currently supported?
We have 44 different hardware blocks that can be connected in arbitrary 
ways but with some constraints. Some of the blocks have several 
inputs(blenders) and some(one type only) have 2 outputs(the block is a 
fork).
There are some limitations. Some blocks only accept Raw Bayer as 
input(ISP). There are some small limitations here and there, like if you 
use a scaler with a high scaling factor, you need to go to memory 
because the internal memory is not sufficient.
Some of the blocks are FIFOs to write/read to memory.
In the end, there are 6 camera inputs and 2 lcd outputs, all of them 
parallel, 2 ISPs with 2 blocks each(bayer/yuv), and 2 stat calculators, 
12 Fifos, 4 color converters, 2 blenders with 4 inputs each, 2 
scalers/rotators, 4 forks, 2 sat, 2 I3D, 2 gamma converters.

Our device might be a little more complex than the devices that are 
currently supported, though I am not aware of all the currently 
supported devices.

Something else, the Media controller doesn't allow to create links from 
userland, which is what I think we would need in order to avoid to 
declare all the possible combinations because that would be hard to 
understand in userland and complicated to handle in the kernel.

We have come from a totally rigid interconnection declared in the BSP to 
become more and more generic but it is probable that becoming too 
generic wouldn't make things any better.


>
>> The conclusion we have reached yet is that we have to design an API
>> that allows us to create v4l2 subdevices that have certain
>> capabilities(scaling,rotating, demosaicing, etc...) and then to
>> create links between them from a userland library.
> Can you create arbitrary devices at will, or do these devices exist on
> hardware all the time?
>
The devices exist on hardware at all time. What I would like to do is to 
create "virtual" subdevices that have some of the capabilities provided 
by the blocks in order to benefit from all the features the hardware 
provides in order to have a media controller tree that is clear and that 
fits the needs of the current application. We are currently thinking 
about designing a kernel driver along with a userland library that 
allows us to reconfigure our hardware paths in order to create those 
virtual subdevices and then expose them with the Media Controller/V4L2 
subdev api.

I have tried to bring the topic up on the  v4l irc because it is a 
complicated matter. I will try again.

There may be a better way to handle such a complicated video controller, 
but I haven't found it yet. If you have any idea, I am totally open to 
reconsider the path we are taking.

In the end, it would be possible to handle each hardware block as an 
independant subdev or input/output as long as we would be able to create 
links with the media api but 2 considerations make it easier to do what 
we are currently doing :
-> In our applications, most of the hardware paths between the blocks 
won't move at runtime, just a few links would have to be reconfigured at 
runtime which could be exposed with the media api.
-> We already have a low level api that allows us to handle the creation 
of a hardware block that provide a list of capabilities.

Thank you,
Julien