[RFC] move sensor control out of kernel

[RFC] move sensor control out of kernel

[eric miao]

I know some one has different opinion, but since the sensor control logic is
getting more and more complicated, and provided that differences between
sensors and vendors are already too many. Is it better to move sensor
control out of kernel.

Most sensors come with a serial control channel, I2C as can be seen
most commonly. Access to the control information can be done by
i2c-dev interface if possible, thus the driver can be freed as an I2C
stub only. So technically, this is practical.

The benefits I can think of now are:

1. simplified sensor driver design

2. sensor control related debugging can be moved to user space thus
    reducing the debugging effort

3. accessing registers in user space can be done by many other ways
    say, UART. E.g.
    ADCM2650 and ADCM2670 differs in the control channel connection,
    one by I2C and the other by UART, the user space control logic has
    only to decide which device node to open: /dev/i2c/xxx or /dev/ttyXX

Another biggest concern to the V4L2 API itself, sensor nowadays has
more control ability than it used to be, some smart sensor provides
even more like auto focus control, lens control, flash mode, and many
other features that current V4L2 API cannot cover.

Besides, along with the complicated image processing chain, it might
be better described by kinds of pipeline, like what gstreamer is doing
now. Moving some or most of the logic to user space will also significantly
reduce the effort of kernel development.

Now the problem is: we don't have a standard in user space :(

Just a topic, any comments. Thanks

-- 
Cheers
- eric

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [RFC] move sensor control out of kernel

[Markus Rechberger]

On 2/29/08, eric miao <eric.y.miao@gmail.com> wrote:
> I know some one has different opinion, but since the sensor control logic is
> getting more and more complicated, and provided that differences between
> sensors and vendors are already too many. Is it better to move sensor
> control out of kernel.
>
> Most sensors come with a serial control channel, I2C as can be seen
> most commonly. Access to the control information can be done by
> i2c-dev interface if possible, thus the driver can be freed as an I2C
> stub only. So technically, this is practical.
>
> The benefits I can think of now are:
>
> 1. simplified sensor driver design
>
> 2. sensor control related debugging can be moved to user space thus
> reducing the debugging effort
>
> 3. accessing registers in user space can be done by many other ways
> say, UART. E.g.
> ADCM2650 and ADCM2670 differs in the control channel connection,
> one by I2C and the other by UART, the user space control logic has
> only to decide which device node to open: /dev/i2c/xxx or /dev/ttyXX
>
> Another biggest concern to the V4L2 API itself, sensor nowadays has
> more control ability than it used to be, some smart sensor provides
> even more like auto focus control, lens control, flash mode, and many
> other features that current V4L2 API cannot cover.
>
> Besides, along with the complicated image processing chain, it might
> be better described by kinds of pipeline, like what gstreamer is doing
> now. Moving some or most of the logic to user space will also significantly
> reduce the effort of kernel development.
>
> Now the problem is: we don't have a standard in user space :(
>

Video4Linux is still a proposal in userspace, vlc and xine only
support v4l1 once you come up with a sane interface it can easily be
adopted by providing a small plugin.
If you look at xine, it already provides a quite mature image
processing chain, the xine API itself is also abstracted to be very
easy.

I also like the way of having something more flexible in userspace,
the question came up how people can emulate a v4l device, there are
projects available for feeding digital datastreams from userspace into
the kernel and providing a video interface for streaming it back to
userspace
I think it's better to discuss such an interface on the Xine, VLC,
gstreamer or any other appropriate ML.

Your idea basically increases the system stability and portability of
the driver.
BSD systems are also able to export an i2c interface.

Markus

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [RFC] move sensor control out of kernel

[Paulius Zaleckas]

Hi,

Take a look at my comments bellow. These are just my thoughts. Maybe I 
am not right...

Best regards,
Paulius Zaleckas

eric miao wrote:
> I know some one has different opinion, but since the sensor control logic is
> getting more and more complicated, and provided that differences between
> sensors and vendors are already too many. Is it better to move sensor
> control out of kernel.

I agree that sensor control drivers are very different and most of them 
work only with specific data bus driver. That is really big problem.
"soc-camera" framework developed by Guennadi Liakhovetski IMO is the 
right solution.

> Most sensors come with a serial control channel, I2C as can be seen
> most commonly. Access to the control information can be done by
> i2c-dev interface if possible, thus the driver can be freed as an I2C
> stub only. So technically, this is practical.

Keep in mind that i2c is used not only for camera sensor control, but 
for other devices also.

> The benefits I can think of now are:
> 
> 1. simplified sensor driver design

It is pretty simple with soc-camera framework also. And the best thing 
is that it is unified. If you implement sensor driver in userspace then 
what kind of interface should V4L data bus driver expose.

> 2. sensor control related debugging can be moved to user space thus
>     reducing the debugging effort
> 
> 3. accessing registers in user space can be done by many other ways
>     say, UART. E.g.
>     ADCM2650 and ADCM2670 differs in the control channel connection,
>     one by I2C and the other by UART, the user space control logic has
>     only to decide which device node to open: /dev/i2c/xxx or /dev/ttyXX

I think it is the same for kernel also... Think about serial, PS/2 and 
USB mouse.

> Another biggest concern to the V4L2 API itself, sensor nowadays has
> more control ability than it used to be, some smart sensor provides
> even more like auto focus control, lens control, flash mode, and many
> other features that current V4L2 API cannot cover.

Implement these controls. Make a patch and send it here. If everything 
is OK it should be merged and V4L will be capable to handle these controls.

> Besides, along with the complicated image processing chain, it might
> be better described by kinds of pipeline, like what gstreamer is doing
> now. Moving some or most of the logic to user space will also significantly
> reduce the effort of kernel development.

Yes. It will also dramatically increase development efforts in userspace 
applications. And there will be a lot of different implementations in 
these applications. Actually a lot more than it is in the kernel now.

> Now the problem is: we don't have a standard in user space :(
> 
> Just a topic, any comments. Thanks
> 

My conclusion: your idea just moves problem from kernel to userspace, 
but doesn't solve it...

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [RFC] move sensor control out of kernel

[eric miao]

On Fri, Feb 29, 2008 at 8:49 PM, Paulius Zaleckas
<paulius.zaleckas@teltonika.lt> wrote:
> Hi,
>
>  Take a look at my comments bellow. These are just my thoughts. Maybe I
>  am not right...
>
>  Best regards,
>  Paulius Zaleckas
>
>
>  eric miao wrote:
>  > I know some one has different opinion, but since the sensor control logic is
>  > getting more and more complicated, and provided that differences between
>  > sensors and vendors are already too many. Is it better to move sensor
>  > control out of kernel.
>
>  I agree that sensor control drivers are very different and most of them
>  work only with specific data bus driver. That is really big problem.
>  "soc-camera" framework developed by Guennadi Liakhovetski IMO is the
>  right solution.
>
>
>  > Most sensors come with a serial control channel, I2C as can be seen
>  > most commonly. Access to the control information can be done by
>  > i2c-dev interface if possible, thus the driver can be freed as an I2C
>  > stub only. So technically, this is practical.
>
>  Keep in mind that i2c is used not only for camera sensor control, but
>  for other devices also.
>
>
>  > The benefits I can think of now are:
>  >
>  > 1. simplified sensor driver design
>
>  It is pretty simple with soc-camera framework also. And the best thing
>  is that it is unified. If you implement sensor driver in userspace then
>  what kind of interface should V4L data bus driver expose.
>

The soc-camera is pretty good. Yet it doesn't solve the issue of
complicated sensor control. mt9m001.c and mt9v022.c are two
good examples, but are all too simple. I have a sensor driver here
with more than 4000 lines of code.

>
>  > 2. sensor control related debugging can be moved to user space thus
>  >     reducing the debugging effort
>  >
>  > 3. accessing registers in user space can be done by many other ways
>  >     say, UART. E.g.
>  >     ADCM2650 and ADCM2670 differs in the control channel connection,
>  >     one by I2C and the other by UART, the user space control logic has
>  >     only to decide which device node to open: /dev/i2c/xxx or /dev/ttyXX
>
>  I think it is the same for kernel also... Think about serial, PS/2 and
>  USB mouse.
>

For kernel, that means you have to provide each bus a driver for the specific
connection type, though the kernel has organized things quite well.

>
>  > Another biggest concern to the V4L2 API itself, sensor nowadays has
>  > more control ability than it used to be, some smart sensor provides
>  > even more like auto focus control, lens control, flash mode, and many
>  > other features that current V4L2 API cannot cover.
>
>  Implement these controls. Make a patch and send it here. If everything
>  is OK it should be merged and V4L will be capable to handle these controls.
>

Mmm...then do you expect an API growing into a monster in the kernel space?

>
>  > Besides, along with the complicated image processing chain, it might
>  > be better described by kinds of pipeline, like what gstreamer is doing
>  > now. Moving some or most of the logic to user space will also significantly
>  > reduce the effort of kernel development.
>
>  Yes. It will also dramatically increase development efforts in userspace
>  applications. And there will be a lot of different implementations in
>  these applications. Actually a lot more than it is in the kernel now.
>

Development in user space is always cheaper than in kernel.

>
>  > Now the problem is: we don't have a standard in user space :(
>  >
>  > Just a topic, any comments. Thanks
>  >
>
>  My conclusion: your idea just moves problem from kernel to userspace,
>  but doesn't solve it...
>

I'm not trying to solve how to control the sensor but how to simplify
the kernel effort and maybe with small reward of stability and ease
of development.

>  --
>  video4linux-list mailing list
>  Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
>  https://www.redhat.com/mailman/listinfo/video4linux-list
>

-- 
Cheers
- eric

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [RFC] move sensor control out of kernel

[Paulius Zaleckas]

eric miao wrote:
>>  > The benefits I can think of now are:
>>  >
>>  > 1. simplified sensor driver design
>>
>>  It is pretty simple with soc-camera framework also. And the best thing
>>  is that it is unified. If you implement sensor driver in userspace then
>>  what kind of interface should V4L data bus driver expose.
>>
> 
> The soc-camera is pretty good. Yet it doesn't solve the issue of
> complicated sensor control. mt9m001.c and mt9v022.c are two
> good examples, but are all too simple. I have a sensor driver here
> with more than 4000 lines of code.

Yes I can imagine driver for analog devices ADV family video decoder :)
Some time ago I wrote driver for ADV...

But I don't imagine application or library with support for all 
devices... Kernel is easy to configure and to compile only drivers you 
need. Also kernel modules are good to load only the driver you need.

Another thing is that there will be different multiple implementations 
of this "4000 lines of code" driver. This means that each of them will 
be tested less than the single one in kernel.

>>  > Another biggest concern to the V4L2 API itself, sensor nowadays has
>>  > more control ability than it used to be, some smart sensor provides
>>  > even more like auto focus control, lens control, flash mode, and many
>>  > other features that current V4L2 API cannot cover.
>>
>>  Implement these controls. Make a patch and send it here. If everything
>>  is OK it should be merged and V4L will be capable to handle these controls.
>>
> 
> Mmm...then do you expect an API growing into a monster in the kernel space?

I guess this will happen anyway, because of tv tuners, dvb-t cards and 
etc. There is no way there to separate "sensor" from the data bus.

And you will have to export data bus interface to the userspace.

>>  My conclusion: your idea just moves problem from kernel to userspace,
>>  but doesn't solve it...
>>
> 
> I'm not trying to solve how to control the sensor but how to simplify
> the kernel effort and maybe with small reward of stability and ease
> of development.

I must agree about kernel stability here.

>>  --
>>  video4linux-list mailing list
>>  Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
>>  https://www.redhat.com/mailman/listinfo/video4linux-list
>>
> 

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [RFC] move sensor control out of kernel

[Guennadi Liakhovetski]

On Fri, 29 Feb 2008, eric miao wrote:
> The soc-camera is pretty good. Yet it doesn't solve the issue of
> complicated sensor control. mt9m001.c and mt9v022.c are two
> good examples, but are all too simple. I have a sensor driver here
> with more than 4000 lines of code.

mt9xxxx drivers also only implement a subset of camera capabilities. The 
approach taken with these two drivers was to implement a minimal 
functional control subset in the kernel, so that "standard" user-space 
applications like xawtv, mplayer, etc. can work with them out of the box, 
and provide access to camera registers over the .get_register / 
.set_register interface for applications requiring finer camera tuning. It 
has disadvantages, of course. But it avoids kernel bloat, nevertheless 
providing reasonable functionality, reduces (kernel) development costs, 
etc. Maybe this interface should be made standard and at least removed 
from under #ifdef CONFIG_VIDEO_ADV_DEBUG?

Thanks
Guennadi
---
Guennadi Liakhovetski

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [RFC] move sensor control out of kernel

[Mauro Carvalho Chehab]

On Fri, 29 Feb 2008 18:35:18 +0200
Paulius Zaleckas <paulius.zaleckas@teltonika.lt> wrote:

> 
> eric miao wrote:
> >>  > The benefits I can think of now are:
> >>  >
> >>  > 1. simplified sensor driver design
> >>
> >>  It is pretty simple with soc-camera framework also. And the best thing
> >>  is that it is unified. If you implement sensor driver in userspace then
> >>  what kind of interface should V4L data bus driver expose.
> >>
> > 
> > The soc-camera is pretty good. Yet it doesn't solve the issue of
> > complicated sensor control. mt9m001.c and mt9v022.c are two
> > good examples, but are all too simple. I have a sensor driver here
> > with more than 4000 lines of code.
> 
> Yes I can imagine driver for analog devices ADV family video decoder :)
> Some time ago I wrote driver for ADV...
> 
> But I don't imagine application or library with support for all 
> devices... Kernel is easy to configure and to compile only drivers you 
> need. Also kernel modules are good to load only the driver you need.
> 
> Another thing is that there will be different multiple implementations 
> of this "4000 lines of code" driver. This means that each of them will 
> be tested less than the single one in kernel.

Having this at kernel avoids duplicating the code. Also, a kernel driver without
the sensor means a broken driver, since it won't work without the userspace
counterpart.

> >>  > Another biggest concern to the V4L2 API itself, sensor nowadays has
> >>  > more control ability than it used to be, some smart sensor provides
> >>  > even more like auto focus control, lens control, flash mode, and many
> >>  > other features that current V4L2 API cannot cover.
> >>
> >>  Implement these controls. Make a patch and send it here. If everything
> >>  is OK it should be merged and V4L will be capable to handle these controls.
> >>
> > 
> > Mmm...then do you expect an API growing into a monster in the kernel space?
> 
> I guess this will happen anyway, because of tv tuners, dvb-t cards and 
> etc. There is no way there to separate "sensor" from the data bus.

This is just a matter of adding newer controls to VIDIOC_[G/S/ENUM]_CTRL. 
This way, generic apps like qv4l2 can properly configure any device. This also
avoids the need of having special userspace apps for certain devices.

If you don't standardize the controls at the API, then you'll have a real mess
at the userspace.

Cheers,
Mauro

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list