Capturing video and still images using one driver

Capturing video and still images using one driver

[Neil Johnson]

video4linux-list,

I am developing on the OMAP3 system using a micron/aptina mt9p031
5-megapixel imager.  This CMOs imager supports full image capture at 5 fps
(2592x1944 pixels) or you can capture subregions using skipping and
binning.  We have proven both capabilities, but would like to be able to
capture both VGA sized video and still images without using separate
drivers.

So far, I have not found any support for capturing large images and video
through a single driver interface.  Does such a capability exist within
v4l2?  One possible way to solve the problem is to allocate N buffers of the
full 5-megapixel size (they end up being 10-MB for each buffer since I'm
using 16-bits per pixel), and then using a small portion of that for video.
These is less desirable since when I'm capturing video, I only need 640x480
size buffers, and I should only need one snapshot buffer at a time (I'm not
streaming them in, just take a snapshot and go back to live video capture).
Is there a way to allocate a side-buffer for the 5-megapixel image and also
allocate "normal" sized buffers for video within the same driver?  Any
recommendations on how to accomplish such a thing?  I would think that
camera-phones using linux would have run up against this.  Thanks,

Neil Johnson
--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: Capturing video and still images using one driver

[Guennadi Liakhovetski]

On Mon, 2 Nov 2009, Neil Johnson wrote:

> video4linux-list,
> 
> I am developing on the OMAP3 system using a micron/aptina mt9p031
> 5-megapixel imager.  This CMOs imager supports full image capture at 5 fps
> (2592x1944 pixels) or you can capture subregions using skipping and
> binning.  We have proven both capabilities, but would like to be able to
> capture both VGA sized video and still images without using separate
> drivers.
> 
> So far, I have not found any support for capturing large images and video
> through a single driver interface.  Does such a capability exist within
> v4l2?  One possible way to solve the problem is to allocate N buffers of the
> full 5-megapixel size (they end up being 10-MB for each buffer since I'm
> using 16-bits per pixel), and then using a small portion of that for video.
> These is less desirable since when I'm capturing video, I only need 640x480
> size buffers, and I should only need one snapshot buffer at a time (I'm not
> streaming them in, just take a snapshot and go back to live video capture).
> Is there a way to allocate a side-buffer for the 5-megapixel image and also
> allocate "normal" sized buffers for video within the same driver?  Any
> recommendations on how to accomplish such a thing?  I would think that
> camera-phones using linux would have run up against this.  Thanks,

I came across the same problem when working on the rj54n1cb0c driver. 
What's even more exciting with that sensor, is that it has separate 
frame-size settings for preview (video) and still capture. So far the 
driver doesn't have any support for those still image frame settings, but 
I was thinking about implenting it using the read V4L method. I.e., you 
would use the mmap method just normally to capture video and use read to 
get still images? OTOH, with my driver I would need to differentiate 
between S_FMT for video and still. Would we have to use different inputs 
for them?

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: Capturing video and still images using one driver

[Neil Johnson]

On Tue, Nov 3, 2009 at 3:02 PM, Guennadi Liakhovetski <g.liakhovetski@gmx.de
> wrote:

> On Mon, 2 Nov 2009, Neil Johnson wrote:
>
> > video4linux-list,
> >
> > I am developing on the OMAP3 system using a micron/aptina mt9p031
> > 5-megapixel imager.  This CMOs imager supports full image capture at 5
> fps
> > (2592x1944 pixels) or you can capture subregions using skipping and
> > binning.  We have proven both capabilities, but would like to be able to
> > capture both VGA sized video and still images without using separate
> > drivers.
> >
> > So far, I have not found any support for capturing large images and video
> > through a single driver interface.  Does such a capability exist within
> > v4l2?  One possible way to solve the problem is to allocate N buffers of
> the
> > full 5-megapixel size (they end up being 10-MB for each buffer since I'm
> > using 16-bits per pixel), and then using a small portion of that for
> video.
> > These is less desirable since when I'm capturing video, I only need
> 640x480
> > size buffers, and I should only need one snapshot buffer at a time (I'm
> not
> > streaming them in, just take a snapshot and go back to live video
> capture).
> > Is there a way to allocate a side-buffer for the 5-megapixel image and
> also
> > allocate "normal" sized buffers for video within the same driver?  Any
> > recommendations on how to accomplish such a thing?  I would think that
> > camera-phones using linux would have run up against this.  Thanks,
>
> I came across the same problem when working on the rj54n1cb0c driver.
> What's even more exciting with that sensor, is that it has separate
> frame-size settings for preview (video) and still capture.


I'm running into this as well.  The MT9P013 has different capture sizes for
video preview (can be 720x480, 1280x720, etc), but the full image capture is
2592x1944.  The biggest problem is that the hardware I'm using (OMAP3530)
has an image signal processing (isp) chain that needs to be properly
configured for the input size and output size.  Right now, the isp gets
initialized when the driver is opened, and then doesn't get touched until
it's closed.  However, the isp needs to be reconfigured if the frame size
changes.  This is getting to be a painful experience.



> So far the
> driver doesn't have any support for those still image frame settings, but
> I was thinking about implenting it using the read V4L method. I.e., you
> would use the mmap method just normally to capture video and use read to
> get still images?

I've considered this, and maybe that's the right way to do it.  The buffer
size for the read would have to be different than the mmap buffers...but
maybe that's doable.  Does "read" use videobuf in most circumstances?  Is it
possible to allocate multiple video buffer queues from a single driver?


> OTOH, with my driver I would need to differentiate
> between S_FMT for video and still. Would we have to use different inputs
> for them?
>
Not sure...maybe that's the way to do it... right now my only option is to
close the device, reopen it with different settings everytime I switch from
video to still image.

>
> Thanks
> Guennadi
> ---
> Guennadi Liakhovetski, Ph.D.
> Freelance Open-Source Software Developer
> http://www.open-technology.de/


Thanks for the feedback.  Hopefully this will be a good learning experience
:)


-- 
Neil Johnson
Systems Engineer
Procerus Technologies
http://www.procerus.com
801-437-0805 (work)
--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: Capturing video and still images using one driver

[Neil Johnson]

Guennadi,

Your suggestions do work well, though my implementation is not ideal.
However, I am now able to switch between high res capture and low res
capture using S_FMT.  I made the switch to using user-ptr buffers in
user space, and now I allocate both large still image buffers and
smaller video buffers when my app starts.  Then, I switch from video
to still image capture (always using streaming, not read()), and then
switch back to video when needed.

Unfortunately, I'm not a huge expert at making my code fit the
video4linux model, so I've basically introduced some hacks that make
it work for me.  I'll try to get my hackish code up to spec so that it
will possibly be useful for others.

Neil

On Tue, Nov 3, 2009 at 3:13 PM, Neil Johnson <realdealneil@gmail.com> wrote:
>
>
> On Tue, Nov 3, 2009 at 3:02 PM, Guennadi Liakhovetski
> <g.liakhovetski@gmx.de> wrote:
>>
>> On Mon, 2 Nov 2009, Neil Johnson wrote:
>>
>> > video4linux-list,
>> >
>> > I am developing on the OMAP3 system using a micron/aptina mt9p031
>> > 5-megapixel imager.  This CMOs imager supports full image capture at 5
>> > fps
>> > (2592x1944 pixels) or you can capture subregions using skipping and
>> > binning.  We have proven both capabilities, but would like to be able to
>> > capture both VGA sized video and still images without using separate
>> > drivers.
>> >
>> > So far, I have not found any support for capturing large images and
>> > video
>> > through a single driver interface.  Does such a capability exist within
>> > v4l2?  One possible way to solve the problem is to allocate N buffers of
>> > the
>> > full 5-megapixel size (they end up being 10-MB for each buffer since I'm
>> > using 16-bits per pixel), and then using a small portion of that for
>> > video.
>> > These is less desirable since when I'm capturing video, I only need
>> > 640x480
>> > size buffers, and I should only need one snapshot buffer at a time (I'm
>> > not
>> > streaming them in, just take a snapshot and go back to live video
>> > capture).
>> > Is there a way to allocate a side-buffer for the 5-megapixel image and
>> > also
>> > allocate "normal" sized buffers for video within the same driver?  Any
>> > recommendations on how to accomplish such a thing?  I would think that
>> > camera-phones using linux would have run up against this.  Thanks,
>>
>> I came across the same problem when working on the rj54n1cb0c driver.
>> What's even more exciting with that sensor, is that it has separate
>> frame-size settings for preview (video) and still capture.
>
> I'm running into this as well.  The MT9P013 has different capture sizes for
> video preview (can be 720x480, 1280x720, etc), but the full image capture is
> 2592x1944.  The biggest problem is that the hardware I'm using (OMAP3530)
> has an image signal processing (isp) chain that needs to be properly
> configured for the input size and output size.  Right now, the isp gets
> initialized when the driver is opened, and then doesn't get touched until
> it's closed.  However, the isp needs to be reconfigured if the frame size
> changes.  This is getting to be a painful experience.
>
>
>>
>> So far the
>> driver doesn't have any support for those still image frame settings, but
>> I was thinking about implenting it using the read V4L method. I.e., you
>> would use the mmap method just normally to capture video and use read to
>> get still images?
>
> I've considered this, and maybe that's the right way to do it.  The buffer
> size for the read would have to be different than the mmap buffers...but
> maybe that's doable.  Does "read" use videobuf in most circumstances?  Is it
> possible to allocate multiple video buffer queues from a single driver?
>
>>
>> OTOH, with my driver I would need to differentiate
>> between S_FMT for video and still. Would we have to use different inputs
>> for them?
>
> Not sure...maybe that's the way to do it... right now my only option is to
> close the device, reopen it with different settings everytime I switch from
> video to still image.
>>
>> Thanks
>> Guennadi
>> ---
>> Guennadi Liakhovetski, Ph.D.
>> Freelance Open-Source Software Developer
>> http://www.open-technology.de/
>
>
> Thanks for the feedback.  Hopefully this will be a good learning experience
> :)
>
>
> --

Re: Capturing video and still images using one driver

[Guennadi Liakhovetski]

(removed the deprecated video ML from CC)

On Wed, 11 Nov 2009, Neil Johnson wrote:

> Guennadi,
> 
> Your suggestions do work well, though my implementation is not ideal.
> However, I am now able to switch between high res capture and low res
> capture using S_FMT.  I made the switch to using user-ptr buffers in
> user space, and now I allocate both large still image buffers and
> smaller video buffers when my app starts.  Then, I switch from video
> to still image capture (always using streaming, not read()), and then
> switch back to video when needed.

You meqn you stop streaming in your application, requeue buffers of 
different size and restart capture?

> Unfortunately, I'm not a huge expert at making my code fit the
> video4linux model, so I've basically introduced some hacks that make
> it work for me.  I'll try to get my hackish code up to spec so that it
> will possibly be useful for others.

Yep, try to make it as good as possible and in any case send it in for 
others to have a look.

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: Capturing video and still images using one driver

[Leon Woestenberg]

Hello Neil,

I found this thread:

On Thu, Nov 12, 2009 at 1:56 AM, Neil Johnson <realdealneil@gmail.com> wrote:
> Unfortunately, I'm not a huge expert at making my code fit the
> video4linux model, so I've basically introduced some hacks that make
> it work for me.  I'll try to get my hackish code up to spec so that it
> will possibly be useful for others.
>
>>> On Mon, 2 Nov 2009, Neil Johnson wrote:
>>> > I am developing on the OMAP3 system using a micron/aptina mt9p031
>>> > 5-megapixel imager.  This CMOs imager supports full image capture at 5
>
I am in the same position (OMAP3 plus MT9P031) and I'm reasonably
familiar with the v4l2 code.

Are you able to share your code so far? My work will be GPL and
submitted upstream.

Regards,
-- 
Leon

Re: Capturing video and still images using one driver

[Robert Jarzmik]

Guennadi Liakhovetski <g.liakhovetski@gmx.de> writes:

> I came across the same problem when working on the rj54n1cb0c driver. 
> What's even more exciting with that sensor, is that it has separate 
> frame-size settings for preview (video) and still capture.

It seems this behaviour is generic across several sensors. As far as I know, the
mt9m111 has 2 modes : low power low resolution, and high power high resolution,
and both are programmable apart (in terms of resolution, zoom, etc ...)

What this makes me think is that a sensor could provide several "contexts" of
use, as :
 - full resolution still image context
 - low resolution still image context
 - full resolution video context
 - low resolution video context

Then, a new/existing v4l2 call would switch the context (perhaps based on buffer
type ?) of the sensor.

Well, that's just some junk I've been thinking over lately.

Cheers.

--
Robert

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: Capturing video and still images using one driver

[Guennadi Liakhovetski]

On Wed, 4 Nov 2009, Robert Jarzmik wrote:

> Guennadi Liakhovetski <g.liakhovetski@gmx.de> writes:
> 
> > I came across the same problem when working on the rj54n1cb0c driver. 
> > What's even more exciting with that sensor, is that it has separate 
> > frame-size settings for preview (video) and still capture.
> 
> It seems this behaviour is generic across several sensors. As far as I know, the
> mt9m111 has 2 modes : low power low resolution, and high power high resolution,
> and both are programmable apart (in terms of resolution, zoom, etc ...)
> 
> What this makes me think is that a sensor could provide several "contexts" of
> use, as :
>  - full resolution still image context
>  - low resolution still image context
>  - full resolution video context
>  - low resolution video context

Why fixed resolutions? Just make it possible to issue S_FMT for video or 
for still imaging... That would work seamlessly with several inputs 
(S_INPUT, S_FMT...).

> Then, a new/existing v4l2 call would switch the context (perhaps based on buffer
> type ?) of the sensor.

...on a second thought, it doesn't seem that smart to me any more to tie 
the streaming vs. still mode distinction to a specific buffer type...

> Well, that's just some junk I've been thinking over lately.

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: Capturing video and still images using one driver

[Robert Jarzmik]

Guennadi Liakhovetski <g.liakhovetski@gmx.de> writes:

>> What this makes me think is that a sensor could provide several "contexts" of
>> use, as :
>>  - full resolution still image context
>>  - low resolution still image context
>>  - full resolution video context
>>  - low resolution video context
>
> Why fixed resolutions? Just make it possible to issue S_FMT for video or 
> for still imaging... That would work seamlessly with several inputs 
> (S_INPUT, S_FMT...).
It's not about _fixed_ resolution. It's rather about power consumption
actually. In mt9m111 sensor, there are 2 modes : A and B. Mode A always eats
tiny amounts of power, but the output resolution is limited to 640x480 IIRC.
Mode B eats more power, but allows full resolution of 1280x1024.

As I saw it, a "context" would allow a range of resolution, a range of clock,
and maybe capability to capture a video stream or not.

What's behind the concept of "context" is : a set of capabilities the hardware
can store internally (in terms of resolution, type of output, power consumption,
etc ...). And of course a "hardware switch" to switch between setups.

>> Then, a new/existing v4l2 call would switch the context (perhaps based on buffer
>> type ?) of the sensor.
>
> ...on a second thought, it doesn't seem that smart to me any more to tie 
> the streaming vs. still mode distinction to a specific buffer type...
I truly don't know. I'll take your word for it.

Cheers.

--
Robert

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: Capturing video and still images using one driver

[Guennadi Liakhovetski]

(forwarding to the new v4l list)

---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

---------- Forwarded message ----------
Date: Thu, 5 Nov 2009 21:37:46 +0100 (CET)
From: Guennadi Liakhovetski <lyakh@axis700.grange>
To: Robert Jarzmik <robert.jarzmik@free.fr>
Cc: Neil Johnson <realdealneil@gmail.com>, video4linux-list@redhat.com
Subject: Re: Capturing video and still images using one driver

On Wed, 4 Nov 2009, Robert Jarzmik wrote:

> Guennadi Liakhovetski <g.liakhovetski@gmx.de> writes:
> 
> > I came across the same problem when working on the rj54n1cb0c driver. 
> > What's even more exciting with that sensor, is that it has separate 
> > frame-size settings for preview (video) and still capture.
> 
> It seems this behaviour is generic across several sensors. As far as I know, the
> mt9m111 has 2 modes : low power low resolution, and high power high resolution,
> and both are programmable apart (in terms of resolution, zoom, etc ...)
> 
> What this makes me think is that a sensor could provide several "contexts" of
> use, as :
>  - full resolution still image context
>  - low resolution still image context
>  - full resolution video context
>  - low resolution video context

Why fixed resolutions? Just make it possible to issue S_FMT for video or 
for still imaging... That would work seamlessly with several inputs 
(S_INPUT, S_FMT...).

> Then, a new/existing v4l2 call would switch the context (perhaps based on buffer
> type ?) of the sensor.

...on a second thought, it doesn't seem that smart to me any more to tie 
the streaming vs. still mode distinction to a specific buffer type...

> Well, that's just some junk I've been thinking over lately.

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/