[RFD] frame-size switching: preview / single-shot use-case

[RFD] frame-size switching: preview / single-shot use-case

[Guennadi Liakhovetski]

Hi

This topic has been slightly discussed several times [1] before, but there 
has been no conclusion, nor I'm aware of any implementation, suitably 
resolving this problem. I've added to CC all involved in earlier 
discussions, that I managed to find.

What seems a typical use-case to me is a system with a vewfinder or a 
display, providing a live preview of the video data from a source, like a 
camera, with a relatively low resolution, and a possibility to take 
high-resolution still photos with a very short delay.

Currently this is pretty difficult to realise, e.g., with soc-camera 
drivers you have to free the videobuf(2) queue, by either closing and 
re-opening the interface, or by issuing an ioctl(VIDIOC_REQBUFS, 
count=0) if your driver is already using videobuf2 and if this is really 
working;), configure with a different resolution and re-allocate 
videobuffers (or use different buffers, allocated per USERPTR). Another 
possibility would be to allocate and use buffers large enough for still 
photos, also for the preview, which would be wasteful, because one can 
well need many more preview than still-shot buffers.

So, it seems to me, we could live with a better solution.

1. We could use separate inputs for different capture modes and support 
per-input videobuf queues. Advantage: no API changes required. 
Disadvantages: confusing, especially, if a driver already exports multiple 
inputs. The driver does not know, whether this mode is required or not, 
always exporting 2 inputs for this purpose doesn't seem like a good idea. 
Eventually, the user might want not 2, but 3 or more of such videobuf 
queues.

2. Use different IO methods, e.g., mmap() for preview and read() for still 
shots. I'm just mentioning this possibility here, because it occurred in 
one of previous threads, but I don't really like it either. What if you 
want to use the same IO method for all? Etc.

3. Not liking either of the above, it seems we need yet a new API for 
this... How about extending VIDIOC_REQBUFS with a videobuf queue index, 
thus using up one of the remaining two 32-bit reserved fields? Then we 
need one more ioctl() like VIDIOC_BUFQ_SELECT to switch from one queue to 
another, after which setting frame format and queuing and dequeuing 
buffers will affect this currently selected queue. We could also keep 
REQBUFS as is and require BUFQ_SELECT to be called before it for any queue 
except the default 0.

Yes, I know, that some video sensors have a double register set for this 
dual-format operation, so, for them it is natural to support two queues, 
and drivers are certainly most welcome to use this feature for, say, the 
first two queues. On other sensors and for any further queues switching 
will have to be done in software.

Ideas? Comments?

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

RE: [RFD] frame-size switching: preview / single-shot use-case

[Karicheri, Muralidharan]

Adding Manjunath Hadli to the list...

Murail

-----Original Message-----
From: Guennadi Liakhovetski [mailto:g.liakhovetski@gmx.de] 
Sent: Tuesday, February 15, 2011 12:34 PM
To: Linux Media Mailing List
Cc: Qing Xu; Hans Verkuil; Neil Johnson; Robert Jarzmik; Uwe Taeubert; Karicheri, Muralidharan; Eino-Ville Talvala
Subject: [RFD] frame-size switching: preview / single-shot use-case

Hi

This topic has been slightly discussed several times [1] before, but there 
has been no conclusion, nor I'm aware of any implementation, suitably 
resolving this problem. I've added to CC all involved in earlier 
discussions, that I managed to find.

What seems a typical use-case to me is a system with a vewfinder or a 
display, providing a live preview of the video data from a source, like a 
camera, with a relatively low resolution, and a possibility to take 
high-resolution still photos with a very short delay.

Currently this is pretty difficult to realise, e.g., with soc-camera 
drivers you have to free the videobuf(2) queue, by either closing and 
re-opening the interface, or by issuing an ioctl(VIDIOC_REQBUFS, 
count=0) if your driver is already using videobuf2 and if this is really 
working;), configure with a different resolution and re-allocate 
videobuffers (or use different buffers, allocated per USERPTR). Another 
possibility would be to allocate and use buffers large enough for still 
photos, also for the preview, which would be wasteful, because one can 
well need many more preview than still-shot buffers.

So, it seems to me, we could live with a better solution.

1. We could use separate inputs for different capture modes and support 
per-input videobuf queues. Advantage: no API changes required. 
Disadvantages: confusing, especially, if a driver already exports multiple 
inputs. The driver does not know, whether this mode is required or not, 
always exporting 2 inputs for this purpose doesn't seem like a good idea. 
Eventually, the user might want not 2, but 3 or more of such videobuf 
queues.

2. Use different IO methods, e.g., mmap() for preview and read() for still 
shots. I'm just mentioning this possibility here, because it occurred in 
one of previous threads, but I don't really like it either. What if you 
want to use the same IO method for all? Etc.

3. Not liking either of the above, it seems we need yet a new API for 
this... How about extending VIDIOC_REQBUFS with a videobuf queue index, 
thus using up one of the remaining two 32-bit reserved fields? Then we 
need one more ioctl() like VIDIOC_BUFQ_SELECT to switch from one queue to 
another, after which setting frame format and queuing and dequeuing 
buffers will affect this currently selected queue. We could also keep 
REQBUFS as is and require BUFQ_SELECT to be called before it for any queue 
except the default 0.

Yes, I know, that some video sensors have a double register set for this 
dual-format operation, so, for them it is natural to support two queues, 
and drivers are certainly most welcome to use this feature for, say, the 
first two queues. On other sensors and for any further queues switching 
will have to be done in software.

Ideas? Comments?

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

RE: [RFD] frame-size switching: preview / single-shot use-case

[Qing Xu]

Hi,

I have a question that why we must check "icf->vb_vidq.bufs[0]" in
s_fmt_vid_cap()? The application mainly calling sequence at switching
fmt could be like this:
streamoff
s_fmt_vid_cap
request_buf
qbuf...qbuf
streamon
qbuf/dqbuf
...
The application should also aware that they are switching the fmt,
so they should release their buffer(per in usrptr method), and re-call
request-buf/qbuf, so based on this assumption, how about we check "bufs[0]"
in request_buf or qbuf according to the new fmt, if we find the buffer
size is not correct, then indicate error in request_buf or qbuf.
However,in s_fmt_vid_cap,
we only need to check whether streaming is on/off, if it is still on,
that means HW resource is not available or IO/buffer is in progress, then
we reject s_fmt. What do you think?

For the idea 3, (what is the difference between idea 1 and 3?)
in our real usage cases of camera(our product is a phone),
we will set many formats, such as:
preview@VGA, photos@5M/3M/QVGA/QCIF..., video@1080p/720p/VGA/QVGA, if we
maintain each queue for each fmt, it seems that there are too many queues,
and, in this way, the application need to be changed, it should quite aware of VIDIOC_BUFQ_SELECT queue-id for each fmt, then it could allocate/release
the required buffer queue by new ioctl. Is my understanding correct?

Thanks!
-Qing

-----Original Message-----
From: Guennadi Liakhovetski [mailto:g.liakhovetski@gmx.de]
Sent: 2011年2月16日 1:34
To: Linux Media Mailing List
Cc: Qing Xu; Hans Verkuil; Neil Johnson; Robert Jarzmik; Uwe Taeubert; Karicheri, Muralidharan; Eino-Ville Talvala
Subject: [RFD] frame-size switching: preview / single-shot use-case

Hi

This topic has been slightly discussed several times [1] before, but there
has been no conclusion, nor I'm aware of any implementation, suitably
resolving this problem. I've added to CC all involved in earlier
discussions, that I managed to find.

What seems a typical use-case to me is a system with a vewfinder or a
display, providing a live preview of the video data from a source, like a
camera, with a relatively low resolution, and a possibility to take
high-resolution still photos with a very short delay.

Currently this is pretty difficult to realise, e.g., with soc-camera
drivers you have to free the videobuf(2) queue, by either closing and
re-opening the interface, or by issuing an ioctl(VIDIOC_REQBUFS,
count=0) if your driver is already using videobuf2 and if this is really
working;), configure with a different resolution and re-allocate
videobuffers (or use different buffers, allocated per USERPTR). Another
possibility would be to allocate and use buffers large enough for still
photos, also for the preview, which would be wasteful, because one can
well need many more preview than still-shot buffers.

So, it seems to me, we could live with a better solution.

1. We could use separate inputs for different capture modes and support
per-input videobuf queues. Advantage: no API changes required.
Disadvantages: confusing, especially, if a driver already exports multiple
inputs. The driver does not know, whether this mode is required or not,
always exporting 2 inputs for this purpose doesn't seem like a good idea.
Eventually, the user might want not 2, but 3 or more of such videobuf
queues.

2. Use different IO methods, e.g., mmap() for preview and read() for still
shots. I'm just mentioning this possibility here, because it occurred in
one of previous threads, but I don't really like it either. What if you
want to use the same IO method for all? Etc.

3. Not liking either of the above, it seems we need yet a new API for
this... How about extending VIDIOC_REQBUFS with a videobuf queue index,
thus using up one of the remaining two 32-bit reserved fields? Then we
need one more ioctl() like VIDIOC_BUFQ_SELECT to switch from one queue to
another, after which setting frame format and queuing and dequeuing
buffers will affect this currently selected queue. We could also keep
REQBUFS as is and require BUFQ_SELECT to be called before it for any queue
except the default 0.

Yes, I know, that some video sensors have a double register set for this
dual-format operation, so, for them it is natural to support two queues,
and drivers are certainly most welcome to use this feature for, say, the
first two queues. On other sensors and for any further queues switching
will have to be done in software.

Ideas? Comments?

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

RE: [RFD] frame-size switching: preview / single-shot use-case

[Guennadi Liakhovetski]

Hi,

On Tue, 15 Feb 2011, Qing Xu wrote:

Please, don't top-post and use a proper quoting.

> Hi,
> 
> I have a question that why we must check "icf->vb_vidq.bufs[0]" in
> s_fmt_vid_cap()? The application mainly calling sequence at switching
> fmt could be like this:
> streamoff
> s_fmt_vid_cap
> request_buf
> qbuf...qbuf
> streamon
> qbuf/dqbuf
> ...
> The application should also aware that they are switching the fmt,
> so they should release their buffer(per in usrptr method), and re-call
> request-buf/qbuf, so based on this assumption, how about we check "bufs[0]"
> in request_buf or qbuf according to the new fmt, if we find the buffer
> size is not correct, then indicate error in request_buf or qbuf.
> However,in s_fmt_vid_cap,
> we only need to check whether streaming is on/off, if it is still on,
> that means HW resource is not available or IO/buffer is in progress, then
> we reject s_fmt. What do you think?

Yes, that restriction is in a way more conservative, than it needs to be. 
I did that to (a) simplify the design and implementation: that way you 
don't have to worry about S_FMT calls in your drivers during a running 
capture, and to (b) verify whether that can be a problem for anyone and to 
hear from them;)

> For the idea 3, (what is the difference between idea 1 and 3?)

(1) proposes to add multiple inputs to the video device, selectable per 
VIDIOC_S_INPUT, (3) doesn't touch inputs and only proposes to add multiple 
videobuf queues to devices, adding a special ioctl().

> in our real usage cases of camera(our product is a phone),
> we will set many formats, such as:
> preview@VGA, photos@5M/3M/QVGA/QCIF..., video@1080p/720p/VGA/QVGA, if we
> maintain each queue for each fmt, it seems that there are too many queues,
> and, in this way, the application need to be changed, it should quite 
> aware of VIDIOC_BUFQ_SELECT queue-id for each fmt, then it could 
> allocate/release
> the required buffer queue by new ioctl. Is my understanding correct?

Of course, with case (3) from my original post, if you want to use those 
multiple queues, you _have_ to modify your application. Same holds for (1) 
and (2) too. As for too many queues - well, you can decide then. Maybe you 
don't need a separate queue for each format, maybe you could use one queue 
for several still-shot queues with similar buffer sizes and buffer 
numbers. Or you don't need them all simultaneously, depending on your 
use-case. If you have an interface, where the user only switches between 
preview and still images, the still image format is usually fixed. To 
select a different format the user has to go to some kind of a 
configuration menu and select a different image format, right? Which would 
be a good time to release the previous still image queue and allocate a 
new one.

Thanks
Guennadi

> 
> Thanks!
> -Qing
> 
> -----Original Message-----
> From: Guennadi Liakhovetski [mailto:g.liakhovetski@gmx.de]
> Sent: 2011Äê2ÔÂ16ÈÕ 1:34
> To: Linux Media Mailing List
> Cc: Qing Xu; Hans Verkuil; Neil Johnson; Robert Jarzmik; Uwe Taeubert; Karicheri, Muralidharan; Eino-Ville Talvala
> Subject: [RFD] frame-size switching: preview / single-shot use-case
> 
> Hi
> 
> This topic has been slightly discussed several times [1] before, but there
> has been no conclusion, nor I'm aware of any implementation, suitably
> resolving this problem. I've added to CC all involved in earlier
> discussions, that I managed to find.
> 
> What seems a typical use-case to me is a system with a vewfinder or a
> display, providing a live preview of the video data from a source, like a
> camera, with a relatively low resolution, and a possibility to take
> high-resolution still photos with a very short delay.
> 
> Currently this is pretty difficult to realise, e.g., with soc-camera
> drivers you have to free the videobuf(2) queue, by either closing and
> re-opening the interface, or by issuing an ioctl(VIDIOC_REQBUFS,
> count=0) if your driver is already using videobuf2 and if this is really
> working;), configure with a different resolution and re-allocate
> videobuffers (or use different buffers, allocated per USERPTR). Another
> possibility would be to allocate and use buffers large enough for still
> photos, also for the preview, which would be wasteful, because one can
> well need many more preview than still-shot buffers.
> 
> So, it seems to me, we could live with a better solution.
> 
> 1. We could use separate inputs for different capture modes and support
> per-input videobuf queues. Advantage: no API changes required.
> Disadvantages: confusing, especially, if a driver already exports multiple
> inputs. The driver does not know, whether this mode is required or not,
> always exporting 2 inputs for this purpose doesn't seem like a good idea.
> Eventually, the user might want not 2, but 3 or more of such videobuf
> queues.
> 
> 2. Use different IO methods, e.g., mmap() for preview and read() for still
> shots. I'm just mentioning this possibility here, because it occurred in
> one of previous threads, but I don't really like it either. What if you
> want to use the same IO method for all? Etc.
> 
> 3. Not liking either of the above, it seems we need yet a new API for
> this... How about extending VIDIOC_REQBUFS with a videobuf queue index,
> thus using up one of the remaining two 32-bit reserved fields? Then we
> need one more ioctl() like VIDIOC_BUFQ_SELECT to switch from one queue to
> another, after which setting frame format and queuing and dequeuing
> buffers will affect this currently selected queue. We could also keep
> REQBUFS as is and require BUFQ_SELECT to be called before it for any queue
> except the default 0.
> 
> Yes, I know, that some video sensors have a double register set for this
> dual-format operation, so, for them it is natural to support two queues,
> and drivers are certainly most welcome to use this feature for, say, the
> first two queues. On other sensors and for any further queues switching
> will have to be done in software.
> 
> Ideas? Comments?
> 
> Thanks
> Guennadi
> ---
> Guennadi Liakhovetski, Ph.D.
> Freelance Open-Source Software Developer
> http://www.open-technology.de/
> 

---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: [RFD] frame-size switching: preview / single-shot use-case

[Hans Verkuil]

Hi Guennadi,

Here is my take on this:

On Wednesday, February 16, 2011 08:42:51 Guennadi Liakhovetski wrote:
> Hi,
> 
> On Tue, 15 Feb 2011, Qing Xu wrote:
> 
> Please, don't top-post and use a proper quoting.
> 
> > Hi,
> > 
> > I have a question that why we must check "icf->vb_vidq.bufs[0]" in
> > s_fmt_vid_cap()? The application mainly calling sequence at switching
> > fmt could be like this:
> > streamoff
> > s_fmt_vid_cap
> > request_buf
> > qbuf...qbuf
> > streamon
> > qbuf/dqbuf
> > ...
> > The application should also aware that they are switching the fmt,
> > so they should release their buffer(per in usrptr method), and re-call
> > request-buf/qbuf, so based on this assumption, how about we check 
"bufs[0]"
> > in request_buf or qbuf according to the new fmt, if we find the buffer
> > size is not correct, then indicate error in request_buf or qbuf.
> > However,in s_fmt_vid_cap,
> > we only need to check whether streaming is on/off, if it is still on,
> > that means HW resource is not available or IO/buffer is in progress, then
> > we reject s_fmt. What do you think?
> 
> Yes, that restriction is in a way more conservative, than it needs to be. 
> I did that to (a) simplify the design and implementation: that way you 
> don't have to worry about S_FMT calls in your drivers during a running 
> capture, and to (b) verify whether that can be a problem for anyone and to 
> hear from them;)
> 
> > For the idea 3, (what is the difference between idea 1 and 3?)
> 
> (1) proposes to add multiple inputs to the video device, selectable per 
> VIDIOC_S_INPUT, (3) doesn't touch inputs and only proposes to add multiple 
> videobuf queues to devices, adding a special ioctl().
> 
> > in our real usage cases of camera(our product is a phone),
> > we will set many formats, such as:
> > preview@VGA, photos@5M/3M/QVGA/QCIF..., video@1080p/720p/VGA/QVGA, if we
> > maintain each queue for each fmt, it seems that there are too many queues,
> > and, in this way, the application need to be changed, it should quite 
> > aware of VIDIOC_BUFQ_SELECT queue-id for each fmt, then it could 
> > allocate/release
> > the required buffer queue by new ioctl. Is my understanding correct?
> 
> Of course, with case (3) from my original post, if you want to use those 
> multiple queues, you _have_ to modify your application. Same holds for (1) 
> and (2) too. As for too many queues - well, you can decide then. Maybe you 
> don't need a separate queue for each format, maybe you could use one queue 
> for several still-shot queues with similar buffer sizes and buffer 
> numbers. Or you don't need them all simultaneously, depending on your 
> use-case. If you have an interface, where the user only switches between 
> preview and still images, the still image format is usually fixed. To 
> select a different format the user has to go to some kind of a 
> configuration menu and select a different image format, right? Which would 
> be a good time to release the previous still image queue and allocate a 
> new one.
> 
> Thanks
> Guennadi
> 
> > 
> > Thanks!
> > -Qing
> > 
> > -----Original Message-----
> > From: Guennadi Liakhovetski [mailto:g.liakhovetski@gmx.de]
> > Sent: 2011Äê2ÔÂ16ÈÕ 1:34
> > To: Linux Media Mailing List
> > Cc: Qing Xu; Hans Verkuil; Neil Johnson; Robert Jarzmik; Uwe Taeubert; 
Karicheri, Muralidharan; Eino-Ville Talvala
> > Subject: [RFD] frame-size switching: preview / single-shot use-case
> > 
> > Hi
> > 
> > This topic has been slightly discussed several times [1] before, but there
> > has been no conclusion, nor I'm aware of any implementation, suitably
> > resolving this problem. I've added to CC all involved in earlier
> > discussions, that I managed to find.
> > 
> > What seems a typical use-case to me is a system with a vewfinder or a
> > display, providing a live preview of the video data from a source, like a
> > camera, with a relatively low resolution, and a possibility to take
> > high-resolution still photos with a very short delay.
> > 
> > Currently this is pretty difficult to realise, e.g., with soc-camera
> > drivers you have to free the videobuf(2) queue, by either closing and
> > re-opening the interface, or by issuing an ioctl(VIDIOC_REQBUFS,
> > count=0) if your driver is already using videobuf2 and if this is really
> > working;), configure with a different resolution and re-allocate
> > videobuffers (or use different buffers, allocated per USERPTR). Another
> > possibility would be to allocate and use buffers large enough for still
> > photos, also for the preview, which would be wasteful, because one can
> > well need many more preview than still-shot buffers.
> > 
> > So, it seems to me, we could live with a better solution.
> > 
> > 1. We could use separate inputs for different capture modes and support
> > per-input videobuf queues. Advantage: no API changes required.
> > Disadvantages: confusing, especially, if a driver already exports multiple
> > inputs. The driver does not know, whether this mode is required or not,
> > always exporting 2 inputs for this purpose doesn't seem like a good idea.
> > Eventually, the user might want not 2, but 3 or more of such videobuf
> > queues.

Don't like this.

> > 
> > 2. Use different IO methods, e.g., mmap() for preview and read() for still
> > shots. I'm just mentioning this possibility here, because it occurred in
> > one of previous threads, but I don't really like it either. What if you
> > want to use the same IO method for all? Etc.

Definitely don't like this.

> > 3. Not liking either of the above, it seems we need yet a new API for
> > this... How about extending VIDIOC_REQBUFS with a videobuf queue index,
> > thus using up one of the remaining two 32-bit reserved fields? Then we
> > need one more ioctl() like VIDIOC_BUFQ_SELECT to switch from one queue to
> > another, after which setting frame format and queuing and dequeuing
> > buffers will affect this currently selected queue. We could also keep
> > REQBUFS as is and require BUFQ_SELECT to be called before it for any queue
> > except the default 0.

What you really are doing here is creating the functionality of two video 
nodes without actually making two nodes.

But from the point of view of the application it makes more sense to actually
have two video nodes. The only difference is that when one starts streaming it
pre-empts the other.

Of course, implementing this correctly will probably require some changes in 
videobuf2 to make it easy.

When it comes to the userspace API I wonder if we shouldn't add a 
VIDIOC_STREAM_FRAMES(u32 framecnt), which streams just 'framecnt' frames. I 
can imagine that it can be hard to stream just a single frame otherwise.

Regards,

	Hans

> > Yes, I know, that some video sensors have a double register set for this
> > dual-format operation, so, for them it is natural to support two queues,
> > and drivers are certainly most welcome to use this feature for, say, the
> > first two queues. On other sensors and for any further queues switching
> > will have to be done in software.
> > 
> > Ideas? Comments?
> > 
> > Thanks
> > Guennadi
> > ---
> > Guennadi Liakhovetski, Ph.D.
> > Freelance Open-Source Software Developer
> > http://www.open-technology.de/
> > 
> 
> ---
> Guennadi Liakhovetski, Ph.D.
> Freelance Open-Source Software Developer
> http://www.open-technology.de/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: [RFD] frame-size switching: preview / single-shot use-case

[Guennadi Liakhovetski]

Hi Hans

Thanks for looking at it!

On Wed, 16 Feb 2011, Hans Verkuil wrote:

> Hi Guennadi,
> 
> Here is my take on this:
> 
> On Wednesday, February 16, 2011 08:42:51 Guennadi Liakhovetski wrote:

[snip]

> > > 3. Not liking either of the above, it seems we need yet a new API for
> > > this... How about extending VIDIOC_REQBUFS with a videobuf queue index,
> > > thus using up one of the remaining two 32-bit reserved fields? Then we
> > > need one more ioctl() like VIDIOC_BUFQ_SELECT to switch from one queue to
> > > another, after which setting frame format and queuing and dequeuing
> > > buffers will affect this currently selected queue. We could also keep
> > > REQBUFS as is and require BUFQ_SELECT to be called before it for any queue
> > > except the default 0.
> 
> What you really are doing here is creating the functionality of two video 
> nodes without actually making two nodes.
> 
> But from the point of view of the application it makes more sense to actually
> have two video nodes. The only difference is that when one starts streaming it
> pre-empts the other.

Well, I don't think I like it all that much... One reason - yes, two 
"independent" video device-nodes, which actually only can stream in turn 
seems somewhat counterintuitive to me, specifically, I don't think there 
is a way to tell the application about this. What if we have more than two 
video devices on the system? Or what if you need more than two video 
queues / formats? Or actually only one? The kernel doesn't know initially 
how many, this is a dynamic resource, not a fixed static interface, IMHO.

> Of course, implementing this correctly will probably require some changes in 
> videobuf2 to make it easy.
> 
> When it comes to the userspace API I wonder if we shouldn't add a 
> VIDIOC_STREAM_FRAMES(u32 framecnt), which streams just 'framecnt' frames. I 
> can imagine that it can be hard to stream just a single frame otherwise.

Don't think you actually need to take just 1 frame for still photography. 
As far as I understand, in many cases they anyway want to take several of 
them to then be able to choose the best one or do some processing. 
Regardless, maybe that ioctl() would make sense, but that's something 
slightly different from what we're trying to figure out here. Yes, it is 
also related to still photography, but it's addressing a different aspect 
of it.

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: [RFD] frame-size switching: preview / single-shot use-case

[Laurent Pinchart]

Hi Guennadi,

On Wednesday 16 February 2011 10:02:18 Guennadi Liakhovetski wrote:
> On Wed, 16 Feb 2011, Hans Verkuil wrote:
> > On Wednesday, February 16, 2011 08:42:51 Guennadi Liakhovetski wrote:
> [snip]

I definitely don't like solutions 1 and 2 either, so I'll only comment on 3.

> > > > 3. Not liking either of the above, it seems we need yet a new API for
> > > > this... How about extending VIDIOC_REQBUFS with a videobuf queue
> > > > index, thus using up one of the remaining two 32-bit reserved
> > > > fields? Then we need one more ioctl() like VIDIOC_BUFQ_SELECT to
> > > > switch from one queue to another, after which setting frame format
> > > > and queuing and dequeuing buffers will affect this currently
> > > > selected queue. We could also keep REQBUFS as is and require
> > > > BUFQ_SELECT to be called before it for any queue except the default
> > > > 0.
> > 
> > What you really are doing here is creating the functionality of two video
> > nodes without actually making two nodes.
> > 
> > But from the point of view of the application it makes more sense to
> > actually have two video nodes. The only difference is that when one
> > starts streaming it pre-empts the other.
> 
> Well, I don't think I like it all that much... One reason - yes, two
> "independent" video device-nodes, which actually only can stream in turn
> seems somewhat counterintuitive to me, specifically, I don't think there
> is a way to tell the application about this. What if we have more than two
> video devices on the system? Or what if you need more than two video
> queues / formats? Or actually only one? The kernel doesn't know initially
> how many, this is a dynamic resource, not a fixed static interface, IMHO.

I agree with this, which is why I don't think two (or more) video nodes would 
be a good solution.

We've hit the exact same issue with the OMAP3 ISP driver. Our current solution 
is to allocate video buffer queues at the file handle level instead of the 
video node level. Applications can open the same video device twice and 
allocate buffers for the viewfinder on one of the instances and for still 
image capture on the other. When switching from viewfinder to still image 
capture, all it needs to do (beside obviously reconfiguring the media 
controller pipeline if required) is to issue VIDIOC_STREAMOFF on the 
viewfinder file handle and VIDIOC_STREAMON on the still capture file handle.

One issue with this approach is that allocating buffers requires knowledge of 
the format and resolution. The driver traditionally computes the buffer size 
from the parameters set by VIDIOC_S_FMT. This would prevent an application 
opening the video node a second time and setting up buffers for still image 
capture a second time while the viewfinder is running, as the VIDIOC_S_FMT on 
the second file handle won't be allowed then.

Changes to the V4L2 spec would be needed to allow this to work properly. One 
possible direction would be to enhance the buffers allocation API to allow 
applications to set the buffer size. I've been thinking about this, and I 
believe this is where the "global buffers pool" API could come into play.

> > Of course, implementing this correctly will probably require some changes
> > in videobuf2 to make it easy.
> > 
> > When it comes to the userspace API I wonder if we shouldn't add a
> > VIDIOC_STREAM_FRAMES(u32 framecnt), which streams just 'framecnt' frames.
> > I can imagine that it can be hard to stream just a single frame
> > otherwise.
> 
> Don't think you actually need to take just 1 frame for still photography.
> As far as I understand, in many cases they anyway want to take several of
> them to then be able to choose the best one or do some processing.
> Regardless, maybe that ioctl() would make sense, but that's something
> slightly different from what we're trying to figure out here. Yes, it is
> also related to still photography, but it's addressing a different aspect
> of it.

-- 
Regards,

Laurent Pinchart

Re: [RFD] frame-size switching: preview / single-shot use-case

[Guennadi Liakhovetski]

Hi Laurent

Thanks for commenting.

On Wed, 16 Feb 2011, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Wednesday 16 February 2011 10:02:18 Guennadi Liakhovetski wrote:
> > On Wed, 16 Feb 2011, Hans Verkuil wrote:
> > > On Wednesday, February 16, 2011 08:42:51 Guennadi Liakhovetski wrote:
> > [snip]
> 
> I definitely don't like solutions 1 and 2 either, so I'll only comment on 3.
> 
> > > > > 3. Not liking either of the above, it seems we need yet a new API for
> > > > > this... How about extending VIDIOC_REQBUFS with a videobuf queue
> > > > > index, thus using up one of the remaining two 32-bit reserved
> > > > > fields? Then we need one more ioctl() like VIDIOC_BUFQ_SELECT to
> > > > > switch from one queue to another, after which setting frame format
> > > > > and queuing and dequeuing buffers will affect this currently
> > > > > selected queue. We could also keep REQBUFS as is and require
> > > > > BUFQ_SELECT to be called before it for any queue except the default
> > > > > 0.
> > > 
> > > What you really are doing here is creating the functionality of two video
> > > nodes without actually making two nodes.
> > > 
> > > But from the point of view of the application it makes more sense to
> > > actually have two video nodes. The only difference is that when one
> > > starts streaming it pre-empts the other.
> > 
> > Well, I don't think I like it all that much... One reason - yes, two
> > "independent" video device-nodes, which actually only can stream in turn
> > seems somewhat counterintuitive to me, specifically, I don't think there
> > is a way to tell the application about this. What if we have more than two
> > video devices on the system? Or what if you need more than two video
> > queues / formats? Or actually only one? The kernel doesn't know initially
> > how many, this is a dynamic resource, not a fixed static interface, IMHO.
> 
> I agree with this, which is why I don't think two (or more) video nodes would 
> be a good solution.
> 
> We've hit the exact same issue with the OMAP3 ISP driver. Our current solution 
> is to allocate video buffer queues at the file handle level instead of the 
> video node level. Applications can open the same video device twice and 
> allocate buffers for the viewfinder on one of the instances and for still 
> image capture on the other. When switching from viewfinder to still image 
> capture, all it needs to do (beside obviously reconfiguring the media 
> controller pipeline if required) is to issue VIDIOC_STREAMOFF on the 
> viewfinder file handle and VIDIOC_STREAMON on the still capture file handle.
> 
> One issue with this approach is that allocating buffers requires knowledge of 
> the format and resolution. The driver traditionally computes the buffer size 
> from the parameters set by VIDIOC_S_FMT. This would prevent an application 
> opening the video node a second time and setting up buffers for still image 
> capture a second time while the viewfinder is running, as the VIDIOC_S_FMT on 
> the second file handle won't be allowed then.
> 
> Changes to the V4L2 spec would be needed to allow this to work properly.

The spec is actually saying about the S_FMT ioctl():

"On success the driver may program the hardware, allocate resources and 
generally prepare for data exchange."

- _may_ program the hardware. So, if we don't do that and instead only 
verify the format and store it for future activation upon a call to 
STREAMON we are not violating the spec, thus, no change is required. OTOH, 
another spec sections "V4L2 close()" says:

"data format parameters, current input or output, control values or other 
properties remain unchanged."

which is usually interpreted as "a sequence open(); ioctl(S_FMT); close(); 
open(); ioctl(STREAMON);" _must_ use the format, set by the call to S_FMT, 
which is not necessarily logical, if we adopt the per-file-descriptor 
format / stream approach.

I think, we have two options:

(1) adopt Laurent's proposal of per-fd contexts, but that would require a 
pretty heave modification of the spec - S_FMT is not kept across close() / 
open() pair.

(2) cleanly separate setting video data format (S_FMT) from specifying the 
allocated buffer size.

Of course, there are further possibilities, like my switching ioctl() 
above, or we could extend the enum v4l2_priority with an extra application 
priority, saying, that this file-descriptor can maintain a separate S_FMT 
/ STREAMON thread, without immediately affecting other descriptors, but 
this seems too obscure to me.

My vote goes for (2) above, which is also what Laurent has mentioned here:

> One 
> possible direction would be to enhance the buffers allocation API to allow 
> applications to set the buffer size. I've been thinking about this, and I 
> believe this is where the "global buffers pool" API could come into play.

I just wouldn't necessarily bind it to "global buffer pools."

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: [RFD] frame-size switching: preview / single-shot use-case

[Guennadi Liakhovetski]

Hi all

Let me try to further elaborate on this topic. So far, I think, the 
following provides the cleanest solution to our quick format-switching / 
multiple video-queue problem:

On Wed, 16 Feb 2011, Guennadi Liakhovetski wrote:

> (2) cleanly separate setting video data format (S_FMT) from specifying the 
> allocated buffer size.

The control flow then would look like:

	struct v4l2_requestbuffers req;

	fd = open();

	/* initialise and configure the first "still-shot" buffer-queue */
	bufq.id = -EINVAL;
	ret = ioctl(fd, VIDIOC_BUFQ_SELECT, &bufq);

	/*
	 * bufq.id now contains a handle to a newly allocated videobuf 
	 * queue, it also becomes the current queue.
	 */

	still_qid = buf.id;

	/*
	 * specify explicitly required buffer size - use one of reserved 
	 * fields
	 */
	req.size = still_x * still_y * bpp;
	ret = ioctl(fd, VIDIOC_REQBUFS, &req);

	for (i = 0; i < req.count; i++) {
		buf.index = i;
		ret = ioctl(fd, VIDIOC_QUERYBUF, &buf);
		start = mmap();
		ret = ioctl(fd, VIDIOC_QBUF, &buf);
	}

	/* Allocate a second "preview" queue */

	bufq.id = -EINVAL;
	ret = ioctl(fd, VIDIOC_BUFQ_SELECT, &bufq);

	preview_qid = bufq.id;

	req.size = preview_x * preview_y * bpp;
	ret = ioctl(fd, VIDIOC_REQBUFS, &req);

	for (i = 0; i < req.count; i++) {
		buf.index = i;
		ret = ioctl(fd, VIDIOC_QUERYBUF, &buf);
		start = mmap();
		ret = ioctl(fd, VIDIOC_QBUF, &buf);
	}

	/* Stay on the preview queue */
	ret = ioctl(fd, VIDIOC_STREAMON, &type);

	for (i = 0; i < n; i++) {
		ret = ioctl(fd, VIDIOC_DQBUF, &buf);
		ret = ioctl(fd, VIDIOC_QBUF, &buf);
	}

	ret = ioctl(fd, VIDIOC_STREAMOFF, &type);

	/* Switch to the still-shot queue */
	bufq.id = still_qid;
	ret = ioctl(fd, VIDIOC_BUFQ_SELECT, &bufq);

	ret = ioctl(fd, VIDIOC_STREAMON, &type);

	for (i = 0; i < n; i++) {
		ret = ioctl(fd, VIDIOC_DQBUF, &buf);
		ret = ioctl(fd, VIDIOC_QBUF, &buf);
	}

	ret = ioctl(fd, VIDIOC_STREAMOFF, &type);

And so on. If the above is _conceptually_ acceptable, we need 1 to 2 new 
ioctl()s and an extension to the VIDIOC_REQBUFS ioctl() to specify the 
buffer size. I wrote "1 or 2 new ioctl()s," because above the new 
VIDIOC_BUFQ_SELECT ioctl() performs two functions: if the handle is 
negative, it allocates a new queue, if it containes a valid (per-device or 
per file-handle?) queue ID, it switches to it. We might also decide to use 
two ioctl()s for these two functions.

Comments?

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: [RFD] frame-size switching: preview / single-shot use-case

[Laurent Pinchart]

Hi Guennadi,

On Thursday 17 February 2011 17:46:15 Guennadi Liakhovetski wrote:
> Hi all
> 
> Let me try to further elaborate on this topic. So far, I think, the
> following provides the cleanest solution to our quick format-switching /
> multiple video-queue problem:
> 
> On Wed, 16 Feb 2011, Guennadi Liakhovetski wrote:
> > (2) cleanly separate setting video data format (S_FMT) from specifying
> > the allocated buffer size.
> 
> The control flow then would look like:
> 
> 	struct v4l2_requestbuffers req;
> 
> 	fd = open();
> 
> 	/* initialise and configure the first "still-shot" buffer-queue */
> 	bufq.id = -EINVAL;
> 	ret = ioctl(fd, VIDIOC_BUFQ_SELECT, &bufq);

If we're going to introduce new ioctls, I think we should do it right from the 
start. Your proposal looks like an interim solution that would be deprecated 
when buffer pools will be implemented. I think we should work directly on 
buffer pools instead.

> 	/*
> 	 * bufq.id now contains a handle to a newly allocated videobuf
> 	 * queue, it also becomes the current queue.
> 	 */
> 
> 	still_qid = buf.id;
> 
> 	/*
> 	 * specify explicitly required buffer size - use one of reserved
> 	 * fields
> 	 */
> 	req.size = still_x * still_y * bpp;
> 	ret = ioctl(fd, VIDIOC_REQBUFS, &req);
> 
> 	for (i = 0; i < req.count; i++) {
> 		buf.index = i;
> 		ret = ioctl(fd, VIDIOC_QUERYBUF, &buf);
> 		start = mmap();
> 		ret = ioctl(fd, VIDIOC_QBUF, &buf);
> 	}
> 
> 	/* Allocate a second "preview" queue */
> 
> 	bufq.id = -EINVAL;
> 	ret = ioctl(fd, VIDIOC_BUFQ_SELECT, &bufq);
> 
> 	preview_qid = bufq.id;
> 
> 	req.size = preview_x * preview_y * bpp;
> 	ret = ioctl(fd, VIDIOC_REQBUFS, &req);
> 
> 	for (i = 0; i < req.count; i++) {
> 		buf.index = i;
> 		ret = ioctl(fd, VIDIOC_QUERYBUF, &buf);
> 		start = mmap();
> 		ret = ioctl(fd, VIDIOC_QBUF, &buf);
> 	}
> 
> 	/* Stay on the preview queue */
> 	ret = ioctl(fd, VIDIOC_STREAMON, &type);
> 
> 	for (i = 0; i < n; i++) {
> 		ret = ioctl(fd, VIDIOC_DQBUF, &buf);
> 		ret = ioctl(fd, VIDIOC_QBUF, &buf);
> 	}
> 
> 	ret = ioctl(fd, VIDIOC_STREAMOFF, &type);
> 
> 	/* Switch to the still-shot queue */
> 	bufq.id = still_qid;
> 	ret = ioctl(fd, VIDIOC_BUFQ_SELECT, &bufq);
> 
> 	ret = ioctl(fd, VIDIOC_STREAMON, &type);
> 
> 	for (i = 0; i < n; i++) {
> 		ret = ioctl(fd, VIDIOC_DQBUF, &buf);
> 		ret = ioctl(fd, VIDIOC_QBUF, &buf);
> 	}
> 
> 	ret = ioctl(fd, VIDIOC_STREAMOFF, &type);
> 
> And so on. If the above is _conceptually_ acceptable, we need 1 to 2 new
> ioctl()s and an extension to the VIDIOC_REQBUFS ioctl() to specify the
> buffer size. I wrote "1 or 2 new ioctl()s," because above the new
> VIDIOC_BUFQ_SELECT ioctl() performs two functions: if the handle is
> negative, it allocates a new queue, if it containes a valid (per-device or
> per file-handle?) queue ID, it switches to it. We might also decide to use
> two ioctl()s for these two functions.

-- 
Regards,

Laurent Pinchart

Re: [RFD] frame-size switching: preview / single-shot use-case

[Mauro Carvalho Chehab]

Em 16-02-2011 08:35, Guennadi Liakhovetski escreveu:
>>>> But from the point of view of the application it makes more sense to
>>>> actually have two video nodes. The only difference is that when one
>>>> starts streaming it pre-empts the other.
>>>
>>> Well, I don't think I like it all that much... One reason - yes, two
>>> "independent" video device-nodes, which actually only can stream in turn
>>> seems somewhat counterintuitive to me, specifically, I don't think there
>>> is a way to tell the application about this. What if we have more than two
>>> video devices on the system? Or what if you need more than two video
>>> queues / formats? Or actually only one? The kernel doesn't know initially
>>> how many, this is a dynamic resource, not a fixed static interface, IMHO.
>>
>> I agree with this, which is why I don't think two (or more) video nodes would 
>> be a good solution.

I agree. Video nodes shouldn't be bind to an specific format. A device with 2
video nodes should be able to retrieve images from two independent video sources.

Unfortunately, ivtv driver was merged with the bad concept of one video node per
different type of formats (partially my fault: I remember I commented about it at 
the time it was submitted, but, as its merge took a long time, and there were 
several other issues that were needed to be solved there, I ended by giving up 
and letting it to come with this API non-compliance, hoping that a fix would
happen at the next kernel release. Unfortunately, it was never fixed).

>> We've hit the exact same issue with the OMAP3 ISP driver. Our current solution 
>> is to allocate video buffer queues at the file handle level instead of the 
>> video node level. Applications can open the same video device twice and 
>> allocate buffers for the viewfinder on one of the instances and for still 
>> image capture on the other. When switching from viewfinder to still image 
>> capture, all it needs to do (beside obviously reconfiguring the media 
>> controller pipeline if required) is to issue VIDIOC_STREAMOFF on the 
>> viewfinder file handle and VIDIOC_STREAMON on the still capture file handle.

This seems to be the proper way. solution (2) of using read()/mmap() is just
an special case of per-file handle stream control, as applications that used 
this approach in the past were, in fact, using two opens, one for read, and 
another for mmap (to be clear, I'm not in favor of 2, I'm just saying that
a per-file handle solution will allow (2) also).

>> One issue with this approach is that allocating buffers requires knowledge of 
>> the format and resolution. The driver traditionally computes the buffer size 
>> from the parameters set by VIDIOC_S_FMT. This would prevent an application 
>> opening the video node a second time and setting up buffers for still image 
>> capture a second time while the viewfinder is running, as the VIDIOC_S_FMT on 
>> the second file handle won't be allowed then.
>>
>> Changes to the V4L2 spec would be needed to allow this to work properly.
> 
> The spec is actually saying about the S_FMT ioctl():
> 
> "On success the driver may program the hardware, allocate resources and 
> generally prepare for data exchange."
> 
> - _may_ program the hardware. So, if we don't do that and instead only 
> verify the format and store it for future activation upon a call to 
> STREAMON we are not violating the spec, thus, no change is required. OTOH, 
> another spec sections "V4L2 close()" says:
> 
> "data format parameters, current input or output, control values or other 
> properties remain unchanged."
> 
> which is usually interpreted as "a sequence open(); ioctl(S_FMT); close(); 
> open(); ioctl(STREAMON);" _must_ use the format, set by the call to S_FMT, 
> which is not necessarily logical, if we adopt the per-file-descriptor 
> format / stream approach.

Every time a "may" appears on a spec, we'll have troubles, as some drivers
will follow the "may" and others won't follow. Changing the behaviour will 
likely cause regressions, whatever direction is taken.

One alternative would be to have a better way to negotiate features than what's
provided by QUERYCAP. If we look for some protocols with a long life, like telnet, 
they don't have a one-way to check/set capabilities. Instead, both parties should
present their capabilities and the client need to negotiate what he wants.

We could do something like:

	ret = ioctl(fd, VIDIOC_QUERYCAP, &cap);
	if  (cap.capabilities & CAN_PER_FD_FMT) {
		setcap.capabilities |= SHOULD_PER_FD_FMT;
		ret = ioctl(fd, VIDIOC_SETCAP, &setcap);
	}

To be sure that the kernel driver will behave fine. Yet, in this particular case,
this would mean that drivers or core will need to handle both per-fd and per-node
S_FMT & friends.


> I think, we have two options:
> 
> (1) adopt Laurent's proposal of per-fd contexts, but that would require a 
> pretty heave modification of the spec - S_FMT is not kept across close() / 
> open() pair.

Whatever done, we'll need to change the specs in a lot of places.

> (2) cleanly separate setting video data format (S_FMT) from specifying the 
> allocated buffer size.

This would break existing applications. Too late for that, except if negotiated
with a "SETCAP" like approach.

There's an additional problem with that: assume that streaming is happening,
and a S_FMT changing the resolution was sent. There's no way to warrant that
the very next frame will have the new resolution. So, a meta-data with the
frame resolution (and format) would be needed.

> Of course, there are further possibilities, like my switching ioctl() 
> above, or we could extend the enum v4l2_priority with an extra application 
> priority, saying, that this file-descriptor can maintain a separate S_FMT 
> / STREAMON thread, without immediately affecting other descriptors, but 
> this seems too obscure to me.
> 
> My vote goes for (2) above, which is also what Laurent has mentioned here:
> 
>> One 
>> possible direction would be to enhance the buffers allocation API to allow 
>> applications to set the buffer size. I've been thinking about this, and I 
>> believe this is where the "global buffers pool" API could come into play.
> 
> I just wouldn't necessarily bind it to "global buffer pools."
> 
> Thanks
> Guennadi
> ---
> Guennadi Liakhovetski, Ph.D.
> Freelance Open-Source Software Developer
> http://www.open-technology.de/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFD] frame-size switching: preview / single-shot use-case

[Guennadi Liakhovetski]

Hi Mauro

Thanks for your comments.

On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote:

> Em 16-02-2011 08:35, Guennadi Liakhovetski escreveu:

[snip]

> > (2) cleanly separate setting video data format (S_FMT) from specifying the 
> > allocated buffer size.
> 
> This would break existing applications. Too late for that, except if negotiated
> with a "SETCAP" like approach.

Sorry, don't see how my proposal from my last mail would change existing 
applications. As long as no explicit buffer-queue management is performed, 
no new queues are allocated, the driver will just implicitly allocate one 
queue and use it. I.e., no change in behaviour.

> There's an additional problem with that: assume that streaming is happening,
> and a S_FMT changing the resolution was sent. There's no way to warrant that
> the very next frame will have the new resolution. So, a meta-data with the
> frame resolution (and format) would be needed.

Sorry, we are not going to allow format changes during a running capture. 
You have to stop streaming, set new formats (possibly switch to another 
queue) and restart streaming.

What am I missing?

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: [RFD] frame-size switching: preview / single-shot use-case

[Mauro Carvalho Chehab]

Em 17-02-2011 17:26, Guennadi Liakhovetski escreveu:
> Hi Mauro
> 
> Thanks for your comments.
> 
> On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote:
> 
>> Em 16-02-2011 08:35, Guennadi Liakhovetski escreveu:
> 
> [snip]
> 
>>> (2) cleanly separate setting video data format (S_FMT) from specifying the 
>>> allocated buffer size.
>>
>> This would break existing applications. Too late for that, except if negotiated
>> with a "SETCAP" like approach.
> 
> Sorry, don't see how my proposal from my last mail would change existing 
> applications. As long as no explicit buffer-queue management is performed, 
> no new queues are allocated, the driver will just implicitly allocate one 
> queue and use it. I.e., no change in behaviour.

Using the same ioctl to explicitly or to implicitly allocating memory depending
on the context would make the API more complicated than it should be.

>> There's an additional problem with that: assume that streaming is happening,
>> and a S_FMT changing the resolution was sent. There's no way to warrant that
>> the very next frame will have the new resolution. So, a meta-data with the
>> frame resolution (and format) would be needed.
> 
> Sorry, we are not going to allow format changes during a running capture. 
> You have to stop streaming, set new formats (possibly switch to another 
> queue) and restart streaming.

> What am I missing?

If you're stopping the stream, the current API will work as-is.

If all of your concerns is about reserving a bigger buffer queue, I think that one
of the reasons for the CMA allocator it for such usage.

Am I missing something?

Cheers,
Mauro

Re: [RFD] frame-size switching: preview / single-shot use-case

[Guennadi Liakhovetski]

On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote:

> Em 17-02-2011 17:26, Guennadi Liakhovetski escreveu:
> > Hi Mauro
> > 
> > Thanks for your comments.
> > 
> > On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote:
> > 
> >> Em 16-02-2011 08:35, Guennadi Liakhovetski escreveu:
> > 
> > [snip]
> > 
> >>> (2) cleanly separate setting video data format (S_FMT) from specifying the 
> >>> allocated buffer size.
> >>
> >> This would break existing applications. Too late for that, except if negotiated
> >> with a "SETCAP" like approach.
> > 
> > Sorry, don't see how my proposal from my last mail would change existing 
> > applications. As long as no explicit buffer-queue management is performed, 
> > no new queues are allocated, the driver will just implicitly allocate one 
> > queue and use it. I.e., no change in behaviour.
> 
> Using the same ioctl to explicitly or to implicitly allocating memory depending
> on the context would make the API more complicated than it should be.

Sorry again, of course, we're adding new functionality, so, it may well 
happen, that the API will become more complicated too. But that was not 
the question. The question was - would it break any existing users? And it 
looks like it wouldn't, at the same time giving new users the required 
additional flexibility and functionality.

> >> There's an additional problem with that: assume that streaming is happening,
> >> and a S_FMT changing the resolution was sent. There's no way to warrant that
> >> the very next frame will have the new resolution. So, a meta-data with the
> >> frame resolution (and format) would be needed.
> > 
> > Sorry, we are not going to allow format changes during a running capture. 
> > You have to stop streaming, set new formats (possibly switch to another 
> > queue) and restart streaming.
> 
> > What am I missing?
> 
> If you're stopping the stream, the current API will work as-is.
> 
> If all of your concerns is about reserving a bigger buffer queue, I think that one
> of the reasons for the CMA allocator it for such usage.

Not just bigger, say, with our preview / still-shot example, we would have 
one queue with a larger number of small buffers for drop-free preview, and 
a small number of larger buffers for still images. Currently you would 
have to allocate a large number of large buffers, which would waste 
memory. Or you would have to reallocate the queue, losing time. AFAICS, 
CMA doesn't manage our memory for us. It only provides an API to reserve 
memory for various uses with various restrictions (alignment, etc.) and 
use different allocators to obtain that memory. So, are you suggesting, 
that with that in place, we would first allocate the preview queue from 
this memory, then free it, when switching to snapshooting, allocate our 
large-buffer queue from the _same_ memory, capture images, free and 
allocate preview queue again? Would that be fast enough?

In fact, it would be kind of smart to reuse the same memory for both 
queues, but if we could do it without re-allocations?...

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: [RFD] frame-size switching: preview / single-shot use-case

[Laurent Pinchart]

Hi Mauro,

On Thursday 17 February 2011 20:04:33 Mauro Carvalho Chehab wrote:
> Em 16-02-2011 08:35, Guennadi Liakhovetski escreveu:
> >>>> But from the point of view of the application it makes more sense to
> >>>> actually have two video nodes. The only difference is that when one
> >>>> starts streaming it pre-empts the other.
> >>> 
> >>> Well, I don't think I like it all that much... One reason - yes, two
> >>> "independent" video device-nodes, which actually only can stream in
> >>> turn seems somewhat counterintuitive to me, specifically, I don't
> >>> think there is a way to tell the application about this. What if we
> >>> have more than two video devices on the system? Or what if you need
> >>> more than two video queues / formats? Or actually only one? The kernel
> >>> doesn't know initially how many, this is a dynamic resource, not a
> >>> fixed static interface, IMHO.
> >> 
> >> I agree with this, which is why I don't think two (or more) video nodes
> >> would be a good solution.
> 
> I agree. Video nodes shouldn't be bind to an specific format. A device with
> 2 video nodes should be able to retrieve images from two independent video
> sources.
> 
> Unfortunately, ivtv driver was merged with the bad concept of one video
> node per different type of formats (partially my fault: I remember I
> commented about it at the time it was submitted, but, as its merge took a
> long time, and there were several other issues that were needed to be
> solved there, I ended by giving up and letting it to come with this API
> non-compliance, hoping that a fix would happen at the next kernel release.
> Unfortunately, it was never fixed).
> 
> >> We've hit the exact same issue with the OMAP3 ISP driver. Our current
> >> solution is to allocate video buffer queues at the file handle level
> >> instead of the video node level. Applications can open the same video
> >> device twice and allocate buffers for the viewfinder on one of the
> >> instances and for still image capture on the other. When switching from
> >> viewfinder to still image capture, all it needs to do (beside obviously
> >> reconfiguring the media controller pipeline if required) is to issue
> >> VIDIOC_STREAMOFF on the viewfinder file handle and VIDIOC_STREAMON on
> >> the still capture file handle.
> 
> This seems to be the proper way. solution (2) of using read()/mmap() is
> just an special case of per-file handle stream control, as applications
> that used this approach in the past were, in fact, using two opens, one
> for read, and another for mmap (to be clear, I'm not in favor of 2, I'm
> just saying that a per-file handle solution will allow (2) also).
> 
> >> One issue with this approach is that allocating buffers requires
> >> knowledge of the format and resolution. The driver traditionally
> >> computes the buffer size from the parameters set by VIDIOC_S_FMT. This
> >> would prevent an application opening the video node a second time and
> >> setting up buffers for still image capture a second time while the
> >> viewfinder is running, as the VIDIOC_S_FMT on the second file handle
> >> won't be allowed then.
> >> 
> >> Changes to the V4L2 spec would be needed to allow this to work properly.
> > 
> > The spec is actually saying about the S_FMT ioctl():
> > 
> > "On success the driver may program the hardware, allocate resources and
> > generally prepare for data exchange."
> > 
> > - _may_ program the hardware. So, if we don't do that and instead only
> > verify the format and store it for future activation upon a call to
> > STREAMON we are not violating the spec, thus, no change is required.
> > OTOH, another spec sections "V4L2 close()" says:
> > 
> > "data format parameters, current input or output, control values or other
> > properties remain unchanged."
> > 
> > which is usually interpreted as "a sequence open(); ioctl(S_FMT);
> > close(); open(); ioctl(STREAMON);" _must_ use the format, set by the
> > call to S_FMT, which is not necessarily logical, if we adopt the
> > per-file-descriptor format / stream approach.
> 
> Every time a "may" appears on a spec, we'll have troubles, as some drivers
> will follow the "may" and others won't follow. Changing the behaviour will
> likely cause regressions, whatever direction is taken.
> 
> One alternative would be to have a better way to negotiate features than
> what's provided by QUERYCAP. If we look for some protocols with a long
> life, like telnet, they don't have a one-way to check/set capabilities.
> Instead, both parties should present their capabilities and the client
> need to negotiate what he wants.
> 
> We could do something like:
> 
> 	ret = ioctl(fd, VIDIOC_QUERYCAP, &cap);
> 	if  (cap.capabilities & CAN_PER_FD_FMT) {
> 		setcap.capabilities |= SHOULD_PER_FD_FMT;
> 		ret = ioctl(fd, VIDIOC_SETCAP, &setcap);
> 	}
> 
> To be sure that the kernel driver will behave fine. Yet, in this particular
> case, this would mean that drivers or core will need to handle both per-fd
> and per-node S_FMT & friends.
> 
> > I think, we have two options:
> > 
> > (1) adopt Laurent's proposal of per-fd contexts, but that would require a
> > pretty heave modification of the spec - S_FMT is not kept across close()
> > / open() pair.
> 
> Whatever done, we'll need to change the specs in a lot of places.
> 
> > (2) cleanly separate setting video data format (S_FMT) from specifying
> > the allocated buffer size.
> 
> This would break existing applications. Too late for that, except if
> negotiated with a "SETCAP" like approach.
> 
> There's an additional problem with that: assume that streaming is
> happening, and a S_FMT changing the resolution was sent. There's no way to
> warrant that the very next frame will have the new resolution. So, a
> meta-data with the frame resolution (and format) would be needed.

It's a completely different topic, but per-frame metadata is something 
interesting. When changing image tuning hardware parameters (such as white 
balance for instance), applications can be very interested in knowing on which 
frame the parameters got applied. Reporting the information to userspace is 
only part of the challenge, being able to pin-point exact frames in the driver 
is also an issue.

> > Of course, there are further possibilities, like my switching ioctl()
> > above, or we could extend the enum v4l2_priority with an extra
> > application priority, saying, that this file-descriptor can maintain a
> > separate S_FMT / STREAMON thread, without immediately affecting other
> > descriptors, but this seems too obscure to me.
> > 
> > My vote goes for (2) above, which is also what Laurent has mentioned here:
> >> One
> >> possible direction would be to enhance the buffers allocation API to
> >> allow applications to set the buffer size. I've been thinking about
> >> this, and I believe this is where the "global buffers pool" API could
> >> come into play.
> > 
> > I just wouldn't necessarily bind it to "global buffer pools."

Maybe not "global", but I would bind it to at least "buffer pools".

-- 
Regards,

Laurent Pinchart

RE: [RFD] frame-size switching: preview / single-shot use-case

[Qing Xu]

Add Kassey and Angela in the loop. Thanks!

-Qing

-----Original Message-----
From: Qing Xu
Sent: 2011年2月16日 10:51
To: 'Guennadi Liakhovetski'; Linux Media Mailing List
Cc: Hans Verkuil; Neil Johnson; Robert Jarzmik; Uwe Taeubert; Karicheri, Muralidharan; Eino-Ville Talvala
Subject: RE: [RFD] frame-size switching: preview / single-shot use-case

Hi,

I have a question that why we must check "icf->vb_vidq.bufs[0]" in
s_fmt_vid_cap()? The application mainly calling sequence at switching
fmt could be like this:
streamoff
s_fmt_vid_cap
request_buf
qbuf...qbuf
streamon
qbuf/dqbuf
...
The application should also aware that they are switching the fmt,
so they should release their buffer(per in usrptr method), and re-call
request-buf/qbuf, so based on this assumption, how about we check "bufs[0]"
in request_buf or qbuf according to the new fmt, if we find the buffer
size is not correct, then indicate error in request_buf or qbuf.
However,in s_fmt_vid_cap,
we only need to check whether streaming is on/off, if it is still on,
that means HW resource is not available or IO/buffer is in progress, then
we reject s_fmt. What do you think?

For the idea 3, (what is the difference between idea 1 and 3?)
in our real usage cases of camera(our product is a phone),
we will set many formats, such as:
preview@VGA, photos@5M/3M/QVGA/QCIF..., video@1080p/720p/VGA/QVGA, if we
maintain each queue for each fmt, it seems that there are too many queues,
and, in this way, the application need to be changed, it should quite aware of VIDIOC_BUFQ_SELECT queue-id for each fmt, then it could allocate/release
the required buffer queue by new ioctl. Is my understanding correct?

Thanks!
-Qing

-----Original Message-----
From: Guennadi Liakhovetski [mailto:g.liakhovetski@gmx.de]
Sent: 2011年2月16日 1:34
To: Linux Media Mailing List
Cc: Qing Xu; Hans Verkuil; Neil Johnson; Robert Jarzmik; Uwe Taeubert; Karicheri, Muralidharan; Eino-Ville Talvala
Subject: [RFD] frame-size switching: preview / single-shot use-case

Hi

This topic has been slightly discussed several times [1] before, but there
has been no conclusion, nor I'm aware of any implementation, suitably
resolving this problem. I've added to CC all involved in earlier
discussions, that I managed to find.

What seems a typical use-case to me is a system with a vewfinder or a
display, providing a live preview of the video data from a source, like a
camera, with a relatively low resolution, and a possibility to take
high-resolution still photos with a very short delay.

Currently this is pretty difficult to realise, e.g., with soc-camera
drivers you have to free the videobuf(2) queue, by either closing and
re-opening the interface, or by issuing an ioctl(VIDIOC_REQBUFS,
count=0) if your driver is already using videobuf2 and if this is really
working;), configure with a different resolution and re-allocate
videobuffers (or use different buffers, allocated per USERPTR). Another
possibility would be to allocate and use buffers large enough for still
photos, also for the preview, which would be wasteful, because one can
well need many more preview than still-shot buffers.

So, it seems to me, we could live with a better solution.

1. We could use separate inputs for different capture modes and support
per-input videobuf queues. Advantage: no API changes required.
Disadvantages: confusing, especially, if a driver already exports multiple
inputs. The driver does not know, whether this mode is required or not,
always exporting 2 inputs for this purpose doesn't seem like a good idea.
Eventually, the user might want not 2, but 3 or more of such videobuf
queues.

2. Use different IO methods, e.g., mmap() for preview and read() for still
shots. I'm just mentioning this possibility here, because it occurred in
one of previous threads, but I don't really like it either. What if you
want to use the same IO method for all? Etc.

3. Not liking either of the above, it seems we need yet a new API for
this... How about extending VIDIOC_REQBUFS with a videobuf queue index,
thus using up one of the remaining two 32-bit reserved fields? Then we
need one more ioctl() like VIDIOC_BUFQ_SELECT to switch from one queue to
another, after which setting frame format and queuing and dequeuing
buffers will affect this currently selected queue. We could also keep
REQBUFS as is and require BUFQ_SELECT to be called before it for any queue
except the default 0.

Yes, I know, that some video sensors have a double register set for this
dual-format operation, so, for them it is natural to support two queues,
and drivers are certainly most welcome to use this feature for, say, the
first two queues. On other sensors and for any further queues switching
will have to be done in software.

Ideas? Comments?

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: [RFD] frame-size switching: preview / single-shot use-case

[Mauro Carvalho Chehab]

Em 15-02-2011 15:33, Guennadi Liakhovetski escreveu:
> Hi
> 
> This topic has been slightly discussed several times [1] before, but there 
> has been no conclusion, nor I'm aware of any implementation, suitably 
> resolving this problem. I've added to CC all involved in earlier 
> discussions, that I managed to find.
> 
> What seems a typical use-case to me is a system with a vewfinder or a 
> display, providing a live preview of the video data from a source, like a 
> camera, with a relatively low resolution, and a possibility to take 
> high-resolution still photos with a very short delay.
> 
> Currently this is pretty difficult to realise, e.g., with soc-camera 
> drivers you have to free the videobuf(2) queue, by either closing and 
> re-opening the interface, or by issuing an ioctl(VIDIOC_REQBUFS, 
> count=0) if your driver is already using videobuf2 and if this is really 
> working;), configure with a different resolution and re-allocate 
> videobuffers (or use different buffers, allocated per USERPTR). Another 
> possibility would be to allocate and use buffers large enough for still 
> photos, also for the preview, which would be wasteful, because one can 
> well need many more preview than still-shot buffers.
> 
> So, it seems to me, we could live with a better solution.
> 
> 1. We could use separate inputs for different capture modes and support 
> per-input videobuf queues. Advantage: no API changes required. 
> Disadvantages: confusing, especially, if a driver already exports multiple 
> inputs. The driver does not know, whether this mode is required or not, 
> always exporting 2 inputs for this purpose doesn't seem like a good idea. 
> Eventually, the user might want not 2, but 3 or more of such videobuf 
> queues.

Very bad. The high res is not a new input.
> 
> 2. Use different IO methods, e.g., mmap() for preview and read() for still 
> shots. I'm just mentioning this possibility here, because it occurred in 
> one of previous threads, but I don't really like it either. What if you 
> want to use the same IO method for all? Etc.

Can be done.
> 
> 3. Not liking either of the above, it seems we need yet a new API for 
> this... How about extending VIDIOC_REQBUFS with a videobuf queue index, 
> thus using up one of the remaining two 32-bit reserved fields? Then we 
> need one more ioctl() like VIDIOC_BUFQ_SELECT to switch from one queue to 
> another, after which setting frame format and queuing and dequeuing 
> buffers will affect this currently selected queue. We could also keep 
> REQBUFS as is and require BUFQ_SELECT to be called before it for any queue 
> except the default 0.
> 
> Yes, I know, that some video sensors have a double register set for this 
> dual-format operation, so, for them it is natural to support two queues, 
> and drivers are certainly most welcome to use this feature for, say, the 
> first two queues. On other sensors and for any further queues switching 
> will have to be done in software.

Seems too hacky to me. 

There's a 4th alternative: open the device twice, and use different settings
on each open. Userspace can stop streaming on one file descriptor and 
start the other one when he wants to take a high-res picture.

Cheers,
Mauro

Re: [RFD] frame-size switching: preview / single-shot use-case

[Michal Nazarewicz]

>>> On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote:
>>>> There's an additional problem with that: assume that streaming is  
>>>> happening, and a S_FMT changing the resolution was sent. There's
>>>> no way to warrant that the very next frame will have the new
>>>> resolution. So, a meta-data with the frame resolution (and format)
>>>> would be needed.

>> Em 17-02-2011 17:26, Guennadi Liakhovetski escreveu:
>>> Sorry, we are not going to allow format changes during a running  
>>> capture. You have to stop streaming, set new formats (possibly
>>> switch to another queue) and restart streaming.
>>>
>>> What am I missing?

> On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote:
>> If you're stopping the stream, the current API will work as-is.
>>
>> If all of your concerns is about reserving a bigger buffer queue, I  
>> think that one of the reasons for the CMA allocator it for such usage.

From: Guennadi Liakhovetski <g.liakhovetski@gmx.de>
> Not just bigger, say, with our preview / still-shot example, we would  
> have one queue with a larger number of small buffers for drop-free
> preview, and a small number of larger buffers for still images.

Ie. waste memory? As in you have both those queues allocated but only
one is used at given time?

> Currently you would have to allocate a large number of large buffers,
> which would waste memory. Or you would have to reallocate the queue,
> losing time.

> AFAICS, CMA doesn't manage our memory for us. It only provides an API
> to reserve memory for various uses with various restrictions (alignment,
> etc.) and use different allocators to obtain that memory.

I'm not sure if I understand you here.  CMA has some API for reserving
memory at boot time but it sure does manage this reserved memory, ie.
when system is running you can allocate chunks of memory from this
reserved block.

Also note, that latest CMA uses only one allocator.

> So, are you suggesting, that with that in place, we would
> first allocate the preview queue from this memory, then free it, when
> switching to snapshooting, allocate our large-buffer queue from the
> _same_ memory, capture images, free and allocate preview queue again?
> Would that be fast enough?

If CMA is considered, the most important thing to note is that CMA may
share memory with page allocator (so that other parts of the system can
use it if CMA-compatible devices are not using it).  When CMA allocates
memory chunk it may potentially need to migrate memory pages which may
take so time (there is room for improvement, but still).

Sharing can be disabled in which case allocation should be quite fast
(the last CMA patchset uses a first-fit bitmap-based gen_allocator API
but O(log n) best-fit algorithm can easily used instead).

To sum things up, if sharing is disabled, CMA should be able to fulfil
your requirements, however it may be undesirable as it wastes space.
If sharing is enabled, on the other hand, the delay may potentially be
noticeable.

> In fact, it would be kind of smart to reuse the same memory for both
> queues, but if we could do it without re-allocations?...

What I would do is allocate a few big buffers and when needed divide them
into smaller chunks (or even allocate one big block and later divide it in
whatever way needed).  I'm not sure if such usage would map well to V4L2
API.

This usage is, as a matter of fact, supported by CMA.  You can allocate
a big block and then run cma_create() on it to create a new CMA context.
Using this context you can allocate a lot of small blocks, then free them
all, to finally allocate few big blocks.

Again, I'm not sure how it maps to V4L2 API.  If you can change formats  
while
retaining V4L device instance's state, this should be doable.

-- 
Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michal "mina86" Nazarewicz    (o o)
ooo +-----<email/xmpp: mnazarewicz@google.com>-----ooO--(_)--Ooo--

Re: [RFD] frame-size switching: preview / single-shot use-case

[Laurent Pinchart]

Hi Michal,

On Friday 18 February 2011 00:09:51 Michal Nazarewicz wrote:
> >>> On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote:
> >>>> There's an additional problem with that: assume that streaming is
> >>>> happening, and a S_FMT changing the resolution was sent. There's
> >>>> no way to warrant that the very next frame will have the new
> >>>> resolution. So, a meta-data with the frame resolution (and format)
> >>>> would be needed.
> >> 
> >> Em 17-02-2011 17:26, Guennadi Liakhovetski escreveu:
> >>> Sorry, we are not going to allow format changes during a running
> >>> capture. You have to stop streaming, set new formats (possibly
> >>> switch to another queue) and restart streaming.
> >>> 
> >>> What am I missing?
> > 
> > On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote:
> >> If you're stopping the stream, the current API will work as-is.
> >> 
> >> If all of your concerns is about reserving a bigger buffer queue, I
> >> think that one of the reasons for the CMA allocator it for such usage.
> 
> From: Guennadi Liakhovetski <g.liakhovetski@gmx.de>
> 
> > Not just bigger, say, with our preview / still-shot example, we would
> > have one queue with a larger number of small buffers for drop-free
> > preview, and a small number of larger buffers for still images.
> 
> Ie. waste memory? As in you have both those queues allocated but only
> one is used at given time?

It's a trade-off between memory and speed. Preallocating still image capture 
buffers will give you better snapshot performances, at the expense of memory.

The basic problems we have here is that taking snapshots is slow with the 
current API if we need to stop capture, free buffers, change the format, 
allocate new buffers (and perform cache management operations) and restart the 
stream. To fix this we're considering a way to preallocate still image capture 
buffers, but I'm open to proposals for other ways to solve the issue :-)

> > Currently you would have to allocate a large number of large buffers,
> > which would waste memory. Or you would have to reallocate the queue,
> > losing time.
> > 
> > AFAICS, CMA doesn't manage our memory for us. It only provides an API
> > to reserve memory for various uses with various restrictions (alignment,
> > etc.) and use different allocators to obtain that memory.
> 
> I'm not sure if I understand you here.  CMA has some API for reserving
> memory at boot time but it sure does manage this reserved memory, ie.
> when system is running you can allocate chunks of memory from this
> reserved block.
> 
> Also note, that latest CMA uses only one allocator.
> 
> > So, are you suggesting, that with that in place, we would
> > first allocate the preview queue from this memory, then free it, when
> > switching to snapshooting, allocate our large-buffer queue from the
> > _same_ memory, capture images, free and allocate preview queue again?
> > Would that be fast enough?
> 
> If CMA is considered, the most important thing to note is that CMA may
> share memory with page allocator (so that other parts of the system can
> use it if CMA-compatible devices are not using it).  When CMA allocates
> memory chunk it may potentially need to migrate memory pages which may
> take so time (there is room for improvement, but still).
> 
> Sharing can be disabled in which case allocation should be quite fast
> (the last CMA patchset uses a first-fit bitmap-based gen_allocator API
> but O(log n) best-fit algorithm can easily used instead).
> 
> To sum things up, if sharing is disabled, CMA should be able to fulfil
> your requirements, however it may be undesirable as it wastes space.
> If sharing is enabled, on the other hand, the delay may potentially be
> noticeable.
> 
> > In fact, it would be kind of smart to reuse the same memory for both
> > queues, but if we could do it without re-allocations?...
> 
> What I would do is allocate a few big buffers and when needed divide them
> into smaller chunks (or even allocate one big block and later divide it in
> whatever way needed).  I'm not sure if such usage would map well to V4L2
> API.
> 
> This usage is, as a matter of fact, supported by CMA.  You can allocate
> a big block and then run cma_create() on it to create a new CMA context.
> Using this context you can allocate a lot of small blocks, then free them
> all, to finally allocate few big blocks.
> 
> Again, I'm not sure how it maps to V4L2 API.  If you can change formats
> while retaining V4L device instance's state, this should be doable.

-- 
Regards,

Laurent Pinchart

Re: [RFD] frame-size switching: preview / single-shot use-case

[Michal Nazarewicz]

On Fri, 18 Feb 2011 11:31:30 +0100, Laurent Pinchart  
<laurent.pinchart@ideasonboard.com> wrote:

> Hi Michal,
>
> On Friday 18 February 2011 00:09:51 Michal Nazarewicz wrote:
>> >>> On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote:
>> >>>> There's an additional problem with that: assume that streaming is
>> >>>> happening, and a S_FMT changing the resolution was sent. There's
>> >>>> no way to warrant that the very next frame will have the new
>> >>>> resolution. So, a meta-data with the frame resolution (and format)
>> >>>> would be needed.
>> >>
>> >> Em 17-02-2011 17:26, Guennadi Liakhovetski escreveu:
>> >>> Sorry, we are not going to allow format changes during a running
>> >>> capture. You have to stop streaming, set new formats (possibly
>> >>> switch to another queue) and restart streaming.
>> >>>
>> >>> What am I missing?
>> >
>> > On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote:
>> >> If you're stopping the stream, the current API will work as-is.
>> >>
>> >> If all of your concerns is about reserving a bigger buffer queue, I
>> >> think that one of the reasons for the CMA allocator it for such  
>> usage.
>>
>> From: Guennadi Liakhovetski <g.liakhovetski@gmx.de>
>>
>> > Not just bigger, say, with our preview / still-shot example, we would
>> > have one queue with a larger number of small buffers for drop-free
>> > preview, and a small number of larger buffers for still images.
>>
>> Ie. waste memory? As in you have both those queues allocated but only
>> one is used at given time?
>
> It's a trade-off between memory and speed. Preallocating still image  
> capture
> buffers will give you better snapshot performances, at the expense of  
> memory.
>
> The basic problems we have here is that taking snapshots is slow with the
> current API if we need to stop capture, free buffers, change the format,
> allocate new buffers (and perform cache management operations) and  
> restart the
> stream. To fix this we're considering a way to preallocate still image  
> capture
> buffers, but I'm open to proposals for other ways to solve the issue :-)

What I'm trying to say is that it would be best if one could configure the
device in such a way that switching between modes would not require the
device to free buffers (even though in user space they could be  
inaccessible).


This is what I have in mind the usage would look like:

1. Open device
		Kernel creates some control structures, the usual stuff.
2. Initialize multi-format (specifying what formats user space will use).
		Kernel calculates amount of memory needed for the most
		demanding format and allocates it.

3. Set format (restricted to one of formats specified in step 2)
		Kernel has memory already allocated, so it only needs to split
		it to buffers needed in given format.
4. Allocate buffers.
		Kernel allocates memory needed for most demanding format
		(calculated in step 2).
		Once memory is allocated, splits it to buffers needed in
		given format.
5. Do the loop... queue, dequeue, all the usual stuff.
		Kernel instructs device to handle buffers, the usual stuff.
6. Free buffers.
		Kernel space destroys the buffers needed for given format
		but DOES NOT free memory.

7. If not done, go to step 3.

8. Finish multi-format mode.
		Kernel actually frees the memory.

9. Close the device.

A V4L device driver could just ignore step 2 and 7 and work in the less  
optimal
mode.


If I understand V4L2 correctly, the API does not allow for step 2 and 8.
In theory, they could be merged with step 1 and 9 respectively, I don't
know id that feasible though.

-- 
Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michal "mina86" Nazarewicz    (o o)
ooo +-----<email/xmpp: mnazarewicz@google.com>-----ooO--(_)--Ooo--

Re: [RFD] frame-size switching: preview / single-shot use-case

[Laurent Pinchart]

Hi Michal,

On Friday 18 February 2011 12:37:24 Michal Nazarewicz wrote:

[snip]

> What I'm trying to say is that it would be best if one could configure the
> device in such a way that switching between modes would not require the
> device to free buffers (even though in user space they could be
> inaccessible).
> 
> 
> This is what I have in mind the usage would look like:
> 
> 1. Open device
> 		Kernel creates some control structures, the usual stuff.
> 2. Initialize multi-format (specifying what formats user space will use).
> 		Kernel calculates amount of memory needed for the most
> 		demanding format and allocates it.

Don't forget that applications can also use USERPTR. We need a low-latency 
solution for that as well.

> 3. Set format (restricted to one of formats specified in step 2)
> 		Kernel has memory already allocated, so it only needs to split
> 		it to buffers needed in given format.
> 4. Allocate buffers.
> 		Kernel allocates memory needed for most demanding format
> 		(calculated in step 2).
> 		Once memory is allocated, splits it to buffers needed in
> 		given format.
> 5. Do the loop... queue, dequeue, all the usual stuff.
> 		Kernel instructs device to handle buffers, the usual stuff.

When buffers are queued cache needs to be cleaned. This is an expensive 
operation, and we need to be able to pre-queue (or at least pre-clean) 
buffers.

> 6. Free buffers.
> 		Kernel space destroys the buffers needed for given format
> 		but DOES NOT free memory.
> 
> 7. If not done, go to step 3.
> 
> 8. Finish multi-format mode.
> 		Kernel actually frees the memory.
> 
> 9. Close the device.
> 
> A V4L device driver could just ignore step 2 and 7 and work in the less
> optimal mode.
> 
> If I understand V4L2 correctly, the API does not allow for step 2 and 8.
> In theory, they could be merged with step 1 and 9 respectively, I don't
> know id that feasible though.

-- 
Regards,

Laurent Pinchart

Re: [RFD] frame-size switching: preview / single-shot use-case

[Michal Nazarewicz]

On Fri, 18 Feb 2011 13:57:25 +0100, Laurent Pinchart  
<laurent.pinchart@ideasonboard.com> wrote:

> Hi Michal,
>
> On Friday 18 February 2011 12:37:24 Michal Nazarewicz wrote:
>
> [snip]
>
>> What I'm trying to say is that it would be best if one could configure  
>> the device in such a way that switching between modes would not require
>> the device to free buffers (even though in user space they could be
>> inaccessible).
>>
>>
>> This is what I have in mind the usage would look like:
>>
>> 1. Open device
>> 		Kernel creates some control structures, the usual stuff.
>> 2. Initialize multi-format (specifying what formats user space will  
>> use).
>> 		Kernel calculates amount of memory needed for the most
>> 		demanding format and allocates it.
>
> Don't forget that applications can also use USERPTR. We need a  
> low-latency solution for that as well.

That would probably work best if user provided one big buffer.  Again,
I don't know how this maps to V4L API.

>
>> 3. Set format (restricted to one of formats specified in step 2)
>> 		Kernel has memory already allocated, so it only needs to split
>> 		it to buffers needed in given format.
>> 4. Allocate buffers.
>> 		Kernel allocates memory needed for most demanding format
>> 		(calculated in step 2).
>> 		Once memory is allocated, splits it to buffers needed in
>> 		given format.
>> 5. Do the loop... queue, dequeue, all the usual stuff.
>> 		Kernel instructs device to handle buffers, the usual stuff.
>
> When buffers are queued cache needs to be cleaned. This is an expensive
> operation, and we need to be able to pre-queue (or at least pre-clean)
> buffers.

Cache operations are always needed, aren't they?  Whatever you do, you
will always have to handle cache coherency (in one way or another) so
there's nothing we can do about it, or is there?

>> 6. Free buffers.
>> 		Kernel space destroys the buffers needed for given format
>> 		but DOES NOT free memory.
>>
>> 7. If not done, go to step 3.
>>
>> 8. Finish multi-format mode.
>> 		Kernel actually frees the memory.
>>
>> 9. Close the device.
>>
>> A V4L device driver could just ignore step 2 and 7 and work in the less
>> optimal mode.
>>
>> If I understand V4L2 correctly, the API does not allow for step 2 and 8.
>> In theory, they could be merged with step 1 and 9 respectively, I don't
>> know id that feasible though.

-- 
Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michal "mina86" Nazarewicz    (o o)
ooo +-----<email/xmpp: mnazarewicz@google.com>-----ooO--(_)--Ooo--

Re: [RFD] frame-size switching: preview / single-shot use-case

[Laurent Pinchart]

Hi Michal,

On Friday 18 February 2011 14:19:44 Michal Nazarewicz wrote:
> On Fri, 18 Feb 2011 13:57:25 +0100, Laurent Pinchart wrote:
> > On Friday 18 February 2011 12:37:24 Michal Nazarewicz wrote:
> > 
> > [snip]
> > 
> >> What I'm trying to say is that it would be best if one could configure
> >> the device in such a way that switching between modes would not require
> >> the device to free buffers (even though in user space they could be
> >> inaccessible).
> >> 
> >> 
> >> This is what I have in mind the usage would look like:
> >> 
> >> 1. Open device
> >> 
> >> 		Kernel creates some control structures, the usual stuff.
> >> 
> >> 2. Initialize multi-format (specifying what formats user space will
> >> use).
> >> 
> >> 		Kernel calculates amount of memory needed for the most
> >> 		demanding format and allocates it.
> > 
> > Don't forget that applications can also use USERPTR. We need a
> > low-latency solution for that as well.
> 
> That would probably work best if user provided one big buffer.  Again,
> I don't know how this maps to V4L API.
> 
> >> 3. Set format (restricted to one of formats specified in step 2)
> >> 
> >> 		Kernel has memory already allocated, so it only needs to split
> >> 		it to buffers needed in given format.
> >> 
> >> 4. Allocate buffers.
> >> 
> >> 		Kernel allocates memory needed for most demanding format
> >> 		(calculated in step 2).
> >> 		Once memory is allocated, splits it to buffers needed in
> >> 		given format.
> >> 
> >> 5. Do the loop... queue, dequeue, all the usual stuff.
> >> 
> >> 		Kernel instructs device to handle buffers, the usual stuff.
> > 
> > When buffers are queued cache needs to be cleaned. This is an expensive
> > operation, and we need to be able to pre-queue (or at least pre-clean)
> > buffers.
> 
> Cache operations are always needed, aren't they?  Whatever you do, you
> will always have to handle cache coherency (in one way or another) so
> there's nothing we can do about it, or is there?

To achieve low latency still image capture, you need to minimize the time 
spent reconfiguring the device from viewfinder to still capture. Cache 
cleaning is always needed, but you can prequeue buffers you can clean the 
cache in advance, avoiding an extra delay when the user presses the still 
image capture button.

> >> 6. Free buffers.
> >> 
> >> 		Kernel space destroys the buffers needed for given format
> >> 		but DOES NOT free memory.
> >> 
> >> 7. If not done, go to step 3.
> >> 
> >> 8. Finish multi-format mode.
> >> 
> >> 		Kernel actually frees the memory.
> >> 
> >> 9. Close the device.
> >> 
> >> A V4L device driver could just ignore step 2 and 7 and work in the less
> >> optimal mode.
> >> 
> >> If I understand V4L2 correctly, the API does not allow for step 2 and 8.
> >> In theory, they could be merged with step 1 and 9 respectively, I don't
> >> know id that feasible though.

-- 
Regards,

Laurent Pinchart

Re: [RFD] frame-size switching: preview / single-shot use-case

[Michal Nazarewicz]

> On Friday 18 February 2011 14:19:44 Michal Nazarewicz wrote:
>> Cache operations are always needed, aren't they?  Whatever you do, you
>> will always have to handle cache coherency (in one way or another) so
>> there's nothing we can do about it, or is there?

On Fri, 18 Feb 2011 14:21:53 +0100, Laurent Pinchart wrote:
> To achieve low latency still image capture, you need to minimize the time
> spent reconfiguring the device from viewfinder to still capture. Cache
> cleaning is always needed, but you can prequeue buffers you can clean the
> cache in advance, avoiding an extra delay when the user presses the still
> image capture button.

If there is enough time to perform those operation while preview is shown  
(ie.
several frames pare second), why would there not be enough time to perform
those operations for still image capture?

If I understand you correctly, what you are describing is a situation where
one has set of buffers for preview and a buffer for still image laying
around waiting to be used, right?

Such scheme will of course work, but I'm just suggesting to think of
a scheme where the unused buffer for still image is reused for preview
frames when preview is shown.

-- 
Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michal "mina86" Nazarewicz    (o o)
ooo +-----<email/xmpp: mnazarewicz@google.com>-----ooO--(_)--Ooo--

Re: [RFD] frame-size switching: preview / single-shot use-case

[Laurent Pinchart]

Hi Michal,

On Friday 18 February 2011 15:05:42 Michal Nazarewicz wrote:
> > On Friday 18 February 2011 14:19:44 Michal Nazarewicz wrote:
> >> Cache operations are always needed, aren't they?  Whatever you do, you
> >> will always have to handle cache coherency (in one way or another) so
> >> there's nothing we can do about it, or is there?
> 
> On Fri, 18 Feb 2011 14:21:53 +0100, Laurent Pinchart wrote:
> > To achieve low latency still image capture, you need to minimize the time
> > spent reconfiguring the device from viewfinder to still capture. Cache
> > cleaning is always needed, but you can prequeue buffers you can clean the
> > cache in advance, avoiding an extra delay when the user presses the still
> > image capture button.
> 
> If there is enough time to perform those operation while preview is shown
> (ie. several frames pare second), why would there not be enough time to
> perform those operations for still image capture?

For still image capture you want to minimize the shot-to-snapshot delay (delay 
between the time when the user presses the shot button and the time when the 
image is captured). Allocating memory for the buffers and prequeuing them 
should thus be done before the user presses the shot button.

> If I understand you correctly, what you are describing is a situation where
> one has set of buffers for preview and a buffer for still image laying
> around waiting to be used, right?

That's correct. That's what we currently do with the OMAP3 ISP.

> Such scheme will of course work, but I'm just suggesting to think of
> a scheme where the unused buffer for still image is reused for preview
> frames when preview is shown.

That will work as well, but it will increase the shot-to-snapshot delay as 
cache management will need to be performed after the user presses the shot 
button.

-- 
Regards,

Laurent Pinchart

Re: [RFD] frame-size switching: preview / single-shot use-case

[Mauro Carvalho Chehab]

Em 18-02-2011 08:31, Laurent Pinchart escreveu:
> It's a trade-off between memory and speed. Preallocating still image capture 
> buffers will give you better snapshot performances, at the expense of memory.
> 
> The basic problems we have here is that taking snapshots is slow with the 
> current API if we need to stop capture, free buffers, change the format, 
> allocate new buffers (and perform cache management operations) and restart the 
> stream. To fix this we're considering a way to preallocate still image capture 
> buffers, but I'm open to proposals for other ways to solve the issue :-)

>From the above operations, considering that CMA is used to reserve a
non-shared memory with enough space for the new buffer size/qtd, I don't
think that the most expensive operation would be to realloc the memory.

The logic to stop/start streaming seems to be the most consuming one, as driver
will need to wait for the current I/O operation to complete, and this can
take hundreds of milisseconds (the duration of one frame).

How much time would CMA need to free and re-allocate the buffers for, let's
say, something in the range of 1-10 MB, on a pre-allocated, non shared memory
space?

Cheers
Mauro

Re: [RFD] frame-size switching: preview / single-shot use-case

[Laurent Pinchart]

Hi Mauro,

On Friday 18 February 2011 13:33:12 Mauro Carvalho Chehab wrote:
> Em 18-02-2011 08:31, Laurent Pinchart escreveu:
> > It's a trade-off between memory and speed. Preallocating still image
> > capture buffers will give you better snapshot performances, at the
> > expense of memory.
> > 
> > The basic problems we have here is that taking snapshots is slow with the
> > current API if we need to stop capture, free buffers, change the format,
> > allocate new buffers (and perform cache management operations) and
> > restart the stream. To fix this we're considering a way to preallocate
> > still image capture buffers, but I'm open to proposals for other ways to
> > solve the issue :-)
> 
> From the above operations, considering that CMA is used to reserve a
> non-shared memory with enough space for the new buffer size/qtd, I don't
> think that the most expensive operation would be to realloc the memory.
> 
> The logic to stop/start streaming seems to be the most consuming one, as
> driver will need to wait for the current I/O operation to complete, and
> this can take hundreds of milisseconds (the duration of one frame).
> 
> How much time would CMA need to free and re-allocate the buffers for, let's
> say, something in the range of 1-10 MB, on a pre-allocated, non shared
> memory space?

CMA won't solve the problem, as not all drivers require continuous memory. The 
OMAP3 ISP has an IOMMU, and two very time-consuming tasks are mapping the 
memory to the IOMMU and cleaning the cache. This is why preallocation *and* 
prequeuing is needed.

-- 
Regards,

Laurent Pinchart

Re: [RFD] frame-size switching: preview / single-shot use-case

[Michal Nazarewicz]

> Em 18-02-2011 08:31, Laurent Pinchart escreveu:
>> It's a trade-off between memory and speed. Preallocating still image  
>> capture buffers will give you better snapshot performances, at the
>> expense of memory.
>>
>> The basic problems we have here is that taking snapshots is slow with  
>> the current API if we need to stop capture, free buffers, change the
>> format, allocate new buffers (and perform cache management operations)
>> and restart the stream. To fix this we're considering a way to
>> preallocate still image capture buffers, but I'm open to proposals for
>> other ways to solve the issue :-)

On Fri, 18 Feb 2011 13:33:12 +0100, Mauro Carvalho Chehab wrote:
> From the above operations, considering that CMA is used to reserve a
> non-shared memory with enough space for the new buffer size/qtd, I don't
> think that the most expensive operation would be to realloc the memory.
>
> The logic to stop/start streaming seems to be the most consuming one, as  
> driver will need to wait for the current I/O operation to complete, and
> this can take hundreds of milisseconds (the duration of one frame).
>
> How much time would CMA need to free and re-allocate the buffers for,  
> let's say, something in the range of 1-10 MB, on a pre-allocated, non
> shared memory space?

Like I said, if memory is shared with page allocator this can potentially
take a lot of time.  If device driver could preallocate some space and
then spit it into buffers when needed, it could be a better solution then
to free chunk and allocate a new one.  CMA was not designed with very low
latency in mind.

-- 
Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michal "mina86" Nazarewicz    (o o)
ooo +-----<email/xmpp: mnazarewicz@google.com>-----ooO--(_)--Ooo--