More videobuf and streaming I/O questions

More videobuf and streaming I/O questions

[Hans Verkuil]

I have a few more questions regarding the streaming I/O API:

1) The spec mentions that the memory field should be set for VIDIOC_DQBUF.
But videobuf doesn't need it and it makes no sense to me either unless it
is for symmetry with VIDIOC_QBUF. Strictly speaking QBUF doesn't need it
either, but it is a good sanity check.

Can I remove the statement in the spec that memory should be set for DQBUF?
The alternative is to add a check against the memory field in videobuf, but
that's rather scary.

2) What to do with REQBUFS when called with a count of 0? Thinking it over I
agree that it shouldn't do an implicit STREAMOFF. But I do think that it is
useful to allow as a simple check whether the I/O method is supported.

So a count of 0 will either return an error if streaming is still in progress
or if the proposed I/O method is not supported, otherwise it will return 0
while leaving count to 0.

This allows one to use REQBUFS to test which I/O methods are supported by
the driver without having the driver allocating any buffers.

This will become more important with embedded systems where almost certainly
additional I/O methods will be introduced (in particular non-contiguous plane
support).

Currently a count of 0 will result in an error in videobuf.

Note that drivers do not generally check for valid values of the memory field
at the moment. So that is another thing we need to improve. But before I start
working on that, I first want to know exactly how REQBUFS should work.

Regards,

	Hans

-- 
Hans Verkuil - video4linux developer - sponsored by TANDBERG

Re: More videobuf and streaming I/O questions

[Laurent Pinchart]

Hi Hans,

On Saturday 20 February 2010 15:00:21 Hans Verkuil wrote:
> I have a few more questions regarding the streaming I/O API:
> 
> 1) The spec mentions that the memory field should be set for VIDIOC_DQBUF.
> But videobuf doesn't need it and it makes no sense to me either unless it
> is for symmetry with VIDIOC_QBUF. Strictly speaking QBUF doesn't need it
> either, but it is a good sanity check.
>
> Can I remove the statement in the spec that memory should be set for DQBUF?
> The alternative is to add a check against the memory field in videobuf, but
> that's rather scary.

In that case I would remove it for QBUF as well, and state that the memory 
field must be ignored by drivers (but should they fill it when returning from 
QBUF/DQBUF ?)

> 2) What to do with REQBUFS when called with a count of 0? Thinking it over
> I agree that it shouldn't do an implicit STREAMOFF. But I do think that it
> is useful to allow as a simple check whether the I/O method is supported.

REQBUFS(0) should also free allocated buffers (if any).

> So a count of 0 will either return an error if streaming is still in
> progress or if the proposed I/O method is not supported, otherwise it will
> return 0 while leaving count to 0.
> 
> This allows one to use REQBUFS to test which I/O methods are supported by
> the driver without having the driver allocating any buffers.
> 
> This will become more important with embedded systems where almost
> certainly additional I/O methods will be introduced (in particular
> non-contiguous plane support).
> 
> Currently a count of 0 will result in an error in videobuf.
> 
> Note that drivers do not generally check for valid values of the memory
> field at the moment. So that is another thing we need to improve. But
> before I start working on that, I first want to know exactly how REQBUFS
> should work.

-- 
Regards,

Laurent Pinchart

Re: More videobuf and streaming I/O questions

[Jean-Francois Moine]

Hi Hans and Laurent,

On Mon, 22 Feb 2010 00:12:18 +0100
Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
> On Saturday 20 February 2010 15:00:21 Hans Verkuil wrote:
> > 1) The spec mentions that the memory field should be set for
> > VIDIOC_DQBUF. But videobuf doesn't need it and it makes no sense to
> > me either unless it is for symmetry with VIDIOC_QBUF. Strictly
> > speaking QBUF doesn't need it either, but it is a good sanity check.
> >
> > Can I remove the statement in the spec that memory should be set
> > for DQBUF? The alternative is to add a check against the memory
> > field in videobuf, but that's rather scary.
> 
> In that case I would remove it for QBUF as well, and state that the
> memory field must be ignored by drivers (but should they fill it when
> returning from QBUF/DQBUF ?)

Agree. It seems that the memory field is not useful at all in the struct
v4l2_buffer if a same process does reqbuf, qbuf, dqbuf and querybuf.


BTW, I had a pending question. The spec says that streamoff 'removes
all buffers from the incoming and outgoing queues' and return to 'the
same state as after calling VIDIOC_REQBUFS'. For output, there is no
problem. For capture, does this mean that the buffers previously queued
by qbuf are implicitly unqueued (i.e. that qbuf must be done again for
all buffers)?

In this case, streamoff does not work with two processes. A first
process is streaming when a second one does streamoff and then
streamon. The first process will stay blocked on polling because no
buffer is queued anymore. It cannot know this fact and the second
process cannot requeue the buffers...

To work correctly, the spec should say that streamoff discards the
content of the filled buffers and that it requeues these buffers as
empty either in the driver's incoming queue (capture) or outgoing queue
(output).

Cheers.

-- 
Ken ar c'hentañ	|	      ** Breizh ha Linux atav! **
Jef		|		http://moinejf.free.fr/

RE: More videobuf and streaming I/O questions

[Pawel Osciak]

Hello,

>On Mon, 22 Feb 2010 00:12:18 +0100
>Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
>> On Saturday 20 February 2010 15:00:21 Hans Verkuil wrote:
>> > 1) The spec mentions that the memory field should be set for
>> > VIDIOC_DQBUF. But videobuf doesn't need it and it makes no sense to
>> > me either unless it is for symmetry with VIDIOC_QBUF. Strictly
>> > speaking QBUF doesn't need it either, but it is a good sanity check.
>> >
>> > Can I remove the statement in the spec that memory should be set
>> > for DQBUF? The alternative is to add a check against the memory
>> > field in videobuf, but that's rather scary.
>>
>> In that case I would remove it for QBUF as well, and state that the
>> memory field must be ignored by drivers (but should they fill it when
>> returning from QBUF/DQBUF ?)
>
>Agree. It seems that the memory field is not useful at all in the struct
>v4l2_buffer if a same process does reqbuf, qbuf, dqbuf and querybuf.

In the current multi-plane buffer proposal, the memory field is required
in querybuf, qbuf and dqbuf for the v4l2-ioctl.c code to be able to
determine whether the planes array should be copied from/to user along
with the buffer.
Just wanted to add another view to the problem, as multiplanes are accepted
yet of course.


As for the REQBUF, I've always thought it'd be nice to be able to ask the
driver for the "recommended" number of buffers that should be used by
issuing a REQBUF with count=0...


Best regards
--
Pawel Osciak
Linux Platform Group
Samsung Poland R&D Center

Re: More videobuf and streaming I/O questions

[Laurent Pinchart]

Hi Jean-François,

On Monday 22 February 2010 10:47:41 Jean-Francois Moine wrote:
> Hi Hans and Laurent,
> 
> On Mon, 22 Feb 2010 00:12:18 +0100
> 
> Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
> > On Saturday 20 February 2010 15:00:21 Hans Verkuil wrote:
> > > 1) The spec mentions that the memory field should be set for
> > > VIDIOC_DQBUF. But videobuf doesn't need it and it makes no sense to
> > > me either unless it is for symmetry with VIDIOC_QBUF. Strictly
> > > speaking QBUF doesn't need it either, but it is a good sanity check.
> > > 
> > > Can I remove the statement in the spec that memory should be set
> > > for DQBUF? The alternative is to add a check against the memory
> > > field in videobuf, but that's rather scary.
> > 
> > In that case I would remove it for QBUF as well, and state that the
> > memory field must be ignored by drivers (but should they fill it when
> > returning from QBUF/DQBUF ?)
> 
> Agree. It seems that the memory field is not useful at all in the struct
> v4l2_buffer if a same process does reqbuf, qbuf, dqbuf and querybuf.
> 
> 
> BTW, I had a pending question. The spec says that streamoff 'removes
> all buffers from the incoming and outgoing queues' and return to 'the
> same state as after calling VIDIOC_REQBUFS'. For output, there is no
> problem. For capture, does this mean that the buffers previously queued
> by qbuf are implicitly unqueued (i.e. that qbuf must be done again for
> all buffers)?

That's correct.

> In this case, streamoff does not work with two processes. A first
> process is streaming when a second one does streamoff and then
> streamon. The first process will stay blocked on polling because no
> buffer is queued anymore. It cannot know this fact and the second
> process cannot requeue the buffers...

I don't think this multiple process use case is valid. The V4L2 streaming API 
wasn't designed to be used in a multi-thread or multi-process context in the 
first place.

> To work correctly, the spec should say that streamoff discards the
> content of the filled buffers and that it requeues these buffers as
> empty either in the driver's incoming queue (capture) or outgoing queue
> (output).

I don't agree. If we did that, buffers couldn't be released after a STREAMOFF. 
Queued buffers belong to the driver, so to free the buffers applications would 
have to call VIDIOC_STREAMOFF and then dequeue all buffers. That's not pretty.

-- 
Regards,

Laurent Pinchart

Re: More videobuf and streaming I/O questions

[Laurent Pinchart]

Hi Pavel,

On Tuesday 23 February 2010 08:41:49 Pawel Osciak wrote:
> >On Mon, 22 Feb 2010 00:12:18 +0100
> >Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
> >> On Saturday 20 February 2010 15:00:21 Hans Verkuil wrote:
> >> > 1) The spec mentions that the memory field should be set for
> >> > VIDIOC_DQBUF. But videobuf doesn't need it and it makes no sense to
> >> > me either unless it is for symmetry with VIDIOC_QBUF. Strictly
> >> > speaking QBUF doesn't need it either, but it is a good sanity check.
> >> > 
> >> > Can I remove the statement in the spec that memory should be set
> >> > for DQBUF? The alternative is to add a check against the memory
> >> > field in videobuf, but that's rather scary.
> >> 
> >> In that case I would remove it for QBUF as well, and state that the
> >> memory field must be ignored by drivers (but should they fill it when
> >> returning from QBUF/DQBUF ?)
> >
> >Agree. It seems that the memory field is not useful at all in the struct
> >v4l2_buffer if a same process does reqbuf, qbuf, dqbuf and querybuf.
> 
> In the current multi-plane buffer proposal, the memory field is required
> in querybuf, qbuf and dqbuf for the v4l2-ioctl.c code to be able to
> determine whether the planes array should be copied from/to user along
> with the buffer.
> Just wanted to add another view to the problem, as multiplanes are accepted
> yet of course.

Thanks for the reminder. I'm not against keeping the requirement for 
applications to set the memory field in the v4l2_buffer structure. QBUF and 
DQBUF should behave the same way though, they should both require the field to 
be set or not require it at all.

> As for the REQBUF, I've always thought it'd be nice to be able to ask the
> driver for the "recommended" number of buffers that should be used by
> issuing a REQBUF with count=0...

How would the driver come up with the number of recommended buffers ?

-- 
Regards,

Laurent Pinchart

RE: More videobuf and streaming I/O questions

[Pawel Osciak]

>On Tuesday 23 February 2010 08:41:49 Pawel Osciak wrote:
>> >On Mon, 22 Feb 2010 00:12:18 +0100
>> >Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
>> As for the REQBUF, I've always thought it'd be nice to be able to ask the
>> driver for the "recommended" number of buffers that should be used by
>> issuing a REQBUF with count=0...
>
>How would the driver come up with the number of recommended buffers ?

>From the top of my head: when encoding a video stream, a codec driver could
decide on the minimum number of input frames required (including reference
frames, etc.).

Or maybe I am missing something, what is your opinion on that?


Best regards
--
Pawel Osciak
Linux Platform Group
Samsung Poland R&D Center

Re: More videobuf and streaming I/O questions

[Mauro Carvalho Chehab]

Pawel Osciak wrote:
>> On Tuesday 23 February 2010 08:41:49 Pawel Osciak wrote:
>>>> On Mon, 22 Feb 2010 00:12:18 +0100
>>>> Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
>>> As for the REQBUF, I've always thought it'd be nice to be able to ask the
>>> driver for the "recommended" number of buffers that should be used by
>>> issuing a REQBUF with count=0...
>> How would the driver come up with the number of recommended buffers ?
> 
> From the top of my head: when encoding a video stream, a codec driver could
> decide on the minimum number of input frames required (including reference
> frames, etc.).
> 
> Or maybe I am missing something, what is your opinion on that?

There are some cases where this feature could be useful. For example, there are
some devices used for surveillance that have one decoder connected to several
inputs. For example, several bttv boards have one bt848 chip for each 8 inputs.
Each input is connected to one camera. The minimum recommended number of buffers
is 16 (2 per each input). This is poorly documented, on some wikis for some of
the boards with such usage.

That's said, there's currently a few missing features for surveillance: the user
software need to manually switch from one input to another, and the video buffer
metadata doesn't indicate the input. 

The better would be to provide a way to let the driver to switch to the next camera 
just after the reception of a new buffer (generally at the IRQ time), instead of 
letting the userspace software to do it at the DQBUF.

-- 

Cheers,
Mauro

Re: More videobuf and streaming I/O questions

[Muralidharan Karicheri]

On Fri, Feb 26, 2010 at 7:04 AM, Mauro Carvalho Chehab
<mchehab@infradead.org> wrote:
> Pawel Osciak wrote:
>>> On Tuesday 23 February 2010 08:41:49 Pawel Osciak wrote:
>>>>> On Mon, 22 Feb 2010 00:12:18 +0100
>>>>> Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
>>>> As for the REQBUF, I've always thought it'd be nice to be able to ask the
>>>> driver for the "recommended" number of buffers that should be used by
>>>> issuing a REQBUF with count=0...
>>> How would the driver come up with the number of recommended buffers ?
>>
>> From the top of my head: when encoding a video stream, a codec driver could
>> decide on the minimum number of input frames required (including reference
>> frames, etc.).
>>
>> Or maybe I am missing something, what is your opinion on that?
>
> There are some cases where this feature could be useful. For example, there are
> some devices used for surveillance that have one decoder connected to several
> inputs. For example, several bttv boards have one bt848 chip for each 8 inputs.
> Each input is connected to one camera. The minimum recommended number of buffers
> is 16 (2 per each input). This is poorly documented, on some wikis for some of
> the boards with such usage.
>
> That's said, there's currently a few missing features for surveillance: the user
> software need to manually switch from one input to another, and the video buffer
> metadata doesn't indicate the input.
>
> The better would be to provide a way to let the driver to switch to the next camera
> just after the reception of a new buffer (generally at the IRQ time), instead of
> letting the userspace software to do it at the DQBUF.
>
This is an interesting use case and I would like to know some details
on this use case.
When you say application manually switch the input, Is it implementing
some kind of
input multiplexing during the session (open, stream on - stream off,
close) ? We have
encountered a similar use case and I was wondering how this can be implemented
in v4l2 driver. In my understanding, a v4l2 device is not allowed to
switch input while
streaming. Does it require 2 buffers per input because every frame
period, you have multiple
frames to queue from the different inputs? Usually a minimum of 3
buffers are typically
required in a SoC case to do streaming. Could you share the details if possible?

Murali
> --
>
> Cheers,
> Mauro
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Murali Karicheri
mkaricheri@gmail.com

Re: More videobuf and streaming I/O questions

[Laurent Pinchart]

Hi Pawel and Mauro,

On Friday 26 February 2010 13:04:54 Mauro Carvalho Chehab wrote:
> Pawel Osciak wrote:
> >> On Tuesday 23 February 2010 08:41:49 Pawel Osciak wrote:
> >>>> On Mon, 22 Feb 2010 00:12:18 +0100
> >>> 
> >>>> Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
> >>> As for the REQBUF, I've always thought it'd be nice to be able to ask
> >>> the driver for the "recommended" number of buffers that should be used
> >>> by issuing a REQBUF with count=0...
> >> 
> >> How would the driver come up with the number of recommended buffers ?
> > 
> > From the top of my head: when encoding a video stream, a codec driver
> > could decide on the minimum number of input frames required (including
> > reference frames, etc.).

Drivers can always raise or lower the number of buffers passed as the 
VIDIOC_REQBUFS argument, so we already have a way to handle hardware 
requirements there.

If we really need a way to tell the driver "please decide on the number of 
buffers for me", we could use a flag/magic value for the buffer count instead 
of using 0. The V4L2 specification clearly states that a count of 0 frees the 
buffers, and several applications rely on that feature.

> > Or maybe I am missing something, what is your opinion on that?
> 
> There are some cases where this feature could be useful. For example, there
> are some devices used for surveillance that have one decoder connected to
> several inputs. For example, several bttv boards have one bt848 chip for
> each 8 inputs. Each input is connected to one camera. The minimum
> recommended number of buffers is 16 (2 per each input).

Why two per input ? There's a single video stream, buffers are not queued 
separately for each input.

Beside, even if the number of recommended buffers was 2 per input, I would 
expect applications to know about that. If an application decides to open a 
single video node and call VIDIOC_S_INPUT during streaming (or configure the 
driver to do it automatically at IRQ time, which is conceptually similar), the 
application should be able to compute the required number of buffers.

> This is poorly documented, on some wikis for some of the boards with such
> usage.
> 
> That's said, there's currently a few missing features for surveillance: the
> user software need to manually switch from one input to another, and the
> video buffer metadata doesn't indicate the input.

There's actually an input field in v4l2_buffer. As far as I know it's only 
used by an out-of-tree, closed source driver that nobody is using anymore (I'm 
the one who requested a reserved field to be turned into the input field back 
then). Now that I'm a bit more knowledgeable about V4L2 and Linux in general, 
I don't think that's the best way to pass metadata around. The v4l2_buffer 
structure won't be able to hold all metadata we need in the future.

> The better would be to provide a way to let the driver to switch to the
> next camera just after the reception of a new buffer (generally at the IRQ
> time), instead of letting the userspace software to do it at the DQBUF.

That would be an improvement, but even then it might be too late. The only way 
to handle analog input switching reliably is to synchronize the input switch 
to the analog signal, and that must be done by the hardware. That kind of 
feature is not commonly found in cheap bttv boards.

-- 
Regards,

Laurent Pinchart

Re: More videobuf and streaming I/O questions

[Laurent Pinchart]

Hi,

On Friday 26 February 2010 14:05:30 Muralidharan Karicheri wrote:
> On Fri, Feb 26, 2010 at 7:04 AM, Mauro Carvalho Chehab wrote:
> > Pawel Osciak wrote:
> >>> On Tuesday 23 February 2010 08:41:49 Pawel Osciak wrote:
> >>>>> On Mon, 22 Feb 2010 00:12:18 +0100
> >>>> 
> >>>>> Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
> >>>> As for the REQBUF, I've always thought it'd be nice to be able to ask
> >>>> the driver for the "recommended" number of buffers that should be
> >>>> used by issuing a REQBUF with count=0...
> >>> 
> >>> How would the driver come up with the number of recommended buffers ?
> >> 
> >> From the top of my head: when encoding a video stream, a codec driver
> >> could decide on the minimum number of input frames required (including
> >> reference frames, etc.).
> >> 
> >> Or maybe I am missing something, what is your opinion on that?
> > 
> > There are some cases where this feature could be useful. For example,
> > there are some devices used for surveillance that have one decoder
> > connected to several inputs. For example, several bttv boards have one
> > bt848 chip for each 8 inputs. Each input is connected to one camera. The
> > minimum recommended number of buffers is 16 (2 per each input). This is
> > poorly documented, on some wikis for some of the boards with such usage.
> > 
> > That's said, there's currently a few missing features for surveillance:
> > the user software need to manually switch from one input to another, and
> > the video buffer metadata doesn't indicate the input.
> > 
> > The better would be to provide a way to let the driver to switch to the
> > next camera just after the reception of a new buffer (generally at the
> > IRQ time), instead of letting the userspace software to do it at the
> > DQBUF.
> 
> This is an interesting use case and I would like to know some details
> on this use case.
> When you say application manually switch the input, Is it implementing some
> kind of input multiplexing during the session (open, stream on - stream off,
> close) ?

No, applications just call VIDIOC_S_INPUT while the stream is active.

> We have encountered a similar use case and I was wondering how this can be
> implemented in v4l2 driver. In my understanding, a v4l2 device is not
> allowed to switch input while streaming.

Switching input while streaming is allowed (if I'm not mistaken), as long as 
the format and size don't change (not sure about lowering the size, that needs 
to be double-checked).

This being said, VIDIOC_S_INPUT was designed to switch analog sources. I'm not 
sure it would be the best would to multiplex streams from two digital sensors 
for instance. Even for analog signals, using the ioctl from userspace usually 
results in at least one corrupt frame if you don't synchronize the switching 
to the analog signals, which requires hardware support.

Can you give us more details about the use case you're thinking about ?

> Does it require 2 buffers per input because every frame period, you have
> multiple frames to queue from the different inputs?

In this case there's a single video stream, so I don't think it would require 
2 buffers per input.

> Usually a minimum of 3 buffers are typically required in a SoC case to do
> streaming. Could you share the details if possible?

-- 
Regards,

Laurent Pinchart

RE: More videobuf and streaming I/O questions

[Karicheri, Muralidharan]

>This being said, VIDIOC_S_INPUT was designed to switch analog sources. I'm
>not
>sure it would be the best would to multiplex streams from two digital
>sensors
>for instance. Even for analog signals, using the ioctl from userspace
>usually
>results in at least one corrupt frame if you don't synchronize the
>switching
>to the analog signals, which requires hardware support.

>
>Can you give us more details about the use case you're thinking about ?


This is for supporting interfacing of TVP5148 to DM365 that can mux from two sources. There is an implementation of this in our internal release. Just exploring how this can be ported to upstream. Not done any work yet on this
from my side.

>
>> Does it require 2 buffers per input because every frame period, you have
>> multiple frames to queue from the different inputs?
>
>In this case there's a single video stream, so I don't think it would
>require
>2 buffers per input.
>
>> Usually a minimum of 3 buffers are typically required in a SoC case to do
>> streaming. Could you share the details if possible?
>
>--
>Regards,
>
>Laurent Pinchart
>--
>To unsubscribe from this list: send the line "unsubscribe linux-media" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html