[Query] VIDIOC_QBUF and VIDIOC_STREAMON order

[Query] VIDIOC_QBUF and VIDIOC_STREAMON order

[Sylwester Nawrocki]

Hello,

As far as I know V4L2 applications are allowed to call VIDIOC_STREAMON before
queuing buffers with VIDIOC_QBUF.

This leads to situation that a H/W is attempted to be enabled by the driver
when it has no any buffer ownership. 

Effectively actual activation of the data pipeline has to be deferred
until first buffer arrived in the driver. Which makes it difficult 
to signal any errors to user during enabling the data pipeline.

Is this allowed to force applications to queue some buffers before calling
STREAMON, by returning an error in vidioc_streamon from the driver, when 
no buffers have been queued at this time?

I suppose this could render some applications to stop working if this kind
of restriction is applied e.g. in camera capture driver.

What the applications really expect?

With the above I refer mostly to a snapshot mode where we have to be careful
not to lose any frame, as there could be only one..


Please share you opinions.


Regards,
-- 
Sylwester Nawrocki
Samsung Poland R&D Center

Re: [Query] VIDIOC_QBUF and VIDIOC_STREAMON order

[Subash Patel]

VIDIOC_STREAMON expects buffers to be queued before hardware part of
image/video pipe is enabled. From my experience of V4L2 user space, I
have always QBUFfed before invoking the STREAMON. Below is the API
specification which also speaks something same:

http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-streamon.html

I think its better to return EINVAL if there are no queued buffers
when VIDIOC_STREAMON is invoked.

Regards,
Subash

On Mon, Mar 14, 2011 at 3:44 PM, Sylwester Nawrocki
<s.nawrocki@samsung.com> wrote:
> Hello,
>
> As far as I know V4L2 applications are allowed to call VIDIOC_STREAMON before
> queuing buffers with VIDIOC_QBUF.
>
> This leads to situation that a H/W is attempted to be enabled by the driver
> when it has no any buffer ownership.
>
> Effectively actual activation of the data pipeline has to be deferred
> until first buffer arrived in the driver. Which makes it difficult
> to signal any errors to user during enabling the data pipeline.
>
> Is this allowed to force applications to queue some buffers before calling
> STREAMON, by returning an error in vidioc_streamon from the driver, when
> no buffers have been queued at this time?
>
> I suppose this could render some applications to stop working if this kind
> of restriction is applied e.g. in camera capture driver.
>
> What the applications really expect?
>
> With the above I refer mostly to a snapshot mode where we have to be careful
> not to lose any frame, as there could be only one..
>
>
> Please share you opinions.
>
>
> Regards,
> --
> Sylwester Nawrocki
> Samsung Poland R&D Center
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [Query] VIDIOC_QBUF and VIDIOC_STREAMON order

[Pawel Osciak]

Hi,

On Mon, Mar 14, 2011 at 03:49, Subash Patel <subashrp@gmail.com> wrote:
> VIDIOC_STREAMON expects buffers to be queued before hardware part of
> image/video pipe is enabled. From my experience of V4L2 user space, I
> have always QBUFfed before invoking the STREAMON. Below is the API
> specification which also speaks something same:
>

Not exactly. It says that the API requires buffers to be queued for
output devices. It does not require any buffers to be queued for input
devices. Sylwester is right here.

This feature of not having to queue input buffers before STREAMON
introduces problems to driver implementations and I am personally not
a big fan of it either. But I'm seeing some additional problems here.
Suppose we forced QBUF to be done before STREAMON. This would work,
but what happens next? What should happen when we want to DQBUF the
last buffer? If the device couldn't start without any buffers queued,
can it continue streaming with all of them dequeued? I would guess
not. So we'd either have to deny DQBUF of the last buffer (which for
me personally is almost unacceptable) or have the last DQBUF
automatically cause a STREAMOFF. So, for the latter, should
applications, after they get all the data they wanted, be made to
always have one more buffer queued as a "throwaway" buffer? This is
probably the only reasonable solution here, but the applications would
have to keep count of their queued buffers and be aware of this.
Also, there might still be situations where being able to STREAMON
without buffers queued would be beneficial. For example, enabling the
device might be a slow/expensive operation and we might prefer to keep
it running even if we don't want any data at the moment. Even for
faster devices, being able to keep them on and periodically take a
snapshot would be faster without having to call STREAMON anyway...

-- 
Best regards,
Pawel Osciak

Re: [Query] VIDIOC_QBUF and VIDIOC_STREAMON order

[Subash Patel]

> Also, there might still be situations where being able to STREAMON
> without buffers queued would be beneficial. For example, enabling the
> device might be a slow/expensive operation and we might prefer to keep
> it running even if we don't want any data at the moment. Even for
> faster devices, being able to keep them on and periodically take a
> snapshot would be faster without having to call STREAMON anyway...

If we are speaking of devices like camera for phone/tabs, then this
will have significant impact on the power consumption.

> Suppose we forced QBUF to be done before STREAMON. This would work,
> but what happens next? What should happen when we want to DQBUF the
> last buffer? If the device couldn't start without any buffers queued,
> can it continue streaming with all of them dequeued? I would guess
> not. So we'd either have to deny DQBUF of the last buffer (which for
> me personally is almost unacceptable) or have the last DQBUF
> automatically cause a STREAMOFF.

I think it depends on hardware's behavior on how it behaves without
any buffers queued. Some hardwares do notify the overflow state
(interrupting for each frame recieved) if there was no buffer queued.
This will ensure even the last frame is DQUEUEd.

Considering the scenario like preview/capture @ 30fps, if there is one
frame loss too, it is acceptable. But that doesnt hold good for
high-speed image capture.

Regards,
Subash

On Tue, Mar 15, 2011 at 8:51 AM, Pawel Osciak <pawel@osciak.com> wrote:
> Hi,
>
> On Mon, Mar 14, 2011 at 03:49, Subash Patel <subashrp@gmail.com> wrote:
>> VIDIOC_STREAMON expects buffers to be queued before hardware part of
>> image/video pipe is enabled. From my experience of V4L2 user space, I
>> have always QBUFfed before invoking the STREAMON. Below is the API
>> specification which also speaks something same:
>>
>
> Not exactly. It says that the API requires buffers to be queued for
> output devices. It does not require any buffers to be queued for input
> devices. Sylwester is right here.
>
> This feature of not having to queue input buffers before STREAMON
> introduces problems to driver implementations and I am personally not
> a big fan of it either. But I'm seeing some additional problems here.
> Suppose we forced QBUF to be done before STREAMON. This would work,
> but what happens next? What should happen when we want to DQBUF the
> last buffer? If the device couldn't start without any buffers queued,
> can it continue streaming with all of them dequeued? I would guess
> not. So we'd either have to deny DQBUF of the last buffer (which for
> me personally is almost unacceptable) or have the last DQBUF
> automatically cause a STREAMOFF. So, for the latter, should
> applications, after they get all the data they wanted, be made to
> always have one more buffer queued as a "throwaway" buffer? This is
> probably the only reasonable solution here, but the applications would
> have to keep count of their queued buffers and be aware of this.
> Also, there might still be situations where being able to STREAMON
> without buffers queued would be beneficial. For example, enabling the
> device might be a slow/expensive operation and we might prefer to keep
> it running even if we don't want any data at the moment. Even for
> faster devices, being able to keep them on and periodically take a
> snapshot would be faster without having to call STREAMON anyway...
>
> --
> Best regards,
> Pawel Osciak
>

Re: [Query] VIDIOC_QBUF and VIDIOC_STREAMON order

[Hans Verkuil]

On Tuesday, March 15, 2011 04:21:05 Pawel Osciak wrote:
> Hi,
> 
> On Mon, Mar 14, 2011 at 03:49, Subash Patel <subashrp@gmail.com> wrote:
> > VIDIOC_STREAMON expects buffers to be queued before hardware part of
> > image/video pipe is enabled. From my experience of V4L2 user space, I
> > have always QBUFfed before invoking the STREAMON. Below is the API
> > specification which also speaks something same:
> >
> 
> Not exactly. It says that the API requires buffers to be queued for
> output devices. It does not require any buffers to be queued for input
> devices. Sylwester is right here.
> 
> This feature of not having to queue input buffers before STREAMON
> introduces problems to driver implementations and I am personally not
> a big fan of it either. But I'm seeing some additional problems here.
> Suppose we forced QBUF to be done before STREAMON. This would work,
> but what happens next? What should happen when we want to DQBUF the
> last buffer? If the device couldn't start without any buffers queued,
> can it continue streaming with all of them dequeued? I would guess
> not. So we'd either have to deny DQBUF of the last buffer (which for
> me personally is almost unacceptable) or have the last DQBUF
> automatically cause a STREAMOFF. So, for the latter, should
> applications, after they get all the data they wanted, be made to
> always have one more buffer queued as a "throwaway" buffer? This is
> probably the only reasonable solution here, but the applications would
> have to keep count of their queued buffers and be aware of this.
> Also, there might still be situations where being able to STREAMON
> without buffers queued would be beneficial. For example, enabling the
> device might be a slow/expensive operation and we might prefer to keep
> it running even if we don't want any data at the moment. Even for
> faster devices, being able to keep them on and periodically take a
> snapshot would be faster without having to call STREAMON anyway...

The problem is that what is possible is highly hardware dependent. All video
capture device I know of (composite in, HDMI in, etc) require that buffers
are queued and they won't release that buffer to userspace until a new free
buffer is available. They DMA continuously and stopping the DMA at the last
buffer and restarting it when a new one appears tends to be too expensive
and leads to additional loss of frames.

In part how this should act depends on the use-case: if you are streaming
video, then requiring buffers to be present before STREAMON and holding on
to a buffer if userspace can't keep up seems quite reasonable to me.

But for snapshot and codec type streams this behavior doesn't make sense.
The main difference is that in this case the DMA is not driven by an external
input, but by internal (userspace) demand.

Something for our meeting to discuss.

Regards,

	Hans

> 
> 

-- 
Hans Verkuil - video4linux developer - sponsored by Cisco

Re: [Query] VIDIOC_QBUF and VIDIOC_STREAMON order

[Sylwester Nawrocki]

Hi Subash,

thanks for your comments.

On 03/14/2011 11:44 AM, Subash Patel wrote:
> VIDIOC_STREAMON expects buffers to be queued before hardware part of
> image/video pipe is enabled. From my experience of V4L2 user space, I have
> always QBUFfed before invoking the STREAMON. Below is the API specification
> which also speaks something same:
>  
> http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-streamon.html
>  

It seems pretty reasonable thing to do in the application, i.e. queue buffer(s)
first. If the device needs buffers to work what STREAMON before QBUF would be
supposed to mean? The application must be aware then that real start of data flow
in the hardware pipeline is being done at the first QBUF. This is not obvious at
all from the current specification.

> I think its better to return EINVAL if there are no queued buffers
> when VIDIOC_STREAMON is invoked.

Unfortunately, as Pawel pointed out the are are some further logic implications
of such approach. Anyway we can handle both QBUF, STREAMON orders in the kernel,
it just makes life a bit harder. 

Adhering exactly to current videobuf2 rules it is not possible to return EINVAL
if there is no buffers QBUFed when invoking STREAMON. And we are going to need
to be able to return an error in buf_queue which could be then propagated up
to the STREAMON caller. This has already been raised in 
"[RFC/PATCH 1/2] v4l: videobuf2: Handle buf_queue errors" from Laurent.

Regards,
Sylwester

>  
> Regards,
> Subash
> 
> On Mon, Mar 14, 2011 at 3:44 PM, Sylwester Nawrocki <s.nawrocki@samsung.com
> <mailto:s.nawrocki@samsung.com>> wrote:
> 
>     Hello,
> 
>     As far as I know V4L2 applications are allowed to call VIDIOC_STREAMON before
>     queuing buffers with VIDIOC_QBUF.
> 
>     This leads to situation that a H/W is attempted to be enabled by the driver
>     when it has no any buffer ownership.
> 
>     Effectively actual activation of the data pipeline has to be deferred
>     until first buffer arrived in the driver. Which makes it difficult
>     to signal any errors to user during enabling the data pipeline.
> 
>     Is this allowed to force applications to queue some buffers before calling
>     STREAMON, by returning an error in vidioc_streamon from the driver, when
>     no buffers have been queued at this time?
> 
>     I suppose this could render some applications to stop working if this kind
>     of restriction is applied e.g. in camera capture driver.
> 
>     What the applications really expect?
> 
>     With the above I refer mostly to a snapshot mode where we have to be careful
>     not to lose any frame, as there could be only one..
> 
> 
>     Please share you opinions.

Re: [Query] VIDIOC_QBUF and VIDIOC_STREAMON order

[Laurent Pinchart]

Hi Hans,

On Tuesday 15 March 2011 08:50:45 Hans Verkuil wrote:
> On Tuesday, March 15, 2011 04:21:05 Pawel Osciak wrote:
> > On Mon, Mar 14, 2011 at 03:49, Subash Patel <subashrp@gmail.com> wrote:
> > > VIDIOC_STREAMON expects buffers to be queued before hardware part of
> > > image/video pipe is enabled. From my experience of V4L2 user space, I
> > > have always QBUFfed before invoking the STREAMON. Below is the API
> > 
> > > specification which also speaks something same:
> > Not exactly. It says that the API requires buffers to be queued for
> > output devices. It does not require any buffers to be queued for input
> > devices. Sylwester is right here.
> > 
> > This feature of not having to queue input buffers before STREAMON
> > introduces problems to driver implementations and I am personally not
> > a big fan of it either. But I'm seeing some additional problems here.
> > Suppose we forced QBUF to be done before STREAMON. This would work,
> > but what happens next? What should happen when we want to DQBUF the
> > last buffer? If the device couldn't start without any buffers queued,
> > can it continue streaming with all of them dequeued? I would guess
> > not. So we'd either have to deny DQBUF of the last buffer (which for
> > me personally is almost unacceptable) or have the last DQBUF
> > automatically cause a STREAMOFF. So, for the latter, should
> > applications, after they get all the data they wanted, be made to
> > always have one more buffer queued as a "throwaway" buffer? This is
> > probably the only reasonable solution here, but the applications would
> > have to keep count of their queued buffers and be aware of this.
> > Also, there might still be situations where being able to STREAMON
> > without buffers queued would be beneficial. For example, enabling the
> > device might be a slow/expensive operation and we might prefer to keep
> > it running even if we don't want any data at the moment. Even for
> > faster devices, being able to keep them on and periodically take a
> > snapshot would be faster without having to call STREAMON anyway...
> 
> The problem is that what is possible is highly hardware dependent. All
> video capture device I know of (composite in, HDMI in, etc) require that
> buffers are queued and they won't release that buffer to userspace until a
> new free buffer is available.

That's funny, all video capture devices I know of behave the opposite way :-) 
They either pause the stream when they run out of buffers and resume it when a 
new buffer gets queued, or they throw away the data when intermediate buffers 
are used (such as with USB devices).

> They DMA continuously and stopping the DMA
> at the last buffer and restarting it when a new one appears tends to be
> too expensive and leads to additional loss of frames.
> 
> In part how this should act depends on the use-case: if you are streaming
> video, then requiring buffers to be present before STREAMON and holding on
> to a buffer if userspace can't keep up seems quite reasonable to me.
> 
> But for snapshot and codec type streams this behavior doesn't make sense.
> The main difference is that in this case the DMA is not driven by an
> external input, but by internal (userspace) demand.
> 
> Something for our meeting to discuss.

-- 
Regards,

Laurent Pinchart

RE: [Query] VIDIOC_QBUF and VIDIOC_STREAMON order

[Jonghun Han]

Hi all,

On Tuesday, March 22, 2011 7:54 PM Laurent Pinchart wrote:
> On Tuesday 15 March 2011 08:50:45 Hans Verkuil wrote:
> > On Tuesday, March 15, 2011 04:21:05 Pawel Osciak wrote:
> > > On Mon, Mar 14, 2011 at 03:49, Subash Patel <subashrp@gmail.com>
wrote:
> > > > VIDIOC_STREAMON expects buffers to be queued before hardware part
> > > > of image/video pipe is enabled. From my experience of V4L2 user
> > > > space, I have always QBUFfed before invoking the STREAMON. Below
> > > > is the API
> > >
> > > > specification which also speaks something same:
> > > Not exactly. It says that the API requires buffers to be queued for
> > > output devices. It does not require any buffers to be queued for
> > > input devices. Sylwester is right here.
> > >
> > > This feature of not having to queue input buffers before STREAMON
> > > introduces problems to driver implementations and I am personally
> > > not a big fan of it either. But I'm seeing some additional problems
here.
> > > Suppose we forced QBUF to be done before STREAMON. This would work,
> > > but what happens next? What should happen when we want to DQBUF the
> > > last buffer? If the device couldn't start without any buffers
> > > queued, can it continue streaming with all of them dequeued? I would
> > > guess not. So we'd either have to deny DQBUF of the last buffer
> > > (which for me personally is almost unacceptable) or have the last
> > > DQBUF automatically cause a STREAMOFF. So, for the latter, should
> > > applications, after they get all the data they wanted, be made to
> > > always have one more buffer queued as a "throwaway" buffer? This is
> > > probably the only reasonable solution here, but the applications
> > > would have to keep count of their queued buffers and be aware of this.
> > > Also, there might still be situations where being able to STREAMON
> > > without buffers queued would be beneficial. For example, enabling
> > > the device might be a slow/expensive operation and we might prefer
> > > to keep it running even if we don't want any data at the moment.
> > > Even for faster devices, being able to keep them on and periodically
> > > take a snapshot would be faster without having to call STREAMON
anyway...
> >
> > The problem is that what is possible is highly hardware dependent. All
> > video capture device I know of (composite in, HDMI in, etc) require
> > that buffers are queued and they won't release that buffer to
> > userspace until a new free buffer is available.
> 
> That's funny, all video capture devices I know of behave the opposite way
:-)
> They either pause the stream when they run out of buffers and resume it
when
> a new buffer gets queued, or they throw away the data when intermediate
> buffers are used (such as with USB devices).
> 

Laurent,
Exynos capture device is the same with your example.
It should pause the stream not to overwrite
when they run out of buffers and resume it when a new buffer gets queued.

Hans,
Do you mean that the buffer is overwritten without pause and resume
until a new free buffer is coming ?

> > They DMA continuously and stopping the DMA at the last buffer and
> > restarting it when a new one appears tends to be too expensive and
> > leads to additional loss of frames.
> >
> > In part how this should act depends on the use-case: if you are
> > streaming video, then requiring buffers to be present before STREAMON
> > and holding on to a buffer if userspace can't keep up seems quite
> reasonable to me.
> >
> > But for snapshot and codec type streams this behavior doesn't make
sense.
> > The main difference is that in this case the DMA is not driven by an
> > external input, but by internal (userspace) demand.
> >
> > Something for our meeting to discuss.
> 
> --
> Regards,
> 
> Laurent Pinchart
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

Best regards,

Jonghun Han