[RFC/PATCH] v4l: added V4L2_BUF_FLAG_EOS flag indicating the last frame in the stream

[RFC/PATCH] v4l: added V4L2_BUF_FLAG_EOS flag indicating the last frame in the stream

[Andrzej Hajda]

v4l: added V4L2_BUF_FLAG_EOS flag indicating the last frame in the stream

Some devices requires indicator if the buffer is the last one in the stream.
Applications and drivers can use this flag in such case.

Signed-off-by: Andrzej Hajda <a.hajda@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---

Hello,

This patch adds new v4l2_buffer flag V4L2_BUF_FLAG_EOS. This flag is set
by applications on the output buffer to indicate the last buffer of the stream.

Some devices (eg. s5p_mfc) requires presence of the end-of-stream indicator
together with the last buffer.
Common practice of sending empty buffer to indicate end-of-strem do not work in
such case.

I would like to ask for review and comments.

Apologies for duplicated e-mails - sendmail problems.

Regards
Andrzej Hajda

---
 Documentation/DocBook/media/v4l/io.xml          |    7 +++++++
 Documentation/DocBook/media/v4l/vidioc-qbuf.xml |    2 ++
 include/linux/videodev2.h                       |    1 +
 3 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/Documentation/DocBook/media/v4l/io.xml b/Documentation/DocBook/media/v4l/io.xml
index b815929..7985ca9 100644
--- a/Documentation/DocBook/media/v4l/io.xml
+++ b/Documentation/DocBook/media/v4l/io.xml
@@ -954,6 +954,13 @@ Typically applications shall use this flag for output buffers if the data
 in this buffer has not been created by the CPU but by some DMA-capable unit,
 in which case caches have not been used.</entry>
 	  </row>
+	  <row>
+	    <entry><constant>V4L2_BUF_FLAG_EOS</constant></entry>
+	    <entry>0x2000</entry>
+	    <entry>Application should set this flag in the output buffer
+in order to inform the driver about the last frame of the stream. Some
+drivers may require it to properly finish processing the stream.</entry>
+	  </row>
 	</tbody>
       </tgroup>
     </table>
diff --git a/Documentation/DocBook/media/v4l/vidioc-qbuf.xml b/Documentation/DocBook/media/v4l/vidioc-qbuf.xml
index 9caa49a..ad49f7d 100644
--- a/Documentation/DocBook/media/v4l/vidioc-qbuf.xml
+++ b/Documentation/DocBook/media/v4l/vidioc-qbuf.xml
@@ -76,6 +76,8 @@ supports capturing from specific video inputs and you want to specify a video
 input, then <structfield>flags</structfield> should be set to
 <constant>V4L2_BUF_FLAG_INPUT</constant> and the field
 <structfield>input</structfield> must be initialized to the desired input.
+Some drivers expects applications set <constant>V4L2_BUF_FLAG_EOS</constant>
+flag on the last buffer of the stream.
 The <structfield>reserved</structfield> field must be set to 0. When using
 the <link linkend="planar-apis">multi-planar API</link>, the
 <structfield>m.planes</structfield> field must contain a userspace pointer
diff --git a/include/linux/videodev2.h b/include/linux/videodev2.h
index c9c9a46..d1d3406 100644
--- a/include/linux/videodev2.h
+++ b/include/linux/videodev2.h
@@ -671,6 +671,7 @@ struct v4l2_buffer {
 /* Cache handling flags */
 #define V4L2_BUF_FLAG_NO_CACHE_INVALIDATE	0x0800
 #define V4L2_BUF_FLAG_NO_CACHE_CLEAN		0x1000
+#define V4L2_BUF_FLAG_EOS	0x2000	/* The last buffer in the stream */
 
 /*
  *	O V E R L A Y   P R E V I E W
-- 
1.7.0.4

RE: [RFC/PATCH] v4l: added V4L2_BUF_FLAG_EOS flag indicating the last frame in the stream

[Kamil Debski]

Hi,

> From: linux-media-owner@vger.kernel.org [mailto:linux-media-
> Sent: 10 April 2012 11:51
> 
> v4l: added V4L2_BUF_FLAG_EOS flag indicating the last frame in the stream
> 
> Some devices requires indicator if the buffer is the last one in the
> stream.
> Applications and drivers can use this flag in such case.
> 
> Signed-off-by: Andrzej Hajda <a.hajda@samsung.com>
> Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
> ---
> 
> Hello,
> 
> This patch adds new v4l2_buffer flag V4L2_BUF_FLAG_EOS. This flag is set
> by applications on the output buffer to indicate the last buffer of the
> stream.
> 
> Some devices (eg. s5p_mfc) requires presence of the end-of-stream
> indicator
> together with the last buffer.
> Common practice of sending empty buffer to indicate end-of-strem do not
> work in
> such case.
> 
> I would like to ask for review and comments.
> 
> Apologies for duplicated e-mails - sendmail problems.
> 
> Regards
> Andrzej Hajda
> 

[snip]

Maybe I could throw some more light at the problem.

The problem is that when the encoding is done it is necessary to mark the
last frame of the video that is encoded. It is needed because the hardware
may need to return some encoded buffers that are kept in the hardware.

Why the buffers are kept in hardware one might ask? The answer to this
question is following. The video frames are enqueued in MFC in presentation
order and the encoded frames are dequeued in decoding order.

Let's see an example:
			           1234567
The presentation order is:   IBBPBBP--
The decoding order here is:  --IPBBPBB
(the P frames have to be decoded before B frames as B frames reference
both preceding and following frame; when no B frames are used then
there is no delay)

So there is a delay of two buffers returned on the CAPTURE side to the
OUTPUT queue. After the last frame is encoded these buffers have to be
returned to the user. Our hardware needs to know that it is the last frame
before it is encoded, so the idea is to add a flag that would mark the
buffer as the last one.

The flag could also be used to mark the last frame during decoding - now
it is done by setting bytesused to 0. The EOS flag could be used in addition
to that.

Comments are welcome.

Best wishes,
--
Kamil Debski
Linux Platform Group
Samsung Poland R&D Center

Re: [RFC/PATCH] v4l: added V4L2_BUF_FLAG_EOS flag indicating the last frame in the stream

[Mauro Carvalho Chehab]

Em 13-04-2012 13:31, Kamil Debski escreveu:
> Hi,
> 
>> From: linux-media-owner@vger.kernel.org [mailto:linux-media-
>> Sent: 10 April 2012 11:51
>>
>> v4l: added V4L2_BUF_FLAG_EOS flag indicating the last frame in the stream
>>
>> Some devices requires indicator if the buffer is the last one in the
>> stream.
>> Applications and drivers can use this flag in such case.
>>
>> Signed-off-by: Andrzej Hajda <a.hajda@samsung.com>
>> Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
>> ---
>>
>> Hello,
>>
>> This patch adds new v4l2_buffer flag V4L2_BUF_FLAG_EOS. This flag is set
>> by applications on the output buffer to indicate the last buffer of the
>> stream.
>>
>> Some devices (eg. s5p_mfc) requires presence of the end-of-stream
>> indicator
>> together with the last buffer.
>> Common practice of sending empty buffer to indicate end-of-strem do not
>> work in
>> such case.
>>
>> I would like to ask for review and comments.
>>
>> Apologies for duplicated e-mails - sendmail problems.
>>
>> Regards
>> Andrzej Hajda
>>
> 
> [snip]
> 
> Maybe I could throw some more light at the problem.
> 
> The problem is that when the encoding is done it is necessary to mark the
> last frame of the video that is encoded. It is needed because the hardware
> may need to return some encoded buffers that are kept in the hardware.

Are you talking only about V4L2_BUF_TYPE_VIDEO_OUTPUT? 

> Why the buffers are kept in hardware one might ask? The answer to this
> question is following. The video frames are enqueued in MFC in presentation
> order and the encoded frames are dequeued in decoding order.
> 
> Let's see an example:
> 			           1234567
> The presentation order is:   IBBPBBP--
> The decoding order here is:  --IPBBPBB
> (the P frames have to be decoded before B frames as B frames reference
> both preceding and following frame; when no B frames are used then
> there is no delay)
> 
> So there is a delay of two buffers returned on the CAPTURE side to the
> OUTPUT queue. After the last frame is encoded these buffers have to be
> returned to the user. Our hardware needs to know that it is the last frame
> before it is encoded, so the idea is to add a flag that would mark the
> buffer as the last one.
> 
> The flag could also be used to mark the last frame during decoding - now
> it is done by setting bytesused to 0. The EOS flag could be used in addition
> to that.
> 
> Comments are welcome.

For V4L2_BUF_TYPE_VIDEO_CAPTURE, a change like the one proposed has issues
to be considered: this kind of issue may happen on any driver delivering MPEG
format. 

So, if we're willing to introduce flags for MPEG-specific handling like that, 
then all drivers delivering mpeg outputs should be patched, and not only the
drivers you're maintaining, otherwise userspace applications can't trust that 
this feature is there.

Btw, the encoder API, designed to mpeg encoders addresses it on a different way:
V4L2_ENC_CMD_STOP stops an mpeg stream, but it waits until the end of a group
of pictures:
    http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-encoder-cmd.html

At least for video capture, this seems to work fine.

For video output, it could make sense to have a flag mark the end of a GoP,
but, again, if we're thinking that the source could be a V4L2 capture, the
GoP end should be marked there, and the patches adding support for it will
need to touch the existing drivers that have mpeg encoders/decoders, in order
to be sure that, after a certain V4L2 API, all of them will support such
feature.

Regards,
Mauro

Re: [RFC/PATCH] v4l: added V4L2_BUF_FLAG_EOS flag indicating the last frame in the stream

[Andrzej Hajda]

On Thu, 2012-04-19 at 10:58 -0300, Mauro Carvalho Chehab wrote:
> Em 13-04-2012 13:31, Kamil Debski escreveu:
> > Hi,
> > 
> >> From: linux-media-owner@vger.kernel.org [mailto:linux-media-
> >> Sent: 10 April 2012 11:51
> >>
> >> v4l: added V4L2_BUF_FLAG_EOS flag indicating the last frame in the stream
> >>
> >> Some devices requires indicator if the buffer is the last one in the
> >> stream.
> >> Applications and drivers can use this flag in such case.
> >>
> >> Signed-off-by: Andrzej Hajda <a.hajda@samsung.com>
> >> Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
> >> ---
> >>
> >> Hello,
> >>
> >> This patch adds new v4l2_buffer flag V4L2_BUF_FLAG_EOS. This flag is set
> >> by applications on the output buffer to indicate the last buffer of the
> >> stream.
> >>
> >> Some devices (eg. s5p_mfc) requires presence of the end-of-stream
> >> indicator
> >> together with the last buffer.
> >> Common practice of sending empty buffer to indicate end-of-strem do not
> >> work in
> >> such case.
> >>
> >> I would like to ask for review and comments.
> >>
> >> Apologies for duplicated e-mails - sendmail problems.
> >>
> >> Regards
> >> Andrzej Hajda
> >>
> > 
> > [snip]
> > 
> > Maybe I could throw some more light at the problem.
> > 
> > The problem is that when the encoding is done it is necessary to mark the
> > last frame of the video that is encoded. It is needed because the hardware
> > may need to return some encoded buffers that are kept in the hardware.
> 
> Are you talking only about V4L2_BUF_TYPE_VIDEO_OUTPUT? 

Hardware encoder requires that the last frame sent to device should be
specially marked. So this flag in our case is only for
V4L2_BUF_TYPE_VIDEO_OUTPUT buffers. As a result of this mark device will
encode and release all frames cached in internal buffers, so the driver
can enqueue them in V4L2_BUF_TYPE_VIDEO_CAPTURE queue.
Sample encoding scenario:
Frame1 -> Dev
Frame2 -> Dev
Dev -> Encoded1
Frame3 marked as the last one -> Dev
Dev -> Encoded2
Dev -> Encoded3
Dev signals end of encoded stream

Without this mark we will not receive encoded frame 3. And there will be
no signal from the device about the end of encoded stream.

> > Why the buffers are kept in hardware one might ask? The answer to this
> > question is following. The video frames are enqueued in MFC in presentation
> > order and the encoded frames are dequeued in decoding order.
> > 
> > Let's see an example:
> > 			           1234567
> > The presentation order is:   IBBPBBP--
> > The decoding order here is:  --IPBBPBB
> > (the P frames have to be decoded before B frames as B frames reference
> > both preceding and following frame; when no B frames are used then
> > there is no delay)
> > 
> > So there is a delay of two buffers returned on the CAPTURE side to the
> > OUTPUT queue. After the last frame is encoded these buffers have to be
> > returned to the user. Our hardware needs to know that it is the last frame
> > before it is encoded, so the idea is to add a flag that would mark the
> > buffer as the last one.
> > 
> > The flag could also be used to mark the last frame during decoding - now
> > it is done by setting bytesused to 0. The EOS flag could be used in addition
> > to that.
> > 
> > Comments are welcome.
> 
> For V4L2_BUF_TYPE_VIDEO_CAPTURE, a change like the one proposed has issues
> to be considered: this kind of issue may happen on any driver delivering MPEG
> format. 
> 
> So, if we're willing to introduce flags for MPEG-specific handling like that, 
> then all drivers delivering mpeg outputs should be patched, and not only the
> drivers you're maintaining, otherwise userspace applications can't trust that 
> this feature is there.

This flag on V4L2_BUF_TYPE_VIDEO_CAPTURE is not required by our
hardware. Since we cannot eliminate empty buffers for signaling end of
encoded stream for compatibility reasons, there will be no real profit
from using it.

> Btw, the encoder API, designed to mpeg encoders addresses it on a different way:
> V4L2_ENC_CMD_STOP stops an mpeg stream, but it waits until the end of a group
> of pictures:
>     http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-encoder-cmd.html
> 
> At least for video capture, this seems to work fine.

In case of our hardware we cannot send STOP command after sending the
last frame, we should sent it together.
Possible workaround I see is that driver will not send
V4L2_BUF_TYPE_VIDEO_OUTPUT frame to device unless there is another one
in V4L2_BUF_TYPE_VIDEO_CAPTURE queue, or it receives V4L2_ENC_CMD_STOP.
This solution seems to me less straightforward that the proposed one and
adds phony requirement on number of V4L2_BUF_TYPE_VIDEO_OUTPUT
buffers(at least 2).

> 
> For video output, it could make sense to have a flag mark the end of a GoP,
> but, again, if we're thinking that the source could be a V4L2 capture, the
> GoP end should be marked there, and the patches adding support for it will
> need to touch the existing drivers that have mpeg encoders/decoders, in order
> to be sure that, after a certain V4L2 API, all of them will support such
> feature.
> 
> Regards,
> Mauro


Regards,
Andrzej

Re: [RFC/PATCH] v4l: added V4L2_BUF_FLAG_EOS flag indicating the last frame in the stream

[Andrzej Hajda]

On Fri, 2012-04-20 at 12:38 +0200, Andrzej Hajda wrote:
> On Thu, 2012-04-19 at 10:58 -0300, Mauro Carvalho Chehab wrote:
> > Em 13-04-2012 13:31, Kamil Debski escreveu:
> > > Hi,
> > > 
> > >> From: linux-media-owner@vger.kernel.org [mailto:linux-media-
> > >> Sent: 10 April 2012 11:51
> > >>
> > >> v4l: added V4L2_BUF_FLAG_EOS flag indicating the last frame in the stream
> > >>
> > >> Some devices requires indicator if the buffer is the last one in the
> > >> stream.
> > >> Applications and drivers can use this flag in such case.
> > >>
> > >> Signed-off-by: Andrzej Hajda <a.hajda@samsung.com>
> > >> Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
> > >> ---
> > >>
> > >> Hello,
> > >>
> > >> This patch adds new v4l2_buffer flag V4L2_BUF_FLAG_EOS. This flag is set
> > >> by applications on the output buffer to indicate the last buffer of the
> > >> stream.
> > >>
> > >> Some devices (eg. s5p_mfc) requires presence of the end-of-stream
> > >> indicator
> > >> together with the last buffer.
> > >> Common practice of sending empty buffer to indicate end-of-strem do not
> > >> work in
> > >> such case.
> > >>
> > >> I would like to ask for review and comments.
> > >>
> > >> Apologies for duplicated e-mails - sendmail problems.
> > >>
> > >> Regards
> > >> Andrzej Hajda
> > >>
> > > 
> > > [snip]
> > > 
> > > Maybe I could throw some more light at the problem.
> > > 
> > > The problem is that when the encoding is done it is necessary to mark the
> > > last frame of the video that is encoded. It is needed because the hardware
> > > may need to return some encoded buffers that are kept in the hardware.
> > 
> > Are you talking only about V4L2_BUF_TYPE_VIDEO_OUTPUT? 
> 
> Hardware encoder requires that the last frame sent to device should be
> specially marked. So this flag in our case is only for
> V4L2_BUF_TYPE_VIDEO_OUTPUT buffers. As a result of this mark device will
> encode and release all frames cached in internal buffers, so the driver
> can enqueue them in V4L2_BUF_TYPE_VIDEO_CAPTURE queue.
> Sample encoding scenario:
> Frame1 -> Dev
> Frame2 -> Dev
> Dev -> Encoded1
> Frame3 marked as the last one -> Dev
> Dev -> Encoded2
> Dev -> Encoded3
> Dev signals end of encoded stream
> 
> Without this mark we will not receive encoded frame 3. And there will be
> no signal from the device about the end of encoded stream.
> 
> > > Why the buffers are kept in hardware one might ask? The answer to this
> > > question is following. The video frames are enqueued in MFC in presentation
> > > order and the encoded frames are dequeued in decoding order.
> > > 
> > > Let's see an example:
> > > 			           1234567
> > > The presentation order is:   IBBPBBP--
> > > The decoding order here is:  --IPBBPBB
> > > (the P frames have to be decoded before B frames as B frames reference
> > > both preceding and following frame; when no B frames are used then
> > > there is no delay)
> > > 
> > > So there is a delay of two buffers returned on the CAPTURE side to the
> > > OUTPUT queue. After the last frame is encoded these buffers have to be
> > > returned to the user. Our hardware needs to know that it is the last frame
> > > before it is encoded, so the idea is to add a flag that would mark the
> > > buffer as the last one.
> > > 
> > > The flag could also be used to mark the last frame during decoding - now
> > > it is done by setting bytesused to 0. The EOS flag could be used in addition
> > > to that.
> > > 
> > > Comments are welcome.
> > 
> > For V4L2_BUF_TYPE_VIDEO_CAPTURE, a change like the one proposed has issues
> > to be considered: this kind of issue may happen on any driver delivering MPEG
> > format. 
> > 
> > So, if we're willing to introduce flags for MPEG-specific handling like that, 
> > then all drivers delivering mpeg outputs should be patched, and not only the
> > drivers you're maintaining, otherwise userspace applications can't trust that 
> > this feature is there.
> 
> This flag on V4L2_BUF_TYPE_VIDEO_CAPTURE is not required by our
> hardware. Since we cannot eliminate empty buffers for signaling end of
> encoded stream for compatibility reasons, there will be no real profit
> from using it.
> 
> > Btw, the encoder API, designed to mpeg encoders addresses it on a different way:
> > V4L2_ENC_CMD_STOP stops an mpeg stream, but it waits until the end of a group
> > of pictures:
> >     http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-encoder-cmd.html
> > 
> > At least for video capture, this seems to work fine.
> 
> In case of our hardware we cannot send STOP command after sending the
> last frame, we should sent it together.
> Possible workaround I see is that driver will not send
> V4L2_BUF_TYPE_VIDEO_OUTPUT frame to device unless there is another one
> in V4L2_BUF_TYPE_VIDEO_CAPTURE queue, or it receives V4L2_ENC_CMD_STOP.

Ups, should be "in V4L2_BUF_TYPE_VIDEO_OUTPUT queue".

> This solution seems to me less straightforward that the proposed one and
> adds phony requirement on number of V4L2_BUF_TYPE_VIDEO_OUTPUT
> buffers(at least 2).
> 
> > 
> > For video output, it could make sense to have a flag mark the end of a GoP,
> > but, again, if we're thinking that the source could be a V4L2 capture, the
> > GoP end should be marked there, and the patches adding support for it will
> > need to touch the existing drivers that have mpeg encoders/decoders, in order
> > to be sure that, after a certain V4L2 API, all of them will support such
> > feature.
> > 
> > Regards,
> > Mauro
> 
> 
> Regards,
> Andrzej
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html