Way to request a time discontinuity in a V4L2 encoder device

Way to request a time discontinuity in a V4L2 encoder device

[Alain VOLMAT]

Hi all,

We have developed a V4L2 mem2mem driver for an H264 encoder running on an IP of one of our SoC and would like to have one more v4l2_buffer flag value for that.

In the context of this driver, we discovered that the current V4L2 encode API is missing the feature to mention to the IP that a time discontinuity has to be created.
Time discontinuity must be triggered when there is a discontinuity in the stream to be encoded, which would for example happen when there is a seek in the data to be encoded. In such case, it means that the IP should reset its bitrate allocation algorithm.

Considering that this information should be triggered on a frame basis, the idea is to have it passed via the flags member of v4l2_buffer, with a new flag V4L2_BUF_FLAG_ENCODE_TIME_DISCONTINUITY.

The usage for this flag value are:
* Queuing a "to be encoded" v4l2_buffer with flags & V4L2_BUF_FLAG_ENCODE_TIME_DISCONTINUITY informs the driver/IP that a time discontinuity (reset in the bitrate allocation algorithm) should be performed before encoding this frame.
* The flags bit should be then propagated until the dequeue to let the application know when a buffer is the first one after a time discontinuity.

Few words about the driver itself, it is available in the following context.
1. STLinux kernel (www.stlinux.com) rather than vanilla kernel since the board support is only available there for now
2. Multicom (http://www.st.com/internet/com/TECHNICAL_RESOURCES/TECHNICAL_LITERATURE/USER_MANUAL/CD18182595.pdf) based V4L2 driver. Multicom is an ST layer to allow to send and serialize commands to our various IP.

Regards,

Alain

Re: Way to request a time discontinuity in a V4L2 encoder device

[Laurent Pinchart]

Hi Alain,

On Wednesday 31 October 2012 14:21:30 Alain VOLMAT wrote:
> Hi all,
> 
> We have developed a V4L2 mem2mem driver for an H264 encoder running on an IP
> of one of our SoC and would like to have one more v4l2_buffer flag value
> for that.
> 
> In the context of this driver, we discovered that the current V4L2 encode
> API is missing the feature to mention to the IP that a time discontinuity
> has to be created. Time discontinuity must be triggered when there is a
> discontinuity in the stream to be encoded, which would for example happen
> when there is a seek in the data to be encoded. In such case, it means that
> the IP should reset its bitrate allocation algorithm.
> 
> Considering that this information should be triggered on a frame basis, the
> idea is to have it passed via the flags member of v4l2_buffer, with a new
> flag V4L2_BUF_FLAG_ENCODE_TIME_DISCONTINUITY.
> 
> The usage for this flag value are:
> * Queuing a "to be encoded" v4l2_buffer with flags &
> V4L2_BUF_FLAG_ENCODE_TIME_DISCONTINUITY informs the driver/IP that a time
> discontinuity (reset in the bitrate allocation algorithm) should be
> performed before encoding this frame. * The flags bit should be then
> propagated until the dequeue to let the application know when a buffer is
> the first one after a time discontinuity.

I wonder whether a buffer flag is the best technical solution for this. 
There's a very limited quantity of flags, and it seems to me like we'll need 
more flags like this one in the future. Using flags thus wouldn't scale.

Encoding a frame obviously requires the contents of the frame itself, but also 
requires meta data such as time discontinuity. Would it be better to add a 
meta data plane that would be used to pass all the frame meta data to the 
encoder ?

> Few words about the driver itself, it is available in the following context.
> 1. STLinux kernel (www.stlinux.com) rather than vanilla kernel since the
> board support is only available there for now 2. Multicom
> (http://www.st.com/internet/com/TECHNICAL_RESOURCES/TECHNICAL_LITERATURE/US
> ER_MANUAL/CD18182595.pdf) based V4L2 driver. Multicom is an ST layer to
> allow to send and serialize commands to our various IP.

-- 
Regards,

Laurent Pinchart

RE: Way to request a time discontinuity in a V4L2 encoder device

[Alain VOLMAT]

Hi Laurent,

Yes indeed, meta plane seems a good candidate. It was the other option.

The pity with that is that the FMT can thus no longer be standard FMT but a specific format that include both plane 0 with real frame data and plane 1 with meta data.
So, standard V4L2 application (that doesn't know about this time discontinuity stuff) wouldn't be able to push things into the encoder since they are not aware of this 2 plane format.

Or maybe we should export 2 format, 1 standard one that doesn't have time discontinuity support, thus not best performance but still can do things and a second format that has 2 planes

Regards,

Alain

> -----Original Message-----
> From: Laurent Pinchart [mailto:laurent.pinchart@ideasonboard.com]
> Sent: Sunday, November 04, 2012 12:54 PM
> To: Alain VOLMAT
> Cc: linux-media
> Subject: Re: Way to request a time discontinuity in a V4L2 encoder device
> 
> Hi Alain,
> 
> On Wednesday 31 October 2012 14:21:30 Alain VOLMAT wrote:
> > Hi all,
> >
> > We have developed a V4L2 mem2mem driver for an H264 encoder running
> on
> > an IP of one of our SoC and would like to have one more v4l2_buffer
> > flag value for that.
> >
> > In the context of this driver, we discovered that the current V4L2
> > encode API is missing the feature to mention to the IP that a time
> > discontinuity has to be created. Time discontinuity must be triggered
> > when there is a discontinuity in the stream to be encoded, which would
> > for example happen when there is a seek in the data to be encoded. In
> > such case, it means that the IP should reset its bitrate allocation algorithm.
> >
> > Considering that this information should be triggered on a frame
> > basis, the idea is to have it passed via the flags member of
> > v4l2_buffer, with a new flag
> V4L2_BUF_FLAG_ENCODE_TIME_DISCONTINUITY.
> >
> > The usage for this flag value are:
> > * Queuing a "to be encoded" v4l2_buffer with flags &
> > V4L2_BUF_FLAG_ENCODE_TIME_DISCONTINUITY informs the driver/IP
> that a
> > time discontinuity (reset in the bitrate allocation algorithm) should
> > be performed before encoding this frame. * The flags bit should be
> > then propagated until the dequeue to let the application know when a
> > buffer is the first one after a time discontinuity.
> 
> I wonder whether a buffer flag is the best technical solution for this.
> There's a very limited quantity of flags, and it seems to me like we'll need
> more flags like this one in the future. Using flags thus wouldn't scale.
> 
> Encoding a frame obviously requires the contents of the frame itself, but also
> requires meta data such as time discontinuity. Would it be better to add a
> meta data plane that would be used to pass all the frame meta data to the
> encoder ?
> 
> > Few words about the driver itself, it is available in the following context.
> > 1. STLinux kernel (www.stlinux.com) rather than vanilla kernel since
> > the board support is only available there for now 2. Multicom
> >
> (http://www.st.com/internet/com/TECHNICAL_RESOURCES/TECHNICAL_LIT
> ERATU
> > RE/US
> > ER_MANUAL/CD18182595.pdf) based V4L2 driver. Multicom is an ST layer
> > to allow to send and serialize commands to our various IP.
> 
> --
> Regards,
> 
> Laurent Pinchart

Re: Way to request a time discontinuity in a V4L2 encoder device

[Sylwester Nawrocki]

Hi,

On 11/05/2012 11:45 AM, Alain VOLMAT wrote:
> Hi Laurent,
>
> Yes indeed, meta plane seems a good candidate. It was the other option.
>
> The pity with that is that the FMT can thus no longer be standard FMT but
>  a specific format that include both plane 0 with real frame data and plane
>1 with meta data.
> So, standard V4L2 application (that doesn't know about this time
>discontinuity stuff) wouldn't be able to push things into the encoder since
>they are not aware of this 2 plane format.
>
> Or maybe we should export 2 format, 1 standard one that doesn't have time
>discontinuity support, thus not best performance but still can do things
>and a second format that has 2 planes

Not sure what media guys think about it, I was considering making it 
possible
for applications (or libv4l or any other library) to request additional 
meta
data plane at a video capture driver, e.g. with VIDIOC_S_FMT ioctl. With 
same
fourcc for e.g. 1-planar buffers with image data and 2-planar buffer with
meta data in plane 1. However this would be somehow device-specific, rather
than a completely generic interface. Since frame-meta data is often device
specific. For camera it would depend on the image sensor whether the 
additional
plane request on a /dev/video? would be fulfilled by a driver or not.

I don't think duplicating 4CCs for the sake of additional meta-data plane is
a good idea.

Your case is a bit different, since you're passing data from application to
a device. Maybe we could somewhat standardize the meta data buffer content,
e.g. by using some standard header specifying what kind of meta data 
follows.
Perhaps struct v4l2_plane::data_offset can be helpful here. This is how 
it's
documented

  * @data_offset:	offset in the plane to the start of data; usually 0,
  *			unless there is a header in front of the data

I mean, the header would specify what actual meta-data is in that additional
plane. Standardising that "standard" meta-data would be another issue.

I think this per buffer device control issue emerged in the past during the
Exynos Multi Format Codec development. There were proposals of per-buffer
v4l2 controls. IIRC it is currently resolved in that driver by doing
VIDIOC_S_CTRL before QBUF. However the meta data plane approach looks more
interesting to me.

--

Regards,
Sylwester

Re: Way to request a time discontinuity in a V4L2 encoder device

[Laurent Pinchart]

Hi Sylwester,

On Monday 05 November 2012 21:47:00 Sylwester Nawrocki wrote:
> On 11/05/2012 11:45 AM, Alain VOLMAT wrote:
> > Hi Laurent,
> > 
> > Yes indeed, meta plane seems a good candidate. It was the other option.
> > 
> > The pity with that is that the FMT can thus no longer be standard FMT but
> > a specific format that include both plane 0 with real frame data and plane
> > 1 with meta data.
> >
> > So, standard V4L2 application (that doesn't know about this time
> > discontinuity stuff) wouldn't be able to push things into the encoder
> > since they are not aware of this 2 plane format.
> >
> > Or maybe we should export 2 format, 1 standard one that doesn't have time
> > discontinuity support, thus not best performance but still can do things
> > and a second format that has 2 planes
> 
> Not sure what media guys think about it, I was considering making it
> possible for applications (or libv4l or any other library) to request
> additional meta data plane at a video capture driver, e.g. with VIDIOC_S_FMT
> ioctl. With same fourcc for e.g. 1-planar buffers with image data and 2-
> planar buffer with meta data in plane 1. However this would be somehow
> device-specific, rather than a completely generic interface. Since frame-
> meta data is often device specific. For camera it would depend on the image
> sensor whether the additional plane request on a /dev/video? would be
> fulfilled by a driver or not.
> 
> I don't think duplicating 4CCs for the sake of additional meta-data plane is
> a good idea.

I agree with you. A generic way to enable meta-data in a separate plane 
without requiring a separate 4CC is a good idea.

> Your case is a bit different, since you're passing data from application to
> a device. Maybe we could somewhat standardize the meta data buffer content,
> e.g. by using some standard header specifying what kind of meta data
> follows.
> Perhaps struct v4l2_plane::data_offset can be helpful here. This is how
> it's documented
> 
>   * @data_offset:	offset in the plane to the start of data; usually 0,
>   *			unless there is a header in front of the data
> 

There's also 11 reserved fields.

> I mean, the header would specify what actual meta-data is in that additional
> plane. Standardising that "standard" meta-data would be another issue.
>
> I think this per buffer device control issue emerged in the past during the
> Exynos Multi Format Codec development. There were proposals of per-buffer
> v4l2 controls. IIRC it is currently resolved in that driver by doing
> VIDIOC_S_CTRL before QBUF. However the meta data plane approach looks more
> interesting to me.

Maybe it's time to discuss the topic again then :-)

-- 
Regards,

Laurent Pinchart

Re: Way to request a time discontinuity in a V4L2 encoder device

[Sakari Ailus]

Hi Sylwester,

On Mon, Nov 05, 2012 at 09:47:00PM +0100, Sylwester Nawrocki wrote:
> Hi,
> 
> On 11/05/2012 11:45 AM, Alain VOLMAT wrote:
> >Hi Laurent,
> >
> >Yes indeed, meta plane seems a good candidate. It was the other option.
> >
> >The pity with that is that the FMT can thus no longer be standard FMT but
> > a specific format that include both plane 0 with real frame data and plane
> >1 with meta data.
> >So, standard V4L2 application (that doesn't know about this time
> >discontinuity stuff) wouldn't be able to push things into the encoder since
> >they are not aware of this 2 plane format.
> >
> >Or maybe we should export 2 format, 1 standard one that doesn't have time
> >discontinuity support, thus not best performance but still can do things
> >and a second format that has 2 planes
> 
> Not sure what media guys think about it, I was considering making it
> possible
> for applications (or libv4l or any other library) to request
> additional meta
> data plane at a video capture driver, e.g. with VIDIOC_S_FMT ioctl.
> With same
> fourcc for e.g. 1-planar buffers with image data and 2-planar buffer with
> meta data in plane 1. However this would be somehow device-specific, rather

How about this: add a special 4cc that tells only that the 4cc is defined
per-plane, and define it in planes instead? We could also add a flags field
to tell that a plane is actually a part of the same image in cases where
true multi-plane formats are being used at the same time.

You could use this to pass frame metadata (when it's produced by the sensor
itself) to the user space as well.

What I haven't yet thought is how this can be told to the user using
ENUMFMT.

> than a completely generic interface. Since frame-meta data is often device
> specific. For camera it would depend on the image sensor whether the

Device-specific metadata should have their own 4ccs as device specific image
formats.

> additional
> plane request on a /dev/video? would be fulfilled by a driver or not.
> 
> I don't think duplicating 4CCs for the sake of additional meta-data plane is
> a good idea.

No, that'd really explode the number of different 4ccs.

> Your case is a bit different, since you're passing data from application to
> a device. Maybe we could somewhat standardize the meta data buffer content,

We need to define a proper format for this. It can include other
per-buffer parameters to / from the device as well.

> e.g. by using some standard header specifying what kind of meta data
> follows.
> Perhaps struct v4l2_plane::data_offset can be helpful here. This is
> how it's
> documented
> 
>  * @data_offset:	offset in the plane to the start of data; usually 0,
>  *			unless there is a header in front of the data
> 
> I mean, the header would specify what actual meta-data is in that additional
> plane. Standardising that "standard" meta-data would be another issue.
> 
> I think this per buffer device control issue emerged in the past during the
> Exynos Multi Format Codec development. There were proposals of per-buffer
> v4l2 controls. IIRC it is currently resolved in that driver by doing
> VIDIOC_S_CTRL before QBUF. However the meta data plane approach looks more
> interesting to me.

That sounds like a simple (and thus good) solution to me: a button control
for resetting the average bitrate calculation. It'd be by far more simple
than using the metadata plane for it. Any known drawbacks that I can't see?
Even if the number of these parameters grow a little extended controls are
fine for the purpose.

Kind regards,

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi     XMPP: sailus@retiisi.org.uk

Re: Way to request a time discontinuity in a V4L2 encoder device

[Laurent Pinchart]

Hi Sakari,

On Sunday 11 November 2012 10:51:06 Sakari Ailus wrote:
> On Mon, Nov 05, 2012 at 09:47:00PM +0100, Sylwester Nawrocki wrote:
> > On 11/05/2012 11:45 AM, Alain VOLMAT wrote:
> > > Hi Laurent,
> > >
> > > Yes indeed, meta plane seems a good candidate. It was the other option.
> > >
> > > The pity with that is that the FMT can thus no longer be standard FMT
> > > but a specific format that include both plane 0 with real frame data and
> > > plane 1 with meta data.
> > > So, standard V4L2 application (that doesn't know about this time
> > > discontinuity stuff) wouldn't be able to push things into the encoder
> > > since they are not aware of this 2 plane format.
> > >
> > > Or maybe we should export 2 format, 1 standard one that doesn't have
> > > time discontinuity support, thus not best performance but still can do
> > > things and a second format that has 2 planes
> > 
> > Not sure what media guys think about it, I was considering making it
> > possible for applications (or libv4l or any other library) to request
> > additional meta data plane at a video capture driver, e.g. with
> > VIDIOC_S_FMT ioctl. With same fourcc for e.g. 1-planar buffers with image
> > data and 2-planar buffer with meta data in plane 1. However this would be
> > somehow device-specific, rather
> 
> How about this: add a special 4cc that tells only that the 4cc is defined
> per-plane, and define it in planes instead? We could also add a flags field
> to tell that a plane is actually a part of the same image in cases where
> true multi-plane formats are being used at the same time.
> 
> You could use this to pass frame metadata (when it's produced by the sensor
> itself) to the user space as well.

That sounds like a good idea to explore.

> What I haven't yet thought is how this can be told to the user using
> ENUMFMT.
> 
> > than a completely generic interface. Since frame-meta data is often device
> > specific. For camera it would depend on the image sensor whether the
> 
> Device-specific metadata should have their own 4ccs as device specific image
> formats.
> 
> > additional plane request on a /dev/video? would be fulfilled by a driver
> > or not.
> > 
> > I don't think duplicating 4CCs for the sake of additional meta-data plane
> > is a good idea.
> 
> No, that'd really explode the number of different 4ccs.
> 
> > Your case is a bit different, since you're passing data from application
> > to a device. Maybe we could somewhat standardize the meta data buffer
> > content,
> 
> We need to define a proper format for this. It can include other per-buffer
> parameters to / from the device as well.
>
> > e.g. by using some standard header specifying what kind of meta data
> > follows. Perhaps struct v4l2_plane::data_offset can be helpful here. This
> > is how it's documented
> > 
> >  * @data_offset:	offset in the plane to the start of data; usually 0,
> >  *			unless there is a header in front of the data
> > 
> > I mean, the header would specify what actual meta-data is in that
> > additional plane. Standardising that "standard" meta-data would be
> > another issue.
> > 
> > I think this per buffer device control issue emerged in the past during
> > the Exynos Multi Format Codec development. There were proposals of per-
> > buffer v4l2 controls. IIRC it is currently resolved in that driver by
> > doing VIDIOC_S_CTRL before QBUF. However the meta data plane approach
> > looks more interesting to me.
> 
> That sounds like a simple (and thus good) solution to me: a button control
> for resetting the average bitrate calculation. It'd be by far more simple
> than using the metadata plane for it. Any known drawbacks that I can't see?
> Even if the number of these parameters grow a little extended controls are
> fine for the purpose.

The main issue as far as I know would be to synchronize the control with 
buffers.

-- 
Regards,

Laurent Pinchart

Re: Way to request a time discontinuity in a V4L2 encoder device

[Sakari Ailus]

Hi Laurent,

On Mon, Nov 12, 2012 at 12:18:10PM +0100, Laurent Pinchart wrote:
> On Sunday 11 November 2012 10:51:06 Sakari Ailus wrote:
> > On Mon, Nov 05, 2012 at 09:47:00PM +0100, Sylwester Nawrocki wrote:
> > > On 11/05/2012 11:45 AM, Alain VOLMAT wrote:
> > > > Hi Laurent,
> > > >
> > > > Yes indeed, meta plane seems a good candidate. It was the other option.
> > > >
> > > > The pity with that is that the FMT can thus no longer be standard FMT
> > > > but a specific format that include both plane 0 with real frame data and
> > > > plane 1 with meta data.
> > > > So, standard V4L2 application (that doesn't know about this time
> > > > discontinuity stuff) wouldn't be able to push things into the encoder
> > > > since they are not aware of this 2 plane format.
> > > >
> > > > Or maybe we should export 2 format, 1 standard one that doesn't have
> > > > time discontinuity support, thus not best performance but still can do
> > > > things and a second format that has 2 planes
> > > 
> > > Not sure what media guys think about it, I was considering making it
> > > possible for applications (or libv4l or any other library) to request
> > > additional meta data plane at a video capture driver, e.g. with
> > > VIDIOC_S_FMT ioctl. With same fourcc for e.g. 1-planar buffers with image
> > > data and 2-planar buffer with meta data in plane 1. However this would be
> > > somehow device-specific, rather
> > 
> > How about this: add a special 4cc that tells only that the 4cc is defined
> > per-plane, and define it in planes instead? We could also add a flags field
> > to tell that a plane is actually a part of the same image in cases where
> > true multi-plane formats are being used at the same time.
> > 
> > You could use this to pass frame metadata (when it's produced by the sensor
> > itself) to the user space as well.
> 
> That sounds like a good idea to explore.
> 
> > What I haven't yet thought is how this can be told to the user using
> > ENUMFMT.
> > 
> > > than a completely generic interface. Since frame-meta data is often device
> > > specific. For camera it would depend on the image sensor whether the
> > 
> > Device-specific metadata should have their own 4ccs as device specific image
> > formats.
> > 
> > > additional plane request on a /dev/video? would be fulfilled by a driver
> > > or not.
> > > 
> > > I don't think duplicating 4CCs for the sake of additional meta-data plane
> > > is a good idea.
> > 
> > No, that'd really explode the number of different 4ccs.
> > 
> > > Your case is a bit different, since you're passing data from application
> > > to a device. Maybe we could somewhat standardize the meta data buffer
> > > content,
> > 
> > We need to define a proper format for this. It can include other per-buffer
> > parameters to / from the device as well.
> >
> > > e.g. by using some standard header specifying what kind of meta data
> > > follows. Perhaps struct v4l2_plane::data_offset can be helpful here. This
> > > is how it's documented
> > > 
> > >  * @data_offset:	offset in the plane to the start of data; usually 0,
> > >  *			unless there is a header in front of the data
> > > 
> > > I mean, the header would specify what actual meta-data is in that
> > > additional plane. Standardising that "standard" meta-data would be
> > > another issue.
> > > 
> > > I think this per buffer device control issue emerged in the past during
> > > the Exynos Multi Format Codec development. There were proposals of per-
> > > buffer v4l2 controls. IIRC it is currently resolved in that driver by
> > > doing VIDIOC_S_CTRL before QBUF. However the meta data plane approach
> > > looks more interesting to me.
> > 
> > That sounds like a simple (and thus good) solution to me: a button control
> > for resetting the average bitrate calculation. It'd be by far more simple
> > than using the metadata plane for it. Any known drawbacks that I can't see?
> > Even if the number of these parameters grow a little extended controls are
> > fine for the purpose.
> 
> The main issue as far as I know would be to synchronize the control with 
> buffers.

The association to a frame could be done to the next frame to be queued. I
don't see an issue here in the case of this control at least. This is
actually almost analogous to keeping the data on a plane of the buffer. The
delivery of the configuration just takes place through extended controls.

-- 
Regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk