V4L2 encoder APIs

V4L2 encoder APIs

[Sebastien LEDUC]

Hi all
I have seen on the linuxTV web site that there were some on-going discussions related to the Codec API.

In our SoCs, it is the HW encoder that is outputting both the slice data and the headers/metadata, but it does it using separate buffers.

So we are looking at how to expose that using V4L2 APIs.

We were thinking that we could use the MPLANE apis to achieve that, where one plane would contain  the header/metadata and another one for the slice data.

Any opinion on this ? 

Thanks in advance for your inputs

Regards,
Sébastien

Re: V4L2 encoder APIs

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 1224 bytes --]

Le mercredi 13 janvier 2016 à 16:04 +0100, Sebastien LEDUC a écrit :
> Hi all
> I have seen on the linuxTV web site that there were some on-going
> discussions related to the Codec API.
> 
> In our SoCs, it is the HW encoder that is outputting both the slice
> data and the headers/metadata, but it does it using separate buffers.
> 
> So we are looking at how to expose that using V4L2 APIs.
> 
> We were thinking that we could use the MPLANE apis to achieve that,
> where one plane would contain  the header/metadata and another one
> for the slice data.
> 
> Any opinion on this ? 

Discussion I had regarding this kind of integration, whether it was
about non-parsing decoder or slice encoder, is that two planes shall be
used (mplane API will allow seperate allocation of those planes). About
slice encoder, a lot of specification work is needed. I believe the
hardest part and what no one could agree on, is the data structure
standard that would be used to represent the required metadata and
headers. In the end, libv4l should probably have some new feature to
transform slice encoded data into byte-stream so existing software can
use those encoder without porting.

cheers,
Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

RE: V4L2 encoder APIs

[Sebastien LEDUC]

> Discussion I had regarding this kind of integration, whether it was about non-
> parsing decoder or slice encoder, is that two planes shall be used (mplane
> API will allow seperate allocation of those planes). About slice encoder, a lot
> of specification work is needed. I believe the hardest part and what no one
> could agree on, is the data structure standard that would be used to
> represent the required metadata and headers. In the end, libv4l should
> probably have some new feature to transform slice encoded data into byte-
> stream so existing software can use those encoder without porting.


Thanks for your answer
It seems that what you are talking about is the case of a slice encoder, where the user side would generate the header/metadata.
In our case the HW is not a slice encoder, as it also generates the header/metadata, but it can generate it in some separate buffer, and this is required in some use-cases.

So I don’t think there is a lot of specification work needed . We just need a way to output several buffers instead of one, and to specify which one is the header buffer, and which one contains slice data.

Regards
Sebastien

Re: V4L2 encoder APIs

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 506 bytes --]

Le mercredi 13 janvier 2016 à 17:46 +0100, Sebastien LEDUC a écrit :
> So I don’t think there is a lot of specification work needed . We
> just need a way to output several buffers instead of one, and to
> specify which one is the header buffer, and which one contains slice
> data.

Ah, ok, I believe you needed to expose metadata per slice to allow
reordering to happen, that might be CODEC specific. In this case, this
could be signalled the same way we signal I-Frames.

cheers,
Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: V4L2 encoder APIs

[Guennadi Liakhovetski]

Hi Sebastien,

On Wed, 13 Jan 2016, Sebastien LEDUC wrote:

> Hi all I have seen on the linuxTV web site that there were some on-going 
> discussions related to the Codec API.
> 
> In our SoCs, it is the HW encoder that is outputting both the slice data 
> and the headers/metadata, but it does it using separate buffers.
> 
> So we are looking at how to expose that using V4L2 APIs.
> 
> We were thinking that we could use the MPLANE apis to achieve that, 
> where one plane would contain the header/metadata and another one for 
> the slice data.
> 
> Any opinion on this ? 

I think this should be handled in the same way as the output direction. We 
are currently discussing this with several V4L developers. For output we 
have to capture different data types to different buffers, running 
multiplexed on the bus, e.g. over CSI-2. Using the MPLANE API would be one 
option, but you don't want to define a new pixel format for each 
combination of each standard pixel format with each accompanying data 
type, be it metadata or anything else. So, you would have to add support 
for per-plane format, which would contradict the current MPLANE API 
concept.

Therefore we're currently considering a different option of transferring 
different buffer types via different buffer queues. Initially we thought 
about simply using multiple video device nodes. That has a disadvantage 
when the number of those streams is variable and potentially large. So, 
another option is to add support for multiple buffer queues per video 
node. Those buffer queues would then have to get some form of 
identification, perhaps a stream ID. That stream ID would also be used to 
associate those streams to links between subdevice pads. That's all still 
very raw... Quite a bit of design and implementation work ahead.

Thanks
Guennadi

> Thanks in advance for your inputs
> 
> Regards,
> Sébastien
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

RE: V4L2 encoder APIs

[Sebastien LEDUC]

Thanks for your answer

> I think this should be handled in the same way as the output direction. We
> are currently discussing this with several V4L developers. For output we have
> to capture different data types to different buffers, running multiplexed on
> the bus, e.g. over CSI-2. Using the MPLANE API would be one option, but you
> don't want to define a new pixel format for each combination of each
> standard pixel format with each accompanying data type, be it metadata or
> anything else. So, you would have to add support for per-plane format,
> which would contradict the current MPLANE API concept.

Dont know what are the other use cases you are discussing
But in our case there is only one combination:
The first plane should contain the header data, and the second one should contain the slice data

At the end the compressed frame is just the concatenation of the first buffer (header) and the second one (slice data)

So there is no reason to have multiple combinations in the future

> 
> Therefore we're currently considering a different option of transferring
> different buffer types via different buffer queues. Initially we thought about
> simply using multiple video device nodes. That has a disadvantage when the
> number of those streams is variable and potentially large. So, another option
> is to add support for multiple buffer queues per video node. Those buffer
> queues would then have to get some form of identification, perhaps a
> stream ID. That stream ID would also be used to associate those streams to
> links between subdevice pads. That's all still very raw... Quite a bit of design
> and implementation work ahead.
> 


In our case using separate buffer queues does not look very appropriate, because there is a strong ordering constraint, and we always want to get a header and slice data together

So my current feeling is really that MPLANE is the best fit for our need

Regards
Sebastien