V4L2 Advanced Codec questions

V4L2 Advanced Codec questions

[Steven Toth]

Hello!

I want to open a couple of HVR22xx items up for discussion.

The HVR-22xx analog encoder is capable of encoded to all kinds of video and 
audio codecs in various containers formats.

 From memory, wm9, mpeg4, mpeg2, divx, AAC, AC3, Windows audio codecs in asf, 
ts, ps, avi containers, depending on various firmware license enablements and 
configuration options. Maybe more, maybe, I'll draw up a complete list when I 
begin to focus on analog.

Any single encoder on the HVR22xx can produce (if licensed) any of the formats 
above. However, due to a lack of CPU horsepower in the RISC engine, the board is 
not completely symmetrical when the encoders are running concurrently. This is 
the main reason why Hauppauge have disabled these features in the windows driver.

It's possible for example to get two concurrent MPEG2 PS streams but only if the 
bitrate is limited to 6Mbps, which we also do in the windows driver.

Apart from the fact that we (the LinuxTV community) will need to determine 
what's possible concurrently, and what isn't, it does raise interesting issues 
for the V4L2 API.

So, how do we expose this advanced codec and hardware encoder limitation 
information through v4l2 to the applications?

Do we, don't we?

Suggestions?

- Steve

Re: V4L2 Advanced Codec questions

[Janne Grunau]

On Thu, Mar 26, 2009 at 11:59:11AM -0400, Steven Toth wrote:
> 
> I want to open a couple of HVR22xx items up for discussion.
> 
> The HVR-22xx analog encoder is capable of encoded to all kinds of video and 
> audio codecs in various containers formats.
> 
>  From memory, wm9, mpeg4, mpeg2, divx, AAC, AC3, Windows audio codecs in asf, 
> ts, ps, avi containers, depending on various firmware license enablements and 
> configuration options. Maybe more, maybe, I'll draw up a complete list when I 
> begin to focus on analog.

Well, the current list is already impressive. I guess MPEG audio layer 2 or 3
are missing. 

> Any single encoder on the HVR22xx can produce (if licensed) any of the formats 
> above. However, due to a lack of CPU horsepower in the RISC engine, the board is 
> not completely symmetrical when the encoders are running concurrently. This is 
> the main reason why Hauppauge have disabled these features in the windows driver.

I guess that will also be a problem within the v4l2 driver since user
space has generally no idea if two video devices share hardware.

> It's possible for example to get two concurrent MPEG2 PS streams but only if the 
> bitrate is limited to 6Mbps, which we also do in the windows driver.
> 
> Apart from the fact that we (the LinuxTV community) will need to determine 
> what's possible concurrently, and what isn't, it does raise interesting issues 
> for the V4L2 API.
> 
> So, how do we expose this advanced codec and hardware encoder limitation 
> information through v4l2 to the applications?

One possibility is adding an mpeg control which request concurrent usage
of the hardware encoder and limits the control values to allow
concurrent usage. If the control is not set, the hardware encoder will
be only useable on one device. User space applications need to discover
the devices sharing hardware and can set the concurrent mode if they wishes.

Current applications won't break but can only use hardware decoding on a
single device. A module option could force concurrent mode in which case
only the encoding parameters are limited but both devices can use
hardware encoding.

Janne

Re: V4L2 Advanced Codec questions

[Hans Verkuil]

On Thursday 26 March 2009 16:59:11 Steven Toth wrote:
> Hello!
>
> I want to open a couple of HVR22xx items up for discussion.
>
> The HVR-22xx analog encoder is capable of encoded to all kinds of video
> and audio codecs in various containers formats.
>
>  From memory, wm9, mpeg4, mpeg2, divx, AAC, AC3, Windows audio codecs in
> asf, ts, ps, avi containers, depending on various firmware license
> enablements and configuration options. Maybe more, maybe, I'll draw up a
> complete list when I begin to focus on analog.
>
> Any single encoder on the HVR22xx can produce (if licensed) any of the
> formats above. However, due to a lack of CPU horsepower in the RISC
> engine, the board is not completely symmetrical when the encoders are
> running concurrently. This is the main reason why Hauppauge have disabled
> these features in the windows driver.
>
> It's possible for example to get two concurrent MPEG2 PS streams but only
> if the bitrate is limited to 6Mbps, which we also do in the windows
> driver.
>
> Apart from the fact that we (the LinuxTV community) will need to
> determine what's possible concurrently, and what isn't, it does raise
> interesting issues for the V4L2 API.
>
> So, how do we expose this advanced codec and hardware encoder limitation
> information through v4l2 to the applications?
>
> Do we, don't we?

Hi Steve,

If I understand it correctly, then a single analog source can be encoded to 
multiple formats at the same time, right?

Or is it that multiple analog sources can each be encoded to some format? Or 
a combination of both?

Is there a limit to the number of concurrent encoders (except for CPU 
horsepower)?

Basically, since you can have multiple encoders, you also need multiple 
videoX nodes, once for each encoder. And I would expect that an application 
can just setup each encoder. Whenever you start an encoder the driver might 
either accept it or return -ENOSPC if there aren't enough resources.

You have to document the restrictions in a document, but otherwise I don't 
see any reason why implementing this would cause any problems.

Adding new containers and codecs is easy: just add the missing ones to enum 
v4l2_mpeg_stream_type, v4l2_mpeg_audio_encoding and 
v4l2_mpeg_video_encoding and add any additional controls that are needed to 
implement each codec/container.

In theory you can reduce the number of possible containers/codecs/bitrates 
in the controls according to the remaining resources. But I think that will 
be too complicated to do for too little gain, not only in the driver but 
also in the application.

Regards,

	Hans

-- 
Hans Verkuil - video4linux developer - sponsored by TANDBERG

Re: V4L2 Advanced Codec questions

[Steven Toth]

Hans Verkuil wrote:
> On Thursday 26 March 2009 16:59:11 Steven Toth wrote:
>> Hello!
>>
>> I want to open a couple of HVR22xx items up for discussion.
>>
>> The HVR-22xx analog encoder is capable of encoded to all kinds of video
>> and audio codecs in various containers formats.
>>
>>  From memory, wm9, mpeg4, mpeg2, divx, AAC, AC3, Windows audio codecs in
>> asf, ts, ps, avi containers, depending on various firmware license
>> enablements and configuration options. Maybe more, maybe, I'll draw up a
>> complete list when I begin to focus on analog.
>>
>> Any single encoder on the HVR22xx can produce (if licensed) any of the
>> formats above. However, due to a lack of CPU horsepower in the RISC
>> engine, the board is not completely symmetrical when the encoders are
>> running concurrently. This is the main reason why Hauppauge have disabled
>> these features in the windows driver.
>>
>> It's possible for example to get two concurrent MPEG2 PS streams but only
>> if the bitrate is limited to 6Mbps, which we also do in the windows
>> driver.
>>
>> Apart from the fact that we (the LinuxTV community) will need to
>> determine what's possible concurrently, and what isn't, it does raise
>> interesting issues for the V4L2 API.
>>
>> So, how do we expose this advanced codec and hardware encoder limitation
>> information through v4l2 to the applications?
>>
>> Do we, don't we?
> 
> Hi Steve,
> 
> If I understand it correctly, then a single analog source can be encoded to 
> multiple formats at the same time, right?

No.

> 
> Or is it that multiple analog sources can each be encoded to some format? Or 
> a combination of both?

Multiple analog sources can produce multiple formats, CPU power permitting.

> 
> Is there a limit to the number of concurrent encoders (except for CPU 
> horsepower)?

Not that I'm aware of, yet.

> 
> Basically, since you can have multiple encoders, you also need multiple 
> videoX nodes, once for each encoder. And I would expect that an application 
> can just setup each encoder. Whenever you start an encoder the driver might 
> either accept it or return -ENOSPC if there aren't enough resources.

This is fine, and expected.

> 
> You have to document the restrictions in a document, but otherwise I don't 
> see any reason why implementing this would cause any problems.
> 
> Adding new containers and codecs is easy: just add the missing ones to enum 
> v4l2_mpeg_stream_type, v4l2_mpeg_audio_encoding and 
> v4l2_mpeg_video_encoding and add any additional controls that are needed to 
> implement each codec/container.

Ahh, this is the magic information I was looking for.

> 
> In theory you can reduce the number of possible containers/codecs/bitrates 
> in the controls according to the remaining resources. But I think that will 
> be too complicated to do for too little gain, not only in the driver but 
> also in the application.

I think this is going to be the major issue and it will start to reflect itself 
through the API into the application. We'll see, maybe not.

Thanks,

Steve

Re: V4L2 Advanced Codec questions

[Hans Verkuil]

On Saturday 28 March 2009 18:03:47 Steven Toth wrote:
> Hans Verkuil wrote:
> > On Thursday 26 March 2009 16:59:11 Steven Toth wrote:
> >> Hello!
> >>
> >> I want to open a couple of HVR22xx items up for discussion.
> >>
> >> The HVR-22xx analog encoder is capable of encoded to all kinds of
> >> video and audio codecs in various containers formats.
> >>
> >>  From memory, wm9, mpeg4, mpeg2, divx, AAC, AC3, Windows audio codecs
> >> in asf, ts, ps, avi containers, depending on various firmware license
> >> enablements and configuration options. Maybe more, maybe, I'll draw up
> >> a complete list when I begin to focus on analog.
> >>
> >> Any single encoder on the HVR22xx can produce (if licensed) any of the
> >> formats above. However, due to a lack of CPU horsepower in the RISC
> >> engine, the board is not completely symmetrical when the encoders are
> >> running concurrently. This is the main reason why Hauppauge have
> >> disabled these features in the windows driver.
> >>
> >> It's possible for example to get two concurrent MPEG2 PS streams but
> >> only if the bitrate is limited to 6Mbps, which we also do in the
> >> windows driver.
> >>
> >> Apart from the fact that we (the LinuxTV community) will need to
> >> determine what's possible concurrently, and what isn't, it does raise
> >> interesting issues for the V4L2 API.
> >>
> >> So, how do we expose this advanced codec and hardware encoder
> >> limitation information through v4l2 to the applications?
> >>
> >> Do we, don't we?
> >
> > Hi Steve,
> >
> > If I understand it correctly, then a single analog source can be
> > encoded to multiple formats at the same time, right?
>
> No.
>
> > Or is it that multiple analog sources can each be encoded to some
> > format? Or a combination of both?
>
> Multiple analog sources can produce multiple formats, CPU power
> permitting.

So it is always: 'one source -> one encoder', right? Never 'one source -> 
multiple encoders'?

>
> > Is there a limit to the number of concurrent encoders (except for CPU
> > horsepower)?
>
> Not that I'm aware of, yet.

There must be a limit to the number of sources? Or can this device act as a 
codec as well: you present it with raw video from memory and it compresses 
it to e.g. mpeg?

>
> > Basically, since you can have multiple encoders, you also need multiple
> > videoX nodes, once for each encoder. And I would expect that an
> > application can just setup each encoder. Whenever you start an encoder
> > the driver might either accept it or return -ENOSPC if there aren't
> > enough resources.
>
> This is fine, and expected.
>
> > You have to document the restrictions in a document, but otherwise I
> > don't see any reason why implementing this would cause any problems.
> >
> > Adding new containers and codecs is easy: just add the missing ones to
> > enum v4l2_mpeg_stream_type, v4l2_mpeg_audio_encoding and
> > v4l2_mpeg_video_encoding and add any additional controls that are
> > needed to implement each codec/container.
>
> Ahh, this is the magic information I was looking for.
>
> > In theory you can reduce the number of possible
> > containers/codecs/bitrates in the controls according to the remaining
> > resources. But I think that will be too complicated to do for too
> > little gain, not only in the driver but also in the application.
>
> I think this is going to be the major issue and it will start to reflect
> itself through the API into the application. We'll see, maybe not.

The first step will be to get a single encoder going :-)

At this stage I don't know enough about the capabilities of the hardware to 
tell how important it is to have such info available. I suspect any 
solution to this might well need something like a media controller device 
about which I've written before, which can be used to present such meta 
information.

I'm ever more of the opinion that v4l needs such a top-level device as v4l 
hardware becomes more complex.

Regards,

	Hans

-- 
Hans Verkuil - video4linux developer - sponsored by TANDBERG