Proposed updates and guidelines for MPEG-2, H.264 and H.265 stateless support

Paul Kocialkowski
2019-05-15T10:09:51Z

Hi,

With the Rockchip stateless VPU driver in the works, we now have a
better idea of what the situation is like on platforms other than
Allwinner. This email shares my conclusions about the situation and how
we should update the MPEG-2, H.264 and H.265 controls accordingly.

- Per-slice decoding

We've discussed this one already[0] and Hans has submitted a patch[1]
to implement the required core bits. When we agree it looks good, we
should lift the restriction that all slices must be concatenated and
have them submitted as individual requests.

One question is what to do about other controls. I feel like it would
make sense to always pass all the required controls for decoding the
slice, including the ones that don't change across slices. But there
may be no particular advantage to this and only downsides. Not doing it
and relying on the "control cache" can work, but we need to specify
that only a single stream can be decoded per opened instance of the
v4l2 device. This is the assumption we're going with for handling
multi-slice anyway, so it shouldn't be an issue.

- Annex-B formats

I don't think we have really reached a conclusion on the pixel formats
we want to expose. The main issue is how to deal with codecs that need
the full slice NALU with start code, where the slice_header is
duplicated in raw bitstream, when others are fine with just the encoded
slice data and the parsed slice header control.

My initial thinking was that we'd need 3 formats:
- One that only takes only the slice compressed data (without raw slice
header and start code);
- One that takes both the NALU data (including start code, raw header
and compressed data) and slice header controls;
- One that takes the NALU data but no slice header.

But I no longer think the latter really makes sense in the context of
stateless video decoding.

A side-note: I think we should definitely have data offsets in every
case, so that implementations can just push the whole NALU regardless
of the format if they're lazy.

- Dropping the DPB concept in H.264/H.265

As far as I could understand, the decoded picture buffer (DPB) is a
concept that only makes sense relative to a decoder implementation. The
spec mentions how to manage it with the Hypothetical reference decoder
(Annex C), but that's about it.

What's really in the bitstream is the list of modified short-term and
long-term references, which is enough for every decoder.

For this reason, I strongly believe we should stop talking about DPB in
the controls and just pass these lists agremented with relevant
information for userspace.

I think it should be up to the driver to maintain a DPB and we could
have helpers for common cases. For instance, the rockchip decoder needs
to keep unused entries around[2] and cedrus has the same requirement
for H.264. However for cedrus/H.265, we don't need to do any book-
keeping in particular and can manage with the lists from the bitstream
directly.

- Using flags

The current MPEG-2 controls have lots of u8 values that can be
represented as flags. Using flags also helps with padding.
It's unlikely that we'll get more than 64 flags, so using a u64 by
default for that sounds fine (we definitely do want to keep some room
available and I don't think using 32 bits as a default is good enough).

I think H.264/HEVC per-control flags should also be moved to u64.

- Clear split of controls and terminology

Some codecs have explicit NAL units that are good fits to match as
controls: e.g. slice header, pps, sps. I think we should stick to the
bitstream element names for those.

For H.264, that would suggest the following changes:
- renaming v4l2_ctrl_h264_decode_param to v4l2_ctrl_h264_slice_header;
- killing v4l2_ctrl_h264_decode_param and having the reference lists
where they belong, which seems to be slice_header;

I'm up for preparing and submitting these control changes and updating
cedrus if they seem agreeable.

What do you think?

Cheers,

Paul

[0]: https://lkml.org/lkml/2019/3/6/82
[1]: https://patchwork.linuxtv.org/patch/55947/
[2]: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/4d7cb46539a93bb6acc802f5a46acddb5aaab378

--
Paul Kocialkowski, Bootlin
Embedded Linux and kernel engineering