Re: [PATCH v4 00/11] media: rkvdec: Add H.264 High 10 and 4:2:2 profile support

[Nicolas Dufresne <nicolas.dufresne@collabora.com>]

Hi,

Le dimanche 16 juin 2024 à 11:47 +0200, Diederik de Haas a écrit :
> On Sunday, 5 November 2023 17:54:59 CEST Jonas Karlman wrote:
> > This is a revival of a 3 year old series [1] now that NV15/NV20/NV30 support
> > for display driver have landed in mainline tree.
> > 
> > This series adds H.264 High 10 and 4:2:2 profile support to the Rockchip
> > Video Decoder driver.
> > 
> > Patch 1 adds helpers for calculating plane bytesperline and sizeimage.
> > Patch 2 adds two new pixelformats for semi-planer 10-bit 4:2:0/4:2:2 YUV.
> > 
> > Patch 3 change to use bytesperline and buffer height to configure strides.
> > Patch 4 change to use values from SPS/PPS control to configure the HW.
> > Patch 5 remove an unnecessary call to validate sps at streaming start.
> > 
> > Patch 6-10 refactor code to support filtering of CAPUTRE formats based
> > on the image format returned from a get_image_fmt ops.
> > 
> > Patch 11 adds final bits to support H.264 High 10 and 4:2:2 profiles.
> > 
> > Tested on a ROCK Pi 4 (RK3399) and Rock64 (RK3328):
> > 
> >   v4l2-compliance 1.24.1, 64 bits, 64-bit time_t
> >   ...
> >   Total for rkvdec device /dev/video1: 46, Succeeded: 46, Failed: 0,
> > Warnings: 0
> > 
> >   Running test suite JVT-FR-EXT with decoder FFmpeg-H.264-V4L2-request
> >   ...
> >   Ran 65/69 tests successfully
> > 
> >   Running test suite JVT-AVC_V1 with decoder FFmpeg-H.264-V4L2-request
> >   ...
> >   Ran 127/135 tests successfully
> > 
> > Before this series:
> > 
> >   Running test suite JVT-FR-EXT with decoder FFmpeg-H.264-V4L2-request
> >   ...
> >   Ran 44/69 tests successfully
> > 
> > ...
> > 
> > Following commits adds support for NV15/NV20/NV30 to VOP driver:
> > 728c15b4b5f3 ("drm/fourcc: Add NV20 and NV30 YUV formats")
> > d4b384228562 ("drm/rockchip: vop: Add NV15, NV20 and NV30 support")
> > 
> > To fully runtime test this series you may need above drm commits and ffmpeg
> > patches from [2], this series and drm patches is also available at [3].
> > 
> > [1]
> > https://lore.kernel.org/linux-media/20200706215430.22859-1-jonas@kwiboo.se/
> > [2] https://github.com/Kwiboo/FFmpeg/commits/v4l2-request-n6.1-dev/ [3]
> > https://github.com/Kwiboo/linux-rockchip/commits/linuxtv-rkvdec-high-10-v4/
> > [4] https://gist.github.com/Kwiboo/f4ac15576b2c72887ae2bc5d58b5c865 [5]
> > https://gist.github.com/Kwiboo/459a1c8f1dcb56e45dc7a7a29cc28adf
> 
> Reviving this old thread now that rkvdec2 'stuff' emerged.
> I have (actually) done quite some tests with this (and "media: rkvdec: Add 
> HEVC backend" patch set) and they have been part of my kernel builds ever 
> since.
> I _think_, but don't know, that is relevant for Andy's question:
> 
> On zondag 16 juni 2024 08:58:20 CEST Andy Yan <andyshrk@163.com> wrote:
> > How can I test these patches? Do they require any additional userspace
> > patches?
> 
> I have the same question and I think you'd need this and the HEVC patch set 
> and then also patch FFmpeg and then it should enable HW acceleration.
> So my question boils down to: with the rkvdec2 patch set, should V4L2-requests 
> now also work with rkvdec, so not just Hantro anymore?

FFmpeg changes are still downstream, and different people (even within
LibreELEC) seems to have slightly different version or alteration. It would be
really nice if this work could move upstream FFMpeg so that we can be more sure
what what "working with FFmpeg v4l2-requests" means.

Meanwhile, support in upstream GStreamer is stable on Hantro G2 and Mediatek
VCODEC. In theory, it works fine with RKVDEC, and it will certainly work with
RKVDEC2 when we get to write the HEVC support.

> 
> BTW: the libdrm commits have been merged upstream quite some time ago, so if 
> you have a recent version of that, you don't need to patch that.
> If you use FFmpeg 7.0, then Jonas has a branch for that too (haven't tried it 
> yet though).
> 
> FWIW: my test results were a bit mixed. I didn't post them before as I don't 
> fully/really understand this 'video stuff', and I didn't want you all to suffer 
> from what was likely a PEBKAC issue.
> 
> On my PineTab2 (rk3566) I had some h.264 videos HW accelerated, but not all. 
> My guess is that it's related to the resolution. 1920x1080 worked, while it 
> didn't work with a 1280x640 video. The video still played, just not HW 
> accelerated. IOW: improvements in some and otherwise it was just rendered by 
> the CPU (I think), just like before.

This is because all rk35XX have two hardware video decoders for H.264. This is
not to be be confused with rkvdec which is gone. It has a modified Hantro G1
core (limited to 1080p60) and rkvdec2 core (driver in progress). I don't think
Rockchip really expected the first one to be ever used, but upstream has been
pushy and its now enabled upstream. That has a side effect, which is that
userspace will have to work harder on these platform to pick the right HW for
the task.

> 
> On my Rock64 I got a pink tint with all videos, like described here:
> https://github.com/mpv-player/mpv/issues/12968
> IIUC, that's actually a problem in the lima driver?

Its not clear from the bug report. This visual artefact has been seen with
wayland compositors lately (notably weston). Notably, this can happen if you try
and import NV12 with mesa (panfrost and lima included) but forcing a TEXTURE_2D
target instead of external target. Normally this should be rejected by mesa, but
is accidentally not, and cause miss-render.

Nicolas

> 
> Cheers,
>   Diederik