Re: [yocto-patches] [meta-rockchip][scarthgap][PATCH 09/10] enable HW VPU decoding for SoCs that have stateless VPUs

[Trevor Woerner <twoerner@gmail.com>]

On Mon 2024-09-09 @ 06:01:30 PM, Quentin Schulz via lists.yoctoproject.org wrote:
> Hi Trevor,
> 
> On 9/9/24 4:55 PM, Trevor Woerner via lists.yoctoproject.org wrote:
> > From: Quentin Schulz <quentin.schulz@cherry.de>
> > 
> > v4l2codecs is the gstreamer plugin for V4L2 stateless video hardware
> > decoding. The Rockchip SoCs that have a VPU all seems to be based on
> > Hantro, RKVDEC or RKVDECv2, all stateless encoding/decoding VPUs.
> > 
> > Therefore, let's enable VPU decoding in Gstreamer whenever possible,
> > when the SoC supports it.
> > 
> > PX30, RK3066, RK3188, RK3288, RK3328, RK3399, RK356x and RK3588(s) all
> > have at least one Hantro VPU.
> > 
> > RK3328, RK3399, RK356x and RK3588(s) all have at least one
> > RKVDEC/RKVDECv2 VPU (though not necessarily supported in the upstream
> > kernel just yet).
> > 
> > === PX30
> > Tested on PX30 Ringneck with with Haikou+Haikou Video Demo adapter:
> > 
> > $ gst-launch-1.0 filesrc location=$FILE ! parsebin ! v4l2slh264dec ! waylandsink
> > 
> > with FILE storing the path to any h264 file, e.g.
> > https://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_720p_h264.mov
> > https://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_h264.mov
> > 
> > Needed packages are:
> > - weston
> > - gstreamer1.0-plugins-bad (for waylandsink and v4l2slh264dec)
> > - gstreamer1.0-plugins-base (for parsebin)
> > 
> 
> Weston output is broken on Scarthgap. c.f.
> https://lore.kernel.org/all/1f84dd9c-bdef-4099-9dba-04b445a7ee0a@cherry.de/
> 
> > A few frames are dropped every other second for 1080p but otherwise
> > smooth.
> > 
> > === RK3399
> > Tested on RK3399 Puma with Haikou:
> > 
> > $ gst-launch-1.0 filesrc location=$FILE ! parsebin ! v4l2slh264dec ! waylandsink
> > 
> > with FILE storing the path to any h264 file, e.g.
> > https://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_h264.mov
> > https://download.blender.org/demo/movies/BBB/bbb_sunflower_2160p_30fps_normal.mp4.zip
> > 
> > Needed packages are:
> > - weston
> > - gstreamer1.0-plugins-bad (for waylandsink and v4l2codecs)
> > - gstreamer1.0-plugins-base (for parsebin)
> > 
> 
> Weston output is broken on Scarthgap. c.f.
> https://lore.kernel.org/all/1f84dd9c-bdef-4099-9dba-04b445a7ee0a@cherry.de/
> 
> > === RK3588
> > 
> > Tested on a RK3588 Tiger with Haikou+Haikou Video Demo adapter - on a
> > downstream v6.6 (upstream-based, not Rockchip BSP-based) with DSI
> > patches - :
> > 
> > $ gst-launch-1.0 filesrc location=$FILE ! parsebin ! v4l2slav1dec ! fakesink
> > 
> > with FILE storing the path to any AV1 file, e.g.
> > http://download.opencontent.netflix.com.s3.amazonaws.com/AV1/cmaf/spark-8b-59.94fps/spark_606kbps_432p.mp4
> > https://woolyss.com/f/av1-nosound-chimera.mp4
> > https://woolyss.com/f/av1-opus-sita.webm
> > 
> > Needed packages are:
> > - gstreamer1.0-plugins-bad (for fakesink and v4l2slav1dec)
> > - gstreamer1.0-plugins-base (for parsebin)
> > 
> 
> Stateless AV1 decoding is not available in the mesa release used on
> Scarthgap.
> 
> So...
> 
> RK3588 seems to have gained upstream support for VDPU121 which should allow
> h264, mpeg2 and vp8 decoding. Building a linux-next kernel after tag
> next-20240829 should have the support AFAICT.
> 
> So, I would retest RK3588 support for those video formats and drop the AV1
> mention from the commit log as it for sure doesn't work because of the
> version of mesa in Scarthgap. We could backport later releases into
> meta-rockchip to gain support for it. I think this **may** be an idea once
> there's support for 24.1 and later merged into OE-Core because it brings
> Panthor (open-source GPU drivers) support to RK3588 which would be a shame
> not to have in an LTS. But let's see when that happens.
> 
> 
> 
> As for the issues with display on PX30 and RK3399, it seems those are
> supposed to be patched in newer releases of gstreamer, c.f.
> https://gitlab.freedesktop.org/wayland/weston/-/issues/839. Note that
> Robert said it would be fixed in gstreamer 1.22 but the linked patch
> only made it to 1.23 (Scarthgap has 1.22, Styhead 1.24).
> 
> Considering what Robert said in the link above, I assume people using
> another compositor than weston may be fine on Scarthgap, so I would suggest
> to backport the patches I submitted in this series to scarthgap anyway? One
> is not forced to use weston to do VPU decoding.
> FYI, I gave ~30min at backporting the gstreamer patches to 1.22.11 but this
> is above my skill set today.
> 
> For those, I would just mention the issue in the README and commit log and
> maybe switch to fakesink in the commit log instead of waylandsink (and drop
> weston from the needed packages).
> 
> What do you think?

To be clear, when I say I've run-tested this build I'm not implying I've
tested every aspect, feature, and functionality of each chip. I've simply
flashed an SD card, applied power, watched it boot to a prompt, log in, and
run something simple like "uname -a" to verify it's using the scarthgap kernel
and not the latest master. I'm not really in a position to test stateless VPU
decoding or anything fancy like that :-)

If there aren't any issues or objections, perhaps I could start by applying
the other patches in this series while considering what to do about this one?
Maybe after the others are applied you'd like to have a go at making a
scarthgap version of this patch (with all the proper commit messages etc)?

Best regards,
	Trevor