[PATCH v3 0/2] [v3]Add hantro g1 video decoder support for RK3588

[PATCH v3 0/2] [v3]Add hantro g1 video decoder support for RK3588

[Jianfeng Liu]

This is the v3 version of this series adding hantro g1 video decoder
support for rk3588.

RK3588 has Hantro G1 video decoder known as VDPU121 in TRM of RK3588 which
is capable to decode MPEG2/H.264/VP8 up to 1920x1088. This vpu ip is also
found in RK3568.

Test results from fluster can be found from thread of v2[1][2].

[1] https://lore.kernel.org/all/CAAXNxMT3f68-ptM7Crhrfmn7iwTyJc9pwz4Beob+5beVODaSHQ@mail.gmail.com
[2] https://lore.kernel.org/all/20231230153159.3748580-1-liujianfeng1994@gmail.com


Changes in v3:
 - Drop code in hantro_drv.c because hantro g1 vpu in rk3588 is compatible
with the one in rk3568, only adding devicetree node should work.
 - Change devicetree node name to video-codec@fdb50000 to match the reg
address.
 - Add dt-bindings rockchip,rk3588-vpu compatible with rockchip,rk3568-vpu
 - Link to v2: https://lore.kernel.org/all/20231228131617.3411561-1-liujianfeng1994@gmail.com

Jianfeng Liu (2):
  arm64: dts: rockchip: Add Hantro G1 VPU support for RK3588
  dt-bindings: media: rockchip-vpu: Add rk3588 vpu compatible string

 .../bindings/media/rockchip-vpu.yaml          |  3 +++
 arch/arm64/boot/dts/rockchip/rk3588s.dtsi     | 20 +++++++++++++++++++
 2 files changed, 23 insertions(+)

-- 
2.34.1

[PATCH v3 1/2] arm64: dts: rockchip: Add Hantro G1 VPU support for RK3588

[Jianfeng Liu]

This patch enables Hantro G1 video decoder in RK3588's
devicetree.

Tested with FFmpeg v4l2_request code taken from [1]
with MPEG2, H.264 and VP8 samples.

[1] https://github.com/LibreELEC/LibreELEC.tv/blob/master/packages/multimedia/ffmpeg/patches/v4l2-request/ffmpeg-001-v4l2-request.patch

Signed-off-by: Jianfeng Liu <liujianfeng1994@gmail.com>
---
 arch/arm64/boot/dts/rockchip/rk3588s.dtsi | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
index 5fb0baf8a..ab2792162 100644
--- a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
@@ -640,6 +640,26 @@ i2c0: i2c@fd880000 {
 		status = "disabled";
 	};
 
+	vpu: video-codec@fdb50000 {
+		compatible = "rockchip,rk3588-vpu", "rockchip,rk3568-vpu";
+		reg = <0x0 0xfdb50000 0x0 0x800>;
+		interrupts = <GIC_SPI 119 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru ACLK_VPU>, <&cru HCLK_VPU>;
+		clock-names = "aclk", "hclk";
+		iommus = <&vdpu_mmu>;
+		power-domains = <&power RK3588_PD_VDPU>;
+	};
+
+	vdpu_mmu: iommu@fdb50800 {
+		compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
+		reg = <0x0 0xfdb50800 0x0 0x40>;
+		interrupts = <GIC_SPI 118 IRQ_TYPE_LEVEL_HIGH 0>;
+		clock-names = "aclk", "iface";
+		clocks = <&cru ACLK_VPU>, <&cru HCLK_VPU>;
+		power-domains = <&power RK3588_PD_VDPU>;
+		#iommu-cells = <0>;
+	};
+
 	vop: vop@fdd90000 {
 		compatible = "rockchip,rk3588-vop";
 		reg = <0x0 0xfdd90000 0x0 0x4200>, <0x0 0xfdd95000 0x0 0x1000>;
-- 
2.34.1

Re: [PATCH v3 1/2] arm64: dts: rockchip: Add Hantro G1 VPU support for RK3588

[Nicolas Dufresne]

Hi Jianfeng,

Le dimanche 31 décembre 2023 à 23:11 +0800, Jianfeng Liu a écrit :
> This patch enables Hantro G1 video decoder in RK3588's
> devicetree.
> 
> Tested with FFmpeg v4l2_request code taken from [1]
> with MPEG2, H.264 and VP8 samples.

In general, we ask submitters to share a fluster [0] score (this is just to
demonstrate that the integration has been well validated). LibreELEC community
have patch to enable this ffmpeg fork. I don't expect any surprise here, and you
can just reply to my email here.

Alternatively, this test tool is shipped in latest Debian, and with the
appropriate GStreamer packages you'll be able to run the VP8 and H.264 tests.

Nicolas

[0] https://github.com/fluendo/fluster

> 
> [1] https://github.com/LibreELEC/LibreELEC.tv/blob/master/packages/multimedia/ffmpeg/patches/v4l2-request/ffmpeg-001-v4l2-request.patch
> 
> Signed-off-by: Jianfeng Liu <liujianfeng1994@gmail.com>
> ---
>  arch/arm64/boot/dts/rockchip/rk3588s.dtsi | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> diff --git a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
> index 5fb0baf8a..ab2792162 100644
> --- a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
> +++ b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
> @@ -640,6 +640,26 @@ i2c0: i2c@fd880000 {
>  		status = "disabled";
>  	};
>  
> +	vpu: video-codec@fdb50000 {
> +		compatible = "rockchip,rk3588-vpu", "rockchip,rk3568-vpu";
> +		reg = <0x0 0xfdb50000 0x0 0x800>;
> +		interrupts = <GIC_SPI 119 IRQ_TYPE_LEVEL_HIGH 0>;
> +		clocks = <&cru ACLK_VPU>, <&cru HCLK_VPU>;
> +		clock-names = "aclk", "hclk";
> +		iommus = <&vdpu_mmu>;
> +		power-domains = <&power RK3588_PD_VDPU>;
> +	};
> +
> +	vdpu_mmu: iommu@fdb50800 {
> +		compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
> +		reg = <0x0 0xfdb50800 0x0 0x40>;
> +		interrupts = <GIC_SPI 118 IRQ_TYPE_LEVEL_HIGH 0>;
> +		clock-names = "aclk", "iface";
> +		clocks = <&cru ACLK_VPU>, <&cru HCLK_VPU>;
> +		power-domains = <&power RK3588_PD_VDPU>;
> +		#iommu-cells = <0>;
> +	};
> +
>  	vop: vop@fdd90000 {
>  		compatible = "rockchip,rk3588-vop";
>  		reg = <0x0 0xfdd90000 0x0 0x4200>, <0x0 0xfdd95000 0x0 0x1000>;

Re: [PATCH v3 1/2] arm64: dts: rockchip: Add Hantro G1 VPU support for RK3588

[amazingfate]

Hi Nicolas,

Wed, 17 Jan 2024 14:13:19 -0500, Nicolas Dufresne wrote:
>In general, we ask submitters to share a fluster [0] score (this is just to
>demonstrate that the integration has been well validated). LibreELEC community
>have patch to enable this ffmpeg fork. I don't expect any surprise here, and you
>can just reply to my email here.

Here are fluster test result with ffmpeg v6.0 patched with v4l2 request patches
from LibreELEC and gstreamer 1.22:
https://gist.github.com/amazingfate/1675389df382ff266d6f1318fd4675fd

I am using Kwiboo's fluster fork[1] to run ffmpeg v4l2 request tests

Jianfeng

[1] https://github.com/Kwiboo/fluster/tree/v4l2-request

Re: [PATCH v3 1/2] arm64: dts: rockchip: Add Hantro G1 VPU support for RK3588

[Nicolas Dufresne]

Le jeudi 18 janvier 2024 à 16:06 +0800, amazingfate a écrit :
> Hi Nicolas,
> 
> Wed, 17 Jan 2024 14:13:19 -0500, Nicolas Dufresne wrote:
> > In general, we ask submitters to share a fluster [0] score (this is just to
> > demonstrate that the integration has been well validated). LibreELEC community
> > have patch to enable this ffmpeg fork. I don't expect any surprise here, and you
> > can just reply to my email here.
> 
> Here are fluster test result with ffmpeg v6.0 patched with v4l2 request patches
> from LibreELEC and gstreamer 1.22:
> https://gist.github.com/amazingfate/1675389df382ff266d6f1318fd4675fd
> 
> I am using Kwiboo's fluster fork[1] to run ffmpeg v4l2 request tests

Thank you very much, these are the expected results for this HW, indicating
everything works as expected.

Nicolas

> 
> Jianfeng
> 
> [1] https://github.com/Kwiboo/fluster/tree/v4l2-request

[PATCH v3 2/2] dt-bindings: media: rockchip-vpu: Add rk3588 vpu compatible string

[Jianfeng Liu]

Add Hantro G1 VPU compatible string for RK3588.
RK3588 has the same Hantro G1 ip as RK3568, which are both
known as VDPU121 in TRM of RK3568 and RK3588.

Signed-off-by: Jianfeng Liu <liujianfeng1994@gmail.com>
---
 Documentation/devicetree/bindings/media/rockchip-vpu.yaml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/Documentation/devicetree/bindings/media/rockchip-vpu.yaml b/Documentation/devicetree/bindings/media/rockchip-vpu.yaml
index c57e1f488..16886bccf 100644
--- a/Documentation/devicetree/bindings/media/rockchip-vpu.yaml
+++ b/Documentation/devicetree/bindings/media/rockchip-vpu.yaml
@@ -31,6 +31,9 @@ properties:
       - items:
           - const: rockchip,rk3228-vpu
           - const: rockchip,rk3399-vpu
+      - items:
+          - const: rockchip,rk3588-vpu
+          - const: rockchip,rk3568-vpu
 
   reg:
     maxItems: 1
-- 
2.34.1

Re: [PATCH v3 2/2] dt-bindings: media: rockchip-vpu: Add rk3588 vpu compatible string

[Conor Dooley]

[-- Attachment #1: Type: text/plain, Size: 362 bytes --]

On Sun, Dec 31, 2023 at 11:11:12PM +0800, Jianfeng Liu wrote:
> Add Hantro G1 VPU compatible string for RK3588.
> RK3588 has the same Hantro G1 ip as RK3568, which are both
> known as VDPU121 in TRM of RK3568 and RK3588.
> 
> Signed-off-by: Jianfeng Liu <liujianfeng1994@gmail.com>

Acked-by: Conor Dooley <conor.dooley@microchip.com>

Cheers,
Conor.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v3 2/2] dt-bindings: media: rockchip-vpu: Add rk3588 vpu compatible string

[Heiko Stübner]

Am Sonntag, 31. Dezember 2023, 16:11:12 CET schrieb Jianfeng Liu:
> Add Hantro G1 VPU compatible string for RK3588.
> RK3588 has the same Hantro G1 ip as RK3568, which are both
> known as VDPU121 in TRM of RK3568 and RK3588.
> 
> Signed-off-by: Jianfeng Liu <liujianfeng1994@gmail.com>
> ---
>  Documentation/devicetree/bindings/media/rockchip-vpu.yaml | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/media/rockchip-vpu.yaml b/Documentation/devicetree/bindings/media/rockchip-vpu.yaml
> index c57e1f488..16886bccf 100644
> --- a/Documentation/devicetree/bindings/media/rockchip-vpu.yaml
> +++ b/Documentation/devicetree/bindings/media/rockchip-vpu.yaml
> @@ -31,6 +31,9 @@ properties:
>        - items:
>            - const: rockchip,rk3228-vpu
>            - const: rockchip,rk3399-vpu
> +      - items:
> +          - const: rockchip,rk3588-vpu
> +          - const: rockchip,rk3568-vpu

at least the rk3588 has multiple v{d/e{.}pu blocks, so this compatible
is very misleading. As you mention in the commit description already,
this describes the vdpu121, so can we please use that in the compatible
as well to distinguish this one from the others that hopefully will come
later?

Thanks
Heiko

Re: [PATCH v3 2/2] dt-bindings: media: rockchip-vpu: Add rk3588 vpu compatible string

[Jianfeng Liu]

Hi Heiko,

Fri, 15 Mar 2024 13:22:18 +0100, Heiko Stübner wrote:
>at least the rk3588 has multiple v{d/e{.}pu blocks, so this compatible
>is very misleading. As you mention in the commit description already,
>this describes the vdpu121, so can we please use that in the compatible
>as well to distinguish this one from the others that hopefully will come
>later?

Since vpdu121 is in both rk3568 and rk3588, maybe it's better to change
rk3568 as well. But rk3568-vpu has been in kernel for a long time so this
would be a big change. I will submit v4 with rockchip,rk3588-vdpu121.

Regards,
Jianfeng

Re: [PATCH v3 0/2] [v3]Add hantro g1 video decoder support for RK3588

[Nicolas Dufresne]

Le dimanche 31 décembre 2023 à 23:11 +0800, Jianfeng Liu a écrit :
> This is the v3 version of this series adding hantro g1 video decoder
> support for rk3588.
> 
> RK3588 has Hantro G1 video decoder known as VDPU121 in TRM of RK3588 which
> is capable to decode MPEG2/H.264/VP8 up to 1920x1088. This vpu ip is also
> found in RK3568.

The only concern I have is that we rejected enabling this VPU on RK3399, since
having two H.264 decoders was confusing userspace softwares at the time, causing
the slow one to be picked sometimes. I know the selection is not fully
implemented in GStreamer either, not sure for LibreELEC ffmpeg fork, Chromium or
Cros-codec.

Of course, its not a problem now, but only when someone provides an rkvdec2
driver that enables the much more capable HW decoder. Shall we hold on H.264
mainline for now ?

Nicolas

> 
> Test results from fluster can be found from thread of v2[1][2].
> 
> [1] https://lore.kernel.org/all/CAAXNxMT3f68-ptM7Crhrfmn7iwTyJc9pwz4Beob+5beVODaSHQ@mail.gmail.com
> [2] https://lore.kernel.org/all/20231230153159.3748580-1-liujianfeng1994@gmail.com
> 
> 
> Changes in v3:
>  - Drop code in hantro_drv.c because hantro g1 vpu in rk3588 is compatible
> with the one in rk3568, only adding devicetree node should work.
>  - Change devicetree node name to video-codec@fdb50000 to match the reg
> address.
>  - Add dt-bindings rockchip,rk3588-vpu compatible with rockchip,rk3568-vpu
>  - Link to v2: https://lore.kernel.org/all/20231228131617.3411561-1-liujianfeng1994@gmail.com
> 
> Jianfeng Liu (2):
>   arm64: dts: rockchip: Add Hantro G1 VPU support for RK3588
>   dt-bindings: media: rockchip-vpu: Add rk3588 vpu compatible string
> 
>  .../bindings/media/rockchip-vpu.yaml          |  3 +++
>  arch/arm64/boot/dts/rockchip/rk3588s.dtsi     | 20 +++++++++++++++++++
>  2 files changed, 23 insertions(+)
> 

Re: [PATCH v3 0/2] [v3]Add hantro g1 video decoder support for RK3588

[Heiko Stübner]

Hi Nicolas,

Am Mittwoch, 17. Januar 2024, 20:16:24 CET schrieb Nicolas Dufresne:
> Le dimanche 31 décembre 2023 à 23:11 +0800, Jianfeng Liu a écrit :
> > This is the v3 version of this series adding hantro g1 video decoder
> > support for rk3588.
> > 
> > RK3588 has Hantro G1 video decoder known as VDPU121 in TRM of RK3588 which
> > is capable to decode MPEG2/H.264/VP8 up to 1920x1088. This vpu ip is also
> > found in RK3568.
> 
> The only concern I have is that we rejected enabling this VPU on RK3399, since
> having two H.264 decoders was confusing userspace softwares at the time, causing
> the slow one to be picked sometimes. I know the selection is not fully
> implemented in GStreamer either, not sure for LibreELEC ffmpeg fork, Chromium or
> Cros-codec.
> 
> Of course, its not a problem now, but only when someone provides an rkvdec2
> driver that enables the much more capable HW decoder. Shall we hold on H.264
> mainline for now ?

hmm, reading the TRM the VDPU121 claims to support:
- VP8
- VC1
- MPEG-1,2,4
- H.263
- JPEG

So at least the TRM doesn't list h.264 as supported at all on VDPU121 .
Not sure what is up with that or is Rockchip also just suggesting that h.264
should be in the VDPU381 [where h.264 is listed as featured] ?


Heiko

Re: [PATCH v3 0/2] [v3]Add hantro g1 video decoder support for RK3588

[amazingfate]

Hi Nicolas,

Wed, 17 Jan 2024 14:16:24 -0500, Nicolas Dufresne wrote:
>The only concern I have is that we rejected enabling this VPU on RK3399, since
>having two H.264 decoders was confusing userspace softwares at the time, causing
>the slow one to be picked sometimes. I know the selection is not fully
>implemented in GStreamer either, not sure for LibreELEC ffmpeg fork, Chromium or
>Cros-codec.

Only H264 decoder is disabled for RK3399 in driver code, and RK3399 does
have this VPU node in devicetree to support other codecs like MPEG or VP8.

>Of course, its not a problem now, but only when someone provides an rkvdec2
>driver that enables the much more capable HW decoder. Shall we hold on H.264
>mainline for now ?

RK356x also has the same situation. That is why this patch shares the same
compatible string with RK356X. Before rkvdec2 launching mainline, this
hantro g1 vpu is the only H264 decoder to use on these socs. We can keep
H.264 enabled, and disable it for RK356x and RK3588 just like what we did
to RK3399 when rkvdec2 launches.

Jianfeng

Re: [PATCH v3 0/2] [v3]Add hantro g1 video decoder support for RK3588

[Piotr Oniszczuk]

> Wiadomość napisana przez amazingfate <liujianfeng1994@gmail.com> w dniu 18.01.2024, o godz. 12:30:
> 
> Before rkvdec2 launching mainline

Ooooo - is there concrete plan for this?

Asking as imho rk35xx can be really nice multimedia soc.

Can be…. but is not due current lack of hevc/vp9 hw decode in mainline due lack of rkvdec2 support
This seems to be clear blocker for many multimedia use cases :-(

Having rkvdec2 support via v4l2_request in mainline imho will make rk35xx killer soc for multimedia….

ps: rkvdec support seems available since years on mainline and works well.
So maybe rkvdec code base can be basis to gradually added support for rkvdec2?
   

Re: [PATCH v3 0/2] [v3]Add hantro g1 video decoder support for RK3588

[Nicolas Dufresne]

Le jeudi 18 janvier 2024 à 14:53 +0100, Piotr Oniszczuk a écrit :
> > Wiadomość napisana przez amazingfate <liujianfeng1994@gmail.com> w dniu 18.01.2024, o godz. 12:30:
> > 
> > Before rkvdec2 launching mainline
> 
> Ooooo - is there concrete plan for this?
> 
> Asking as imho rk35xx can be really nice multimedia soc.
> 
> Can be…. but is not due current lack of hevc/vp9 hw decode in mainline due lack of rkvdec2 support
> This seems to be clear blocker for many multimedia use cases :-(
> 
> Having rkvdec2 support via v4l2_request in mainline imho will make rk35xx killer soc for multimedia….
> 
> ps: rkvdec support seems available since years on mainline and works well.
> So maybe rkvdec code base can be basis to gradually added support for rkvdec2?
>   

You'd be surprised that we are still upstreaming HEVC/10bit and 4:2:2 support
(well Jonas is) and are missing a proper solution to some needed HW reset logic
(to fix concurrent decoding). Though all these exist in some form in LibreELEC
patchset.

When this is behind, Jonas reported that he might be interested in looking into
rkvdec2. Consider that his is volunteer work, but it would indeed be amazing.
The HW work blob free (no firmware needed) supports up to 8K (rk3588).  Some
early work on adding encoder interface for this type of HW is also in progress.
The AV1 decoder for rk3588 (which is a chip from Verisillicon, not RK) is
already upstream.

regards,
Nicolas