[PATCH v2 0/2] Enable JPEG encoding on rk3588

[PATCH v2 0/2] Enable JPEG encoding on rk3588

[Emmanuel Gil Peyrot]

Only the JPEG encoder is available for now, although there are patches
for the undocumented VP8 encoder floating around[0].

This has been tested on a rock-5b, resulting in four /dev/video*
encoders.  The userspace program I’ve been using to test them is
Onix[1], using the jpeg-encoder example, it will pick one of these four
at random (but displays the one it picked):
% ffmpeg -i <input image> -pix_fmt yuvj420p temp.yuv
% jpeg-encoder temp.yuv <width> <height> NV12 <quality> output.jpeg

[0] https://patchwork.kernel.org/project/linux-rockchip/list/?series=789885
[1] https://crates.io/crates/onix

Changes since v1:
- Dropped patches 1 and 4.
- Use the proper compatible form, since this device should be fully
  compatible with the VEPU of rk356x.
- Describe where the VEPU121 name comes from, and list other encoders
  and decoders present in this SoC.
- Properly test the device tree changes, I previously couldn’t since I
  was using a too recent version of python-jsonschema…

Emmanuel Gil Peyrot (2):
  media: dt-binding: media: Document rk3588’s VEPU121
  arm64: dts: rockchip: Add VEPU121 to rk3588

 .../bindings/media/rockchip,rk3568-vepu.yaml  |  8 +-
 arch/arm64/boot/dts/rockchip/rk3588s.dtsi     | 80 +++++++++++++++++++
 2 files changed, 86 insertions(+), 2 deletions(-)

-- 
2.44.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 1/2] media: dt-binding: media: Document rk3588’s VEPU121

[Emmanuel Gil Peyrot]

This encoder-only device is present four times on this SoC, and should
support everything the rk3568 vepu supports (so JPEG, H.264 and VP8
encoding).

According to the TRM[1], there is also the VEPU580 encoder which
supports H.264 and H.265, and various VDPU* decoders, of which only the
VDPU981 is currently supported.  This patch describes only the VEPU121.

[1] https://github.com/FanX-Tek/rk3588-TRM-and-Datasheet

Signed-off-by: Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
---
 .../devicetree/bindings/media/rockchip,rk3568-vepu.yaml   | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml b/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
index 9d90d8d0565a..4c6cb21da041 100644
--- a/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
+++ b/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
@@ -15,8 +15,12 @@ description:
 
 properties:
   compatible:
-    enum:
-      - rockchip,rk3568-vepu
+    oneOf:
+      - const: rockchip,rk3568-vepu
+      - items:
+          - enum:
+              - rockchip,rk3588-vepu121
+          - const: rockchip,rk3568-vepu
 
   reg:
     maxItems: 1
-- 
2.44.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 1/2] media: dt-binding: media: Document rk3588’s VEPU121

[Conor Dooley]

[-- Attachment #1.1: Type: text/plain, Size: 1501 bytes --]

On Wed, Mar 27, 2024 at 02:41:11PM +0100, Emmanuel Gil Peyrot wrote:
> This encoder-only device is present four times on this SoC, and should
> support everything the rk3568 vepu supports (so JPEG, H.264 and VP8
> encoding).
> 
> According to the TRM[1], there is also the VEPU580 encoder which
> supports H.264 and H.265, and various VDPU* decoders, of which only the
> VDPU981 is currently supported.  This patch describes only the VEPU121.
> 
> [1] https://github.com/FanX-Tek/rk3588-TRM-and-Datasheet
> 
> Signed-off-by: Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>

Acked-by: Conor Dooley <conor.dooley@microchip.com>

Thanks,
Conor.

> ---
>  .../devicetree/bindings/media/rockchip,rk3568-vepu.yaml   | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml b/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
> index 9d90d8d0565a..4c6cb21da041 100644
> --- a/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
> +++ b/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
> @@ -15,8 +15,12 @@ description:
>  
>  properties:
>    compatible:
> -    enum:
> -      - rockchip,rk3568-vepu
> +    oneOf:
> +      - const: rockchip,rk3568-vepu
> +      - items:
> +          - enum:
> +              - rockchip,rk3588-vepu121
> +          - const: rockchip,rk3568-vepu
>  
>    reg:
>      maxItems: 1
> -- 
> 2.44.0
> 

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 1/2] media: dt-binding: media: Document rk3588’s VEPU121

[Nicolas Dufresne]

Le mercredi 27 mars 2024 à 17:23 +0000, Conor Dooley a écrit :
> On Wed, Mar 27, 2024 at 02:41:11PM +0100, Emmanuel Gil Peyrot wrote:
> > This encoder-only device is present four times on this SoC, and should
> > support everything the rk3568 vepu supports (so JPEG, H.264 and VP8
> > encoding).
> > 
> > According to the TRM[1], there is also the VEPU580 encoder which
> > supports H.264 and H.265, and various VDPU* decoders, of which only the
> > VDPU981 is currently supported.  This patch describes only the VEPU121.
> > 
> > [1] https://github.com/FanX-Tek/rk3588-TRM-and-Datasheet
> > 
> > Signed-off-by: Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
> 
> Acked-by: Conor Dooley <conor.dooley@microchip.com>

I'd like to prevent this change until we fix the driver. It should not expose 1
video device per core, it should instead do schedule around these cores.

Nicolas

> 
> Thanks,
> Conor.
> 
> > ---
> >  .../devicetree/bindings/media/rockchip,rk3568-vepu.yaml   | 8 ++++++--
> >  1 file changed, 6 insertions(+), 2 deletions(-)
> > 
> > diff --git a/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml b/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
> > index 9d90d8d0565a..4c6cb21da041 100644
> > --- a/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
> > +++ b/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
> > @@ -15,8 +15,12 @@ description:
> >  
> >  properties:
> >    compatible:
> > -    enum:
> > -      - rockchip,rk3568-vepu
> > +    oneOf:
> > +      - const: rockchip,rk3568-vepu
> > +      - items:
> > +          - enum:
> > +              - rockchip,rk3588-vepu121
> > +          - const: rockchip,rk3568-vepu
> >  
> >    reg:
> >      maxItems: 1
> > -- 
> > 2.44.0
> > 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 1/2] media: dt-binding: media: Document rk3588’s VEPU121

[Sebastian Reichel]

[-- Attachment #1.1: Type: text/plain, Size: 1518 bytes --]

Hi,

On Wed, Mar 27, 2024 at 02:41:11PM +0100, Emmanuel Gil Peyrot wrote:
> This encoder-only device is present four times on this SoC, and should
> support everything the rk3568 vepu supports (so JPEG, H.264 and VP8
> encoding).
> 
> According to the TRM[1], there is also the VEPU580 encoder which
> supports H.264 and H.265, and various VDPU* decoders, of which only the
> VDPU981 is currently supported.  This patch describes only the VEPU121.
> 
> [1] https://github.com/FanX-Tek/rk3588-TRM-and-Datasheet
> 
> Signed-off-by: Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
> ---

Reviewed-by: Sebastian Reichel <sebastian.reichel@collabora.com>

-- Sebastian

>  .../devicetree/bindings/media/rockchip,rk3568-vepu.yaml   | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml b/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
> index 9d90d8d0565a..4c6cb21da041 100644
> --- a/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
> +++ b/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
> @@ -15,8 +15,12 @@ description:
>  
>  properties:
>    compatible:
> -    enum:
> -      - rockchip,rk3568-vepu
> +    oneOf:
> +      - const: rockchip,rk3568-vepu
> +      - items:
> +          - enum:
> +              - rockchip,rk3588-vepu121
> +          - const: rockchip,rk3568-vepu
>  
>    reg:
>      maxItems: 1
> -- 
> 2.44.0
> 

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 2/2] arm64: dts: rockchip: Add VEPU121 to rk3588

[Emmanuel Gil Peyrot]

The TRM (version 1.0 page 385) lists five VEPU121 cores, but only four
interrupts are listed (on page 24), so I’ve only enabled four of them
for now.

Signed-off-by: Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
---
 arch/arm64/boot/dts/rockchip/rk3588s.dtsi | 80 +++++++++++++++++++++++
 1 file changed, 80 insertions(+)

diff --git a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
index 87b83c87bd55..510ed3db9d01 100644
--- a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
@@ -2488,6 +2488,86 @@ gpio4: gpio@fec50000 {
 		};
 	};
 
+	jpeg_enc0: video-codec@fdba0000 {
+		compatible = "rockchip,rk3588-vepu121", "rockchip,rk3568-vepu";
+		reg = <0x0 0xfdba0000 0x0 0x800>;
+		interrupts = <GIC_SPI 122 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru ACLK_JPEG_ENCODER0>, <&cru HCLK_JPEG_ENCODER0>;
+		clock-names = "aclk", "hclk";
+		iommus = <&jpeg_enc0_mmu>;
+		power-domains = <&power RK3588_PD_VDPU>;
+	};
+
+	jpeg_enc0_mmu: iommu@fdba0800 {
+		compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
+		reg = <0x0 0xfdba0800 0x0 0x40>;
+		interrupts = <GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru ACLK_JPEG_ENCODER0>, <&cru HCLK_JPEG_ENCODER0>;
+		clock-names = "aclk", "iface";
+		power-domains = <&power RK3588_PD_VDPU>;
+		#iommu-cells = <0>;
+	};
+
+	jpeg_enc1: video-codec@fdba4000 {
+		compatible = "rockchip,rk3588-vepu121", "rockchip,rk3568-vepu";
+		reg = <0x0 0xfdba4000 0x0 0x800>;
+		interrupts = <GIC_SPI 124 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru ACLK_JPEG_ENCODER1>, <&cru HCLK_JPEG_ENCODER1>;
+		clock-names = "aclk", "hclk";
+		iommus = <&jpeg_enc1_mmu>;
+		power-domains = <&power RK3588_PD_VDPU>;
+	};
+
+	jpeg_enc1_mmu: iommu@fdba4800 {
+		compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
+		reg = <0x0 0xfdba4800 0x0 0x40>;
+		interrupts = <GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru ACLK_JPEG_ENCODER1>, <&cru HCLK_JPEG_ENCODER1>;
+		clock-names = "aclk", "iface";
+		power-domains = <&power RK3588_PD_VDPU>;
+		#iommu-cells = <0>;
+	};
+
+	jpeg_enc2: video-codec@fdba8000 {
+		compatible = "rockchip,rk3588-vepu121", "rockchip,rk3568-vepu";
+		reg = <0x0 0xfdba8000 0x0 0x800>;
+		interrupts = <GIC_SPI 126 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru ACLK_JPEG_ENCODER2>, <&cru HCLK_JPEG_ENCODER2>;
+		clock-names = "aclk", "hclk";
+		iommus = <&jpeg_enc2_mmu>;
+		power-domains = <&power RK3588_PD_VDPU>;
+	};
+
+	jpeg_enc2_mmu: iommu@fdba8800 {
+		compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
+		reg = <0x0 0xfdba8800 0x0 0x40>;
+		interrupts = <GIC_SPI 125 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru ACLK_JPEG_ENCODER2>, <&cru HCLK_JPEG_ENCODER2>;
+		clock-names = "aclk", "iface";
+		power-domains = <&power RK3588_PD_VDPU>;
+		#iommu-cells = <0>;
+	};
+
+	jpeg_enc3: video-codec@fdbac000 {
+		compatible = "rockchip,rk3588-vepu121", "rockchip,rk3568-vepu";
+		reg = <0x0 0xfdbac000 0x0 0x800>;
+		interrupts = <GIC_SPI 128 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru ACLK_JPEG_ENCODER3>, <&cru HCLK_JPEG_ENCODER3>;
+		clock-names = "aclk", "hclk";
+		iommus = <&jpeg_enc3_mmu>;
+		power-domains = <&power RK3588_PD_VDPU>;
+	};
+
+	jpeg_enc3_mmu: iommu@fdbac800 {
+		compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
+		reg = <0x0 0xfdbac800 0x0 0x40>;
+		interrupts = <GIC_SPI 127 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru ACLK_JPEG_ENCODER3>, <&cru HCLK_JPEG_ENCODER3>;
+		clock-names = "aclk", "iface";
+		power-domains = <&power RK3588_PD_VDPU>;
+		#iommu-cells = <0>;
+	};
+
 	av1d: video-codec@fdc70000 {
 		compatible = "rockchip,rk3588-av1-vpu";
 		reg = <0x0 0xfdc70000 0x0 0x800>;
-- 
2.44.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/2] arm64: dts: rockchip: Add VEPU121 to rk3588

[Sebastian Reichel]

[-- Attachment #1.1: Type: text/plain, Size: 4157 bytes --]

Hi,

On Wed, Mar 27, 2024 at 02:41:12PM +0100, Emmanuel Gil Peyrot wrote:
> The TRM (version 1.0 page 385) lists five VEPU121 cores, but only four
> interrupts are listed (on page 24), so I’ve only enabled four of them
> for now.
> 
> Signed-off-by: Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
> ---

Reviewed-by: Sebastian Reichel <sebastian.reichel@collabora.com>

-- Sebastian

>  arch/arm64/boot/dts/rockchip/rk3588s.dtsi | 80 +++++++++++++++++++++++
>  1 file changed, 80 insertions(+)
> 
> diff --git a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
> index 87b83c87bd55..510ed3db9d01 100644
> --- a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
> +++ b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
> @@ -2488,6 +2488,86 @@ gpio4: gpio@fec50000 {
>  		};
>  	};
>  
> +	jpeg_enc0: video-codec@fdba0000 {
> +		compatible = "rockchip,rk3588-vepu121", "rockchip,rk3568-vepu";
> +		reg = <0x0 0xfdba0000 0x0 0x800>;
> +		interrupts = <GIC_SPI 122 IRQ_TYPE_LEVEL_HIGH 0>;
> +		clocks = <&cru ACLK_JPEG_ENCODER0>, <&cru HCLK_JPEG_ENCODER0>;
> +		clock-names = "aclk", "hclk";
> +		iommus = <&jpeg_enc0_mmu>;
> +		power-domains = <&power RK3588_PD_VDPU>;
> +	};
> +
> +	jpeg_enc0_mmu: iommu@fdba0800 {
> +		compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
> +		reg = <0x0 0xfdba0800 0x0 0x40>;
> +		interrupts = <GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH 0>;
> +		clocks = <&cru ACLK_JPEG_ENCODER0>, <&cru HCLK_JPEG_ENCODER0>;
> +		clock-names = "aclk", "iface";
> +		power-domains = <&power RK3588_PD_VDPU>;
> +		#iommu-cells = <0>;
> +	};
> +
> +	jpeg_enc1: video-codec@fdba4000 {
> +		compatible = "rockchip,rk3588-vepu121", "rockchip,rk3568-vepu";
> +		reg = <0x0 0xfdba4000 0x0 0x800>;
> +		interrupts = <GIC_SPI 124 IRQ_TYPE_LEVEL_HIGH 0>;
> +		clocks = <&cru ACLK_JPEG_ENCODER1>, <&cru HCLK_JPEG_ENCODER1>;
> +		clock-names = "aclk", "hclk";
> +		iommus = <&jpeg_enc1_mmu>;
> +		power-domains = <&power RK3588_PD_VDPU>;
> +	};
> +
> +	jpeg_enc1_mmu: iommu@fdba4800 {
> +		compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
> +		reg = <0x0 0xfdba4800 0x0 0x40>;
> +		interrupts = <GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH 0>;
> +		clocks = <&cru ACLK_JPEG_ENCODER1>, <&cru HCLK_JPEG_ENCODER1>;
> +		clock-names = "aclk", "iface";
> +		power-domains = <&power RK3588_PD_VDPU>;
> +		#iommu-cells = <0>;
> +	};
> +
> +	jpeg_enc2: video-codec@fdba8000 {
> +		compatible = "rockchip,rk3588-vepu121", "rockchip,rk3568-vepu";
> +		reg = <0x0 0xfdba8000 0x0 0x800>;
> +		interrupts = <GIC_SPI 126 IRQ_TYPE_LEVEL_HIGH 0>;
> +		clocks = <&cru ACLK_JPEG_ENCODER2>, <&cru HCLK_JPEG_ENCODER2>;
> +		clock-names = "aclk", "hclk";
> +		iommus = <&jpeg_enc2_mmu>;
> +		power-domains = <&power RK3588_PD_VDPU>;
> +	};
> +
> +	jpeg_enc2_mmu: iommu@fdba8800 {
> +		compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
> +		reg = <0x0 0xfdba8800 0x0 0x40>;
> +		interrupts = <GIC_SPI 125 IRQ_TYPE_LEVEL_HIGH 0>;
> +		clocks = <&cru ACLK_JPEG_ENCODER2>, <&cru HCLK_JPEG_ENCODER2>;
> +		clock-names = "aclk", "iface";
> +		power-domains = <&power RK3588_PD_VDPU>;
> +		#iommu-cells = <0>;
> +	};
> +
> +	jpeg_enc3: video-codec@fdbac000 {
> +		compatible = "rockchip,rk3588-vepu121", "rockchip,rk3568-vepu";
> +		reg = <0x0 0xfdbac000 0x0 0x800>;
> +		interrupts = <GIC_SPI 128 IRQ_TYPE_LEVEL_HIGH 0>;
> +		clocks = <&cru ACLK_JPEG_ENCODER3>, <&cru HCLK_JPEG_ENCODER3>;
> +		clock-names = "aclk", "hclk";
> +		iommus = <&jpeg_enc3_mmu>;
> +		power-domains = <&power RK3588_PD_VDPU>;
> +	};
> +
> +	jpeg_enc3_mmu: iommu@fdbac800 {
> +		compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
> +		reg = <0x0 0xfdbac800 0x0 0x40>;
> +		interrupts = <GIC_SPI 127 IRQ_TYPE_LEVEL_HIGH 0>;
> +		clocks = <&cru ACLK_JPEG_ENCODER3>, <&cru HCLK_JPEG_ENCODER3>;
> +		clock-names = "aclk", "iface";
> +		power-domains = <&power RK3588_PD_VDPU>;
> +		#iommu-cells = <0>;
> +	};
> +
>  	av1d: video-codec@fdc70000 {
>  		compatible = "rockchip,rk3588-av1-vpu";
>  		reg = <0x0 0xfdc70000 0x0 0x800>;
> -- 
> 2.44.0
> 

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 0/2] Enable JPEG encoding on rk3588

[Nicolas Dufresne]

Hi,

Le mercredi 27 mars 2024 à 14:41 +0100, Emmanuel Gil Peyrot a écrit :
> Only the JPEG encoder is available for now, although there are patches
> for the undocumented VP8 encoder floating around[0].

[0] seems like a broken link. The VP8 encoder RFC is for RK3399 (and Hantro H1
posted by ST more recently). The TRM says "VEPU121(JPEG encoder only)", which
suggest that the H.264 and VP8 encoders usually found on the VEPU121 are
removed. As Rockchip have remove the synthesize register while modifying the H1
IP, it is difficult to verify. Confusingly the H.264 specific registers are
documented in the TRM around VEPU121.

> 
> This has been tested on a rock-5b, resulting in four /dev/video*
> encoders.  The userspace program I’ve been using to test them is
> Onix[1], using the jpeg-encoder example, it will pick one of these four
> at random (but displays the one it picked):
> % ffmpeg -i <input image> -pix_fmt yuvj420p temp.yuv
> % jpeg-encoder temp.yuv <width> <height> NV12 <quality> output.jpeg

I don't like that we exposing each identical cores a separate video nodes. I
think we should aim for 1 device, and then multi-plex and schedule de cores from
inside the Linux kernel.

Not doing this now means we'll never have an optimal hardware usage
distribution. Just consider two userspace software wanting to do jpeg encoding.
If they both take a guess, they may endup using a single core. Where with proper
scheduling in V4L2, the kernel will be able to properly distribute the load. I
insist on this, since if we merge you changes it becomes an ABI and we can't
change it anymore.

I understand that this impose a rework of the mem2mem framework so that we can
run multiple jobs, but this will be needed anyway on RK3588, since the rkvdec2,
which we don't have a driver yet is also multi-core, but you need to use 2 cores
when the resolution is close to 8K.

Nicolas

> 
> [0] https://patchwork.kernel.org/project/linux-rockchip/list/?series=789885
> [1] https://crates.io/crates/onix
> 
> Changes since v1:
> - Dropped patches 1 and 4.
> - Use the proper compatible form, since this device should be fully
>   compatible with the VEPU of rk356x.
> - Describe where the VEPU121 name comes from, and list other encoders
>   and decoders present in this SoC.
> - Properly test the device tree changes, I previously couldn’t since I
>   was using a too recent version of python-jsonschema…
> 
> Emmanuel Gil Peyrot (2):
>   media: dt-binding: media: Document rk3588’s VEPU121
>   arm64: dts: rockchip: Add VEPU121 to rk3588
> 
>  .../bindings/media/rockchip,rk3568-vepu.yaml  |  8 +-
>  arch/arm64/boot/dts/rockchip/rk3588s.dtsi     | 80 +++++++++++++++++++
>  2 files changed, 86 insertions(+), 2 deletions(-)
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 0/2] Enable JPEG encoding on rk3588

[Link Mauve]

On Thu, Apr 04, 2024 at 01:41:15PM -0400, Nicolas Dufresne wrote:
> Hi,

Hi,

> 
> Le mercredi 27 mars 2024 à 14:41 +0100, Emmanuel Gil Peyrot a écrit :
> > Only the JPEG encoder is available for now, although there are patches
> > for the undocumented VP8 encoder floating around[0].
> 
> [0] seems like a broken link. The VP8 encoder RFC is for RK3399 (and Hantro H1
> posted by ST more recently). The TRM says "VEPU121(JPEG encoder only)", which
> suggest that the H.264 and VP8 encoders usually found on the VEPU121 are
> removed. As Rockchip have remove the synthesize register while modifying the H1
> IP, it is difficult to verify. Confusingly the H.264 specific registers are
> documented in the TRM around VEPU121.

Ah, the link became, and was indeed ST’s series:
https://patchwork.kernel.org/project/linux-rockchip/list/?series=789885&archive=both

But the TRM part 1 says the VEPU121 supports H.264 encoding (page 367),
and it’s likely they didn’t remove just VP8 support since the codec
features are pretty close to H.264’s.

> 
> > 
> > This has been tested on a rock-5b, resulting in four /dev/video*
> > encoders.  The userspace program I’ve been using to test them is
> > Onix[1], using the jpeg-encoder example, it will pick one of these four
> > at random (but displays the one it picked):
> > % ffmpeg -i <input image> -pix_fmt yuvj420p temp.yuv
> > % jpeg-encoder temp.yuv <width> <height> NV12 <quality> output.jpeg
> 
> I don't like that we exposing each identical cores a separate video nodes. I
> think we should aim for 1 device, and then multi-plex and schedule de cores from
> inside the Linux kernel.

I agree, but this should be handled in the driver not in the device
tree, and it can be done later.

> 
> Not doing this now means we'll never have an optimal hardware usage
> distribution. Just consider two userspace software wanting to do jpeg encoding.
> If they both take a guess, they may endup using a single core. Where with proper
> scheduling in V4L2, the kernel will be able to properly distribute the load. I
> insist on this, since if we merge you changes it becomes an ABI and we can't
> change it anymore.

Will it really become ABI just like that?  Userspace should always
discover the video nodes and their capabilities and not hardcode e.g. a
specific /dev/videoN file for a specific codec.  I would argue that this
series would let userspace do JPEG encoding right away, even if in a
less optimal way than if the driver would round-robin them through a
single video node, but that can always be added in a future version.

> 
> I understand that this impose a rework of the mem2mem framework so that we can
> run multiple jobs, but this will be needed anyway on RK3588, since the rkvdec2,
> which we don't have a driver yet is also multi-core, but you need to use 2 cores
> when the resolution is close to 8K.

I think the mediatek JPEG driver already supports that, would it be ok
to do it the same way?

> 
> Nicolas
> 
> > 
> > [0] https://patchwork.kernel.org/project/linux-rockchip/list/?series=789885
> > [1] https://crates.io/crates/onix
> > 
> > Changes since v1:
> > - Dropped patches 1 and 4.
> > - Use the proper compatible form, since this device should be fully
> >   compatible with the VEPU of rk356x.
> > - Describe where the VEPU121 name comes from, and list other encoders
> >   and decoders present in this SoC.
> > - Properly test the device tree changes, I previously couldn’t since I
> >   was using a too recent version of python-jsonschema…
> > 
> > Emmanuel Gil Peyrot (2):
> >   media: dt-binding: media: Document rk3588’s VEPU121
> >   arm64: dts: rockchip: Add VEPU121 to rk3588
> > 
> >  .../bindings/media/rockchip,rk3568-vepu.yaml  |  8 +-
> >  arch/arm64/boot/dts/rockchip/rk3588s.dtsi     | 80 +++++++++++++++++++
> >  2 files changed, 86 insertions(+), 2 deletions(-)
> > 
> 

-- 
Link Mauve

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 0/2] Enable JPEG encoding on rk3588

[Nicolas Dufresne]

Le vendredi 05 avril 2024 à 16:21 +0200, Link Mauve a écrit :
> On Thu, Apr 04, 2024 at 01:41:15PM -0400, Nicolas Dufresne wrote:
> > Hi,
> 
> Hi,
> 
> > 
> > Le mercredi 27 mars 2024 à 14:41 +0100, Emmanuel Gil Peyrot a écrit :
> > > Only the JPEG encoder is available for now, although there are patches
> > > for the undocumented VP8 encoder floating around[0].
> > 
> > [0] seems like a broken link. The VP8 encoder RFC is for RK3399 (and Hantro H1
> > posted by ST more recently). The TRM says "VEPU121(JPEG encoder only)", which
> > suggest that the H.264 and VP8 encoders usually found on the VEPU121 are
> > removed. As Rockchip have remove the synthesize register while modifying the H1
> > IP, it is difficult to verify. Confusingly the H.264 specific registers are
> > documented in the TRM around VEPU121.
> 
> Ah, the link became, and was indeed ST’s series:
> https://patchwork.kernel.org/project/linux-rockchip/list/?series=789885&archive=both
> 
> But the TRM part 1 says the VEPU121 supports H.264 encoding (page 367),
> and it’s likely they didn’t remove just VP8 support since the codec
> features are pretty close to H.264’s.
> 
> > 
> > > 
> > > This has been tested on a rock-5b, resulting in four /dev/video*
> > > encoders.  The userspace program I’ve been using to test them is
> > > Onix[1], using the jpeg-encoder example, it will pick one of these four
> > > at random (but displays the one it picked):
> > > % ffmpeg -i <input image> -pix_fmt yuvj420p temp.yuv
> > > % jpeg-encoder temp.yuv <width> <height> NV12 <quality> output.jpeg
> > 
> > I don't like that we exposing each identical cores a separate video nodes. I
> > think we should aim for 1 device, and then multi-plex and schedule de cores from
> > inside the Linux kernel.
> 
> I agree, but this should be handled in the driver not in the device
> tree, and it can be done later.

As the behaviour we want is that these cores becomes a group and get schedule
together, its certainly a good time to slow down and evaluate if that part needs
to be improve in the DT too.

Hantro G1/H1 and VEPU/VDPU121 combos originally shared the same sram region. Its
not clear if any of these cores have this limitation and if this should be
expressed in the DT / driver.

> 
> > 
> > Not doing this now means we'll never have an optimal hardware usage
> > distribution. Just consider two userspace software wanting to do jpeg encoding.
> > If they both take a guess, they may endup using a single core. Where with proper
> > scheduling in V4L2, the kernel will be able to properly distribute the load. I
> > insist on this, since if we merge you changes it becomes an ABI and we can't
> > change it anymore.
> 
> Will it really become ABI just like that?  Userspace should always
> discover the video nodes and their capabilities and not hardcode e.g. a
> specific /dev/videoN file for a specific codec.  I would argue that this
> series would let userspace do JPEG encoding right away, even if in a
> less optimal way than if the driver would round-robin them through a
> single video node, but that can always be added in a future version.

Might be on the gray side, but there is good chances software written for your
specific board can stop working after te grouping is done.

> 
> > 
> > I understand that this impose a rework of the mem2mem framework so that we can
> > run multiple jobs, but this will be needed anyway on RK3588, since the rkvdec2,
> > which we don't have a driver yet is also multi-core, but you need to use 2 cores
> > when the resolution is close to 8K.
> 
> I think the mediatek JPEG driver already supports that, would it be ok
> to do it the same way?

I don't know for JPEG, the MTK vcoder do support cascading cores. This is
different from concurrent cores. In MTK architecture, for some of the codec,
there is LAT (entropy decoder) and CORE (the reconstruction block) that are
split.

Nicolas

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 0/2] Enable JPEG encoding on rk3588

[Link Mauve]

On Sun, Apr 07, 2024 at 10:08:58AM +0200, Nicolas Dufresne wrote:
> Le vendredi 05 avril 2024 à 16:21 +0200, Link Mauve a écrit :
> > On Thu, Apr 04, 2024 at 01:41:15PM -0400, Nicolas Dufresne wrote:
> > > Hi,
> > 
> > Hi,
> > 
> > > 
> > > Le mercredi 27 mars 2024 à 14:41 +0100, Emmanuel Gil Peyrot a écrit :
> > > > Only the JPEG encoder is available for now, although there are patches
> > > > for the undocumented VP8 encoder floating around[0].
> > > 
> > > [0] seems like a broken link. The VP8 encoder RFC is for RK3399 (and Hantro H1
> > > posted by ST more recently). The TRM says "VEPU121(JPEG encoder only)", which
> > > suggest that the H.264 and VP8 encoders usually found on the VEPU121 are
> > > removed. As Rockchip have remove the synthesize register while modifying the H1
> > > IP, it is difficult to verify. Confusingly the H.264 specific registers are
> > > documented in the TRM around VEPU121.
> > 
> > Ah, the link became, and was indeed ST’s series:
> > https://patchwork.kernel.org/project/linux-rockchip/list/?series=789885&archive=both
> > 
> > But the TRM part 1 says the VEPU121 supports H.264 encoding (page 367),
> > and it’s likely they didn’t remove just VP8 support since the codec
> > features are pretty close to H.264’s.
> > 
> > > 
> > > > 
> > > > This has been tested on a rock-5b, resulting in four /dev/video*
> > > > encoders.  The userspace program I’ve been using to test them is
> > > > Onix[1], using the jpeg-encoder example, it will pick one of these four
> > > > at random (but displays the one it picked):
> > > > % ffmpeg -i <input image> -pix_fmt yuvj420p temp.yuv
> > > > % jpeg-encoder temp.yuv <width> <height> NV12 <quality> output.jpeg
> > > 
> > > I don't like that we exposing each identical cores a separate video nodes. I
> > > think we should aim for 1 device, and then multi-plex and schedule de cores from
> > > inside the Linux kernel.
> > 
> > I agree, but this should be handled in the driver not in the device
> > tree, and it can be done later.
> 
> As the behaviour we want is that these cores becomes a group and get schedule
> together, its certainly a good time to slow down and evaluate if that part needs
> to be improve in the DT too.
> 
> Hantro G1/H1 and VEPU/VDPU121 combos originally shared the same sram region. Its
> not clear if any of these cores have this limitation and if this should be
> expressed in the DT / driver.

The TRM on page 369 mentions that:
> Please note that VDPU121 and VDPU381 and VDPU720 and AV1 and VEPU121 and VEPU580
> is different IP cores, so these processing core can be work together.

I understand that as them not sharing any memory and being able to work
independently.

> 
> > 
> > > 
> > > Not doing this now means we'll never have an optimal hardware usage
> > > distribution. Just consider two userspace software wanting to do jpeg encoding.
> > > If they both take a guess, they may endup using a single core. Where with proper
> > > scheduling in V4L2, the kernel will be able to properly distribute the load. I
> > > insist on this, since if we merge you changes it becomes an ABI and we can't
> > > change it anymore.
> > 
> > Will it really become ABI just like that?  Userspace should always
> > discover the video nodes and their capabilities and not hardcode e.g. a
> > specific /dev/videoN file for a specific codec.  I would argue that this
> > series would let userspace do JPEG encoding right away, even if in a
> > less optimal way than if the driver would round-robin them through a
> > single video node, but that can always be added in a future version.
> 
> Might be on the gray side, but there is good chances software written for your
> specific board can stop working after te grouping is done.

I will send a new series shortly which enables only one of these four
cores, the functionality should be the same, it will just expose only
one video node which can get a four times throughput upgrade later once
we implement multi-core support in the driver.

> 
> > 
> > > 
> > > I understand that this impose a rework of the mem2mem framework so that we can
> > > run multiple jobs, but this will be needed anyway on RK3588, since the rkvdec2,
> > > which we don't have a driver yet is also multi-core, but you need to use 2 cores
> > > when the resolution is close to 8K.
> > 
> > I think the mediatek JPEG driver already supports that, would it be ok
> > to do it the same way?
> 
> I don't know for JPEG, the MTK vcoder do support cascading cores. This is
> different from concurrent cores. In MTK architecture, for some of the codec,
> there is LAT (entropy decoder) and CORE (the reconstruction block) that are
> split.

Ah, that’s different then, I only had a cursory look at them back when I
implemented the JPEG decoder for sunxi.

> 
> Nicolas

-- 
Link Mauve

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel