[RFC PATCH 0/3] media: rockchip: VEPU510 H.264 encoder for RK3576

[RFC PATCH 0/3] media: rockchip: VEPU510 H.264 encoder for RK3576

[Jiaxing Hu]

This is an RFC for a from-scratch mainline V4L2 driver for the H.264
hardware video encoder (VEPU510) on the Rockchip RK3576.  It is a
stateful mem2mem encoder (NV12 in, H.264 Annex-B out) modelled on the
verisilicon/hantro driver, not on the downstream MPP-service model.

I'm posting it as an RFC because intra frames work but inter frames do
not, and I'd like a second pair of eyes on the inter-frame problem
before this is worth a real submission.

What works
----------

Intra-only (I-frame) encoding is confirmed on real hardware (Radxa
ROCK 4D).  With GOP size 1 the driver produces a valid H.264 stream the
reference decoder accepts.  This already exercises the full path: V4L2
m2m, the register programming, the software SPS/PPS prepend, and the
hardware slice output.

There is no upstream userspace involved (an encoder needs no request
API); I drive it with a small ioctl test program that feeds NV12 frames
and writes the CAPTURE buffers to a .h264 file.

The open problem: inter (P-frame) frames hang
---------------------------------------------

Every P-frame stalls the encoder's own hardware watchdog (INT_STA bit 8,
~20 ms after the kick) and produces essentially no bitstream.  I have
spent a lot of time narrowing this; it is sharply localized but I cannot
close it:

  - The P-frame register writes match a real register-write trace of the
    vendor stack encoding the same content, byte for byte (I traced the
    vendor kernel's writes and diffed against this driver's).

  - The reconstruction the previous frame writes is *valid*: dumping the
    recon buffer after the I-frame shows correct reconstructed pixels.

  - If the P-frame reads its own (older, settled) recon slot instead of
    the immediately-preceding frame's, it completes.  Reading the
    immediately-preceding frame's *fresh* reconstruction as the
    reference is what hangs.

  - It is not FBC: storing the reconstruction uncompressed
    (enc_pic.rec_fbc_dis = 1) still hangs.

  - The first frame of a session always works; the first P-frame hangs;
    after a couple of failures + core resets, later frames sometimes
    start completing.  It behaves like a warm-up / settling problem on
    the reference-read path, not a wrong register value.

So the encoder programs identically to the vendor and reads a valid
reference, but stalls fetching the previous frame's reconstruction as
the inter reference on the first inter frame of a session.  My best
guess is that the vendor does something between consecutive frame
submissions -- a completion/drain wait, a cache/coherency step, or a
per-frame re-arm -- that I am missing, but I have not found it.  If
anyone recognizes this on VEPU5xx, or knows what the reference-read path
needs between frames, a pointer would be very welcome.

The shape of this -- the first operation of a power session works, the
next stalls, and a reset/warm-up sometimes helps -- is the same one I
ran into bringing up this SoC's NPU (accel/rocket RKNN), where the
second/chained submit in a session would not fire [1].  I can't claim
they share a root cause, but if this is a known RK3576-wide submit /
re-arm quirk rather than an encoder-specific bug, that would be good to
know.

[1] https://lore.kernel.org/all/20260718031146.3368811-1-gahing@gahingwoo.com/

Notes for review
----------------

  - The PARAM/SQI register classes are programmed from mpp's constant
    "default tuning" tables (not derived per-frame), as noted in the
    code.  They are required: leaving them unwritten stalls even the
    I-frame.

  - Rate control is fixed-QP only for now (the bitrate control is
    advisory).

  - H.264 baseline/main, single slice, 4:2:0 only.

  - Tested at 176x144; other resolutions are not yet validated.

Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>

Jiaxing Hu (3):
  dt-bindings: media: add Rockchip RK3576 VEPU H.264 encoder
  media: rockchip: add VEPU510 H.264 encoder driver for RK3576
  arm64: dts: rockchip: rk3576: add VEPU H.264 encoder nodes

 .../bindings/media/rockchip,rk3576-vepu.yaml       |   94 ++
 arch/arm64/boot/dts/rockchip/rk3576.dtsi           |   50 +
 drivers/media/platform/rockchip/Kconfig            |    1 +
 drivers/media/platform/rockchip/Makefile           |    1 +
 drivers/media/platform/rockchip/rkvenc/Kconfig     |   14 +
 drivers/media/platform/rockchip/rkvenc/Makefile    |    6 +
 .../media/platform/rockchip/rkvenc/rkvenc-h264.c   | 1095 ++++++++++++++++++++
 .../media/platform/rockchip/rkvenc/rkvenc-regs.h   |  929 +++++++++++++++++
 drivers/media/platform/rockchip/rkvenc/rkvenc.c    |  892 ++++++++++++++++
 drivers/media/platform/rockchip/rkvenc/rkvenc.h    |  212 ++++
 10 files changed, 3294 insertions(+)
--
2.43.0

[RFC PATCH 1/3] dt-bindings: media: add Rockchip RK3576 VEPU H.264 encoder

[Jiaxing Hu]

Add the binding for the VEPU510 H.264 hardware video encoder found on the
Rockchip RK3576.  Each SoC has two independent encoder cores, each behind
its own IOMMU.

Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>
---
 .../bindings/media/rockchip,rk3576-vepu.yaml       | 94 ++++++++++++++++++++++
 1 file changed, 94 insertions(+)

diff --git a/Documentation/devicetree/bindings/media/rockchip,rk3576-vepu.yaml b/Documentation/devicetree/bindings/media/rockchip,rk3576-vepu.yaml
new file mode 100644
index 000000000..942118cb3
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/rockchip,rk3576-vepu.yaml
@@ -0,0 +1,94 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/media/rockchip,rk3576-vepu.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Rockchip VEPU510 Video Encoder (RK3576)
+
+maintainers:
+  - Jiaxing Hu <gahing@gahingwoo.com>
+
+description: |-
+  RK3576 has two identical VEPU510 hardware H.264/H.265 video encoder
+  cores. Each core is a standalone V4L2 mem2mem device in this binding;
+  there is no shared hardware descriptor/link-list engine tying the two
+  cores together (unlike the RK3576 video decoder's CCU), so each core
+  node is independent and self-contained.
+
+properties:
+  compatible:
+    const: rockchip,rk3576-vepu
+
+  reg:
+    maxItems: 1
+
+  interrupts:
+    maxItems: 1
+
+  clocks:
+    items:
+      - description: The Video Encoder AXI interface clock
+      - description: The Video Encoder AHB interface clock
+      - description: The Video Encoder core clock
+
+  clock-names:
+    items:
+      - const: aclk_vcodec
+      - const: hclk_vcodec
+      - const: clk_core
+
+  assigned-clocks: true
+
+  assigned-clock-rates: true
+
+  resets:
+    items:
+      - description: The Video Encoder AXI interface reset
+      - description: The Video Encoder AHB interface reset
+      - description: The Video Encoder core reset
+
+  reset-names:
+    items:
+      - const: video_a
+      - const: video_h
+      - const: video_core
+
+  power-domains:
+    maxItems: 1
+
+  iommus:
+    maxItems: 1
+
+required:
+  - compatible
+  - reg
+  - interrupts
+  - clocks
+  - clock-names
+  - power-domains
+  - iommus
+
+additionalProperties: false
+
+examples:
+  - |
+    #include <dt-bindings/interrupt-controller/arm-gic.h>
+    #include <dt-bindings/clock/rockchip,rk3576-cru.h>
+    #include <dt-bindings/power/rockchip,rk3576-power.h>
+
+    vepu0: video-codec@27a00000 {
+        compatible = "rockchip,rk3576-vepu";
+        reg = <0x27a00000 0x6000>;
+        interrupts = <GIC_SPI 310 IRQ_TYPE_LEVEL_HIGH>;
+        clocks = <&cru ACLK_VEPU0>, <&cru HCLK_VEPU0>, <&cru CLK_VEPU0_CORE>;
+        clock-names = "aclk_vcodec", "hclk_vcodec", "clk_core";
+        assigned-clocks = <&cru ACLK_VEPU0>, <&cru CLK_VEPU0_CORE>;
+        assigned-clock-rates = <400000000>, <702000000>;
+        resets = <&cru SRST_A_VEPU0>, <&cru SRST_H_VEPU0>, <&cru SRST_VEPU0_CORE>;
+        reset-names = "video_a", "video_h", "video_core";
+        power-domains = <&power RK3576_PD_VEPU0>;
+        iommus = <&vepu0_mmu>;
+    };
+
+...
-- 
2.43.0

[RFC PATCH 3/3] arm64: dts: rockchip: rk3576: add VEPU H.264 encoder nodes

[Jiaxing Hu]

Add the two VEPU510 encoder cores at 0x27a00000 / 0x27a10000 and their
IOMMUs.

Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>
---
 arch/arm64/boot/dts/rockchip/rk3576.dtsi | 50 ++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/arch/arm64/boot/dts/rockchip/rk3576.dtsi b/arch/arm64/boot/dts/rockchip/rk3576.dtsi
index b622c6fee..9bf4f2c41 100644
--- a/arch/arm64/boot/dts/rockchip/rk3576.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3576.dtsi
@@ -1320,6 +1320,56 @@ vdec_mmu: iommu@27b00800 {
 			#iommu-cells = <0>;
 		};
 
+		vepu0: video-codec@27a00000 {
+			compatible = "rockchip,rk3576-vepu";
+			reg = <0x0 0x27a00000 0x0 0x6000>;
+			interrupts = <GIC_SPI 310 IRQ_TYPE_LEVEL_HIGH>;
+			clocks = <&cru ACLK_VEPU0>, <&cru HCLK_VEPU0>, <&cru CLK_VEPU0_CORE>;
+			clock-names = "aclk_vcodec", "hclk_vcodec", "clk_core";
+			assigned-clocks = <&cru ACLK_VEPU0>, <&cru CLK_VEPU0_CORE>;
+			assigned-clock-rates = <400000000>, <702000000>;
+			resets = <&cru SRST_A_VEPU0>, <&cru SRST_H_VEPU0>, <&cru SRST_VEPU0_CORE>;
+			reset-names = "video_a", "video_h", "video_core";
+			power-domains = <&power RK3576_PD_VEPU0>;
+			iommus = <&vepu0_mmu>;
+		};
+
+		vepu0_mmu: iommu@27a0f000 {
+			compatible = "rockchip,rk3576-iommu", "rockchip,rk3568-iommu";
+			reg = <0x0 0x27a0f000 0x0 0x40>;
+			interrupts = <GIC_SPI 311 IRQ_TYPE_LEVEL_HIGH>;
+			clocks = <&cru ACLK_VEPU0>, <&cru HCLK_VEPU0>;
+			clock-names = "aclk", "iface";
+			power-domains = <&power RK3576_PD_VEPU0>;
+			rockchip,disable-mmu-reset;
+			#iommu-cells = <0>;
+		};
+
+		vepu1: video-codec@27a10000 {
+			compatible = "rockchip,rk3576-vepu";
+			reg = <0x0 0x27a10000 0x0 0x6000>;
+			interrupts = <GIC_SPI 387 IRQ_TYPE_LEVEL_HIGH>;
+			clocks = <&cru ACLK_VEPU1>, <&cru HCLK_VEPU1>, <&cru CLK_VEPU1_CORE>;
+			clock-names = "aclk_vcodec", "hclk_vcodec", "clk_core";
+			assigned-clocks = <&cru ACLK_VEPU1>, <&cru CLK_VEPU1_CORE>;
+			assigned-clock-rates = <400000000>, <702000000>;
+			resets = <&cru SRST_A_VEPU1>, <&cru SRST_H_VEPU1>, <&cru SRST_VEPU1_CORE>;
+			reset-names = "video_a", "video_h", "video_core";
+			power-domains = <&power RK3576_PD_VEPU1>;
+			iommus = <&vepu1_mmu>;
+		};
+
+		vepu1_mmu: iommu@27a1f000 {
+			compatible = "rockchip,rk3576-iommu", "rockchip,rk3568-iommu";
+			reg = <0x0 0x27a1f000 0x0 0x40>;
+			interrupts = <GIC_SPI 388 IRQ_TYPE_LEVEL_HIGH>;
+			clocks = <&cru ACLK_VEPU1>, <&cru HCLK_VEPU1>;
+			clock-names = "aclk", "iface";
+			power-domains = <&power RK3576_PD_VEPU1>;
+			rockchip,disable-mmu-reset;
+			#iommu-cells = <0>;
+		};
+
 		vop: vop@27d00000 {
 			compatible = "rockchip,rk3576-vop";
 			reg = <0x0 0x27d00000 0x0 0x3000>, <0x0 0x27d05000 0x0 0x1000>;
-- 
2.43.0

Re: [RFC PATCH 2/3] media: rockchip: add VEPU510 H.264 encoder driver for RK3576

[Heiko Stübner]

Am Mittwoch, 22. Juli 2026, 09:34:16 Mitteleuropäische Sommerzeit schrieb Jiaxing Hu:
> Add a from-scratch stateful V4L2 mem2mem driver for the Rockchip RK3576
> VEPU510 H.264 hardware video encoder (raw NV12 in, H.264 Annex-B out),
> modelled on the verisilicon/hantro device_run()/codec_ops split rather
> than the downstream MPP-service/task-queue model.
> 
> The two encoder cores (rkvenc0/rkvenc1) are exposed as two independent
> V4L2 M2M device nodes, each driving one physical core standalone; the
> downstream vendor CCU cross-core load-balancing is not implemented.
> 
> Register field semantics were worked out by trial-and-error against
> Rockchip's own open-source userspace codec library (rockchip-linux/mpp)
> and cross-checked against a real register-write trace of the downstream
> vendor stack running the same encode.
> 
> Intra (I-frame) encoding is confirmed working on real hardware (Radxa
> ROCK 4D): it produces valid H.264 that the reference decoders accept.
> Inter (P-frame) encoding still hits a hardware-watchdog stall in the
> reference-read datapath -- this is the main open question for this RFC;
> see the cover letter for the full symptom analysis.
> 
> Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>

[...]

> diff --git a/drivers/media/platform/rockchip/rkvenc/rkvenc.c b/drivers/media/platform/rockchip/rkvenc/rkvenc.c
> new file mode 100644
> index 000000000..d70bcc5fb
> --- /dev/null
> +++ b/drivers/media/platform/rockchip/rkvenc/rkvenc.c
> @@ -0,0 +1,892 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Rockchip VEPU510 (RK3576) hardware video encoder driver.
> + *
> + * Copyright (C) 2026 Jiaxing Hu <gahing@gahingwoo.com>
> + *
> + * Architecture notes (see also rkvenc-regs.h):
> + *
> + *  - This is a *stateful* V4L2 mem2mem encoder (raw NV12 in on OUTPUT,
> + *    H.264 Annex-B out on CAPTURE) modelled on
> + *    drivers/media/platform/verisilicon's hantro_drv.c device_run()/
> + *    codec_ops{run,done} split, NOT on rkvdec's stateless request-API
> + *    decoder pattern — an encoder has no per-frame bitstream to parse,
> + *    so there is nothing analogous to rkvdec_run_preamble/postamble here.
> + *
> + *  - The vendor downstream driver (rockchip-linux/kernel,
> + *    drivers/video/rockchip/mpp/mpp_rkvenc2.c) groups rkvenc0/rkvenc1
> + *    under a "CCU" (rockchip,rkv-encoder-rk3576-ccu) that does pure
> + *    software task-queue load balancing across both cores, plus an
> + *    optional DCHS (dual-core-handshake) register protocol used only
> + *    when *deliberately* splitting one frame's rows across both cores.
> + *    There is no hardware descriptor/link-list engine behind it (unlike
> + *    the decoder's CCU). v1 of this driver does not implement either:
> + *    rkvenc0 and rkvenc1 are exposed as two independent V4L2 M2M device
> + *    nodes, each driving one physical core standalone — the same
> + *    simplification rkvdec itself makes for multi-core VDPU hardware
> + *    (see rkvdec_disable_multicore()).

At least the comment and from a casual look also the code get this
backwards. The idea is to explicitly _not_ expose multiple video devices.

See
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/platform/rockchip/rkvdec/rkvdec.c#n1613

Any (future) scheduling should happen inside the driver, and not get
offloaded onto _every_ userspace application individually.


Heiko