[RFC PATCH 0/3] media: rockchip: VEPU510 H.264 encoder for RK3576

[RFC PATCH 0/3] media: rockchip: VEPU510 H.264 encoder for RK3576

[Jiaxing Hu]

This is an RFC for a from-scratch mainline V4L2 driver for the H.264
hardware video encoder (VEPU510) on the Rockchip RK3576.  It is a
stateful mem2mem encoder (NV12 in, H.264 Annex-B out) modelled on the
verisilicon/hantro driver, not on the downstream MPP-service model.

I'm posting it as an RFC because intra frames work but inter frames do
not, and I'd like a second pair of eyes on the inter-frame problem
before this is worth a real submission.

What works
----------

Intra-only (I-frame) encoding is confirmed on real hardware (Radxa
ROCK 4D).  With GOP size 1 the driver produces a valid H.264 stream the
reference decoder accepts.  This already exercises the full path: V4L2
m2m, the register programming, the software SPS/PPS prepend, and the
hardware slice output.

There is no upstream userspace involved (an encoder needs no request
API); I drive it with a small ioctl test program that feeds NV12 frames
and writes the CAPTURE buffers to a .h264 file.

The open problem: inter (P-frame) frames hang
---------------------------------------------

Every P-frame stalls the encoder's own hardware watchdog (INT_STA bit 8,
~20 ms after the kick) and produces essentially no bitstream.  I have
spent a lot of time narrowing this; it is sharply localized but I cannot
close it:

  - The P-frame register writes match a real register-write trace of the
    vendor stack encoding the same content, byte for byte (I traced the
    vendor kernel's writes and diffed against this driver's).

  - The reconstruction the previous frame writes is *valid*: dumping the
    recon buffer after the I-frame shows correct reconstructed pixels.

  - If the P-frame reads its own (older, settled) recon slot instead of
    the immediately-preceding frame's, it completes.  Reading the
    immediately-preceding frame's *fresh* reconstruction as the
    reference is what hangs.

  - It is not FBC: storing the reconstruction uncompressed
    (enc_pic.rec_fbc_dis = 1) still hangs.

  - The first frame of a session always works; the first P-frame hangs;
    after a couple of failures + core resets, later frames sometimes
    start completing.  It behaves like a warm-up / settling problem on
    the reference-read path, not a wrong register value.

So the encoder programs identically to the vendor and reads a valid
reference, but stalls fetching the previous frame's reconstruction as
the inter reference on the first inter frame of a session.  My best
guess is that the vendor does something between consecutive frame
submissions -- a completion/drain wait, a cache/coherency step, or a
per-frame re-arm -- that I am missing, but I have not found it.  If
anyone recognizes this on VEPU5xx, or knows what the reference-read path
needs between frames, a pointer would be very welcome.

The shape of this -- the first operation of a power session works, the
next stalls, and a reset/warm-up sometimes helps -- is the same one I
ran into bringing up this SoC's NPU (accel/rocket RKNN), where the
second/chained submit in a session would not fire [1].  I can't claim
they share a root cause, but if this is a known RK3576-wide submit /
re-arm quirk rather than an encoder-specific bug, that would be good to
know.

[1] https://lore.kernel.org/all/20260718031146.3368811-1-gahing@gahingwoo.com/

Notes for review
----------------

  - The PARAM/SQI register classes are programmed from mpp's constant
    "default tuning" tables (not derived per-frame), as noted in the
    code.  They are required: leaving them unwritten stalls even the
    I-frame.

  - Rate control is fixed-QP only for now (the bitrate control is
    advisory).

  - H.264 baseline/main, single slice, 4:2:0 only.

  - Tested at 176x144; other resolutions are not yet validated.

Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>

Jiaxing Hu (3):
  dt-bindings: media: add Rockchip RK3576 VEPU H.264 encoder
  media: rockchip: add VEPU510 H.264 encoder driver for RK3576
  arm64: dts: rockchip: rk3576: add VEPU H.264 encoder nodes

 .../bindings/media/rockchip,rk3576-vepu.yaml       |   94 ++
 arch/arm64/boot/dts/rockchip/rk3576.dtsi           |   50 +
 drivers/media/platform/rockchip/Kconfig            |    1 +
 drivers/media/platform/rockchip/Makefile           |    1 +
 drivers/media/platform/rockchip/rkvenc/Kconfig     |   14 +
 drivers/media/platform/rockchip/rkvenc/Makefile    |    6 +
 .../media/platform/rockchip/rkvenc/rkvenc-h264.c   | 1095 ++++++++++++++++++++
 .../media/platform/rockchip/rkvenc/rkvenc-regs.h   |  929 +++++++++++++++++
 drivers/media/platform/rockchip/rkvenc/rkvenc.c    |  892 ++++++++++++++++
 drivers/media/platform/rockchip/rkvenc/rkvenc.h    |  212 ++++
 10 files changed, 3294 insertions(+)
--
2.43.0

[RFC PATCH 1/3] dt-bindings: media: add Rockchip RK3576 VEPU H.264 encoder

[Jiaxing Hu]

Add the binding for the VEPU510 H.264 hardware video encoder found on the
Rockchip RK3576.  Each SoC has two independent encoder cores, each behind
its own IOMMU.

Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>
---
 .../bindings/media/rockchip,rk3576-vepu.yaml       | 94 ++++++++++++++++++++++
 1 file changed, 94 insertions(+)

diff --git a/Documentation/devicetree/bindings/media/rockchip,rk3576-vepu.yaml b/Documentation/devicetree/bindings/media/rockchip,rk3576-vepu.yaml
new file mode 100644
index 000000000..942118cb3
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/rockchip,rk3576-vepu.yaml
@@ -0,0 +1,94 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/media/rockchip,rk3576-vepu.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Rockchip VEPU510 Video Encoder (RK3576)
+
+maintainers:
+  - Jiaxing Hu <gahing@gahingwoo.com>
+
+description: |-
+  RK3576 has two identical VEPU510 hardware H.264/H.265 video encoder
+  cores. Each core is a standalone V4L2 mem2mem device in this binding;
+  there is no shared hardware descriptor/link-list engine tying the two
+  cores together (unlike the RK3576 video decoder's CCU), so each core
+  node is independent and self-contained.
+
+properties:
+  compatible:
+    const: rockchip,rk3576-vepu
+
+  reg:
+    maxItems: 1
+
+  interrupts:
+    maxItems: 1
+
+  clocks:
+    items:
+      - description: The Video Encoder AXI interface clock
+      - description: The Video Encoder AHB interface clock
+      - description: The Video Encoder core clock
+
+  clock-names:
+    items:
+      - const: aclk_vcodec
+      - const: hclk_vcodec
+      - const: clk_core
+
+  assigned-clocks: true
+
+  assigned-clock-rates: true
+
+  resets:
+    items:
+      - description: The Video Encoder AXI interface reset
+      - description: The Video Encoder AHB interface reset
+      - description: The Video Encoder core reset
+
+  reset-names:
+    items:
+      - const: video_a
+      - const: video_h
+      - const: video_core
+
+  power-domains:
+    maxItems: 1
+
+  iommus:
+    maxItems: 1
+
+required:
+  - compatible
+  - reg
+  - interrupts
+  - clocks
+  - clock-names
+  - power-domains
+  - iommus
+
+additionalProperties: false
+
+examples:
+  - |
+    #include <dt-bindings/interrupt-controller/arm-gic.h>
+    #include <dt-bindings/clock/rockchip,rk3576-cru.h>
+    #include <dt-bindings/power/rockchip,rk3576-power.h>
+
+    vepu0: video-codec@27a00000 {
+        compatible = "rockchip,rk3576-vepu";
+        reg = <0x27a00000 0x6000>;
+        interrupts = <GIC_SPI 310 IRQ_TYPE_LEVEL_HIGH>;
+        clocks = <&cru ACLK_VEPU0>, <&cru HCLK_VEPU0>, <&cru CLK_VEPU0_CORE>;
+        clock-names = "aclk_vcodec", "hclk_vcodec", "clk_core";
+        assigned-clocks = <&cru ACLK_VEPU0>, <&cru CLK_VEPU0_CORE>;
+        assigned-clock-rates = <400000000>, <702000000>;
+        resets = <&cru SRST_A_VEPU0>, <&cru SRST_H_VEPU0>, <&cru SRST_VEPU0_CORE>;
+        reset-names = "video_a", "video_h", "video_core";
+        power-domains = <&power RK3576_PD_VEPU0>;
+        iommus = <&vepu0_mmu>;
+    };
+
+...
-- 
2.43.0

[RFC PATCH 3/3] arm64: dts: rockchip: rk3576: add VEPU H.264 encoder nodes

[Jiaxing Hu]

Add the two VEPU510 encoder cores at 0x27a00000 / 0x27a10000 and their
IOMMUs.

Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>
---
 arch/arm64/boot/dts/rockchip/rk3576.dtsi | 50 ++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/arch/arm64/boot/dts/rockchip/rk3576.dtsi b/arch/arm64/boot/dts/rockchip/rk3576.dtsi
index b622c6fee..9bf4f2c41 100644
--- a/arch/arm64/boot/dts/rockchip/rk3576.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3576.dtsi
@@ -1320,6 +1320,56 @@ vdec_mmu: iommu@27b00800 {
 			#iommu-cells = <0>;
 		};
 
+		vepu0: video-codec@27a00000 {
+			compatible = "rockchip,rk3576-vepu";
+			reg = <0x0 0x27a00000 0x0 0x6000>;
+			interrupts = <GIC_SPI 310 IRQ_TYPE_LEVEL_HIGH>;
+			clocks = <&cru ACLK_VEPU0>, <&cru HCLK_VEPU0>, <&cru CLK_VEPU0_CORE>;
+			clock-names = "aclk_vcodec", "hclk_vcodec", "clk_core";
+			assigned-clocks = <&cru ACLK_VEPU0>, <&cru CLK_VEPU0_CORE>;
+			assigned-clock-rates = <400000000>, <702000000>;
+			resets = <&cru SRST_A_VEPU0>, <&cru SRST_H_VEPU0>, <&cru SRST_VEPU0_CORE>;
+			reset-names = "video_a", "video_h", "video_core";
+			power-domains = <&power RK3576_PD_VEPU0>;
+			iommus = <&vepu0_mmu>;
+		};
+
+		vepu0_mmu: iommu@27a0f000 {
+			compatible = "rockchip,rk3576-iommu", "rockchip,rk3568-iommu";
+			reg = <0x0 0x27a0f000 0x0 0x40>;
+			interrupts = <GIC_SPI 311 IRQ_TYPE_LEVEL_HIGH>;
+			clocks = <&cru ACLK_VEPU0>, <&cru HCLK_VEPU0>;
+			clock-names = "aclk", "iface";
+			power-domains = <&power RK3576_PD_VEPU0>;
+			rockchip,disable-mmu-reset;
+			#iommu-cells = <0>;
+		};
+
+		vepu1: video-codec@27a10000 {
+			compatible = "rockchip,rk3576-vepu";
+			reg = <0x0 0x27a10000 0x0 0x6000>;
+			interrupts = <GIC_SPI 387 IRQ_TYPE_LEVEL_HIGH>;
+			clocks = <&cru ACLK_VEPU1>, <&cru HCLK_VEPU1>, <&cru CLK_VEPU1_CORE>;
+			clock-names = "aclk_vcodec", "hclk_vcodec", "clk_core";
+			assigned-clocks = <&cru ACLK_VEPU1>, <&cru CLK_VEPU1_CORE>;
+			assigned-clock-rates = <400000000>, <702000000>;
+			resets = <&cru SRST_A_VEPU1>, <&cru SRST_H_VEPU1>, <&cru SRST_VEPU1_CORE>;
+			reset-names = "video_a", "video_h", "video_core";
+			power-domains = <&power RK3576_PD_VEPU1>;
+			iommus = <&vepu1_mmu>;
+		};
+
+		vepu1_mmu: iommu@27a1f000 {
+			compatible = "rockchip,rk3576-iommu", "rockchip,rk3568-iommu";
+			reg = <0x0 0x27a1f000 0x0 0x40>;
+			interrupts = <GIC_SPI 388 IRQ_TYPE_LEVEL_HIGH>;
+			clocks = <&cru ACLK_VEPU1>, <&cru HCLK_VEPU1>;
+			clock-names = "aclk", "iface";
+			power-domains = <&power RK3576_PD_VEPU1>;
+			rockchip,disable-mmu-reset;
+			#iommu-cells = <0>;
+		};
+
 		vop: vop@27d00000 {
 			compatible = "rockchip,rk3576-vop";
 			reg = <0x0 0x27d00000 0x0 0x3000>, <0x0 0x27d05000 0x0 0x1000>;
-- 
2.43.0

Re: [RFC PATCH 2/3] media: rockchip: add VEPU510 H.264 encoder driver for RK3576

[Heiko Stübner]

Am Mittwoch, 22. Juli 2026, 09:34:16 Mitteleuropäische Sommerzeit schrieb Jiaxing Hu:
> Add a from-scratch stateful V4L2 mem2mem driver for the Rockchip RK3576
> VEPU510 H.264 hardware video encoder (raw NV12 in, H.264 Annex-B out),
> modelled on the verisilicon/hantro device_run()/codec_ops split rather
> than the downstream MPP-service/task-queue model.
> 
> The two encoder cores (rkvenc0/rkvenc1) are exposed as two independent
> V4L2 M2M device nodes, each driving one physical core standalone; the
> downstream vendor CCU cross-core load-balancing is not implemented.
> 
> Register field semantics were worked out by trial-and-error against
> Rockchip's own open-source userspace codec library (rockchip-linux/mpp)
> and cross-checked against a real register-write trace of the downstream
> vendor stack running the same encode.
> 
> Intra (I-frame) encoding is confirmed working on real hardware (Radxa
> ROCK 4D): it produces valid H.264 that the reference decoders accept.
> Inter (P-frame) encoding still hits a hardware-watchdog stall in the
> reference-read datapath -- this is the main open question for this RFC;
> see the cover letter for the full symptom analysis.
> 
> Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>

[...]

> diff --git a/drivers/media/platform/rockchip/rkvenc/rkvenc.c b/drivers/media/platform/rockchip/rkvenc/rkvenc.c
> new file mode 100644
> index 000000000..d70bcc5fb
> --- /dev/null
> +++ b/drivers/media/platform/rockchip/rkvenc/rkvenc.c
> @@ -0,0 +1,892 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Rockchip VEPU510 (RK3576) hardware video encoder driver.
> + *
> + * Copyright (C) 2026 Jiaxing Hu <gahing@gahingwoo.com>
> + *
> + * Architecture notes (see also rkvenc-regs.h):
> + *
> + *  - This is a *stateful* V4L2 mem2mem encoder (raw NV12 in on OUTPUT,
> + *    H.264 Annex-B out on CAPTURE) modelled on
> + *    drivers/media/platform/verisilicon's hantro_drv.c device_run()/
> + *    codec_ops{run,done} split, NOT on rkvdec's stateless request-API
> + *    decoder pattern — an encoder has no per-frame bitstream to parse,
> + *    so there is nothing analogous to rkvdec_run_preamble/postamble here.
> + *
> + *  - The vendor downstream driver (rockchip-linux/kernel,
> + *    drivers/video/rockchip/mpp/mpp_rkvenc2.c) groups rkvenc0/rkvenc1
> + *    under a "CCU" (rockchip,rkv-encoder-rk3576-ccu) that does pure
> + *    software task-queue load balancing across both cores, plus an
> + *    optional DCHS (dual-core-handshake) register protocol used only
> + *    when *deliberately* splitting one frame's rows across both cores.
> + *    There is no hardware descriptor/link-list engine behind it (unlike
> + *    the decoder's CCU). v1 of this driver does not implement either:
> + *    rkvenc0 and rkvenc1 are exposed as two independent V4L2 M2M device
> + *    nodes, each driving one physical core standalone — the same
> + *    simplification rkvdec itself makes for multi-core VDPU hardware
> + *    (see rkvdec_disable_multicore()).

At least the comment and from a casual look also the code get this
backwards. The idea is to explicitly _not_ expose multiple video devices.

See
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/platform/rockchip/rkvdec/rkvdec.c#n1613

Any (future) scheduling should happen inside the driver, and not get
offloaded onto _every_ userspace application individually.


Heiko

Re: [RFC PATCH 0/3] media: rockchip: VEPU510 H.264 encoder for RK3576

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 8617 bytes --]

Hi,

Le mercredi 22 juillet 2026 à 19:34 +1200, Jiaxing Hu a écrit :
> This is an RFC for a from-scratch mainline V4L2 driver for the H.264
> hardware video encoder (VEPU510) on the Rockchip RK3576.  It is a
> stateful mem2mem encoder (NV12 in, H.264 Annex-B out) modelled on the
> verisilicon/hantro driver, not on the downstream MPP-service model.
> 
> I'm posting it as an RFC because intra frames work but inter frames do
> not, and I'd like a second pair of eyes on the inter-frame problem
> before this is worth a real submission.

This is really nice to see interest in enabling encoder for this chips set. I'm
adding Detlev in CC here, as he's actively working on RK3588 encoder, which is
quite a similar chip. But there is quite a twist his approach which I'll explain
shortly.

What I believe I've seen walking through your RFC is an encoder based on V4L2
Stateful encoder interface:

https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-encoder.html

While this isn't invalid, specially for encoders, for this type of hardware, the
community agreed direction was to introduce V4L2 Stateless Encoder
specification, using the media request and compound controls to maintain a lower
level interface for this HW. See Paul's proposal for IMX8MP Hantro VC8000E chips
(H.264 only for now):

https://lore.kernel.org/all/20260522101653.2565125-1-paulk@sys-base.io/

The downside is that we need a new spec document for this class of driver to
reach upstream (not part of Paul's series). Plus, for each codec, appropriate
encode parameter controls needs to be designed, and they must be proven to work
across multiple hardware (generally at least 2). I still think there is a need
for V4L2 drivers like this. Being high level, they offer a great interface to
protect against HW actions that would be harmful to the Linux operating system.
Specially when you don't have an IOMMU like the IMX8M Plus. But at the same
time, V4L2 memory model often clash with modern graphic stack, making the
integration quite painful.

Here's the twist to Detlev (and myself) approach. In order to make the kernel
module even thinner, and to avoid having to specify new V4L2 interfaces, we
decided to look into making drivers similar to how we do GPU drivers today. This
is possible today thanks to the Vulkan Video standard. So Detlev driver, which
finally got P-Frames support few days ago (he'll explain why this didn't work
initially for him), is split in two part:

- The kernel module, which expose hardware specific interface (he decided to
implement it in Rust)
- The userspace driver, which he's implementing in Mesa project

What I really like of this approach is that the kernel driver is much more
stable, as it expose the HW at a lower level in a codec agnostic way. For the
final application, the interface is the same regardless if its a GPU attached
codec or a standalone codec. Its even the same interface if you are on Windows,
except for the low level zero-copy interop. The rest of the development can
happen in userspace, which is a lot faster to develop and easier to debug.

One thing we notice, is that for performance reason, keeping rate control as
close to the IRQ as possible is important. Its impossible to batch any work if
you have to constantly wait for the previous work to be done in order to compute
the next set of QP values. For that reason, we plan to eventually share these
in-kernel implementation with V4L2. In our imagination, eBPF could also be a
usable option.

I will leave to Detlev to share some early code once he's ready, but considering
you both are working on the same family of encoder, it would certainly be a lot
nicer if both endup with the same type of drivers. Offering two interface for
the same hardware would impose a lot of extra effort that I would rather avoid.

regards,
Nicolas

> 
> What works
> ----------
> 
> Intra-only (I-frame) encoding is confirmed on real hardware (Radxa
> ROCK 4D).  With GOP size 1 the driver produces a valid H.264 stream the
> reference decoder accepts.  This already exercises the full path: V4L2
> m2m, the register programming, the software SPS/PPS prepend, and the
> hardware slice output.
> 
> There is no upstream userspace involved (an encoder needs no request
> API); I drive it with a small ioctl test program that feeds NV12 frames
> and writes the CAPTURE buffers to a .h264 file.
> 
> The open problem: inter (P-frame) frames hang
> ---------------------------------------------
> 
> Every P-frame stalls the encoder's own hardware watchdog (INT_STA bit 8,
> ~20 ms after the kick) and produces essentially no bitstream.  I have
> spent a lot of time narrowing this; it is sharply localized but I cannot
> close it:
> 
>   - The P-frame register writes match a real register-write trace of the
>     vendor stack encoding the same content, byte for byte (I traced the
>     vendor kernel's writes and diffed against this driver's).
> 
>   - The reconstruction the previous frame writes is *valid*: dumping the
>     recon buffer after the I-frame shows correct reconstructed pixels.
> 
>   - If the P-frame reads its own (older, settled) recon slot instead of
>     the immediately-preceding frame's, it completes.  Reading the
>     immediately-preceding frame's *fresh* reconstruction as the
>     reference is what hangs.
> 
>   - It is not FBC: storing the reconstruction uncompressed
>     (enc_pic.rec_fbc_dis = 1) still hangs.
> 
>   - The first frame of a session always works; the first P-frame hangs;
>     after a couple of failures + core resets, later frames sometimes
>     start completing.  It behaves like a warm-up / settling problem on
>     the reference-read path, not a wrong register value.
> 
> So the encoder programs identically to the vendor and reads a valid
> reference, but stalls fetching the previous frame's reconstruction as
> the inter reference on the first inter frame of a session.  My best
> guess is that the vendor does something between consecutive frame
> submissions -- a completion/drain wait, a cache/coherency step, or a
> per-frame re-arm -- that I am missing, but I have not found it.  If
> anyone recognizes this on VEPU5xx, or knows what the reference-read path
> needs between frames, a pointer would be very welcome.
> 
> The shape of this -- the first operation of a power session works, the
> next stalls, and a reset/warm-up sometimes helps -- is the same one I
> ran into bringing up this SoC's NPU (accel/rocket RKNN), where the
> second/chained submit in a session would not fire [1].  I can't claim
> they share a root cause, but if this is a known RK3576-wide submit /
> re-arm quirk rather than an encoder-specific bug, that would be good to
> know.
> 
> [1] https://lore.kernel.org/all/20260718031146.3368811-1-gahing@gahingwoo.com/
> 
> Notes for review
> ----------------
> 
>   - The PARAM/SQI register classes are programmed from mpp's constant
>     "default tuning" tables (not derived per-frame), as noted in the
>     code.  They are required: leaving them unwritten stalls even the
>     I-frame.
> 
>   - Rate control is fixed-QP only for now (the bitrate control is
>     advisory).
> 
>   - H.264 baseline/main, single slice, 4:2:0 only.
> 
>   - Tested at 176x144; other resolutions are not yet validated.
> 
> Signed-off-by: Jiaxing Hu <gahing@gahingwoo.com>
> 
> Jiaxing Hu (3):
>   dt-bindings: media: add Rockchip RK3576 VEPU H.264 encoder
>   media: rockchip: add VEPU510 H.264 encoder driver for RK3576
>   arm64: dts: rockchip: rk3576: add VEPU H.264 encoder nodes
> 
>  .../bindings/media/rockchip,rk3576-vepu.yaml       |   94 ++
>  arch/arm64/boot/dts/rockchip/rk3576.dtsi           |   50 +
>  drivers/media/platform/rockchip/Kconfig            |    1 +
>  drivers/media/platform/rockchip/Makefile           |    1 +
>  drivers/media/platform/rockchip/rkvenc/Kconfig     |   14 +
>  drivers/media/platform/rockchip/rkvenc/Makefile    |    6 +
>  .../media/platform/rockchip/rkvenc/rkvenc-h264.c   | 1095
> ++++++++++++++++++++
>  .../media/platform/rockchip/rkvenc/rkvenc-regs.h   |  929 +++++++++++++++++
>  drivers/media/platform/rockchip/rkvenc/rkvenc.c    |  892 ++++++++++++++++
>  drivers/media/platform/rockchip/rkvenc/rkvenc.h    |  212 ++++
>  10 files changed, 3294 insertions(+)
> --
> 2.43.0
> 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [RFC PATCH 0/3] media: rockchip: VEPU510 H.264 encoder for RK3576

[Jiaxing Hu]

On Tue, 22 Jul 2026 20:29:53 +0200, Nicolas Dufresne wrote:
> This is really nice to see interest in enabling encoder for this chips
> set. I'm adding Detlev in CC here, as he's actively working on RK3588
> encoder, which is quite a similar chip.

Thanks, and thanks for looping Detlev in.

> So Detlev driver, which finally got P-Frames support few days ago
> (he'll explain why this didn't work initially for him) [...]

This is the most useful sentence in the thread for me. RK3576's VEPU510
and RK3588's VEPU580 are the same VEPU5xx family -- same vendor HAL flow,
identical reconstruction/reference-read (recn_refr) path -- and that
reference read is exactly where every P-frame stalls for me: the first
inter frame of a session hangs the encoder watchdog fetching the previous
frame's reconstruction, even though the reconstruction is valid and my
register writes match the vendor byte for byte. So whatever Detlev hit is
very likely the same bug.

Detlev -- whatever you can share about why P-frames didn't work at first
would help me enormously.

> While this isn't invalid [...] the community agreed direction was to
> introduce V4L2 Stateless Encoder specification [...]
> [...] we decided to look into making drivers similar to how we do GPU
> drivers today. This is possible today thanks to the Vulkan Video
> standard.

I'm not attached to the stateful interface -- I used it to get something
running on real silicon, not out of conviction. Before I commit to
reworking, I'd like to be sure I understand what "converge" means here,
because I see two fairly different directions in the thread: Paul's V4L2
stateless H.264 encoder (kernel-side reflist/rbsp/RC core), and the
Vulkan-Video split you describe for Detlev (thin kernel module + Mesa
userspace). Those differ a lot in kind and in effort. Which is the
intended target for the Rockchip encoders, or is that still open? Is
there a branch, early code or spec draft I should read beyond Paul's
series?

Either way the RK3576 hardware enablement and the reference-read fix are
shared with RK3588, so I'd much rather contribute the VEPU510 side to a
common base than maintain a parallel one -- I'd just like to know which
base that is before rebuilding on it.

Thanks,
Jiaxing

Re: [RFC PATCH 2/3] media: rockchip: add VEPU510 H.264 encoder driver for RK3576

[Jiaxing Hu]

On Wed, 22 Jul 2026 12:00:27 +0200, Heiko Stübner wrote:
> At least the comment and from a casual look also the code get this
> backwards. The idea is to explicitly _not_ expose multiple video
> devices.
> [...]
> Any (future) scheduling should happen inside the driver, and not get
> offloaded onto _every_ userspace application individually.

You're right, thanks. Exposing rkvenc0/rkvenc1 as two nodes and pushing
the core choice onto userspace is backwards; rkvdec_disable_multicore()
is the pattern to follow -- probe only the first core, keep any
multi-core scheduling inside the driver. I'll fix that.

Depending on where the interface discussion in the 0/3 thread lands
(Nicolas is pointing at a lower-level stateless / Vulkan-Video direction
that Detlev is already building for RK3588) this driver's shape may
change quite a bit, but the "don't offload core selection to userspace"
point holds either way.

Thanks,
Jiaxing

Re: [RFC PATCH 0/3] media: rockchip: VEPU510 H.264 encoder for RK3576

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 2981 bytes --]

Hi,

Le jeudi 23 juillet 2026 à 12:46 +1200, Jiaxing Hu a écrit :
> I'm not attached to the stateful interface -- I used it to get something
> running on real silicon, not out of conviction. Before I commit to
> reworking, I'd like to be sure I understand what "converge" means here,
> because I see two fairly different directions in the thread: Paul's V4L2
> stateless H.264 encoder (kernel-side reflist/rbsp/RC core), and the
> Vulkan-Video split you describe for Detlev (thin kernel module + Mesa
> userspace). Those differ a lot in kind and in effort. Which is the
> intended target for the Rockchip encoders, or is that still open? Is
> there a branch, early code or spec draft I should read beyond Paul's
> series?

This is a fair point, and something that will certainly bring some level of
confusion in the short term. My personal goal, and what Detlev is actively
working on, is to move all codecs drivers toward Vulkan Video, using the most
meaningful driver interface for the purpose. The drivers needs to stay safe
though, so it matters that the HW can protect memory access.

Detlev and I picked the RKVENC because there was no upstream driver for it, so a
good place for a clean transition. It has an IOMMU and can ensure per process
memory access protection. If a process miss-program the encoder and make it
corrupt on other encoding session it owns, its all right, its not different then
writing a C program with multiple thread and corrupting another thread. But
corruption other process session, or the system is not acceptable, and something
Linux driver must protect against.

Paul does not have this hardware protection, so I think he needs a higher level
interface for the driver, and at the point, why not base it on existing kernel
interface. Though, V4l2 is massively larger (probably around 100x) interface
then what we are drafting currently for RKVENC in a drm style driver.

The common part is the rate control. Typically, stateful encoder are firmware
based, and the rate control is in the firmware. We need an in kernel rate
control for performance reason. The programming latency is going to be too high
otherwise. The point of convergence, is that we can share helpers for that
purpose, and Paul's work is really clean, and already in the shape of helpers
not tied to v4l2.

> Either way the RK3576 hardware enablement and the reference-read fix are
> shared with RK3588, so I'd much rather contribute the VEPU510 side to a
> common base than maintain a parallel one -- I'd just like to know which
> base that is before rebuilding on it.

This is our experience with decoder too, its relatively easy to support both,
differences being minor. I will ping Detlev today and see if he can help here. I
know in his case he was trying to avoid reconstructed frame compression, and it
only started working when he enable that compression. Though, he had mmu faults
prior to that.

cheers,
Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [RFC PATCH 0/3] media: rockchip: VEPU510 H.264 encoder for RK3576

[Paul Kocialkowski]

[-- Attachment #1: Type: text/plain, Size: 4570 bytes --]

Hi,

Le Thu 23 Jul 26, 12:14, Nicolas Dufresne a écrit :
> Le jeudi 23 juillet 2026 à 12:46 +1200, Jiaxing Hu a écrit :
> > I'm not attached to the stateful interface -- I used it to get something
> > running on real silicon, not out of conviction. Before I commit to
> > reworking, I'd like to be sure I understand what "converge" means here,
> > because I see two fairly different directions in the thread: Paul's V4L2
> > stateless H.264 encoder (kernel-side reflist/rbsp/RC core), and the
> > Vulkan-Video split you describe for Detlev (thin kernel module + Mesa
> > userspace). Those differ a lot in kind and in effort. Which is the
> > intended target for the Rockchip encoders, or is that still open? Is
> > there a branch, early code or spec draft I should read beyond Paul's
> > series?

I would recommend taking a look at the slides of the talks I gave at the
Linux Media Summit last year and this year, which should give you more
context and insight about the design choices.

- https://paulk.fr/talks/2025-linux-media-summit/2025-linux-media-summit-v4l2-stateless-video-encoding-uapi.pdf
- https://paulk.fr/talks/2026-linux-media-summit/2026-linux-media-summit-v4l2-stateless-video-encoding-uapi-progress-update.pdf

Generally speaking the idea of this proposal is to stick with the same
design idea that we had for stateless decoders, including V4L2 M2M queues,
dedicated controls and media requests.

Like Nicolas explained, some hardware can be a good fit for both
approaches. I think we agreed that it wouldn't be a problem to have two
drivers (one v4l2, one drm-ish) for the same hardware, so the question is
really about what you prefer to do and get involved with.

My V4L2-based proposal is already rather usable, although it has not yet
reached its final form. I have a GStreamer implementation ready that has
been used extensively too, but it will also need to be reworked before
it can reach upstream.

The DRM-ish proposal is at an earlier stage and will likely require a
complete redesign of your driver, moving most things in userspace.

> Paul does not have this hardware protection, so I think he needs a higher level
> interface for the driver, and at the point, why not base it on existing kernel
> interface. Though, V4l2 is massively larger (probably around 100x) interface
> then what we are drafting currently for RKVENC in a drm style driver.
> 
> The common part is the rate control. Typically, stateful encoder are firmware
> based, and the rate control is in the firmware. We need an in kernel rate
> control for performance reason. The programming latency is going to be too high
> otherwise. The point of convergence, is that we can share helpers for that
> purpose, and Paul's work is really clean, and already in the shape of helpers
> not tied to v4l2.

Well there is some level of dependency to v4l2 in some places, but I'm
sure we can extract relevant parts and make them common so that a drm-ish
subsystem can reuse them easily.

> > Either way the RK3576 hardware enablement and the reference-read fix are
> > shared with RK3588, so I'd much rather contribute the VEPU510 side to a
> > common base than maintain a parallel one -- I'd just like to know which
> > base that is before rebuilding on it.
> 
> This is our experience with decoder too, its relatively easy to support both,
> differences being minor. I will ping Detlev today and see if he can help here. I
> know in his case he was trying to avoid reconstructed frame compression, and it
> only started working when he enable that compression. Though, he had mmu faults
> prior to that.

In my opinion it would be fine to have two drivers, but it's really up
to you. Since you already have a v4l2 base for it, I would encourage you
to move it to my proposal, which will probably be finalized in a shorter
time frame than the other proposal and lets you reuse the code you've
already written. Then you could also add RK3576 support to the drm-ish
Vulkan Video proposal without too much work when it is ready, reusing the
RK3588 work.

But again, it's really your decision. And it's hard to say how long
things will take to finalize. But I think it would be beneficial to have
two drivers for this hardware and not trash the work you've done.

All the best,

Paul

-- 
Paul Kocialkowski,

Independent contractor - sys-base - https://www.sys-base.io/
Free software developer - https://www.paulk.fr/

Expert in multimedia, graphics and embedded hardware support with Linux.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC PATCH 0/3] media: rockchip: VEPU510 H.264 encoder for RK3576

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 1534 bytes --]

Hi Paul.

Le jeudi 23 juillet 2026 à 20:41 +0200, Paul Kocialkowski a écrit :
> In my opinion it would be fine to have two drivers, but it's really up
> to you. Since you already have a v4l2 base for it, I would encourage you
> to move it to my proposal, which will probably be finalized in a shorter
> time frame than the other proposal and lets you reuse the code you've
> already written. Then you could also add RK3576 support to the drm-ish
> Vulkan Video proposal without too much work when it is ready, reusing the
> RK3588 work.

My main concern is that once you exposed an API, the transition path is near
impossible unless you accept to support both APIs concurrently. So I may raise a
slight objection on the above proposal. Once we have the code shared (Detlev is
working on it), you'll see that Detlev (and Daniel Almeida) implementation is
just a step ahead, and adding RK3576 to an RK3588 is just a minor update (its
the same chip, different minor version). On top of which, once the kernel driver
will have settled, adding more codecs will mostly (or entirely) happen in
userspace.

With no offense, the RFC here requires significant work before it can be
upstreamed. A lot of the code is hex tables generated from inspecting a running
driver. So there is a lot of work to decipher this into clean code.

On that aspect, if you are missing any reference code or technical
documentation, let us know, though I believe most of this is public information
for this chip.

regards,
Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [RFC PATCH 0/3] media: rockchip: VEPU510 H.264 encoder for RK3576

[Paul Kocialkowski]

[-- Attachment #1: Type: text/plain, Size: 2685 bytes --]

Hi Nicolas,

Le Thu 23 Jul 26, 15:39, Nicolas Dufresne a écrit :
> Hi Paul.
> 
> Le jeudi 23 juillet 2026 à 20:41 +0200, Paul Kocialkowski a écrit :
> > In my opinion it would be fine to have two drivers, but it's really up
> > to you. Since you already have a v4l2 base for it, I would encourage you
> > to move it to my proposal, which will probably be finalized in a shorter
> > time frame than the other proposal and lets you reuse the code you've
> > already written. Then you could also add RK3576 support to the drm-ish
> > Vulkan Video proposal without too much work when it is ready, reusing the
> > RK3588 work.
> 
> My main concern is that once you exposed an API, the transition path is near
> impossible unless you accept to support both APIs concurrently. So I may raise a
> slight objection on the above proposal.

Well there could be a recommended implementation using your design and a
more experimental v4l2 implementation that is discouraged to use but
still available.

But that means another driver to maintain and more long-term involvement
from Jiaxing Hu. I am not sure it is worth it, but I don't think we
should entirely close the door on the idea. I also understand that there
can be a bit of frustration with trashing code and a desire to continue
working on it (especially now that it can benefit from the technical
knowledge acquired from your work). But I'm just speculating here.

Still I would be interested to eventually be able to compare performance
between the two approaches on the same hardware :)

> Once we have the code shared (Detlev is
> working on it), you'll see that Detlev (and Daniel Almeida) implementation is
> just a step ahead, and adding RK3576 to an RK3588 is just a minor update (its
> the same chip, different minor version). On top of which, once the kernel driver
> will have settled, adding more codecs will mostly (or entirely) happen in
> userspace.
> 
> With no offense, the RFC here requires significant work before it can be
> upstreamed. A lot of the code is hex tables generated from inspecting a running
> driver. So there is a lot of work to decipher this into clean code.

Yes I'm sure there is still a lot of work to do.

All the best,

Paul

> On that aspect, if you are missing any reference code or technical
> documentation, let us know, though I believe most of this is public information
> for this chip.
> 
> regards,
> Nicolas



-- 
Paul Kocialkowski,

Independent contractor - sys-base - https://www.sys-base.io/
Free software developer - https://www.paulk.fr/

Expert in multimedia, graphics and embedded hardware support with Linux.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC PATCH 0/3] media: rockchip: VEPU510 H.264 encoder for RK3576

[Jiaxing Hu]

Hi Nicolas, Paul, Detlev,

Thanks both for laying out the trade-offs so clearly.

Decision from my side: I'll converge on the DRM-ish / Vulkan Video
direction and sync with Detlev, rather than push the stateful V4L2
driver further. Nicolas' point about UAPI lock-in is the deciding one
for me -- I don't want to expose an encoder UAPI that then has to be
supported forever -- and if RK3576 is just a minor delta on Detlev's
RK3588 base, that's exactly where my hardware work is worth the most.

Paul -- thank you for the generous offer. I read both your Media Summit
decks and your posted series ([PATCH 00/14], the generic in-kernel
h264-enc core + rbsp + rate control on VC8000E); it's clearly further
along than I'd assumed. I'm not closing the door on a V4L2 version
later, but I'd rather not commit to maintaining two drivers up front, so
I'll treat that as a possible follow-up, not the primary path.

Where I think I can be useful right now is the hardware, since that part
is shared with RK3588 regardless of interface:

> I know in his case he was trying to avoid reconstructed frame
> compression, and it only started working when he enable that
> compression. Though, he had mmu faults prior to that.

Useful data point, but let me be precise about where I already am, so I
don't send Detlev chasing something I've done. This driver already runs
with reconstruction compression enabled (enc_pic.rec_fbc_dis = 0) -- the
state Detlev needed -- and the first inter frame still hangs. Disabling
it (rec_fbc_dis = 1) was one of my experiments and also hung, so FBC
state alone doesn't explain my stall.

The part that does line up is the mmu fault. I still have a residual
rk_iommu write fault on this path; I traced it from a boot-varying
garbage IOVA down to a benign IOVA 0, and I'd concluded it was a
separate AXI transaction from the recon write, independent of the
P-frame hang. Detlev's report -- that his mmu faults and his missing
P-frames cleared together -- is a direct reason to distrust that
"independent" conclusion and re-check whether the fault and the stall
share a root cause on RK3576 too.

Detlev -- when you have a moment: on RK3588, were the mmu fault and the
P-frame failure the same underlying problem? And did enabling
reconstruction compression clear the fault on its own, or did the
reference-read mapping (how the previous frame's reconstruction is
mapped for the encoder to fetch) need a separate change as well?
Whatever you can share, I'm happy to test on RK3576 and feed back a
Tested-by.

One point on the current code, since it came up:

> A lot of the code is hex tables generated from inspecting a running
> driver.

Fair, and I won't pretend otherwise -- the PARAM/SQI classes are
currently shipped as fixed tables taken from a captured encode rather
than computed. The one thing I'd add is that those particular values are
mpp's public default-tuning tables (the RDO lambda/cost and
subjective-quality tables in Rockchip's open-source mpp HAL, Apache-2.0),
so they're constants I can cite to mpp source rather than opaque state.
But I fully agree the driver needs significant cleanup before it's
upstream-worthy, and in the DRM-ish model most of that tuning moves to
userspace anyway, so a lot of it won't survive the transition in its
current form.

So: please point me at Detlev's shared branch / early code whenever it's
in a shape to build against, and I'll start porting the RK3576 hardware
bring-up onto it.

Thanks,
Jiaxing