Linux-ARM-Kernel Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling
@ 2026-11-20 23:41 Daniel Golle
  2026-11-20 23:41 ` [PATCH v2 1/7] media: rkvdec: hevc: apply request controls first Daniel Golle
                   ` (6 more replies)
  0 siblings, 7 replies; 8+ messages in thread
From: Daniel Golle @ 2026-11-20 23:41 UTC (permalink / raw)
  To: Detlev Casanova, Ezequiel Garcia, Mauro Carvalho Chehab,
	Heiko Stuebner, Nicolas Dufresne, Alex Bee, Hans Verkuil,
	Jonas Karlman, Sebastian Fricke, linux-media, linux-rockchip,
	linux-arm-kernel, linux-kernel

This series fixes how the VDPU381 and VDPU383 HEVC back ends of the
rkvdec driver handle the EXT_SPS_ST_RPS and EXT_SPS_LT_RPS controls.
The defects were found while bringing up the VDPU346 of the RK3568,
which uses the VDPU381 back end, on top of Christian Hewitt's VDPU346
series [1], which has since been continued in [2].

Patch 1 looks up the controls after the request was applied instead of
before, which fixes a use-after-free.

Patches 2 and 3 are about jobs which cannot be decoded because the RPS
controls are missing. The VDPU383 back end now completes the request
when it refuses such a job, and the VDPU381 back end refuses the job
as well instead of starting the hardware.

Patch 4 refuses RPS controls with fewer elements than the SPS counts,
and patch 5 limits the size of predicted short-term sets. Both fix out
of bounds accesses which userspace can trigger.

Patch 6 fixes wrong decoding caused by a short-term RPS cache which
only looked at the first set, and patch 7 adds a missing allocation
check.

Patches 1 to 6 carry Cc: stable, like the EXT SPS RPS bounds merged in
v7.3 (f0b9d7e5be06, 796b5c6d4f16, 052c5ed5a1d9).

Tested on a NanoPi R5C (RK3568) with Linux 6.18 and the VDPU346 series
backported: GStreamer fluster JCT-VC-HEVC_V1 stays at 140/147, an FFmpeg
v4l2request build that sets the RPS controls confirms the reads and the
short-term set cache, and the VDPU381 refusal removes a watchdog reset
of the RK3568. The same VDPU381 back end was exercised on a Radxa ROCK
5B (RK3588): H.264, HEVC and VP9 decode through rkvdec with no kernel
message across the fluster runs (HEVC 146/147), a per-patch A/B sweep
and a 30-minute stability run, at the baseline throughput. The VDPU383
back end has no hardware here, the code-only patches have no reproducer,
and nothing was tested on an RK3576.

All of this was done with v1. v2 changes the order of the patches and
the commit messages, the code after the series is the same as in v1,
and the tests were not repeated.

An LLM coding agent did the analysis, and helped with writing the
patches. A second instance, given only the code, reviewed the series,
and has tested and validated it on real hardware independently.

Changes since v1:
- Move "media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets"
  before the check of the control size (5/7 -> 3/7). A control which is
  too short is handled like a missing one, and this way that means a
  refused job in both back ends as soon as the check exists, instead
  of a decode with a stale RPS table on the VDPU381 (Nicolas Dufresne)
- Rename "media: rkvdec: hevc: bound the RPS reads by the control size"
  to "media: rkvdec: hevc: refuse RPS controls shorter than the SPS
  counts" (3/7 -> 4/7)
- Rewrite all commit messages: say what is wrong and what the patch
  achieves, without walking through the code (Nicolas Dufresne)
- State in patch 4 why the size cannot be checked when the control is
  set (Nicolas Dufresne)
- Rebase on linux-next
- No change to the code after the series

v1: https://lore.kernel.org/r/cover.1794645268.git.daniel@makrotopia.org

[1] https://lore.kernel.org/r/20260110053727.2295260-1-christianshewitt@gmail.com
[2] https://lore.kernel.org/r/cover.1794654074.git.daniel@makrotopia.org

Daniel Golle (7):
  media: rkvdec: hevc: apply request controls first
  media: rkvdec: vdpu383: complete refused HEVC requests
  media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets
  media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts
  media: rkvdec: hevc: bound predicted short-term RPS sets
  media: rkvdec: hevc: cache every short-term RPS set
  media: rkvdec: hevc: check the RPS scratch allocation

 .../rockchip/rkvdec/rkvdec-hevc-common.c      | 52 +++++++++++++------
 .../rockchip/rkvdec/rkvdec-hevc-common.h      | 11 +++-
 .../rockchip/rkvdec/rkvdec-vdpu381-hevc.c     | 24 +++++----
 .../rockchip/rkvdec/rkvdec-vdpu383-hevc.c     | 14 +++--
 4 files changed, 67 insertions(+), 34 deletions(-)


base-commit: 909ef7a9a30fa139e69e555e3606680f5cca7726
-- 
2.56.1


^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2026-11-20 23:44 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 23:41 [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling Daniel Golle
2026-11-20 23:41 ` [PATCH v2 1/7] media: rkvdec: hevc: apply request controls first Daniel Golle
2026-11-20 23:41 ` [PATCH v2 2/7] media: rkvdec: vdpu383: complete refused HEVC requests Daniel Golle
2026-11-20 23:42 ` [PATCH v2 3/7] media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets Daniel Golle
2026-11-20 23:42 ` [PATCH v2 4/7] media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts Daniel Golle
2026-11-20 23:43 ` [PATCH v2 5/7] media: rkvdec: hevc: bound predicted short-term RPS sets Daniel Golle
2026-11-20 23:43 ` [PATCH v2 6/7] media: rkvdec: hevc: cache every short-term RPS set Daniel Golle
2026-11-20 23:44 ` [PATCH v2 7/7] media: rkvdec: hevc: check the RPS scratch allocation Daniel Golle

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox