From: Daniel Golle <daniel@makrotopia.org>
To: Detlev Casanova <detlev.casanova@collabora.com>,
Ezequiel Garcia <ezequiel@vanguardiasur.com.ar>,
Mauro Carvalho Chehab <mchehab@kernel.org>,
Heiko Stuebner <heiko@sntech.de>,
Nicolas Dufresne <nicolas.dufresne@collabora.com>,
Alex Bee <knaerzche@gmail.com>,
Hans Verkuil <hverkuil+cisco@kernel.org>,
Jonas Karlman <jonas@kwiboo.se>,
Sebastian Fricke <sebastian.fricke@collabora.com>,
linux-media@vger.kernel.org, linux-rockchip@lists.infradead.org,
linux-arm-kernel@lists.infradead.org,
linux-kernel@vger.kernel.org
Subject: [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling
Date: Fri, 20 Nov 2026 23:41:00 +0000 [thread overview]
Message-ID: <cover.1795217229.git.daniel@makrotopia.org> (raw)
This series fixes how the VDPU381 and VDPU383 HEVC back ends of the
rkvdec driver handle the EXT_SPS_ST_RPS and EXT_SPS_LT_RPS controls.
The defects were found while bringing up the VDPU346 of the RK3568,
which uses the VDPU381 back end, on top of Christian Hewitt's VDPU346
series [1], which has since been continued in [2].
Patch 1 looks up the controls after the request was applied instead of
before, which fixes a use-after-free.
Patches 2 and 3 are about jobs which cannot be decoded because the RPS
controls are missing. The VDPU383 back end now completes the request
when it refuses such a job, and the VDPU381 back end refuses the job
as well instead of starting the hardware.
Patch 4 refuses RPS controls with fewer elements than the SPS counts,
and patch 5 limits the size of predicted short-term sets. Both fix out
of bounds accesses which userspace can trigger.
Patch 6 fixes wrong decoding caused by a short-term RPS cache which
only looked at the first set, and patch 7 adds a missing allocation
check.
Patches 1 to 6 carry Cc: stable, like the EXT SPS RPS bounds merged in
v7.3 (f0b9d7e5be06, 796b5c6d4f16, 052c5ed5a1d9).
Tested on a NanoPi R5C (RK3568) with Linux 6.18 and the VDPU346 series
backported: GStreamer fluster JCT-VC-HEVC_V1 stays at 140/147, an FFmpeg
v4l2request build that sets the RPS controls confirms the reads and the
short-term set cache, and the VDPU381 refusal removes a watchdog reset
of the RK3568. The same VDPU381 back end was exercised on a Radxa ROCK
5B (RK3588): H.264, HEVC and VP9 decode through rkvdec with no kernel
message across the fluster runs (HEVC 146/147), a per-patch A/B sweep
and a 30-minute stability run, at the baseline throughput. The VDPU383
back end has no hardware here, the code-only patches have no reproducer,
and nothing was tested on an RK3576.
All of this was done with v1. v2 changes the order of the patches and
the commit messages, the code after the series is the same as in v1,
and the tests were not repeated.
An LLM coding agent did the analysis, and helped with writing the
patches. A second instance, given only the code, reviewed the series,
and has tested and validated it on real hardware independently.
Changes since v1:
- Move "media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets"
before the check of the control size (5/7 -> 3/7). A control which is
too short is handled like a missing one, and this way that means a
refused job in both back ends as soon as the check exists, instead
of a decode with a stale RPS table on the VDPU381 (Nicolas Dufresne)
- Rename "media: rkvdec: hevc: bound the RPS reads by the control size"
to "media: rkvdec: hevc: refuse RPS controls shorter than the SPS
counts" (3/7 -> 4/7)
- Rewrite all commit messages: say what is wrong and what the patch
achieves, without walking through the code (Nicolas Dufresne)
- State in patch 4 why the size cannot be checked when the control is
set (Nicolas Dufresne)
- Rebase on linux-next
- No change to the code after the series
v1: https://lore.kernel.org/r/cover.1794645268.git.daniel@makrotopia.org
[1] https://lore.kernel.org/r/20260110053727.2295260-1-christianshewitt@gmail.com
[2] https://lore.kernel.org/r/cover.1794654074.git.daniel@makrotopia.org
Daniel Golle (7):
media: rkvdec: hevc: apply request controls first
media: rkvdec: vdpu383: complete refused HEVC requests
media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets
media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts
media: rkvdec: hevc: bound predicted short-term RPS sets
media: rkvdec: hevc: cache every short-term RPS set
media: rkvdec: hevc: check the RPS scratch allocation
.../rockchip/rkvdec/rkvdec-hevc-common.c | 52 +++++++++++++------
.../rockchip/rkvdec/rkvdec-hevc-common.h | 11 +++-
.../rockchip/rkvdec/rkvdec-vdpu381-hevc.c | 24 +++++----
.../rockchip/rkvdec/rkvdec-vdpu383-hevc.c | 14 +++--
4 files changed, 67 insertions(+), 34 deletions(-)
base-commit: 909ef7a9a30fa139e69e555e3606680f5cca7726
--
2.56.1
next reply other threads:[~2026-11-20 23:41 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 23:41 Daniel Golle [this message]
2026-11-20 23:41 ` [PATCH v2 1/7] media: rkvdec: hevc: apply request controls first Daniel Golle
2026-11-20 23:41 ` [PATCH v2 2/7] media: rkvdec: vdpu383: complete refused HEVC requests Daniel Golle
2026-11-20 23:42 ` [PATCH v2 3/7] media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets Daniel Golle
2026-11-20 23:42 ` [PATCH v2 4/7] media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts Daniel Golle
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1795217229.git.daniel@makrotopia.org \
--to=daniel@makrotopia.org \
--cc=detlev.casanova@collabora.com \
--cc=ezequiel@vanguardiasur.com.ar \
--cc=heiko@sntech.de \
--cc=hverkuil+cisco@kernel.org \
--cc=jonas@kwiboo.se \
--cc=knaerzche@gmail.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=linux-rockchip@lists.infradead.org \
--cc=mchehab@kernel.org \
--cc=nicolas.dufresne@collabora.com \
--cc=sebastian.fricke@collabora.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).