All of lore.kernel.org
 help / color / mirror / Atom feed
From: Daniel Golle <daniel@makrotopia.org>
To: Detlev Casanova <detlev.casanova@collabora.com>,
	Ezequiel Garcia <ezequiel@vanguardiasur.com.ar>,
	Mauro Carvalho Chehab <mchehab@kernel.org>,
	Heiko Stuebner <heiko@sntech.de>,
	Nicolas Dufresne <nicolas.dufresne@collabora.com>,
	Alex Bee <knaerzche@gmail.com>,
	Hans Verkuil <hverkuil+cisco@kernel.org>,
	Jonas Karlman <jonas@kwiboo.se>,
	Sebastian Fricke <sebastian.fricke@collabora.com>,
	linux-media@vger.kernel.org, linux-rockchip@lists.infradead.org,
	linux-arm-kernel@lists.infradead.org,
	linux-kernel@vger.kernel.org
Subject: [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling
Date: Fri, 20 Nov 2026 23:41:00 +0000	[thread overview]
Message-ID: <cover.1795217229.git.daniel@makrotopia.org> (raw)

This series fixes how the VDPU381 and VDPU383 HEVC back ends of the
rkvdec driver handle the EXT_SPS_ST_RPS and EXT_SPS_LT_RPS controls.
The defects were found while bringing up the VDPU346 of the RK3568,
which uses the VDPU381 back end, on top of Christian Hewitt's VDPU346
series [1], which has since been continued in [2].

Patch 1 looks up the controls after the request was applied instead of
before, which fixes a use-after-free.

Patches 2 and 3 are about jobs which cannot be decoded because the RPS
controls are missing. The VDPU383 back end now completes the request
when it refuses such a job, and the VDPU381 back end refuses the job
as well instead of starting the hardware.

Patch 4 refuses RPS controls with fewer elements than the SPS counts,
and patch 5 limits the size of predicted short-term sets. Both fix out
of bounds accesses which userspace can trigger.

Patch 6 fixes wrong decoding caused by a short-term RPS cache which
only looked at the first set, and patch 7 adds a missing allocation
check.

Patches 1 to 6 carry Cc: stable, like the EXT SPS RPS bounds merged in
v7.3 (f0b9d7e5be06, 796b5c6d4f16, 052c5ed5a1d9).

Tested on a NanoPi R5C (RK3568) with Linux 6.18 and the VDPU346 series
backported: GStreamer fluster JCT-VC-HEVC_V1 stays at 140/147, an FFmpeg
v4l2request build that sets the RPS controls confirms the reads and the
short-term set cache, and the VDPU381 refusal removes a watchdog reset
of the RK3568. The same VDPU381 back end was exercised on a Radxa ROCK
5B (RK3588): H.264, HEVC and VP9 decode through rkvdec with no kernel
message across the fluster runs (HEVC 146/147), a per-patch A/B sweep
and a 30-minute stability run, at the baseline throughput. The VDPU383
back end has no hardware here, the code-only patches have no reproducer,
and nothing was tested on an RK3576.

All of this was done with v1. v2 changes the order of the patches and
the commit messages, the code after the series is the same as in v1,
and the tests were not repeated.

An LLM coding agent did the analysis, and helped with writing the
patches. A second instance, given only the code, reviewed the series,
and has tested and validated it on real hardware independently.

Changes since v1:
- Move "media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets"
  before the check of the control size (5/7 -> 3/7). A control which is
  too short is handled like a missing one, and this way that means a
  refused job in both back ends as soon as the check exists, instead
  of a decode with a stale RPS table on the VDPU381 (Nicolas Dufresne)
- Rename "media: rkvdec: hevc: bound the RPS reads by the control size"
  to "media: rkvdec: hevc: refuse RPS controls shorter than the SPS
  counts" (3/7 -> 4/7)
- Rewrite all commit messages: say what is wrong and what the patch
  achieves, without walking through the code (Nicolas Dufresne)
- State in patch 4 why the size cannot be checked when the control is
  set (Nicolas Dufresne)
- Rebase on linux-next
- No change to the code after the series

v1: https://lore.kernel.org/r/cover.1794645268.git.daniel@makrotopia.org

[1] https://lore.kernel.org/r/20260110053727.2295260-1-christianshewitt@gmail.com
[2] https://lore.kernel.org/r/cover.1794654074.git.daniel@makrotopia.org

Daniel Golle (7):
  media: rkvdec: hevc: apply request controls first
  media: rkvdec: vdpu383: complete refused HEVC requests
  media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets
  media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts
  media: rkvdec: hevc: bound predicted short-term RPS sets
  media: rkvdec: hevc: cache every short-term RPS set
  media: rkvdec: hevc: check the RPS scratch allocation

 .../rockchip/rkvdec/rkvdec-hevc-common.c      | 52 +++++++++++++------
 .../rockchip/rkvdec/rkvdec-hevc-common.h      | 11 +++-
 .../rockchip/rkvdec/rkvdec-vdpu381-hevc.c     | 24 +++++----
 .../rockchip/rkvdec/rkvdec-vdpu383-hevc.c     | 14 +++--
 4 files changed, 67 insertions(+), 34 deletions(-)


base-commit: 909ef7a9a30fa139e69e555e3606680f5cca7726
-- 
2.56.1

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

WARNING: multiple messages have this Message-ID (diff)
From: Daniel Golle <daniel@makrotopia.org>
To: Detlev Casanova <detlev.casanova@collabora.com>,
	Ezequiel Garcia <ezequiel@vanguardiasur.com.ar>,
	Mauro Carvalho Chehab <mchehab@kernel.org>,
	Heiko Stuebner <heiko@sntech.de>,
	Nicolas Dufresne <nicolas.dufresne@collabora.com>,
	Alex Bee <knaerzche@gmail.com>,
	Hans Verkuil <hverkuil+cisco@kernel.org>,
	Jonas Karlman <jonas@kwiboo.se>,
	Sebastian Fricke <sebastian.fricke@collabora.com>,
	linux-media@vger.kernel.org, linux-rockchip@lists.infradead.org,
	linux-arm-kernel@lists.infradead.org,
	linux-kernel@vger.kernel.org
Subject: [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling
Date: Fri, 20 Nov 2026 23:41:00 +0000	[thread overview]
Message-ID: <cover.1795217229.git.daniel@makrotopia.org> (raw)

This series fixes how the VDPU381 and VDPU383 HEVC back ends of the
rkvdec driver handle the EXT_SPS_ST_RPS and EXT_SPS_LT_RPS controls.
The defects were found while bringing up the VDPU346 of the RK3568,
which uses the VDPU381 back end, on top of Christian Hewitt's VDPU346
series [1], which has since been continued in [2].

Patch 1 looks up the controls after the request was applied instead of
before, which fixes a use-after-free.

Patches 2 and 3 are about jobs which cannot be decoded because the RPS
controls are missing. The VDPU383 back end now completes the request
when it refuses such a job, and the VDPU381 back end refuses the job
as well instead of starting the hardware.

Patch 4 refuses RPS controls with fewer elements than the SPS counts,
and patch 5 limits the size of predicted short-term sets. Both fix out
of bounds accesses which userspace can trigger.

Patch 6 fixes wrong decoding caused by a short-term RPS cache which
only looked at the first set, and patch 7 adds a missing allocation
check.

Patches 1 to 6 carry Cc: stable, like the EXT SPS RPS bounds merged in
v7.3 (f0b9d7e5be06, 796b5c6d4f16, 052c5ed5a1d9).

Tested on a NanoPi R5C (RK3568) with Linux 6.18 and the VDPU346 series
backported: GStreamer fluster JCT-VC-HEVC_V1 stays at 140/147, an FFmpeg
v4l2request build that sets the RPS controls confirms the reads and the
short-term set cache, and the VDPU381 refusal removes a watchdog reset
of the RK3568. The same VDPU381 back end was exercised on a Radxa ROCK
5B (RK3588): H.264, HEVC and VP9 decode through rkvdec with no kernel
message across the fluster runs (HEVC 146/147), a per-patch A/B sweep
and a 30-minute stability run, at the baseline throughput. The VDPU383
back end has no hardware here, the code-only patches have no reproducer,
and nothing was tested on an RK3576.

All of this was done with v1. v2 changes the order of the patches and
the commit messages, the code after the series is the same as in v1,
and the tests were not repeated.

An LLM coding agent did the analysis, and helped with writing the
patches. A second instance, given only the code, reviewed the series,
and has tested and validated it on real hardware independently.

Changes since v1:
- Move "media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets"
  before the check of the control size (5/7 -> 3/7). A control which is
  too short is handled like a missing one, and this way that means a
  refused job in both back ends as soon as the check exists, instead
  of a decode with a stale RPS table on the VDPU381 (Nicolas Dufresne)
- Rename "media: rkvdec: hevc: bound the RPS reads by the control size"
  to "media: rkvdec: hevc: refuse RPS controls shorter than the SPS
  counts" (3/7 -> 4/7)
- Rewrite all commit messages: say what is wrong and what the patch
  achieves, without walking through the code (Nicolas Dufresne)
- State in patch 4 why the size cannot be checked when the control is
  set (Nicolas Dufresne)
- Rebase on linux-next
- No change to the code after the series

v1: https://lore.kernel.org/r/cover.1794645268.git.daniel@makrotopia.org

[1] https://lore.kernel.org/r/20260110053727.2295260-1-christianshewitt@gmail.com
[2] https://lore.kernel.org/r/cover.1794654074.git.daniel@makrotopia.org

Daniel Golle (7):
  media: rkvdec: hevc: apply request controls first
  media: rkvdec: vdpu383: complete refused HEVC requests
  media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets
  media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts
  media: rkvdec: hevc: bound predicted short-term RPS sets
  media: rkvdec: hevc: cache every short-term RPS set
  media: rkvdec: hevc: check the RPS scratch allocation

 .../rockchip/rkvdec/rkvdec-hevc-common.c      | 52 +++++++++++++------
 .../rockchip/rkvdec/rkvdec-hevc-common.h      | 11 +++-
 .../rockchip/rkvdec/rkvdec-vdpu381-hevc.c     | 24 +++++----
 .../rockchip/rkvdec/rkvdec-vdpu383-hevc.c     | 14 +++--
 4 files changed, 67 insertions(+), 34 deletions(-)


base-commit: 909ef7a9a30fa139e69e555e3606680f5cca7726
-- 
2.56.1

             reply	other threads:[~2026-11-20 23:41 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20 23:41 Daniel Golle [this message]
2026-11-20 23:41 ` [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling Daniel Golle
2026-11-20 23:41 ` [PATCH v2 1/7] media: rkvdec: hevc: apply request controls first Daniel Golle
2026-11-20 23:41   ` Daniel Golle
2026-11-20 23:41 ` [PATCH v2 2/7] media: rkvdec: vdpu383: complete refused HEVC requests Daniel Golle
2026-11-20 23:41   ` Daniel Golle
2026-11-20 23:42 ` [PATCH v2 3/7] media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets Daniel Golle
2026-11-20 23:42   ` Daniel Golle
2026-11-20 23:42 ` [PATCH v2 4/7] media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts Daniel Golle
2026-11-20 23:42   ` Daniel Golle
2026-11-20 23:43 ` [PATCH v2 5/7] media: rkvdec: hevc: bound predicted short-term RPS sets Daniel Golle
2026-11-20 23:43   ` Daniel Golle
2026-11-20 23:43 ` [PATCH v2 6/7] media: rkvdec: hevc: cache every short-term RPS set Daniel Golle
2026-11-20 23:43   ` Daniel Golle
2026-11-20 23:44 ` [PATCH v2 7/7] media: rkvdec: hevc: check the RPS scratch allocation Daniel Golle
2026-11-20 23:44   ` Daniel Golle

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1795217229.git.daniel@makrotopia.org \
    --to=daniel@makrotopia.org \
    --cc=detlev.casanova@collabora.com \
    --cc=ezequiel@vanguardiasur.com.ar \
    --cc=heiko@sntech.de \
    --cc=hverkuil+cisco@kernel.org \
    --cc=jonas@kwiboo.se \
    --cc=knaerzche@gmail.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-media@vger.kernel.org \
    --cc=linux-rockchip@lists.infradead.org \
    --cc=mchehab@kernel.org \
    --cc=nicolas.dufresne@collabora.com \
    --cc=sebastian.fricke@collabora.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.