* [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling
@ 2026-11-20 23:41 Daniel Golle
2026-11-20 23:41 ` [PATCH v2 1/7] media: rkvdec: hevc: apply request controls first Daniel Golle
` (6 more replies)
0 siblings, 7 replies; 8+ messages in thread
From: Daniel Golle @ 2026-11-20 23:41 UTC (permalink / raw)
To: Detlev Casanova, Ezequiel Garcia, Mauro Carvalho Chehab,
Heiko Stuebner, Nicolas Dufresne, Alex Bee, Hans Verkuil,
Jonas Karlman, Sebastian Fricke, linux-media, linux-rockchip,
linux-arm-kernel, linux-kernel
This series fixes how the VDPU381 and VDPU383 HEVC back ends of the
rkvdec driver handle the EXT_SPS_ST_RPS and EXT_SPS_LT_RPS controls.
The defects were found while bringing up the VDPU346 of the RK3568,
which uses the VDPU381 back end, on top of Christian Hewitt's VDPU346
series [1], which has since been continued in [2].
Patch 1 looks up the controls after the request was applied instead of
before, which fixes a use-after-free.
Patches 2 and 3 are about jobs which cannot be decoded because the RPS
controls are missing. The VDPU383 back end now completes the request
when it refuses such a job, and the VDPU381 back end refuses the job
as well instead of starting the hardware.
Patch 4 refuses RPS controls with fewer elements than the SPS counts,
and patch 5 limits the size of predicted short-term sets. Both fix out
of bounds accesses which userspace can trigger.
Patch 6 fixes wrong decoding caused by a short-term RPS cache which
only looked at the first set, and patch 7 adds a missing allocation
check.
Patches 1 to 6 carry Cc: stable, like the EXT SPS RPS bounds merged in
v7.3 (f0b9d7e5be06, 796b5c6d4f16, 052c5ed5a1d9).
Tested on a NanoPi R5C (RK3568) with Linux 6.18 and the VDPU346 series
backported: GStreamer fluster JCT-VC-HEVC_V1 stays at 140/147, an FFmpeg
v4l2request build that sets the RPS controls confirms the reads and the
short-term set cache, and the VDPU381 refusal removes a watchdog reset
of the RK3568. The same VDPU381 back end was exercised on a Radxa ROCK
5B (RK3588): H.264, HEVC and VP9 decode through rkvdec with no kernel
message across the fluster runs (HEVC 146/147), a per-patch A/B sweep
and a 30-minute stability run, at the baseline throughput. The VDPU383
back end has no hardware here, the code-only patches have no reproducer,
and nothing was tested on an RK3576.
All of this was done with v1. v2 changes the order of the patches and
the commit messages, the code after the series is the same as in v1,
and the tests were not repeated.
An LLM coding agent did the analysis, and helped with writing the
patches. A second instance, given only the code, reviewed the series,
and has tested and validated it on real hardware independently.
Changes since v1:
- Move "media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets"
before the check of the control size (5/7 -> 3/7). A control which is
too short is handled like a missing one, and this way that means a
refused job in both back ends as soon as the check exists, instead
of a decode with a stale RPS table on the VDPU381 (Nicolas Dufresne)
- Rename "media: rkvdec: hevc: bound the RPS reads by the control size"
to "media: rkvdec: hevc: refuse RPS controls shorter than the SPS
counts" (3/7 -> 4/7)
- Rewrite all commit messages: say what is wrong and what the patch
achieves, without walking through the code (Nicolas Dufresne)
- State in patch 4 why the size cannot be checked when the control is
set (Nicolas Dufresne)
- Rebase on linux-next
- No change to the code after the series
v1: https://lore.kernel.org/r/cover.1794645268.git.daniel@makrotopia.org
[1] https://lore.kernel.org/r/20260110053727.2295260-1-christianshewitt@gmail.com
[2] https://lore.kernel.org/r/cover.1794654074.git.daniel@makrotopia.org
Daniel Golle (7):
media: rkvdec: hevc: apply request controls first
media: rkvdec: vdpu383: complete refused HEVC requests
media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets
media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts
media: rkvdec: hevc: bound predicted short-term RPS sets
media: rkvdec: hevc: cache every short-term RPS set
media: rkvdec: hevc: check the RPS scratch allocation
.../rockchip/rkvdec/rkvdec-hevc-common.c | 52 +++++++++++++------
.../rockchip/rkvdec/rkvdec-hevc-common.h | 11 +++-
.../rockchip/rkvdec/rkvdec-vdpu381-hevc.c | 24 +++++----
.../rockchip/rkvdec/rkvdec-vdpu383-hevc.c | 14 +++--
4 files changed, 67 insertions(+), 34 deletions(-)
base-commit: 909ef7a9a30fa139e69e555e3606680f5cca7726
--
2.56.1
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH v2 1/7] media: rkvdec: hevc: apply request controls first
2026-11-20 23:41 [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling Daniel Golle
@ 2026-11-20 23:41 ` Daniel Golle
2026-11-20 23:41 ` [PATCH v2 2/7] media: rkvdec: vdpu383: complete refused HEVC requests Daniel Golle
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Daniel Golle @ 2026-11-20 23:41 UTC (permalink / raw)
To: Detlev Casanova, Ezequiel Garcia, Mauro Carvalho Chehab,
Heiko Stuebner, Nicolas Dufresne, Alex Bee, Hans Verkuil,
Jonas Karlman, Sebastian Fricke, linux-media, linux-rockchip,
linux-arm-kernel, linux-kernel
The HEVC run preamble looks up the control values before the request is
applied. Applying a request can reallocate the array of a dynamic array
control, so the pointers taken before may point to freed memory. The
slice count and the information whether the RPS controls were set are
also those of the previous request.
Userspace can trigger the reallocation by passing more elements of such
a control than in any earlier request.
Apply the request first and look up the controls afterwards, as the VP9
code does.
Fixes: 3595375c2301 ("media: rkvdec: Add HEVC backend")
Cc: stable@vger.kernel.org
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
Changes since v1:
- Rewrite the commit message
- No code change
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
index 2b8e04dd15727..c249e3f16f42c 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
@@ -429,6 +429,8 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx,
{
struct v4l2_ctrl *ctrl;
+ rkvdec_run_preamble(ctx, &run->base);
+
ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
V4L2_CID_STATELESS_HEVC_DECODE_PARAMS);
run->decode_params = ctrl ? ctrl->p_cur.p : NULL;
@@ -460,6 +462,4 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx,
} else {
run->ext_sps_lt_rps = NULL;
}
-
- rkvdec_run_preamble(ctx, &run->base);
}
--
2.56.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v2 2/7] media: rkvdec: vdpu383: complete refused HEVC requests
2026-11-20 23:41 [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling Daniel Golle
2026-11-20 23:41 ` [PATCH v2 1/7] media: rkvdec: hevc: apply request controls first Daniel Golle
@ 2026-11-20 23:41 ` Daniel Golle
2026-11-20 23:42 ` [PATCH v2 3/7] media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets Daniel Golle
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Daniel Golle @ 2026-11-20 23:41 UTC (permalink / raw)
To: Detlev Casanova, Ezequiel Garcia, Mauro Carvalho Chehab,
Heiko Stuebner, Nicolas Dufresne, Alex Bee, Hans Verkuil,
Jonas Karlman, Sebastian Fricke, linux-media, linux-rockchip,
linux-arm-kernel, linux-kernel
When the RPS controls are missing, the VDPU383 HEVC back end fails the
job after the request was applied, but without completing the request's
controls. The request then never completes: userspace waits for it
forever and MEDIA_REQUEST_IOC_REINIT fails with -EBUSY.
Complete the controls before returning the error, as the VP9 code does.
Fixes: e3b5b77e3689 ("media: rkvdec: Add HEVC support for the VDPU383 variant")
Cc: stable@vger.kernel.org
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
Changes since v1:
- Rewrite the commit message
- No code change
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c
index 3462d995d4cf0..e9d8b590f0d58 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c
@@ -553,6 +553,7 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
(!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
dev_err_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
+ rkvdec_run_postamble(ctx, &run.base);
return -EINVAL;
}
--
2.56.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v2 3/7] media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets
2026-11-20 23:41 [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling Daniel Golle
2026-11-20 23:41 ` [PATCH v2 1/7] media: rkvdec: hevc: apply request controls first Daniel Golle
2026-11-20 23:41 ` [PATCH v2 2/7] media: rkvdec: vdpu383: complete refused HEVC requests Daniel Golle
@ 2026-11-20 23:42 ` Daniel Golle
2026-11-20 23:42 ` [PATCH v2 4/7] media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts Daniel Golle
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Daniel Golle @ 2026-11-20 23:42 UTC (permalink / raw)
To: Detlev Casanova, Ezequiel Garcia, Mauro Carvalho Chehab,
Heiko Stuebner, Nicolas Dufresne, Alex Bee, Hans Verkuil,
Jonas Karlman, Sebastian Fricke, linux-media, linux-rockchip,
linux-arm-kernel, linux-kernel
When a stream uses reference picture sets and userspace never set the
EXT_SPS_ST_RPS or EXT_SPS_LT_RPS control, the VDPU381 HEVC back end
only logs a warning and starts the decoder anyway. The hardware then
uses an RPS table which does not belong to the stream and gives wrong
pictures at best. On the VDPU346 of the RK3568, which uses this back
end, such a decode ends in a failed IOMMU reset and the system stops
responding until the hardware watchdog resets it.
Fail the job with -EINVAL before the hardware is started, as the
VDPU383 back end does.
Fixes: c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")
Cc: stable@vger.kernel.org
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
Changes since v1:
- Move before the check of the control size, was 5/7
- Test ctx->has_sps_st_rps and ctx->has_sps_lt_rps as a result of the
new position, the next patch replaces them in both back ends
- Rewrite the commit message
.../rockchip/rkvdec/rkvdec-vdpu381-hevc.c | 19 ++++++++-----------
1 file changed, 8 insertions(+), 11 deletions(-)
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
index e1936e87f45be..7627282c15f0a 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
@@ -596,21 +596,18 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
rkvdec_hevc_run_preamble(ctx, &run);
- rkvdec_hevc_assemble_hw_scaling_list(ctx, &run, &tbl->scaling_list,
- &hevc_ctx->scaling_matrix_cache);
- assemble_hw_pps(ctx, &run);
-
- /*
- * On vdpu381, not setting the long and short term ref sets will just output wrong frames.
- * Let's just warn about it and let the decoder run anyway.
- */
if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
(!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
- dev_warn_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
- } else {
- rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+ dev_err_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
+ rkvdec_run_postamble(ctx, &run.base);
+ return -EINVAL;
}
+ rkvdec_hevc_assemble_hw_scaling_list(ctx, &run, &tbl->scaling_list,
+ &hevc_ctx->scaling_matrix_cache);
+ assemble_hw_pps(ctx, &run);
+ rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+
config_registers(ctx, &run);
rkvdec_run_postamble(ctx, &run.base);
--
2.56.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v2 4/7] media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts
2026-11-20 23:41 [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling Daniel Golle
` (2 preceding siblings ...)
2026-11-20 23:42 ` [PATCH v2 3/7] media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets Daniel Golle
@ 2026-11-20 23:42 ` Daniel Golle
2026-11-20 23:43 ` [PATCH v2 5/7] media: rkvdec: hevc: bound predicted short-term RPS sets Daniel Golle
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Daniel Golle @ 2026-11-20 23:42 UTC (permalink / raw)
To: Detlev Casanova, Ezequiel Garcia, Mauro Carvalho Chehab,
Heiko Stuebner, Nicolas Dufresne, Alex Bee, Hans Verkuil,
Jonas Karlman, Sebastian Fricke, linux-media, linux-rockchip,
linux-arm-kernel, linux-kernel
The SPS control tells how many short-term and long-term reference
picture sets a stream has. The sets themselves are passed in two
separate array controls, EXT_SPS_ST_RPS and EXT_SPS_LT_RPS. The driver
reads as many array elements as the SPS announces, without checking
that userspace passed that many. A larger count in the SPS makes it
read past the end of the array.
Compare the array size with the SPS count when a decode job is
prepared, and refuse the job if the array is too short. The controls
can be set independently, so this is the first point where both values
are known.
Fixes: c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")
Cc: stable@vger.kernel.org
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
Changes since v1:
- Move after the VDPU381 refusal, was 3/7, so that a control which is
too short fails the job in both back ends (Nicolas Dufresne)
- Change the subject, was "media: rkvdec: hevc: bound the RPS reads by
the control size"
- Rewrite the commit message, and say why the size cannot be checked
when the control is set (Nicolas Dufresne)
- The VDPU381 hunk now applies to the refusal instead of the warning,
no other code change
.../platform/rockchip/rkvdec/rkvdec-hevc-common.c | 12 ++++++------
.../platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c | 4 ++--
.../platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c | 4 ++--
3 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
index c249e3f16f42c..797a16517ed59 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
@@ -448,18 +448,18 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx,
V4L2_CID_STATELESS_HEVC_SCALING_MATRIX);
run->scaling_matrix = ctrl ? ctrl->p_cur.p : NULL;
+ run->ext_sps_st_rps = NULL;
if (ctx->has_sps_st_rps) {
ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
V4L2_CID_STATELESS_HEVC_EXT_SPS_ST_RPS);
- run->ext_sps_st_rps = ctrl ? ctrl->p_cur.p : NULL;
- } else {
- run->ext_sps_st_rps = NULL;
+ if (ctrl && ctrl->elems >= run->sps->num_short_term_ref_pic_sets)
+ run->ext_sps_st_rps = ctrl->p_cur.p;
}
+ run->ext_sps_lt_rps = NULL;
if (ctx->has_sps_lt_rps) {
ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
V4L2_CID_STATELESS_HEVC_EXT_SPS_LT_RPS);
- run->ext_sps_lt_rps = ctrl ? ctrl->p_cur.p : NULL;
- } else {
- run->ext_sps_lt_rps = NULL;
+ if (ctrl && ctrl->elems >= run->sps->num_long_term_ref_pics_sps)
+ run->ext_sps_lt_rps = ctrl->p_cur.p;
}
}
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
index 7627282c15f0a..db836536e601b 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
@@ -596,8 +596,8 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
rkvdec_hevc_run_preamble(ctx, &run);
- if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
- (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
+ if ((!run.ext_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
+ (!run.ext_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
dev_err_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
rkvdec_run_postamble(ctx, &run.base);
return -EINVAL;
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c
index e9d8b590f0d58..c409d21c08598 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c
@@ -550,8 +550,8 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
* To be on the safe side for this new v4l2 control, write an error in the log and mark
* the buffer as failed by returning an error here.
*/
- if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
- (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
+ if ((!run.ext_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
+ (!run.ext_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
dev_err_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
rkvdec_run_postamble(ctx, &run.base);
return -EINVAL;
--
2.56.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v2 5/7] media: rkvdec: hevc: bound predicted short-term RPS sets
2026-11-20 23:41 [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling Daniel Golle
` (3 preceding siblings ...)
2026-11-20 23:42 ` [PATCH v2 4/7] media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts Daniel Golle
@ 2026-11-20 23:43 ` Daniel Golle
2026-11-20 23:43 ` [PATCH v2 6/7] media: rkvdec: hevc: cache every short-term RPS set Daniel Golle
2026-11-20 23:44 ` [PATCH v2 7/7] media: rkvdec: hevc: check the RPS scratch allocation Daniel Golle
6 siblings, 0 replies; 8+ messages in thread
From: Daniel Golle @ 2026-11-20 23:43 UTC (permalink / raw)
To: Detlev Casanova, Ezequiel Garcia, Mauro Carvalho Chehab,
Heiko Stuebner, Nicolas Dufresne, Alex Bee, Hans Verkuil,
Jonas Karlman, Sebastian Fricke, linux-media, linux-rockchip,
linux-arm-kernel, linux-kernel
A short-term reference picture set can be predicted from an earlier
set, and can then hold one entry more than that set. The control
validation allows 16 entries per set, which is also the size of the
arrays the driver calculates the sets in. A set predicted from a
16-entry set overflows them, on the stack and on the heap.
ITU-T H.265 7.4.8 limits a set to 15 entries, and the hardware table
has room for 15 entries per set as well. Skip the prediction when the
reference set has more than 15 entries, as is already done for an
invalid reference index.
Fixes: c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")
Cc: stable@vger.kernel.org
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
Changes since v1:
- Was 4/7
- Rewrite the commit message
- No code change
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
index 797a16517ed59..3b281c16a75ed 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
@@ -223,6 +223,8 @@ static void st_ref_pic_set_prediction(struct rkvdec_hevc_run *run, int idx,
(rps_data->abs_delta_rps_minus1 + 1); /* 7-60 */
ref_rps = &calculated_rps_st_sets[ref_rps_idx];
+ if (ref_rps->num_delta_pocs >= ARRAY_SIZE(use_delta_flag))
+ return;
for (j = 0; j <= ref_rps->num_delta_pocs; j++) {
used_by_curr_pic_flag[j] = !!(rps_data->used_by_curr_pic & (1 << j));
--
2.56.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v2 6/7] media: rkvdec: hevc: cache every short-term RPS set
2026-11-20 23:41 [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling Daniel Golle
` (4 preceding siblings ...)
2026-11-20 23:43 ` [PATCH v2 5/7] media: rkvdec: hevc: bound predicted short-term RPS sets Daniel Golle
@ 2026-11-20 23:43 ` Daniel Golle
2026-11-20 23:44 ` [PATCH v2 7/7] media: rkvdec: hevc: check the RPS scratch allocation Daniel Golle
6 siblings, 0 replies; 8+ messages in thread
From: Daniel Golle @ 2026-11-20 23:43 UTC (permalink / raw)
To: Detlev Casanova, Ezequiel Garcia, Mauro Carvalho Chehab,
Heiko Stuebner, Nicolas Dufresne, Alex Bee, Hans Verkuil,
Jonas Karlman, Sebastian Fricke, linux-media, linux-rockchip,
linux-arm-kernel, linux-kernel
The driver rebuilds the short-term RPS table of the hardware only when
the EXT_SPS_ST_RPS control changed, but it compares just the first set
with a cached copy. A change in a later set or in the number of sets
goes unnoticed, and the hardware keeps the old table. As the cache
starts zeroed, a stream whose first set is all zero never gets a table
at all. Inter pictures are then decoded with wrong references, as seen
with JCT-VC DELTAQP_A_BRCM_4 on an RK3568.
Cache all sets together with their number. The control framework limits
the number of sets to 64, which is the size of the cache.
Fixes: c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")
Cc: stable@vger.kernel.org
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
Changes since v1:
- Rewrite the commit message
- No code change
.../platform/rockchip/rkvdec/rkvdec-hevc-common.c | 12 ++++++++----
.../platform/rockchip/rkvdec/rkvdec-hevc-common.h | 9 ++++++++-
.../platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c | 2 +-
.../platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c | 2 +-
4 files changed, 18 insertions(+), 7 deletions(-)
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
index 3b281c16a75ed..e3449ab0d415b 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
@@ -330,14 +330,17 @@ static void st_ref_pic_set_calculate(struct rkvdec_hevc_run *run, int idx,
}
static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
- struct v4l2_ctrl_hevc_ext_sps_st_rps *cache)
+ struct rkvdec_hevc_st_rps_cache *cache)
{
+ u8 num_sets = run->sps->num_short_term_ref_pic_sets;
+ size_t size;
int idx;
if (!run->ext_sps_st_rps)
return;
- if (!memcmp(cache, run->ext_sps_st_rps, sizeof(struct v4l2_ctrl_hevc_ext_sps_st_rps)))
+ size = num_sets * sizeof(*cache->sets);
+ if (cache->num_sets == num_sets && !memcmp(cache->sets, run->ext_sps_st_rps, size))
return;
struct calculated_rps_st_set *calculated_rps_st_sets =
@@ -357,11 +360,12 @@ static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rk
kfree(calculated_rps_st_sets);
- memcpy(cache, run->ext_sps_st_rps, sizeof(struct v4l2_ctrl_hevc_ext_sps_st_rps));
+ memcpy(cache->sets, run->ext_sps_st_rps, size);
+ cache->num_sets = num_sets;
}
void rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
- struct v4l2_ctrl_hevc_ext_sps_st_rps *st_cache)
+ struct rkvdec_hevc_st_rps_cache *st_cache)
{
rkvdec_hevc_prepare_hw_st_rps(run, rps, st_cache);
rkvdec_hevc_assemble_hw_lt_rps(run, rps);
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h
index 2a9b7719ab2d5..f6a3dec6ff4ad 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h
@@ -35,10 +35,17 @@
#define RKVDEC_RPS_HEVC_SIZE ALIGN(1032 + 64 * 384, 128)
+#define RKVDEC_HEVC_MAX_SHORT_TERM_REF_PIC_SETS 64
+
struct rkvdec_rps {
u32 info[RKVDEC_RPS_HEVC_SIZE / 8 / 4];
} __packed;
+struct rkvdec_hevc_st_rps_cache {
+ struct v4l2_ctrl_hevc_ext_sps_st_rps sets[RKVDEC_HEVC_MAX_SHORT_TERM_REF_PIC_SETS];
+ u8 num_sets;
+};
+
struct rkvdec_hevc_run {
struct rkvdec_run base;
const struct v4l2_ctrl_hevc_slice_params *slices_params;
@@ -65,7 +72,7 @@ void compute_tiles_non_uniform(struct rkvdec_hevc_run *run, u16 log2_min_cb_size
u16 width, u16 height, s32 pic_in_cts_width,
s32 pic_in_cts_height, u16 *column_width, u16 *row_height);
void rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
- struct v4l2_ctrl_hevc_ext_sps_st_rps *st_cache);
+ struct rkvdec_hevc_st_rps_cache *st_cache);
void rkvdec_hevc_assemble_hw_scaling_list(struct rkvdec_ctx *ctx,
struct rkvdec_hevc_run *run,
struct scaling_factor *scaling_factor,
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
index db836536e601b..8388421a68a48 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
@@ -117,7 +117,7 @@ struct rkvdec_hevc_priv_tbl {
struct rkvdec_hevc_ctx {
struct rkvdec_aux_buf priv_tbl;
struct v4l2_ctrl_hevc_scaling_matrix scaling_matrix_cache;
- struct v4l2_ctrl_hevc_ext_sps_st_rps st_cache;
+ struct rkvdec_hevc_st_rps_cache st_cache;
struct rkvdec_vdpu381_regs_hevc regs;
};
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c
index c409d21c08598..47f236295f1e5 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c
@@ -124,7 +124,7 @@ struct rkvdec_hevc_priv_tbl {
struct rkvdec_hevc_ctx {
struct rkvdec_aux_buf priv_tbl;
struct v4l2_ctrl_hevc_scaling_matrix scaling_matrix_cache;
- struct v4l2_ctrl_hevc_ext_sps_st_rps st_cache;
+ struct rkvdec_hevc_st_rps_cache st_cache;
struct vdpu383_regs_h26x regs;
};
--
2.56.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v2 7/7] media: rkvdec: hevc: check the RPS scratch allocation
2026-11-20 23:41 [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling Daniel Golle
` (5 preceding siblings ...)
2026-11-20 23:43 ` [PATCH v2 6/7] media: rkvdec: hevc: cache every short-term RPS set Daniel Golle
@ 2026-11-20 23:44 ` Daniel Golle
6 siblings, 0 replies; 8+ messages in thread
From: Daniel Golle @ 2026-11-20 23:44 UTC (permalink / raw)
To: Detlev Casanova, Ezequiel Garcia, Mauro Carvalho Chehab,
Heiko Stuebner, Nicolas Dufresne, Alex Bee, Hans Verkuil,
Jonas Karlman, Sebastian Fricke, linux-media, linux-rockchip,
linux-arm-kernel, linux-kernel
The buffer for the calculated short-term sets is used without checking
that its allocation succeeded. Return -ENOMEM and fail the decode job
in both back ends.
Fixes: c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
Changes since v1:
- Rewrite the commit message
- No code change
.../rockchip/rkvdec/rkvdec-hevc-common.c | 26 ++++++++++++++-----
.../rockchip/rkvdec/rkvdec-hevc-common.h | 4 +--
.../rockchip/rkvdec/rkvdec-vdpu381-hevc.c | 7 ++++-
.../rockchip/rkvdec/rkvdec-vdpu383-hevc.c | 7 ++++-
4 files changed, 33 insertions(+), 11 deletions(-)
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
index e3449ab0d415b..80df05bacaeba 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
@@ -329,24 +329,27 @@ static void st_ref_pic_set_calculate(struct rkvdec_hevc_run *run, int idx,
st_rps->num_delta_pocs = st_rps->num_positive_pics + st_rps->num_negative_pics;
}
-static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
- struct rkvdec_hevc_st_rps_cache *cache)
+static int rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
+ struct rkvdec_hevc_st_rps_cache *cache)
{
u8 num_sets = run->sps->num_short_term_ref_pic_sets;
size_t size;
int idx;
if (!run->ext_sps_st_rps)
- return;
+ return 0;
size = num_sets * sizeof(*cache->sets);
if (cache->num_sets == num_sets && !memcmp(cache->sets, run->ext_sps_st_rps, size))
- return;
+ return 0;
struct calculated_rps_st_set *calculated_rps_st_sets =
kzalloc(sizeof(struct calculated_rps_st_set) *
run->sps->num_short_term_ref_pic_sets, GFP_KERNEL);
+ if (!calculated_rps_st_sets)
+ return -ENOMEM;
+
for (idx = 0; idx < run->sps->num_short_term_ref_pic_sets; idx++) {
const struct v4l2_ctrl_hevc_ext_sps_st_rps *rps_data = &run->ext_sps_st_rps[idx];
@@ -362,13 +365,22 @@ static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rk
memcpy(cache->sets, run->ext_sps_st_rps, size);
cache->num_sets = num_sets;
+
+ return 0;
}
-void rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
- struct rkvdec_hevc_st_rps_cache *st_cache)
+int rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
+ struct rkvdec_hevc_st_rps_cache *st_cache)
{
- rkvdec_hevc_prepare_hw_st_rps(run, rps, st_cache);
+ int ret;
+
+ ret = rkvdec_hevc_prepare_hw_st_rps(run, rps, st_cache);
+ if (ret)
+ return ret;
+
rkvdec_hevc_assemble_hw_lt_rps(run, rps);
+
+ return 0;
}
struct vb2_buffer *
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h
index f6a3dec6ff4ad..32f192743c8c8 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h
@@ -71,8 +71,8 @@ void compute_tiles_uniform(struct rkvdec_hevc_run *run, u16 log2_min_cb_size,
void compute_tiles_non_uniform(struct rkvdec_hevc_run *run, u16 log2_min_cb_size,
u16 width, u16 height, s32 pic_in_cts_width,
s32 pic_in_cts_height, u16 *column_width, u16 *row_height);
-void rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
- struct rkvdec_hevc_st_rps_cache *st_cache);
+int rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
+ struct rkvdec_hevc_st_rps_cache *st_cache);
void rkvdec_hevc_assemble_hw_scaling_list(struct rkvdec_ctx *ctx,
struct rkvdec_hevc_run *run,
struct scaling_factor *scaling_factor,
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
index 8388421a68a48..2a239114759a2 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
@@ -593,6 +593,7 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
struct rkvdec_hevc_run run;
struct rkvdec_hevc_ctx *hevc_ctx = ctx->priv;
struct rkvdec_hevc_priv_tbl *tbl = hevc_ctx->priv_tbl.cpu;
+ int ret;
rkvdec_hevc_run_preamble(ctx, &run);
@@ -606,7 +607,11 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
rkvdec_hevc_assemble_hw_scaling_list(ctx, &run, &tbl->scaling_list,
&hevc_ctx->scaling_matrix_cache);
assemble_hw_pps(ctx, &run);
- rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+ ret = rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+ if (ret) {
+ rkvdec_run_postamble(ctx, &run.base);
+ return ret;
+ }
config_registers(ctx, &run);
diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c
index 47f236295f1e5..52554c75f47f4 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c
@@ -542,6 +542,7 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
struct rkvdec_hevc_ctx *hevc_ctx = ctx->priv;
struct rkvdec_hevc_priv_tbl *tbl = hevc_ctx->priv_tbl.cpu;
u32 timeout_threshold;
+ int ret;
rkvdec_hevc_run_preamble(ctx, &run);
@@ -560,7 +561,11 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
rkvdec_hevc_assemble_hw_scaling_list(ctx, &run, &tbl->scaling_list,
&hevc_ctx->scaling_matrix_cache);
assemble_hw_pps(ctx, &run);
- rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+ ret = rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+ if (ret) {
+ rkvdec_run_postamble(ctx, &run.base);
+ return ret;
+ }
config_registers(ctx, &run);
--
2.56.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
end of thread, other threads:[~2026-11-20 23:44 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 23:41 [PATCH v2 0/7] media: rkvdec: Fix the HEVC reference picture set handling Daniel Golle
2026-11-20 23:41 ` [PATCH v2 1/7] media: rkvdec: hevc: apply request controls first Daniel Golle
2026-11-20 23:41 ` [PATCH v2 2/7] media: rkvdec: vdpu383: complete refused HEVC requests Daniel Golle
2026-11-20 23:42 ` [PATCH v2 3/7] media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets Daniel Golle
2026-11-20 23:42 ` [PATCH v2 4/7] media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts Daniel Golle
2026-11-20 23:43 ` [PATCH v2 5/7] media: rkvdec: hevc: bound predicted short-term RPS sets Daniel Golle
2026-11-20 23:43 ` [PATCH v2 6/7] media: rkvdec: hevc: cache every short-term RPS set Daniel Golle
2026-11-20 23:44 ` [PATCH v2 7/7] media: rkvdec: hevc: check the RPS scratch allocation Daniel Golle
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox