[PATCH] media: rkvdec: hevc: cap EXT SPS RPS control counts before descriptor assembly

[PATCH] media: rkvdec: hevc: cap EXT SPS RPS control counts before descriptor assembly

[Michael Bommarito]

V4L2_CID_STATELESS_HEVC_EXT_SPS_ST_RPS and
V4L2_CID_STATELESS_HEVC_EXT_SPS_LT_RPS are registered as dynamic-size
controls with a per-control element cap of 65. The V4L2 control core
enforces only the payload-element cap. It does not bound the spec-derived
count fields that the rkvdec HEVC helper later uses to walk fixed hardware
descriptor tables and temporary helper arrays:

  - struct rkvdec_rps::refs[32]
  - struct rkvdec_rps::short_term_ref_sets[64]
  - struct calculated_rps_st_set::delta_poc_s0[16] / delta_poc_s1[16]

A userspace V4L2 client that can open the Rockchip RKVDEC m2m decoder node
may submit SPS/RPS controls whose counts exceed those capacities or whose
prediction reference index underflows. rkvdec_hevc_assemble_hw_rps() then
walks past the descriptor table or temporary-array bounds.

KASAN under a small KUnit harness wrapping the real helper reports
slab-out-of-bounds in all of:

  - num_short_term_ref_pic_sets = 65 (write past short_term_ref_sets[64])
  - num_long_term_ref_pics_sps = 33  (write past refs[32], intra-struct)
  - ext_sps_st_rps[i].num_negative_pics or num_positive_pics > 16
    (write past delta_poc_s0[16] inside calculated_rps_st_set)
  - INTER_REF_PIC_SET_PRED with delta_idx_minus1 + 1 > i
    (u8 ref_rps_idx underflow then OOB read on calculated_rps_st_sets)

Validate the SPS/RPS counts before calling the assembly helpers. The cap
values match both the HEVC spec ranges (num_short_term_ref_pic_sets <= 64,
num_long_term_ref_pics_sps <= 32) and the fixed driver descriptor and
helper-array capacities. Reject controls whose counts exceed those, and
reject prediction entries whose reference index would underflow.

Fixes: c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")
Signed-off-by: Michael Bommarito <michael.bommarito@gmail.com>
Assisted-by: Claude:claude-opus-4-7
---

I don't have a RK3588 / RK3576 board to confirm this through a real
/dev/videoN request path yet, but was convinced enough by:

  1. Static reach: registration of EXT_SPS_ST_RPS / EXT_SPS_LT_RPS with
     .dims = { 65 } at drivers/media/platform/rockchip/rkvdec/rkvdec.c
     :239-287, the SPS-count-driven loops in rkvdec-hevc-common.c
     :213-225 and :228-251, and v4l2-ctrls-core.c :1213-1277, which
     validates only EXT RPS flags and not the spec-derived count fields.

  2. A KUnit harness (separate, not in this patch) that allocates one
     struct rkvdec_rps + a single calculated_rps_st_sets element via
     kunit_kzalloc / kzalloc and calls the real
     rkvdec_hevc_assemble_hw_rps() helper. Under UML + KASAN_GENERIC with
     the kasan_multi_shot boot param, on a stock tree it produces these
     reports:

       BUG: KASAN: slab-out-of-bounds in
         rkvdec_hevc_assemble_hw_rps+0xb0c/0x1080
         (num_short_term_ref_pic_sets = 65, write of size 36 0 bytes past
         the rps allocation)

       BUG: KASAN: slab-out-of-bounds in
         rkvdec_hevc_assemble_hw_rps (num_negative_pics = 64, write past
         the single-element kzalloc'd calculated_rps_st_sets buffer)

       BUG: KASAN: slab-use-after-free / slab-out-of-bounds reads via
         u8 ref_rps_idx underflow at calculated_rps_st_sets[255]
         (INTER_REF_PIC_SET_PRED with delta_idx_minus1 = 0, idx = 0)

     The num_long_term_ref_pics_sps = 33 case is invisible to KASAN
     (the OOB write lands inside struct rkvdec_rps) but corrupts
     short_term_ref_sets[0]; the harness asserts that case explicitly.

  3. Same harness on the patched tree: all five cases (four adversarial
     plus a legitimate-limit regression with ST=64, LT=32, num_neg=1)
     pass clean, no KASAN reports.

If hardware-side validation actually does reject these counts before
rkvdec_hevc_assemble_hw_rps() runs and this patch is unnecessary, please
say so and I will withdraw it. If it is reachable, I will follow up with
a runtime hardware splat once the Orange Pi board I bought arrives.

Let me know if you want a patch set with the KUnit harnesses too.

checkpatch.pl: 0 errors / 0 warnings.

 .../rockchip/rkvdec/rkvdec-hevc-common.c      | 49 +++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
index 3119f3bc9f98..895fb16bc572 100644
--- a/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
+++ b/drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
@@ -408,9 +408,58 @@ static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rk
 	memcpy(cache, run->ext_sps_st_rps, sizeof(struct v4l2_ctrl_hevc_ext_sps_st_rps));
 }
 
+/*
+ * V4L2 caps the EXT_SPS RPS payload length but not the SPS-derived counts
+ * that the helpers walk. Caps match the HEVC spec ranges.
+ */
+#define RKVDEC_HEVC_MAX_SHORT_TERM_REF_PIC_SETS	64
+#define RKVDEC_HEVC_MAX_LONG_TERM_REF_PICS_SPS	32
+#define RKVDEC_HEVC_MAX_RPS_NEG_POS_PICS	16
+
+static int rkvdec_hevc_validate_rps_ctrls(struct rkvdec_hevc_run *run)
+{
+	const struct v4l2_ctrl_hevc_sps *sps = run->sps;
+
+	if (run->ext_sps_lt_rps &&
+	    sps->num_long_term_ref_pics_sps >
+	    RKVDEC_HEVC_MAX_LONG_TERM_REF_PICS_SPS)
+		return -EINVAL;
+
+	if (run->ext_sps_st_rps) {
+		unsigned int i;
+
+		if (sps->num_short_term_ref_pic_sets >
+		    RKVDEC_HEVC_MAX_SHORT_TERM_REF_PIC_SETS)
+			return -EINVAL;
+
+		for (i = 0; i < sps->num_short_term_ref_pic_sets; i++) {
+			const struct v4l2_ctrl_hevc_ext_sps_st_rps *r =
+				&run->ext_sps_st_rps[i];
+
+			if (r->num_negative_pics >
+			    RKVDEC_HEVC_MAX_RPS_NEG_POS_PICS ||
+			    r->num_positive_pics >
+			    RKVDEC_HEVC_MAX_RPS_NEG_POS_PICS)
+				return -EINVAL;
+
+			if ((r->flags &
+			     V4L2_HEVC_EXT_SPS_ST_RPS_FLAG_INTER_REF_PIC_SET_PRED) &&
+			    (unsigned int)r->delta_idx_minus1 + 1 > i)
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
 void rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
 				 struct v4l2_ctrl_hevc_ext_sps_st_rps *st_cache)
 {
+	if (rkvdec_hevc_validate_rps_ctrls(run)) {
+		pr_err_ratelimited("rkvdec: rejecting HEVC SPS/RPS controls with out-of-range counts\n");
+		return;
+	}
+
 	rkvdec_hevc_prepare_hw_st_rps(run, rps, st_cache);
 	rkvdec_hevc_assemble_hw_lt_rps(run, rps);
 }
-- 
2.53.0