[PATCH] media: ipu6: return -EBUSY from busy video S_FMT

[Michael Bommarito <michael.bommarito@gmail.com>]

ipu6_isys_vidioc_try_fmt_vid_cap() rejects video format changes once
the vb2 queue is busy, but ipu6_isys_vidioc_s_fmt_vid_cap() ignores
that return value and still commits the user supplied format to
av->pix_fmt:

	ipu6_isys_vidioc_try_fmt_vid_cap(file, fh, f);
	av->pix_fmt = f->fmt.pix;

That lets userspace allocate and queue buffers under one capture
geometry, then change av->pix_fmt before STREAMON.  The queued vb2
buffers keep their original plane sizes and payloads, while the IPU6
stream setup later derives firmware output width, height and stride
from the mutated av->pix_fmt.

On a Dell Precision 5490 with a Meteor Lake IPU (8086:7d19), Ubuntu
26.04's 7.0.0-14-generic kernel, and an ov01a10 sensor path through
IVSC to /dev/video32, the following sequence reproduced the bad state:

  1. set /dev/video32 to 640x480 BG10
  2. REQBUFS and QBUF two mmap buffers
  3. call VIDIOC_S_FMT again with 1280x800 BG10

The second S_FMT returned success and G_FMT reported 1280x800, while
QUERYBUF still reported the old 615680 byte queued buffer lengths.
A subsequent STREAMON on the same live sensor route wedged the host
display/system hard enough to require a power cycle.  No KASAN,
IOMMU, oops or persistent crash record was captured, so this should
be treated as a local availability failure rather than a demonstrated
privilege escalation primitive.

Return the try-format error before committing av->pix_fmt.  This
matches the metadata S_FMT path, which already checks vb2_is_busy()
before assigning av->meta_fmt, and preserves the intended "no format
changes while buffers exist" invariant.

Tested on the same machine: with the patched modules loaded the busy
S_FMT sequence above now returns -EBUSY and av->pix_fmt is left alone,
while v4l2-compliance -d /dev/video0 still reports 48/48 pass.  The
existing compliance suite does not exercise S_FMT after REQBUFS, so
adding a busy-queue case there would catch future regressions of this
shape.

Fixes: d3bd039cd2a0 ("media: intel/ipu6: support line-based metadata capture support")
Cc: stable@vger.kernel.org
Signed-off-by: Michael Bommarito <michael.bommarito@gmail.com>
Assisted-by: Claude:claude-opus-4-7
Assisted-by: Codex:gpt-5-5-xhigh
---
 drivers/media/pci/intel/ipu6/ipu6-isys-video.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/media/pci/intel/ipu6/ipu6-isys-video.c b/drivers/media/pci/intel/ipu6/ipu6-isys-video.c
index 3ac48d2076da..88fceab6ed86 100644
--- a/drivers/media/pci/intel/ipu6/ipu6-isys-video.c
+++ b/drivers/media/pci/intel/ipu6/ipu6-isys-video.c
@@ -323,8 +323,12 @@ static int ipu6_isys_vidioc_s_fmt_vid_cap(struct file *file, void *fh,
 {
 	struct ipu6_isys_video *av = video_drvdata(file);
+	int ret;
 
-	ipu6_isys_vidioc_try_fmt_vid_cap(file, fh, f);
+	ret = ipu6_isys_vidioc_try_fmt_vid_cap(file, fh, f);
+	if (ret)
+		return ret;
+
 	av->pix_fmt = f->fmt.pix;
 
 	return 0;
 }
-- 
2.53.0