All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/3] media: imx: staging cleanup and FIM export to v4l2-core
@ 2026-11-21 14:14 Jiaqi Shi
  2026-11-21 14:14 ` [PATCH 1/3] media: imx: remove legacy video node control and format handling Jiaqi Shi
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Jiaqi Shi @ 2026-11-21 14:14 UTC (permalink / raw)
  To: linux-media, imx, linux-staging
  Cc: slongerbeam, p.zabel, mchehab, gregkh, Frank.Li, kernel, festevam,
	Jiaqi Shi

This series continues the i.MX staging media driver cleanup by removing
legacy video node handling and exporting the Frame Interval Monitor to
v4l2-core.

Patch 1 removes control inheritance and source-dependent format handling
from capture video nodes, as both are deprecated in favor of the
subdev-based userspace API.

Patch 2 moves the generic FIM implementation to v4l2-core so other
drivers can reuse it, while keeping the i.MX-specific input capture
controls in the i.MX adapter. The error calculation uses signed
arithmetic to avoid underflow when the measured interval is smaller
than the nominal interval.

Patch 3 allows the FIM input capture channel control to be written,
which was previously rejected with -EINVAL.

Compile-tested with allmodconfig; checkpatch.pl --strict is clean.
No runtime testing due to lack of hardware.

Jiaqi Shi (3):
  media: imx: remove legacy video node control and format handling
  media: v4l2-core: add frame interval monitor helper
  media: imx: allow setting FIM input capture channel

 Documentation/admin-guide/media/imx.rst       |  10 +-
 drivers/media/v4l2-core/Kconfig               |   4 +
 drivers/media/v4l2-core/Makefile              |   1 +
 drivers/media/v4l2-core/v4l2-fim.c            | 324 ++++++++++++++++
 drivers/staging/media/imx/Kconfig             |   1 +
 drivers/staging/media/imx/TODO                |  13 -
 drivers/staging/media/imx/imx-ic-prpencvf.c   |   5 +-
 drivers/staging/media/imx/imx-media-capture.c | 192 +---------
 drivers/staging/media/imx/imx-media-csi.c     |   5 +-
 .../staging/media/imx/imx-media-dev-common.c  | 113 +-----
 drivers/staging/media/imx/imx-media-fim.c     | 355 +++---------------
 drivers/staging/media/imx/imx-media.h         |   7 +-
 include/media/v4l2-fim.h                      |  40 ++
 13 files changed, 444 insertions(+), 626 deletions(-)
 create mode 100644 drivers/media/v4l2-core/v4l2-fim.c
 delete mode 100644 drivers/staging/media/imx/TODO
 create mode 100644 include/media/v4l2-fim.h

-- 
2.53.0


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH 1/3] media: imx: remove legacy video node control and format handling
  2026-11-21 14:14 [PATCH 0/3] media: imx: staging cleanup and FIM export to v4l2-core Jiaqi Shi
@ 2026-11-21 14:14 ` Jiaqi Shi
  2026-11-21 14:40   ` sashiko-bot
  2026-11-21 14:14 ` [PATCH 2/3] media: v4l2-core: add frame interval monitor helper Jiaqi Shi
  2026-11-21 14:14 ` [PATCH 3/3] media: imx: allow setting FIM input capture channel Jiaqi Shi
  2 siblings, 1 reply; 7+ messages in thread
From: Jiaqi Shi @ 2026-11-21 14:14 UTC (permalink / raw)
  To: linux-media, imx, linux-staging
  Cc: slongerbeam, p.zabel, mchehab, gregkh, Frank.Li, kernel, festevam,
	Jiaqi Shi

Remove control inheritance from pipeline subdevices to video nodes.
Userspace should access controls through subdev nodes instead.

Use source-independent format ioctls for all capture nodes and
initialize them to the existing UYVY 640x480 default instead of the
source subdev active format. Keep stream-time format validation.

Update the usage documentation and remove the completed TODO entries.

Signed-off-by: Jiaqi Shi <shijiaqi_develop@163.com>
---
 Documentation/admin-guide/media/imx.rst       |  10 +-
 drivers/staging/media/imx/TODO                |  10 -
 drivers/staging/media/imx/imx-ic-prpencvf.c   |   5 +-
 drivers/staging/media/imx/imx-media-capture.c | 192 ++----------------
 drivers/staging/media/imx/imx-media-csi.c     |   5 +-
 .../staging/media/imx/imx-media-dev-common.c  | 113 +----------
 drivers/staging/media/imx/imx-media.h         |   7 +-
 7 files changed, 21 insertions(+), 321 deletions(-)

diff --git a/Documentation/admin-guide/media/imx.rst b/Documentation/admin-guide/media/imx.rst
index bb68100d8acb..b1c5fecfaef7 100644
--- a/Documentation/admin-guide/media/imx.rst
+++ b/Documentation/admin-guide/media/imx.rst
@@ -394,13 +394,9 @@ scaling up to 1024x1024, CSC, flip, and rotation:
 Usage Notes
 -----------
 
-To aid in configuration and for backward compatibility with V4L2
-applications that access controls only from video device nodes, the
-capture device interfaces inherit controls from the active entities
-in the current pipeline, so controls can be accessed either directly
-from the subdev or from the active capture device interface. For
-example, the FIM controls are available either from the ipuX_csiY
-subdevs or from the active capture device.
+Controls for pipeline entities are accessed through their subdev device
+nodes. For example, the FIM controls are available from the ipuX_csiY
+subdevs.
 
 The following are specific usage notes for the Sabre* reference
 boards:
diff --git a/drivers/staging/media/imx/TODO b/drivers/staging/media/imx/TODO
index 11c9e10d34ae..43cd030d4bbd 100644
--- a/drivers/staging/media/imx/TODO
+++ b/drivers/staging/media/imx/TODO
@@ -1,13 +1,3 @@
 
 - The Frame Interval Monitor could be exported to v4l2-core for
   general use.
-
-- This media driver supports inheriting V4L2 controls to the
-  video capture devices, from the subdevices in the capture device's
-  pipeline. The controls for each capture device are updated in the
-  link_notify callback when the pipeline is modified. This feature should be
-  removed, userspace should use the subdev-based userspace API instead.
-
-- Similarly to the legacy control handling, legacy format handling where
-  formats on the video nodes are influenced by the active format of the
-  connected subdev should be removed.
diff --git a/drivers/staging/media/imx/imx-ic-prpencvf.c b/drivers/staging/media/imx/imx-ic-prpencvf.c
index 3430ddeac356..b3f2177e1a5d 100644
--- a/drivers/staging/media/imx/imx-ic-prpencvf.c
+++ b/drivers/staging/media/imx/imx-ic-prpencvf.c
@@ -1286,7 +1286,7 @@ static int prp_registered(struct v4l2_subdev *sd)
 
 	ret = imx_media_capture_device_register(priv->vdev, 0);
 	if (ret)
-		goto remove_vdev;
+		return ret;
 
 	ret = prp_init_controls(priv);
 	if (ret)
@@ -1296,8 +1296,6 @@ static int prp_registered(struct v4l2_subdev *sd)
 
 unreg_vdev:
 	imx_media_capture_device_unregister(priv->vdev);
-remove_vdev:
-	imx_media_capture_device_remove(priv->vdev);
 	return ret;
 }
 
@@ -1306,7 +1304,6 @@ static void prp_unregistered(struct v4l2_subdev *sd)
 	struct prp_priv *priv = sd_to_priv(sd);
 
 	imx_media_capture_device_unregister(priv->vdev);
-	imx_media_capture_device_remove(priv->vdev);
 
 	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
 }
diff --git a/drivers/staging/media/imx/imx-media-capture.c b/drivers/staging/media/imx/imx-media-capture.c
index 9216a221b72f..93d05cd96798 100644
--- a/drivers/staging/media/imx/imx-media-capture.c
+++ b/drivers/staging/media/imx/imx-media-capture.c
@@ -12,7 +12,6 @@
 #include <linux/slab.h>
 #include <linux/spinlock.h>
 #include <linux/timer.h>
-#include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
 #include <media/v4l2-event.h>
 #include <media/v4l2-fwnode.h>
@@ -41,10 +40,6 @@ struct capture_priv {
 	struct vb2_queue q;			/* The videobuf2 queue */
 	struct list_head ready_q;		/* List of queued buffers */
 	spinlock_t q_lock;			/* Protect ready_q */
-
-	struct v4l2_ctrl_handler ctrl_hdlr;	/* Controls inherited from subdevs */
-
-	bool legacy_api;			/* Use the legacy (pre-MC) API */
 };
 
 #define to_capture_priv(v) container_of(v, struct capture_priv, vdev)
@@ -346,133 +341,6 @@ static int capture_legacy_enum_frameintervals(struct file *file, void *fh,
 	return 0;
 }
 
-static int capture_legacy_enum_fmt_vid_cap(struct file *file, void *fh,
-					   struct v4l2_fmtdesc *f)
-{
-	struct capture_priv *priv = video_drvdata(file);
-	const struct imx_media_pixfmt *cc_src;
-	struct v4l2_subdev_format fmt_src = {
-		.pad = priv->src_sd_pad,
-		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
-	};
-	u32 fourcc;
-	int ret;
-
-	ret = v4l2_subdev_call(priv->src_sd, pad, get_fmt, NULL, &fmt_src);
-	if (ret) {
-		dev_err(priv->dev, "failed to get src_sd format\n");
-		return ret;
-	}
-
-	cc_src = imx_media_find_ipu_format(fmt_src.format.code,
-					   PIXFMT_SEL_YUV_RGB);
-	if (cc_src) {
-		enum imx_pixfmt_sel fmt_sel =
-			(cc_src->cs == IPUV3_COLORSPACE_YUV) ?
-			PIXFMT_SEL_YUV : PIXFMT_SEL_RGB;
-
-		ret = imx_media_enum_pixel_formats(&fourcc, f->index, fmt_sel,
-						   0);
-		if (ret)
-			return ret;
-	} else {
-		cc_src = imx_media_find_mbus_format(fmt_src.format.code,
-						    PIXFMT_SEL_ANY);
-		if (WARN_ON(!cc_src))
-			return -EINVAL;
-
-		if (f->index != 0)
-			return -EINVAL;
-		fourcc = cc_src->fourcc;
-	}
-
-	f->pixelformat = fourcc;
-
-	return 0;
-}
-
-static const struct imx_media_pixfmt *
-__capture_legacy_try_fmt(struct capture_priv *priv,
-			 struct v4l2_subdev_format *fmt_src,
-			 struct v4l2_pix_format *pixfmt)
-{
-	const struct imx_media_pixfmt *cc;
-
-	cc = capture_find_format(fmt_src->format.code, pixfmt->pixelformat);
-	if (WARN_ON(!cc))
-		return NULL;
-
-	/* allow IDMAC interweave but enforce field order from source */
-	if (V4L2_FIELD_IS_INTERLACED(pixfmt->field)) {
-		switch (fmt_src->format.field) {
-		case V4L2_FIELD_SEQ_TB:
-			fmt_src->format.field = V4L2_FIELD_INTERLACED_TB;
-			break;
-		case V4L2_FIELD_SEQ_BT:
-			fmt_src->format.field = V4L2_FIELD_INTERLACED_BT;
-			break;
-		default:
-			break;
-		}
-	}
-
-	imx_media_mbus_fmt_to_pix_fmt(pixfmt, &fmt_src->format, cc);
-
-	return cc;
-}
-
-static int capture_legacy_try_fmt_vid_cap(struct file *file, void *fh,
-					  struct v4l2_format *f)
-{
-	struct capture_priv *priv = video_drvdata(file);
-	struct v4l2_subdev_format fmt_src = {
-		.pad = priv->src_sd_pad,
-		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
-	};
-	int ret;
-
-	ret = v4l2_subdev_call(priv->src_sd, pad, get_fmt, NULL, &fmt_src);
-	if (ret)
-		return ret;
-
-	if (!__capture_legacy_try_fmt(priv, &fmt_src, &f->fmt.pix))
-		return -EINVAL;
-
-	return 0;
-}
-
-static int capture_legacy_s_fmt_vid_cap(struct file *file, void *fh,
-					struct v4l2_format *f)
-{
-	struct capture_priv *priv = video_drvdata(file);
-	struct v4l2_subdev_format fmt_src = {
-		.pad = priv->src_sd_pad,
-		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
-	};
-	const struct imx_media_pixfmt *cc;
-	int ret;
-
-	if (vb2_is_busy(&priv->q)) {
-		dev_err(priv->dev, "%s queue busy\n", __func__);
-		return -EBUSY;
-	}
-
-	ret = v4l2_subdev_call(priv->src_sd, pad, get_fmt, NULL, &fmt_src);
-	if (ret)
-		return ret;
-
-	cc = __capture_legacy_try_fmt(priv, &fmt_src, &f->fmt.pix);
-	if (!cc)
-		return -EINVAL;
-
-	priv->vdev.cc = cc;
-	priv->vdev.fmt = f->fmt.pix;
-	priv->vdev.compose.width = fmt_src.format.width;
-	priv->vdev.compose.height = fmt_src.format.height;
-
-	return 0;
-}
-
 static int capture_legacy_querystd(struct file *file, void *fh,
 				   v4l2_std_id *std)
 {
@@ -553,8 +421,6 @@ static int capture_legacy_subscribe_event(struct v4l2_fh *fh,
 		return v4l2_event_subscribe(fh, sub, 0, NULL);
 	case V4L2_EVENT_SOURCE_CHANGE:
 		return v4l2_src_change_event_subscribe(fh, sub);
-	case V4L2_EVENT_CTRL:
-		return v4l2_ctrl_subscribe_event(fh, sub);
 	default:
 		return -EINVAL;
 	}
@@ -566,10 +432,10 @@ static const struct v4l2_ioctl_ops capture_legacy_ioctl_ops = {
 	.vidioc_enum_framesizes		= capture_legacy_enum_framesizes,
 	.vidioc_enum_frameintervals	= capture_legacy_enum_frameintervals,
 
-	.vidioc_enum_fmt_vid_cap	= capture_legacy_enum_fmt_vid_cap,
+	.vidioc_enum_fmt_vid_cap	= capture_enum_fmt_vid_cap,
 	.vidioc_g_fmt_vid_cap		= capture_g_fmt_vid_cap,
-	.vidioc_try_fmt_vid_cap		= capture_legacy_try_fmt_vid_cap,
-	.vidioc_s_fmt_vid_cap		= capture_legacy_s_fmt_vid_cap,
+	.vidioc_try_fmt_vid_cap		= capture_try_fmt_vid_cap,
+	.vidioc_s_fmt_vid_cap		= capture_s_fmt_vid_cap,
 
 	.vidioc_querystd		= capture_legacy_querystd,
 	.vidioc_g_std			= capture_legacy_g_std,
@@ -869,36 +735,21 @@ void imx_media_capture_device_error(struct imx_media_video_dev *vdev)
 }
 EXPORT_SYMBOL_GPL(imx_media_capture_device_error);
 
-static int capture_init_format(struct capture_priv *priv)
+static void capture_init_format(struct capture_priv *priv)
 {
-	struct v4l2_subdev_format fmt_src = {
-		.pad = priv->src_sd_pad,
-		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
-	};
 	struct imx_media_video_dev *vdev = &priv->vdev;
-	int ret;
-
-	if (priv->legacy_api) {
-		ret = v4l2_subdev_call(priv->src_sd, pad, get_fmt, NULL,
-				       &fmt_src);
-		if (ret) {
-			dev_err(priv->dev, "failed to get source format\n");
-			return ret;
-		}
-	} else {
-		fmt_src.format.code = MEDIA_BUS_FMT_UYVY8_2X8;
-		fmt_src.format.width = IMX_MEDIA_DEF_PIX_WIDTH;
-		fmt_src.format.height = IMX_MEDIA_DEF_PIX_HEIGHT;
-	}
+	struct v4l2_mbus_framefmt fmt_src = {
+		.code = MEDIA_BUS_FMT_UYVY8_2X8,
+		.width = IMX_MEDIA_DEF_PIX_WIDTH,
+		.height = IMX_MEDIA_DEF_PIX_HEIGHT,
+	};
 
-	imx_media_mbus_fmt_to_pix_fmt(&vdev->fmt, &fmt_src.format, NULL);
-	vdev->compose.width = fmt_src.format.width;
-	vdev->compose.height = fmt_src.format.height;
+	imx_media_mbus_fmt_to_pix_fmt(&vdev->fmt, &fmt_src, NULL);
+	vdev->compose.width = fmt_src.width;
+	vdev->compose.height = fmt_src.height;
 
 	vdev->cc = imx_media_find_pixel_format(vdev->fmt.pixelformat,
 					       PIXFMT_SEL_ANY);
-
-	return 0;
 }
 
 int imx_media_capture_device_register(struct imx_media_video_dev *vdev,
@@ -916,9 +767,7 @@ int imx_media_capture_device_register(struct imx_media_video_dev *vdev,
 	vfd->v4l2_dev = v4l2_dev;
 
 	/* Initialize the default format and compose rectangle. */
-	ret = capture_init_format(priv);
-	if (ret < 0)
-		return ret;
+	capture_init_format(priv);
 
 	/* Register the video device. */
 	ret = video_register_device(vfd, VFL_TYPE_VIDEO, -1);
@@ -974,7 +823,6 @@ imx_media_capture_device_init(struct device *dev, struct v4l2_subdev *src_sd,
 	priv->src_sd = src_sd;
 	priv->src_sd_pad = pad;
 	priv->dev = dev;
-	priv->legacy_api = legacy_api;
 
 	mutex_init(&priv->mutex);
 	INIT_LIST_HEAD(&priv->ready_q);
@@ -1031,24 +879,10 @@ imx_media_capture_device_init(struct device *dev, struct v4l2_subdev *src_sd,
 		return ERR_PTR(ret);
 	}
 
-	if (legacy_api) {
-		/* Initialize the control handler. */
-		v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
-		vfd->ctrl_handler = &priv->ctrl_hdlr;
-	}
-
 	return &priv->vdev;
 }
 EXPORT_SYMBOL_GPL(imx_media_capture_device_init);
 
-void imx_media_capture_device_remove(struct imx_media_video_dev *vdev)
-{
-	struct capture_priv *priv = to_capture_priv(vdev);
-
-	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
-}
-EXPORT_SYMBOL_GPL(imx_media_capture_device_remove);
-
 MODULE_DESCRIPTION("i.MX5/6 v4l2 video capture interface driver");
 MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
 MODULE_LICENSE("GPL");
diff --git a/drivers/staging/media/imx/imx-media-csi.c b/drivers/staging/media/imx/imx-media-csi.c
index 43c637b333f2..3a285fd10ac9 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -1823,12 +1823,10 @@ static int csi_registered(struct v4l2_subdev *sd)
 
 	ret = imx_media_capture_device_register(priv->vdev, 0);
 	if (ret)
-		goto remove_vdev;
+		goto free_fim;
 
 	return 0;
 
-remove_vdev:
-	imx_media_capture_device_remove(priv->vdev);
 free_fim:
 	if (priv->fim)
 		imx_media_fim_free(priv->fim);
@@ -1842,7 +1840,6 @@ static void csi_unregistered(struct v4l2_subdev *sd)
 	struct csi_priv *priv = v4l2_get_subdevdata(sd);
 
 	imx_media_capture_device_unregister(priv->vdev);
-	imx_media_capture_device_remove(priv->vdev);
 
 	if (priv->fim)
 		imx_media_fim_free(priv->fim);
diff --git a/drivers/staging/media/imx/imx-media-dev-common.c b/drivers/staging/media/imx/imx-media-dev-common.c
index 0d0ee8627a2d..8a0ce7e394ae 100644
--- a/drivers/staging/media/imx/imx-media-dev-common.c
+++ b/drivers/staging/media/imx/imx-media-dev-common.c
@@ -6,7 +6,6 @@
  * Copyright (c) 2016 Mentor Graphics Inc.
  */
 
-#include <media/v4l2-ctrls.h>
 #include <media/v4l2-event.h>
 #include <media/v4l2-ioctl.h>
 #include <media/v4l2-mc.h>
@@ -189,116 +188,6 @@ int imx_media_probe_complete(struct v4l2_async_notifier *notifier)
 }
 EXPORT_SYMBOL_GPL(imx_media_probe_complete);
 
-/*
- * adds controls to a video device from an entity subdevice.
- * Continues upstream from the entity's sink pads.
- */
-static int imx_media_inherit_controls(struct imx_media_dev *imxmd,
-				      struct video_device *vfd,
-				      struct media_entity *entity)
-{
-	int i, ret = 0;
-
-	if (is_media_entity_v4l2_subdev(entity)) {
-		struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
-
-		dev_dbg(imxmd->md.dev,
-			"adding controls to %s from %s\n",
-			vfd->entity.name, sd->entity.name);
-
-		ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
-					    sd->ctrl_handler,
-					    NULL, true);
-		if (ret)
-			return ret;
-	}
-
-	/* move upstream */
-	for (i = 0; i < entity->num_pads; i++) {
-		struct media_pad *pad, *spad = &entity->pads[i];
-
-		if (!(spad->flags & MEDIA_PAD_FL_SINK))
-			continue;
-
-		pad = media_pad_remote_pad_first(spad);
-		if (!pad || !is_media_entity_v4l2_subdev(pad->entity))
-			continue;
-
-		ret = imx_media_inherit_controls(imxmd, vfd, pad->entity);
-		if (ret)
-			break;
-	}
-
-	return ret;
-}
-
-static int imx_media_link_notify(struct media_link *link, u32 flags,
-				 unsigned int notification)
-{
-	struct imx_media_dev *imxmd = container_of(link->graph_obj.mdev,
-						   struct imx_media_dev, md);
-	struct media_entity *source = link->source->entity;
-	struct imx_media_pad_vdev *pad_vdev;
-	struct list_head *pad_vdev_list;
-	struct video_device *vfd;
-	struct v4l2_subdev *sd;
-	int pad_idx, ret;
-
-	ret = v4l2_pipeline_link_notify(link, flags, notification);
-	if (ret)
-		return ret;
-
-	/* don't bother if source is not a subdev */
-	if (!is_media_entity_v4l2_subdev(source))
-		return 0;
-
-	sd = media_entity_to_v4l2_subdev(source);
-	pad_idx = link->source->index;
-
-	pad_vdev_list = to_pad_vdev_list(sd, pad_idx);
-	if (!pad_vdev_list) {
-		/* nothing to do if source sd has no pad vdev list */
-		return 0;
-	}
-
-	/*
-	 * Before disabling a link, reset controls for all video
-	 * devices reachable from this link.
-	 *
-	 * After enabling a link, refresh controls for all video
-	 * devices reachable from this link.
-	 */
-	if (notification == MEDIA_DEV_NOTIFY_PRE_LINK_CH &&
-	    !(flags & MEDIA_LNK_FL_ENABLED)) {
-		list_for_each_entry(pad_vdev, pad_vdev_list, list) {
-			vfd = pad_vdev->vdev->vfd;
-			if (!vfd->ctrl_handler)
-				continue;
-			dev_dbg(imxmd->md.dev,
-				"reset controls for %s\n",
-				vfd->entity.name);
-			v4l2_ctrl_handler_free(vfd->ctrl_handler);
-			v4l2_ctrl_handler_init(vfd->ctrl_handler, 0);
-		}
-	} else if (notification == MEDIA_DEV_NOTIFY_POST_LINK_CH &&
-		   (link->flags & MEDIA_LNK_FL_ENABLED)) {
-		list_for_each_entry(pad_vdev, pad_vdev_list, list) {
-			vfd = pad_vdev->vdev->vfd;
-			if (!vfd->ctrl_handler)
-				continue;
-			dev_dbg(imxmd->md.dev,
-				"refresh controls for %s\n",
-				vfd->entity.name);
-			ret = imx_media_inherit_controls(imxmd, vfd,
-							 &vfd->entity);
-			if (ret)
-				break;
-		}
-	}
-
-	return ret;
-}
-
 static void imx_media_notify(struct v4l2_subdev *sd, unsigned int notification,
 			     void *arg)
 {
@@ -326,7 +215,7 @@ static const struct v4l2_async_notifier_operations imx_media_notifier_ops = {
 };
 
 static const struct media_device_ops imx_media_md_ops = {
-	.link_notify = imx_media_link_notify,
+	.link_notify = v4l2_pipeline_link_notify,
 };
 
 struct imx_media_dev *imx_media_dev_init(struct device *dev,
diff --git a/drivers/staging/media/imx/imx-media.h b/drivers/staging/media/imx/imx-media.h
index 537558a7bece..832c9d292353 100644
--- a/drivers/staging/media/imx/imx-media.h
+++ b/drivers/staging/media/imx/imx-media.h
@@ -125,10 +125,8 @@ static inline struct imx_media_buffer *to_imx_media_vb(struct vb2_buffer *vb)
 }
 
 /*
- * to support control inheritance to video devices, this
- * retrieves a pad's list_head of video devices that can
- * be reached from the pad. Note that only the lists in
- * source pads get populated, sink pads have empty lists.
+ * Retrieve a pad's list of video devices reachable from the pad for
+ * subdev event forwarding. Only source pad lists are populated.
  */
 static inline struct list_head *
 to_pad_vdev_list(struct v4l2_subdev *sd, int pad_index)
@@ -271,7 +269,6 @@ int imx_media_ic_unregister(struct v4l2_subdev *sd);
 struct imx_media_video_dev *
 imx_media_capture_device_init(struct device *dev, struct v4l2_subdev *src_sd,
 			      int pad, bool legacy_api);
-void imx_media_capture_device_remove(struct imx_media_video_dev *vdev);
 int imx_media_capture_device_register(struct imx_media_video_dev *vdev,
 				      u32 link_flags);
 void imx_media_capture_device_unregister(struct imx_media_video_dev *vdev);
-- 
2.53.0


^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH 2/3] media: v4l2-core: add frame interval monitor helper
  2026-11-21 14:14 [PATCH 0/3] media: imx: staging cleanup and FIM export to v4l2-core Jiaqi Shi
  2026-11-21 14:14 ` [PATCH 1/3] media: imx: remove legacy video node control and format handling Jiaqi Shi
@ 2026-11-21 14:14 ` Jiaqi Shi
  2026-11-21 14:27   ` sashiko-bot
  2026-11-21 14:14 ` [PATCH 3/3] media: imx: allow setting FIM input capture channel Jiaqi Shi
  2 siblings, 1 reply; 7+ messages in thread
From: Jiaqi Shi @ 2026-11-21 14:14 UTC (permalink / raw)
  To: linux-media, imx, linux-staging
  Cc: slongerbeam, p.zabel, mchehab, gregkh, Frank.Li, kernel, festevam,
	Jiaqi Shi

Move the generic frame interval monitoring implementation from i.MX
to v4l2-core. The helper supports driver-selected control and event
IDs and accepts EOF or external timestamps.

Keep the i.MX-specific input capture controls in the i.MX adapter,
sharing the original control handler and preserving the existing UAPI
and event semantics.

Select V4L2_FIM from VIDEO_IMX_MEDIA and remove the completed TODO.

Signed-off-by: Jiaqi Shi <shijiaqi_develop@163.com>
---
 drivers/media/v4l2-core/Kconfig           |   4 +
 drivers/media/v4l2-core/Makefile          |   1 +
 drivers/media/v4l2-core/v4l2-fim.c        | 324 ++++++++++++++++++++
 drivers/staging/media/imx/Kconfig         |   1 +
 drivers/staging/media/imx/TODO            |   3 -
 drivers/staging/media/imx/imx-media-fim.c | 352 ++++------------------
 include/media/v4l2-fim.h                  |  40 +++
 7 files changed, 421 insertions(+), 304 deletions(-)
 create mode 100644 drivers/media/v4l2-core/v4l2-fim.c
 delete mode 100644 drivers/staging/media/imx/TODO
 create mode 100644 include/media/v4l2-fim.h

diff --git a/drivers/media/v4l2-core/Kconfig b/drivers/media/v4l2-core/Kconfig
index d50ccac9733c..cff961b24c04 100644
--- a/drivers/media/v4l2-core/Kconfig
+++ b/drivers/media/v4l2-core/Kconfig
@@ -40,6 +40,10 @@ config VIDEO_TUNER
 config V4L2_JPEG_HELPER
 	tristate
 
+# Used by drivers that need v4l2-fim.ko
+config V4L2_FIM
+	tristate
+
 # Used by drivers that need v4l2-h264.ko
 config V4L2_H264
 	tristate
diff --git a/drivers/media/v4l2-core/Makefile b/drivers/media/v4l2-core/Makefile
index 329f0eadce99..d5f6699a2669 100644
--- a/drivers/media/v4l2-core/Makefile
+++ b/drivers/media/v4l2-core/Makefile
@@ -26,6 +26,7 @@ videodev-$(CONFIG_VIDEO_V4L2_I2C) += v4l2-i2c.o
 
 obj-$(CONFIG_V4L2_ASYNC) += v4l2-async.o
 obj-$(CONFIG_V4L2_CCI) += v4l2-cci.o
+obj-$(CONFIG_V4L2_FIM) += v4l2-fim.o
 obj-$(CONFIG_V4L2_FLASH_LED_CLASS) += v4l2-flash-led-class.o
 obj-$(CONFIG_V4L2_FWNODE) += v4l2-fwnode.o
 obj-$(CONFIG_V4L2_H264) += v4l2-h264.o
diff --git a/drivers/media/v4l2-core/v4l2-fim.c b/drivers/media/v4l2-core/v4l2-fim.c
new file mode 100644
index 000000000000..57d595196663
--- /dev/null
+++ b/drivers/media/v4l2-core/v4l2-fim.c
@@ -0,0 +1,324 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Frame Interval Monitor.
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.
+ */
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-fim.h>
+#include <media/v4l2-subdev.h>
+
+enum {
+	FIM_CL_ENABLE = 0,
+	FIM_CL_NUM,
+	FIM_CL_TOLERANCE_MIN,
+	FIM_CL_TOLERANCE_MAX,
+	FIM_CL_NUM_SKIP,
+};
+
+#define FIM_CL_ENABLE_DEF          0 /* FIM disabled by default */
+#define FIM_CL_NUM_DEF             8 /* average 8 frames */
+#define FIM_CL_NUM_SKIP_DEF        2 /* skip 2 frames after restart */
+#define FIM_CL_TOLERANCE_MIN_DEF  50 /* usec */
+#define FIM_CL_TOLERANCE_MAX_DEF   0 /* no max tolerance (unbounded) */
+
+struct v4l2_fim {
+	struct v4l2_subdev *sd;
+	struct v4l2_fim_config config;
+	struct v4l2_ctrl *ctrl[V4L2_FIM_NUM_CONTROLS];
+
+	spinlock_t lock; /* protect control values and monitoring state */
+
+	bool enabled;
+	int num_avg;
+	int num_skip;
+	unsigned long tolerance_min;
+	unsigned long tolerance_max;
+	bool external_timestamp;
+
+	int counter;
+	ktime_t last_ts;
+	unsigned long sum;
+	unsigned long nominal;
+	bool stream_on;
+};
+
+static void update_fim_nominal(struct v4l2_fim *fim,
+			       const struct v4l2_fract *fi)
+{
+	if (fi->denominator == 0) {
+		dev_dbg(fim->sd->dev, "no frame interval, FIM disabled\n");
+		fim->enabled = false;
+		return;
+	}
+
+	fim->nominal = DIV_ROUND_CLOSEST_ULL(1000000ULL * (u64)fi->numerator,
+					     fi->denominator);
+
+	dev_dbg(fim->sd->dev, "FI=%lu usec\n", fim->nominal);
+}
+
+static void reset_fim(struct v4l2_fim *fim, bool curval)
+{
+	struct v4l2_ctrl *en = fim->ctrl[FIM_CL_ENABLE];
+	struct v4l2_ctrl *num = fim->ctrl[FIM_CL_NUM];
+	struct v4l2_ctrl *skip = fim->ctrl[FIM_CL_NUM_SKIP];
+	struct v4l2_ctrl *tol_min = fim->ctrl[FIM_CL_TOLERANCE_MIN];
+	struct v4l2_ctrl *tol_max = fim->ctrl[FIM_CL_TOLERANCE_MAX];
+
+	if (fim->config.external_timestamp)
+		fim->external_timestamp =
+			fim->config.external_timestamp(fim->config.priv, curval);
+
+	if (curval) {
+		fim->enabled = en->cur.val;
+		fim->num_avg = num->cur.val;
+		fim->num_skip = skip->cur.val;
+		fim->tolerance_min = tol_min->cur.val;
+		fim->tolerance_max = tol_max->cur.val;
+	} else {
+		fim->enabled = en->val;
+		fim->num_avg = num->val;
+		fim->num_skip = skip->val;
+		fim->tolerance_min = tol_min->val;
+		fim->tolerance_max = tol_max->val;
+	}
+
+	/* disable tolerance range if max <= min */
+	if (fim->tolerance_max <= fim->tolerance_min)
+		fim->tolerance_max = 0;
+
+	/* num_skip must be >= 1 if input capture not used */
+	if (!fim->external_timestamp)
+		fim->num_skip = max_t(int, fim->num_skip, 1);
+
+	fim->counter = -fim->num_skip;
+	fim->sum = 0;
+}
+
+void v4l2_fim_reset(struct v4l2_fim *fim, bool curval)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&fim->lock, flags);
+	reset_fim(fim, curval);
+	spin_unlock_irqrestore(&fim->lock, flags);
+}
+EXPORT_SYMBOL_GPL(v4l2_fim_reset);
+
+static void send_fim_event(struct v4l2_fim *fim)
+{
+	struct v4l2_event ev = {
+		.type = fim->config.event_type,
+	};
+
+	v4l2_subdev_notify_event(fim->sd, &ev);
+}
+
+/*
+ * Monitor an averaged frame interval. If the average deviates too much
+ * from the nominal frame rate, send the frame interval error event. The
+ * frame intervals are averaged in order to quiet noise from
+ * (presumably random) interrupt latency.
+ */
+static void frame_interval_monitor(struct v4l2_fim *fim, ktime_t timestamp)
+{
+	long long interval, error;
+	unsigned long error_avg;
+	bool send_event = false;
+
+	if (!fim->enabled || ++fim->counter <= 0)
+		goto out_update_ts;
+
+	/* max error is less than 100µs, so use 32-bit division or fail */
+	interval = ktime_to_ns(ktime_sub(timestamp, fim->last_ts));
+	error = abs(interval - NSEC_PER_USEC * (s64)fim->nominal);
+	if (error > U32_MAX)
+		error = U32_MAX;
+	else
+		error = abs((u32)error / NSEC_PER_USEC);
+
+	if (fim->tolerance_max && error >= fim->tolerance_max) {
+		dev_dbg(fim->sd->dev,
+			"FIM: %llu ignored, out of tolerance bounds\n",
+			error);
+		fim->counter--;
+		goto out_update_ts;
+	}
+
+	fim->sum += error;
+
+	if (fim->counter == fim->num_avg) {
+		error_avg = DIV_ROUND_CLOSEST(fim->sum, fim->num_avg);
+
+		if (error_avg > fim->tolerance_min)
+			send_event = true;
+
+		dev_dbg(fim->sd->dev, "FIM: error: %lu usec%s\n",
+			error_avg, send_event ? " (!!!)" : "");
+
+		fim->counter = 0;
+		fim->sum = 0;
+	}
+
+out_update_ts:
+	fim->last_ts = timestamp;
+	if (send_event)
+		send_fim_event(fim);
+}
+
+void v4l2_fim_eof_monitor(struct v4l2_fim *fim, ktime_t timestamp)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&fim->lock, flags);
+
+	if (!fim->external_timestamp)
+		frame_interval_monitor(fim, timestamp);
+
+	spin_unlock_irqrestore(&fim->lock, flags);
+}
+EXPORT_SYMBOL_GPL(v4l2_fim_eof_monitor);
+
+void v4l2_fim_monitor_timestamp(struct v4l2_fim *fim, ktime_t timestamp)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&fim->lock, flags);
+	frame_interval_monitor(fim, timestamp);
+	spin_unlock_irqrestore(&fim->lock, flags);
+}
+EXPORT_SYMBOL_GPL(v4l2_fim_monitor_timestamp);
+
+void v4l2_fim_set_stream(struct v4l2_fim *fim,
+			 const struct v4l2_fract *fi, bool on)
+{
+	unsigned long flags;
+
+	v4l2_ctrl_lock(fim->ctrl[FIM_CL_ENABLE]);
+
+	if (fim->stream_on == on)
+		goto out;
+
+	if (on) {
+		spin_lock_irqsave(&fim->lock, flags);
+		reset_fim(fim, true);
+		update_fim_nominal(fim, fi);
+		spin_unlock_irqrestore(&fim->lock, flags);
+
+		if (fim->external_timestamp && fim->enabled && !fim->num_skip &&
+		    fim->config.wait_first_timestamp)
+			fim->config.wait_first_timestamp(fim->config.priv);
+	}
+
+	fim->stream_on = on;
+out:
+	v4l2_ctrl_unlock(fim->ctrl[FIM_CL_ENABLE]);
+}
+EXPORT_SYMBOL_GPL(v4l2_fim_set_stream);
+
+bool v4l2_fim_is_streaming(struct v4l2_fim *fim)
+{
+	return fim->stream_on;
+}
+EXPORT_SYMBOL_GPL(v4l2_fim_is_streaming);
+
+static int fim_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct v4l2_fim *fim = ctrl->priv;
+	unsigned long flags;
+
+	spin_lock_irqsave(&fim->lock, flags);
+	reset_fim(fim, false);
+	spin_unlock_irqrestore(&fim->lock, flags);
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops fim_ctrl_ops = {
+	.s_ctrl = fim_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config fim_ctrl[] = {
+	[FIM_CL_ENABLE] = {
+		.ops = &fim_ctrl_ops,
+		.name = "FIM Enable",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def = FIM_CL_ENABLE_DEF,
+		.min = 0,
+		.max = 1,
+		.step = 1,
+	},
+	[FIM_CL_NUM] = {
+		.ops = &fim_ctrl_ops,
+		.name = "FIM Num Average",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def = FIM_CL_NUM_DEF,
+		.min =  1,
+		.max = 64,
+		.step = 1,
+	},
+	[FIM_CL_TOLERANCE_MIN] = {
+		.ops = &fim_ctrl_ops,
+		.name = "FIM Tolerance Min",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def = FIM_CL_TOLERANCE_MIN_DEF,
+		.min =    2,
+		.max =  200,
+		.step =  1,
+	},
+	[FIM_CL_TOLERANCE_MAX] = {
+		.ops = &fim_ctrl_ops,
+		.name = "FIM Tolerance Max",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def = FIM_CL_TOLERANCE_MAX_DEF,
+		.min =    0,
+		.max =  500,
+		.step =   1,
+	},
+	[FIM_CL_NUM_SKIP] = {
+		.ops = &fim_ctrl_ops,
+		.name = "FIM Num Skip",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def = FIM_CL_NUM_SKIP_DEF,
+		.min =   0,
+		.max = 256,
+		.step = 1,
+	},
+};
+
+struct v4l2_fim *v4l2_fim_init(struct v4l2_subdev *sd,
+			       struct v4l2_ctrl_handler *hdlr,
+			       const struct v4l2_fim_config *config)
+{
+	struct v4l2_fim *fim;
+	int i;
+
+	fim = devm_kzalloc(sd->dev, sizeof(*fim), GFP_KERNEL);
+	if (!fim)
+		return ERR_PTR(-ENOMEM);
+
+	fim->sd = sd;
+	fim->config = *config;
+	spin_lock_init(&fim->lock);
+
+	for (i = 0; i < V4L2_FIM_NUM_CONTROLS; i++) {
+		struct v4l2_ctrl_config cfg = fim_ctrl[i];
+
+		cfg.id = config->ctrl_base + i;
+		fim->ctrl[i] = v4l2_ctrl_new_custom(hdlr, &cfg, fim);
+	}
+	if (hdlr->error)
+		return ERR_PTR(hdlr->error);
+
+	v4l2_ctrl_cluster(V4L2_FIM_NUM_CONTROLS, fim->ctrl);
+
+	return fim;
+}
+EXPORT_SYMBOL_GPL(v4l2_fim_init);
+
+MODULE_DESCRIPTION("V4L2 frame interval monitor helpers");
+MODULE_LICENSE("GPL");
diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
index 1cd48028b641..f559b9979039 100644
--- a/drivers/staging/media/imx/Kconfig
+++ b/drivers/staging/media/imx/Kconfig
@@ -6,6 +6,7 @@ config VIDEO_IMX_MEDIA
 	depends on VIDEO_DEV
 	depends on IMX_IPUV3_CORE
 	select MEDIA_CONTROLLER
+	select V4L2_FIM
 	select V4L2_FWNODE
 	select V4L2_MEM2MEM_DEV
 	select VIDEOBUF2_DMA_CONTIG
diff --git a/drivers/staging/media/imx/TODO b/drivers/staging/media/imx/TODO
deleted file mode 100644
index 43cd030d4bbd..000000000000
--- a/drivers/staging/media/imx/TODO
+++ /dev/null
@@ -1,3 +0,0 @@
-
-- The Frame Interval Monitor could be exported to v4l2-core for
-  general use.
diff --git a/drivers/staging/media/imx/imx-media-fim.c b/drivers/staging/media/imx/imx-media-fim.c
index ccbc0371fba2..b6467e29ac28 100644
--- a/drivers/staging/media/imx/imx-media-fim.c
+++ b/drivers/staging/media/imx/imx-media-fim.c
@@ -4,68 +4,31 @@
  *
  * Copyright (c) 2016 Mentor Graphics Inc.
  */
-#include <linux/delay.h>
+#include <linux/completion.h>
 #include <linux/irq.h>
-#include <linux/module.h>
-#include <linux/platform_device.h>
+#include <linux/jiffies.h>
 #include <linux/slab.h>
-#include <linux/spinlock.h>
 #include <media/v4l2-ctrls.h>
+#include <media/v4l2-fim.h>
 #include <media/v4l2-subdev.h>
 #include <media/imx.h>
 #include "imx-media.h"
 
-enum {
-	FIM_CL_ENABLE = 0,
-	FIM_CL_NUM,
-	FIM_CL_TOLERANCE_MIN,
-	FIM_CL_TOLERANCE_MAX,
-	FIM_CL_NUM_SKIP,
-	FIM_NUM_CONTROLS,
-};
-
 enum {
 	FIM_CL_ICAP_EDGE = 0,
 	FIM_CL_ICAP_CHANNEL,
 	FIM_NUM_ICAP_CONTROLS,
 };
 
-#define FIM_CL_ENABLE_DEF          0 /* FIM disabled by default */
-#define FIM_CL_NUM_DEF             8 /* average 8 frames */
-#define FIM_CL_NUM_SKIP_DEF        2 /* skip 2 frames after restart */
-#define FIM_CL_TOLERANCE_MIN_DEF  50 /* usec */
-#define FIM_CL_TOLERANCE_MAX_DEF   0 /* no max tolerance (unbounded) */
-
 struct imx_media_fim {
-	/* the owning subdev of this fim instance */
 	struct v4l2_subdev *sd;
-
-	/* FIM's control handler */
+	struct v4l2_fim *core;
 	struct v4l2_ctrl_handler ctrl_handler;
+	struct v4l2_ctrl *icap_ctrl[FIM_NUM_ICAP_CONTROLS];
 
-	/* control clusters */
-	struct v4l2_ctrl  *ctrl[FIM_NUM_CONTROLS];
-	struct v4l2_ctrl  *icap_ctrl[FIM_NUM_ICAP_CONTROLS];
-
-	spinlock_t        lock; /* protect control values */
-
-	/* current control values */
-	bool              enabled;
-	int               num_avg;
-	int               num_skip;
-	unsigned long     tolerance_min; /* usec */
-	unsigned long     tolerance_max; /* usec */
-	/* input capture method of measuring FI */
-	int               icap_channel;
-	int               icap_flags;
-
-	int               counter;
-	ktime_t		  last_ts;
-	unsigned long     sum;       /* usec */
-	unsigned long     nominal;   /* usec */
-
+	int icap_channel;
+	int icap_flags;
 	struct completion icap_first_event;
-	bool              stream_on;
 };
 
 static bool icap_enabled(struct imx_media_fim *fim)
@@ -73,171 +36,53 @@ static bool icap_enabled(struct imx_media_fim *fim)
 	return fim->icap_flags != IRQ_TYPE_NONE;
 }
 
-static void update_fim_nominal(struct imx_media_fim *fim,
-			       const struct v4l2_fract *fi)
-{
-	if (fi->denominator == 0) {
-		dev_dbg(fim->sd->dev, "no frame interval, FIM disabled\n");
-		fim->enabled = false;
-		return;
-	}
-
-	fim->nominal = DIV_ROUND_CLOSEST_ULL(1000000ULL * (u64)fi->numerator,
-					     fi->denominator);
-
-	dev_dbg(fim->sd->dev, "FI=%lu usec\n", fim->nominal);
-}
-
-static void reset_fim(struct imx_media_fim *fim, bool curval)
+static bool imx_fim_external_timestamp(void *priv, bool curval)
 {
+	struct imx_media_fim *fim = priv;
 	struct v4l2_ctrl *icap_chan = fim->icap_ctrl[FIM_CL_ICAP_CHANNEL];
 	struct v4l2_ctrl *icap_edge = fim->icap_ctrl[FIM_CL_ICAP_EDGE];
-	struct v4l2_ctrl *en = fim->ctrl[FIM_CL_ENABLE];
-	struct v4l2_ctrl *num = fim->ctrl[FIM_CL_NUM];
-	struct v4l2_ctrl *skip = fim->ctrl[FIM_CL_NUM_SKIP];
-	struct v4l2_ctrl *tol_min = fim->ctrl[FIM_CL_TOLERANCE_MIN];
-	struct v4l2_ctrl *tol_max = fim->ctrl[FIM_CL_TOLERANCE_MAX];
 
 	if (curval) {
-		fim->enabled = en->cur.val;
 		fim->icap_flags = icap_edge->cur.val;
 		fim->icap_channel = icap_chan->cur.val;
-		fim->num_avg = num->cur.val;
-		fim->num_skip = skip->cur.val;
-		fim->tolerance_min = tol_min->cur.val;
-		fim->tolerance_max = tol_max->cur.val;
 	} else {
-		fim->enabled = en->val;
 		fim->icap_flags = icap_edge->val;
 		fim->icap_channel = icap_chan->val;
-		fim->num_avg = num->val;
-		fim->num_skip = skip->val;
-		fim->tolerance_min = tol_min->val;
-		fim->tolerance_max = tol_max->val;
 	}
 
-	/* disable tolerance range if max <= min */
-	if (fim->tolerance_max <= fim->tolerance_min)
-		fim->tolerance_max = 0;
-
-	/* num_skip must be >= 1 if input capture not used */
-	if (!icap_enabled(fim))
-		fim->num_skip = max_t(int, fim->num_skip, 1);
-
-	fim->counter = -fim->num_skip;
-	fim->sum = 0;
+	return icap_enabled(fim);
 }
 
-static void send_fim_event(struct imx_media_fim *fim, unsigned long error)
-{
-	static const struct v4l2_event ev = {
-		.type = V4L2_EVENT_IMX_FRAME_INTERVAL_ERROR,
-	};
-
-	v4l2_subdev_notify_event(fim->sd, &ev);
-}
-
-/*
- * Monitor an averaged frame interval. If the average deviates too much
- * from the nominal frame rate, send the frame interval error event. The
- * frame intervals are averaged in order to quiet noise from
- * (presumably random) interrupt latency.
- */
-static void frame_interval_monitor(struct imx_media_fim *fim,
-				   ktime_t timestamp)
-{
-	long long interval, error;
-	unsigned long error_avg;
-	bool send_event = false;
-
-	if (!fim->enabled || ++fim->counter <= 0)
-		goto out_update_ts;
-
-	/* max error is less than l00µs, so use 32-bit division or fail */
-	interval = ktime_to_ns(ktime_sub(timestamp, fim->last_ts));
-	error = abs(interval - NSEC_PER_USEC * (u64)fim->nominal);
-	if (error > U32_MAX)
-		error = U32_MAX;
-	else
-		error = abs((u32)error / NSEC_PER_USEC);
-
-	if (fim->tolerance_max && error >= fim->tolerance_max) {
-		dev_dbg(fim->sd->dev,
-			"FIM: %llu ignored, out of tolerance bounds\n",
-			error);
-		fim->counter--;
-		goto out_update_ts;
-	}
-
-	fim->sum += error;
-
-	if (fim->counter == fim->num_avg) {
-		error_avg = DIV_ROUND_CLOSEST(fim->sum, fim->num_avg);
-
-		if (error_avg > fim->tolerance_min)
-			send_event = true;
-
-		dev_dbg(fim->sd->dev, "FIM: error: %lu usec%s\n",
-			error_avg, send_event ? " (!!!)" : "");
-
-		fim->counter = 0;
-		fim->sum = 0;
-	}
-
-out_update_ts:
-	fim->last_ts = timestamp;
-	if (send_event)
-		send_fim_event(fim, error_avg);
-}
-
-/*
- * In case we are monitoring the first frame interval after streamon
- * (when fim->num_skip = 0), we need a valid fim->last_ts before we
- * can begin. This only applies to the input capture method. It is not
- * possible to accurately measure the first FI after streamon using the
- * EOF method, so fim->num_skip minimum is set to 1 in that case, so this
- * function is a noop when the EOF method is used.
- */
-static void fim_acquire_first_ts(struct imx_media_fim *fim)
+static void imx_fim_wait_first_timestamp(void *priv)
 {
+	struct imx_media_fim *fim = priv;
 	unsigned long ret;
 
-	if (!fim->enabled || fim->num_skip > 0)
-		return;
-
 	ret = wait_for_completion_timeout(
-		&fim->icap_first_event,
-		msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+			&fim->icap_first_event,
+			msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
 	if (ret == 0)
 		v4l2_warn(fim->sd, "wait first icap event timeout\n");
 }
 
-/* FIM Controls */
 static int fim_s_ctrl(struct v4l2_ctrl *ctrl)
 {
-	struct imx_media_fim *fim = container_of(ctrl->handler,
-						 struct imx_media_fim,
-						 ctrl_handler);
-	unsigned long flags;
+	struct imx_media_fim *fim = ctrl->priv;
 	int ret = 0;
 
-	spin_lock_irqsave(&fim->lock, flags);
-
 	switch (ctrl->id) {
-	case V4L2_CID_IMX_FIM_ENABLE:
-		break;
 	case V4L2_CID_IMX_FIM_ICAP_EDGE:
-		if (fim->stream_on)
+		if (v4l2_fim_is_streaming(fim->core))
 			ret = -EBUSY;
 		break;
 	default:
 		ret = -EINVAL;
+		break;
 	}
 
 	if (!ret)
-		reset_fim(fim, false);
+		v4l2_fim_reset(fim->core, false);
 
-	spin_unlock_irqrestore(&fim->lock, flags);
 	return ret;
 }
 
@@ -245,66 +90,13 @@ static const struct v4l2_ctrl_ops fim_ctrl_ops = {
 	.s_ctrl = fim_s_ctrl,
 };
 
-static const struct v4l2_ctrl_config fim_ctrl[] = {
-	[FIM_CL_ENABLE] = {
-		.ops = &fim_ctrl_ops,
-		.id = V4L2_CID_IMX_FIM_ENABLE,
-		.name = "FIM Enable",
-		.type = V4L2_CTRL_TYPE_BOOLEAN,
-		.def = FIM_CL_ENABLE_DEF,
-		.min = 0,
-		.max = 1,
-		.step = 1,
-	},
-	[FIM_CL_NUM] = {
-		.ops = &fim_ctrl_ops,
-		.id = V4L2_CID_IMX_FIM_NUM,
-		.name = "FIM Num Average",
-		.type = V4L2_CTRL_TYPE_INTEGER,
-		.def = FIM_CL_NUM_DEF,
-		.min =  1, /* no averaging */
-		.max = 64, /* average 64 frames */
-		.step = 1,
-	},
-	[FIM_CL_TOLERANCE_MIN] = {
-		.ops = &fim_ctrl_ops,
-		.id = V4L2_CID_IMX_FIM_TOLERANCE_MIN,
-		.name = "FIM Tolerance Min",
-		.type = V4L2_CTRL_TYPE_INTEGER,
-		.def = FIM_CL_TOLERANCE_MIN_DEF,
-		.min =    2,
-		.max =  200,
-		.step =   1,
-	},
-	[FIM_CL_TOLERANCE_MAX] = {
-		.ops = &fim_ctrl_ops,
-		.id = V4L2_CID_IMX_FIM_TOLERANCE_MAX,
-		.name = "FIM Tolerance Max",
-		.type = V4L2_CTRL_TYPE_INTEGER,
-		.def = FIM_CL_TOLERANCE_MAX_DEF,
-		.min =    0,
-		.max =  500,
-		.step =   1,
-	},
-	[FIM_CL_NUM_SKIP] = {
-		.ops = &fim_ctrl_ops,
-		.id = V4L2_CID_IMX_FIM_NUM_SKIP,
-		.name = "FIM Num Skip",
-		.type = V4L2_CTRL_TYPE_INTEGER,
-		.def = FIM_CL_NUM_SKIP_DEF,
-		.min =   0, /* skip no frames */
-		.max = 256, /* skip 256 frames */
-		.step =  1,
-	},
-};
-
 static const struct v4l2_ctrl_config fim_icap_ctrl[] = {
 	[FIM_CL_ICAP_EDGE] = {
 		.ops = &fim_ctrl_ops,
 		.id = V4L2_CID_IMX_FIM_ICAP_EDGE,
 		.name = "FIM Input Capture Edge",
 		.type = V4L2_CTRL_TYPE_INTEGER,
-		.def =  IRQ_TYPE_NONE, /* input capture disabled by default */
+		.def =  IRQ_TYPE_NONE,
 		.min =  IRQ_TYPE_NONE,
 		.max =  IRQ_TYPE_EDGE_BOTH,
 		.step = 1,
@@ -321,108 +113,66 @@ static const struct v4l2_ctrl_config fim_icap_ctrl[] = {
 	},
 };
 
-static int init_fim_controls(struct imx_media_fim *fim)
-{
-	struct v4l2_ctrl_handler *hdlr = &fim->ctrl_handler;
-	int i, ret;
-
-	v4l2_ctrl_handler_init(hdlr, FIM_NUM_CONTROLS + FIM_NUM_ICAP_CONTROLS);
-
-	for (i = 0; i < FIM_NUM_CONTROLS; i++)
-		fim->ctrl[i] = v4l2_ctrl_new_custom(hdlr,
-						    &fim_ctrl[i],
-						    NULL);
-	for (i = 0; i < FIM_NUM_ICAP_CONTROLS; i++)
-		fim->icap_ctrl[i] = v4l2_ctrl_new_custom(hdlr,
-							 &fim_icap_ctrl[i],
-							 NULL);
-	if (hdlr->error) {
-		ret = hdlr->error;
-		goto err_free;
-	}
-
-	v4l2_ctrl_cluster(FIM_NUM_CONTROLS, fim->ctrl);
-	v4l2_ctrl_cluster(FIM_NUM_ICAP_CONTROLS, fim->icap_ctrl);
-
-	return 0;
-err_free:
-	v4l2_ctrl_handler_free(hdlr);
-	return ret;
-}
-
-/*
- * Monitor frame intervals via EOF interrupt. This method is
- * subject to uncertainty errors introduced by interrupt latency.
- *
- * This is a noop if the Input Capture method is being used, since
- * the frame_interval_monitor() is called by the input capture event
- * callback handler in that case.
- */
 void imx_media_fim_eof_monitor(struct imx_media_fim *fim, ktime_t timestamp)
 {
-	unsigned long flags;
-
-	spin_lock_irqsave(&fim->lock, flags);
-
-	if (!icap_enabled(fim))
-		frame_interval_monitor(fim, timestamp);
-
-	spin_unlock_irqrestore(&fim->lock, flags);
+	v4l2_fim_eof_monitor(fim->core, timestamp);
 }
 
-/* Called by the subdev in its s_stream callback */
 void imx_media_fim_set_stream(struct imx_media_fim *fim,
-			      const struct v4l2_fract *fi,
+			      const struct v4l2_fract *frame_interval,
 			      bool on)
 {
-	unsigned long flags;
-
-	v4l2_ctrl_lock(fim->ctrl[FIM_CL_ENABLE]);
-
-	if (fim->stream_on == on)
-		goto out;
-
-	if (on) {
-		spin_lock_irqsave(&fim->lock, flags);
-		reset_fim(fim, true);
-		update_fim_nominal(fim, fi);
-		spin_unlock_irqrestore(&fim->lock, flags);
-
-		if (icap_enabled(fim))
-			fim_acquire_first_ts(fim);
-	}
-
-	fim->stream_on = on;
-out:
-	v4l2_ctrl_unlock(fim->ctrl[FIM_CL_ENABLE]);
+	v4l2_fim_set_stream(fim->core, frame_interval, on);
 }
 
 int imx_media_fim_add_controls(struct imx_media_fim *fim)
 {
-	/* add the FIM controls to the calling subdev ctrl handler */
 	return v4l2_ctrl_add_handler(fim->sd->ctrl_handler,
 				     &fim->ctrl_handler, NULL, true);
 }
 
-/* Called by the subdev in its subdev registered callback */
 struct imx_media_fim *imx_media_fim_init(struct v4l2_subdev *sd)
 {
+	struct v4l2_fim_config config = {
+		.ctrl_base = V4L2_CID_IMX_FIM_ENABLE,
+		.event_type = V4L2_EVENT_IMX_FRAME_INTERVAL_ERROR,
+		.external_timestamp = imx_fim_external_timestamp,
+		.wait_first_timestamp = imx_fim_wait_first_timestamp,
+	};
 	struct imx_media_fim *fim;
-	int ret;
+	int i, ret;
 
 	fim = devm_kzalloc(sd->dev, sizeof(*fim), GFP_KERNEL);
 	if (!fim)
 		return ERR_PTR(-ENOMEM);
 
 	fim->sd = sd;
+	config.priv = fim;
+
+	v4l2_ctrl_handler_init(&fim->ctrl_handler,
+			       V4L2_FIM_NUM_CONTROLS + FIM_NUM_ICAP_CONTROLS);
+
+	fim->core = v4l2_fim_init(sd, &fim->ctrl_handler, &config);
+	if (IS_ERR(fim->core)) {
+		ret = PTR_ERR(fim->core);
+		goto err_free;
+	}
 
-	spin_lock_init(&fim->lock);
+	for (i = 0; i < FIM_NUM_ICAP_CONTROLS; i++)
+		fim->icap_ctrl[i] = v4l2_ctrl_new_custom(&fim->ctrl_handler,
+							 &fim_icap_ctrl[i], fim);
+	if (fim->ctrl_handler.error) {
+		ret = fim->ctrl_handler.error;
+		goto err_free;
+	}
 
-	ret = init_fim_controls(fim);
-	if (ret)
-		return ERR_PTR(ret);
+	v4l2_ctrl_cluster(FIM_NUM_ICAP_CONTROLS, fim->icap_ctrl);
 
 	return fim;
+
+err_free:
+	v4l2_ctrl_handler_free(&fim->ctrl_handler);
+	return ERR_PTR(ret);
 }
 
 void imx_media_fim_free(struct imx_media_fim *fim)
diff --git a/include/media/v4l2-fim.h b/include/media/v4l2-fim.h
new file mode 100644
index 000000000000..4596d0492676
--- /dev/null
+++ b/include/media/v4l2-fim.h
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * V4L2 frame interval monitor helpers
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.
+ */
+
+#ifndef _V4L2_FIM_H
+#define _V4L2_FIM_H
+
+#include <linux/ktime.h>
+#include <linux/types.h>
+
+struct v4l2_ctrl_handler;
+struct v4l2_fract;
+struct v4l2_subdev;
+
+struct v4l2_fim;
+
+#define V4L2_FIM_NUM_CONTROLS	5
+
+struct v4l2_fim_config {
+	u32 ctrl_base;
+	u32 event_type;
+	void *priv;
+	bool (*external_timestamp)(void *priv, bool curval);
+	void (*wait_first_timestamp)(void *priv);
+};
+
+struct v4l2_fim *v4l2_fim_init(struct v4l2_subdev *sd,
+			       struct v4l2_ctrl_handler *hdlr,
+			       const struct v4l2_fim_config *config);
+void v4l2_fim_reset(struct v4l2_fim *fim, bool curval);
+void v4l2_fim_set_stream(struct v4l2_fim *fim,
+			 const struct v4l2_fract *frame_interval, bool on);
+void v4l2_fim_eof_monitor(struct v4l2_fim *fim, ktime_t timestamp);
+void v4l2_fim_monitor_timestamp(struct v4l2_fim *fim, ktime_t timestamp);
+bool v4l2_fim_is_streaming(struct v4l2_fim *fim);
+
+#endif
-- 
2.53.0


^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH 3/3] media: imx: allow setting FIM input capture channel
  2026-11-21 14:14 [PATCH 0/3] media: imx: staging cleanup and FIM export to v4l2-core Jiaqi Shi
  2026-11-21 14:14 ` [PATCH 1/3] media: imx: remove legacy video node control and format handling Jiaqi Shi
  2026-11-21 14:14 ` [PATCH 2/3] media: v4l2-core: add frame interval monitor helper Jiaqi Shi
@ 2026-11-21 14:14 ` Jiaqi Shi
  2026-11-21 14:24   ` sashiko-bot
  2 siblings, 1 reply; 7+ messages in thread
From: Jiaqi Shi @ 2026-11-21 14:14 UTC (permalink / raw)
  To: linux-media, imx, linux-staging
  Cc: slongerbeam, p.zabel, mchehab, gregkh, Frank.Li, kernel, festevam,
	Jiaqi Shi

fim_s_ctrl() rejected all controls except V4L2_CID_IMX_FIM_ICAP_EDGE
with -EINVAL, which made V4L2_CID_IMX_FIM_ICAP_CHANNEL read-only in
practice even though it is registered as a writable control.

Add an explicit case for ICAP_CHANNEL so it can be updated, and fix
the wait_for_completion_timeout() continuation style while here.

Signed-off-by: Jiaqi Shi <shijiaqi_develop@163.com>
---
 drivers/staging/media/imx/imx-media-fim.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-fim.c b/drivers/staging/media/imx/imx-media-fim.c
index b6467e29ac28..300635a3b552 100644
--- a/drivers/staging/media/imx/imx-media-fim.c
+++ b/drivers/staging/media/imx/imx-media-fim.c
@@ -58,9 +58,8 @@ static void imx_fim_wait_first_timestamp(void *priv)
 	struct imx_media_fim *fim = priv;
 	unsigned long ret;
 
-	ret = wait_for_completion_timeout(
-			&fim->icap_first_event,
-			msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+	ret = wait_for_completion_timeout(&fim->icap_first_event,
+					  msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
 	if (ret == 0)
 		v4l2_warn(fim->sd, "wait first icap event timeout\n");
 }
@@ -75,6 +74,8 @@ static int fim_s_ctrl(struct v4l2_ctrl *ctrl)
 		if (v4l2_fim_is_streaming(fim->core))
 			ret = -EBUSY;
 		break;
+	case V4L2_CID_IMX_FIM_ICAP_CHANNEL:
+		break;
 	default:
 		ret = -EINVAL;
 		break;
-- 
2.53.0


^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [PATCH 3/3] media: imx: allow setting FIM input capture channel
  2026-11-21 14:14 ` [PATCH 3/3] media: imx: allow setting FIM input capture channel Jiaqi Shi
@ 2026-11-21 14:24   ` sashiko-bot
  0 siblings, 0 replies; 7+ messages in thread
From: sashiko-bot @ 2026-11-21 14:24 UTC (permalink / raw)
  To: Jiaqi Shi, imx; +Cc: Frank.Li, media-ci

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] media: imx: unreachable ICAP_CHANNEL case in fim_s_ctrl() due to
  clustering
--

commit 3f6e136c4d9e74ffb8117949ce551b405c749b3a
Author: Jiaqi Shi <shijiaqi_develop@163.com>

media: imx: allow setting FIM input capture channel

This commit adds a V4L2_CID_IMX_FIM_ICAP_CHANNEL case to fim_s_ctrl() and
reformats a wait_for_completion_timeout() call.

> diff --git a/drivers/staging/media/imx/imx-media-fim.c b/drivers/staging/media/imx/imx-media-fim.c
> index b6467e29ac288..300635a3b5527 100644
> --- a/drivers/staging/media/imx/imx-media-fim.c
> +++ b/drivers/staging/media/imx/imx-media-fim.c
[ ... ]
> @@ -75,6 +74,8 @@ static int fim_s_ctrl(struct v4l2_ctrl *ctrl)
>  		if (v4l2_fim_is_streaming(fim->core))
>  			ret = -EBUSY;
>  		break;
> +	case V4L2_CID_IMX_FIM_ICAP_CHANNEL:
> +		break;
>  	default:
>  		ret = -EINVAL;
>  		break;

[Severity: Low]
Is this new V4L2_CID_IMX_FIM_ICAP_CHANNEL case reachable?

imx_media_fim_init() clusters fim->icap_ctrl[] via
v4l2_ctrl_cluster(FIM_NUM_ICAP_CONTROLS, fim->icap_ctrl) with
V4L2_CID_IMX_FIM_ICAP_EDGE at index 0 as the cluster master.  When
userspace updates any control in a cluster, try_or_set_cluster() always
invokes s_ctrl on ctrl->cluster[0], so fim_s_ctrl() is always called with
ctrl->id == V4L2_CID_IMX_FIM_ICAP_EDGE.  Because v4l2_fim_reset() then
reads both icap_edge->val and icap_chan->val in
imx_fim_external_timestamp(), setting V4L2_CID_IMX_FIM_ICAP_CHANNEL was
not rejected with -EINVAL, and this new case is dead code that also omits
the v4l2_fim_is_streaming(fim->core) check.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261121141412.319802-1-shijiaqi_develop@163.com?part=3

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH 2/3] media: v4l2-core: add frame interval monitor helper
  2026-11-21 14:14 ` [PATCH 2/3] media: v4l2-core: add frame interval monitor helper Jiaqi Shi
@ 2026-11-21 14:27   ` sashiko-bot
  0 siblings, 0 replies; 7+ messages in thread
From: sashiko-bot @ 2026-11-21 14:27 UTC (permalink / raw)
  To: Jiaqi Shi, imx; +Cc: Frank.Li, media-ci

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] media: v4l2-fim: cross-cluster ctrl->val read in reset_fim()
--

commit 4ee73622d4ea40f9f5507b8b2948de21bfddfccc
Author: Jiaqi Shi <shijiaqi_develop@163.com>

media: v4l2-core: add frame interval monitor helper

This commit extracts the frame interval monitoring helper from the i.MX
staging driver into v4l2-core while keeping the input capture controls in
imx-media-fim.c.

> diff --git a/drivers/media/v4l2-core/v4l2-fim.c b/drivers/media/v4l2-core/v4l2-fim.c
> new file mode 100644
> index 0000000000000..57d595196663b
> --- /dev/null
> +++ b/drivers/media/v4l2-core/v4l2-fim.c
> @@ -0,0 +1,324 @@

[ ... ]

> +static void reset_fim(struct v4l2_fim *fim, bool curval)
> +{
> +	struct v4l2_ctrl *en = fim->ctrl[FIM_CL_ENABLE];
> +	struct v4l2_ctrl *num = fim->ctrl[FIM_CL_NUM];
> +	struct v4l2_ctrl *skip = fim->ctrl[FIM_CL_NUM_SKIP];
> +	struct v4l2_ctrl *tol_min = fim->ctrl[FIM_CL_TOLERANCE_MIN];
> +	struct v4l2_ctrl *tol_max = fim->ctrl[FIM_CL_TOLERANCE_MAX];
> +
> +	if (fim->config.external_timestamp)
> +		fim->external_timestamp =
> +			fim->config.external_timestamp(fim->config.priv, curval);
> +
> +	if (curval) {
> +		fim->enabled = en->cur.val;
> +		fim->num_avg = num->cur.val;
> +		fim->num_skip = skip->cur.val;
> +		fim->tolerance_min = tol_min->cur.val;
> +		fim->tolerance_max = tol_max->cur.val;
> +	} else {
> +		fim->enabled = en->val;
> +		fim->num_avg = num->val;
> +		fim->num_skip = skip->val;
> +		fim->tolerance_min = tol_min->val;
> +		fim->tolerance_max = tol_max->val;
> +	}

[Severity: Medium]
Can reset_fim() and fim->config.external_timestamp() read uncommitted
ctrl->val values across separate control clusters when curval is false?

v4l2_fim_init() clusters the five core FIM controls into fim->ctrl, while
imx_media_fim_init() clusters the two input capture controls separately
into fim->icap_ctrl.  The V4L2 control framework only synchronizes
ctrl->val with ctrl->cur.val for the cluster currently being tried or set,
and leaves trial or rejected values in ctrl->val after VIDIOC_TRY_EXT_CTRLS
or a failed s_ctrl.

If setting V4L2_CID_IMX_FIM_ICAP_EDGE while streaming fails with -EBUSY
(or is tried via VIDIOC_TRY_EXT_CTRLS), icap_edge->val retains the
uncommitted value.  A subsequent update to any core FIM control calls
fim_s_ctrl() -> reset_fim(fim, false) ->
imx_fim_external_timestamp(..., false), which reads icap_edge->val and
enables external_timestamp mode during active streaming, disabling EOF
monitoring.  Conversely, updating fim->icap_ctrl calls
v4l2_fim_reset(fim->core, false), causing reset_fim() to read uncommitted
->val fields from the core fim->ctrl cluster.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261121141412.319802-1-shijiaqi_develop@163.com?part=2

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH 1/3] media: imx: remove legacy video node control and format handling
  2026-11-21 14:14 ` [PATCH 1/3] media: imx: remove legacy video node control and format handling Jiaqi Shi
@ 2026-11-21 14:40   ` sashiko-bot
  0 siblings, 0 replies; 7+ messages in thread
From: sashiko-bot @ 2026-11-21 14:40 UTC (permalink / raw)
  To: Jiaqi Shi, imx; +Cc: Frank.Li, media-ci

Thank you for your contribution! Sashiko AI review found 5 potential
issues to consider:
- [High] media: imx: unresolved default colorimetry and field in
  capture_init_format()
- [High] media: imx: insufficient pixel format validation in
  capture_validate_fmt()
- [High] media: imx: unclamped dimensions in __capture_try_fmt() cause
  divide-by-zero
- [Medium] media: imx: callers of imx_media_capture_device_init() still
  set legacy_api
- [Low] media: imx: outdated v4l2-ctl format examples in
  Documentation/admin-guide
--

commit 91db31570bbb5c6e57f4f438a93a25e73be6926a
Author: Jiaqi Shi <shijiaqi_develop@163.com>

media: imx: remove legacy video node control and format handling

This commit removes V4L2 control inheritance from pipeline subdevices to
capture video nodes and switches capture video nodes to source-independent
format initialization and format ioctls.

> diff --git a/Documentation/admin-guide/media/imx.rst b/Documentation/admin-guide/media/imx.rst
> index bb68100d8acba..b1c5fecfaef74 100644
> --- a/Documentation/admin-guide/media/imx.rst
> +++ b/Documentation/admin-guide/media/imx.rst
> @@ -394,13 +394,9 @@ scaling up to 1024x1024, CSC, flip, and rotation:
>  Usage Notes
>  -----------
>  
> -To aid in configuration and for backward compatibility with V4L2
> -applications that access controls only from video device nodes, the
> -capture device interfaces inherit controls from the active entities
> -in the current pipeline, so controls can be accessed either directly
> -from the subdev or from the active capture device interface. For
> -example, the FIM controls are available either from the ipuX_csiY
> -subdevs or from the active capture device.
> +Controls for pipeline entities are accessed through their subdev device
> +nodes. For example, the FIM controls are available from the ipuX_csiY
> +subdevs.

[Severity: Low]
Should the v4l2-ctl --set-fmt-video examples later in this document also
be updated to specify width and height when the upstream subdev pad is not
640x480?

Because capture nodes now initialize to 640x480 and VIDIOC_S_FMT no longer
inherits dimensions from the source subdev, the 720x480, 720x576, and
800x600 pipeline examples that only pass field= or pixelformat=RGB3 leave
the capture node at 640x480 and fail in capture_validate_fmt() with -EPIPE
at stream start.

[ ... ]

> diff --git a/drivers/staging/media/imx/imx-media-capture.c b/drivers/staging/media/imx/imx-media-capture.c
> index 9216a221b72f4..93d05cd967980 100644
> --- a/drivers/staging/media/imx/imx-media-capture.c
> +++ b/drivers/staging/media/imx/imx-media-capture.c

[ ... ]

> @@ -566,10 +432,10 @@ static const struct v4l2_ioctl_ops capture_legacy_ioctl_ops = {
>  	.vidioc_enum_framesizes		= capture_legacy_enum_framesizes,
>  	.vidioc_enum_frameintervals	= capture_legacy_enum_frameintervals,
>  
> -	.vidioc_enum_fmt_vid_cap	= capture_legacy_enum_fmt_vid_cap,
> +	.vidioc_enum_fmt_vid_cap	= capture_enum_fmt_vid_cap,
>  	.vidioc_g_fmt_vid_cap		= capture_g_fmt_vid_cap,
> -	.vidioc_try_fmt_vid_cap		= capture_legacy_try_fmt_vid_cap,
> -	.vidioc_s_fmt_vid_cap		= capture_legacy_s_fmt_vid_cap,
> +	.vidioc_try_fmt_vid_cap		= capture_try_fmt_vid_cap,
> +	.vidioc_s_fmt_vid_cap		= capture_s_fmt_vid_cap,

[Severity: High]
Does capture_validate_fmt() allow incompatible pixel formats on the video
node now that capture_s_fmt_vid_cap() accepts any PIXFMT_SEL_ANY format?

Previously, capture_legacy_s_fmt_vid_cap() called capture_find_format()
with both fmt_src->format.code and pixfmt->pixelformat, which rejected
Bayer formats on IPU internal links and required an exact format match on
non-IPU passthrough links.  Now capture_validate_fmt() is the only check,
and it only compares the two-value colorspace enum:

drivers/staging/media/imx/imx-media-capture.c:capture_validate_fmt() {
    ...
	cc = capture_find_format(fmt_src.format.code, 0);
	if (!cc || priv->vdev.cc->cs != cc->cs)
		return -EPIPE;
    ...
}

Because all RGB, Bayer, and greyscale formats share IPUV3_COLORSPACE_RGB
and all YUV formats share IPUV3_COLORSPACE_YUV, mismatched formats pass
validation.  On a CSI passthrough link, configuring an 8-bpp or 12-bpp
format on the video node while the source subdev outputs a 16-bpp format
with the same colorspace programs 16-bpp IDMAC DMA transfers into buffers
sized for 8 or 12 bpp, overflowing the DMA buffer.  On an IPU internal RGB
link, V4L2_PIX_FMT_Y10 or V4L2_PIX_FMT_Y12 reaches the default WARN_ON(1)
in ipu_cpmem_set_image(), while 8-bpp Bayer or greyscale formats cause
ipu_cpmem_set_fmt() to fail with its return value ignored, leaving the
CPMEM BPP field at 0 (32 bpp) for 8-bpp buffers.

Passing priv->vdev.fmt.pixelformat instead of 0 to capture_find_format()
in capture_validate_fmt() would restore the format check.

[Severity: High]
Can VIDIOC_S_FMT followed by VIDIOC_REQBUFS trigger a divide-by-zero in
capture_queue_setup()?

capture_legacy_s_fmt_vid_cap() previously took width and height from the
source subdev, where dimensions were already clamped to non-zero hardware
bounds.  capture_s_fmt_vid_cap() instead passes the user's unclamped
width and height through __capture_try_fmt() to
imx_media_mbus_fmt_to_pix_fmt(), so width = 0 or height = 0 sets
priv->vdev.fmt.sizeimage to 0.  A subsequent VIDIOC_REQBUFS or
VIDIOC_CREATE_BUFS then divides by pix->sizeimage:

drivers/staging/media/imx/imx-media-capture.c:capture_queue_setup() {
    ...
	count = min_t(__u32, VID_MEM_LIMIT / pix->sizeimage, count);
    ...
}

[ ... ]

> @@ -869,36 +735,21 @@ void imx_media_capture_device_error(struct imx_media_video_dev *vdev)
>  }
>  EXPORT_SYMBOL_GPL(imx_media_capture_device_error);
>  
> -static int capture_init_format(struct capture_priv *priv)
> +static void capture_init_format(struct capture_priv *priv)
>  {
> -	struct v4l2_subdev_format fmt_src = {
> -		.pad = priv->src_sd_pad,
> -		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
> -	};
>  	struct imx_media_video_dev *vdev = &priv->vdev;
> -	int ret;
> -
> -	if (priv->legacy_api) {
> -		ret = v4l2_subdev_call(priv->src_sd, pad, get_fmt, NULL,
> -				       &fmt_src);
> -		if (ret) {
> -			dev_err(priv->dev, "failed to get source format\n");
> -			return ret;
> -		}
> -	} else {
> -		fmt_src.format.code = MEDIA_BUS_FMT_UYVY8_2X8;
> -		fmt_src.format.width = IMX_MEDIA_DEF_PIX_WIDTH;
> -		fmt_src.format.height = IMX_MEDIA_DEF_PIX_HEIGHT;
> -	}
> +	struct v4l2_mbus_framefmt fmt_src = {
> +		.code = MEDIA_BUS_FMT_UYVY8_2X8,
> +		.width = IMX_MEDIA_DEF_PIX_WIDTH,
> +		.height = IMX_MEDIA_DEF_PIX_HEIGHT,
> +	};
>  
> -	imx_media_mbus_fmt_to_pix_fmt(&vdev->fmt, &fmt_src.format, NULL);
> -	vdev->compose.width = fmt_src.format.width;
> -	vdev->compose.height = fmt_src.format.height;
> +	imx_media_mbus_fmt_to_pix_fmt(&vdev->fmt, &fmt_src, NULL);
> +	vdev->compose.width = fmt_src.width;
> +	vdev->compose.height = fmt_src.height;
>  
>  	vdev->cc = imx_media_find_pixel_format(vdev->fmt.pixelformat,
>  					       PIXFMT_SEL_ANY);
> -
> -	return 0;
>  }

[Severity: High]
Does leaving fmt_src.field and the colorimetry members zero-initialized in
capture_init_format() and __capture_try_fmt() break IC color space
conversion at stream start?

Previously, vdev->fmt was initialized and updated from the source subdev's
active format, which imx_media_init_mbus_fmt() and
imx_media_try_colorimetry() populated with V4L2_FIELD_NONE and resolved
colorimetry values (such as V4L2_YCBCR_ENC_601 and
V4L2_QUANTIZATION_LIM_RANGE).  Now fmt_src leaves field as V4L2_FIELD_ANY
(0) and ycbcr_enc and quantization as DEFAULT (0), and __capture_try_fmt()
also passes zeroed values through without resolving them.

When streaming starts on an IC capture node (ipuX_ic_prpenc or
ipuX_ic_prpvf), prp_setup_norotation() and prp_setup_rotation() pass
vdev->fmt.ycbcr_enc and vdev->fmt.quantization to ipu_ic_calc_csc() ->
calc_csc_coeffs(), which rejects V4L2_YCBCR_ENC_DEFAULT with -ENOTSUPP
whenever YUV-to-RGB or RGB-to-YUV conversion is needed:

drivers/gpu/ipu-v3/ipu-ic-csc.c:calc_csc_coeffs() {
    ...
	switch (csc->out_cs.enc) {
	case V4L2_YCBCR_ENC_601:
		params_tbl = (csc->in_cs.cs == IPUV3_COLORSPACE_YUV) ?
			yuv2rgb_601 : rgb2yuv_601;
		break;
	case V4L2_YCBCR_ENC_709:
		params_tbl = (csc->in_cs.cs == IPUV3_COLORSPACE_YUV) ?
			yuv2rgb_709 : rgb2yuv_709;
		break;
	default:
		return -ENOTSUPP;
	}
    ...
}

On default YUV-to-YUV IC captures, V4L2_QUANTIZATION_DEFAULT also maps to
full range instead of limited range, programming the yuvl2yuvf range
expansion CSC matrix instead of an identity pass-through.

[ ... ]

> @@ -974,7 +823,6 @@ imx_media_capture_device_init(struct device *dev, struct v4l2_subdev *src_sd,
>  	priv->src_sd = src_sd;
>  	priv->src_sd_pad = pad;
>  	priv->dev = dev;
> -	priv->legacy_api = legacy_api;

[Severity: Medium]
Should prp_registered() and csi_registered() stop setting legacy_api = true
(or should legacy_api be removed altogether)?

Both callers of imx_media_capture_device_init() still pass legacy_api =
true, which leaves V4L2_CAP_IO_MC unset on every capture device, keeps
capture_ioctl_ops unused, and keeps capture_legacy_enum_framesizes() active.
Without V4L2_CAP_IO_MC, v4l_enum_fmt() in the V4L2 core zeroes
f->mbus_code before calling capture_enum_fmt_vid_cap(), and on
ipuX_ic_prpenc/prpvf capture nodes capture_legacy_enum_framesizes() passes
non-IPU mbus codes to prp_enum_frame_size(), which rejects 22 of the 23
enumerated pixel formats with -EINVAL.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261121141412.319802-1-shijiaqi_develop@163.com?part=1

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2026-11-21 14:40 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-21 14:14 [PATCH 0/3] media: imx: staging cleanup and FIM export to v4l2-core Jiaqi Shi
2026-11-21 14:14 ` [PATCH 1/3] media: imx: remove legacy video node control and format handling Jiaqi Shi
2026-11-21 14:40   ` sashiko-bot
2026-11-21 14:14 ` [PATCH 2/3] media: v4l2-core: add frame interval monitor helper Jiaqi Shi
2026-11-21 14:27   ` sashiko-bot
2026-11-21 14:14 ` [PATCH 3/3] media: imx: allow setting FIM input capture channel Jiaqi Shi
2026-11-21 14:24   ` sashiko-bot

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.