[PATCH v2 00/10] media: Introduce video device state management

Jai Luthra
2025-09-19T09:57:07Z

Hi,

This patch series introduces state management for video devices.

Currently, V4L2 subdevices have a well-established state management
system [1] that allows the framework to store the subdevice state
at a central location.

Video devices however lack this, leading to inconsistent state handling
across drivers and making it difficult to implement features like
hardware multiplexing in complex media pipelines [2].

The series is composed of three parts:

- Core Infrastructure (PATCH 1-4)
Introduces the basic state structure, try state support,
initialization callbacks, and driver helpers
- Framework Integration (PATCH 5-7)
Integrates state passing through the ioctl wrappers and driver
implementations
- Driver Examples (PATCH 8-10)
Use the state to store formats in TI J721E CSI2RX and Rockchip
RKISP1 drivers

This should also provide a foundation for drivers to extend or subclass
the state structure with device-specific variables in future.

I have tested capture and format negotiation with the converted drivers
on SK AM62A (j721e-csi2rx) and Debix Model A (rkisp1).

[1]: https://lore.kernel.org/linux-media/20210610145606.3468235-1-tomi.valkeinen@ideasonboard.com/
[2]: https://lore.kernel.org/linux-media/20250724-multicontext-mainline-2025-v2-0-c9b316773486@ideasonboard.com/

Signed-off-by: Jai Luthra <jai.luthra@ideasonboard.com>
---
Changes in v2:
- Drop RFC tag
- Rebase on top of latest media tree, that includes multiple changes by
Jacopo and Laurent, which removes all usage of passing v4l2_fh through
the opaque pointer in ioctl implementations, replacing it with a NULL
pointer
- Allocate the state dynamically, and free it when done with the device
or the filehandle
- Introduce an init_state callback so that drivers can initialize the
state after allocation
- Pass the video device state (as an explicitly typed pointer) to all
ioctl implemenation across all drivers
- Split the single patch introducing video device state into separate
patches each making a single logical change before the state is
actually passed to the drivers in PATCH 7
- In drivers, use a helper to access the current format from the state
- Move the queue busy check back to the drivers, as some codecs and VBI
devices may support changing formats, so the framework shouldn't
return -EBUSY
- Drop pixel format info and configuration storage from rkisp1 internal
strucutre, and calculate them on the fly
- Fix other review comments from Sakari, Laurent and Jacopo, except
those around dropping the enum and using try_fmt hook to update only
the state, and s_fmt hook to update hardware registers
- Link to v1: https://lore.kernel.org/r/20250703-vdev-state-v1-0-d647a5e4986d@ideasonboard.com

---
Jai Luthra (10):
media: v4l2-core: Introduce state management for video devices
media: v4l2-dev: Add support for try state
media: v4l2-dev: Add callback for initializing video device state
media: v4l2-dev: Add helpers to get current format from the state
media: v4l2-ioctl: Add video_device_state argument to v4l2 ioctl wrappers
media: Replace void * with video_device_state * in all driver ioctl implementations
media: v4l2-ioctl: Pass device state for G/S/TRY_FMT ioctls
media: ti: j721e-csi2rx: Use video_device_state
media: rkisp1: Use video_device_state
media: rkisp1: Calculate format information on demand

drivers/input/rmi4/rmi_f54.c | 22 +-
drivers/input/touchscreen/atmel_mxt_ts.c | 23 +-
drivers/input/touchscreen/sur40.c | 35 +-
.../extron-da-hd-4k-plus/extron-da-hd-4k-plus.c | 34 +-
drivers/media/common/saa7146/saa7146_video.c | 37 +-
drivers/media/common/videobuf2/videobuf2-v4l2.c | 30 +-
drivers/media/dvb-frontends/rtl2832_sdr.c | 38 +-
drivers/media/i2c/video-i2c.c | 39 ++-
drivers/media/pci/bt8xx/bttv-driver.c | 84 +++--
drivers/media/pci/bt8xx/bttv-vbi.c | 11 +-
drivers/media/pci/bt8xx/bttvp.h | 9 +-
drivers/media/pci/cobalt/cobalt-v4l2.c | 98 +++---
drivers/media/pci/cx18/cx18-driver.c | 9 +-
drivers/media/pci/cx18/cx18-ioctl.c | 114 +++---
drivers/media/pci/cx18/cx18-ioctl.h | 9 +-
drivers/media/pci/cx23885/cx23885-417.c | 53 +--
drivers/media/pci/cx23885/cx23885-ioctl.c | 8 +-
drivers/media/pci/cx23885/cx23885-ioctl.h | 9 +-
drivers/media/pci/cx23885/cx23885-vbi.c | 5 +-
drivers/media/pci/cx23885/cx23885-video.c | 82 +++--
drivers/media/pci/cx23885/cx23885.h | 4 +-
drivers/media/pci/cx25821/cx25821-video.c | 57 ++-
drivers/media/pci/cx88/cx88-blackbird.c | 47 ++-
drivers/media/pci/cx88/cx88-vbi.c | 2 +-
drivers/media/pci/cx88/cx88-video.c | 56 ++-
drivers/media/pci/cx88/cx88.h | 2 +-
drivers/media/pci/dt3155/dt3155.c | 25 +-
drivers/media/pci/intel/ipu3/ipu3-cio2.c | 29 +-
drivers/media/pci/intel/ipu6/ipu6-isys-video.c | 41 ++-
drivers/media/pci/ivtv/ivtv-driver.c | 8 +-
drivers/media/pci/ivtv/ivtv-ioctl.c | 225 ++++++++----
drivers/media/pci/ivtv/ivtv-ioctl.h | 6 +-
drivers/media/pci/mgb4/mgb4_vin.c | 55 ++-
drivers/media/pci/mgb4/mgb4_vout.c | 49 ++-
drivers/media/pci/saa7134/saa7134-empress.c | 12 +-
drivers/media/pci/saa7134/saa7134-video.c | 96 ++++--
drivers/media/pci/saa7134/saa7134.h | 38 +-
drivers/media/pci/saa7146/hexium_gemini.c | 10 +-
drivers/media/pci/saa7146/hexium_orion.c | 10 +-
drivers/media/pci/saa7146/mxb.c | 46 ++-
drivers/media/pci/saa7164/saa7164-encoder.c | 39 ++-
drivers/media/pci/saa7164/saa7164-vbi.c | 27 +-
drivers/media/pci/saa7164/saa7164.h | 9 +-
drivers/media/pci/solo6x10/solo6x10-v4l2-enc.c | 46 ++-
drivers/media/pci/solo6x10/solo6x10-v4l2.c | 32 +-
drivers/media/pci/ttpci/budget-av.c | 10 +-
drivers/media/pci/tw5864/tw5864-video.c | 49 ++-
drivers/media/pci/tw68/tw68-video.c | 43 ++-
drivers/media/pci/tw686x/tw686x-video.c | 53 ++-
drivers/media/pci/zoran/zoran_driver.c | 44 ++-
drivers/media/platform/allegro-dvt/allegro-core.c | 49 ++-
.../media/platform/amlogic/c3/isp/c3-isp-capture.c | 18 +-
.../media/platform/amlogic/c3/isp/c3-isp-params.c | 9 +-
.../media/platform/amlogic/c3/isp/c3-isp-stats.c | 9 +-
drivers/media/platform/amlogic/meson-ge2d/ge2d.c | 36 +-
drivers/media/platform/amphion/vdec.c | 30 +-
drivers/media/platform/amphion/venc.c | 49 ++-
drivers/media/platform/aspeed/aspeed-video.c | 50 ++-
drivers/media/platform/atmel/atmel-isi.c | 36 +-
drivers/media/platform/broadcom/bcm2835-unicam.c | 33 +-
.../media/platform/chips-media/coda/coda-common.c | 69 ++--
.../platform/chips-media/wave5/wave5-helper.c | 2 +-
.../platform/chips-media/wave5/wave5-helper.h | 3 +-
.../platform/chips-media/wave5/wave5-vpu-dec.c | 54 ++-
.../platform/chips-media/wave5/wave5-vpu-enc.c | 62 +++-
.../media/platform/imagination/e5010-jpeg-enc.c | 34 +-
drivers/media/platform/intel/pxa_camera.c | 34 +-
drivers/media/platform/m2m-deinterlace.c | 34 +-
drivers/media/platform/marvell/mcam-core.c | 52 +--
.../media/platform/mediatek/jpeg/mtk_jpeg_core.c | 44 ++-
drivers/media/platform/mediatek/mdp/mtk_mdp_m2m.c | 34 +-
.../media/platform/mediatek/mdp3/mtk-mdp3-m2m.c | 25 +-
.../mediatek/vcodec/decoder/mtk_vcodec_dec.c | 52 ++-
.../mediatek/vcodec/encoder/mtk_vcodec_enc.c | 50 ++-
.../media/platform/microchip/microchip-isc-base.c | 33 +-
drivers/media/platform/nuvoton/npcm-video.c | 43 ++-
drivers/media/platform/nvidia/tegra-vde/v4l2.c | 38 +-
drivers/media/platform/nxp/dw100/dw100.c | 35 +-
drivers/media/platform/nxp/imx-jpeg/mxc-jpeg.c | 52 ++-
drivers/media/platform/nxp/imx-pxp.c | 34 +-
drivers/media/platform/nxp/imx7-media-csi.c | 21 +-
drivers/media/platform/nxp/imx8-isi/imx8-isi-m2m.c | 15 +-
.../media/platform/nxp/imx8-isi/imx8-isi-video.c | 18 +-
drivers/media/platform/nxp/mx2_emmaprp.c | 31 +-
drivers/media/platform/qcom/camss/camss-video.c | 27 +-
drivers/media/platform/qcom/iris/iris_vidc.c | 48 ++-
drivers/media/platform/qcom/venus/vdec.c | 29 +-
drivers/media/platform/qcom/venus/venc.c | 38 +-
.../media/platform/raspberrypi/pisp_be/pisp_be.c | 42 ++-
drivers/media/platform/raspberrypi/rp1-cfe/cfe.c | 40 ++-
.../media/platform/renesas/rcar-vin/rcar-v4l2.c | 21 +-
drivers/media/platform/renesas/rcar_drif.c | 30 +-
drivers/media/platform/renesas/rcar_fdp1.c | 18 +-
drivers/media/platform/renesas/rcar_jpu.c | 21 +-
drivers/media/platform/renesas/renesas-ceu.c | 36 +-
.../media/platform/renesas/rzg2l-cru/rzg2l-video.c | 18 +-
drivers/media/platform/renesas/sh_vou.c | 41 ++-
drivers/media/platform/renesas/vsp1/vsp1_histo.c | 9 +-
drivers/media/platform/renesas/vsp1/vsp1_video.c | 18 +-
drivers/media/platform/rockchip/rga/rga.c | 24 +-
.../platform/rockchip/rkisp1/rkisp1-capture.c | 289 ++++++++--------
.../media/platform/rockchip/rkisp1/rkisp1-common.h | 10 -
.../media/platform/rockchip/rkisp1/rkisp1-params.c | 15 +-
.../media/platform/rockchip/rkisp1/rkisp1-stats.c | 9 +-
drivers/media/platform/rockchip/rkvdec/rkvdec.c | 34 +-
.../media/platform/samsung/exynos-gsc/gsc-m2m.c | 44 ++-
.../platform/samsung/exynos4-is/fimc-capture.c | 45 ++-
.../platform/samsung/exynos4-is/fimc-isp-video.c | 32 +-
.../media/platform/samsung/exynos4-is/fimc-lite.c | 36 +-
.../media/platform/samsung/exynos4-is/fimc-m2m.c | 21 +-
.../platform/samsung/s3c-camif/camif-capture.c | 54 ++-
drivers/media/platform/samsung/s5p-g2d/g2d.c | 26 +-
.../media/platform/samsung/s5p-jpeg/jpeg-core.c | 34 +-
.../media/platform/samsung/s5p-mfc/s5p_mfc_dec.c | 48 ++-
.../media/platform/samsung/s5p-mfc/s5p_mfc_enc.c | 51 ++-
drivers/media/platform/st/sti/bdisp/bdisp-v4l2.c | 26 +-
drivers/media/platform/st/sti/delta/delta-v4l2.c | 44 ++-
drivers/media/platform/st/sti/hva/hva-v4l2.c | 46 ++-
drivers/media/platform/st/stm32/dma2d/dma2d.c | 18 +-
drivers/media/platform/st/stm32/stm32-dcmi.c | 42 ++-
.../st/stm32/stm32-dcmipp/dcmipp-bytecap.c | 20 +-
.../media/platform/sunxi/sun4i-csi/sun4i_v4l2.c | 25 +-
.../platform/sunxi/sun6i-csi/sun6i_csi_capture.c | 24 +-
drivers/media/platform/sunxi/sun8i-di/sun8i-di.c | 31 +-
.../platform/sunxi/sun8i-rotate/sun8i_rotate.c | 34 +-
.../media/platform/synopsys/hdmirx/snps_hdmirx.c | 47 ++-
drivers/media/platform/ti/am437x/am437x-vpfe.c | 52 ++-
drivers/media/platform/ti/cal/cal-video.c | 54 ++-
drivers/media/platform/ti/davinci/vpif_capture.c | 94 ++---
drivers/media/platform/ti/davinci/vpif_display.c | 58 ++--
.../media/platform/ti/j721e-csi2rx/j721e-csi2rx.c | 127 +++----
drivers/media/platform/ti/omap/omap_vout.c | 49 ++-
drivers/media/platform/ti/omap3isp/ispvideo.c | 53 ++-
drivers/media/platform/ti/vpe/vpe.c | 25 +-
drivers/media/platform/verisilicon/hantro_v4l2.c | 47 ++-
drivers/media/platform/via/via-camera.c | 42 ++-
drivers/media/platform/xilinx/xilinx-dma.c | 15 +-
drivers/media/radio/dsbr100.c | 15 +-
drivers/media/radio/radio-cadet.c | 18 +-
drivers/media/radio/radio-isa.c | 18 +-
drivers/media/radio/radio-keene.c | 15 +-
drivers/media/radio/radio-ma901.c | 15 +-
drivers/media/radio/radio-miropcm20.c | 15 +-
drivers/media/radio/radio-mr800.c | 18 +-
drivers/media/radio/radio-raremono.c | 19 +-
drivers/media/radio/radio-sf16fmi.c | 15 +-
drivers/media/radio/radio-si476x.c | 28 +-
drivers/media/radio/radio-tea5764.c | 15 +-
drivers/media/radio/radio-tea5777.c | 21 +-
drivers/media/radio/radio-timb.c | 26 +-
drivers/media/radio/si470x/radio-si470x-common.c | 25 +-
drivers/media/radio/si4713/radio-platform-si4713.c | 18 +-
drivers/media/radio/si4713/radio-usb-si4713.c | 15 +-
drivers/media/radio/tea575x.c | 21 +-
drivers/media/test-drivers/vicodec/vicodec-core.c | 54 +--
drivers/media/test-drivers/vim2m.c | 56 +--
drivers/media/test-drivers/vimc/vimc-capture.c | 20 +-
drivers/media/test-drivers/visl/visl-video.c | 34 +-
drivers/media/test-drivers/vivid/vivid-core.c | 203 ++++++-----
drivers/media/test-drivers/vivid/vivid-meta-cap.c | 6 +-
drivers/media/test-drivers/vivid/vivid-meta-cap.h | 6 +-
drivers/media/test-drivers/vivid/vivid-meta-out.c | 6 +-
drivers/media/test-drivers/vivid/vivid-meta-out.h | 9 +-
drivers/media/test-drivers/vivid/vivid-radio-rx.c | 16 +-
drivers/media/test-drivers/vivid/vivid-radio-rx.h | 16 +-
drivers/media/test-drivers/vivid/vivid-radio-tx.c | 6 +-
drivers/media/test-drivers/vivid/vivid-radio-tx.h | 6 +-
drivers/media/test-drivers/vivid/vivid-sdr-cap.c | 30 +-
drivers/media/test-drivers/vivid/vivid-sdr-cap.h | 30 +-
drivers/media/test-drivers/vivid/vivid-touch-cap.c | 22 +-
drivers/media/test-drivers/vivid/vivid-touch-cap.h | 22 +-
drivers/media/test-drivers/vivid/vivid-vbi-cap.c | 26 +-
drivers/media/test-drivers/vivid/vivid-vbi-cap.h | 22 +-
drivers/media/test-drivers/vivid/vivid-vbi-out.c | 22 +-
drivers/media/test-drivers/vivid/vivid-vbi-out.h | 18 +-
drivers/media/test-drivers/vivid/vivid-vid-cap.c | 114 +++---
drivers/media/test-drivers/vivid/vivid-vid-cap.h | 117 +++++--
.../media/test-drivers/vivid/vivid-vid-common.c | 24 +-
.../media/test-drivers/vivid/vivid-vid-common.h | 34 +-
drivers/media/test-drivers/vivid/vivid-vid-out.c | 122 ++++---
drivers/media/test-drivers/vivid/vivid-vid-out.h | 95 +++--
drivers/media/usb/airspy/airspy.c | 37 +-
drivers/media/usb/au0828/au0828-video.c | 74 ++--
drivers/media/usb/cx231xx/cx231xx-417.c | 28 +-
drivers/media/usb/cx231xx/cx231xx-video.c | 79 +++--
drivers/media/usb/cx231xx/cx231xx.h | 39 ++-
drivers/media/usb/dvb-usb/cxusb-analog.c | 48 ++-
drivers/media/usb/em28xx/em28xx-video.c | 87 +++--
drivers/media/usb/go7007/go7007-v4l2.c | 75 ++--
drivers/media/usb/gspca/gspca.c | 51 ++-
drivers/media/usb/hackrf/hackrf.c | 39 ++-
drivers/media/usb/hdpvr/hdpvr-video.c | 60 ++--
drivers/media/usb/msi2500/msi2500.c | 31 +-
drivers/media/usb/pvrusb2/pvrusb2-v4l2.c | 98 ++++--
drivers/media/usb/pwc/pwc-v4l.c | 44 ++-
drivers/media/usb/s2255/s2255drv.c | 55 ++-
drivers/media/usb/stk1160/stk1160-v4l.c | 40 ++-
drivers/media/usb/usbtv/usbtv-video.c | 24 +-
drivers/media/usb/uvc/uvc_metadata.c | 17 +-
drivers/media/usb/uvc/uvc_v4l2.c | 59 +++-
drivers/media/v4l2-core/v4l2-ctrls-api.c | 2 +-
drivers/media/v4l2-core/v4l2-dev.c | 66 ++++
drivers/media/v4l2-core/v4l2-fh.c | 6 +
drivers/media/v4l2-core/v4l2-ioctl.c | 381 +++++++++++----------
drivers/media/v4l2-core/v4l2-mem2mem.c | 56 +--
drivers/staging/media/atomisp/pci/atomisp_ioctl.c | 72 ++--
drivers/staging/media/av7110/av7110_v4l.c | 60 +++-
.../media/deprecated/atmel/atmel-isc-base.c | 33 +-
drivers/staging/media/imx/imx-media-capture.c | 53 ++-
drivers/staging/media/imx/imx-media-csc-scaler.c | 23 +-
drivers/staging/media/ipu3/ipu3-v4l2.c | 29 +-
drivers/staging/media/ipu7/ipu7-isys-video.c | 30 +-
drivers/staging/media/meson/vdec/vdec.c | 26 +-
drivers/staging/media/starfive/camss/stf-video.c | 18 +-
drivers/staging/media/sunxi/cedrus/cedrus_video.c | 29 +-
.../media/sunxi/sun6i-isp/sun6i_isp_capture.c | 24 +-
.../media/sunxi/sun6i-isp/sun6i_isp_params.c | 9 +-
drivers/staging/media/tegra-video/vi.c | 68 ++--
drivers/staging/most/video/video.c | 39 ++-
.../vc04_services/bcm2835-camera/bcm2835-camera.c | 58 ++--
drivers/usb/gadget/function/uvc_v4l2.c | 50 ++-
include/media/v4l2-ctrls.h | 5 +-
include/media/v4l2-dev.h | 84 +++++
include/media/v4l2-fh.h | 2 +
include/media/v4l2-ioctl.h | 238 ++++++-------
include/media/v4l2-mem2mem.h | 48 ++-
include/media/videobuf2-v4l2.h | 33 +-
227 files changed, 5946 insertions(+), 3322 deletions(-)
---
base-commit: 680fa38ea3a19131871a90bac05a610e6247d928
change-id: 20250703-vdev-state-0743baa0ad4b

Best regards,
--
Jai Luthra <jai.luthra@ideasonboard.com>