[PATCH 00/26] media: Add support for multi-context operations

[PATCH 00/26] media: Add support for multi-context operations

[Jacopo Mondi]

Modern ISPs are designed to handle multiple "streams" of data, not necessarily
related to the same image or video stream.

The hardware resources are generally time-multiplexed between different
execution contexts at the hardware or firmware level and in order to operate the
ISP with multiple video sources it is necessary for drivers to keep track of
per-context data and resources.

In V4L2 the M2M framework supports multiple contexts through multiple opens of
the same video device. This doesn't however support drivers exposing multiple
video devices and sub-devices. Several out-of-tree drivers implement
multi-context support by registering multiple 'logical' instances of the same
media graph, one for each context. This effectively multiplies the number of
video device nodes and subdevice nodes. Userspace applications open one media
graph instance and operate on the corresponding video devices, under the
impression of dealing with a dedicated instance of the sole underlying hardware
resource.

This solution is however a short term hack, it doesn't scale well when the
number of contexts grow. ISPs such as the Mali C55 have been designed to process
8 cameras concurrently, and other ISPs may do more.

For this reason, a solution to expose and manage multiple execution contexts
without duplicating the number of media, video and sub-devices registered to
userspace is needed to improve support for multi-context devices in V4L2.

The series enables userspace to multiplex the usage of a media device and of
video devices without duplicating the number of devnodes in userspace, by
introducing the following concept in the framework:

- Media Device Context: a context created at media-device open time and stored
  in the media-fh file handle. To a media device context is associated a list
  of media entity contexts which are 'bound' to it.

- Video Device Context: represents an isolated execution context of a
  video device. By storing the data and the configuration of a video
  device userspace is allowed to effectively multiplex the usage of a
  device node.

- Video Subdevice Context: represents an isolated execution context of
  v4l2 subdevice. It stores that subdev state which is now available in
  three locations:
  - in the file handle: for TRY formats
  - in the subdevice: for ACTIVE formats to support operations of subdevices
    drivers that do not implement context support
  - in the subdevice default context: for ACTIVE formats to support
    operations of subdevice drivers that implement context support and
    are operated by non-context-aware userspace
  - in the subdevice file-handle context: for ACTIVE formats to support
    operations of subdevices drivers that implement context support and
    are  bound to a media device context

  Both the Video Device Context and the V4L2 Subdevice
  Context extend the Media Entity Context base type so that the MC and
  V4L2 layers are kept independent one from each other.

- A Video Device Context is created by a new ioctl VIDIOC_BIND_CONTEXT and is
  stored in the v4l2-fh file handle.

- A V4L2 Subdevice context is created by a new ioctl
  VIDIOC_SUBDEV_BIND_CONTEXT and is stored in the v4l2_subdev_fh file
  handle.

  The VIDIOC_BIND_CONTEXT and VIDIOC_SUBDEV_BIND_CONTEXT ioctls associates a
  Video/Subdevice Context to a Media Device Context. By binding a set of video
  devices and subdevices to a media device context userspace can create several
  isolated 'execution contexts' which can be operated independently one from
  each other.

- A V4L2 Video Device and V4L2 Subdevice default context is made
  created to allow drivers that implement multi-context support but are
  operated by userspace that is not context aware (IOW doesn't call
  VIDIOC_BIND_CONTEXT).

The first 20 patches implement the above described changes in the
framework:

  media: v4l2-subdev: Validate media links with context
  media: mc-entity: Add link_validate_context
  media: media-entity: Support context in pipeline_start
  media: v4l2-subdev: Get state from context
  media: v4l2-subdev: Add subdev state accessor helpers
  media: v4l2_subdev: Introduce default context
  media: Documentation: Add VIDIOC_SUBDEV_BIND_CONTEXT
  media: v4l2-subdv: Introduce VIDIOC_SUBDEV_BIND_CONTEXT
  media: v4l2-subdev: Introduce v4l2 subdev context
  media: videobuf2-v4l2: Support vb2_queue embedded in a context
  media: v4l2-dev: Add video_device_context_from_queue()
  media: v4l2-dev: Add video_device_context_from_file()
  media: Introduce default contexts
  media: v4l2-dev: Documentation: Add VIDIOC_BIND_CONTEXT
  media: v4l2-ioctl: Introduce VIDIOC_BIND_CONTEXT
  media: v4l2-dev: Introduce video device context
  media: media-device: Introduce media device context
  media: media-entity: Introduce media_entity_context
  media: mc: Maintain a list of open file handles in a media device
  media: mc: Add per-file-handle data support

For testing the implementation I used a Raspberry Pi5, implementing
support for multiple contexts in the ISP driver. The PiSP BE driver
however only implements video devices, as the single ISP subdev is not
exposed to userspace and doesn't require any configuration. To test the
V4L2 subdev context operations I had to first expose the ISP subdev to
userspace and implement link validation and pipeline allocation for it.
This is implemented in the following 6 patches NOT FOR INCLUSION in
mainline but just here for reference

 [DNI] media: pisp_be: Register devnode to userspace
 [DNI] media: pisp_be: Implement link validation
 [DNI] media: pisp_be: Implement set/get_pad_fmt
 [DNI] media: pisp_be: Add support for subdev state
 [DNI] media: pisp_be: Start and stop the media pipeline

Finally, the last patch (again NOT FOR INCLUSION) ports the ISP driver
to support multi-context operations and provides a reference for the
multi-context API usage by drivers:

 [DNI] media: pisp_be: Add support for multi-context

The multi-context support has been tested with a version of libcamera
that binds all video devices and the ISP subdevice in an execution
context and allows to stream two cameras at the same time using the same
media graph.

In order to validate the fact that introducing context support in
drivers doesn't break existing userspace which is not context-aware,
capturing from a single camera with [mainline libcamera + ISP subdev
configuration] has been tested as well.

CI pipeline:
- https://gitlab.freedesktop.org/linux-media/users/jmondi/-/pipelines/1472617
- checkpatch errors are due to "DNI" prefix in last patches
- static check errors seems unrelated to this set of patches but to
  existing code
- checkpatch error are due to the usage of "Generic" which checkpatch
  doesn't like

Branch for testing:
- kernel patches:
  https://gitlab.freedesktop.org/linux-media/users/jmondi/-/commits/multicontext/mainline/2025
- libcamera multi-context:
  https://git.libcamera.org/libcamera/jmondi/libcamera.git/log/?h=multicontext/rpi/mc
- libcamera single context:
  https://git.libcamera.org/libcamera/jmondi/libcamera.git/log/?h=multicontext/rpi/subdev-no-multi-context

rfc->v1 Compared to the RFC version sent in September 2024:
  - Implement support for V4L2 Subdevice context
  - Implement context-aware pipeline start and link validation
  - Break out from Sakari's media lifetime series the only 2 required
    patches that introduce media_device_fh
  - Test the BE ISP subdevice by implementing support for the ISP
    subdevice in the BE driver

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
Jacopo Mondi (24):
      media: media-entity: Introduce media_entity_context
      media: media-device: Introduce media device context
      media: v4l2-dev: Introduce video device context
      media: v4l2-ioctl: Introduce VIDIOC_BIND_CONTEXT
      media: Documentation: Add VIDIOC_BIND_CONTEXT
      media: v4l2-dev: Introduce default contexts
      media: v4l2-dev: Add video_device_context_from_file()
      media: v4l2-dev: Add video_device_context_from_queue()
      media: videobuf2-v4l2: Support vb2_queue embedded in a context
      media: v4l2-subdev: Introduce v4l2 subdev context
      media: v4l2-subdev: Introduce VIDIOC_SUBDEV_BIND_CONTEXT
      media: Documentation: Add VIDIOC_SUBDEV_BIND_CONTEXT
      media: v4l2_subdev: Introduce default context
      media: v4l2-subdev: Add subdev state accessor helpers
      media: v4l2-subdev: Get state from context
      media: media-entity: Support context in pipeline_start
      media: mc-entity: Add link_validate_context
      media: v4l2-subdev: Validate media links with context
      [DNI] media: pisp_be: Start and stop the media pipeline
      [DNI] media: pisp_be: Add support for subdev state
      [DNI] media: pisp_be: Implement set/get_pad_fmt
      [DNI] media: pisp_be: Implement link validation
      [DNI] media: pisp_be: Register devnode to userspace
      [DNI] media: pisp_be: Add support for multi-context

Laurent Pinchart (1):
      media: mc: Add per-file-handle data support

Sakari Ailus (1):
      media: mc: Maintain a list of open file handles in a media device

 .../userspace-api/media/v4l/user-func.rst          |   2 +
 .../media/v4l/vidioc-bind-context.rst              |  80 +++
 .../media/v4l/vidioc-subdev-bind-context.rst       |  81 +++
 drivers/media/common/videobuf2/videobuf2-v4l2.c    | 135 ++--
 drivers/media/mc/mc-device.c                       | 212 +++++-
 drivers/media/mc/mc-devnode.c                      |  20 +-
 drivers/media/mc/mc-entity.c                       | 220 +++++-
 .../media/platform/raspberrypi/pisp_be/pisp_be.c   | 773 ++++++++++++++++-----
 drivers/media/v4l2-core/v4l2-dev.c                 | 158 ++++-
 drivers/media/v4l2-core/v4l2-device.c              |  11 +-
 drivers/media/v4l2-core/v4l2-fh.c                  |   1 +
 drivers/media/v4l2-core/v4l2-ioctl.c               |  64 ++
 drivers/media/v4l2-core/v4l2-subdev.c              | 414 ++++++++++-
 include/media/media-device.h                       | 223 ++++++
 include/media/media-devnode.h                      |  21 +-
 include/media/media-entity.h                       | 163 ++++-
 include/media/media-fh.h                           |  37 +
 include/media/v4l2-dev.h                           | 252 +++++++
 include/media/v4l2-fh.h                            |   3 +
 include/media/v4l2-ioctl.h                         |   7 +
 include/media/v4l2-subdev.h                        | 293 ++++++--
 include/uapi/linux/v4l2-subdev.h                   |  11 +
 include/uapi/linux/videodev2.h                     |  11 +
 23 files changed, 2845 insertions(+), 347 deletions(-)
---
base-commit: d968e50b5c26642754492dea23cbd3592bde62d8
change-id: 20250716-multicontext-mainline-2025-3479c7c24b7a

Best regards,
-- 
Jacopo Mondi <jacopo.mondi@ideasonboard.com>

[PATCH 01/26] media: mc: Add per-file-handle data support

[Jacopo Mondi]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

The media devnode core associates devnodes with files by storing the
devnode pointer in the file structure private_data field. In order to
allow tracking of per-file-handle data introduce a new media devnode
file handle structure that stores the devnode pointer, and store a
pointer to that structure in the file private_data field.

Users of the media devnode code (the only existing user being
media_device) are responsible for managing their own subclass of the
media_devnode_fh structure.

Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Prepare struct media_device_fh to be used for maintaining file handle list
to avoid shuffling things here and there right after.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Acked-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/mc/mc-device.c  | 14 +++++++++++++-
 drivers/media/mc/mc-devnode.c | 20 +++++++++-----------
 include/media/media-device.h  |  7 +++++++
 include/media/media-devnode.h | 18 +++++++++++++++++-
 include/media/media-fh.h      | 32 ++++++++++++++++++++++++++++++++
 5 files changed, 78 insertions(+), 13 deletions(-)

diff --git a/drivers/media/mc/mc-device.c b/drivers/media/mc/mc-device.c
index c0dd4ae5722725f1744bc6fd6282d5c765438059..f298f8f67b8a84194d66126461e84228c0b20394 100644
--- a/drivers/media/mc/mc-device.c
+++ b/drivers/media/mc/mc-device.c
@@ -22,6 +22,7 @@
 #include <media/media-device.h>
 #include <media/media-devnode.h>
 #include <media/media-entity.h>
+#include <media/media-fh.h>
 #include <media/media-request.h>
 
 #ifdef CONFIG_MEDIA_CONTROLLER
@@ -35,7 +36,6 @@
 #define MEDIA_ENT_SUBTYPE_MASK			0x0000ffff
 #define MEDIA_ENT_T_DEVNODE_UNKNOWN		(MEDIA_ENT_F_OLD_BASE | \
 						 MEDIA_ENT_SUBTYPE_MASK)
-
 /* -----------------------------------------------------------------------------
  * Userspace API
  */
@@ -47,11 +47,23 @@ static inline void __user *media_get_uptr(__u64 arg)
 
 static int media_device_open(struct file *filp)
 {
+	struct media_device_fh *fh;
+
+	fh = kzalloc(sizeof(*fh), GFP_KERNEL);
+	if (!fh)
+		return -ENOMEM;
+
+	filp->private_data = &fh->fh;
+
 	return 0;
 }
 
 static int media_device_close(struct file *filp)
 {
+	struct media_device_fh *fh = media_device_fh(filp);
+
+	kfree(fh);
+
 	return 0;
 }
 
diff --git a/drivers/media/mc/mc-devnode.c b/drivers/media/mc/mc-devnode.c
index 56444edaf13651874331e7c04e86b0a585067d38..312eb48ffc2f2a0c013e4744204995df0ff5b12c 100644
--- a/drivers/media/mc/mc-devnode.c
+++ b/drivers/media/mc/mc-devnode.c
@@ -142,6 +142,7 @@ static long media_compat_ioctl(struct file *filp, unsigned int cmd,
 static int media_open(struct inode *inode, struct file *filp)
 {
 	struct media_devnode *devnode;
+	struct media_devnode_fh *fh;
 	int ret;
 
 	/* Check if the media device is available. This needs to be done with
@@ -162,17 +163,15 @@ static int media_open(struct inode *inode, struct file *filp)
 	get_device(&devnode->dev);
 	mutex_unlock(&media_devnode_lock);
 
-	filp->private_data = devnode;
-
-	if (devnode->fops->open) {
-		ret = devnode->fops->open(filp);
-		if (ret) {
-			put_device(&devnode->dev);
-			filp->private_data = NULL;
-			return ret;
-		}
+	ret = devnode->fops->open(filp);
+	if (ret) {
+		put_device(&devnode->dev);
+		return ret;
 	}
 
+	fh = filp->private_data;
+	fh->devnode = devnode;
+
 	return 0;
 }
 
@@ -181,8 +180,7 @@ static int media_release(struct inode *inode, struct file *filp)
 {
 	struct media_devnode *devnode = media_devnode_data(filp);
 
-	if (devnode->fops->release)
-		devnode->fops->release(filp);
+	devnode->fops->release(filp);
 
 	filp->private_data = NULL;
 
diff --git a/include/media/media-device.h b/include/media/media-device.h
index 53d2a16a70b0d9d6e5cc28fe1fc5d5ef384410d5..2fc750efef7c43814f019f12078e9c96c1bd6bf9 100644
--- a/include/media/media-device.h
+++ b/include/media/media-device.h
@@ -108,6 +108,10 @@ struct media_device_ops {
  *		     other operations that stop or start streaming.
  * @request_id: Used to generate unique request IDs
  *
+ * @fh_list:	List of file handles in the media device
+ *		(struct media_device_fh.mdev_list).
+ * @fh_list_lock: Serialise access to fh_list list.
+ *
  * This structure represents an abstract high-level media device. It allows easy
  * access to entities and provides basic media device-level support. The
  * structure can be allocated directly or embedded in a larger structure.
@@ -180,6 +184,9 @@ struct media_device {
 
 	struct mutex req_queue_mutex;
 	atomic_t request_id;
+
+	struct list_head fh_list;
+	spinlock_t fh_list_lock;
 };
 
 /* We don't need to include usb.h here */
diff --git a/include/media/media-devnode.h b/include/media/media-devnode.h
index d27c1c646c2805171be3997d72210dd4d1a38e32..6c2e253dde498779dffd103dc5d00e50e14a0249 100644
--- a/include/media/media-devnode.h
+++ b/include/media/media-devnode.h
@@ -55,6 +55,20 @@ struct media_file_operations {
 	int (*release) (struct file *);
 };
 
+/**
+ * struct media_devnode_fh - Media device node file handle
+ * @devnode:	pointer to the media device node
+ *
+ * This structure serves as a base for per-file-handle data storage. Media
+ * device node users embed media_devnode_fh in their custom file handle data
+ * structures and store the media_devnode_fh in the file private_data in order
+ * to let the media device node core locate the media_devnode corresponding to a
+ * file handle.
+ */
+struct media_devnode_fh {
+	struct media_devnode *devnode;
+};
+
 /**
  * struct media_devnode - Media device node
  * @media_dev:	pointer to struct &media_device
@@ -146,7 +160,9 @@ void media_devnode_unregister(struct media_devnode *devnode);
  */
 static inline struct media_devnode *media_devnode_data(struct file *filp)
 {
-	return filp->private_data;
+	struct media_devnode_fh *fh = filp->private_data;
+
+	return fh->devnode;
 }
 
 /**
diff --git a/include/media/media-fh.h b/include/media/media-fh.h
new file mode 100644
index 0000000000000000000000000000000000000000..6f00744b81d6000a4b0c503fe6968dd7adcbb1c3
--- /dev/null
+++ b/include/media/media-fh.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Media device file handle
+ *
+ * Copyright (C) 2019--2023 Intel Corporation
+ */
+
+#ifndef MEDIA_FH_H
+#define MEDIA_FH_H
+
+#include <linux/list.h>
+#include <linux/file.h>
+
+#include <media/media-devnode.h>
+
+/**
+ * struct media_device_fh - File handle specific information on MC
+ *
+ * @fh: The media device file handle
+ * @mdev_list: This file handle in media device's list of file handles
+ */
+struct media_device_fh {
+	struct media_devnode_fh fh;
+	struct list_head mdev_list;
+};
+
+static inline struct media_device_fh *media_device_fh(struct file *filp)
+{
+	return container_of(filp->private_data, struct media_device_fh, fh);
+}
+
+#endif /* MEDIA_FH_H */

-- 
2.49.0

[PATCH 02/26] media: mc: Maintain a list of open file handles in a media device

[Jacopo Mondi]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

The list of file handles is needed to deliver media events as well as for
other purposes in the future.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
[Access media_device with devnode->media_dev]
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/mc/mc-device.c  | 19 ++++++++++++++++++-
 drivers/media/mc/mc-devnode.c |  2 +-
 include/media/media-devnode.h |  3 ++-
 3 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/drivers/media/mc/mc-device.c b/drivers/media/mc/mc-device.c
index f298f8f67b8a84194d66126461e84228c0b20394..e0cad87087d3863bf14207049a54e5e4dea1cdd4 100644
--- a/drivers/media/mc/mc-device.c
+++ b/drivers/media/mc/mc-device.c
@@ -45,8 +45,9 @@ static inline void __user *media_get_uptr(__u64 arg)
 	return (void __user *)(uintptr_t)arg;
 }
 
-static int media_device_open(struct file *filp)
+static int media_device_open(struct media_devnode *devnode, struct file *filp)
 {
+	struct media_device *mdev = devnode->media_dev;
 	struct media_device_fh *fh;
 
 	fh = kzalloc(sizeof(*fh), GFP_KERNEL);
@@ -55,13 +56,23 @@ static int media_device_open(struct file *filp)
 
 	filp->private_data = &fh->fh;
 
+	spin_lock_irq(&mdev->fh_list_lock);
+	list_add(&fh->mdev_list, &mdev->fh_list);
+	spin_unlock_irq(&mdev->fh_list_lock);
+
 	return 0;
 }
 
 static int media_device_close(struct file *filp)
 {
+	struct media_devnode *devnode = media_devnode_data(filp);
+	struct media_device *mdev = devnode->media_dev;
 	struct media_device_fh *fh = media_device_fh(filp);
 
+	spin_lock_irq(&mdev->fh_list_lock);
+	list_del(&fh->mdev_list);
+	spin_unlock_irq(&mdev->fh_list_lock);
+
 	kfree(fh);
 
 	return 0;
@@ -698,10 +709,12 @@ void media_device_init(struct media_device *mdev)
 	INIT_LIST_HEAD(&mdev->pads);
 	INIT_LIST_HEAD(&mdev->links);
 	INIT_LIST_HEAD(&mdev->entity_notify);
+	INIT_LIST_HEAD(&mdev->fh_list);
 
 	mutex_init(&mdev->req_queue_mutex);
 	mutex_init(&mdev->graph_mutex);
 	ida_init(&mdev->entity_internal_idx);
+	spin_lock_init(&mdev->fh_list_lock);
 
 	atomic_set(&mdev->request_id, 0);
 
@@ -809,6 +822,10 @@ void media_device_unregister(struct media_device *mdev)
 		return;
 	}
 
+	spin_lock_irq(&mdev->fh_list_lock);
+	list_del_init(&mdev->fh_list);
+	spin_unlock_irq(&mdev->fh_list_lock);
+
 	/* Clear the devnode register bit to avoid races with media dev open */
 	media_devnode_unregister_prepare(mdev->devnode);
 
diff --git a/drivers/media/mc/mc-devnode.c b/drivers/media/mc/mc-devnode.c
index 312eb48ffc2f2a0c013e4744204995df0ff5b12c..50435f102aa7cd6a7cde759eeef73a99b9b80239 100644
--- a/drivers/media/mc/mc-devnode.c
+++ b/drivers/media/mc/mc-devnode.c
@@ -163,7 +163,7 @@ static int media_open(struct inode *inode, struct file *filp)
 	get_device(&devnode->dev);
 	mutex_unlock(&media_devnode_lock);
 
-	ret = devnode->fops->open(filp);
+	ret = devnode->fops->open(devnode, filp);
 	if (ret) {
 		put_device(&devnode->dev);
 		return ret;
diff --git a/include/media/media-devnode.h b/include/media/media-devnode.h
index 6c2e253dde498779dffd103dc5d00e50e14a0249..26b19373c6646bfd11cfded220c9e61c81130580 100644
--- a/include/media/media-devnode.h
+++ b/include/media/media-devnode.h
@@ -22,6 +22,7 @@
 #include <linux/cdev.h>
 
 struct media_device;
+struct media_devnode;
 
 /*
  * Flag to mark the media_devnode struct as registered. Drivers must not touch
@@ -51,7 +52,7 @@ struct media_file_operations {
 	__poll_t (*poll) (struct file *, struct poll_table_struct *);
 	long (*ioctl) (struct file *, unsigned int, unsigned long);
 	long (*compat_ioctl) (struct file *, unsigned int, unsigned long);
-	int (*open) (struct file *);
+	int (*open) (struct media_devnode *, struct file *);
 	int (*release) (struct file *);
 };
 

-- 
2.49.0

[PATCH 03/26] media: media-entity: Introduce media_entity_context

[Jacopo Mondi]

Introduce the 'struct media_entity_context' type, which serves for
reference counting and introduce two new media entity operations to
allow drivers to allocate and free a media entity context.

The newly introduced type will be used as a base type for the
device context types (video_device_context and v4l2_subdevice_context)
that will be introduced in the next patches.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c |  46 ++++++++++++++++++++
 include/media/media-entity.h | 101 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 147 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 045590905582054c46656e20463271b1f93fa6b4..b4a9f0a0aa7353d7a3333f20903980956b3df4a7 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -1673,3 +1673,49 @@ struct media_link *__media_entity_next_link(struct media_entity *entity,
 	return NULL;
 }
 EXPORT_SYMBOL_GPL(__media_entity_next_link);
+
+static void media_entity_release_context(struct kref *refcount)
+{
+	struct media_entity_context *ctx =
+		container_of(refcount, struct media_entity_context, refcount);
+
+	ctx->entity->ops->destroy_context(ctx);
+}
+
+struct media_entity_context *
+media_entity_context_get(struct media_entity_context *ctx)
+{
+	if (!ctx)
+		return ERR_PTR(-EINVAL);
+
+	kref_get(&ctx->refcount);
+
+	return ctx;
+}
+EXPORT_SYMBOL_GPL(media_entity_context_get);
+
+void media_entity_context_put(struct media_entity_context *ctx)
+{
+	if (!ctx)
+		return;
+
+	kref_put(&ctx->refcount, media_entity_release_context);
+}
+EXPORT_SYMBOL_GPL(media_entity_context_put);
+
+void media_entity_init_context(struct media_entity *entity,
+			       struct media_entity_context *ctx)
+{
+	if (!ctx)
+		return;
+
+	ctx->entity = entity;
+	kref_init(&ctx->refcount);
+	INIT_LIST_HEAD(&ctx->list);
+}
+EXPORT_SYMBOL_GPL(media_entity_init_context);
+
+void media_entity_cleanup_context(struct media_entity_context *ctx)
+{
+}
+EXPORT_SYMBOL_GPL(media_entity_cleanup_context);
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 64cf590b11343f68a456c5870ca2f32917c122f9..32298fe8a18c6ee3c1dbcff9ef869548904417a7 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -15,6 +15,7 @@
 #include <linux/bug.h>
 #include <linux/container_of.h>
 #include <linux/fwnode.h>
+#include <linux/kref.h>
 #include <linux/list.h>
 #include <linux/media.h>
 #include <linux/minmax.h>
@@ -248,6 +249,37 @@ struct media_pad {
 	struct media_pipeline *pipe;
 };
 
+/**
+ * struct media_entity_context - A media entity execution context
+ * @mdev_context: The media device context this media entity is bound to.
+ *		  The field is initialized when the entity is bound to a media
+ *		  device context.
+ * @entity: The media entity this context belongs to
+ * @refcount: The kref reference counter
+ * list: The list entry to link the entity context in the media device context
+ *
+ * This type represent the 'base class' used to implement execution context for
+ * video device contexts and subdevice contexts. Those types embedds an instance
+ * of 'struct media_entity_context' as their first member, allowing the MC core
+ * to implement type polymorphism and handle video device and subdevice contexts
+ * transparently.
+ *
+ * The main function of this type is to provide reference counting for the
+ * 'dervived' device context types. The video device and subdevice core
+ * populates the 'context_release' function pointer that implement specific
+ * clean-up operations, similar to what a 'virtual destructor' would do in C++.
+ *
+ * Drivers are not expected to use this type directly, but only the MC core
+ * will.
+ */
+struct media_device_context;
+struct media_entity_context {
+	struct media_device_context *mdev_context;
+	struct media_entity *entity;
+	struct kref refcount;
+	struct list_head list;
+};
+
 /**
  * struct media_entity_operations - Media entity operations
  * @get_fwnode_pad:	Return the pad number based on a fwnode endpoint or
@@ -269,6 +301,15 @@ struct media_pad {
  *			media_entity_has_pad_interdep().
  *			Optional: If the operation isn't implemented all pads
  *			will be considered as interdependent.
+ * @alloc_context:	Allocate a media entity context. Drivers are allowed to
+ *			sub-class the entity context type by defining a driver
+ *			specific type that embeds an instance of either a
+ *			video_device_context or subdevice_context as first
+ *			member, and allocate the size of a driver-specific type
+ *			in the implementation of this operation. Returns 0 for
+ *			success, or an error code < 0 otherwise.
+ * @destroy_context:	Release a media entity context previously allocated by
+ *			the driver.
  *
  * .. note::
  *
@@ -284,6 +325,9 @@ struct media_entity_operations {
 	int (*link_validate)(struct media_link *link);
 	bool (*has_pad_interdep)(struct media_entity *entity, unsigned int pad0,
 				 unsigned int pad1);
+	int (*alloc_context)(struct media_entity *entity,
+			     struct media_entity_context **context);
+	void (*destroy_context)(struct media_entity_context *context);
 };
 
 /**
@@ -1448,3 +1492,60 @@ struct media_link *__media_entity_next_link(struct media_entity *entity,
 					     MEDIA_LNK_FL_DATA_LINK))
 
 #endif
+
+/**
+ * media_entity_context_get - Increase the media entity context reference count
+ *			      and return a reference to it
+ *
+ * @ctx: the media entity context
+ *
+ * Increase the media entity context reference count. The reference count
+ * is increased by the V4L2 core when:
+ *
+ * * a new context is allocated when bounding a media entity to a media device
+ *   context (by kref_init())
+ * * the media pipeline the context is part of starts streaming
+ *
+ * The entity context gets automatically decreased by the V4L2 core when:
+ *
+ * * a context is unbound
+ * * the pipeline stops streaming
+ */
+struct media_entity_context *
+media_entity_context_get(struct media_entity_context *ctx);
+
+/**
+ * media_entity_context_put - Decrease the media entity context reference count
+ *
+ * @ctx: the media entity context
+ *
+ * Decrease the media entity context reference count. The reference count
+ * is decreased by the V4L2 core when:
+ *
+ * * the file handle the context is associated with is closed
+ * * the media pipeline the context is part of is stopped
+ */
+void media_entity_context_put(struct media_entity_context *ctx);
+
+/**
+ * media_entity_init_context - Initialize the media entity context
+ *
+ * @entity: the media entity this context belongs to
+ * @ctx: the media entity context
+ *
+ * Initialize the media entity context by initializing the kref reference
+ * counter. The intended caller of this function are the video device context
+ * and subdevic context initialize functions.
+ */
+void media_entity_init_context(struct media_entity *entity,
+			       struct media_entity_context *ctx);
+
+/**
+ * media_entity_cleanup_context - Cleanup the media entity context
+ *
+ * @ctx: the media entity context
+ *
+ * Cleanup the media entity context. The intended caller of this function are
+ * the video device and subdevice context cleanup functions.
+ */
+void media_entity_cleanup_context(struct media_entity_context *ctx);

-- 
2.49.0

[PATCH 04/26] media: media-device: Introduce media device context

[Jacopo Mondi]

Introduce a new type in the media-fh.h header that represent a media
device context.

A media device context is allocated when the media device is open
and released when the last reference to it is put. A new pair of
media_device_ops is added to allow device drivers to allocate and
release a media context.

The media context groups together the media entity contexts that are
associated with it to form an isolated execution context.

Provide helpers in mc-device.c for drivers and for the v4l2-core to
handle media device contexts and to bind/unbind entity contexts
to it. Once an entity context has been bound to a media device
context it is possible to retrieve it by using a pointer to the entity
the device is represented by.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/mc/mc-device.c | 168 ++++++++++++++++++++++++++++++++++
 drivers/media/mc/mc-entity.c |   1 +
 include/media/media-device.h | 211 +++++++++++++++++++++++++++++++++++++++++++
 include/media/media-fh.h     |   5 +
 4 files changed, 385 insertions(+)

diff --git a/drivers/media/mc/mc-device.c b/drivers/media/mc/mc-device.c
index e0cad87087d3863bf14207049a54e5e4dea1cdd4..d8f12db933d22ae7466051698d853f4bdc599400 100644
--- a/drivers/media/mc/mc-device.c
+++ b/drivers/media/mc/mc-device.c
@@ -12,7 +12,9 @@
 #include <linux/export.h>
 #include <linux/idr.h>
 #include <linux/ioctl.h>
+#include <linux/kref.h>
 #include <linux/media.h>
+#include <linux/mutex.h>
 #include <linux/slab.h>
 #include <linux/types.h>
 #include <linux/pci.h>
@@ -49,11 +51,31 @@ static int media_device_open(struct media_devnode *devnode, struct file *filp)
 {
 	struct media_device *mdev = devnode->media_dev;
 	struct media_device_fh *fh;
+	int ret;
 
 	fh = kzalloc(sizeof(*fh), GFP_KERNEL);
 	if (!fh)
 		return -ENOMEM;
 
+	if (mdev->ops && mdev->ops->alloc_context) {
+		if (WARN_ON(!mdev->ops->destroy_context)) {
+			kfree(fh);
+			return -EINVAL;
+		}
+
+		ret = mdev->ops->alloc_context(mdev, &fh->context);
+		if (ret) {
+			kfree(fh);
+			return ret;
+		}
+
+		/*
+		 * Make sure the driver implementing alloc_context has
+		 * called media_device_init_context()
+		 */
+		WARN_ON(!fh->context->initialized);
+	}
+
 	filp->private_data = &fh->fh;
 
 	spin_lock_irq(&mdev->fh_list_lock);
@@ -73,6 +95,8 @@ static int media_device_close(struct file *filp)
 	list_del(&fh->mdev_list);
 	spin_unlock_irq(&mdev->fh_list_lock);
 
+	media_device_context_put(fh->context);
+
 	kfree(fh);
 
 	return 0;
@@ -860,6 +884,150 @@ void media_device_unregister(struct media_device *mdev)
 }
 EXPORT_SYMBOL_GPL(media_device_unregister);
 
+/* -----------------------------------------------------------------------------
+ * Context handling
+ */
+
+static void media_device_release_context(struct kref *refcount)
+{
+	struct media_device_context *context =
+		container_of(refcount, struct media_device_context, refcount);
+
+	/*
+	 * All the associated entity contexts should have been released if we
+	 * get here.
+	 */
+	WARN_ON(!list_empty(&context->contexts));
+
+	context->mdev->ops->destroy_context(context);
+}
+
+struct media_device_context *
+media_device_context_get(struct media_device_context *ctx)
+{
+	if (!ctx)
+		return ERR_PTR(-EINVAL);
+
+	kref_get(&ctx->refcount);
+
+	return ctx;
+}
+EXPORT_SYMBOL_GPL(media_device_context_get);
+
+void media_device_context_put(struct media_device_context *ctx)
+{
+	if (!ctx)
+		return;
+
+	kref_put(&ctx->refcount, media_device_release_context);
+}
+EXPORT_SYMBOL_GPL(media_device_context_put);
+
+struct media_device_context *media_device_context_get_from_fd(unsigned int fd)
+{
+	struct media_device_context *ctx;
+	struct file *filp = fget(fd);
+	struct media_device_fh *fh;
+
+	if (!filp)
+		return NULL;
+
+	fh = media_device_fh(filp);
+	ctx = media_device_context_get(fh->context);
+	fput(filp);
+
+	return ctx;
+}
+EXPORT_SYMBOL_GPL(media_device_context_get_from_fd);
+
+int media_device_init_context(struct media_device *mdev,
+			      struct media_device_context *ctx)
+{
+	ctx->mdev = mdev;
+	INIT_LIST_HEAD(&ctx->contexts);
+	mutex_init(&ctx->lock);
+	kref_init(&ctx->refcount);
+
+	ctx->initialized = true;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(media_device_init_context);
+
+void media_device_cleanup_context(struct media_device_context *ctx)
+{
+	mutex_destroy(&ctx->lock);
+	list_del_init(&ctx->contexts);
+}
+EXPORT_SYMBOL_GPL(media_device_cleanup_context);
+
+int media_device_bind_context(struct media_device_context *mdev_context,
+			      struct media_entity_context *context)
+{
+	struct media_entity_context *entry;
+
+	if (WARN_ON(!mdev_context || !context))
+		return -EINVAL;
+
+	guard(mutex)(&mdev_context->lock);
+
+	/* Make sure the entity has not been bound already. */
+	list_for_each_entry(entry, &mdev_context->contexts, list) {
+		if (entry == context)
+			return -EINVAL;
+	}
+
+	list_add_tail(&context->list, &mdev_context->contexts);
+	context->mdev_context = media_device_context_get(mdev_context);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(media_device_bind_context);
+
+int media_device_unbind_context(struct media_entity_context *context)
+{
+	struct media_device_context *mdev_context = context->mdev_context;
+	struct media_entity_context *entry;
+	struct media_entity_context *tmp;
+
+	if (WARN_ON(!mdev_context || !context))
+		return -EINVAL;
+
+	guard(mutex)(&mdev_context->lock);
+	list_for_each_entry_safe(entry, tmp, &mdev_context->contexts, list) {
+		if (entry != context)
+			continue;
+
+		list_del(&entry->list);
+		media_device_context_put(mdev_context);
+		entry->mdev_context = NULL;
+
+		return 0;
+	}
+
+	WARN(true, "Media entity context is not bound to any media context\n");
+
+	return -EINVAL;
+}
+EXPORT_SYMBOL_GPL(media_device_unbind_context);
+
+struct media_entity_context *
+media_device_get_entity_context(struct media_device_context *mdev_context,
+				struct media_entity *entity)
+{
+	struct media_entity_context *entry;
+
+	guard(mutex)(&mdev_context->lock);
+
+	list_for_each_entry(entry, &mdev_context->contexts, list) {
+		if (entry->entity == entity)
+			return media_entity_context_get(entry);
+	}
+
+	return ERR_PTR(-EINVAL);
+}
+EXPORT_SYMBOL(media_device_get_entity_context);
+
 #if IS_ENABLED(CONFIG_PCI)
 void media_device_pci_init(struct media_device *mdev,
 			   struct pci_dev *pci_dev,
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index b4a9f0a0aa7353d7a3333f20903980956b3df4a7..7bc276c725f974539ea06e3882d004b81be1de68 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -1717,5 +1717,6 @@ EXPORT_SYMBOL_GPL(media_entity_init_context);
 
 void media_entity_cleanup_context(struct media_entity_context *ctx)
 {
+	media_device_unbind_context(ctx);
 }
 EXPORT_SYMBOL_GPL(media_entity_cleanup_context);
diff --git a/include/media/media-device.h b/include/media/media-device.h
index 2fc750efef7c43814f019f12078e9c96c1bd6bf9..2ea8fce9ba75700286961f1622584372a954cb8a 100644
--- a/include/media/media-device.h
+++ b/include/media/media-device.h
@@ -18,10 +18,73 @@
 
 #include <media/media-devnode.h>
 #include <media/media-entity.h>
+#include <media/media-fh.h>
 
 struct ida;
 struct media_device;
 
+/**
+ * struct media_device_context - Media device context
+ * @mdev: The media device this context is associated with
+ * @refcount: The kref reference counter
+ * @lock: Protects the entities contexts list
+ * @contexts: List of entity contexts associated with this media device context
+ * @initialized: Flag set to true by media_device_init_context()
+ *
+ * A media device context is created every time the media device gets opened by
+ * userspace. It is then uniquely identified for applications by the numerical
+ * file descriptor returned by a successful call to open() and is associated
+ * with an instance of :c:type:`media_device_fh`.
+ *
+ * Media device contexts are ref-counted and thus freed once the last reference
+ * to them is released.
+ *
+ * A media device context groups together the media entity contexts registered
+ * on a video device or v4l2 subdevice that has been associated with a media
+ * device context. The association between a media entity context and media
+ * device context is called 'bounding', and the result of bounding is to create
+ * an 'execution context' independent from other execution contexts.
+ *
+ * An entity context is bound to a media device context by a call to the
+ * VIDIOC_BIND_CONTEXT ioctl on video devices and by a call to
+ * VIDIOC_SUBDEV_BIND_CONTEXT on subdevices by userspace. The bounding operation
+ * groups together entity contexts to the same media device context. As video
+ * devices and v4l2 subdevices devnodes can be opened multiple times, each file
+ * descriptor resulting from a successful open() call can be bound to a
+ * different media device context.
+ *
+ * Creating execution contexts by bounding video entity contexts to a media
+ * device context allows userspace to effectively multiplex the usage of a
+ * media graph and of the device nodes that are part of it.
+ *
+ * In order to create an execution context userspace should:
+ *
+ * 1) Open the media device to create a media device context identified by the
+ * file descriptor returned by a successful 'open()' call
+ * 2) Open the video device or v4l2 subdevice and bind the file descriptors to
+ * the media device context by calling the VIDIOC_BIND_CONTEXT and
+ * VIDIOC_SUBDEV_BIND_CONTEXT ioctls
+ *
+ * All devices bound to the same media device context are now part of the same
+ * execution context. From this point on all the operations performed on a file
+ * descriptor bound to a media device context are independent from operations
+ * performed on a file descriptor bound to a different execution context.
+ *
+ * Binding an entity context to a media device context increases the media
+ * device context reference count. This guarantees that references to media
+ * device context are valid as long as there are valid entity contexts that
+ * refers to it. Symmetrically, unbinding an entity context from a media
+ * device context decreases the media device context reference count.
+ */
+struct media_device_context {
+	struct media_device *mdev;
+	struct kref refcount;
+	/* Protects the 'contexts' list */
+	struct mutex lock;
+	struct list_head contexts;
+	bool initialized;
+};
+
 /**
  * struct media_entity_notify - Media Entity Notify
  *
@@ -62,6 +125,13 @@ struct media_entity_notify {
  *	       request (and thus the buffer) must be available to the driver.
  *	       And once a buffer is queued, then the driver can complete
  *	       or delete objects from the request before req_queue exits.
+ * @alloc_context: Allocate a media device context. The operation allows drivers to
+ *		   allocate a driver-specific structure that embeds a
+ *		   media_device_context instance as first member where to store
+ *		   driver-specific information that are global to all device
+ *		   contexts part of media device context. Returns 0 on success a
+ *		   negative error code otherwise.
+ * @destroy_context: Release a media device context.
  */
 struct media_device_ops {
 	int (*link_notify)(struct media_link *link, u32 flags,
@@ -70,6 +140,9 @@ struct media_device_ops {
 	void (*req_free)(struct media_request *req);
 	int (*req_validate)(struct media_request *req);
 	void (*req_queue)(struct media_request *req);
+	int (*alloc_context)(struct media_device *mdev,
+			     struct media_device_context **ctx);
+	void (*destroy_context)(struct media_device_context *ctx);
 };
 
 /**
@@ -298,6 +371,144 @@ int __must_check __media_device_register(struct media_device *mdev,
  */
 void media_device_unregister(struct media_device *mdev);
 
+/* -----------------------------------------------------------------------------
+ * media device context handling
+ */
+
+/**
+ * media_device_context_get - Increase the media device context reference count
+ *			      and return a reference to it
+ * @ctx: The media device context
+ */
+struct media_device_context *
+media_device_context_get(struct media_device_context *ctx);
+
+/**
+ * media_device_context_put - Decrease the media device context reference count
+ * @ctx: The media device context
+ */
+void media_device_context_put(struct media_device_context *ctx);
+
+/**
+ * media_device_context_get_from_fd - Get the media device context associated with a
+ *				      numerical file descriptor
+ *
+ * @fd: the numerical file descriptor
+ *
+ * A media device context is created whenever the media device devnode is opened
+ * by userspace. It is then associated uniquely with a numerical file descriptor
+ * which is unique in the userspace process context.
+ *
+ * This function allows to retrieve the media device associated with such
+ * numerical file descriptor and increases the media device context reference
+ * count to guarantee the returned reference stays valid at least until the
+ * caller does not call media_device_context_put().
+ *
+ * Caller of this function are required to put the returned media device context
+ * once they are done with it.
+ *
+ * The intended caller of this function is the VIDIOC_BIND_CONTEXT ioctl handler
+ * which need to get the media device contexts associated to a numerical file
+ * descriptor.
+ */
+struct media_device_context *media_device_context_get_from_fd(unsigned int fd);
+
+/**
+ * media_device_init_context - Initialize the media device context
+ *
+ * @mdev: The media device this context belongs to
+ * @ctx: The media device context to initialize
+ *
+ * Initialize the fields of a media device context. Device drivers that support
+ * multi context operations shall call this function in their implementation of
+ * media_device_operations.alloc_context()
+ */
+int media_device_init_context(struct media_device *mdev,
+			      struct media_device_context *ctx);
+
+/**
+ * media_device_cleanup_context - Cleanup the media device context
+ *
+ * @ctx: The media device context to clean up
+ *
+ * Cleanup a media device context. Device drivers that support multi context
+ * operations shall call this function in their implementation of
+ * media_device_operations.destroy_context() before releasing the memory allocated
+ * by media_device_operations.alloc_context().
+ */
+void media_device_cleanup_context(struct media_device_context *ctx);
+
+/**
+ * media_device_bind_context - Bind an entity context to a media device context
+ *
+ * @mdev_context: pointer to struct &media_device_context
+ * @context: the entity context to bind
+ *
+ * This function creates a mapping entry in the media device context that
+ * associates an entity context to the media entity it belongs to and stores it
+ * in a linked list so that they can be retrieved later.
+ *
+ * Binding an entity context to a media device context increases the media
+ * device context refcount.
+ *
+ * The intended caller of this function is the VIDIOC_BIND_CONTEXT ioctl handler
+ * that binds a newly created context to a media device context.
+ */
+int media_device_bind_context(struct media_device_context *mdev_context,
+			      struct media_entity_context *context);
+
+/**
+ * media_device_unbind_context - Unbind an entity context from a media device
+ *				 context
+ *
+ * @context: the entity context to unbind
+ *
+ * An entity context is unbound from a media device context when the file handle
+ * it is associated with gets closed.
+ *
+ * Unbinding an entity context from a media device context decreases the media
+ * device context refcount.
+ *
+ * Returns 0 if the context was bound to a media device context, -EINVAL
+ * otherwise.
+ */
+int media_device_unbind_context(struct media_entity_context *context);
+
+/**
+ * media_device_get_entity_context - Get the entity context associated with
+ *				     a media entity in a media device context
+ *
+ * @mdev_context: pointer to struct &media_device_context
+ * @entity: pointer to struct &media_entity that the entity context is
+ *	    associated with
+ *
+ * An entity context is uniquely associated with a media device context after it
+ * has been bound to it by a call to the VIDIOC_BIND_CONTEXT ioctl. This helper
+ * function retrieves the entity context associated with a media device context
+ * for a specific entity that represents a video device or a v4l2 subdevice.
+ *
+ * The reference count of the returned entity context is increased to guarantee
+ * the returned reference stays valid until the caller does not call
+ * media_entity_context_put().
+ *
+ * Drivers are not expected to call this function directly but should instead
+ * use the helpers provided by the video_device and v4l2_subdevice layers,
+ * video_device_context_get() and v4l2_subdev_get_context() respectively.
+ * Drivers are always required to decrease the returned context reference count
+ * by calling video_device_context_put() and v4l2_subdev_put_context().
+ *
+ * If no entity context has been associated with the media device context
+ * provided as first argument an error  pointer is returned. Drivers are
+ * required to always check the value returned by this function.
+ */
+struct media_entity_context *
+media_device_get_entity_context(struct media_device_context *mdev_context,
+				struct media_entity *entity);
+
+/*------------------------------------------------------------------------------
+ * Media entity handling
+ */
+
 /**
  * media_device_register_entity() - registers a media entity inside a
  *	previously registered media device.
diff --git a/include/media/media-fh.h b/include/media/media-fh.h
index 6f00744b81d6000a4b0c503fe6968dd7adcbb1c3..48ec266416dd288a008bc5f93db5eb7ec6b8859c 100644
--- a/include/media/media-fh.h
+++ b/include/media/media-fh.h
@@ -13,15 +13,20 @@
 
 #include <media/media-devnode.h>
 
+struct media_device_context;
+
 /**
  * struct media_device_fh - File handle specific information on MC
  *
  * @fh: The media device file handle
  * @mdev_list: This file handle in media device's list of file handles
+ * @context: The media device context associated with the file handle
  */
 struct media_device_fh {
 	struct media_devnode_fh fh;
 	struct list_head mdev_list;
+
+	struct media_device_context *context;
 };
 
 static inline struct media_device_fh *media_device_fh(struct file *filp)

-- 
2.49.0

[PATCH 05/26] media: v4l2-dev: Introduce video device context

[Jacopo Mondi]

Introduce a new type in video-device that represents a video device
context. It extends 'struct media_entity_context' to include
video-device specific data and configuration. The new type is intended
to be extended by drivers that can store driver-specific information
in their video device derived types.

The next patch will introduce the VIDIOC_BIND_CONTEXT ioctl that allows
to create a video device context and uniquely associate it with a media
context.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-dev.c |  43 +++++++++++++
 include/media/v4l2-dev.h           | 126 +++++++++++++++++++++++++++++++++++++
 include/media/v4l2-fh.h            |   3 +
 3 files changed, 172 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index c369235113d98ae26c30a1aa386e7d60d541a66e..c83c37843c9e7beb899a4b2bd176273c3dec381b 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -1200,6 +1200,49 @@ struct media_pipeline *video_device_pipeline(struct video_device *vdev)
 }
 EXPORT_SYMBOL_GPL(video_device_pipeline);
 
+struct video_device_context *
+video_device_context_get(struct media_device_context *mdev_context,
+			 struct video_device *vdev)
+{
+	struct media_entity *entity = &vdev->entity;
+	struct media_entity_context *ctx =
+		media_device_get_entity_context(mdev_context, entity);
+
+	if (!ctx)
+		return NULL;
+
+	return container_of(ctx, struct video_device_context, base);
+}
+EXPORT_SYMBOL_GPL(video_device_context_get);
+
+void video_device_context_put(struct video_device_context *ctx)
+{
+	if (!ctx)
+		return;
+
+	media_entity_context_put(&ctx->base);
+}
+EXPORT_SYMBOL_GPL(video_device_context_put);
+
+int video_device_init_context(struct video_device *vdev,
+			      struct video_device_context *ctx)
+{
+	media_entity_init_context(&vdev->entity, &ctx->base);
+
+	ctx->vdev = vdev;
+	mutex_init(&ctx->queue_lock);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(video_device_init_context);
+
+void video_device_cleanup_context(struct video_device_context *ctx)
+{
+	mutex_destroy(&ctx->queue_lock);
+	media_entity_cleanup_context(&ctx->base);
+}
+EXPORT_SYMBOL_GPL(video_device_cleanup_context);
+
 #endif /* CONFIG_MEDIA_CONTROLLER */
 
 /*
diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
index a69801274800f4f9add723b51fe0a31331e88f97..9276e095fb17414a9eb7845db0aa81572c42ca45 100644
--- a/include/media/v4l2-dev.h
+++ b/include/media/v4l2-dev.h
@@ -18,6 +18,7 @@
 #include <linux/videodev2.h>
 
 #include <media/media-entity.h>
+#include <media/videobuf2-core.h>
 
 #define VIDEO_MAJOR	81
 
@@ -660,6 +661,131 @@ __must_check int video_device_pipeline_alloc_start(struct video_device *vdev);
  */
 struct media_pipeline *video_device_pipeline(struct video_device *vdev);
 
+/**
+ * struct video_device_context - The video device context
+ * @base: The media entity context base class member
+ * @vdev: The video device this context belongs to
+ * @queue_lock: Protects the vb2 queue
+ * @queue: The vb2 queue
+ *
+ * This structure represents an isolated execution context of a video device.
+ * This type 'derives' the base 'struct media_entity_context' type which
+ * implements refcounting on our behalf and allows instances of this type to be
+ * linked in the media_device_context contexts list.
+ *
+ * By storing the data and the configuration of a video device in a per-file
+ * handle context, userspace is allowed to multiplex the usage of a single video
+ * device devnode by opening it multiple times and by associating it with a
+ * media device context. This operation is called 'bounding' and is performed
+ * using the VIDIOC_BIND_CONTEXT ioctl.
+ *
+ * A video device context is created and stored in the v4l2-fh file handle
+ * associated with an open file descriptor when a video device is 'bound' to a
+ * media device context. The 'bounding' operation realizes a permanent
+ * association valid until the video device context is released.
+ *
+ * A video device can be bound to the same media device context once only.
+ * Trying to bind the same video device to the same media device context a
+ * second time, without releasing the already established context by closing the
+ * bound file descriptor first, will result in an error.
+ *
+ * To create a video device context userspace shall use the VIDIOC_BIND_CONTEXT
+ * ioctl that creates the video device context and uniquely associates it with a
+ * media device file descriptor.
+ *
+ * Once a video device file descriptor has been bound to a media device context,
+ * all the operations performed on the video device file descriptor will be
+ * directed on the just created video device context. This means, in example,
+ * that the video device format and the buffer queue are isolated from the ones
+ * associated with a different file descriptor obtained by opening again the
+ * same video device devnode but bound to a different media device context.
+ *
+ * Drivers that implement multiplexing support have to provide a valid
+ * implementation of the context-related operations in the
+ * media entity operations.
+ *
+ * Drivers are allowed to sub-class the video_device_context structure by
+ * defining a driver-specific type which embeds a struct video_device_context
+ * instance as first member, and allocate the driver-specific structure size in
+ * their implementation of the `alloc_context` operation.
+ *
+ * Video device contexts are ref-counted by embedding an instance of 'struct
+ * media_entity_context' and are freed once all the references to it are
+ * released.
+ *
+ * A video device context ref-count is increased when:
+ * - The context is created by bounding a video device to a media device context
+ * - The media pipeline starts streaming
+ * A video device context ref-count is decreased when:
+ * - The associated file handle is closed
+ * - The media pipeline stops streaming
+ *
+ * The ref-count is increased by a call to video_device_context_get() and is
+ * reponsibility of the caller to decrease the reference count with a call to
+ * video_device_context_put().
+ */
+struct video_device_context {
+	struct media_entity_context base;
+
+	struct video_device *vdev;
+	/* Protects the vb2 queue. */
+	struct mutex queue_lock;
+	struct vb2_queue queue;
+};
+
+/**
+ * video_device_context_get - Helper to get a video device context from a
+ *			      media device context
+ *
+ * @mdev_context: The media device context
+ * @vdev: The video device the context refers to
+ *
+ * Helper function that wraps media_device_get_entity_context() and returns
+ * the video device context associated with a video device in a media device
+ * context.
+ *
+ * The reference count of the returned video device context is increased.
+ * Callers of this function are required to decrease the reference count of
+ * the context reference with a call to video_device_context_put().
+ */
+struct video_device_context *
+video_device_context_get(struct media_device_context *mdev_context,
+			 struct video_device *vdev);
+
+/**
+ * video_device_context_put - Helper to decrease a video context reference
+ *			      count
+ *
+ * @ctx: The video context to release
+ */
+void video_device_context_put(struct video_device_context *ctx);
+
+/**
+ * video_device_init_context - Initialize the video device context
+ *
+ * @vdev: The video device this context belongs to
+ * @ctx: The context to initialize
+ *
+ * Initialize the video device context. The intended callers of this function
+ * are driver-specific implementations of the media_entity_ops.alloc_context()
+ * function that allocates their driver specific types that derive from
+ * struct video_device_context.
+ */
+int video_device_init_context(struct video_device *vdev,
+			      struct video_device_context *ctx);
+
+/**
+ * video_device_cleanup_context - Cleanup the video device context
+ *
+ * @ctx: The context to cleanup.
+ *
+ * Cleanup the video device context. The intended callers of this function are
+ * driver specific implementation of the media_entity_ops.destroy_context()
+ * function before releasing the memory previously allocated by
+ * media_entity_ops.alloc_context().
+ */
+void video_device_cleanup_context(struct video_device_context *ctx);
+
 #endif /* CONFIG_MEDIA_CONTROLLER */
 
 #endif /* _V4L2_DEV_H */
diff --git a/include/media/v4l2-fh.h b/include/media/v4l2-fh.h
index b5b3e00c8e6a0b082d9cd8a0c972a5094adcb6f2..a8de8613a026589ede94dc3e70c8a49ae08f4582 100644
--- a/include/media/v4l2-fh.h
+++ b/include/media/v4l2-fh.h
@@ -20,6 +20,7 @@
 
 struct video_device;
 struct v4l2_ctrl_handler;
+struct video_device_context;
 
 /**
  * struct v4l2_fh - Describes a V4L2 file handler
@@ -38,6 +39,7 @@ struct v4l2_ctrl_handler;
  * @sequence: event sequence number
  *
  * @m2m_ctx: pointer to &struct v4l2_m2m_ctx
+ * @context: The video device context
  */
 struct v4l2_fh {
 	struct list_head	list;
@@ -54,6 +56,7 @@ struct v4l2_fh {
 	u32			sequence;
 
 	struct v4l2_m2m_ctx	*m2m_ctx;
+	struct video_device_context *context;
 };
 
 /**

-- 
2.49.0

[PATCH 06/26] media: v4l2-ioctl: Introduce VIDIOC_BIND_CONTEXT

[Jacopo Mondi]

Introduce a new ioctl in V4L2 to allocate a video device context and
associate it with a media device context.

The ioctl is valid only if support for MEDIA_CONTROLLER is compiled in
as it calls into the entity ops to let driver allocate a new context and
binds the newly created context with the media context associated
with the file descriptor provided by userspace as the new ioctl
argument.

The newly allocated video context is then stored in the v4l2-fh that
represent the open file handle on which the ioctl has been called.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-dev.c   | 10 ++++++
 drivers/media/v4l2-core/v4l2-fh.c    |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c | 64 ++++++++++++++++++++++++++++++++++++
 include/media/v4l2-ioctl.h           |  7 ++++
 include/uapi/linux/videodev2.h       | 11 +++++++
 5 files changed, 93 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index c83c37843c9e7beb899a4b2bd176273c3dec381b..bc6502b4ce21cc0ad53136e1637d1c926e31dd89 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -606,6 +606,10 @@ static void determine_valid_ioctls(struct video_device *vdev)
 		__set_bit(_IOC_NR(VIDIOC_ENUM_FREQ_BANDS), valid_ioctls);
 
 	if (is_vid) {
+#ifdef CONFIG_MEDIA_CONTROLLER
+		__set_bit(_IOC_NR(VIDIOC_BIND_CONTEXT), valid_ioctls);
+#endif
+
 		/* video specific ioctls */
 		if ((is_rx && (ops->vidioc_enum_fmt_vid_cap ||
 			       ops->vidioc_enum_fmt_vid_overlay)) ||
@@ -661,12 +665,18 @@ static void determine_valid_ioctls(struct video_device *vdev)
 		SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_meta_cap);
 		SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_meta_cap);
 		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_meta_cap);
+#ifdef CONFIG_MEDIA_CONTROLLER
+		__set_bit(_IOC_NR(VIDIOC_BIND_CONTEXT), valid_ioctls);
+#endif
 	} else if (is_meta && is_tx) {
 		/* metadata output specific ioctls */
 		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_meta_out);
 		SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_meta_out);
 		SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_meta_out);
 		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_meta_out);
+#ifdef CONFIG_MEDIA_CONTROLLER
+		__set_bit(_IOC_NR(VIDIOC_BIND_CONTEXT), valid_ioctls);
+#endif
 	}
 	if (is_vbi) {
 		/* vbi specific ioctls */
diff --git a/drivers/media/v4l2-core/v4l2-fh.c b/drivers/media/v4l2-core/v4l2-fh.c
index 90eec79ee995a2d214590beeacc91b9f8f33236d..f7af444d2344541ccae1eae230b39d4cbc47f6bd 100644
--- a/drivers/media/v4l2-core/v4l2-fh.c
+++ b/drivers/media/v4l2-core/v4l2-fh.c
@@ -93,6 +93,7 @@ int v4l2_fh_release(struct file *filp)
 	struct v4l2_fh *fh = filp->private_data;
 
 	if (fh) {
+		video_device_context_put(fh->context);
 		v4l2_fh_del(fh);
 		v4l2_fh_exit(fh);
 		kfree(fh);
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 46da373066f4ec786b87ef18b8372abee621332f..bade64cc62b66dd6237ccd5338aa6dd8ab00ef8c 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -9,6 +9,7 @@
  */
 
 #include <linux/compat.h>
+#include <linux/file.h>
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/slab.h>
@@ -350,6 +351,13 @@ static void v4l_print_format(const void *arg, bool write_only)
 	}
 }
 
+static void v4l_print_context(const void *arg, bool write_only)
+{
+	const struct v4l2_context *c = arg;
+
+	pr_cont("context=%u\n", c->context_fd);
+}
+
 static void v4l_print_framebuffer(const void *arg, bool write_only)
 {
 	const struct v4l2_framebuffer *p = arg;
@@ -2151,6 +2159,61 @@ static int v4l_overlay(const struct v4l2_ioctl_ops *ops,
 	return ops->vidioc_overlay(file, fh, *(unsigned int *)arg);
 }
 
+static int v4l_bind_context(const struct v4l2_ioctl_ops *ops,
+			    struct file *file, void *fh, void *arg)
+{
+	struct video_device *vdev = video_devdata(file);
+	struct media_device_context *mdev_context;
+	struct v4l2_fh *vfh =
+		test_bit(V4L2_FL_USES_V4L2_FH, &vdev->flags) ? fh : NULL;
+	struct v4l2_context *c = arg;
+	int ret;
+
+	/*
+	 * TODO: do not __set_bit(_IOC_NR(VIDIOC_BIND_CONTEXT), valid_ioctls)
+	 * if V4L2_FL_USES_V4L2_FH isn't set or the driver does not implement
+	 * alloc_context and destroy_context.
+	 */
+	if (!vfh)
+		return -ENOTTY;
+
+	if (!vdev->entity.ops || !vdev->entity.ops->alloc_context ||
+	    !vdev->entity.ops->destroy_context)
+		return -ENOTTY;
+
+	mdev_context = media_device_context_get_from_fd(c->context_fd);
+	if (!mdev_context)
+		return -EINVAL;
+
+	/* Let the driver allocate the per-file handle context. */
+	ret = vdev->entity.ops->alloc_context(&vdev->entity,
+					      (struct media_entity_context **)
+					      &vfh->context);
+	if (ret)
+		goto err_put_mdev_context;
+
+	/*
+	 * Bind the newly created video device context to the media device
+	 * context identified by the file descriptor.
+	 */
+	ret = media_device_bind_context(mdev_context,
+					(struct media_entity_context *)
+					vfh->context);
+	if (ret)
+		goto err_put_context;
+
+	media_device_context_put(mdev_context);
+
+	return 0;
+
+err_put_context:
+	video_device_context_put(vfh->context);
+err_put_mdev_context:
+	media_device_context_put(mdev_context);
+
+	return ret;
+}
+
 static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -2998,6 +3061,7 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
 	IOCTL_INFO(VIDIOC_DBG_G_CHIP_INFO, v4l_dbg_g_chip_info, v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, match)),
 	IOCTL_INFO(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, v4l_print_query_ext_ctrl, INFO_FL_CTRL | INFO_FL_CLEAR(v4l2_query_ext_ctrl, id)),
 	IOCTL_INFO(VIDIOC_REMOVE_BUFS, v4l_remove_bufs, v4l_print_remove_buffers, INFO_FL_PRIO | INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_remove_buffers, type)),
+	IOCTL_INFO(VIDIOC_BIND_CONTEXT, v4l_bind_context, v4l_print_context, 0),
 };
 #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls)
 
diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
index 82695c3a300a73219f262fb556ed61a8f09d273e..6d9edfd9ca912972ad15acdc07014dee1ed36ab6 100644
--- a/include/media/v4l2-ioctl.h
+++ b/include/media/v4l2-ioctl.h
@@ -18,6 +18,7 @@
 #include <linux/videodev2.h>
 
 struct v4l2_fh;
+struct video_device_context;
 
 /**
  * struct v4l2_ioctl_ops - describe operations for each V4L2 ioctl
@@ -149,6 +150,8 @@ struct v4l2_fh;
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for metadata capture
  * @vidioc_try_fmt_meta_out: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for metadata output
+ * @vidioc_bind_context: pointer to the function that implements
+ *	:ref:`VIDIOC_BIND_CONTEXT <vidioc_bind_context>` ioctl
  * @vidioc_reqbufs: pointer to the function that implements
  *	:ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
  * @vidioc_querybuf: pointer to the function that implements
@@ -402,6 +405,10 @@ struct v4l2_ioctl_ops {
 	int (*vidioc_try_fmt_meta_out)(struct file *file, void *fh,
 				       struct v4l2_format *f);
 
+	/* Context handlers */
+	int (*vidioc_bind_context)(struct file *file, void *fh,
+				   struct video_device_context *c);
+
 	/* Buffer handlers */
 	int (*vidioc_reqbufs)(struct file *file, void *fh,
 			      struct v4l2_requestbuffers *b);
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 3dd9fa45dde1066d52a68581625a39e7ec92c9b7..0b9aa89e2479620dbbaa54f1aadff7aaa7a3d0f7 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -1057,6 +1057,14 @@ struct v4l2_jpegcompression {
 					* always use APP0 */
 };
 
+/*
+ *     V I D E O   D E V I C E   C O N T E X T
+ */
+
+struct v4l2_context {
+	__u32 context_fd;
+};
+
 /*
  *	M E M O R Y - M A P P I N G   B U F F E R S
  */
@@ -2818,6 +2826,9 @@ struct v4l2_remove_buffers {
 #define VIDIOC_REMOVE_BUFS	_IOWR('V', 104, struct v4l2_remove_buffers)
 
 
+/* Context handling */
+#define VIDIOC_BIND_CONTEXT	_IOW('V', 105, struct v4l2_context)
+
 /* Reminder: when adding new ioctls please add support for them to
    drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
 

-- 
2.49.0

[PATCH 07/26] media: Documentation: Add VIDIOC_BIND_CONTEXT

[Jacopo Mondi]

Document the newly introduced VIDIOC_BIND_CONTEXT ioctl.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 .../userspace-api/media/v4l/user-func.rst          |  1 +
 .../media/v4l/vidioc-bind-context.rst              | 80 ++++++++++++++++++++++
 2 files changed, 81 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
index 6f661138801cde2810997740ee8305085fe73a43..0d9aff56ab653b2a4f6afe4828f88bc5637addf1 100644
--- a/Documentation/userspace-api/media/v4l/user-func.rst
+++ b/Documentation/userspace-api/media/v4l/user-func.rst
@@ -12,6 +12,7 @@ Function Reference
 
     func-close
     func-ioctl
+    vidioc-bind-context
     vidioc-create-bufs
     vidioc-cropcap
     vidioc-dbg-g-chip-info
diff --git a/Documentation/userspace-api/media/v4l/vidioc-bind-context.rst b/Documentation/userspace-api/media/v4l/vidioc-bind-context.rst
new file mode 100644
index 0000000000000000000000000000000000000000..74c885169e22c495056a9364e722e2140d8f21c5
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-bind-context.rst
@@ -0,0 +1,80 @@
+.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. c:namespace:: V4L
+
+.. _vidioc_bind_context:
+
+*************************
+ioctl VIDIOC_BIND_CONTEXT
+*************************
+
+Name
+====
+
+VIDIOC_BIND_CONTEXT - Bind a video device file handle to a media device context
+
+Synopsis
+========
+
+.. c:macro:: VIDIOC_BIND_CONTEXT
+
+``int ioctl(int fd, VIDIOC_BIND_CONTEXT, struct v4l2_context *argp)``
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :c:func:`open()`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_context`.
+
+Description
+===========
+
+Applications call the ``VIDIOC_BIND_CONTEXT`` ioctl to bind a video device file
+handle to a media device  context. Binding a video device file handle to a media
+device context creates an isolated execution context which allows to multiplex
+the usage of a video device. This means, in practice, that the video device
+configuration (format, sizes etc) applied on a file handle bound to a media
+device context won't be visible on file handles bound to a different media
+device context (or not bound at all).
+
+By opening a media device applications create a media device context to which
+video devices and subdevices file handles can be bound to. The file descriptor
+returned by a call to :c:func:`open()` on the media device identifies uniquely
+the media device context. Application populates the ``context_fd`` field of
+:c:type:`v4l2_context` with the file descriptor of an open media device to
+identify the media context to which they want to bind a video device to.
+
+Applications can open a video device node multiple times, and call
+``VIDIOC_BIND_CONTEXT`` on each file handle returned by a successful call to
+:c:func:`open()` to isolate the operations performed on that file handle from
+any operation performed on other file handles bound to different contexts. This
+means, in example, that the video device format, sizes and the buffers are
+isolated from the ones associated with a file descriptor, obtained by opening
+the same video device but bound to a different media device context (or not
+bound at all).
+
+The bounding operation realizes a permanent association valid until the video
+device context is released by closing the file handle.
+
+A video device file handle can be bound to the same media device context once
+only. Trying to bind the same file handle to the same media device context a
+second time, without releasing the already established context by closing the
+bound file descriptor first, will result in an error.
+
+Bounding is an opt-in feature that applications are free to ignore. Any
+operation directed to a non bound file handle will continue to work as it used
+to, and the video device configuration (formats, sizes etc) will be visible
+across all the other non-bound file handles.
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+EINVAL
+    The media device context file handle ``context_fd`` is not valid or the
+    video device file handle is already bound to a context.

-- 
2.49.0

[PATCH 08/26] media: v4l2-dev: Introduce default contexts

[Jacopo Mondi]

Introduce the media device and video device default contexts.

Drivers ported to use multi-context support that used to work with a
non-context aware userspace (which doesn't call VIDIOC_BIND_CONTEXT)
shall continue to work even if they are context aware.

Provide a default context in the media device and in the video device
structures and let drivers allocate and release them with the newly
introduced operations.

Bind the video device default context to the default context of the
media device associated with the v4l2_dev.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/mc/mc-device.c       | 11 +++++++++
 drivers/media/v4l2-core/v4l2-dev.c | 46 +++++++++++++++++++++++++++++++++++---
 include/media/media-device.h       |  5 +++++
 include/media/v4l2-dev.h           |  3 +++
 4 files changed, 62 insertions(+), 3 deletions(-)

diff --git a/drivers/media/mc/mc-device.c b/drivers/media/mc/mc-device.c
index d8f12db933d22ae7466051698d853f4bdc599400..e1f34f884fee2e8c3750f9c1e85142ff2f6b7bf0 100644
--- a/drivers/media/mc/mc-device.c
+++ b/drivers/media/mc/mc-device.c
@@ -746,12 +746,23 @@ void media_device_init(struct media_device *mdev)
 		media_set_bus_info(mdev->bus_info, sizeof(mdev->bus_info),
 				   mdev->dev);
 
+	mdev->default_context = NULL;
+	if (mdev->ops &&
+	    mdev->ops->alloc_context && mdev->ops->destroy_context) {
+		if (mdev->ops->alloc_context(mdev, &mdev->default_context)) {
+			dev_err(mdev->dev,
+				"Failed to initialize media device default context\n");
+			return;
+		}
+	}
+
 	dev_dbg(mdev->dev, "Media device initialized\n");
 }
 EXPORT_SYMBOL_GPL(media_device_init);
 
 void media_device_cleanup(struct media_device *mdev)
 {
+	media_device_context_put(mdev->default_context);
 	ida_destroy(&mdev->entity_internal_idx);
 	mdev->entity_internal_idx_max = 0;
 	media_graph_walk_cleanup(&mdev->pm_count_walk);
diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index bc6502b4ce21cc0ad53136e1637d1c926e31dd89..2d78096931efd88215bc847b105e198a54035546 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -31,6 +31,8 @@
 #include <media/v4l2-ioctl.h>
 #include <media/v4l2-event.h>
 
+#include <media/media-device.h>
+
 #define VIDEO_NUM_DEVICES	256
 #define VIDEO_NAME              "video4linux"
 
@@ -220,8 +222,13 @@ static void v4l2_device_release(struct device *cd)
 	if (v4l2_dev->release == NULL)
 		v4l2_dev = NULL;
 
-	/* Release video_device and perform other
-	   cleanups as needed. */
+	/* Release default context. */
+#ifdef CONFIG_MEDIA_CONTROLLER
+	video_device_context_put(vdev->default_context);
+#endif
+	vdev->default_context = NULL;
+
+	/* Release video_device and perform other cleanups as needed. */
 	vdev->release(vdev);
 
 	/* Decrease v4l2_device refcount */
@@ -1086,7 +1093,36 @@ int __video_register_device(struct video_device *vdev,
 	/* Part 5: Register the entity. */
 	ret = video_register_media_controller(vdev);
 
-	/* Part 6: Activate this minor. The char device can now be used. */
+	/*
+	 * Part 6: Complete the video device registration by initializing the
+	 * default context. The defaul context serves for context-aware driver
+	 * to operate with a non-context-aware userspace that never creates
+	 * new contexts. If the video device driver is not context aware, it
+	 * will never implement 'context_ops' and will never use the default
+	 * context.
+	 */
+	vdev->default_context = NULL;
+#ifdef CONFIG_MEDIA_CONTROLLER
+	if (vdev->entity.ops && vdev->entity.ops->alloc_context &&
+	    vdev->entity.ops->destroy_context) {
+		ret = vdev->entity.ops->alloc_context(&vdev->entity,
+					(struct media_entity_context **)
+					&vdev->default_context);
+		if (ret) {
+			pr_err("%s: default context alloc failed\n", __func__);
+			goto cleanup;
+		}
+
+		ret = media_device_bind_context(vdev->v4l2_dev->mdev->default_context,
+						&vdev->default_context->base);
+		if (ret) {
+			pr_err("%s: default context bind failed\n", __func__);
+			goto cleanup;
+		}
+	}
+#endif
+
+	/* Part 7: Activate this minor. The char device can now be used. */
 	set_bit(V4L2_FL_REGISTERED, &vdev->flags);
 	mutex_unlock(&videodev_lock);
 
@@ -1094,6 +1130,10 @@ int __video_register_device(struct video_device *vdev,
 
 cleanup:
 	mutex_lock(&videodev_lock);
+#ifdef CONFIG_MEDIA_CONTROLLER
+	video_device_context_put(vdev->default_context);
+#endif
+	vdev->default_context = NULL;
 	if (vdev->cdev)
 		cdev_del(vdev->cdev);
 	video_devices[vdev->minor] = NULL;
diff --git a/include/media/media-device.h b/include/media/media-device.h
index 2ea8fce9ba75700286961f1622584372a954cb8a..b3cc6793a8b5eff4c26e57b01e1a62ab71e8195b 100644
--- a/include/media/media-device.h
+++ b/include/media/media-device.h
@@ -184,6 +184,9 @@ struct media_device_ops {
  * @fh_list:	List of file handles in the media device
  *		(struct media_device_fh.mdev_list).
  * @fh_list_lock: Serialise access to fh_list list.
+ * @default_context: The default video device context. Used by drivers that
+ *		     support multi-context operation when operated by a
+ *		     non-context aware userspace.
  *
  * This structure represents an abstract high-level media device. It allows easy
  * access to entities and provides basic media device-level support. The
@@ -260,6 +263,8 @@ struct media_device {
 
 	struct list_head fh_list;
 	spinlock_t fh_list_lock;
+
+	struct media_device_context *default_context;
 };
 
 /* We don't need to include usb.h here */
diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
index 9276e095fb17414a9eb7845db0aa81572c42ca45..9e1cf58623acc4ab5f503a9663fd999b38130226 100644
--- a/include/media/v4l2-dev.h
+++ b/include/media/v4l2-dev.h
@@ -237,6 +237,8 @@ struct v4l2_file_operations {
  * @ctrl_handler: Control handler associated with this device node.
  *	 May be NULL.
  * @queue: &struct vb2_queue associated with this device node. May be NULL.
+ * @default_context: &struct video_device_context associated with this device
+ *		     node
  * @prio: pointer to &struct v4l2_prio_state with device's Priority state.
  *	 If NULL, then v4l2_dev->prio will be used.
  * @name: video device name
@@ -283,6 +285,7 @@ struct video_device {
 
 	struct vb2_queue *queue;
 
+	struct video_device_context *default_context;
 	struct v4l2_prio_state *prio;
 
 	/* device info */

-- 
2.49.0

[PATCH 09/26] media: v4l2-dev: Add video_device_context_from_file()

[Jacopo Mondi]

Introduce an helper function to get a video_device_context from file.
If no context has been bound to the file, return the video device
default context.

As a video device context lifetime is bound to the one of the file
handle it is associated with, and the intended user of this helper are
the v4l2_ioctl_ops handlers that operates on an open file handle, the
context returned by this function is guaranteed to be valid for the
whole function scope of the caller.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-dev.c | 15 +++++++++++++++
 include/media/v4l2-dev.h           | 30 ++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index 2d78096931efd88215bc847b105e198a54035546..96696959314abfb1864ea5d96742e579b5a41f6f 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -1180,6 +1180,21 @@ struct dentry *v4l2_debugfs_root(void)
 EXPORT_SYMBOL_GPL(v4l2_debugfs_root);
 #endif
 
+struct video_device_context *
+video_device_context_from_file(struct file *filp, struct video_device *vfd)
+{
+	struct v4l2_fh *vfh =
+		test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags) ? filp->private_data
+							    : NULL;
+
+	/* If the file handle has been bound to a context return it. */
+	if (vfh && vfh->context)
+		return vfh->context;
+
+	return vfd->default_context;
+}
+EXPORT_SYMBOL_GPL(video_device_context_from_file);
+
 #if defined(CONFIG_MEDIA_CONTROLLER)
 
 __must_check int video_device_pipeline_start(struct video_device *vdev,
diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
index 9e1cf58623acc4ab5f503a9663fd999b38130226..a41afb81663bc9cb3bfc06dcf9b11ceeaf7c3415 100644
--- a/include/media/v4l2-dev.h
+++ b/include/media/v4l2-dev.h
@@ -789,6 +789,36 @@ int video_device_init_context(struct video_device *vdev,
  */
 void video_device_cleanup_context(struct video_device_context *ctx);
 
+/**
+ * video_device_context_from_file - Get the video device context associated with
+ *				    an open file handle
+ * @filp: The file handle
+ * @vdev: The video device
+ *
+ * If a video device context has been bound to an open file handle by a call
+ * to the VIDIOC_BIND_CONTEXT ioctl this function returns it, otherwise returns
+ * the default video device context.
+ *
+ * The intended callers of this helper are the driver-specific v4l2_ioctl_ops
+ * handlers, which receive as first argument a file pointer. By using this
+ * helper drivers can get the context associated with such file, if any.
+ * Otherwise, if the video device  has not been bound to any media device
+ * context, the default video device context is returned.
+ *
+ * As video device contexts are reference counted and their lifetime is
+ * guaranteed to be at least the one of the file handle they are associated
+ * with, callers of this function are guaranteed to always receive a valid
+ * context reference by this function. The reference will remain valid for the
+ * whole function scope of the caller that has received the open file handle as
+ * argument. Likewise, accessing the media context from the video device context
+ * returned by this function is always safe within the same function scope.
+ *
+ * This function does not increase the reference count of the returned video
+ * device context.
+ */
+struct video_device_context *
+video_device_context_from_file(struct file *filp, struct video_device *vdev);
+
 #endif /* CONFIG_MEDIA_CONTROLLER */
 
 #endif /* _V4L2_DEV_H */

-- 
2.49.0

[PATCH 10/26] media: v4l2-dev: Add video_device_context_from_queue()

[Jacopo Mondi]

Introduce an helper function to get a video_device_context from
a videobuf2 queue.

Device drivers that support multi-context operations will receive in
their vb2_ops callbacks implementation a vb2_queue argument that is
embedded in a video device context, either a context created by binding
the video device to a media device context or the video device default
context. This helper allows those drivers to retrieve the context
the vb2_queue is embedded with.

As the intended callers of this helpers are vb2_ops callbacks
implementation, called by the videobuf2 core in response to an operation
on a file handle, the returned context is guaranteed to be valid for
the whole duration of the callback.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 include/media/v4l2-dev.h | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
index a41afb81663bc9cb3bfc06dcf9b11ceeaf7c3415..bab4b13b109362bec84d8d16440b6ea895206b60 100644
--- a/include/media/v4l2-dev.h
+++ b/include/media/v4l2-dev.h
@@ -819,6 +819,42 @@ void video_device_cleanup_context(struct video_device_context *ctx);
 struct video_device_context *
 video_device_context_from_file(struct file *filp, struct video_device *vdev);
 
+/**
+ * video_device_context_from_queue - Get a video device context associated with
+ *				     a vb2 queue
+ *
+ * @q: the videobuf2 queue
+ *
+ * Return the video device context this videobuf2 queue belongs to.
+ *
+ * Device drivers that support multi-context operations will always be provided
+ * with a device context to work with, either a context created by userspace
+ * by binding the video device to a media device context or the video device
+ * default context. The videobuf2 queue that context-aware drivers operates will
+ * always be contained in either one of those contexts.
+ *
+ * This function should be used by driver-specific callbacks implementation of
+ * the vb2_ops which receive a videobuf2 as argument. Being the vb2_ops callback
+ * called by the videobuf2 core in response to a userspace operation on an open
+ * file handle, callers of this function are guaranteed to always receive a
+ * valid context reference by this function. The reference will remain valid for
+ * the whole function scope of the vb2_ops callback that has received the
+ * videobuf2 queue as argument. Likewise, accessing the media device context
+ * from the video device context returned by this function is always safe within
+ * the same function scope.
+ *
+ * Drivers that do not support multi-context operations shall never use
+ * this function.
+ *
+ * This function does not increase the reference count of the returned video
+ * device context.
+ */
+static inline struct video_device_context *
+video_device_context_from_queue(struct vb2_queue *q)
+{
+	return container_of(q, struct video_device_context, queue);
+}
+
 #endif /* CONFIG_MEDIA_CONTROLLER */
 
 #endif /* _V4L2_DEV_H */

-- 
2.49.0

[PATCH 11/26] media: videobuf2-v4l2: Support vb2_queue embedded in a context

[Jacopo Mondi]

Support device drivers that implement multi-context operations in the
videobuf2 core by providing an helper to retrieve the vb2_queue from
the context associated with an open file handle. If no context is
associated with a file handle, retrieve it from the video device
default context, created by the core for multi-context aware drivers.

Fall-back to use the vb2_queue from the video_device to support existing
drivers which are not context aware.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/common/videobuf2/videobuf2-v4l2.c | 135 +++++++++++++++---------
 1 file changed, 84 insertions(+), 51 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 1cd26faee50338aefeb670c6865da7c2d43f44d3..01269e03102f9ed7731732cc5d3444664eaa7bec 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -998,23 +998,42 @@ EXPORT_SYMBOL_GPL(vb2_poll);
  * and so they simplify the driver code.
  */
 
+/*
+ * Helper to get the vb2 queue either from:
+ * 1) The video context bound to the open file handle
+ * 2) The default context for context-aware drivers if userspace has not bound
+ *    a context to the file handle
+ * 3) From the video device for non-context aware drivers
+ */
+static struct vb2_queue *get_vb2_queue(struct file *file,
+				       struct video_device *vdev)
+{
+	struct video_device_context *ctx =
+				video_device_context_from_file(file, vdev);
+
+	return ctx ? &ctx->queue
+		   : vdev->default_context ? &vdev->default_context->queue
+		   : vdev->queue;
+}
+
 /* vb2 ioctl helpers */
 
 int vb2_ioctl_remove_bufs(struct file *file, void *priv,
 			  struct v4l2_remove_buffers *d)
 {
 	struct video_device *vdev = video_devdata(file);
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
 
-	if (vdev->queue->type != d->type)
+	if (q->type != d->type)
 		return -EINVAL;
 
 	if (d->count == 0)
 		return 0;
 
-	if (vb2_queue_is_busy(vdev->queue, file))
+	if (vb2_queue_is_busy(q, file))
 		return -EBUSY;
 
-	return vb2_core_remove_bufs(vdev->queue, d->index, d->count);
+	return vb2_core_remove_bufs(q, d->index, d->count);
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_remove_bufs);
 
@@ -1022,21 +1041,21 @@ int vb2_ioctl_reqbufs(struct file *file, void *priv,
 			  struct v4l2_requestbuffers *p)
 {
 	struct video_device *vdev = video_devdata(file);
-	int res = vb2_verify_memory_type(vdev->queue, p->memory, p->type);
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
+	int res = vb2_verify_memory_type(q, p->memory, p->type);
 	u32 flags = p->flags;
 
-	vb2_set_flags_and_caps(vdev->queue, p->memory, &flags,
-			       &p->capabilities, NULL);
+	vb2_set_flags_and_caps(q, p->memory, &flags, &p->capabilities, NULL);
 	p->flags = flags;
 	if (res)
 		return res;
-	if (vb2_queue_is_busy(vdev->queue, file))
+	if (vb2_queue_is_busy(q, file))
 		return -EBUSY;
-	res = vb2_core_reqbufs(vdev->queue, p->memory, p->flags, &p->count);
+	res = vb2_core_reqbufs(q, p->memory, p->flags, &p->count);
 	/* If count == 0, then the owner has released all buffers and he
 	   is no longer owner of the queue. Otherwise we have a new owner. */
 	if (res == 0)
-		vdev->queue->owner = p->count ? file->private_data : NULL;
+		q->owner = p->count ? file->private_data : NULL;
 	return res;
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_reqbufs);
@@ -1045,11 +1064,12 @@ int vb2_ioctl_create_bufs(struct file *file, void *priv,
 			  struct v4l2_create_buffers *p)
 {
 	struct video_device *vdev = video_devdata(file);
-	int res = vb2_verify_memory_type(vdev->queue, p->memory, p->format.type);
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
+	int res = vb2_verify_memory_type(q, p->memory, p->format.type);
 
-	p->index = vb2_get_num_buffers(vdev->queue);
-	vb2_set_flags_and_caps(vdev->queue, p->memory, &p->flags,
-			       &p->capabilities, &p->max_num_buffers);
+	p->index = vb2_get_num_buffers(q);
+	vb2_set_flags_and_caps(q, p->memory, &p->flags, &p->capabilities,
+			       &p->max_num_buffers);
 	/*
 	 * If count == 0, then just check if memory and type are valid.
 	 * Any -EBUSY result from vb2_verify_memory_type can be mapped to 0.
@@ -1058,12 +1078,12 @@ int vb2_ioctl_create_bufs(struct file *file, void *priv,
 		return res != -EBUSY ? res : 0;
 	if (res)
 		return res;
-	if (vb2_queue_is_busy(vdev->queue, file))
+	if (vb2_queue_is_busy(q, file))
 		return -EBUSY;
 
-	res = vb2_create_bufs(vdev->queue, p);
+	res = vb2_create_bufs(q, p);
 	if (res == 0)
-		vdev->queue->owner = file->private_data;
+		q->owner = file->private_data;
 	return res;
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_create_bufs);
@@ -1072,69 +1092,76 @@ int vb2_ioctl_prepare_buf(struct file *file, void *priv,
 			  struct v4l2_buffer *p)
 {
 	struct video_device *vdev = video_devdata(file);
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
 
-	if (vb2_queue_is_busy(vdev->queue, file))
+	if (vb2_queue_is_busy(q, file))
 		return -EBUSY;
-	return vb2_prepare_buf(vdev->queue, vdev->v4l2_dev->mdev, p);
+	return vb2_prepare_buf(q, vdev->v4l2_dev->mdev, p);
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_prepare_buf);
 
 int vb2_ioctl_querybuf(struct file *file, void *priv, struct v4l2_buffer *p)
 {
 	struct video_device *vdev = video_devdata(file);
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
 
 	/* No need to call vb2_queue_is_busy(), anyone can query buffers. */
-	return vb2_querybuf(vdev->queue, p);
+	return vb2_querybuf(q, p);
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_querybuf);
 
 int vb2_ioctl_qbuf(struct file *file, void *priv, struct v4l2_buffer *p)
 {
 	struct video_device *vdev = video_devdata(file);
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
 
-	if (vb2_queue_is_busy(vdev->queue, file))
+	if (vb2_queue_is_busy(q, file))
 		return -EBUSY;
-	return vb2_qbuf(vdev->queue, vdev->v4l2_dev->mdev, p);
+	return vb2_qbuf(q, vdev->v4l2_dev->mdev, p);
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_qbuf);
 
 int vb2_ioctl_dqbuf(struct file *file, void *priv, struct v4l2_buffer *p)
 {
 	struct video_device *vdev = video_devdata(file);
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
 
-	if (vb2_queue_is_busy(vdev->queue, file))
+	if (vb2_queue_is_busy(q, file))
 		return -EBUSY;
-	return vb2_dqbuf(vdev->queue, p, file->f_flags & O_NONBLOCK);
+	return vb2_dqbuf(q, p, file->f_flags & O_NONBLOCK);
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_dqbuf);
 
 int vb2_ioctl_streamon(struct file *file, void *priv, enum v4l2_buf_type i)
 {
 	struct video_device *vdev = video_devdata(file);
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
 
-	if (vb2_queue_is_busy(vdev->queue, file))
+	if (vb2_queue_is_busy(q, file))
 		return -EBUSY;
-	return vb2_streamon(vdev->queue, i);
+	return vb2_streamon(q, i);
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_streamon);
 
 int vb2_ioctl_streamoff(struct file *file, void *priv, enum v4l2_buf_type i)
 {
 	struct video_device *vdev = video_devdata(file);
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
 
-	if (vb2_queue_is_busy(vdev->queue, file))
+	if (vb2_queue_is_busy(q, file))
 		return -EBUSY;
-	return vb2_streamoff(vdev->queue, i);
+	return vb2_streamoff(q, i);
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_streamoff);
 
 int vb2_ioctl_expbuf(struct file *file, void *priv, struct v4l2_exportbuffer *p)
 {
 	struct video_device *vdev = video_devdata(file);
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
 
-	if (vb2_queue_is_busy(vdev->queue, file))
+	if (vb2_queue_is_busy(q, file))
 		return -EBUSY;
-	return vb2_expbuf(vdev->queue, p);
+	return vb2_expbuf(q, p);
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_expbuf);
 
@@ -1143,20 +1170,22 @@ EXPORT_SYMBOL_GPL(vb2_ioctl_expbuf);
 int vb2_fop_mmap(struct file *file, struct vm_area_struct *vma)
 {
 	struct video_device *vdev = video_devdata(file);
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
 
-	return vb2_mmap(vdev->queue, vma);
+	return vb2_mmap(q, vma);
 }
 EXPORT_SYMBOL_GPL(vb2_fop_mmap);
 
 int _vb2_fop_release(struct file *file, struct mutex *lock)
 {
 	struct video_device *vdev = video_devdata(file);
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
 
 	if (lock)
 		mutex_lock(lock);
-	if (!vdev->queue->owner || file->private_data == vdev->queue->owner) {
-		vb2_queue_release(vdev->queue);
-		vdev->queue->owner = NULL;
+	if (!q->owner || file->private_data == q->owner) {
+		vb2_queue_release(q);
+		q->owner = NULL;
 	}
 	if (lock)
 		mutex_unlock(lock);
@@ -1167,7 +1196,8 @@ EXPORT_SYMBOL_GPL(_vb2_fop_release);
 int vb2_fop_release(struct file *file)
 {
 	struct video_device *vdev = video_devdata(file);
-	struct mutex *lock = vdev->queue->lock ? vdev->queue->lock : vdev->lock;
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
+	struct mutex *lock = q->lock ? q->lock : vdev->lock;
 
 	return _vb2_fop_release(file, lock);
 }
@@ -1177,19 +1207,20 @@ ssize_t vb2_fop_write(struct file *file, const char __user *buf,
 		size_t count, loff_t *ppos)
 {
 	struct video_device *vdev = video_devdata(file);
-	struct mutex *lock = vdev->queue->lock ? vdev->queue->lock : vdev->lock;
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
+	struct mutex *lock = q->lock ? q->lock : vdev->lock;
 	int err = -EBUSY;
 
-	if (!(vdev->queue->io_modes & VB2_WRITE))
+	if (!(q->io_modes & VB2_WRITE))
 		return -EINVAL;
 	if (lock && mutex_lock_interruptible(lock))
 		return -ERESTARTSYS;
-	if (vb2_queue_is_busy(vdev->queue, file))
+	if (vb2_queue_is_busy(q, file))
 		goto exit;
-	err = vb2_write(vdev->queue, buf, count, ppos,
-		       file->f_flags & O_NONBLOCK);
-	if (vdev->queue->fileio)
-		vdev->queue->owner = file->private_data;
+	err = vb2_write(q, buf, count, ppos,
+			file->f_flags & O_NONBLOCK);
+	if (q->fileio)
+		q->owner = file->private_data;
 exit:
 	if (lock)
 		mutex_unlock(lock);
@@ -1201,20 +1232,21 @@ ssize_t vb2_fop_read(struct file *file, char __user *buf,
 		size_t count, loff_t *ppos)
 {
 	struct video_device *vdev = video_devdata(file);
-	struct mutex *lock = vdev->queue->lock ? vdev->queue->lock : vdev->lock;
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
+	struct mutex *lock = q->lock ? q->lock : vdev->lock;
 	int err = -EBUSY;
 
-	if (!(vdev->queue->io_modes & VB2_READ))
+	if (!(q->io_modes & VB2_READ))
 		return -EINVAL;
 	if (lock && mutex_lock_interruptible(lock))
 		return -ERESTARTSYS;
-	if (vb2_queue_is_busy(vdev->queue, file))
+	if (vb2_queue_is_busy(q, file))
 		goto exit;
-	vdev->queue->owner = file->private_data;
-	err = vb2_read(vdev->queue, buf, count, ppos,
+	q->owner = file->private_data;
+	err = vb2_read(q, buf, count, ppos,
 		       file->f_flags & O_NONBLOCK);
-	if (!vdev->queue->fileio)
-		vdev->queue->owner = NULL;
+	if (!q->fileio)
+		q->owner = NULL;
 exit:
 	if (lock)
 		mutex_unlock(lock);
@@ -1225,7 +1257,7 @@ EXPORT_SYMBOL_GPL(vb2_fop_read);
 __poll_t vb2_fop_poll(struct file *file, poll_table *wait)
 {
 	struct video_device *vdev = video_devdata(file);
-	struct vb2_queue *q = vdev->queue;
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
 	struct mutex *lock = q->lock ? q->lock : vdev->lock;
 	__poll_t res;
 	void *fileio;
@@ -1241,7 +1273,7 @@ __poll_t vb2_fop_poll(struct file *file, poll_table *wait)
 
 	fileio = q->fileio;
 
-	res = vb2_poll(vdev->queue, file, wait);
+	res = vb2_poll(q, file, wait);
 
 	/* If fileio was started, then we have a new queue owner. */
 	if (!fileio && q->fileio)
@@ -1257,8 +1289,9 @@ unsigned long vb2_fop_get_unmapped_area(struct file *file, unsigned long addr,
 		unsigned long len, unsigned long pgoff, unsigned long flags)
 {
 	struct video_device *vdev = video_devdata(file);
+	struct vb2_queue *q = get_vb2_queue(file, vdev);
 
-	return vb2_get_unmapped_area(vdev->queue, addr, len, pgoff, flags);
+	return vb2_get_unmapped_area(q, addr, len, pgoff, flags);
 }
 EXPORT_SYMBOL_GPL(vb2_fop_get_unmapped_area);
 #endif

-- 
2.49.0

[PATCH 12/26] media: v4l2-subdev: Introduce v4l2 subdev context

[Jacopo Mondi]

Introduce a new type in v4l2 subdev that represents a v4l2 subdevice
contex. It extends 'struct media_entity_context' and is intended to be
extended by drivers that can store driver-specific information
in their derived types.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c |  39 +++++++++++
 include/media/v4l2-subdev.h           | 126 ++++++++++++++++++++++++++++++++++
 2 files changed, 165 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 4fd25fea3b58477056729665706ddbacc436379c..7307f57439499c8d5360c89f492944828ac23973 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1577,6 +1577,45 @@ bool v4l2_subdev_has_pad_interdep(struct media_entity *entity,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_has_pad_interdep);
 
+struct v4l2_subdev_context *
+v4l2_subdev_context_get(struct media_device_context *mdev_context,
+			struct v4l2_subdev *sd)
+{
+	struct media_entity *entity = &sd->entity;
+	struct media_entity_context *ctx =
+		 media_device_get_entity_context(mdev_context, entity);
+
+	if (!ctx)
+		return NULL;
+
+	return container_of(ctx, struct v4l2_subdev_context, base);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_context_get);
+
+void v4l2_subdev_context_put(struct v4l2_subdev_context *ctx)
+{
+	if (!ctx)
+		return;
+
+	media_entity_context_put(&ctx->base);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_context_put);
+
+int v4l2_subdev_init_context(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_context *context)
+{
+	media_entity_init_context(&sd->entity, &context->base);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_init_context);
+
+void v4l2_subdev_cleanup_context(struct v4l2_subdev_context *context)
+{
+	media_entity_cleanup_context(&context->base);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup_context);
+
 struct v4l2_subdev_state *
 __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
 			  struct lock_class_key *lock_key)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 5dcf4065708f32e7d3b5da003771810d5f7973b8..9d257b859acafb11cfe6976e906e7baabd0206f6 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -757,6 +757,78 @@ struct v4l2_subdev_state {
 	struct v4l2_subdev_stream_configs stream_configs;
 };
 
+/**
+ * struct v4l2_subdev_context - The v4l2 subdevice context
+ * @base: The media entity context base class member
+ * @state: The subdevice state associated with this context
+ *
+ * This structure represents an isolated execution context of a subdevice.
+ * This type 'derives' the base 'struct media_entity_context' type which
+ * implements refcounting on our behalf and allows instances of this type to be
+ * linked in the media_device_context contexts list.
+ *
+ * The subdevice context stores the subdev state in a per-file handle context,
+ * userspace is allowed to multiplex the usage of a subdevice devnode by opening
+ * it multiple times and by associating it with a media device context. This
+ * operation is called 'bounding' and is performed using the
+ * VIDIOC_SUBDEV_BIND_CONTEXT ioctl.
+ *
+ * A subdevice context is created and stored in the v4l2_fh file handle
+ * associated with an open file descriptor when a subdevice is 'bound' to a
+ * media device context. The 'bounding' operation realizes a permanent
+ * association valid until the subdevice context is released.
+ *
+ * A subdevice can be bound to the same media device context once only.
+ * Trying to bind the same subdevice to the same media device context a
+ * second time, without releasing the already established context by closing the
+ * bound file descriptor first, will result in an error.
+ *
+ * To create a subdevice context userspace shall use the
+ * VIDIOC_SUBDEV_BIND_CONTEXT ioctl that creates the subdevice context and
+ * uniquely associates it with a media device file descriptor.
+ *
+ * Once a subdevice file descriptor has been bound to a media device context,
+ * all the operations performed on the subdevice file descriptor will be
+ * directed on the just created subdevice context. This means, in example, that
+ * the subdevice state and configuration is isolated from the ones associated
+ * with a different file descriptor obtained by opening again the same subdevice
+ * devnode but bound to a different media device context.
+ *
+ * Drivers that implement multiplexing support have to provide a valid
+ * implementation of the context-related operations in the media entity
+ * operations.
+ *
+ * Drivers are allowed to sub-class the v4l2_subdevice_context structure by
+ * defining a driver-specific type which embeds a struct v4l2_subdevice_context
+ * instance as first member, and allocate the driver-specific structure size in
+ * their implementation of the `alloc_context` operation.
+ *
+ * Subdevice contexts are ref-counted by embedding an instance of 'struct
+ * media_entity_context' and are freed once all the references to it are
+ * released.
+ *
+ * A subdevice context ref-count is increased when:
+ * - The context is created by bounding a video device to a media device context
+ * - The media pipeline it is part of starts streaming
+ * A subdevice context ref-count is decreased when:
+ * - The associated file handle is closed
+ * - The media pipeline it is part of stops streaming
+ *
+ * The ref-count is increased by a call to v4l2_subdev_context_get() and is
+ * reponsibility of the caller to decrease the reference count with a call to
+ * v4l2_subdev_context_put().
+ */
+struct v4l2_subdev_context {
+	struct media_entity_context base;
+	/*
+	 * TODO: active_state should most likely be changed from a pointer to an
+	 * embedded field. For the time being it's kept as a pointer to more
+	 * easily catch uses of active_state in the cases where the driver
+	 * doesn't support it.
+	 */
+	struct v4l2_subdev_state *state;
+};
+
 /**
  * struct v4l2_subdev_pad_ops - v4l2-subdev pad level operations
  *
@@ -1152,6 +1224,7 @@ struct v4l2_subdev_fh {
 	struct module *owner;
 #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
 	struct v4l2_subdev_state *state;
+	struct v4l2_subdev_context *context;
 	u64 client_caps;
 #endif
 };
@@ -1285,6 +1358,59 @@ int v4l2_subdev_link_validate(struct media_link *link);
 bool v4l2_subdev_has_pad_interdep(struct media_entity *entity,
 				  unsigned int pad0, unsigned int pad1);
 
+/**
+ * v4l2_subdev_context_get - Helper to get a v4l2 subdev context from a
+ *			     media device context
+ *
+ * @mdev_context: The media device context
+ * @sd: The V4L2 subdevice the context refers to
+ *
+ * Helper function that wraps media_device_get_entity_context() and returns
+ * the v4l2 subdevice context associated with a subdevice in a media device
+ * context.
+ *
+ * The reference count of the returned v4l2 subdevice context is increased.
+ * Callers of this function are required to decrease the reference count of
+ * the context reference with a call to v4l2_subdev_context_put().
+ */
+struct v4l2_subdev_context *
+v4l2_subdev_context_get(struct media_device_context *mdev_context,
+			struct v4l2_subdev *sd);
+
+/**
+ * v4l2_subdev_context_put - Helper to decrease a v4l2 subdevice context
+ *			     reference count
+ *
+ * @ctx: The v4l2 subdevice context to put
+ */
+void v4l2_subdev_context_put(struct v4l2_subdev_context *ctx);
+
+/**
+ * v4l2_subdev_init_context - Initialize the v4l2 subdevice context
+ *
+ * @sd: The subdevice the context belongs to
+ * @ctx: The context to initialize
+ *
+ * Initialize the v4l2 subdevice context. The intended callers of this function
+ * are driver-specific implementations of the media_entity_ops.alloc_context()
+ * function that allocates their driver specific types that derive from
+ * struct v4l2_subdev_context.
+ */
+int v4l2_subdev_init_context(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_context *ctx);
+
+/**
+ * v4l2_subdev_cleanup_context - Cleanup the v4l2 subdevice context
+ *
+ * @ctx: The context to cleanup.
+ *
+ * Cleanup the v4l2 subdevice context. The intended callers of this function are
+ * driver specific implementation of the media_entity_ops.destroy_context()
+ * function before releasing the memory previously allocated by
+ * media_entity_ops.alloc_context().
+ */
+void v4l2_subdev_cleanup_context(struct v4l2_subdev_context *ctx);
+
 /**
  * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
  *

-- 
2.49.0

[PATCH 13/26] media: v4l2-subdev: Introduce VIDIOC_SUBDEV_BIND_CONTEXT

[Jacopo Mondi]

Introduce a new ioctl in V4L2 subdev to allocate a subdevice context
and associate it with a media device context.

The ioctl handler calls the entity ops to let the driver allocate a
new context and initialize the subdev state there contained.

The new subdevice context is bound to the media device context
associated with the file descriptor provided by userspace as the new
ioctl argument.

The newly allocated context is stored in the v4l2_subdev_fh handle
that represent the open file handle on which the ioctl has been called.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 59 +++++++++++++++++++++++++++++++++++
 include/uapi/linux/v4l2-subdev.h      | 11 +++++++
 2 files changed, 70 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 7307f57439499c8d5360c89f492944828ac23973..300f84317623dd082a4cd2caec97057f972e82a3 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -607,6 +607,46 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
 			     v4l2_subdev_get_unlocked_active_state(sd);
 }
 
+static int subdev_do_bind_context(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_context **context,
+				  struct media_device_context *mdev_context)
+{
+	static struct lock_class_key key;
+	struct v4l2_subdev_state *state;
+	int ret;
+
+	ret = sd->entity.ops->alloc_context(&sd->entity,
+					    (struct media_entity_context **)
+					    context);
+	if (ret)
+		return ret;
+
+	state = __v4l2_subdev_state_alloc(sd, "context->state->lock", &key);
+	if (IS_ERR(state)) {
+		ret = PTR_ERR(state);
+		goto err_put_context;
+	}
+	(*context)->state = state;
+
+	/*
+	 * Bind the newly created video device context to the media device
+	 * context identified by the file descriptor.
+	 */
+	ret = media_device_bind_context(mdev_context,
+					(struct media_entity_context *)
+					*context);
+	if (ret)
+		goto err_free_state;
+
+	return 0;
+
+err_free_state:
+	__v4l2_subdev_state_free((*context)->state);
+err_put_context:
+	v4l2_subdev_context_put(*context);
+	return ret;
+}
+
 static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 			    struct v4l2_subdev_state *state)
 {
@@ -1089,6 +1129,25 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		return 0;
 	}
 
+	case VIDIOC_SUBDEV_BIND_CONTEXT: {
+		struct v4l2_subdev_bind_context *c = arg;
+		struct media_device_context *mdev_context;
+		int ret;
+
+		if (!sd->entity.ops || !sd->entity.ops->alloc_context ||
+		    !sd->entity.ops->destroy_context)
+			return -ENOTTY;
+
+		mdev_context = media_device_context_get_from_fd(c->context_fd);
+		if (!mdev_context)
+			return -EINVAL;
+
+		ret = subdev_do_bind_context(sd, &subdev_fh->context,
+					     mdev_context);
+		media_device_context_put(mdev_context);
+		return ret;
+	}
+
 	case VIDIOC_SUBDEV_G_CLIENT_CAP: {
 		struct v4l2_subdev_client_capability *client_cap = arg;
 
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 2347e266cf7516b4073c1edd43b97a3ddddb183b..6184cc0153a9dd9fbfa6729e1b6127ba4a961395 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -243,6 +243,16 @@ struct v4l2_subdev_routing {
 	__u32 reserved[11];
 };
 
+/**
+ * struct v4l2_subdev_bind_context - Subdev context information
+ *
+ * @context_fd: The file descriptor of the media_device instance the subdevice
+ *		has to be bound to
+ */
+struct v4l2_subdev_bind_context {
+	__u32 context_fd;
+};
+
 /*
  * The client is aware of streams. Setting this flag enables the use of 'stream'
  * fields (referring to the stream number) with various ioctls. If this is not
@@ -285,6 +295,7 @@ struct v4l2_subdev_client_capability {
 #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
 #define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
 #define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
+#define VIDIOC_SUBDEV_BIND_CONTEXT		_IOWR('V', 50, struct v4l2_subdev_bind_context)
 #define VIDIOC_SUBDEV_G_CLIENT_CAP		_IOR('V',  101, struct v4l2_subdev_client_capability)
 #define VIDIOC_SUBDEV_S_CLIENT_CAP		_IOWR('V',  102, struct v4l2_subdev_client_capability)
 

-- 
2.49.0

[PATCH 14/26] media: Documentation: Add VIDIOC_SUBDEV_BIND_CONTEXT

[Jacopo Mondi]

Document the newly introduced VIDIOC_SUBDEV_BIND_CONTEXT ioctl.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 .../userspace-api/media/v4l/user-func.rst          |  1 +
 .../media/v4l/vidioc-subdev-bind-context.rst       | 81 ++++++++++++++++++++++
 2 files changed, 82 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
index 0d9aff56ab653b2a4f6afe4828f88bc5637addf1..236847a1d6cb2a266bea30ecf7583979099b0343 100644
--- a/Documentation/userspace-api/media/v4l/user-func.rst
+++ b/Documentation/userspace-api/media/v4l/user-func.rst
@@ -66,6 +66,7 @@ Function Reference
     vidioc-remove-bufs
     vidioc-s-hw-freq-seek
     vidioc-streamon
+    vidioc-subdev-bind-context
     vidioc-subdev-enum-frame-interval
     vidioc-subdev-enum-frame-size
     vidioc-subdev-enum-mbus-code
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-bind-context.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-bind-context.rst
new file mode 100644
index 0000000000000000000000000000000000000000..5cba529a3cdcb63c7257f871d667fa792c0ca382
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-bind-context.rst
@@ -0,0 +1,81 @@
+.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. c:namespace:: V4L
+
+.. _vidioc_subdev_bind_context:
+
+********************************
+ioctl VIDIOC_SUBDEV_BIND_CONTEXT
+********************************
+
+Name
+====
+
+VIDIOC_SUBDEV_BIND_CONTEXT - Bind a subdevice file handle to a media device
+context
+
+Synopsis
+========
+
+.. c:macro:: VIDIOC_SUBDEV_BIND_CONTEXT
+
+``int ioctl(int fd, VIDIOC_SUBDEV_BIND_CONTEXT, struct v4l2_subdev_bind_context *argp)``
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :c:func:`open()`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_subdev_bind_context`.
+
+Description
+===========
+
+Applications call the ``VIDIOC_SUBDEV_BIND_CONTEXT`` ioctl to bind a subdevice
+file handle to a media device  context. Binding a subdevice file handle to a
+media device context creates an isolated execution context which allows to
+multiplex the usage of a video device. This means, in practice, that the
+subdevice configuration (format, sizes etc) applied on a file handle bound to a
+media device context won't be visible on file handles bound to a different media
+device context (or not bound at all).
+
+By opening a media device applications create a media device context to which
+video devices and subdevices file handles can be bound to. The file descriptor
+returned by a call to :c:func:`open()` on the media device identifies uniquely
+the media device context. Application populates the ``context_fd`` field of
+:c:type:`v4l2_subdev_bind_context` with the file descriptor of an open media
+device to identify the media context to which they want to bind a subdevice
+to.
+
+Applications can open a subdevice node multiple times, and call
+``VIDIOC_BIND_CONTEXT`` on each file handle returned by a successful call to
+:c:func:`open()` to isolate the operations performed on that file handle from
+any operation performed on other file handles bound to different contexts. This
+means, in example, that the subdevice format and sizes are isolated from the
+ones associated with a file descriptor, obtained by opening the same subdevice
+but bound to a different media device context (or not bound at all).
+
+The bounding operation realizes a permanent association valid until the
+subdevice context is released by closing the file handle.
+
+A subdevice file handle can be bound to the same media device context once
+only. Trying to bind the same file handle to the same media device context a
+second time, without releasing the already established context by closing the
+bound file descriptor first, will result in an error.
+
+Bounding is an opt-in feature that applications are free to ignore. Any
+operation directed to a non bound file handle will continue to work as it used
+to, and the video device configuration (formats, sizes etc) will be visible
+across all the other non-bound file handles.
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+EINVAL
+    The media device context file handle ``context_fd`` is not valid or the
+    subdevice file handle is already bound to a context.

-- 
2.49.0

[PATCH 15/26] media: v4l2_subdev: Introduce default context

[Jacopo Mondi]

Introduce a default context for v4l2 subdvice.

Drivers ported to use multi-context support that used to work with a
non-context aware userspace (which doesn't call VIDIOC_SUBDEV_BIND_CONTEXT)
shall continue to work even if they are context aware.

Provide a default context in the v4l2 subdev and bind it to the media
device default context when the subdevice is fully registered by
providing a v4l2_subdev_registered() function.

Release the context when the subdevice gets unregistered by the core,
providing a v4l2_subdev_unregistered() helper.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-device.c | 11 +++-----
 drivers/media/v4l2-core/v4l2-subdev.c | 50 +++++++++++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 29 ++++++++++++++++++++
 3 files changed, 83 insertions(+), 7 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-device.c b/drivers/media/v4l2-core/v4l2-device.c
index 5e537454f5cd71b3c50a2a2864642f7d5548047b..bf3ebd77b7bb8b13c849a89d01e6d889e8a2e4fd 100644
--- a/drivers/media/v4l2-core/v4l2-device.c
+++ b/drivers/media/v4l2-core/v4l2-device.c
@@ -146,11 +146,9 @@ int __v4l2_device_register_subdev(struct v4l2_device *v4l2_dev,
 	}
 #endif
 
-	if (sd->internal_ops && sd->internal_ops->registered) {
-		err = sd->internal_ops->registered(sd);
-		if (err)
-			goto error_unregister;
-	}
+	err = v4l2_subdev_registered(sd);
+	if (err)
+		goto error_unregister;
 
 	sd->owner = module;
 
@@ -274,8 +272,7 @@ void v4l2_device_unregister_subdev(struct v4l2_subdev *sd)
 	list_del(&sd->list);
 	spin_unlock(&v4l2_dev->lock);
 
-	if (sd->internal_ops && sd->internal_ops->unregistered)
-		sd->internal_ops->unregistered(sd);
+	v4l2_subdev_unregistered(sd);
 	sd->v4l2_dev = NULL;
 
 #if defined(CONFIG_MEDIA_CONTROLLER)
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 300f84317623dd082a4cd2caec97057f972e82a3..438f51980e5ac0f092ba6b0a979a376133968ddf 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1275,6 +1275,56 @@ const struct v4l2_file_operations v4l2_subdev_fops = {
 	.poll = subdev_poll,
 };
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+static int v4l2_subdev_register_default_context(struct v4l2_subdev *sd)
+{
+	struct media_device_context *mdev_context;
+
+	/* If the driver does not support contexts, return here. */
+	if (!sd->entity.ops || !sd->entity.ops->alloc_context ||
+	    !sd->entity.ops->destroy_context)
+		return 0;
+
+	mdev_context = sd->entity.graph_obj.mdev->default_context;
+	return subdev_do_bind_context(sd, &sd->default_context, mdev_context);
+}
+#endif /* CONFIG_MEDIA_CONTROLLER */
+
+int v4l2_subdev_registered(struct v4l2_subdev *sd)
+{
+	int ret;
+
+#ifdef CONFIG_MEDIA_CONTROLLER
+	ret = v4l2_subdev_register_default_context(sd);
+	if (ret)
+		return ret;
+#endif /* CONFIG_MEDIA_CONTROLLER */
+
+	if (sd->internal_ops && sd->internal_ops->registered) {
+		ret = sd->internal_ops->registered(sd);
+		if (ret)
+			goto err_registered;
+	}
+
+	return 0;
+
+err_registered:
+	if (sd->default_context)
+		v4l2_subdev_context_put(sd->default_context);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_registered);
+
+void v4l2_subdev_unregistered(struct v4l2_subdev *sd)
+{
+	if (sd->default_context)
+		v4l2_subdev_context_put(sd->default_context);
+
+	if (sd->internal_ops && sd->internal_ops->unregistered)
+		sd->internal_ops->unregistered(sd);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_unregistered);
+
 #ifdef CONFIG_MEDIA_CONTROLLER
 
 int v4l2_subdev_get_fwnode_pad_1_to_1(struct media_entity *entity,
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 9d257b859acafb11cfe6976e906e7baabd0206f6..1fa42a9f322be0be44fc9308744f4f4ae0cf1606 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1130,6 +1130,10 @@ struct v4l2_subdev_platform_data {
  * @active_state: Active state for the subdev (NULL for subdevs tracking the
  *		  state internally). Initialized by calling
  *		  v4l2_subdev_init_finalize().
+ * @default_context: Default context for the subdev, allows to operate
+ *		     context-aware drivers with a context-unaware userspace.
+ *		     It is initialized when the subdev is registered in
+ *		     v4l2_subdev_registered().
  * @enabled_pads: Bitmask of enabled pads used by v4l2_subdev_enable_streams()
  *		  and v4l2_subdev_disable_streams() helper functions for
  *		  fallback cases.
@@ -1182,6 +1186,7 @@ struct v4l2_subdev {
 	 * doesn't support it.
 	 */
 	struct v4l2_subdev_state *active_state;
+	struct v4l2_subdev_context *default_context;
 	u64 enabled_pads;
 	bool s_stream_enabled;
 };
@@ -1286,6 +1291,30 @@ static inline void *v4l2_get_subdev_hostdata(const struct v4l2_subdev *sd)
 	return sd->host_priv;
 }
 
+/**
+ * v4l2_subdev_registered - Subdevice registered notification
+ *
+ * @sd: The subdevice that has been registered
+ *
+ * Notify that a subdevice has been registered by the core. This function wraps
+ * a call to sd->internal_ops->registered (if available) and instantiates the
+ * default v4l2 subdevice context.
+ *
+ * Returns 0 on success, a negative error code otherwise.
+ */
+int v4l2_subdev_registered(struct v4l2_subdev *sd);
+
+/**
+ * v4l2_subdev_unregistered - Subdevice unregistered notification
+ *
+ * @sd: The subdevice that has been unregistered
+ *
+ * Notify that a subdevice has been unregistered by the core. This function
+ * wraps a call to sd->internal_ops->unregistered (if available) and deletes
+ * the default v4l2 subdevice context.
+ */
+void v4l2_subdev_unregistered(struct v4l2_subdev *sd);
+
 #ifdef CONFIG_MEDIA_CONTROLLER
 
 /**

-- 
2.49.0

[PATCH 16/26] media: v4l2-subdev: Add subdev state accessor helpers

[Jacopo Mondi]

The v4l2-subdev.c file offers three helpers to access the subdevice
active state from a v4l2_subdev pointer:

- v4l2_subdev_get_unlocked_active_state(sd)
- v4l2_subdev_get_locked_active_state(sd)
- v4l2_subdev_lock_and_get_active_state(sd)

With the introduction of struct v4l2_subdev_context which contains a
subdev_state as well, the actual "active" state is stored in three
possible places:
- A context associated with a v4l2_subdev_fh for context aware drivers
  operated by context aware userspace
- The default subdevice context for context aware drivers operated by
  non-context aware userspace
- The subdevice active state for non-context aware drivers

Provide helpers similar in spirit the existing ones but accept as
argument either a subdevice the context itself and retrieve the active
state from the correct place.

Helpers will be used in following patches for link validation.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 101 +++++++++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 104 +++++++++++++++++++---------------
 2 files changed, 159 insertions(+), 46 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 438f51980e5ac0f092ba6b0a979a376133968ddf..7372f61127c871cec44a3d1900e2b8bef34632b9 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -20,6 +20,7 @@
 #include <linux/version.h>
 #include <linux/videodev2.h>
 
+#include <media/media-device.h>
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
 #include <media/v4l2-event.h>
@@ -189,6 +190,106 @@ static inline int check_pad(struct v4l2_subdev *sd, u32 pad)
 	return 0;
 }
 
+/* subdev state accessor helpers */
+
+/*
+ * Access the state from the subdevice.
+ *
+ * If the driver is context-aware use the state stored in the default context
+ * otherwise use the active state stored in the subdevice.
+ */
+
+struct v4l2_subdev_state *
+v4l2_subdev_get_unlocked_active_state_from_sd(struct v4l2_subdev *sd)
+{
+	if (!sd)
+		return NULL;
+
+	if (sd->default_context) {
+		lockdep_assert_not_held(sd->default_context->state->lock);
+
+		return sd->default_context->state;
+	}
+
+	if (sd->active_state)
+		lockdep_assert_not_held(sd->active_state->lock);
+	return sd->active_state;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_get_unlocked_active_state_from_sd);
+
+struct v4l2_subdev_state *
+v4l2_subdev_get_locked_active_state_from_sd(struct v4l2_subdev *sd)
+{
+	if (!sd)
+		return NULL;
+
+	if (sd->default_context) {
+		lockdep_assert_held(sd->default_context->state->lock);
+
+		return sd->default_context->state;
+	}
+
+	if (sd->active_state)
+		lockdep_assert_held(sd->active_state->lock);
+	return sd->active_state;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_get_locked_active_state_from_sd);
+
+struct v4l2_subdev_state *
+v4l2_subdev_lock_and_get_active_state_from_sd(struct v4l2_subdev *sd)
+{
+	if (!sd)
+		return NULL;
+
+	if (sd->default_context) {
+		v4l2_subdev_lock_state(sd->default_context->state);
+
+		return sd->default_context->state;
+	}
+
+	if (sd->active_state)
+		v4l2_subdev_lock_state(sd->active_state);
+	return sd->active_state;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_lock_and_get_active_state_from_sd);
+
+/* Access the subdevice state from a subdvice context. */
+struct v4l2_subdev_state *
+v4l2_subdev_get_unlocked_active_state_from_ctx(struct v4l2_subdev_context *ctx)
+{
+	if (!ctx)
+		return NULL;
+
+	if (ctx->state)
+		lockdep_assert_not_held(ctx->state->lock);
+	return ctx->state;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_get_unlocked_active_state_from_ctx);
+
+struct v4l2_subdev_state *
+v4l2_subdev_get_locked_active_state_from_ctx(struct v4l2_subdev_context *ctx)
+{
+	if (!ctx)
+		return NULL;
+
+	if (ctx->state)
+		lockdep_assert_held(ctx->state->lock);
+	return ctx->state;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_get_locked_active_state_from_ctx);
+
+struct v4l2_subdev_state *
+v4l2_subdev_lock_and_get_active_state_from_ctx(struct v4l2_subdev_context *ctx)
+{
+	if (!ctx)
+		return NULL;
+
+	if (ctx->state)
+		v4l2_subdev_lock_state(ctx->state);
+	return ctx->state;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_lock_and_get_active_state_from_ctx);
+
 static int check_state(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 		       u32 which, u32 pad, u32 stream)
 {
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 1fa42a9f322be0be44fc9308744f4f4ae0cf1606..8087c0ae3bc0a0a95512b4b0ff5257522a104ca0 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -29,16 +29,17 @@
 
 #define	V4L2_DEVICE_NOTIFY_EVENT		_IOW('v', 2, struct v4l2_event)
 
-struct v4l2_device;
+struct led_classdev;
+struct media_device_context;
+struct tuner_setup;
 struct v4l2_ctrl_handler;
+struct v4l2_device;
 struct v4l2_event;
 struct v4l2_event_subscription;
 struct v4l2_fh;
+struct v4l2_mbus_frame_desc;
 struct v4l2_subdev;
 struct v4l2_subdev_fh;
-struct tuner_setup;
-struct v4l2_mbus_frame_desc;
-struct led_classdev;
 
 /**
  * struct v4l2_decode_vbi_line - used to decode_vbi_line
@@ -1968,64 +1969,75 @@ static inline void v4l2_subdev_unlock_states(struct v4l2_subdev_state *state1,
 		mutex_unlock(state2->lock);
 }
 
+struct v4l2_subdev_state *
+v4l2_subdev_get_unlocked_active_state_from_sd(struct v4l2_subdev *sd);
+struct v4l2_subdev_state *
+v4l2_subdev_get_locked_active_state_from_sd(struct v4l2_subdev *sd);
+struct v4l2_subdev_state *
+v4l2_subdev_lock_and_get_active_state_from_sd(struct v4l2_subdev *sd);
+
+struct v4l2_subdev_state *
+v4l2_subdev_get_unlocked_active_state_from_ctx(struct v4l2_subdev_context *ctx);
+struct v4l2_subdev_state *
+v4l2_subdev_get_locked_active_state_from_ctx(struct v4l2_subdev_context *ctx);
+struct v4l2_subdev_state *
+v4l2_subdev_lock_and_get_active_state_from_ctx(struct v4l2_subdev_context *ctx);
+
 /**
- * v4l2_subdev_get_unlocked_active_state() - Checks that the active subdev state
- *					     is unlocked and returns it
- * @sd: The subdevice
+ * v4l2_subdev_get_unlocked_active_state() - Checks that the subdev state is
+ *					     unlocked and returns it
+ * @sdctx: The subdevice, or the subdevice context
  *
- * Returns the active state for the subdevice, or NULL if the subdev does not
- * support active state. If the state is not NULL, calls
- * lockdep_assert_not_held() to issue a warning if the state is locked.
+ * Returns the subdevice state, or NULL if it is not valid. If the state is
+ * not NULL, calls lockdep_assert_not_held() to issue a warning if the state
+ * is locked.
  *
- * This function is to be used e.g. when getting the active state for the sole
- * purpose of passing it forward, without accessing the state fields.
+ * This function is to be used e.g. when getting the state for the sole purpose
+ * of passing it forward, without accessing the state fields.
  */
-static inline struct v4l2_subdev_state *
-v4l2_subdev_get_unlocked_active_state(struct v4l2_subdev *sd)
-{
-	if (sd->active_state)
-		lockdep_assert_not_held(sd->active_state->lock);
-	return sd->active_state;
-}
+#define v4l2_subdev_get_unlocked_active_state(sdctx)				\
+	_Generic((sdctx),							\
+		struct v4l2_subdev *: 						\
+			v4l2_subdev_get_unlocked_active_state_from_sd,		\
+		struct v4l2_subdev_context *:					\
+			v4l2_subdev_get_unlocked_active_state_from_ctx)		\
+	(sdctx)
 
 /**
- * v4l2_subdev_get_locked_active_state() - Checks that the active subdev state
- *					   is locked and returns it
- *
- * @sd: The subdevice
+ * v4l2_subdev_get_locked_active_state() - Checks that the subdev state is
+ *					   locked and returns it
+ * @sdctx: The subdevice, or the subdevice context
  *
- * Returns the active state for the subdevice, or NULL if the subdev does not
- * support active state. If the state is not NULL, calls lockdep_assert_held()
- * to issue a warning if the state is not locked.
+ * Returns the subdevice state, or NULL is not valid. If the state is not NULL,
+ * calls lockdep_assert_held() to issue a warning if the state is not locked.
  *
- * This function is to be used when the caller knows that the active state is
+ * This function is to be used when the caller knows that the context state is
  * already locked.
  */
-static inline struct v4l2_subdev_state *
-v4l2_subdev_get_locked_active_state(struct v4l2_subdev *sd)
-{
-	if (sd->active_state)
-		lockdep_assert_held(sd->active_state->lock);
-	return sd->active_state;
-}
+#define v4l2_subdev_get_locked_active_state(sdctx)				\
+	_Generic((sdctx),							\
+		struct v4l2_subdev *:						\
+			v4l2_subdev_get_locked_active_state_from_sd,		\
+		struct v4l2_subdev_context *:					\
+			v4l2_subdev_get_locked_active_state_from_ctx)		\
+	(sdctx)
 
 /**
- * v4l2_subdev_lock_and_get_active_state() - Locks and returns the active subdev
- *					     state for the subdevice
- * @sd: The subdevice
+ * v4l2_subdev_lock_and_get_active_state_from_ctx() - Locks and returns the
+ *						      subdevice state
+ * @sdctx: The subdevice, or the subdevice context
  *
- * Returns the locked active state for the subdevice, or NULL if the subdev
- * does not support active state.
+ * Returns the locked subdevice state, or NULL if it is not valid.
  *
  * The state must be unlocked with v4l2_subdev_unlock_state() after use.
  */
-static inline struct v4l2_subdev_state *
-v4l2_subdev_lock_and_get_active_state(struct v4l2_subdev *sd)
-{
-	if (sd->active_state)
-		v4l2_subdev_lock_state(sd->active_state);
-	return sd->active_state;
-}
+#define v4l2_subdev_lock_and_get_active_state(sdctx)				\
+	_Generic((sdctx),							\
+		struct v4l2_subdev *:						\
+			v4l2_subdev_lock_and_get_active_state_from_sd,		\
+		struct v4l2_subdev_context *:					\
+			v4l2_subdev_get_locked_active_state_from_ctx)		\
+	(sdctx)
 
 /**
  * v4l2_subdev_init - initializes the sub-device struct

-- 
2.49.0

[PATCH 17/26] media: v4l2-subdev: Get state from context

[Jacopo Mondi]

The V4L2 subdev ioctl handler retrieves the correct subdev
state inspecting the 'which' field of the ioctl argument.

So far the subdev state either come from the file handle in case of
V4L2_SUBDEV_FORMAT_TRY or from the active state stored in the subdev
directly in case of V4L2_SUBDEV_FORMAT_ACTIVE.

With the introduction of multi-contexts support, there will be a subdev
state associated to each bound context. If we have a valid context,
use the state from there in case of V4L2_SUBDEV_FORMAT_ACTIVE, and
default to the subdev active state in case the context is not valid.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 7372f61127c871cec44a3d1900e2b8bef34632b9..66c539d880127844893620d325a2b05ac4aa9e96 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -703,9 +703,20 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
 		break;
 	}
 
-	return which == V4L2_SUBDEV_FORMAT_TRY ?
-			     subdev_fh->state :
-			     v4l2_subdev_get_unlocked_active_state(sd);
+	/*
+	 * If which is FORMAT_TRY return the state stored in the file handle.
+	 * If a context has been allocated because the subdev has been bound
+	 * then return the state stored in the context. Otherwise default to the
+	 * subdevice active state.
+	 */
+
+	if (which == V4L2_SUBDEV_FORMAT_TRY)
+		return subdev_fh->state;
+
+	if (subdev_fh->context)
+		return v4l2_subdev_get_unlocked_active_state(subdev_fh->context);
+
+	return v4l2_subdev_get_unlocked_active_state(sd);
 }
 
 static int subdev_do_bind_context(struct v4l2_subdev *sd,

-- 
2.49.0

[PATCH 18/26] media: media-entity: Support context in pipeline_start

[Jacopo Mondi]

Add helpers to propagate an execution context when starting the media
pipeline. Starting a pipeline with an execution context implies that
when creating the streaming pipeline all the entities part of the
pipeline shall have a valid context associated with the media context in
use.

When creating the media pipeline increase the device context use
count to guarantee that during the streaming phase drivers will
always operate with valid contexts.

Add 'overrides' for all the video_device_pipeline_start() and
media_pipeline_start() variants to support contexts.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c       | 150 ++++++++++++++++++++++++++++++++-----
 drivers/media/v4l2-core/v4l2-dev.c |  44 +++++++++++
 include/media/media-entity.h       |  56 +++++++++++++-
 include/media/v4l2-dev.h           |  57 ++++++++++++++
 4 files changed, 287 insertions(+), 20 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 7bc276c725f974539ea06e3882d004b81be1de68..f421d6d74630bb96400d39d805c5db5d3d1ff913 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -768,8 +768,50 @@ static int media_pipeline_populate(struct media_pipeline *pipe,
 	return ret;
 }
 
-__must_check int __media_pipeline_start(struct media_pad *origin,
-					struct media_pipeline *pipe)
+static int
+media_pipeline_validate_context(struct media_device_context *mdev_context,
+				struct media_entity *entity,
+				struct media_pipeline_pad *ppad)
+{
+	struct media_entity_context *context;
+
+	if (!mdev_context)
+		return 0;
+
+	/*
+	 * It's not mandatory for all entities in the pipeline to support
+	 * contexts.
+	 */
+	if (!entity->ops || !entity->ops->alloc_context ||
+	    !entity->ops->destroy_context)
+		return 0;
+
+	/*
+	 * But if they do they should be bound to the same media device context
+	 * as all other entities.
+	 *
+	 * media_device_get_entity_context increases the ref-counting of the
+	 * context. Store a reference in the ppad for later decreasing the
+	 * reference count when the pipeline is stopped.
+	 *
+	 * Fail validation if no context is associated with this media context
+	 * and be loud about that as userspace should be informed it has to
+	 * bind all entities of the pipeline in the same context.
+	 */
+	context = media_device_get_entity_context(mdev_context, entity);
+	if (WARN_ON(IS_ERR(context)))
+		return -EPIPE;
+
+	media_entity_context_get(context);
+	ppad->context = context;
+
+	return 0;
+}
+
+__must_check int
+__media_pipeline_start_context(struct media_pad *origin,
+			       struct media_pipeline *pipe,
+			       struct media_device_context *mdev_context)
 {
 	struct media_device *mdev = origin->graph_obj.mdev;
 	struct media_pipeline_pad *err_ppad;
@@ -829,7 +871,15 @@ __must_check int __media_pipeline_start(struct media_pad *origin,
 		}
 
 		/*
-		 * 2. Validate all active links whose sink is the current pad.
+		 * 2. If we have a media context, ensure the entity has a device
+		 * context associated with it.
+		 */
+		ret = media_pipeline_validate_context(mdev_context, entity, ppad);
+		if (ret)
+			goto error;
+
+		/*
+		 * 3. Validate all active links whose sink is the current pad.
 		 * Validation of the source pads is performed in the context of
 		 * the connected sink pad to avoid duplicating checks.
 		 */
@@ -875,7 +925,7 @@ __must_check int __media_pipeline_start(struct media_pad *origin,
 		}
 
 		/*
-		 * 3. If the pad has the MEDIA_PAD_FL_MUST_CONNECT flag set,
+		 * 4. If the pad has the MEDIA_PAD_FL_MUST_CONNECT flag set,
 		 * ensure that it has either no link or an enabled link.
 		 */
 		if ((pad->flags & MEDIA_PAD_FL_MUST_CONNECT) &&
@@ -905,6 +955,9 @@ __must_check int __media_pipeline_start(struct media_pad *origin,
 		if (err_ppad == ppad)
 			break;
 
+		if (err_ppad->context)
+			media_entity_context_put(err_ppad->context);
+
 		err_ppad->pad->pipe = NULL;
 	}
 
@@ -912,19 +965,35 @@ __must_check int __media_pipeline_start(struct media_pad *origin,
 
 	return ret;
 }
+EXPORT_SYMBOL_GPL(__media_pipeline_start_context);
+
+__must_check int __media_pipeline_start(struct media_pad *origin,
+					struct media_pipeline *pipe)
+{
+	return __media_pipeline_start_context(origin, pipe, NULL);
+}
 EXPORT_SYMBOL_GPL(__media_pipeline_start);
 
-__must_check int media_pipeline_start(struct media_pad *origin,
-				      struct media_pipeline *pipe)
+__must_check int
+media_pipeline_start_context(struct media_pad *origin,
+			     struct media_pipeline *pipe,
+			     struct media_device_context *context)
 {
 	struct media_device *mdev = origin->graph_obj.mdev;
 	int ret;
 
 	mutex_lock(&mdev->graph_mutex);
-	ret = __media_pipeline_start(origin, pipe);
+	ret = __media_pipeline_start_context(origin, pipe, context);
 	mutex_unlock(&mdev->graph_mutex);
 	return ret;
 }
+EXPORT_SYMBOL_GPL(media_pipeline_start_context);
+
+__must_check int media_pipeline_start(struct media_pad *origin,
+				      struct media_pipeline *pipe)
+{
+	return media_pipeline_start_context(origin, pipe, NULL);
+}
 EXPORT_SYMBOL_GPL(media_pipeline_start);
 
 void __media_pipeline_stop(struct media_pad *pad)
@@ -942,8 +1011,11 @@ void __media_pipeline_stop(struct media_pad *pad)
 	if (--pipe->start_count)
 		return;
 
-	list_for_each_entry(ppad, &pipe->pads, list)
+	list_for_each_entry(ppad, &pipe->pads, list) {
+		if (ppad->context)
+			media_entity_context_put(ppad->context);
 		ppad->pad->pipe = NULL;
+	}
 
 	media_pipeline_cleanup(pipe);
 
@@ -962,14 +1034,13 @@ void media_pipeline_stop(struct media_pad *pad)
 }
 EXPORT_SYMBOL_GPL(media_pipeline_stop);
 
-__must_check int media_pipeline_alloc_start(struct media_pad *pad)
+static struct media_pipeline *
+media_pipeline_alloc(struct media_pad *pad)
 {
 	struct media_device *mdev = pad->graph_obj.mdev;
-	struct media_pipeline *new_pipe = NULL;
 	struct media_pipeline *pipe;
-	int ret;
 
-	mutex_lock(&mdev->graph_mutex);
+	lockdep_assert_held(&mdev->graph_mutex);
 
 	/*
 	 * Is the pad already part of a pipeline? If not, we need to allocate
@@ -977,19 +1048,33 @@ __must_check int media_pipeline_alloc_start(struct media_pad *pad)
 	 */
 	pipe = media_pad_pipeline(pad);
 	if (!pipe) {
-		new_pipe = kzalloc(sizeof(*new_pipe), GFP_KERNEL);
-		if (!new_pipe) {
-			ret = -ENOMEM;
-			goto out;
-		}
+		pipe = kzalloc(sizeof(*pipe), GFP_KERNEL);
+		if (!pipe)
+			return ERR_PTR(-ENOMEM);
 
-		pipe = new_pipe;
 		pipe->allocated = true;
 	}
 
+	return pipe;
+}
+
+__must_check int media_pipeline_alloc_start(struct media_pad *pad)
+{
+	struct media_device *mdev = pad->graph_obj.mdev;
+	struct media_pipeline *pipe;
+	int ret;
+
+	mutex_lock(&mdev->graph_mutex);
+
+	pipe = media_pipeline_alloc(pad);
+	if (IS_ERR(pipe)) {
+		ret = PTR_ERR(pipe);
+		goto out;
+	}
+
 	ret = __media_pipeline_start(pad, pipe);
 	if (ret)
-		kfree(new_pipe);
+		kfree(pipe);
 
 out:
 	mutex_unlock(&mdev->graph_mutex);
@@ -998,6 +1083,33 @@ __must_check int media_pipeline_alloc_start(struct media_pad *pad)
 }
 EXPORT_SYMBOL_GPL(media_pipeline_alloc_start);
 
+__must_check
+int media_pipeline_alloc_start_context(struct media_pad *pad,
+				       struct media_device_context *mdev_context)
+{
+	struct media_device *mdev = pad->graph_obj.mdev;
+	struct media_pipeline *pipe;
+	int ret;
+
+	mutex_lock(&mdev->graph_mutex);
+
+	pipe = media_pipeline_alloc(pad);
+	if (IS_ERR(pipe)) {
+		ret = PTR_ERR(pipe);
+		goto out;
+	}
+
+	ret = __media_pipeline_start_context(pad, pipe, mdev_context);
+	if (ret)
+		kfree(pipe);
+
+out:
+	mutex_unlock(&mdev->graph_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(media_pipeline_alloc_start_context);
+
 struct media_pad *
 __media_pipeline_pad_iter_next(struct media_pipeline *pipe,
 			       struct media_pipeline_pad_iter *iter,
diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index 96696959314abfb1864ea5d96742e579b5a41f6f..d2136f5a84b165cf1f8f7b37cef981d74c3e5ac2 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -1221,6 +1221,36 @@ __must_check int __video_device_pipeline_start(struct video_device *vdev,
 }
 EXPORT_SYMBOL_GPL(__video_device_pipeline_start);
 
+__must_check int
+video_device_context_pipeline_start(struct video_device_context *context,
+				    struct media_pipeline *pipe)
+{
+	struct video_device *vdev = context->vdev;
+	struct media_entity *entity = &vdev->entity;
+
+	if (entity->num_pads != 1)
+		return -ENODEV;
+
+	return media_pipeline_start_context(&entity->pads[0], pipe,
+					    context->base.mdev_context);
+}
+EXPORT_SYMBOL(video_device_context_pipeline_start);
+
+__must_check int
+__video_device_context_pipeline_start(struct video_device_context *context,
+				      struct media_pipeline *pipe)
+{
+	struct video_device *vdev = context->vdev;
+	struct media_entity *entity = &vdev->entity;
+
+	if (entity->num_pads != 1)
+		return -ENODEV;
+
+	return __media_pipeline_start_context(&entity->pads[0], pipe,
+					      context->base.mdev_context);
+}
+EXPORT_SYMBOL(__video_device_context_pipeline_start);
+
 void video_device_pipeline_stop(struct video_device *vdev)
 {
 	struct media_entity *entity = &vdev->entity;
@@ -1254,6 +1284,20 @@ __must_check int video_device_pipeline_alloc_start(struct video_device *vdev)
 }
 EXPORT_SYMBOL_GPL(video_device_pipeline_alloc_start);
 
+__must_check int
+video_device_context_pipeline_alloc_start(struct video_device_context *context)
+{
+	struct video_device *vdev = context->vdev;
+	struct media_entity *entity = &vdev->entity;
+
+	if (entity->num_pads != 1)
+		return -ENODEV;
+
+	return media_pipeline_alloc_start_context(&entity->pads[0],
+						  context->base.mdev_context);
+}
+EXPORT_SYMBOL_GPL(video_device_context_pipeline_alloc_start);
+
 struct media_pipeline *video_device_pipeline(struct video_device *vdev)
 {
 	struct media_entity *entity = &vdev->entity;
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 32298fe8a18c6ee3c1dbcff9ef869548904417a7..b60c311ab390beb6931fe8f2bbe8939e11cda452 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -119,16 +119,20 @@ struct media_pipeline {
  * @list:		Entry in the media_pad pads list
  * @pipe:		The media_pipeline that the pad is part of
  * @pad:		The media pad
+ * @context:		Reference to a video device or subdevice context
  *
  * This structure associate a pad with a media pipeline. Instances of
  * media_pipeline_pad are created by media_pipeline_start() when it builds the
  * pipeline, and stored in the &media_pad.pads list. media_pipeline_stop()
- * removes the entries from the list and deletes them.
+ * removes the entries from the list and deletes them. The context field is
+ * populated only if a valid context has been associated with the pad.
  */
+struct media_entity_context;
 struct media_pipeline_pad {
 	struct list_head list;
 	struct media_pipeline *pipe;
 	struct media_pad *pad;
+	struct media_entity_context *context;
 };
 
 /**
@@ -1212,6 +1216,39 @@ __must_check int media_pipeline_start(struct media_pad *origin,
 __must_check int __media_pipeline_start(struct media_pad *origin,
 					struct media_pipeline *pipe);
 
+/**
+ * media_pipeline_start_context - Mark a pipeline as streaming
+ * @origin: Starting pad
+ * @pipe: Media pipeline to be assigned to all pads in the pipeline.
+ * @context: The media device context the pipeline belongs to
+ *
+ * Mark all pads connected to a given pad through enabled links, either
+ * directly or indirectly, as streaming. The given pipeline object is assigned
+ * to every pad in the pipeline and stored in the media_pad pipe field.
+ *
+ * Calls to this function can be nested, in which case the same number of
+ * media_pipeline_stop() calls will be required to stop streaming. The
+ * pipeline pointer must be identical for all nested calls to
+ * media_pipeline_start().
+ */
+__must_check int
+media_pipeline_start_context(struct media_pad *origin,
+			     struct media_pipeline *pipe,
+			     struct media_device_context *context);
+
+/**
+ * __media_pipeline_start_context - Mark a pipeline as streaming
+ * @origin: Starting pad
+ * @pipe: Media pipeline to be assigned to all pads in the pipeline.
+ * @context: The media device context the pipeline belongs to
+ *
+ * ..note:: This is the non-locking version of media_pipeline_start_context()
+ */
+__must_check int
+__media_pipeline_start_context(struct media_pad *origin,
+			       struct media_pipeline *pipe,
+			       struct media_device_context *context);
+
 /**
  * media_pipeline_stop - Mark a pipeline as not streaming
  * @pad: Starting pad
@@ -1318,6 +1355,23 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
  */
 __must_check int media_pipeline_alloc_start(struct media_pad *pad);
 
+/**
+ * media_pipeline_alloc_start_context - Mark a pipeline as streaming
+ * @pad: Starting pad
+ * @context: The media device context the pipeline belongs to
+ *
+ * media_pipeline_alloc_start_context() is similar to
+ * media_pipeline_start_context() but instead of working on a given pipeline the
+ * function will use an existing pipeline if the pad is already part of a
+ * pipeline, or allocate a new pipeline.
+ *
+ * Calls to media_pipeline_alloc_start_context() must be matched with
+ * media_pipeline_stop().
+ */
+__must_check int
+media_pipeline_alloc_start_context(struct media_pad *pad,
+				   struct media_device_context *context);
+
 /**
  * media_devnode_create() - creates and initializes a device node interface
  *
diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
index bab4b13b109362bec84d8d16440b6ea895206b60..93095df692e3628d4be003d25843fa744d6b20a4 100644
--- a/include/media/v4l2-dev.h
+++ b/include/media/v4l2-dev.h
@@ -602,6 +602,48 @@ __must_check int video_device_pipeline_start(struct video_device *vdev,
 __must_check int __video_device_pipeline_start(struct video_device *vdev,
 					       struct media_pipeline *pipe);
 
+/**
+ * video_device_context_pipeline_start - Mark a pipeline as streaming starting
+ *					 from a video device context
+ * @context: The video device context that starts the streaming
+ * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ *
+ * ..note:: This is the multi-context version of video_device_pipeline_start()
+ *          Documentation of this function only describes specific aspects of
+ *          this version. Refer to the video_device_pipeline_start()
+ *          documentation for a complete reference.
+ *
+ * Validate that all entities connected to a video device through enabled links
+ * by ensuring that a context associated with the same media device context
+ * exists for them. Increase the reference counting of each of the contexts part
+ * of the pipeline to guarantee their lifetime is maintained as long as the
+ * pipeline is streaming.
+ *
+ * Context validation and refcounting of all entities that are part of a
+ * streaming pipeline ensures that device drivers can safely access device
+ * contexts in a media device context during streaming. References to contexts
+ * retried by a call to media_device_get_entity_context(), are guaranteed to be
+ * valid as long as the pipeline is streaming. Likewise, the media device
+ * context that contains the device contexts is guaranteed to be valid as long
+ * as the pipeline is streaming.
+ */
+__must_check int
+video_device_context_pipeline_start(struct video_device_context *context,
+				    struct media_pipeline *pipe);
+
+/**
+ * __video_device_context_pipeline_start - Mark a pipeline as streaming starting
+ *					   from a video device context
+ * @context: The video device context that starts the streaming
+ * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ *
+ * ..note:: This is the non-locking version of
+ *	    __video_device_context_pipeline_start()
+ */
+__must_check int
+__video_device_context_pipeline_start(struct video_device_context *context,
+				      struct media_pipeline *pipe);
+
 /**
  * video_device_pipeline_stop - Mark a pipeline as not streaming
  * @vdev: Starting video device
@@ -646,6 +688,21 @@ void __video_device_pipeline_stop(struct video_device *vdev);
  */
 __must_check int video_device_pipeline_alloc_start(struct video_device *vdev);
 
+/**
+ * video_device_context_pipeline_alloc_start - Mark a pipeline as streaming
+ * @context: The video device context that starts the streaming
+ *
+ * video_device_context_pipeline_alloc_start() is similar to
+ * video_device_context_pipeline_start() but instead of working on a given
+ * pipeline the function will use an existing pipeline if the video device is
+ * already part of a pipeline, or allocate a new pipeline.
+ *
+ * Calls to video_device_context_pipeline_alloc_start() must be matched with
+ * video_device_pipeline_stop().
+ */
+__must_check int
+video_device_context_pipeline_alloc_start(struct video_device_context *context);
+
 /**
  * video_device_pipeline - Get the media pipeline a video device is part of
  * @vdev: The video device

-- 
2.49.0

[PATCH 19/26] media: mc-entity: Add link_validate_context

[Jacopo Mondi]

Add an 'override' of the link_validate() media entity operation that
accepts a media_device_context as second argument to allow entities
to validate the link state in the provided media_device context.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 23 +++++++++++++++++++----
 include/media/media-entity.h |  6 ++++++
 2 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index f421d6d74630bb96400d39d805c5db5d3d1ff913..675ceaede0d10a2420b8ea6a89e5963dcfde5ffe 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -884,6 +884,8 @@ __media_pipeline_start_context(struct media_pad *origin,
 		 * the connected sink pad to avoid duplicating checks.
 		 */
 		for_each_media_entity_data_link(entity, link) {
+			const struct media_entity_operations *ops;
+
 			/* Skip links unrelated to the current pad. */
 			if (link->sink != pad && link->source != pad)
 				continue;
@@ -902,13 +904,23 @@ __media_pipeline_start_context(struct media_pad *origin,
 			if (link->sink != pad)
 				continue;
 
-			if (!entity->ops || !entity->ops->link_validate)
+			ops = entity->ops;
+			if (!ops || (!ops->link_validate &&
+				     !ops->link_validate_context))
 				continue;
 
-			ret = entity->ops->link_validate(link);
+			if (mdev_context && ops->link_validate_context)
+				ret = ops->link_validate_context(link,
+								 mdev_context);
+			else
+				ret = entity->ops->link_validate(link);
+
 			if (ret) {
 				dev_dbg(mdev->dev,
-					"Link '%s':%u -> '%s':%u failed validation: %d\n",
+					"%sink '%s':%u -> '%s':%u failed validation: %d\n",
+					(mdev_context &&
+					 ops->link_validate_context) ?
+					"Context l" : "L",
 					link->source->entity->name,
 					link->source->index,
 					link->sink->entity->name,
@@ -917,7 +929,10 @@ __media_pipeline_start_context(struct media_pad *origin,
 			}
 
 			dev_dbg(mdev->dev,
-				"Link '%s':%u -> '%s':%u is valid\n",
+				"%sink '%s':%u -> '%s':%u is valid\n",
+				(mdev_context &&
+				 ops->link_validate_context) ?
+				"Context l" : "L",
 				link->source->entity->name,
 				link->source->index,
 				link->sink->entity->name,
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index b60c311ab390beb6931fe8f2bbe8939e11cda452..b053a0baee4031a464edf506d3d131bacb810f81 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -295,6 +295,10 @@ struct media_entity_context {
  * @link_validate:	Return whether a link is valid from the entity point of
  *			view. The media_pipeline_start() function
  *			validates all links by calling this operation. Optional.
+ * @link_validate_context: Return whether a link is valid from the entity
+ *			context point of view. The
+ *			media_pipeline_start_context() function validates all
+ *			links calling this operation. Optional.
  * @has_pad_interdep:	Return whether two pads of the entity are
  *			interdependent. If two pads are interdependent they are
  *			part of the same pipeline and enabling one of the pads
@@ -327,6 +331,8 @@ struct media_entity_operations {
 			  const struct media_pad *local,
 			  const struct media_pad *remote, u32 flags);
 	int (*link_validate)(struct media_link *link);
+	int (*link_validate_context)(struct media_link *link,
+				     struct media_device_context *mdev_context);
 	bool (*has_pad_interdep)(struct media_entity *entity, unsigned int pad0,
 				 unsigned int pad1);
 	int (*alloc_context)(struct media_entity *entity,

-- 
2.49.0

[PATCH 20/26] media: v4l2-subdev: Validate media links with context

[Jacopo Mondi]

The v4l2-subdev.c file provides an helper for subdevs to implement
the media entity .link_validate() operation which can be used by
subdevice drivers.

Provide an 'overload' of the v4l2_subdev_link_validate() function
that supports contexts to be used by context-aware subdev drivers to
implement .link_validate_context().

When v4l2_subdev_link_validate() is used to validate a subdev-to-subdev
link, propagate the media_device_context to all the call chain and
introduce (and use) helpers to get the subdevice state from either the
context or the subdev in case no context is available.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 148 +++++++++++++++++++++++++++-------
 include/media/v4l2-subdev.h           |  34 +++++++-
 2 files changed, 151 insertions(+), 31 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 66c539d880127844893620d325a2b05ac4aa9e96..59f42a3a9755a77ea74442605dbbc2af1b67a0ea 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1460,6 +1460,71 @@ int v4l2_subdev_get_fwnode_pad_1_to_1(struct media_entity *entity,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_get_fwnode_pad_1_to_1);
 
+/*
+ * Retrieve the subdevice state from the media device context or,
+ * if there is no context, use the active state from the subdevice.
+ *
+ * These three functions wraps the usual subdev state helpers:
+ *
+ * - get_unlocked
+ * - get_locked
+ * - lock_and_get
+ */
+
+static struct v4l2_subdev_state *
+v4l2_subdev_get_unlocked_state_from_mdev_ctx(struct v4l2_subdev *sd,
+					     struct media_device_context *mdev_ctx)
+{
+	struct v4l2_subdev_context *ctx = NULL;
+
+	if (mdev_ctx) {
+		ctx =  v4l2_subdev_context_get(mdev_ctx, sd);
+		if (WARN_ON(!ctx))
+			return NULL;
+	}
+
+	if (ctx)
+		return v4l2_subdev_get_unlocked_active_state(ctx);
+
+	return v4l2_subdev_get_unlocked_active_state(sd);
+}
+
+static struct v4l2_subdev_state *
+v4l2_subdev_get_locked_state_from_mdev_ctx(struct v4l2_subdev *sd,
+					   struct media_device_context *mdev_ctx)
+{
+	struct v4l2_subdev_context *ctx = NULL;
+
+	if (mdev_ctx) {
+		ctx =  v4l2_subdev_context_get(mdev_ctx, sd);
+		if (WARN_ON(!ctx))
+			return NULL;
+	}
+
+	if (ctx)
+		return v4l2_subdev_get_locked_active_state(ctx);
+
+	return v4l2_subdev_get_locked_active_state(sd);
+}
+
+static struct v4l2_subdev_state *
+v4l2_subdev_lock_and_get_state_from_mdev_ctx(struct v4l2_subdev *sd,
+					     struct media_device_context *mdev_ctx)
+{
+	struct v4l2_subdev_context *ctx = NULL;
+
+	if (mdev_ctx) {
+		ctx =  v4l2_subdev_context_get(mdev_ctx, sd);
+		if (WARN_ON(!ctx))
+			return NULL;
+	}
+
+	if (ctx)
+		return v4l2_subdev_lock_and_get_active_state(ctx);
+
+	return v4l2_subdev_lock_and_get_active_state(sd);
+}
+
 int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
 				      struct media_link *link,
 				      struct v4l2_subdev_format *source_fmt,
@@ -1518,8 +1583,9 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
 
 static int
-v4l2_subdev_link_validate_get_format(struct media_pad *pad, u32 stream,
-				     struct v4l2_subdev_format *fmt,
+v4l2_subdev_link_validate_get_format(struct media_pad *pad,
+				     struct media_device_context *mdev_context,
+				     u32 stream, struct v4l2_subdev_format *fmt,
 				     bool states_locked)
 {
 	struct v4l2_subdev_state *state;
@@ -1533,9 +1599,11 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad, u32 stream,
 	fmt->stream = stream;
 
 	if (states_locked)
-		state = v4l2_subdev_get_locked_active_state(sd);
+		state = v4l2_subdev_get_locked_state_from_mdev_ctx(sd,
+								mdev_context);
 	else
-		state = v4l2_subdev_lock_and_get_active_state(sd);
+		state = v4l2_subdev_lock_and_get_state_from_mdev_ctx(sd,
+								mdev_context);
 
 	ret = v4l2_subdev_call(sd, pad, get_fmt, state, fmt);
 
@@ -1548,6 +1616,7 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad, u32 stream,
 #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
 
 static void __v4l2_link_validate_get_streams(struct media_pad *pad,
+					     struct media_device_context *mdev_context,
 					     u64 *streams_mask,
 					     bool states_locked)
 {
@@ -1560,10 +1629,11 @@ static void __v4l2_link_validate_get_streams(struct media_pad *pad,
 	*streams_mask = 0;
 
 	if (states_locked)
-		state = v4l2_subdev_get_locked_active_state(subdev);
+		state = v4l2_subdev_get_locked_state_from_mdev_ctx(subdev,
+								mdev_context);
 	else
-		state = v4l2_subdev_lock_and_get_active_state(subdev);
-
+		state = v4l2_subdev_lock_and_get_state_from_mdev_ctx(subdev,
+								mdev_context);
 	if (WARN_ON(!state))
 		return;
 
@@ -1592,6 +1662,7 @@ static void __v4l2_link_validate_get_streams(struct media_pad *pad,
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 static void v4l2_link_validate_get_streams(struct media_pad *pad,
+					   struct media_device_context *mdev_context,
 					   u64 *streams_mask,
 					   bool states_locked)
 {
@@ -1604,14 +1675,17 @@ static void v4l2_link_validate_get_streams(struct media_pad *pad,
 	}
 
 #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
-	__v4l2_link_validate_get_streams(pad, streams_mask, states_locked);
+	__v4l2_link_validate_get_streams(pad, mdev_context, streams_mask,
+					 states_locked);
 #else
 	/* This shouldn't happen */
 	*streams_mask = 0;
 #endif
 }
 
-static int v4l2_subdev_link_validate_locked(struct media_link *link, bool states_locked)
+static int v4l2_subdev_link_validate_locked(struct media_link *link,
+					    struct media_device_context *mdev_context,
+					    bool states_locked)
 {
 	struct v4l2_subdev *sink_subdev =
 		media_entity_to_v4l2_subdev(link->sink->entity);
@@ -1626,8 +1700,10 @@ static int v4l2_subdev_link_validate_locked(struct media_link *link, bool states
 		link->source->entity->name, link->source->index,
 		link->sink->entity->name, link->sink->index);
 
-	v4l2_link_validate_get_streams(link->source, &source_streams_mask, states_locked);
-	v4l2_link_validate_get_streams(link->sink, &sink_streams_mask, states_locked);
+	v4l2_link_validate_get_streams(link->source, mdev_context,
+				       &source_streams_mask, states_locked);
+	v4l2_link_validate_get_streams(link->sink, mdev_context,
+				       &sink_streams_mask, states_locked);
 
 	/*
 	 * It is ok to have more source streams than sink streams as extra
@@ -1654,7 +1730,8 @@ static int v4l2_subdev_link_validate_locked(struct media_link *link, bool states
 			link->source->entity->name, link->source->index, stream,
 			link->sink->entity->name, link->sink->index, stream);
 
-		ret = v4l2_subdev_link_validate_get_format(link->source, stream,
+		ret = v4l2_subdev_link_validate_get_format(link->source,
+							   mdev_context, stream,
 							   &source_fmt, states_locked);
 		if (ret < 0) {
 			dev_dbg(dev,
@@ -1664,7 +1741,8 @@ static int v4l2_subdev_link_validate_locked(struct media_link *link, bool states
 			continue;
 		}
 
-		ret = v4l2_subdev_link_validate_get_format(link->sink, stream,
+		ret = v4l2_subdev_link_validate_get_format(link->sink,
+							   mdev_context, stream,
 							   &sink_fmt, states_locked);
 		if (ret < 0) {
 			dev_dbg(dev,
@@ -1693,7 +1771,8 @@ static int v4l2_subdev_link_validate_locked(struct media_link *link, bool states
 	return 0;
 }
 
-int v4l2_subdev_link_validate(struct media_link *link)
+int __v4l2_subdev_link_validate(struct media_link *link,
+				struct media_device_context *mdev_context)
 {
 	struct v4l2_subdev *source_sd, *sink_sd;
 	struct v4l2_subdev_state *source_state, *sink_state;
@@ -1716,28 +1795,35 @@ int v4l2_subdev_link_validate(struct media_link *link)
 	if (is_media_entity_v4l2_video_device(link->source->entity)) {
 		struct media_entity *source = link->source->entity;
 
-		if (!source->ops || !source->ops->link_validate) {
+		if (!source->ops ||
+		    (mdev_context && !source->ops->link_validate_context) ||
+		    (!mdev_context && !source->ops->link_validate)) {
 			/*
-			 * Many existing drivers do not implement the required
-			 * .link_validate() operation for their video devices.
-			 * Print a warning to get the drivers fixed, and return
-			 * 0 to avoid breaking userspace. This should
-			 * eventually be turned into a WARN_ON() when all
-			 * drivers will have been fixed.
+			 * Many existing drivers do not implement the correct
+			 * .link_validate() or .link_validate_context()
+			 * operations for their video devices. Print a warning
+			 * to get the drivers fixed, and return 0 to avoid
+			 * breaking userspace. This should eventually be turned
+			 * into a WARN_ON() when all drivers will have been
+			 * fixed.
 			 */
-			pr_warn_once("video device '%s' does not implement .link_validate(), driver bug!\n",
+			pr_warn_once("video device '%s' does not implement the correct .link_validate operation: driver bug!\n",
 				     source->name);
 			return 0;
 		}
 
 		/*
 		 * Avoid infinite loops in case a video device incorrectly uses
-		 * this helper function as its .link_validate() handler.
+		 * this helper function as its .link_validate[_context]()
+		 * handler.
 		 */
-		if (WARN_ON(source->ops->link_validate == v4l2_subdev_link_validate))
+		if (WARN_ON(source->ops->link_validate == v4l2_subdev_link_validate ||
+			    source->ops->link_validate_context == v4l2_subdev_link_validate_context))
 			return -EINVAL;
 
-		return source->ops->link_validate(link);
+		return (mdev_context && source->ops->link_validate_context) ?
+		       source->ops->link_validate_context(link, mdev_context) :
+		       source->ops->link_validate(link);
 	}
 
 	/*
@@ -1750,22 +1836,24 @@ int v4l2_subdev_link_validate(struct media_link *link)
 	sink_sd = media_entity_to_v4l2_subdev(link->sink->entity);
 	source_sd = media_entity_to_v4l2_subdev(link->source->entity);
 
-	sink_state = v4l2_subdev_get_unlocked_active_state(sink_sd);
-	source_state = v4l2_subdev_get_unlocked_active_state(source_sd);
-
+	sink_state = v4l2_subdev_get_unlocked_state_from_mdev_ctx(sink_sd,
+								  mdev_context);
+	source_state = v4l2_subdev_get_unlocked_state_from_mdev_ctx(source_sd,
+								    mdev_context);
 	states_locked = sink_state && source_state;
 
 	if (states_locked)
 		v4l2_subdev_lock_states(sink_state, source_state);
 
-	ret = v4l2_subdev_link_validate_locked(link, states_locked);
+	ret = v4l2_subdev_link_validate_locked(link, mdev_context, states_locked);
 
 	if (states_locked)
 		v4l2_subdev_unlock_states(sink_state, source_state);
 
 	return ret;
+
 }
-EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
+EXPORT_SYMBOL_GPL(__v4l2_subdev_link_validate);
 
 bool v4l2_subdev_has_pad_interdep(struct media_entity *entity,
 				  unsigned int pad0, unsigned int pad1)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 8087c0ae3bc0a0a95512b4b0ff5257522a104ca0..16b6b265e711cf2293ce2478ef90a622beb869e5 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1223,6 +1223,7 @@ struct v4l2_subdev {
  * @vfh: pointer to &struct v4l2_fh
  * @state: pointer to &struct v4l2_subdev_state
  * @owner: module pointer to the owner of this file handle
+ * @context: pointer to subdevice context associated with the file handle
  * @client_caps: bitmask of ``V4L2_SUBDEV_CLIENT_CAP_*``
  */
 struct v4l2_subdev_fh {
@@ -1351,6 +1352,9 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
 				      struct v4l2_subdev_format *source_fmt,
 				      struct v4l2_subdev_format *sink_fmt);
 
+int __v4l2_subdev_link_validate(struct media_link *link,
+				struct media_device_context *mdev_context);
+
 /**
  * v4l2_subdev_link_validate - validates a media link
  *
@@ -1368,7 +1372,35 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
  * the video devices also implement their &media_entity_ops.link_validate
  * operation.
  */
-int v4l2_subdev_link_validate(struct media_link *link);
+static inline int v4l2_subdev_link_validate(struct media_link *link)
+{
+	return __v4l2_subdev_link_validate(link, NULL);
+}
+
+/**
+ * v4l2_subdev_link_validate_context - validates a media link in a media context
+ *
+ * @link: pointer to &struct media_link
+ * @mdev_context: the media device context
+ *
+ * This function calls the subdev's link_validate_context ops to validate
+ * if a media link is valid for streaming in a media device context. It also
+ * internally calls v4l2_subdev_link_validate_default() to ensure that width,
+ * height and the media bus pixel code are equal on both source and sink of the
+ * link.
+ *
+ * The function can be used as a drop-in &media_entity_ops.link_validate_context
+ * implementation for v4l2_subdev instances. It supports all links between
+ * subdevs, as well as links between subdevs and video devices, provided that
+ * the video devices also implement their
+ * &media_entity_ops.link_validate_context operation.
+ */
+static inline int
+v4l2_subdev_link_validate_context(struct media_link *link,
+				  struct media_device_context *mdev_context)
+{
+	return __v4l2_subdev_link_validate(link, mdev_context);
+}
 
 /**
  * v4l2_subdev_has_pad_interdep - MC has_pad_interdep implementation for subdevs

-- 
2.49.0

[PATCH DNI 21/26] media: pisp_be: Start and stop the media pipeline

[Jacopo Mondi]

Call video_device_pipeline_alloc_start() and
video_device_pipeline_stop() at streaming start/stop time for all
video nodes.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/platform/raspberrypi/pisp_be/pisp_be.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c b/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
index b30891718d8df9c48ce1b83ad9fcafb201105625..423cb21298309c2ba51214b129fbf6e875370c98 100644
--- a/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
+++ b/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
@@ -878,6 +878,10 @@ static int pispbe_node_start_streaming(struct vb2_queue *q, unsigned int count)
 	if (ret < 0)
 		goto err_return_buffers;
 
+	ret = video_device_pipeline_alloc_start(&node->vfd);
+	if (ret)
+		goto err_return_buffers;
+
 	scoped_guard(spinlock_irq, &pispbe->hw_lock) {
 		node->pispbe->streaming_map |=  BIT(node->id);
 		node->pispbe->sequence = 0;
@@ -933,6 +937,8 @@ static void pispbe_node_stop_streaming(struct vb2_queue *q)
 
 	vb2_wait_for_all_buffers(&node->queue);
 
+	video_device_pipeline_stop(&node->vfd);
+
 	spin_lock_irq(&pispbe->hw_lock);
 	pispbe->streaming_map &= ~BIT(node->id);
 

-- 
2.49.0

[PATCH DNI 22/26] media: pisp_be: Add support for subdev state

[Jacopo Mondi]

Add support for subdev state in the ISP subdevice by providing an
init_state() callback and by calling init_finalize() before registering
the subdev.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 .../media/platform/raspberrypi/pisp_be/pisp_be.c   | 41 ++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c b/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
index 423cb21298309c2ba51214b129fbf6e875370c98..5aec4f8979053500c870e71ce7171bbd1cac9606 100644
--- a/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
+++ b/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
@@ -1496,12 +1496,49 @@ static const struct v4l2_subdev_ops pispbe_sd_ops = {
 	.pad = &pispbe_pad_ops,
 };
 
+static int pispbe_init_state(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_state *state)
+{
+	struct v4l2_mbus_framefmt *fmt;
+
+	for (unsigned int i = 0; i < PISPBE_NUM_NODES; i++) {
+		fmt = v4l2_subdev_state_get_format(state, i);
+
+		switch (i) {
+		case MAIN_INPUT_NODE:
+			fallthrough;
+		case OUTPUT0_NODE:
+			fallthrough;
+		case OUTPUT1_NODE:
+			fmt->width = 1920;
+			fmt->height = 1080;
+			fmt->code = MEDIA_BUS_FMT_FIXED;
+			break;
+		case CONFIG_NODE:
+			fmt->width =  sizeof(struct pisp_be_tiles_config);
+			fmt->height = 1;
+			fmt->code = MEDIA_BUS_FMT_METADATA_FIXED;
+			break;
+		default:
+			/* No need to configure other nodes. */
+			continue;
+		}
+	}
+
+	return 0;
+}
+
+static const struct v4l2_subdev_internal_ops pispbe_subdev_internal_ops = {
+	.init_state = pispbe_init_state,
+};
+
 static int pispbe_init_subdev(struct pispbe_dev *pispbe)
 {
 	struct v4l2_subdev *sd = &pispbe->sd;
 	int ret;
 
 	v4l2_subdev_init(sd, &pispbe_sd_ops);
+	sd->internal_ops = &pispbe_subdev_internal_ops;
 	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
 	sd->owner = THIS_MODULE;
 	sd->dev = pispbe->dev;
@@ -1517,6 +1554,10 @@ static int pispbe_init_subdev(struct pispbe_dev *pispbe)
 	if (ret)
 		goto error;
 
+	ret = v4l2_subdev_init_finalize(sd);
+	if (ret)
+		goto error;
+
 	ret = v4l2_device_register_subdev(&pispbe->v4l2_dev, sd);
 	if (ret)
 		goto error;

-- 
2.49.0

[PATCH DNI 23/26] media: pisp_be: Implement set/get_pad_fmt

[Jacopo Mondi]

Implement operation handlers for the set and get pad_fmt subdev
pad operations.

Format is only modifiable on the image nodes (of both output and capture
types). Only sizes can be modified.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 .../media/platform/raspberrypi/pisp_be/pisp_be.c   | 29 ++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c b/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
index 5aec4f8979053500c870e71ce7171bbd1cac9606..22e440f387b5e5560b2cc80a8b3bf6064dc12d7c 100644
--- a/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
+++ b/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
@@ -1488,7 +1488,36 @@ static int pispbe_init_node(struct pispbe_dev *pispbe, unsigned int id)
 	return ret;
 }
 
+static int pispbe_subdev_set_pad_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_format *format)
+{
+	struct v4l2_mbus_framefmt *fmt = &format->format;
+
+	/* Only allow setting sizes on the image input and output pads. */
+
+	switch (format->pad) {
+	case TDN_INPUT_NODE:
+	case STITCH_INPUT_NODE:
+	case TDN_OUTPUT_NODE:
+	case STITCH_OUTPUT_NODE:
+		return -EINVAL;
+	}
+
+	fmt->width = clamp(fmt->width, PISP_BACK_END_MIN_TILE_WIDTH,
+			   PISP_BACK_END_MAX_TILE_WIDTH);
+	fmt->height = clamp(fmt->height, PISP_BACK_END_MIN_TILE_HEIGHT,
+			    PISP_BACK_END_MAX_TILE_HEIGHT);
+	fmt->code = MEDIA_BUS_FMT_FIXED;
+
+	*v4l2_subdev_state_get_format(state, format->pad) = *fmt;
+
+	return 0;
+}
+
 static const struct v4l2_subdev_pad_ops pispbe_pad_ops = {
+	.set_fmt = pispbe_subdev_set_pad_fmt,
+	.get_fmt = v4l2_subdev_get_fmt,
 	.link_validate = v4l2_subdev_link_validate_default,
 };
 

-- 
2.49.0

[PATCH DNI 24/26] media: pisp_be: Implement link validation

[Jacopo Mondi]

Implement link validation to validate that the sizes of the format on
the video device matches the size programmed on the ISP subdevice
connected pad.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 .../media/platform/raspberrypi/pisp_be/pisp_be.c   | 67 ++++++++++++++++++++++
 1 file changed, 67 insertions(+)

diff --git a/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c b/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
index 22e440f387b5e5560b2cc80a8b3bf6064dc12d7c..2a8c09a9c70952c9f99e542271e994d62392c617 100644
--- a/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
+++ b/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
@@ -1362,6 +1362,67 @@ static const struct v4l2_ioctl_ops pispbe_node_ioctl_ops = {
 	.vidioc_streamoff = vb2_ioctl_streamoff,
 };
 
+static int pispbe_link_validate(struct media_link *link)
+{
+	const struct v4l2_mbus_framefmt *sd_fmt;
+	struct v4l2_pix_format_mplane *pix_mp;
+	struct v4l2_subdev_state *state;
+	struct media_entity *vdev_ent;
+	struct media_entity *sd_ent;
+	struct pispbe_node *node;
+	struct v4l2_subdev *sd;
+
+	if (is_media_entity_v4l2_video_device(link->source->entity)) {
+		vdev_ent = link->source->entity;
+		sd_ent = link->sink->entity;
+	} else {
+		sd_ent = link->source->entity;
+		vdev_ent = link->sink->entity;
+	}
+
+	node = container_of(media_entity_to_video_device(vdev_ent),
+			    struct pispbe_node, vfd);
+	switch (node->id) {
+	case TDN_INPUT_NODE:
+		fallthrough;
+	case STITCH_INPUT_NODE:
+		fallthrough;
+	case TDN_OUTPUT_NODE:
+		fallthrough;
+	case STITCH_OUTPUT_NODE:
+		fallthrough;
+	case CONFIG_NODE:
+		/* Skip validation for these nodes. */
+		return 0;
+	}
+	pix_mp = &node->format.fmt.pix_mp;
+
+	sd = media_entity_to_v4l2_subdev(sd_ent);
+	state = v4l2_subdev_get_unlocked_active_state(sd);
+	sd_fmt = v4l2_subdev_state_get_format(state, node->id);
+
+	/* Only check for sizes. */
+	if (pix_mp->width != sd_fmt->width) {
+		dev_dbg(node->pispbe->dev,
+			"%s: width does not match (vdev %u, sd %u)\n",
+			__func__, pix_mp->width, sd_fmt->width);
+		return -EPIPE;
+	}
+
+	if (pix_mp->height != sd_fmt->height) {
+		dev_dbg(node->pispbe->dev,
+			"%s: height does not match (vdev %u, sd %u)\n",
+			__func__, pix_mp->height, sd_fmt->height);
+		return -EPIPE;
+	}
+
+	return 0;
+}
+
+static const struct media_entity_operations pispbe_node_entity_ops = {
+	.link_validate = pispbe_link_validate,
+};
+
 static const struct video_device pispbe_videodev = {
 	.name = PISPBE_NAME,
 	.vfl_dir = VFL_DIR_M2M, /* gets overwritten */
@@ -1445,6 +1506,7 @@ static int pispbe_init_node(struct pispbe_dev *pispbe, unsigned int id)
 	vdev->device_caps = V4L2_CAP_STREAMING | node_desc[id].caps;
 
 	node->pad.flags = output ? MEDIA_PAD_FL_SOURCE : MEDIA_PAD_FL_SINK;
+	entity->ops = &pispbe_node_entity_ops;
 	ret = media_entity_pads_init(entity, 1, &node->pad);
 	if (ret) {
 		dev_err(pispbe->dev,
@@ -1561,6 +1623,10 @@ static const struct v4l2_subdev_internal_ops pispbe_subdev_internal_ops = {
 	.init_state = pispbe_init_state,
 };
 
+static const struct media_entity_operations pispbe_subdev_entity_ops = {
+	.link_validate = v4l2_subdev_link_validate,
+};
+
 static int pispbe_init_subdev(struct pispbe_dev *pispbe)
 {
 	struct v4l2_subdev *sd = &pispbe->sd;
@@ -1569,6 +1635,7 @@ static int pispbe_init_subdev(struct pispbe_dev *pispbe)
 	v4l2_subdev_init(sd, &pispbe_sd_ops);
 	sd->internal_ops = &pispbe_subdev_internal_ops;
 	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+	sd->entity.ops = &pispbe_subdev_entity_ops;
 	sd->owner = THIS_MODULE;
 	sd->dev = pispbe->dev;
 	strscpy(sd->name, PISPBE_NAME, sizeof(sd->name));

-- 
2.49.0

[PATCH DNI 25/26] media: pisp_be: Register devnode to userspace

[Jacopo Mondi]

To set formats on the subdev, we need to have the devnode registered to
userspace.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/platform/raspberrypi/pisp_be/pisp_be.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c b/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
index 2a8c09a9c70952c9f99e542271e994d62392c617..df3cdd81843376abf98bb184cde74d4d66b0ecfe 100644
--- a/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
+++ b/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
@@ -1635,6 +1635,7 @@ static int pispbe_init_subdev(struct pispbe_dev *pispbe)
 	v4l2_subdev_init(sd, &pispbe_sd_ops);
 	sd->internal_ops = &pispbe_subdev_internal_ops;
 	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+	sd->flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
 	sd->entity.ops = &pispbe_subdev_entity_ops;
 	sd->owner = THIS_MODULE;
 	sd->dev = pispbe->dev;
@@ -1658,7 +1659,7 @@ static int pispbe_init_subdev(struct pispbe_dev *pispbe)
 	if (ret)
 		goto error;
 
-	return 0;
+	return v4l2_device_register_subdev_nodes(&pispbe->v4l2_dev);
 
 error:
 	media_entity_cleanup(&sd->entity);

-- 
2.49.0

[PATCH DNI 26/26] media: pisp_be: Add support for multi-context

[Jacopo Mondi]

Add support for context multiplexing to the PiSP BackEnd driver.

Move information that used to be per-node to the video context and
global information to the media context.

Implement media and video context allocation and release operations
to allocate and initialize a context and initialize the video queue there
contained.

Remove the per-node video device format and the buffer queues and
port the driver to use the ones in the video device context.

The operations that used to work with a pispbe_node now operates on a
pispbe_context.

- v4l2 ioctl ops: receive a file pointer, retrieve the associated context
  from the open file handle
- vb2 ops: receive a queue, retrieve the context from the queue
- internal driver ops: given a media device context retrieve the video
  context from the other video devices when assembling a job

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 .../media/platform/raspberrypi/pisp_be/pisp_be.c   | 651 +++++++++++++++------
 1 file changed, 473 insertions(+), 178 deletions(-)

diff --git a/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c b/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
index df3cdd81843376abf98bb184cde74d4d66b0ecfe..742fc9dddbead84fb2c9a615c9dccb3afcc66a55 100644
--- a/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
+++ b/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c
@@ -146,6 +146,29 @@ static const struct pispbe_node_description node_desc[PISPBE_NUM_NODES] = {
 	((node)->buf_type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) || \
 	((node)->buf_type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE))
 
+/* ----------------------------------------------------------------------------
+ * Media device context
+ */
+
+struct pispbe_media_context {
+	struct media_device_context mdev_context;
+	u32 streaming_map;
+	unsigned int sequence;
+	dma_addr_t config_dma_addr;
+	struct pisp_be_tiles_config *config;
+};
+
+static struct pispbe_media_context *
+pispbe_media_context(struct media_device_context *mdev_context)
+{
+	return container_of(mdev_context, struct pispbe_media_context,
+			    mdev_context);
+}
+
+/* ----------------------------------------------------------------------------
+ * Video device context
+ */
+
 /*
  * Structure to describe a single node /dev/video<N> which represents a single
  * input or output queue to the PiSP Back End device.
@@ -161,14 +184,96 @@ struct pispbe_node {
 	struct pispbe_dev *pispbe;
 	/* Video device lock */
 	struct mutex node_lock;
-	/* vb2_queue lock */
-	struct mutex queue_lock;
-	struct list_head ready_queue;
-	struct vb2_queue queue;
+};
+
+static struct pispbe_node *pispbe_node_from_vdev(struct video_device *vdev)
+{
+	return container_of(vdev, struct pispbe_node, vfd);
+}
+
+/* Structure to describe a single execution context for a node. */
+
+ struct pispbe_node_context {
+	struct video_device_context vdev_context;
+
 	struct v4l2_format format;
+	struct list_head ready_queue;
 	const struct pisp_be_format *pisp_format;
+	struct pispbe_dev *pispbe;
+	struct pispbe_node *node;
+};
+
+static struct pispbe_node_context *
+pispbe_node_context(struct video_device_context *ctx)
+{
+	return container_of(ctx, struct pispbe_node_context, vdev_context);
+}
+
+static struct pispbe_node_context *
+pispbe_node_context_from_entity(struct media_entity_context *ctx)
+{
+	struct video_device_context *vdev_context =
+			container_of(ctx, struct video_device_context, base);
+
+	return pispbe_node_context(vdev_context);
+}
+
+static struct pispbe_node_context *
+pispbe_node_context_from_queue(struct vb2_queue *queue)
+{
+	return pispbe_node_context(video_device_context_from_queue(queue));
+}
+
+static struct pispbe_node_context *
+pispbe_node_context_from_file(struct file *file, struct video_device *vfd)
+{
+	return pispbe_node_context(video_device_context_from_file(file, vfd));
+}
+
+static struct pispbe_node_context *
+pispbe_get_dev_context(struct pispbe_media_context *pispbe_mdev_context,
+			struct video_device *vdev)
+{
+	struct video_device_context *ctx =
+		video_device_context_get(&pispbe_mdev_context->mdev_context,
+					 vdev);
+
+	return ctx ? pispbe_node_context(ctx) : NULL;
+}
+
+static void pispbe_put_dev_context(struct pispbe_node_context *ctx)
+{
+	video_device_context_put(&ctx->vdev_context);
+}
+
+static struct pispbe_media_context *
+pispbe_media_context_from_dev(struct pispbe_node_context *context)
+{
+	return pispbe_media_context(context->vdev_context.base.mdev_context);
+}
+
+/* ----------------------------------------------------------------------------
+ * ISP subdevice context
+ */
+struct pispbe_subdev_context {
+	struct v4l2_subdev_context sd_context;
 };
 
+static struct pispbe_subdev_context *
+pispbe_subdev_context(struct v4l2_subdev_context *ctx)
+{
+	return container_of(ctx, struct pispbe_subdev_context, sd_context);
+}
+
+static struct pispbe_subdev_context *
+pispbe_subdev_context_from_entity(struct media_entity_context *ctx)
+{
+	struct v4l2_subdev_context *sd_context =
+		container_of(ctx, struct v4l2_subdev_context, base);
+
+	return pispbe_subdev_context(sd_context);
+}
+
 /* For logging only, use the entity name with "pispbe" and separator removed */
 #define NODE_NAME(node) \
 		(node_desc[(node)->id].ent_name + sizeof(PISPBE_NAME))
@@ -181,6 +286,7 @@ struct pispbe_job {
 	 * then captures, then metadata last.
 	 */
 	struct pispbe_buffer *buf[PISPBE_NUM_NODES];
+	struct pispbe_media_context *context;
 };
 
 struct pispbe_hw_enables {
@@ -196,6 +302,7 @@ struct pispbe_job_descriptor {
 	struct pisp_be_tiles_config *config;
 	struct pispbe_hw_enables hw_enables;
 	dma_addr_t tiles;
+	struct pispbe_media_context *context;
 };
 
 /*
@@ -205,7 +312,6 @@ struct pispbe_job_descriptor {
 struct pispbe_dev {
 	struct device *dev;
 	struct pispbe_dev *pispbe;
-	struct pisp_be_tiles_config *config;
 	void __iomem *be_reg_base;
 	struct clk *clk;
 	struct v4l2_device v4l2_dev;
@@ -213,17 +319,15 @@ struct pispbe_dev {
 	struct media_device mdev;
 	struct media_pad pad[PISPBE_NUM_NODES]; /* output pads first */
 	struct pispbe_node node[PISPBE_NUM_NODES];
-	dma_addr_t config_dma_addr;
-	unsigned int sequence;
-	u32 streaming_map;
 	struct pispbe_job queued_job, running_job;
-	/* protects "hw_busy" flag, streaming_map and job_queue */
+	/* protects "hw_busy" flag and job_queue */
 	spinlock_t hw_lock;
 	bool hw_busy; /* non-zero if a job is queued or is being started */
 	struct list_head job_queue;
 	int irq;
 	u32 hw_version;
 	u8 done, started;
+	struct media_pipeline pipe;
 };
 
 static u32 pispbe_rd(struct pispbe_dev *pispbe, unsigned int offset)
@@ -307,30 +411,31 @@ struct pispbe_buffer {
 };
 
 static int pispbe_get_planes_addr(dma_addr_t addr[3], struct pispbe_buffer *buf,
-				  struct pispbe_node *node)
+				  struct pispbe_node_context *context)
 {
-	unsigned int num_planes = node->format.fmt.pix_mp.num_planes;
+	struct v4l2_format *format = &context->format;
+	unsigned int num_planes = format->fmt.pix_mp.num_planes;
 	unsigned int plane_factor = 0;
 	unsigned int size;
 	unsigned int p;
 
-	if (!buf || !node->pisp_format)
+	if (!buf || !context->pisp_format)
 		return 0;
 
 	/*
 	 * Determine the base plane size. This will not be the same
-	 * as node->format.fmt.pix_mp.plane_fmt[0].sizeimage for a single
+	 * as format->fmt.pix_mp.plane_fmt[0].sizeimage for a single
 	 * plane buffer in an mplane format.
 	 */
-	size = node->format.fmt.pix_mp.plane_fmt[0].bytesperline *
-	       node->format.fmt.pix_mp.height;
+	size = format->fmt.pix_mp.plane_fmt[0].bytesperline *
+	       format->fmt.pix_mp.height;
 
 	for (p = 0; p < num_planes && p < PISPBE_MAX_PLANES; p++) {
 		addr[p] = vb2_dma_contig_plane_dma_addr(&buf->vb.vb2_buf, p);
-		plane_factor += node->pisp_format->plane_factor[p];
+		plane_factor += context->pisp_format->plane_factor[p];
 	}
 
-	for (; p < PISPBE_MAX_PLANES && node->pisp_format->plane_factor[p]; p++) {
+	for (; p < PISPBE_MAX_PLANES && context->pisp_format->plane_factor[p]; p++) {
 		/*
 		 * Calculate the address offset of this plane as needed
 		 * by the hardware. This is specifically for non-mplane
@@ -338,7 +443,7 @@ static int pispbe_get_planes_addr(dma_addr_t addr[3], struct pispbe_buffer *buf,
 		 * for the V4L2_PIX_FMT_YUV420 format.
 		 */
 		addr[p] = addr[0] + ((size * plane_factor) >> 3);
-		plane_factor += node->pisp_format->plane_factor[p];
+		plane_factor += context->pisp_format->plane_factor[p];
 	}
 
 	return num_planes;
@@ -354,11 +459,13 @@ static dma_addr_t pispbe_get_addr(struct pispbe_buffer *buf)
 
 static void pispbe_xlate_addrs(struct pispbe_dev *pispbe,
 			       struct pispbe_job_descriptor *job,
+			       struct pispbe_media_context *mdev_context,
 			       struct pispbe_buffer *buf[PISPBE_NUM_NODES])
 {
 	struct pispbe_hw_enables *hw_en = &job->hw_enables;
 	struct pisp_be_tiles_config *config = job->config;
 	dma_addr_t *addrs = job->hw_dma_addrs;
+	struct pispbe_node_context *ctx;
 	int ret;
 
 	/* Take a copy of the "enable" bitmaps so we can modify them. */
@@ -369,8 +476,10 @@ static void pispbe_xlate_addrs(struct pispbe_dev *pispbe,
 	 * Main input first. There are 3 address pointers, corresponding to up
 	 * to 3 planes.
 	 */
-	ret = pispbe_get_planes_addr(addrs, buf[MAIN_INPUT_NODE],
-				     &pispbe->node[MAIN_INPUT_NODE]);
+	ctx = pispbe_get_dev_context(mdev_context,
+				     &pispbe->node[MAIN_INPUT_NODE].vfd);
+	ret = pispbe_get_planes_addr(addrs, buf[MAIN_INPUT_NODE], ctx);
+	pispbe_put_dev_context(ctx);
 	if (ret <= 0) {
 		/* Shouldn't happen, we have validated an input is available. */
 		dev_warn(pispbe->dev, "ISP-BE missing input\n");
@@ -426,9 +535,11 @@ static void pispbe_xlate_addrs(struct pispbe_dev *pispbe,
 
 	/* Main image output channels. */
 	for (unsigned int i = 0; i < PISP_BACK_END_NUM_OUTPUTS; i++) {
+		ctx = pispbe_get_dev_context(mdev_context,
+					     &pispbe->node[OUTPUT0_NODE + i].vfd);
 		ret = pispbe_get_planes_addr(addrs + 7 + 3 * i,
-					     buf[OUTPUT0_NODE + i],
-					     &pispbe->node[OUTPUT0_NODE + i]);
+					     buf[OUTPUT0_NODE + i], ctx);
+		pispbe_put_dev_context(ctx);
 		if (ret <= 0)
 			hw_en->rgb_enables &= ~(PISP_BE_RGB_ENABLE_OUTPUT0 << i);
 	}
@@ -449,37 +560,48 @@ static void pispbe_xlate_addrs(struct pispbe_dev *pispbe,
  *
  * Returns 0 if a job has been successfully prepared, < 0 otherwise.
  */
-static int pispbe_prepare_job(struct pispbe_dev *pispbe)
+static int pispbe_prepare_job(struct pispbe_dev *pispbe,
+			      struct pispbe_node_context *context)
 {
+	struct pispbe_media_context *mdev_context =
+					pispbe_media_context_from_dev(context);
 	struct pispbe_job_descriptor __free(kfree) *job = NULL;
 	struct pispbe_buffer *buf[PISPBE_NUM_NODES] = {};
+	struct pispbe_node_context *ctx;
 	unsigned int streaming_map;
 	unsigned int config_index;
-	struct pispbe_node *node;
 
 	lockdep_assert_irqs_enabled();
 
 	scoped_guard(spinlock_irq, &pispbe->hw_lock) {
 		static const u32 mask = BIT(CONFIG_NODE) | BIT(MAIN_INPUT_NODE);
 
-		if ((pispbe->streaming_map & mask) != mask)
+		if ((mdev_context->streaming_map & mask) != mask)
 			return -ENODEV;
 
 		/*
 		 * Take a copy of streaming_map: nodes activated after this
 		 * point are ignored when preparing this job.
 		 */
-		streaming_map = pispbe->streaming_map;
+		streaming_map = mdev_context->streaming_map;
 	}
 
 	job = kzalloc(sizeof(*job), GFP_KERNEL);
 	if (!job)
 		return -ENOMEM;
 
-	node = &pispbe->node[CONFIG_NODE];
-	buf[CONFIG_NODE] = list_first_entry_or_null(&node->ready_queue,
+	job->context = mdev_context;
+
+	ctx = pispbe_get_dev_context(mdev_context,
+				     &pispbe->node[CONFIG_NODE].vfd);
+	if (!ctx)
+		return -ENODEV;
+
+	buf[CONFIG_NODE] = list_first_entry_or_null(&ctx->ready_queue,
 						    struct pispbe_buffer,
 						    ready_list);
+	pispbe_put_dev_context(ctx);
+
 	if (!buf[CONFIG_NODE])
 		return -ENODEV;
 
@@ -487,8 +609,8 @@ static int pispbe_prepare_job(struct pispbe_dev *pispbe)
 	job->buffers[CONFIG_NODE] = buf[CONFIG_NODE];
 
 	config_index = buf[CONFIG_NODE]->vb.vb2_buf.index;
-	job->config = &pispbe->config[config_index];
-	job->tiles = pispbe->config_dma_addr +
+	job->config = &mdev_context->config[config_index];
+	job->tiles = mdev_context->config_dma_addr +
 		     config_index * sizeof(struct pisp_be_tiles_config) +
 		     offsetof(struct pisp_be_tiles_config, tiles);
 
@@ -529,12 +651,17 @@ static int pispbe_prepare_job(struct pispbe_dev *pispbe)
 			ignore_buffers = true;
 		}
 
-		node = &pispbe->node[i];
-
 		/* Pull a buffer from each V4L2 queue to form the queued job */
-		buf[i] = list_first_entry_or_null(&node->ready_queue,
+		ctx = pispbe_get_dev_context(mdev_context,
+					     &pispbe->node[i].vfd);
+		if (!ctx && !ignore_buffers)
+			goto err_return_buffers;
+
+		buf[i] = list_first_entry_or_null(&ctx->ready_queue,
 						  struct pispbe_buffer,
 						  ready_list);
+		pispbe_put_dev_context(ctx);
+
 		if (buf[i]) {
 			list_del(&buf[i]->ready_list);
 			job->buffers[i] = buf[i];
@@ -545,7 +672,7 @@ static int pispbe_prepare_job(struct pispbe_dev *pispbe)
 	}
 
 	/* Convert buffers to DMA addresses for the hardware */
-	pispbe_xlate_addrs(pispbe, job, buf);
+	pispbe_xlate_addrs(pispbe, job, mdev_context, buf);
 
 	scoped_guard(spinlock_irq, &pispbe->hw_lock) {
 		list_add_tail(&job->queue, &pispbe->job_queue);
@@ -558,19 +685,25 @@ static int pispbe_prepare_job(struct pispbe_dev *pispbe)
 
 err_return_buffers:
 	for (unsigned int i = 0; i < PISPBE_NUM_NODES; i++) {
-		struct pispbe_node *n =  &pispbe->node[i];
-
 		if (!buf[i])
 			continue;
 
 		/* Return the buffer to the ready_list queue */
-		list_add(&buf[i]->ready_list, &n->ready_queue);
+		ctx = pispbe_get_dev_context(mdev_context,
+					     &pispbe->node[i].vfd);
+		if (!ctx)
+			continue;
+
+		list_add(&buf[i]->ready_list, &ctx->ready_queue);
+		pispbe_put_dev_context(ctx);
 	}
 
 	return -ENODEV;
 }
 
-static void pispbe_schedule(struct pispbe_dev *pispbe, bool clear_hw_busy)
+static void pispbe_schedule(struct pispbe_dev *pispbe,
+			    struct pispbe_media_context *mdev_context,
+			    bool clear_hw_busy)
 {
 	struct pispbe_job_descriptor *job;
 
@@ -591,9 +724,11 @@ static void pispbe_schedule(struct pispbe_dev *pispbe, bool clear_hw_busy)
 
 		for (unsigned int i = 0; i < PISPBE_NUM_NODES; i++)
 			pispbe->queued_job.buf[i] = job->buffers[i];
-		pispbe->queued_job.valid = true;
 
 		pispbe->hw_busy = true;
+
+		pispbe->queued_job.context = job->context;
+		pispbe->queued_job.valid = true;
 	}
 
 	/*
@@ -615,13 +750,13 @@ static void pispbe_isr_jobdone(struct pispbe_dev *pispbe,
 	for (unsigned int i = 0; i < PISPBE_NUM_NODES; i++) {
 		if (buf[i]) {
 			buf[i]->vb.vb2_buf.timestamp = ts;
-			buf[i]->vb.sequence = pispbe->sequence;
+			buf[i]->vb.sequence = job->context->sequence;
 			vb2_buffer_done(&buf[i]->vb.vb2_buf,
 					VB2_BUF_STATE_DONE);
 		}
 	}
 
-	pispbe->sequence++;
+	job->context->sequence++;
 }
 
 static irqreturn_t pispbe_isr(int irq, void *dev)
@@ -674,18 +809,20 @@ static irqreturn_t pispbe_isr(int irq, void *dev)
 	}
 
 	/* check if there's more to do before going to sleep */
-	pispbe_schedule(pispbe, can_queue_another);
+	pispbe_schedule(pispbe, NULL, can_queue_another);
 
 	return IRQ_HANDLED;
 }
 
 static int pisp_be_validate_config(struct pispbe_dev *pispbe,
+				   struct pispbe_media_context *mdev_context,
 				   struct pisp_be_tiles_config *config)
 {
 	u32 bayer_enables = config->config.global.bayer_enables;
 	u32 rgb_enables = config->config.global.rgb_enables;
+	struct pispbe_node_context *context;
+	struct v4l2_pix_format_mplane *mp;
 	struct device *dev = pispbe->dev;
-	struct v4l2_format *fmt;
 	unsigned int bpl, size;
 
 	if (!(bayer_enables & PISP_BE_BAYER_ENABLE_INPUT) ==
@@ -702,36 +839,50 @@ static int pisp_be_validate_config(struct pispbe_dev *pispbe,
 	}
 
 	/* Ensure output config strides and buffer sizes match the V4L2 formats. */
-	fmt = &pispbe->node[TDN_OUTPUT_NODE].format;
+	context = pispbe_get_dev_context(mdev_context,
+					 &pispbe->node[TDN_OUTPUT_NODE].vfd);
+	if (!context)
+		return -EINVAL;
+
+	mp = &context->format.fmt.pix_mp;
+	pispbe_put_dev_context(context);
+
 	if (bayer_enables & PISP_BE_BAYER_ENABLE_TDN_OUTPUT) {
 		bpl = config->config.tdn_output_format.stride;
 		size = bpl * config->config.tdn_output_format.height;
 
-		if (fmt->fmt.pix_mp.plane_fmt[0].bytesperline < bpl) {
+		if (mp->plane_fmt[0].bytesperline < bpl) {
 			dev_dbg(dev, "%s: bpl mismatch on tdn_output\n",
 				__func__);
 			return -EINVAL;
 		}
 
-		if (fmt->fmt.pix_mp.plane_fmt[0].sizeimage < size) {
+		if (mp->plane_fmt[0].sizeimage < size) {
 			dev_dbg(dev, "%s: size mismatch on tdn_output\n",
 				__func__);
 			return -EINVAL;
 		}
 	}
 
-	fmt = &pispbe->node[STITCH_OUTPUT_NODE].format;
+	context = pispbe_get_dev_context(mdev_context,
+					 &pispbe->node[STITCH_OUTPUT_NODE].vfd);
+	if (!context)
+		return -EINVAL;
+
+	mp = &context->format.fmt.pix_mp;
+	pispbe_put_dev_context(context);
+
 	if (bayer_enables & PISP_BE_BAYER_ENABLE_STITCH_OUTPUT) {
 		bpl = config->config.stitch_output_format.stride;
 		size = bpl * config->config.stitch_output_format.height;
 
-		if (fmt->fmt.pix_mp.plane_fmt[0].bytesperline < bpl) {
+		if (mp->plane_fmt[0].bytesperline < bpl) {
 			dev_dbg(dev, "%s: bpl mismatch on stitch_output\n",
 				__func__);
 			return -EINVAL;
 		}
 
-		if (fmt->fmt.pix_mp.plane_fmt[0].sizeimage < size) {
+		if (mp->plane_fmt[0].sizeimage < size) {
 			dev_dbg(dev, "%s: size mismatch on stitch_output\n",
 				__func__);
 			return -EINVAL;
@@ -746,8 +897,15 @@ static int pisp_be_validate_config(struct pispbe_dev *pispbe,
 		    PISP_IMAGE_FORMAT_WALLPAPER_ROLL)
 			continue; /* TODO: Size checks for wallpaper formats */
 
-		fmt = &pispbe->node[OUTPUT0_NODE + j].format;
-		for (unsigned int i = 0; i < fmt->fmt.pix_mp.num_planes; i++) {
+		context = pispbe_get_dev_context(mdev_context,
+					  &pispbe->node[OUTPUT0_NODE + j].vfd);
+		if (!context)
+			return -EINVAL;
+
+		mp = &context->format.fmt.pix_mp;
+		pispbe_put_dev_context(context);
+
+		for (unsigned int i = 0; i < mp->num_planes; i++) {
 			bpl = !i ? config->config.output_format[j].image.stride
 			    : config->config.output_format[j].image.stride2;
 			size = bpl * config->config.output_format[j].image.height;
@@ -756,13 +914,15 @@ static int pisp_be_validate_config(struct pispbe_dev *pispbe,
 						PISP_IMAGE_FORMAT_SAMPLING_420)
 				size >>= 1;
 
-			if (fmt->fmt.pix_mp.plane_fmt[i].bytesperline < bpl) {
+			if (mp->plane_fmt[i].bytesperline < bpl) {
 				dev_dbg(dev, "%s: bpl mismatch on output %d\n",
 					__func__, j);
+				dev_dbg(dev, "Expected %u, got %u\n",
+					bpl, mp->plane_fmt[i].bytesperline);
 				return -EINVAL;
 			}
 
-			if (fmt->fmt.pix_mp.plane_fmt[i].sizeimage < size) {
+			if (mp->plane_fmt[i].sizeimage < size) {
 				dev_dbg(dev, "%s: size mismatch on output\n",
 					__func__);
 				return -EINVAL;
@@ -777,10 +937,12 @@ static int pispbe_node_queue_setup(struct vb2_queue *q, unsigned int *nbuffers,
 				   unsigned int *nplanes, unsigned int sizes[],
 				   struct device *alloc_devs[])
 {
+	struct pispbe_node_context *context = pispbe_node_context_from_queue(q);
+	struct v4l2_format *format = &context->format;
 	struct pispbe_node *node = vb2_get_drv_priv(q);
 	struct pispbe_dev *pispbe = node->pispbe;
 	unsigned int num_planes = NODE_IS_MPLANE(node) ?
-				  node->format.fmt.pix_mp.num_planes : 1;
+				  format->fmt.pix_mp.num_planes : 1;
 
 	if (*nplanes) {
 		if (*nplanes != num_planes)
@@ -788,8 +950,8 @@ static int pispbe_node_queue_setup(struct vb2_queue *q, unsigned int *nbuffers,
 
 		for (unsigned int i = 0; i < *nplanes; i++) {
 			unsigned int size = NODE_IS_MPLANE(node) ?
-				node->format.fmt.pix_mp.plane_fmt[i].sizeimage :
-				node->format.fmt.meta.buffersize;
+				format->fmt.pix_mp.plane_fmt[i].sizeimage :
+				format->fmt.meta.buffersize;
 
 			if (sizes[i] < size)
 				return -EINVAL;
@@ -801,8 +963,8 @@ static int pispbe_node_queue_setup(struct vb2_queue *q, unsigned int *nbuffers,
 	*nplanes = num_planes;
 	for (unsigned int i = 0; i < *nplanes; i++) {
 		unsigned int size = NODE_IS_MPLANE(node) ?
-				node->format.fmt.pix_mp.plane_fmt[i].sizeimage :
-				node->format.fmt.meta.buffersize;
+				    format->fmt.pix_mp.plane_fmt[i].sizeimage :
+				    format->fmt.meta.buffersize;
 		sizes[i] = size;
 	}
 
@@ -815,15 +977,20 @@ static int pispbe_node_queue_setup(struct vb2_queue *q, unsigned int *nbuffers,
 
 static int pispbe_node_buffer_prepare(struct vb2_buffer *vb)
 {
-	struct pispbe_node *node = vb2_get_drv_priv(vb->vb2_queue);
+	struct vb2_queue *q = vb->vb2_queue;
+	struct pispbe_node_context *context = pispbe_node_context_from_queue(q);
+	struct pispbe_media_context *mdev_context =
+					pispbe_media_context_from_dev(context);
+	struct v4l2_format *format = &context->format;
+	struct pispbe_node *node = context->node;
 	struct pispbe_dev *pispbe = node->pispbe;
 	unsigned int num_planes = NODE_IS_MPLANE(node) ?
-				  node->format.fmt.pix_mp.num_planes : 1;
+				  format->fmt.pix_mp.num_planes : 1;
 
 	for (unsigned int i = 0; i < num_planes; i++) {
 		unsigned long size = NODE_IS_MPLANE(node) ?
-				node->format.fmt.pix_mp.plane_fmt[i].sizeimage :
-				node->format.fmt.meta.buffersize;
+				     format->fmt.pix_mp.plane_fmt[i].sizeimage :
+				     format->fmt.meta.buffersize;
 
 		if (vb2_plane_size(vb, i) < size) {
 			dev_dbg(pispbe->dev,
@@ -836,12 +1003,12 @@ static int pispbe_node_buffer_prepare(struct vb2_buffer *vb)
 	}
 
 	if (node->id == CONFIG_NODE) {
-		void *dst = &node->pispbe->config[vb->index];
+		void *dst = &mdev_context->config[vb->index];
 		void *src = vb2_plane_vaddr(vb, 0);
 
 		memcpy(dst, src, sizeof(struct pisp_be_tiles_config));
 
-		return pisp_be_validate_config(pispbe, dst);
+		return pisp_be_validate_config(pispbe, mdev_context, dst);
 	}
 
 	return 0;
@@ -849,27 +1016,34 @@ static int pispbe_node_buffer_prepare(struct vb2_buffer *vb)
 
 static void pispbe_node_buffer_queue(struct vb2_buffer *buf)
 {
-	struct vb2_v4l2_buffer *vbuf =
-		container_of(buf, struct vb2_v4l2_buffer, vb2_buf);
-	struct pispbe_buffer *buffer =
-		container_of(vbuf, struct pispbe_buffer, vb);
-	struct pispbe_node *node = vb2_get_drv_priv(buf->vb2_queue);
+	struct vb2_v4l2_buffer *vbuf = container_of(buf, struct vb2_v4l2_buffer,
+						    vb2_buf);
+	struct pispbe_buffer *buffer = container_of(vbuf, struct pispbe_buffer,
+						    vb);
+	struct vb2_queue *q = buf->vb2_queue;
+	struct pispbe_node_context *context = pispbe_node_context_from_queue(q);
+	struct pispbe_media_context *mdev_context =
+					pispbe_media_context_from_dev(context);
+	struct pispbe_node *node = context->node;
 	struct pispbe_dev *pispbe = node->pispbe;
 
 	dev_dbg(pispbe->dev, "%s: for node %s\n", __func__, NODE_NAME(node));
-	list_add_tail(&buffer->ready_list, &node->ready_queue);
+	list_add_tail(&buffer->ready_list, &context->ready_queue);
 
 	/*
 	 * Every time we add a buffer, check if there's now some work for the hw
 	 * to do.
 	 */
-	if (!pispbe_prepare_job(pispbe))
-		pispbe_schedule(pispbe, false);
+	if (!pispbe_prepare_job(pispbe, context))
+		pispbe_schedule(pispbe, mdev_context, false);
 }
 
 static int pispbe_node_start_streaming(struct vb2_queue *q, unsigned int count)
 {
-	struct pispbe_node *node = vb2_get_drv_priv(q);
+	struct pispbe_node_context *context = pispbe_node_context_from_queue(q);
+	struct pispbe_media_context *mdev_context =
+					pispbe_media_context_from_dev(context);
+	struct pispbe_node *node = context->node;
 	struct pispbe_dev *pispbe = node->pispbe;
 	struct pispbe_buffer *buf, *tmp;
 	int ret;
@@ -878,28 +1052,28 @@ static int pispbe_node_start_streaming(struct vb2_queue *q, unsigned int count)
 	if (ret < 0)
 		goto err_return_buffers;
 
-	ret = video_device_pipeline_alloc_start(&node->vfd);
+	ret = video_device_context_pipeline_alloc_start(&context->vdev_context);
 	if (ret)
 		goto err_return_buffers;
 
 	scoped_guard(spinlock_irq, &pispbe->hw_lock) {
-		node->pispbe->streaming_map |=  BIT(node->id);
-		node->pispbe->sequence = 0;
+		mdev_context->streaming_map |=  BIT(node->id);
+		mdev_context->sequence = 0;
 	}
 
 	dev_dbg(pispbe->dev, "%s: for node %s (count %u)\n",
 		__func__, NODE_NAME(node), count);
 	dev_dbg(pispbe->dev, "Nodes streaming now 0x%x\n",
-		node->pispbe->streaming_map);
+		mdev_context->streaming_map);
 
 	/* Maybe we're ready to run. */
-	if (!pispbe_prepare_job(pispbe))
-		pispbe_schedule(pispbe, false);
+	if (!pispbe_prepare_job(pispbe, context))
+		pispbe_schedule(pispbe, mdev_context, false);
 
 	return 0;
 
 err_return_buffers:
-	list_for_each_entry_safe(buf, tmp, &node->ready_queue, ready_list) {
+	list_for_each_entry_safe(buf, tmp, &context->ready_queue, ready_list) {
 		list_del(&buf->ready_list);
 		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_QUEUED);
 	}
@@ -909,7 +1083,10 @@ static int pispbe_node_start_streaming(struct vb2_queue *q, unsigned int count)
 
 static void pispbe_node_stop_streaming(struct vb2_queue *q)
 {
-	struct pispbe_node *node = vb2_get_drv_priv(q);
+	struct pispbe_node_context *context = pispbe_node_context_from_queue(q);
+	struct pispbe_media_context *mdev_context =
+					pispbe_media_context_from_dev(context);
+	struct pispbe_node *node = context->node;
 	struct pispbe_dev *pispbe = node->pispbe;
 	struct pispbe_job_descriptor *job, *temp;
 	struct pispbe_buffer *buf;
@@ -926,7 +1103,7 @@ static void pispbe_node_stop_streaming(struct vb2_queue *q)
 	 */
 	dev_dbg(pispbe->dev, "%s: for node %s\n", __func__, NODE_NAME(node));
 	do {
-		buf = list_first_entry_or_null(&node->ready_queue,
+		buf = list_first_entry_or_null(&context->ready_queue,
 					       struct pispbe_buffer,
 					       ready_list);
 		if (buf) {
@@ -935,14 +1112,14 @@ static void pispbe_node_stop_streaming(struct vb2_queue *q)
 		}
 	} while (buf);
 
-	vb2_wait_for_all_buffers(&node->queue);
+	vb2_wait_for_all_buffers(&context->vdev_context.queue);
 
 	video_device_pipeline_stop(&node->vfd);
 
 	spin_lock_irq(&pispbe->hw_lock);
-	pispbe->streaming_map &= ~BIT(node->id);
+	mdev_context->streaming_map &= ~BIT(node->id);
 
-	if (pispbe->streaming_map == 0) {
+	if (mdev_context->streaming_map == 0) {
 		/*
 		 * If all nodes have stopped streaming release all jobs
 		 * without holding the lock.
@@ -960,7 +1137,7 @@ static void pispbe_node_stop_streaming(struct vb2_queue *q)
 	pm_runtime_put_autosuspend(pispbe->dev);
 
 	dev_dbg(pispbe->dev, "Nodes streaming now 0x%x\n",
-		pispbe->streaming_map);
+		mdev_context->streaming_map);
 }
 
 static const struct vb2_ops pispbe_node_queue_ops = {
@@ -1001,6 +1178,8 @@ static int pispbe_node_g_fmt_vid_cap(struct file *file, void *priv,
 {
 	struct pispbe_node *node = video_drvdata(file);
 	struct pispbe_dev *pispbe = node->pispbe;
+	struct pispbe_node_context *context =
+			pispbe_node_context_from_file(file, &node->vfd);
 
 	if (!NODE_IS_CAPTURE(node) || NODE_IS_META(node)) {
 		dev_dbg(pispbe->dev,
@@ -1009,7 +1188,7 @@ static int pispbe_node_g_fmt_vid_cap(struct file *file, void *priv,
 		return -EINVAL;
 	}
 
-	*f = node->format;
+	*f = context->format;
 	dev_dbg(pispbe->dev, "Get capture format for node %s\n",
 		NODE_NAME(node));
 
@@ -1021,6 +1200,8 @@ static int pispbe_node_g_fmt_vid_out(struct file *file, void *priv,
 {
 	struct pispbe_node *node = video_drvdata(file);
 	struct pispbe_dev *pispbe = node->pispbe;
+	struct pispbe_node_context *context =
+			pispbe_node_context_from_file(file, &node->vfd);
 
 	if (NODE_IS_CAPTURE(node) || NODE_IS_META(node)) {
 		dev_dbg(pispbe->dev,
@@ -1029,7 +1210,7 @@ static int pispbe_node_g_fmt_vid_out(struct file *file, void *priv,
 		return -EINVAL;
 	}
 
-	*f = node->format;
+	*f = context->format;
 	dev_dbg(pispbe->dev, "Get output format for node %s\n",
 		NODE_NAME(node));
 
@@ -1041,6 +1222,8 @@ static int pispbe_node_g_fmt_meta_out(struct file *file, void *priv,
 {
 	struct pispbe_node *node = video_drvdata(file);
 	struct pispbe_dev *pispbe = node->pispbe;
+	struct pispbe_node_context *context =
+			pispbe_node_context_from_file(file, &node->vfd);
 
 	if (!NODE_IS_META(node) || NODE_IS_CAPTURE(node)) {
 		dev_dbg(pispbe->dev,
@@ -1049,7 +1232,7 @@ static int pispbe_node_g_fmt_meta_out(struct file *file, void *priv,
 		return -EINVAL;
 	}
 
-	*f = node->format;
+	*f = context->format;
 	dev_dbg(pispbe->dev, "Get output format for meta node %s\n",
 		NODE_NAME(node));
 
@@ -1219,17 +1402,19 @@ static int pispbe_node_s_fmt_vid_cap(struct file *file, void *priv,
 {
 	struct pispbe_node *node = video_drvdata(file);
 	struct pispbe_dev *pispbe = node->pispbe;
+	struct pispbe_node_context *context =
+			pispbe_node_context_from_file(file, &node->vfd);
 	int ret;
 
 	ret = pispbe_node_try_fmt_vid_cap(file, priv, f);
 	if (ret < 0)
 		return ret;
 
-	if (vb2_is_busy(&node->queue))
+	if (vb2_is_busy(&context->vdev_context.queue))
 		return -EBUSY;
 
-	node->format = *f;
-	node->pisp_format = pispbe_find_fmt(f->fmt.pix_mp.pixelformat);
+	context->format = *f;
+	context->pisp_format = pispbe_find_fmt(f->fmt.pix_mp.pixelformat);
 
 	dev_dbg(pispbe->dev, "Set capture format for node %s to %p4cc\n",
 		NODE_NAME(node), &f->fmt.pix_mp.pixelformat);
@@ -1242,17 +1427,19 @@ static int pispbe_node_s_fmt_vid_out(struct file *file, void *priv,
 {
 	struct pispbe_node *node = video_drvdata(file);
 	struct pispbe_dev *pispbe = node->pispbe;
+	struct pispbe_node_context *context =
+			pispbe_node_context_from_file(file, &node->vfd);
 	int ret;
 
 	ret = pispbe_node_try_fmt_vid_out(file, priv, f);
 	if (ret < 0)
 		return ret;
 
-	if (vb2_is_busy(&node->queue))
+	if (vb2_is_busy(&context->vdev_context.queue))
 		return -EBUSY;
 
-	node->format = *f;
-	node->pisp_format = pispbe_find_fmt(f->fmt.pix_mp.pixelformat);
+	context->format = *f;
+	context->pisp_format = pispbe_find_fmt(f->fmt.pix_mp.pixelformat);
 
 	dev_dbg(pispbe->dev, "Set output format for node %s to %p4cc\n",
 		NODE_NAME(node), &f->fmt.pix_mp.pixelformat);
@@ -1265,17 +1452,19 @@ static int pispbe_node_s_fmt_meta_out(struct file *file, void *priv,
 {
 	struct pispbe_node *node = video_drvdata(file);
 	struct pispbe_dev *pispbe = node->pispbe;
+	struct pispbe_node_context *context =
+			pispbe_node_context_from_file(file, &node->vfd);
 	int ret;
 
 	ret = pispbe_node_try_fmt_meta_out(file, priv, f);
 	if (ret < 0)
 		return ret;
 
-	if (vb2_is_busy(&node->queue))
+	if (vb2_is_busy(&context->vdev_context.queue))
 		return -EBUSY;
 
-	node->format = *f;
-	node->pisp_format = &meta_out_supported_formats[0];
+	context->format = *f;
+	context->pisp_format = &meta_out_supported_formats[0];
 
 	dev_dbg(pispbe->dev, "Set output format for meta node %s to %p4cc\n",
 		NODE_NAME(node), &f->fmt.meta.dataformat);
@@ -1287,8 +1476,10 @@ static int pispbe_node_enum_fmt(struct file *file, void  *priv,
 				struct v4l2_fmtdesc *f)
 {
 	struct pispbe_node *node = video_drvdata(file);
+	struct pispbe_node_context *context =
+			pispbe_node_context_from_file(file, &node->vfd);
 
-	if (f->type != node->queue.type)
+	if (f->type != context->vdev_context.queue.type)
 		return -EINVAL;
 
 	if (NODE_IS_META(node)) {
@@ -1362,13 +1553,17 @@ static const struct v4l2_ioctl_ops pispbe_node_ioctl_ops = {
 	.vidioc_streamoff = vb2_ioctl_streamoff,
 };
 
-static int pispbe_link_validate(struct media_link *link)
+static int pispbe_link_validate(struct media_link *link,
+				struct media_device_context *mdev_context)
 {
+	struct pispbe_subdev_context *isp_context;
+	struct pispbe_node_context *node_context;
 	const struct v4l2_mbus_framefmt *sd_fmt;
 	struct v4l2_pix_format_mplane *pix_mp;
 	struct v4l2_subdev_state *state;
 	struct media_entity *vdev_ent;
 	struct media_entity *sd_ent;
+	struct video_device *vdev;
 	struct pispbe_node *node;
 	struct v4l2_subdev *sd;
 
@@ -1380,8 +1575,12 @@ static int pispbe_link_validate(struct media_link *link)
 		vdev_ent = link->sink->entity;
 	}
 
-	node = container_of(media_entity_to_video_device(vdev_ent),
-			    struct pispbe_node, vfd);
+	vdev = media_entity_to_video_device(vdev_ent);
+	node_context = pispbe_node_context(video_device_context_get(mdev_context,
+								    vdev));
+
+	node = node_context->node;
+
 	switch (node->id) {
 	case TDN_INPUT_NODE:
 		fallthrough;
@@ -1395,10 +1594,13 @@ static int pispbe_link_validate(struct media_link *link)
 		/* Skip validation for these nodes. */
 		return 0;
 	}
-	pix_mp = &node->format.fmt.pix_mp;
+	pix_mp = &node_context->format.fmt.pix_mp;
 
 	sd = media_entity_to_v4l2_subdev(sd_ent);
-	state = v4l2_subdev_get_unlocked_active_state(sd);
+	isp_context = pispbe_subdev_context(v4l2_subdev_context_get(mdev_context,
+								    sd));
+
+	state = v4l2_subdev_get_unlocked_active_state(&isp_context->sd_context);
 	sd_fmt = v4l2_subdev_state_get_format(state, node->id);
 
 	/* Only check for sizes. */
@@ -1419,28 +1621,16 @@ static int pispbe_link_validate(struct media_link *link)
 	return 0;
 }
 
-static const struct media_entity_operations pispbe_node_entity_ops = {
-	.link_validate = pispbe_link_validate,
-};
-
-static const struct video_device pispbe_videodev = {
-	.name = PISPBE_NAME,
-	.vfl_dir = VFL_DIR_M2M, /* gets overwritten */
-	.fops = &pispbe_fops,
-	.ioctl_ops = &pispbe_node_ioctl_ops,
-	.minor = -1,
-	.release = video_device_release_empty,
-};
-
-static void pispbe_node_def_fmt(struct pispbe_node *node)
+static void pispbe_node_def_fmt(struct pispbe_node *node,
+				struct pispbe_node_context *context)
 {
+	struct v4l2_format *format = &context->format;
+
 	if (NODE_IS_META(node) && NODE_IS_OUTPUT(node)) {
 		/* Config node */
-		struct v4l2_format *f = &node->format;
-
-		f->fmt.meta.dataformat = V4L2_META_FMT_RPI_BE_CFG;
-		f->fmt.meta.buffersize = sizeof(struct pisp_be_tiles_config);
-		f->type = node->buf_type;
+		format->fmt.meta.dataformat = V4L2_META_FMT_RPI_BE_CFG;
+		format->fmt.meta.buffersize = sizeof(struct pisp_be_tiles_config);
+		format->type = node->buf_type;
 	} else {
 		struct v4l2_format f = {
 			.fmt.pix_mp.pixelformat = V4L2_PIX_FMT_YUV420,
@@ -1449,36 +1639,39 @@ static void pispbe_node_def_fmt(struct pispbe_node *node)
 			.type = node->buf_type,
 		};
 		pispbe_try_format(&f, node);
-		node->format = f;
+		*format = f;
 	}
 
-	node->pisp_format = pispbe_find_fmt(node->format.fmt.pix_mp.pixelformat);
+	context->pisp_format = pispbe_find_fmt(format->fmt.pix_mp.pixelformat);
 }
 
-/*
- * Initialise a struct pispbe_node and register it as /dev/video<N>
- * to represent one of the PiSP Back End's input or output streams.
- */
-static int pispbe_init_node(struct pispbe_dev *pispbe, unsigned int id)
+static int pispbe_alloc_node_context(struct media_entity *entity,
+				     struct media_entity_context **ctx)
 {
-	bool output = NODE_DESC_IS_OUTPUT(&node_desc[id]);
-	struct pispbe_node *node = &pispbe->node[id];
-	struct media_entity *entity = &node->vfd.entity;
-	struct video_device *vdev = &node->vfd;
-	struct vb2_queue *q = &node->queue;
+	struct video_device *vdev = container_of(entity, struct video_device,
+						 entity);
+	struct pispbe_node *node = pispbe_node_from_vdev(vdev);
+	struct pispbe_dev *pispbe = node->pispbe;
+	struct pispbe_node_context *context;
+	struct vb2_queue *q;
 	int ret;
 
-	node->id = id;
-	node->pispbe = pispbe;
-	node->buf_type = node_desc[id].buf_type;
+	*ctx = kzalloc(sizeof(*context), GFP_KERNEL);
+	if (!*ctx)
+		return -ENOMEM;
+	context = (struct pispbe_node_context *)*ctx;
 
-	mutex_init(&node->node_lock);
-	mutex_init(&node->queue_lock);
-	INIT_LIST_HEAD(&node->ready_queue);
+	video_device_init_context(vdev, &context->vdev_context);
+
+	context->pispbe = pispbe;
+	context->node = node;
 
-	node->format.type = node->buf_type;
-	pispbe_node_def_fmt(node);
+	INIT_LIST_HEAD(&context->ready_queue);
 
+	context->format.type = node->buf_type;
+	pispbe_node_def_fmt(node, context);
+
+	q = &context->vdev_context.queue;
 	q->type = node->buf_type;
 	q->io_modes = VB2_MMAP | VB2_DMABUF;
 	q->mem_ops = &vb2_dma_contig_memops;
@@ -1488,21 +1681,70 @@ static int pispbe_init_node(struct pispbe_dev *pispbe, unsigned int id)
 	q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
 	q->dev = pispbe->dev;
 	/* get V4L2 to handle node->queue locking */
-	q->lock = &node->queue_lock;
+	q->lock = &context->vdev_context.queue_lock;
 
 	ret = vb2_queue_init(q);
 	if (ret < 0) {
 		dev_err(pispbe->dev, "vb2_queue_init failed\n");
-		goto err_mutex_destroy;
+		goto err_cleanup;
 	}
 
+	return 0;
+
+err_cleanup:
+	kfree(*ctx);
+	return ret;
+}
+
+static void pispbe_free_node_context(struct media_entity_context *ctx)
+{
+	struct pispbe_node_context *context =
+					pispbe_node_context_from_entity(ctx);
+
+	list_del_init(&context->ready_queue);
+	video_device_cleanup_context(&context->vdev_context);
+	kfree(context);
+}
+
+static const struct media_entity_operations pispbe_node_entity_ops = {
+	.alloc_context = pispbe_alloc_node_context,
+	.destroy_context = pispbe_free_node_context,
+	.link_validate_context = pispbe_link_validate,
+};
+
+static const struct video_device pispbe_videodev = {
+	.name = PISPBE_NAME,
+	.vfl_dir = VFL_DIR_M2M, /* gets overwritten */
+	.fops = &pispbe_fops,
+	.ioctl_ops = &pispbe_node_ioctl_ops,
+	.minor = -1,
+	.release = video_device_release_empty,
+};
+
+/*
+ * Initialise a struct pispbe_node and register it as /dev/video<N>
+ * to represent one of the PiSP Back End's input or output streams.
+ */
+static int pispbe_init_node(struct pispbe_dev *pispbe, unsigned int id)
+{
+	bool output = NODE_DESC_IS_OUTPUT(&node_desc[id]);
+	struct pispbe_node *node = &pispbe->node[id];
+	struct media_entity *entity = &node->vfd.entity;
+	struct video_device *vdev = &node->vfd;
+	int ret;
+
+	node->id = id;
+	node->pispbe = pispbe;
+	node->buf_type = node_desc[id].buf_type;
+
+	mutex_init(&node->node_lock);
+
 	*vdev = pispbe_videodev; /* default initialization */
 	strscpy(vdev->name, node_desc[id].ent_name, sizeof(vdev->name));
 	vdev->v4l2_dev = &pispbe->v4l2_dev;
 	vdev->vfl_dir = output ? VFL_DIR_TX : VFL_DIR_RX;
 	/* get V4L2 to serialise our ioctls */
 	vdev->lock = &node->node_lock;
-	vdev->queue = &node->queue;
 	vdev->device_caps = V4L2_CAP_STREAMING | node_desc[id].caps;
 
 	node->pad.flags = output ? MEDIA_PAD_FL_SOURCE : MEDIA_PAD_FL_SINK;
@@ -1512,15 +1754,16 @@ static int pispbe_init_node(struct pispbe_dev *pispbe, unsigned int id)
 		dev_err(pispbe->dev,
 			"Failed to register media pads for %s device node\n",
 			NODE_NAME(node));
-		goto err_unregister_queue;
+		goto err_mutex_destroy;
 	}
 
 	ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
+
 	if (ret) {
 		dev_err(pispbe->dev,
 			"Failed to register video %s device node\n",
 			NODE_NAME(node));
-		goto err_unregister_queue;
+		goto err_mutex_destroy;
 	}
 	video_set_drvdata(vdev, node);
 
@@ -1542,14 +1785,15 @@ static int pispbe_init_node(struct pispbe_dev *pispbe, unsigned int id)
 
 err_unregister_video_dev:
 	video_unregister_device(&node->vfd);
-err_unregister_queue:
-	vb2_queue_release(&node->queue);
 err_mutex_destroy:
 	mutex_destroy(&node->node_lock);
-	mutex_destroy(&node->queue_lock);
 	return ret;
 }
 
+/* ----------------------------------------------------------------------------
+ * ISP subdevice operations
+ */
+
 static int pispbe_subdev_set_pad_fmt(struct v4l2_subdev *sd,
 				     struct v4l2_subdev_state *state,
 				     struct v4l2_subdev_format *format)
@@ -1623,8 +1867,83 @@ static const struct v4l2_subdev_internal_ops pispbe_subdev_internal_ops = {
 	.init_state = pispbe_init_state,
 };
 
+static int pispbe_alloc_subdev_context(struct media_entity *entity,
+				       struct media_entity_context **ctx)
+{
+	struct v4l2_subdev *sd = container_of(entity, struct v4l2_subdev,
+					      entity);
+	struct pispbe_subdev_context *context;
+
+	*ctx = kzalloc(sizeof(*context), GFP_KERNEL);
+	if (!*ctx)
+		return -ENOMEM;
+	context = (struct pispbe_subdev_context *)*ctx;
+
+	v4l2_subdev_init_context(sd, &context->sd_context);
+
+	return 0;
+}
+
+static void pispbe_free_subdev_context(struct media_entity_context *ctx)
+{
+	struct pispbe_subdev_context *context =
+					pispbe_subdev_context_from_entity(ctx);
+
+	v4l2_subdev_cleanup_context(&context->sd_context);
+	kfree(context);
+}
+
 static const struct media_entity_operations pispbe_subdev_entity_ops = {
-	.link_validate = v4l2_subdev_link_validate,
+	.alloc_context = pispbe_alloc_subdev_context,
+	.destroy_context = pispbe_free_subdev_context,
+	.link_validate_context = v4l2_subdev_link_validate_context,
+};
+
+static int pispbe_media_alloc_context(struct media_device *mdev,
+				      struct media_device_context **ctx)
+{
+	struct pispbe_media_context *context;
+
+	*ctx = kzalloc(sizeof(*context), GFP_KERNEL);
+	if (!*ctx)
+		return -ENOMEM;
+	context = (struct pispbe_media_context *)*ctx;
+
+	media_device_init_context(mdev, &context->mdev_context);
+
+	context->streaming_map = 0;
+
+	context->config =
+		dma_alloc_coherent(mdev->dev,
+				   sizeof(struct pisp_be_tiles_config) *
+				   PISP_BE_NUM_CONFIG_BUFFERS,
+				   &context->config_dma_addr, GFP_KERNEL);
+	if (!context->config) {
+		dev_err(mdev->dev,
+			"Unable to allocate cached config buffers.\n");
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+static void pispbe_media_destroy_context(struct media_device_context *ctx)
+{
+	struct pispbe_media_context *context =
+					(struct pispbe_media_context *)ctx;
+
+	if (context->config)
+		dma_free_coherent(ctx->mdev->dev,
+				  sizeof(struct pisp_be_tiles_config) *
+				  PISP_BE_NUM_CONFIG_BUFFERS,
+				  context->config, context->config_dma_addr);
+
+	media_device_cleanup_context(&context->mdev_context);
+	kfree(context);
+}
+
+static const struct media_device_ops pispbe_media_device_ops = {
+	.alloc_context = pispbe_media_alloc_context,
+	.destroy_context = pispbe_media_destroy_context,
 };
 
 static int pispbe_init_subdev(struct pispbe_dev *pispbe)
@@ -1677,6 +1996,7 @@ static int pispbe_init_devices(struct pispbe_dev *pispbe)
 	mdev = &pispbe->mdev;
 	mdev->hw_revision = pispbe->hw_version;
 	mdev->dev = pispbe->dev;
+	mdev->ops = &pispbe_media_device_ops;
 	strscpy(mdev->model, PISPBE_NAME, sizeof(mdev->model));
 	media_device_init(mdev);
 
@@ -1704,26 +2024,11 @@ static int pispbe_init_devices(struct pispbe_dev *pispbe)
 	if (ret)
 		goto err_unregister_nodes;
 
-	pispbe->config =
-		dma_alloc_coherent(pispbe->dev,
-				   sizeof(struct pisp_be_tiles_config) *
-					PISP_BE_NUM_CONFIG_BUFFERS,
-				   &pispbe->config_dma_addr, GFP_KERNEL);
-	if (!pispbe->config) {
-		dev_err(pispbe->dev, "Unable to allocate cached config buffers.\n");
-		ret = -ENOMEM;
-		goto err_unregister_mdev;
-	}
-
 	return 0;
 
-err_unregister_mdev:
-	media_device_unregister(mdev);
 err_unregister_nodes:
-	while (num_regist-- > 0) {
+	while (num_regist-- > 0)
 		video_unregister_device(&pispbe->node[num_regist].vfd);
-		vb2_queue_release(&pispbe->node[num_regist].queue);
-	}
 	v4l2_device_unregister_subdev(&pispbe->sd);
 	media_entity_cleanup(&pispbe->sd.entity);
 err_unregister_v4l2:
@@ -1735,14 +2040,6 @@ static int pispbe_init_devices(struct pispbe_dev *pispbe)
 
 static void pispbe_destroy_devices(struct pispbe_dev *pispbe)
 {
-	if (pispbe->config) {
-		dma_free_coherent(pispbe->dev,
-				  sizeof(struct pisp_be_tiles_config) *
-					PISP_BE_NUM_CONFIG_BUFFERS,
-				  pispbe->config,
-				  pispbe->config_dma_addr);
-	}
-
 	dev_dbg(pispbe->dev, "Unregister from media controller\n");
 
 	v4l2_device_unregister_subdev(&pispbe->sd);
@@ -1751,9 +2048,7 @@ static void pispbe_destroy_devices(struct pispbe_dev *pispbe)
 
 	for (int i = PISPBE_NUM_NODES - 1; i >= 0; i--) {
 		video_unregister_device(&pispbe->node[i].vfd);
-		vb2_queue_release(&pispbe->node[i].queue);
 		mutex_destroy(&pispbe->node[i].node_lock);
-		mutex_destroy(&pispbe->node[i].queue_lock);
 	}
 
 	media_device_cleanup(&pispbe->mdev);

-- 
2.49.0

Re: [PATCH 09/26] media: v4l2-dev: Add video_device_context_from_file()

[kernel test robot]

Hi Jacopo,

kernel test robot noticed the following build warnings:

[auto build test WARNING on d968e50b5c26642754492dea23cbd3592bde62d8]

url:    https://github.com/intel-lab-lkp/linux/commits/Jacopo-Mondi/media-mc-Add-per-file-handle-data-support/20250717-190546
base:   d968e50b5c26642754492dea23cbd3592bde62d8
patch link:    https://lore.kernel.org/r/20250717-multicontext-mainline-2025-v1-9-81ac18979c03%40ideasonboard.com
patch subject: [PATCH 09/26] media: v4l2-dev: Add video_device_context_from_file()
config: x86_64-buildonly-randconfig-004-20250718 (https://download.01.org/0day-ci/archive/20250718/202507181702.W5iinHRF-lkp@intel.com/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250718/202507181702.W5iinHRF-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202507181702.W5iinHRF-lkp@intel.com/

All warnings (new ones prefixed by >>):

>> drivers/media/v4l2-core/v4l2-dev.c:1184:1: warning: no previous prototype for function 'video_device_context_from_file' [-Wmissing-prototypes]
    1184 | video_device_context_from_file(struct file *filp, struct video_device *vfd)
         | ^
   drivers/media/v4l2-core/v4l2-dev.c:1183:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
    1183 | struct video_device_context *
         | ^
         | static 
   1 warning generated.


vim +/video_device_context_from_file +1184 drivers/media/v4l2-core/v4l2-dev.c

  1182	
  1183	struct video_device_context *
> 1184	video_device_context_from_file(struct file *filp, struct video_device *vfd)
  1185	{
  1186		struct v4l2_fh *vfh =
  1187			test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags) ? filp->private_data
  1188								    : NULL;
  1189	
  1190		/* If the file handle has been bound to a context return it. */
  1191		if (vfh && vfh->context)
  1192			return vfh->context;
  1193	
  1194		return vfd->default_context;
  1195	}
  1196	EXPORT_SYMBOL_GPL(video_device_context_from_file);
  1197	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH 11/26] media: videobuf2-v4l2: Support vb2_queue embedded in a context

[kernel test robot]

Hi Jacopo,

kernel test robot noticed the following build errors:

[auto build test ERROR on d968e50b5c26642754492dea23cbd3592bde62d8]

url:    https://github.com/intel-lab-lkp/linux/commits/Jacopo-Mondi/media-mc-Add-per-file-handle-data-support/20250717-190546
base:   d968e50b5c26642754492dea23cbd3592bde62d8
patch link:    https://lore.kernel.org/r/20250717-multicontext-mainline-2025-v1-11-81ac18979c03%40ideasonboard.com
patch subject: [PATCH 11/26] media: videobuf2-v4l2: Support vb2_queue embedded in a context
config: x86_64-buildonly-randconfig-004-20250718 (https://download.01.org/0day-ci/archive/20250718/202507181803.MKWlcVqu-lkp@intel.com/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250718/202507181803.MKWlcVqu-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202507181803.MKWlcVqu-lkp@intel.com/

All errors (new ones prefixed by >>):

>> drivers/media/common/videobuf2/videobuf2-v4l2.c:1012:5: error: call to undeclared function 'video_device_context_from_file'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    1012 |                                 video_device_context_from_file(file, vdev);
         |                                 ^
>> drivers/media/common/videobuf2/videobuf2-v4l2.c:1011:31: error: incompatible integer to pointer conversion initializing 'struct video_device_context *' with an expression of type 'int' [-Wint-conversion]
    1011 |         struct video_device_context *ctx =
         |                                      ^
    1012 |                                 video_device_context_from_file(file, vdev);
         |                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> drivers/media/common/videobuf2/videobuf2-v4l2.c:1014:19: error: incomplete definition of type 'struct video_device_context'
    1014 |         return ctx ? &ctx->queue
         |                       ~~~^
   include/media/v4l2-dev.h:288:9: note: forward declaration of 'struct video_device_context'
     288 |         struct video_device_context *default_context;
         |                ^
   drivers/media/common/videobuf2/videobuf2-v4l2.c:1015:54: error: incomplete definition of type 'struct video_device_context'
    1015 |                    : vdev->default_context ? &vdev->default_context->queue
         |                                               ~~~~~~~~~~~~~~~~~~~~~^
   include/media/v4l2-dev.h:288:9: note: forward declaration of 'struct video_device_context'
     288 |         struct video_device_context *default_context;
         |                ^
   4 errors generated.


vim +/video_device_context_from_file +1012 drivers/media/common/videobuf2/videobuf2-v4l2.c

   992	
   993	/*
   994	 * The following functions are not part of the vb2 core API, but are helper
   995	 * functions that plug into struct v4l2_ioctl_ops, struct v4l2_file_operations
   996	 * and struct vb2_ops.
   997	 * They contain boilerplate code that most if not all drivers have to do
   998	 * and so they simplify the driver code.
   999	 */
  1000	
  1001	/*
  1002	 * Helper to get the vb2 queue either from:
  1003	 * 1) The video context bound to the open file handle
  1004	 * 2) The default context for context-aware drivers if userspace has not bound
  1005	 *    a context to the file handle
  1006	 * 3) From the video device for non-context aware drivers
  1007	 */
  1008	static struct vb2_queue *get_vb2_queue(struct file *file,
  1009					       struct video_device *vdev)
  1010	{
> 1011		struct video_device_context *ctx =
> 1012					video_device_context_from_file(file, vdev);
  1013	
> 1014		return ctx ? &ctx->queue
  1015			   : vdev->default_context ? &vdev->default_context->queue
  1016			   : vdev->queue;
  1017	}
  1018	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki