[PATCH 0/3] Add media jobs framework

[PATCH 0/3] Add media jobs framework

[Daniel Scally]

Hello all

This series adds a new API to the media controller framework, which
I'm calling "media jobs". The framework is intended to facilitate
communication between separate drivers which need to work together
to fully operate a media pipeline. For example, the need for the
framework arose when writing support for the ISP in the RZ/V2H; the
ISP is fed by a DMA engine which is not part of the same IP block,
and so is driven by its own driver (though sharing a media graph).
The ISP driver needs to be able to communicate with the DMA engine
driver to instruct it to push the next frame. Because the DMA engine
might be different on a different platform that used the ISP, direct
calls into functions exported by the DMA engine driver wouldn't be
scalable, and so this driver agnostic route was adopted. The
framework allows drivers to define the steps that need to be taken
(for example writing configuration data, reading statistics data,
writing buffer addresses and triggering data transmission) to complete
a "job" (of which the only current example is the processing of a
frame of data through the pipeline, though I expect that other use
cases could become apparent too) and to then schedule them into a
work queue once driver definable dependencies have been met. The
dependencies might, for example, be the queuing of buffers to V4L2
capture / output devices.

The framework allows precise definition of the ordering of the steps
regardless of the order in which the drivers populate the jobs. Steps
can be flagged as being placed at a particular position from the front
or back of the queue (I.E. last, or third from last) or as requiring
no particular order. This would allow the construction of a job like:

Step 0 (ISP Driver): Program the hardware with parameters
Step 1 to N-1 (Both drivers): Program the hardware with buffers
Step N (DMA Engine Driver): Send a frame of data to the ISP
... ISP processes data ...
Step N + 1 (ISP Driver): Read out statistics data from the last frame

The mali-c55 ISP driver and the DMA engine feeding it on the RZ/V2H
(called the rzv2h-ivc driver) both use the framework, and will be
posted shortly after this series with references back to it. I will
reply to this message with links to those series for convenience.

The first patch in this set is not strictly part of the framework,
but also facilitates multiple drivers with V4L2 Video Devices
sharing a single media graph. We have a requirement to delay the
start of streaming until all of a pipeline's video devices have had
their .start_streaming() operations called; these new entity ops
provide a mechanism through which each driver can inform the other
that the last video device in the pipeline has now been started.

Thanks
Dan

Daniel Scally (3):
  media: mc: entity: Add pipeline_started/stopped ops
  media: mc: Add media jobs framework
  media: Documentation: Add documentation for media jobs

 Documentation/driver-api/media/mc-core.rst | 154 +++++++
 drivers/media/mc/Makefile                  |   2 +-
 drivers/media/mc/mc-entity.c               |  45 +++
 drivers/media/mc/mc-jobs.c                 | 446 +++++++++++++++++++++
 include/media/media-entity.h               |  24 ++
 include/media/media-jobs.h                 | 354 ++++++++++++++++
 6 files changed, 1024 insertions(+), 1 deletion(-)
 create mode 100644 drivers/media/mc/mc-jobs.c
 create mode 100644 include/media/media-jobs.h

-- 
2.34.1

[PATCH 1/3] media: mc: entity: Add pipeline_started/stopped ops

[Daniel Scally]

Add two new members to struct media_entity_operations, along with new
functions in media-entity.c to traverse a media pipeline and call the
new operations. The new functions are intended to be used to signal
to a media pipeline that it has fully started, with the entity ops
allowing drivers to define some action to be taken when those
conditions are met.

The combination of the new functions and operations allows drivers
which are part of a multi-driver pipeline to delay actually starting
streaming until all of the conditions for streaming succcessfully are
met across all drivers.

Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 45 ++++++++++++++++++++++++++++++++++++
 include/media/media-entity.h | 24 +++++++++++++++++++
 2 files changed, 69 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 045590905582..e36b1710669d 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -1053,6 +1053,51 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_entity_iter_next);
 
+int media_pipeline_started(struct media_pipeline *pipe)
+{
+	struct media_pipeline_entity_iter iter;
+	struct media_entity *entity;
+	int ret;
+
+	ret = media_pipeline_entity_iter_init(pipe, &iter);
+	if (ret)
+		return ret;
+
+	media_pipeline_for_each_entity(pipe, &iter, entity) {
+		ret = media_entity_call(entity, pipeline_started);
+		if (ret && ret != -ENOIOCTLCMD)
+			goto err_notify_stopped;
+	}
+
+	media_pipeline_entity_iter_cleanup(&iter);
+
+	return ret == -ENOIOCTLCMD ? 0 : ret;
+
+err_notify_stopped:
+	media_pipeline_stopped(pipe);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(media_pipeline_started);
+
+int media_pipeline_stopped(struct media_pipeline *pipe)
+{
+	struct media_pipeline_entity_iter iter;
+	struct media_entity *entity;
+	int ret;
+
+	ret = media_pipeline_entity_iter_init(pipe, &iter);
+	if (ret)
+		return ret;
+
+	media_pipeline_for_each_entity(pipe, &iter, entity)
+		media_entity_call(entity, pipeline_stopped);
+
+	media_pipeline_entity_iter_cleanup(&iter);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(media_pipeline_stopped);
+
 /* -----------------------------------------------------------------------------
  * Links management
  */
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 64cf590b1134..e858326b95cb 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -269,6 +269,10 @@ struct media_pad {
  *			media_entity_has_pad_interdep().
  *			Optional: If the operation isn't implemented all pads
  *			will be considered as interdependent.
+ * @pipeline_started:	Notify this entity that the pipeline it is a part of has
+ *			been started
+ * @pipeline_stopped:	Notify this entity that the pipeline it is a part of has
+ *			been stopped
  *
  * .. note::
  *
@@ -284,6 +288,8 @@ struct media_entity_operations {
 	int (*link_validate)(struct media_link *link);
 	bool (*has_pad_interdep)(struct media_entity *entity, unsigned int pad0,
 				 unsigned int pad1);
+	int (*pipeline_started)(struct media_entity *entity);
+	void (*pipeline_stopped)(struct media_entity *entity);
 };
 
 /**
@@ -1261,6 +1267,24 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
 	     entity != NULL;							\
 	     entity = __media_pipeline_entity_iter_next((pipe), iter, entity))
 
+/**
+ * media_pipeline_started - Inform entities in a pipeline that it has started
+ * @pipe:	The pipeline
+ *
+ * Iterate on all entities in a media pipeline and call their pipeline_started
+ * member of media_entity_operations.
+ */
+int media_pipeline_started(struct media_pipeline *pipe);
+
+/**
+ * media_pipeline_stopped - Inform entities in a pipeline that it has stopped
+ * @pipe:	The pipeline
+ *
+ * Iterate on all entities in a media pipeline and call their pipeline_stopped
+ * member of media_entity_operations.
+ */
+int media_pipeline_stopped(struct media_pipeline *pipe);
+
 /**
  * media_pipeline_alloc_start - Mark a pipeline as streaming
  * @pad: Starting pad
-- 
2.34.1

[PATCH 2/3] media: mc: Add media jobs framework

[Daniel Scally]

Add a new framework to the media subsystem describing media jobs.
This framework is intended to be able to model the interactions
between multiple different drivers that need to be run in concert
to fully control a media pipeline, for example an ISP driver and a
driver controlling a DMA device that feeds data from memory in to
that ISP.

The new framework allows all drivers involved to add explicit steps
that need to be performed, and to control the ordering of those steps
precisely. Once the job with its steps has been created it's then
scheduled to be run with a workqueue which executes each step in the
defined order.

Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
 drivers/media/mc/Makefile  |   2 +-
 drivers/media/mc/mc-jobs.c | 446 +++++++++++++++++++++++++++++++++++++
 include/media/media-jobs.h | 354 +++++++++++++++++++++++++++++
 3 files changed, 801 insertions(+), 1 deletion(-)
 create mode 100644 drivers/media/mc/mc-jobs.c
 create mode 100644 include/media/media-jobs.h

diff --git a/drivers/media/mc/Makefile b/drivers/media/mc/Makefile
index 2b7af42ba59c..9148bbfd1578 100644
--- a/drivers/media/mc/Makefile
+++ b/drivers/media/mc/Makefile
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 
 mc-objs	:= mc-device.o mc-devnode.o mc-entity.o \
-	   mc-request.o
+	   mc-jobs.o mc-request.o
 
 ifneq ($(CONFIG_USB),)
 	mc-objs += mc-dev-allocator.o
diff --git a/drivers/media/mc/mc-jobs.c b/drivers/media/mc/mc-jobs.c
new file mode 100644
index 000000000000..1f04cdf63d27
--- /dev/null
+++ b/drivers/media/mc/mc-jobs.c
@@ -0,0 +1,446 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Media jobs framework
+ *
+ * Copyright 2025 Ideas on Board Oy
+ *
+ * Author: Daniel Scally <dan.scally@ideasonboard.com>
+ */
+
+#include <linux/cleanup.h>
+#include <linux/kref.h>
+#include <linux/list.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+
+#include <media/media-device.h>
+#include <media/media-entity.h>
+#include <media/media-jobs.h>
+
+int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
+			    void *data, unsigned int flags, unsigned int pos)
+{
+	struct media_job_step *step, *tmp;
+	unsigned int num = flags;
+	unsigned int count = 0;
+
+	guard(spinlock)(&job->lock);
+
+	if (!flags) {
+		WARN_ONCE(1, "%s(): No flag bits set\n", __func__);
+		return -EINVAL;
+	}
+
+	/* Count the number of set flags; they're mutually exclusive. */
+	while (num) {
+		num &= (num - 1);
+		count++;
+	}
+
+	if (count > 1) {
+		WARN_ONCE(1, "%s(): Multiple flag bits set\n", __func__);
+		return -EINVAL;
+	}
+
+	step = kzalloc(sizeof(*step), GFP_KERNEL);
+	if (!step)
+		return -ENOMEM;
+
+	step->run_step = run_step;
+	step->data = data;
+	step->flags = flags;
+	step->pos = pos;
+
+	/*
+	 * We need to decide where to place the step. If the list is empty that
+	 * is really easy (and also the later code is much easier if the code is
+	 * guaranteed not to be empty...)
+	 */
+	if (list_empty(&job->steps)) {
+		list_add_tail(&step->list, &job->steps);
+		return 0;
+	}
+
+	/*
+	 * If we've been asked to place it at a specific position from the end
+	 * of the list, we cycle back through it until either we exhaust the
+	 * list or find an entry that needs to go further from the back than the
+	 * new one.
+	 */
+	if ((flags & MEDIA_JOBS_FL_STEP_FROM_BACK)) {
+		list_for_each_entry_reverse(tmp, &job->steps, list) {
+			if (tmp->flags == flags && tmp->pos == pos)
+				return -EINVAL;
+
+			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_BACK ||
+			    tmp->pos > pos)
+				break;
+		}
+
+		/*
+		 * If the entry we broke on is also one placed from the back and
+		 * should be closer to the back than the new one, we place the
+		 * new one in front of it...otherwise place the new one behind
+		 * it.
+		 */
+		if (tmp->flags == flags && tmp->pos < pos)
+			list_add_tail(&step->list, &tmp->list);
+		else
+			list_add(&step->list, &tmp->list);
+
+		return 0;
+	}
+
+	/*
+	 * If we've been asked to place it a specific position from the front of
+	 * the list we do the same kind of operation, but going from the front
+	 * instead.
+	 */
+	if (flags & MEDIA_JOBS_FL_STEP_FROM_FRONT) {
+		list_for_each_entry(tmp, &job->steps, list) {
+			if (tmp->flags == flags && tmp->pos == pos)
+				return -EINVAL;
+
+			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_FRONT ||
+			    tmp->pos > pos)
+				break;
+		}
+
+		/*
+		 * If the entry we broke on is also placed from the front and
+		 * should be closed to the front than the new one, we place the
+		 * new one behind it, otherwise in front of it.
+		 */
+		if (tmp->flags == flags && tmp->pos < pos)
+			list_add(&step->list, &tmp->list);
+		else
+			list_add_tail(&step->list, &tmp->list);
+
+		return 0;
+	}
+
+	/*
+	 * If the step is flagged as "can go anywhere" we just need to try to
+	 * find the first "from the back" entry and add it immediately before
+	 * that. If we can't find one, add it after whatever we did find.
+	 */
+	if (flags & MEDIA_JOBS_FL_STEP_ANYWHERE) {
+		list_for_each_entry(tmp, &job->steps, list)
+			if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK))
+				break;
+
+		if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK) ||
+		    list_entry_is_head(tmp, &job->steps, list))
+			list_add_tail(&step->list, &tmp->list);
+		else
+			list_add(&step->list, &tmp->list);
+
+		return 0;
+	}
+
+	/* Shouldn't get here, unless the flag value is wrong. */
+	WARN_ONCE(1, "%s(): Invalid flag value\n", __func__);
+	return -EINVAL;
+}
+EXPORT_SYMBOL_GPL(media_jobs_add_job_step);
+
+int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
+			   void *data)
+{
+	struct media_job_dep *dep;
+
+	if (!ops || !ops->check_dep || !data)
+		return -EINVAL;
+
+	guard(spinlock)(&job->lock);
+
+	/* Confirm the same dependency hasn't already been added */
+	list_for_each_entry(dep, &job->deps, list)
+		if (dep->ops == ops && dep->data == data)
+			return -EINVAL;
+
+	dep = kzalloc(sizeof(*dep), GFP_KERNEL);
+	if (!dep)
+		return -ENOMEM;
+
+	dep->ops = ops;
+	dep->data = data;
+	list_add(&dep->list, &job->deps);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(media_jobs_add_job_dep);
+
+static bool media_jobs_check_pending_job(struct media_job *job,
+					 enum media_job_types type,
+					 struct media_job_dep_ops *dep_ops,
+					 void *data)
+{
+	struct media_job_dep *dep;
+
+	guard(spinlock)(&job->lock);
+
+	if (job->type != type)
+		return false;
+
+	list_for_each_entry(dep, &job->deps, list) {
+		if (dep->ops == dep_ops && dep->data == data) {
+			if (dep->met)
+				return false;
+
+			break;
+		}
+	}
+
+	dep->met = true;
+	return true;
+}
+
+static struct media_job *media_jobs_get_job(struct media_job_scheduler *sched,
+					    enum media_job_types type,
+					    struct media_job_dep_ops *dep_ops,
+					    void *dep_data)
+{
+	struct media_job_setup_func *jsf;
+	struct media_job *job;
+	int ret;
+
+	list_for_each_entry(job, &sched->pending, list)
+		if (media_jobs_check_pending_job(job, type, dep_ops, dep_data))
+			return job;
+
+	job = kzalloc(sizeof(*job), GFP_KERNEL);
+	if (!job)
+		return ERR_PTR(-ENOMEM);
+
+	spin_lock_init(&job->lock);
+	INIT_LIST_HEAD(&job->deps);
+	INIT_LIST_HEAD(&job->steps);
+	job->type = type;
+	job->sched = sched;
+
+	list_for_each_entry(jsf, &sched->setup_funcs, list) {
+		if (jsf->type != type)
+			continue;
+
+		ret = jsf->job_setup(job, jsf->data);
+		if (ret) {
+			kfree(job);
+			return ERR_PTR(ret);
+		}
+	}
+
+	list_add_tail(&job->list, &sched->pending);
+
+	/* This marks the dependency as met */
+	media_jobs_check_pending_job(job, type, dep_ops, dep_data);
+
+	return job;
+}
+
+static void media_jobs_free_job(struct media_job *job, bool reset)
+{
+	struct media_job_step *step, *stmp;
+	struct media_job_dep *dep, *dtmp;
+
+	scoped_guard(spinlock, &job->lock) {
+		list_for_each_entry_safe(dep, dtmp, &job->deps, list) {
+			if (reset && dep->ops->reset_dep)
+				dep->ops->reset_dep(dep->data);
+
+			list_del(&dep->list);
+			kfree(dep);
+		}
+
+		list_for_each_entry_safe(step, stmp, &job->steps, list) {
+			list_del(&step->list);
+			kfree(step);
+		}
+	}
+
+	list_del(&job->list);
+	kfree(job);
+}
+
+int media_jobs_try_queue_job(struct media_job_scheduler *sched,
+			     enum media_job_types type,
+			     struct media_job_dep_ops *dep_ops, void *dep_data)
+{
+	struct media_job_dep *dep;
+	struct media_job *job;
+
+	if (!sched)
+		return 0;
+
+	guard(spinlock)(&sched->lock);
+
+	job = media_jobs_get_job(sched, type, dep_ops, dep_data);
+	if (IS_ERR(job))
+		return PTR_ERR(job);
+
+	list_for_each_entry(dep, &job->deps, list)
+		if (!dep->ops->check_dep(dep->data))
+			return 0; /* Not a failure */
+
+	list_for_each_entry(dep, &job->deps, list)
+		if (dep->ops->clear_dep)
+			dep->ops->clear_dep(dep->data);
+
+	list_move_tail(&job->list, &sched->queue);
+	queue_work(sched->async_wq, &sched->work);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(media_jobs_try_queue_job);
+
+static void __media_jobs_run_jobs(struct work_struct *work)
+{
+	struct media_job_scheduler *sched = container_of(work,
+							 struct media_job_scheduler,
+							 work);
+	struct media_job_step *step;
+	struct media_job *job;
+
+	while (true) {
+		scoped_guard(spinlock, &sched->lock) {
+			if (list_empty(&sched->queue))
+				return;
+
+			job = list_first_entry(&sched->queue, struct media_job,
+					       list);
+		}
+
+		list_for_each_entry(step, &job->steps, list)
+			step->run_step(step->data);
+
+		media_jobs_free_job(job, false);
+	}
+}
+
+void media_jobs_run_jobs(struct media_job_scheduler *sched)
+{
+	if (!sched)
+		return;
+
+	queue_work(sched->async_wq, &sched->work);
+}
+EXPORT_SYMBOL_GPL(media_jobs_run_jobs);
+
+static void __media_jobs_cancel_jobs(struct media_job_scheduler *sched)
+{
+	struct media_job *job, *jtmp;
+
+	list_for_each_entry_safe(job, jtmp, &sched->pending, list)
+		media_jobs_free_job(job, true);
+
+	list_for_each_entry_safe(job, jtmp, &sched->queue, list)
+		media_jobs_free_job(job, true);
+}
+
+void media_jobs_cancel_jobs(struct media_job_scheduler *sched)
+{
+	if (!sched)
+		return;
+
+	guard(spinlock)(&sched->lock);
+	__media_jobs_cancel_jobs(sched);
+}
+EXPORT_SYMBOL_GPL(media_jobs_cancel_jobs);
+
+int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
+				  int (*job_setup)(struct media_job *job, void *data),
+				  void *data, enum media_job_types type)
+{
+	struct media_job_setup_func *new_setup_func;
+
+	guard(spinlock)(&sched->lock);
+
+	new_setup_func = kzalloc(sizeof(*new_setup_func), GFP_KERNEL);
+	if (!new_setup_func)
+		return -ENOMEM;
+
+	new_setup_func->type = type;
+	new_setup_func->job_setup = job_setup;
+	new_setup_func->data = data;
+	list_add_tail(&new_setup_func->list, &sched->setup_funcs);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(media_jobs_add_job_setup_func);
+
+static void __media_jobs_put_scheduler(struct kref *kref)
+{
+	struct media_job_scheduler *sched =
+		container_of(kref, struct media_job_scheduler, kref);
+	struct media_job_setup_func *func, *ftmp;
+
+	cancel_work_sync(&sched->work);
+	destroy_workqueue(sched->async_wq);
+
+	scoped_guard(spinlock, &sched->lock) {
+		__media_jobs_cancel_jobs(sched);
+
+		list_for_each_entry_safe(func, ftmp, &sched->setup_funcs, list) {
+			list_del(&func->list);
+			kfree(func);
+		}
+	}
+
+	list_del(&sched->list);
+	kfree(sched);
+}
+
+void media_jobs_put_scheduler(struct media_job_scheduler *sched)
+{
+	kref_put(&sched->kref, __media_jobs_put_scheduler);
+}
+EXPORT_SYMBOL_GPL(media_jobs_put_scheduler);
+
+struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev)
+{
+	struct media_job_scheduler *sched;
+	char workqueue_name[32];
+	int ret;
+
+	guard(mutex)(&media_job_schedulers_lock);
+
+	list_for_each_entry(sched, &media_job_schedulers, list) {
+		if (sched->mdev == mdev) {
+			kref_get(&sched->kref);
+			return sched;
+		}
+	}
+
+	ret = snprintf(workqueue_name, sizeof(workqueue_name),
+		       "mc jobs (%s)", mdev->driver_name);
+	if (!ret)
+		return ERR_PTR(-EINVAL);
+
+	sched = kzalloc(sizeof(*sched), GFP_KERNEL);
+	if (!sched)
+		return ERR_PTR(-ENOMEM);
+
+	sched->async_wq = alloc_workqueue(workqueue_name, 0, 0);
+	if (!sched->async_wq) {
+		kfree(sched);
+		return ERR_PTR(-EINVAL);
+	}
+
+	sched->mdev = mdev;
+	kref_init(&sched->kref);
+	spin_lock_init(&sched->lock);
+	INIT_LIST_HEAD(&sched->setup_funcs);
+	INIT_LIST_HEAD(&sched->pending);
+	INIT_LIST_HEAD(&sched->queue);
+	INIT_WORK(&sched->work, __media_jobs_run_jobs);
+
+	list_add_tail(&sched->list, &media_job_schedulers);
+
+	return sched;
+}
+EXPORT_SYMBOL_GPL(media_jobs_get_scheduler);
+
+LIST_HEAD(media_job_schedulers);
+
+/* Synchronise access to the global schedulers list */
+DEFINE_MUTEX(media_job_schedulers_lock);
diff --git a/include/media/media-jobs.h b/include/media/media-jobs.h
new file mode 100644
index 000000000000..a97270861251
--- /dev/null
+++ b/include/media/media-jobs.h
@@ -0,0 +1,354 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Media jobs framework
+ *
+ * Copyright 2025 Ideas on Board Oy
+ *
+ * Author: Daniel Scally <dan.scally@ideasonboard.com>
+ */
+
+#include <linux/kref.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+#include <linux/workqueue.h>
+
+#ifndef _MEDIA_JOBS_H
+#define _MEDIA_JOBS_H
+
+struct media_device;
+struct media_entity;
+struct media_job;
+struct media_job_dep;
+
+/**
+ * define MEDIA_JOBS_FL_STEP_ANYWHERE - \
+ *    Flag a media job step as able to run anytime
+ *
+ * This flag informs the framework that a job step does not need a particular
+ * position in the list of job steps and can be placed anywhere.
+ */
+#define MEDIA_JOBS_FL_STEP_ANYWHERE			BIT(0)
+
+/**
+ * define MEDIA_JOBS_FL_STEP_FROM_FRONT - \
+ *    Flag a media job step as needing to be placed near the start of the list
+ *
+ * This flag informs the framework that a job step needs to be placed at a set
+ * position from the start of the list of job steps.
+ */
+#define MEDIA_JOBS_FL_STEP_FROM_FRONT			BIT(1)
+
+/**
+ * define MEDIA_JOBS_FL_STEP_FROM_BACK - \
+ *    Flag a media job step as needing to be placed near the end of the list
+ *
+ * This flag informs the framework that a job step needs to be placed at a set
+ * position from the end of the list of job steps.
+ */
+#define MEDIA_JOBS_FL_STEP_FROM_BACK			BIT(2)
+
+/**
+ * enum media_job_types - Type of media job
+ *
+ * @MEDIA_JOB_TYPE_PIPELINE_PULSE:	A data event moving through the media
+ *					pipeline
+ *
+ * This enumeration details different types of media jobs. The type can be used
+ * to differentiate between which steps and dependencies a driver needs to add
+ * to a job when it is created.
+ */
+enum media_job_types {
+	MEDIA_JOB_TYPE_PIPELINE_PULSE,
+};
+
+/**
+ * struct media_job_scheduler - A job scheduler for a particular media device
+ *
+ * @mdev:		Media device this scheduler is for
+ * @list:		List head to attach to the global list of schedulers
+ * @kref:		Reference counter
+ * @lock:		Lock to protect access to the scheduler
+ * @setup_funcs:	List of &struct media_job_setup_func to populate jobs
+ * @pending:		List of &struct media_jobs created but not yet queued
+ * @queue:		List of &struct media_jobs queued to the scheduler
+ * @work:		Work item to run the jobs
+ * @async_wq:		Workqueue to run the work on
+ *
+ * This struct is the main job scheduler struct - drivers wanting to use this
+ * framework should acquire an instance through media_jobs_get_scheduler() and
+ * subsequently populate it with job setup functions.
+ */
+struct media_job_scheduler {
+	struct media_device *mdev;
+	struct list_head list;
+	struct kref kref;
+
+	spinlock_t lock; /* Synchronise access to the struct's lists */
+	struct list_head setup_funcs;
+	struct list_head pending;
+	struct list_head queue;
+	struct work_struct work;
+	struct workqueue_struct *async_wq;
+};
+
+/**
+ * struct media_job_setup_func - A function to populate a media job with steps
+ *				 and dependencies
+ *
+ * @list:	The list object to attach to the scheduler
+ * @type:	The &enum media_job_types that this function populates a job for
+ * @job_setup:	Function pointer to the driver's job setup function
+ * @data:	Pointer to the driver data for use with @job_setup
+ *
+ * This struct holds data about the functions a driver registers with the jobs
+ * framework in order to populate a new job with steps and dependencies.
+ */
+struct media_job_setup_func {
+	struct list_head list;
+	enum media_job_types type;
+	int (*job_setup)(struct media_job *job, void *data);
+	void *data;
+};
+
+/**
+ * struct media_job - A representation of a job to be run through the pipeline
+ *
+ * @lock:	Lock to protect access to the job's lists
+ * @list:	List head to attach the job to &struct media_job_scheduler in
+ *		either the pending or queue lists
+ * @steps:	List of &struct media_job_step to run the job
+ * @deps:	List of &struct media_job_dep to check that the job can be
+ *		queued
+ * @sched:	Pointer to the media job scheduler
+ * @type:	The type of the job
+ *
+ * This struct holds lists of steps that need to be performed to carry out a
+ * job in the pipeline. A separate list of dependencies allows the queueing of
+ * the job to be delayed until all drivers are ready to carry it out.
+ */
+struct media_job {
+	spinlock_t lock; /* Synchronise access to the struct's lists 6*/
+	struct list_head list;
+	struct list_head steps;
+	struct list_head deps;
+	struct media_job_scheduler *sched;
+	enum media_job_types type;
+};
+
+/**
+ * struct media_job_step - A holder for a function to run as part of a job
+ *
+ * @list:	List head to attach the job step to a &struct media_job.steps
+ * @run_step:	The function to run to perform the step
+ * @data:	Data to pass to the .run_step() function
+ * @flags:	Flags to control how the step is ordered within the job's list
+ *		of steps
+ * @pos:	Position indicator to control how the step is ordered within the
+ *		job's list of steps
+ *
+ * This struct defines a function that needs to be run as part of the execution
+ * of a job in a media pipeline, along with information that help the scheduler
+ * determine what order it should be ran in in reference to the other steps that
+ * are part of the same job.
+ */
+struct media_job_step {
+	struct list_head list;
+	void (*run_step)(void *data);
+	void *data;
+	unsigned int flags;
+	unsigned int pos;
+};
+
+/**
+ * struct media_job_dep_ops - Operations to manage a media job dependency
+ *
+ * @check_dep:	A function to ask the driver whether the dependency is met
+ * @clear_dep:	A function to tell the driver that the job has been queued
+ * @reset_dep:	A function to tell the driver that the job has been cancelled
+ *
+ * Media jobs have dependencies, such as requiring buffers to be queued. These
+ * operations allow a driver to define how the media jobs framework should check
+ * whether or not those dependencies are met and how it should inform them that
+ * it is taking action based on the state of those dependencies.
+ */
+struct media_job_dep_ops {
+	bool (*check_dep)(void *data);
+	void (*clear_dep)(void *data);
+	void (*reset_dep)(void *data);
+};
+
+/**
+ * struct media_job_dep - Representation of media job dependency
+ *
+ * @list:	List head to attach to a &struct media_job.deps
+ * @ops:	A pointer to the dependency's operations functions
+ * @met:	A flag to record whether or not the dependency is met
+ * @data:	Data to pass to the dependency's operations
+ *
+ * This struct represents a dependency of a media job. The operations member
+ * holds pointers to functions allowing the framework to interact with the
+ * driver to check whether or not the dependency is met.
+ */
+struct media_job_dep {
+	struct list_head list;
+	struct media_job_dep_ops *ops;
+	bool met;
+	void *data;
+};
+
+/**
+ * media_jobs_try_queue_job - Try to queue a &struct media_job
+ *
+ * @sched:	Pointer to the job scheduler
+ * @type:	The type of the media job
+ * @dep_ops:	A pointer to the dependency operations for this job
+ * @dep_data:	A pointer to the dependency data for this job
+ *
+ * Try to queue a media job with the scheduler. This function should be called
+ * by the drivers whenever a dependency for a media job is met - for example
+ * when a buffer is queued to the driver. The framework will check to see if an
+ * existing job on the scheduler's pending list shares the same type, dependency
+ * operations and dependency data. If it does then that existing job will be
+ * considered. If there is no extant job with those same parameters, a new job
+ * is allocated and populated by calling the setup functions registered with
+ * the framework.
+ *
+ * The function iterates over the dependencies that are registered with the job
+ * and checks to see if they are met. If they're all met, they're cleared and
+ * the job is placed onto the scheduler's queue.
+ *
+ * To help reduce conditionals in drivers where a driver supports both the use
+ * of the media jobs framework and operation without it, this function is a no
+ * op if @sched is NULL.
+ *
+ * Return: 0 on success or a negative error number
+ */
+int media_jobs_try_queue_job(struct media_job_scheduler *sched,
+			     enum media_job_types type,
+			     struct media_job_dep_ops *dep_ops, void *dep_data);
+
+/**
+ * media_jobs_add_job_step - Add a step to a media job
+ *
+ * @job:	Pointer to the &struct media_job
+ * @run_step:	Pointer to the function to run to execute the step
+ * @data:	Pointer to the data to pass to @run_ste
+ * @flags:	One of the MEDIA_JOBS_FL_STEP_* flags
+ * @pos:	A position indicator to use with @flags
+ *
+ * This function adds a step to the job and should be called from the drivers'
+ * job setup functions as registered with the framework through
+ * media_jobs_add_job_setup_func(). The @flags and @pos parameters are used
+ * to determine the ordering of the steps within the job:
+ *
+ * If @flags has the MEDIA_JOBS_FL_STEP_ANYWHERE bit set, the step is placed
+ * after all steps with MEDIA_JOBS_FL_STEP_FROM_FRONT and before all steps with
+ * MEDIA_JOBS_FL_STEP_FROM_BACK bit set, but otherwise in whatever order this
+ * function is called.
+ *
+ * If @flags has the MEDIA_JOBS_FL_STEP_FROM_FRONT bit set then the step is
+ * placed @pos steps from the front of the list. Attempting to place multiple
+ * steps in the same position will result in an error.
+ *
+ * If @flags has the MEDIA_JOBS_FL_STEP_FROM_BACK bit set then the step is
+ * placed @pos steps from the back of the list. Attempting to place multiple
+ * steps in the same position will result in an error.
+ *
+ * Return: 0 on success or a negative error number
+ */
+int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
+			    void *data, unsigned int flags, unsigned int pos);
+
+/**
+ * media_jobs_add_job_dep - Add a dependency to a media job
+ *
+ * @job:	Pointer to the &struct media_job
+ * @ops:	Pointer to the &struct media_job_dep_ops
+ * @data:	Pointer to the data to pass to the dependency's operations
+ *
+ * This function adds a dependency to the job and should be called from the
+ * drivers job setup functions as registered with the framework through the
+ * media_jobs_add_job_setup_func() function.
+ *
+ * Return: 0 on success or a negative error number
+ */
+int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
+			   void *data);
+
+/**
+ * media_jobs_add_job_setup_func - Add a function that populates a media job
+ *
+ * @sched:	Pointer to the media jobs scheduler
+ * @job_setup:	Pointer to the new job setup function
+ * @data:	Data to pass to the job setup function
+ * @type:	The type of job that this function should be called for
+ *
+ * Drivers that wish to utilise the framework need to use this function to
+ * register a callback that adds job steps and dependencies when one is created.
+ * The function must call media_jobs_add_job_step() and media_jobs_add_job_dep()
+ * to populate the job.
+ *
+ * Return: 0 on success or a negative error number
+ */
+int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
+				  int (*job_setup)(struct media_job *job, void *data),
+				  void *data, enum media_job_types type);
+
+/**
+ * media_jobs_put_scheduler - Put a reference to the media jobs scheduler
+ *
+ * @sched:	Pointer to the media jobs scheduler
+ *
+ * This function puts a reference to the media jobs scheduler, and is intended
+ * to be called in error and exit paths for consuming drivers
+ */
+void media_jobs_put_scheduler(struct media_job_scheduler *sched);
+
+/**
+ * media_jobs_get_scheduler - Get a media jobs scheduler
+ *
+ * @mdev:	Pointer to the media device associated with the scheduler
+ *
+ * This function gets a pointer to a &struct media_job_scheduler associated with
+ * the media device passed to @mdev. If one is not available then it is
+ * allocated and returned. This allows multiple drivers sharing a media graph to
+ * work with the same media job scheduler.
+ *
+ * Return: 0 on success or a negative error number
+ */
+struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev);
+
+/**
+ * media_jobs_run_jobs - Run any media jobs that are ready in the queue
+ *
+ * @sched:	Pointer to the media job scheduler
+ *
+ * This function triggers the workqueue that processes any jobs that have been
+ * queued, and should be called whenever the pipeline is ready to do so.
+ *
+ * To help reduce conditionals in drivers where a driver supports both the use
+ * of the media jobs framework and operation without it, this function is a no
+ * op if @sched is NULL.
+ */
+void media_jobs_run_jobs(struct media_job_scheduler *sched);
+
+/**
+ * media_jobs_cancel_jobs - cancel all waiting jobs
+ *
+ * @sched:	Pointer to the media job scheduler
+ *
+ * This function iterates over any pending and queued jobs, resets their
+ * dependencies and frees the job
+ *
+ * To help reduce conditionals in drivers where a driver supports both the use
+ * of the media jobs framework and operation without it, this function is a no
+ * op if @sched is NULL.
+ */
+void media_jobs_cancel_jobs(struct media_job_scheduler *sched);
+
+extern struct list_head media_job_schedulers;
+extern struct mutex media_job_schedulers_lock;
+
+#endif /* _MEDIA_JOBS_H */
-- 
2.34.1

[PATCH 3/3] media: Documentation: Add documentation for media jobs

[Daniel Scally]

Add a segment to mc-core.rst that explains the purpose behind the
media jobs framework and how to use it.

Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst | 154 +++++++++++++++++++++
 1 file changed, 154 insertions(+)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 1d010bd7ec49..53f13f857c1f 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -327,6 +327,158 @@ Call :c:func:`media_device_register()`, if media devnode isn't registered
 Call :c:func:`media_device_delete()` to free the media_device. Freeing is
 handled by the kref put handler.
 
+Media Jobs Framework
+^^^^^^^^^^^^^^^^^^^^
+
+The media jobs framework exists to facilitate situations in which multiple
+drivers must work together to properly operate a media pipeline in a driver
+agnostic way. The archetypical example is of a memory to memory ISP that does
+not include its own DMA input engine, and which must interact with the driver
+for one that has been integrated. Because the DMA engine and its driver may be
+different between each implementation, hardcoding calls of functions exported by
+the DMA engine driver would not be appropriate. The media jobs framework allows
+the drivers to define the steps that each must execute to correctly push data
+through the pipeline and then schedule the sequence of steps to run in a work
+queue.
+
+To start with each driver must acquire a reference to a
+:c:type:`media_jobs_scheduler` by calling :c:func:`media_jobs_get_scheduler()`,
+passing the pointer to their :c:type:`media_device`. This ensures that all of
+the drivers are working with the same scheduler. Drivers must then call
+:c:func:`media_jobs_add_job_setup_func()` to register a function that populates
+each job with the dependencies that must be cleared to allow it to operate, and
+the steps that must be carried out to execute it. For example:
+
+.. code-block:: c
+
+   static void isp_driver_run_step(void *data)
+   {
+       struct isp *isp = data;
+
+       /*
+        * Logic here to actually execute the necessary steps, for example we
+        * might configure some hardware registers.
+        */
+       ...;
+   }
+
+   static struct media_job_dep_ops ops = {
+       ...,
+   };
+
+   static int isp_driver_add_job_setup_func(struct media_job *job, void *data)
+   {
+       int ret;
+
+       ret = media_jobs_add_job_dep(job, &ops, data);
+       if (ret)
+           return ret;
+
+       ret = media_jobs_add_job_step(job, isp_driver_run_step, data,
+                                     MEDIA_JOBS_FL_STEP_ANYWHERE, 0);
+       if (ret)
+           return ret;
+
+       return 0;
+   }
+
+The flags parameter of `media_jobs_add_job_step()` must be one of
+:c:macro:`MEDIA_JOBS_FL_STEP_ANYWHERE`, :c:macro:`MEDIA_JOBS_FL_STEP_FROM_FRONT`
+or :c:macro:`MEDIA_JOBS_FL_STEP_FROM_BACK`. The flag and pos parameters together
+define the order of the step within the job. Steps added with
+`MEDIA_JOBS_FL_STEP_ANYWHERE` will go after all steps that are added with
+`MEDIA_JOBS_FL_STEP_FROM_FRONT` and all steps with `MEDIA_JOBS_FL_STEP_ANYWHERE`
+that either have a lower `pos` or were previously added. They will go before all
+those added with `MEDIA_JOBS_FL_STEP_FROM_BACK` and all steps with
+`MEDIA_JOBS_FL_STEP_ANYWHERE` that have a higher `pos`. Steps added with
+`MEDIA_JOBS_FL_STEP_FROM_FRONT` will go `pos` places from the front of the list,
+and steps added with `flags` set to `MEDIA_JOBS_FL_STEP_FROM_BACK`` will go
+`pos` places from the end of the list. This allows multiple drivers to quite
+precisely define which steps need to be executed and what order they should be
+executed in.
+
+Adding a step with the same `flags` and `pos` as a previously added step will
+result in an error.
+
+The functions held in :c:type:`media_job_dep_ops` define how the media jobs
+framework handles job dependencies. It is expected that there will be some hard
+dependencies before a job can be executed; for example pushing a buffer of image
+data through an ISP pipeline necessarily requires that an input buffer be ready
+and an output buffer be ready to accept the processed data. The operations ask
+the driver if the dependencies are met, tell the driver that a job has been
+queued and reset the dependencies in the event the job is cancelled:
+
+.. code-block:: c
+
+   struct isp {
+
+       ...;
+
+       struct {
+           struct list_head pending;
+	   struct list_head processing;
+       } buffers;
+   }
+
+   static bool isp_driver_check_dep(void *data)
+   {
+       struct isp *isp = data;
+
+       /*
+        * Do we have a buffer queued ready to accept the ISP's output data?
+        */
+       if (list_empty(isp->buffers.pending))
+           return false;
+
+       return true;
+   }
+
+   static void isp_driver_clear_dep(void *data)
+   {
+       struct isp *isp = data;
+       struct buf *buf;
+
+       /*
+        * We need to "consume" the buffer so that it's not also considered as
+        * meeting this dependency for the next attempt to queue a job
+        */
+       buf = list_first_entry(&isp->buffers.pending, struct buf, list);
+       list_move_tail(&buf->list, isp->buffers.processing);
+   }
+
+   static void isp_driver_reset_dep(void *data)
+   {
+       struct isp *isp = data;
+       struct buf *buf;
+
+       /*
+        * If a queued job is cancelled then we need to return the dependency to
+        * its original state, which in this example means returning it to the
+        * pending queue.
+        */
+       buf = list_first_entry(&isp->buffers.pending, struct buf, list);
+       list_move_tail(&buf->list, isp->buffers.pending);
+   }
+
+   static struct media_job_dep_ops ops = {
+       .check_dep = isp_driver_check_dep,
+       .clear_dep = isp_driver_clear_dep,
+       .reset_dep = isp_driver_reset_dep,
+   };
+
+The actual creation and queueing of the jobs should be done by the drivers by
+calling :c:func:`media_jobs_try_queue_job()` at any time a dependency of the
+job is met - for example (following the earlier example) when a buffer is queued
+to either the ISP or DMA engine's driver. When all of the dependencies that are
+necessary for a job to be queued are met, this function will push a job to the
+scheduler's queue.
+
+The scheduler has a workqueue that runs the jobs. This is triggered by calls to
+the :c:func:`media_jobs_run_jobs()` function, which must be called periodically
+as the pipeline is running. When the streaming is finished the drivers should
+shut down the workqueue and cancel the queued jobs by calling
+:c:func:`media_jobs_cancel_jobs()`.
+
 API Definitions
 ^^^^^^^^^^^^^^^
 
@@ -336,6 +488,8 @@ API Definitions
 
 .. kernel-doc:: include/media/media-entity.h
 
+.. kernel-doc:: include/media/media-jobs.h
+
 .. kernel-doc:: include/media/media-request.h
 
 .. kernel-doc:: include/media/media-dev-allocator.h
-- 
2.34.1

Re: [PATCH 0/3] Add media jobs framework

[Dan Scally]

Hi all

On 19/05/2025 15:04, Daniel Scally wrote:
> Hello all
>
> This series adds a new API to the media controller framework, which
> I'm calling "media jobs". The framework is intended to facilitate
> communication between separate drivers which need to work together
> to fully operate a media pipeline. For example, the need for the
> framework arose when writing support for the ISP in the RZ/V2H; the
> ISP is fed by a DMA engine which is not part of the same IP block,
> and so is driven by its own driver (though sharing a media graph).
> The ISP driver needs to be able to communicate with the DMA engine
> driver to instruct it to push the next frame. Because the DMA engine
> might be different on a different platform that used the ISP, direct
> calls into functions exported by the DMA engine driver wouldn't be
> scalable, and so this driver agnostic route was adopted. The
> framework allows drivers to define the steps that need to be taken
> (for example writing configuration data, reading statistics data,
> writing buffer addresses and triggering data transmission) to complete
> a "job" (of which the only current example is the processing of a
> frame of data through the pipeline, though I expect that other use
> cases could become apparent too) and to then schedule them into a
> work queue once driver definable dependencies have been met. The
> dependencies might, for example, be the queuing of buffers to V4L2
> capture / output devices.
>
> The framework allows precise definition of the ordering of the steps
> regardless of the order in which the drivers populate the jobs. Steps
> can be flagged as being placed at a particular position from the front
> or back of the queue (I.E. last, or third from last) or as requiring
> no particular order. This would allow the construction of a job like:
>
> Step 0 (ISP Driver): Program the hardware with parameters
> Step 1 to N-1 (Both drivers): Program the hardware with buffers
> Step N (DMA Engine Driver): Send a frame of data to the ISP
> ... ISP processes data ...
> Step N + 1 (ISP Driver): Read out statistics data from the last frame
>
> The mali-c55 ISP driver and the DMA engine feeding it on the RZ/V2H
> (called the rzv2h-ivc driver) both use the framework, and will be
> posted shortly after this series with references back to it. I will
> reply to this message with links to those series for convenience.

The two submissions using the new framework:

IVC: https://lore.kernel.org/linux-media/20250519145754.454005-1-dan.scally@ideasonboard.com/T/

ISP: https://lore.kernel.org/linux-media/20250519143409.451100-1-dan.scally@ideasonboard.com/T/


Thanks

Dan

>
> The first patch in this set is not strictly part of the framework,
> but also facilitates multiple drivers with V4L2 Video Devices
> sharing a single media graph. We have a requirement to delay the
> start of streaming until all of a pipeline's video devices have had
> their .start_streaming() operations called; these new entity ops
> provide a mechanism through which each driver can inform the other
> that the last video device in the pipeline has now been started.
>
> Thanks
> Dan
>
> Daniel Scally (3):
>    media: mc: entity: Add pipeline_started/stopped ops
>    media: mc: Add media jobs framework
>    media: Documentation: Add documentation for media jobs
>
>   Documentation/driver-api/media/mc-core.rst | 154 +++++++
>   drivers/media/mc/Makefile                  |   2 +-
>   drivers/media/mc/mc-entity.c               |  45 +++
>   drivers/media/mc/mc-jobs.c                 | 446 +++++++++++++++++++++
>   include/media/media-entity.h               |  24 ++
>   include/media/media-jobs.h                 | 354 ++++++++++++++++
>   6 files changed, 1024 insertions(+), 1 deletion(-)
>   create mode 100644 drivers/media/mc/mc-jobs.c
>   create mode 100644 include/media/media-jobs.h
>

Re: [PATCH 1/3] media: mc: entity: Add pipeline_started/stopped ops

[Jacopo Mondi]

Hi Dan

On Mon, May 19, 2025 at 03:04:01PM +0100, Daniel Scally wrote:
> Add two new members to struct media_entity_operations, along with new
> functions in media-entity.c to traverse a media pipeline and call the
> new operations. The new functions are intended to be used to signal
> to a media pipeline that it has fully started, with the entity ops
> allowing drivers to define some action to be taken when those
> conditions are met.
>
> The combination of the new functions and operations allows drivers
> which are part of a multi-driver pipeline to delay actually starting
> streaming until all of the conditions for streaming succcessfully are
> met across all drivers.

Maybe s/succcessfully are/are successfully/

Or was this (the three 'c' apart) intentional ?

>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
>  drivers/media/mc/mc-entity.c | 45 ++++++++++++++++++++++++++++++++++++
>  include/media/media-entity.h | 24 +++++++++++++++++++
>  2 files changed, 69 insertions(+)
>
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 045590905582..e36b1710669d 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -1053,6 +1053,51 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
>  }
>  EXPORT_SYMBOL_GPL(__media_pipeline_entity_iter_next);
>
> +int media_pipeline_started(struct media_pipeline *pipe)
> +{
> +	struct media_pipeline_entity_iter iter;
> +	struct media_entity *entity;
> +	int ret;
> +
> +	ret = media_pipeline_entity_iter_init(pipe, &iter);
> +	if (ret)
> +		return ret;
> +
> +	media_pipeline_for_each_entity(pipe, &iter, entity) {
> +		ret = media_entity_call(entity, pipeline_started);
> +		if (ret && ret != -ENOIOCTLCMD)
> +			goto err_notify_stopped;
> +	}
> +
> +	media_pipeline_entity_iter_cleanup(&iter);
> +
> +	return ret == -ENOIOCTLCMD ? 0 : ret;

Shouldn't you just return 0 ? If a ret < 0 is encoutered in the loop
we just to the below label

> +
> +err_notify_stopped:
> +	media_pipeline_stopped(pipe);

Do you need to media_pipeline_entity_iter_cleanup() ?

> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(media_pipeline_started);
> +
> +int media_pipeline_stopped(struct media_pipeline *pipe)
> +{
> +	struct media_pipeline_entity_iter iter;
> +	struct media_entity *entity;
> +	int ret;
> +
> +	ret = media_pipeline_entity_iter_init(pipe, &iter);
> +	if (ret)
> +		return ret;
> +
> +	media_pipeline_for_each_entity(pipe, &iter, entity)
> +		media_entity_call(entity, pipeline_stopped);
> +
> +	media_pipeline_entity_iter_cleanup(&iter);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(media_pipeline_stopped);
> +
>  /* -----------------------------------------------------------------------------
>   * Links management
>   */
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 64cf590b1134..e858326b95cb 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -269,6 +269,10 @@ struct media_pad {
>   *			media_entity_has_pad_interdep().
>   *			Optional: If the operation isn't implemented all pads
>   *			will be considered as interdependent.
> + * @pipeline_started:	Notify this entity that the pipeline it is a part of has
> + *			been started
> + * @pipeline_stopped:	Notify this entity that the pipeline it is a part of has
> + *			been stopped
>   *
>   * .. note::
>   *
> @@ -284,6 +288,8 @@ struct media_entity_operations {
>  	int (*link_validate)(struct media_link *link);
>  	bool (*has_pad_interdep)(struct media_entity *entity, unsigned int pad0,
>  				 unsigned int pad1);
> +	int (*pipeline_started)(struct media_entity *entity);
> +	void (*pipeline_stopped)(struct media_entity *entity);
>  };
>
>  /**
> @@ -1261,6 +1267,24 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
>  	     entity != NULL;							\
>  	     entity = __media_pipeline_entity_iter_next((pipe), iter, entity))
>
> +/**
> + * media_pipeline_started - Inform entities in a pipeline that it has started
> + * @pipe:	The pipeline
> + *
> + * Iterate on all entities in a media pipeline and call their pipeline_started
> + * member of media_entity_operations.
> + */
> +int media_pipeline_started(struct media_pipeline *pipe);
> +
> +/**
> + * media_pipeline_stopped - Inform entities in a pipeline that it has stopped
> + * @pipe:	The pipeline
> + *
> + * Iterate on all entities in a media pipeline and call their pipeline_stopped
> + * member of media_entity_operations.
> + */
> +int media_pipeline_stopped(struct media_pipeline *pipe);
> +

All good, but I don't see these operations being used at all in this
series ?

>  /**
>   * media_pipeline_alloc_start - Mark a pipeline as streaming
>   * @pad: Starting pad
> --
> 2.34.1
>
>

Re: [PATCH 2/3] media: mc: Add media jobs framework

[Jacopo Mondi]

Hi Dan,
   just a few notes here and there, I will have to check users in more
detail to comment on the interface

On Mon, May 19, 2025 at 03:04:02PM +0100, Daniel Scally wrote:
> Add a new framework to the media subsystem describing media jobs.
> This framework is intended to be able to model the interactions
> between multiple different drivers that need to be run in concert
> to fully control a media pipeline, for example an ISP driver and a
> driver controlling a DMA device that feeds data from memory in to
> that ISP.
>
> The new framework allows all drivers involved to add explicit steps
> that need to be performed, and to control the ordering of those steps
> precisely. Once the job with its steps has been created it's then
> scheduled to be run with a workqueue which executes each step in the
> defined order.
>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
>  drivers/media/mc/Makefile  |   2 +-
>  drivers/media/mc/mc-jobs.c | 446 +++++++++++++++++++++++++++++++++++++
>  include/media/media-jobs.h | 354 +++++++++++++++++++++++++++++
>  3 files changed, 801 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/media/mc/mc-jobs.c
>  create mode 100644 include/media/media-jobs.h
>
> diff --git a/drivers/media/mc/Makefile b/drivers/media/mc/Makefile
> index 2b7af42ba59c..9148bbfd1578 100644
> --- a/drivers/media/mc/Makefile
> +++ b/drivers/media/mc/Makefile
> @@ -1,7 +1,7 @@
>  # SPDX-License-Identifier: GPL-2.0
>
>  mc-objs	:= mc-device.o mc-devnode.o mc-entity.o \
> -	   mc-request.o
> +	   mc-jobs.o mc-request.o
>
>  ifneq ($(CONFIG_USB),)
>  	mc-objs += mc-dev-allocator.o
> diff --git a/drivers/media/mc/mc-jobs.c b/drivers/media/mc/mc-jobs.c
> new file mode 100644
> index 000000000000..1f04cdf63d27
> --- /dev/null
> +++ b/drivers/media/mc/mc-jobs.c
> @@ -0,0 +1,446 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Media jobs framework
> + *
> + * Copyright 2025 Ideas on Board Oy
> + *
> + * Author: Daniel Scally <dan.scally@ideasonboard.com>
> + */
> +
> +#include <linux/cleanup.h>
> +#include <linux/kref.h>
> +#include <linux/list.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>

Some of these are included by the header file

> +
> +#include <media/media-device.h>
> +#include <media/media-entity.h>
> +#include <media/media-jobs.h>
> +
> +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
> +			    void *data, unsigned int flags, unsigned int pos)
> +{
> +	struct media_job_step *step, *tmp;
> +	unsigned int num = flags;
> +	unsigned int count = 0;
> +
> +	guard(spinlock)(&job->lock);
> +
> +	if (!flags) {
> +		WARN_ONCE(1, "%s(): No flag bits set\n", __func__);
> +		return -EINVAL;
> +	}

You could check this before taking the lock

> +
> +	/* Count the number of set flags; they're mutually exclusive. */

if you want just to count flags you can use hweightX from bitops.h

> +	while (num) {
> +		num &= (num - 1);
> +		count++;
> +	}
> +
> +	if (count > 1) {
> +		WARN_ONCE(1, "%s(): Multiple flag bits set\n", __func__);
> +		return -EINVAL;
> +	}
> +
> +	step = kzalloc(sizeof(*step), GFP_KERNEL);
> +	if (!step)
> +		return -ENOMEM;
> +
> +	step->run_step = run_step;
> +	step->data = data;
> +	step->flags = flags;
> +	step->pos = pos;
> +
> +	/*
> +	 * We need to decide where to place the step. If the list is empty that
> +	 * is really easy (and also the later code is much easier if the code is
> +	 * guaranteed not to be empty...)
> +	 */
> +	if (list_empty(&job->steps)) {
> +		list_add_tail(&step->list, &job->steps);
> +		return 0;
> +	}
> +
> +	/*
> +	 * If we've been asked to place it at a specific position from the end
> +	 * of the list, we cycle back through it until either we exhaust the
> +	 * list or find an entry that needs to go further from the back than the
> +	 * new one.
> +	 */
> +	if ((flags & MEDIA_JOBS_FL_STEP_FROM_BACK)) {
> +		list_for_each_entry_reverse(tmp, &job->steps, list) {
> +			if (tmp->flags == flags && tmp->pos == pos)
> +				return -EINVAL;
> +
> +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_BACK ||
> +			    tmp->pos > pos)
> +				break;
> +		}
> +
> +		/*
> +		 * If the entry we broke on is also one placed from the back and
> +		 * should be closer to the back than the new one, we place the
> +		 * new one in front of it...otherwise place the new one behind
> +		 * it.
> +		 */
> +		if (tmp->flags == flags && tmp->pos < pos)
> +			list_add_tail(&step->list, &tmp->list);
> +		else
> +			list_add(&step->list, &tmp->list);
> +
> +		return 0;
> +	}
> +
> +	/*
> +	 * If we've been asked to place it a specific position from the front of
> +	 * the list we do the same kind of operation, but going from the front
> +	 * instead.
> +	 */
> +	if (flags & MEDIA_JOBS_FL_STEP_FROM_FRONT) {
> +		list_for_each_entry(tmp, &job->steps, list) {
> +			if (tmp->flags == flags && tmp->pos == pos)
> +				return -EINVAL;
> +
> +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_FRONT ||
> +			    tmp->pos > pos)
> +				break;
> +		}
> +
> +		/*
> +		 * If the entry we broke on is also placed from the front and
> +		 * should be closed to the front than the new one, we place the
> +		 * new one behind it, otherwise in front of it.
> +		 */
> +		if (tmp->flags == flags && tmp->pos < pos)
> +			list_add(&step->list, &tmp->list);
> +		else
> +			list_add_tail(&step->list, &tmp->list);
> +
> +		return 0;
> +	}

There probably is room to factorize out the common code parts from
here

> +
> +	/*
> +	 * If the step is flagged as "can go anywhere" we just need to try to
> +	 * find the first "from the back" entry and add it immediately before
> +	 * that. If we can't find one, add it after whatever we did find.
> +	 */
> +	if (flags & MEDIA_JOBS_FL_STEP_ANYWHERE) {
> +		list_for_each_entry(tmp, &job->steps, list)
> +			if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK))
> +				break;
> +
> +		if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK) ||
> +		    list_entry_is_head(tmp, &job->steps, list))
> +			list_add_tail(&step->list, &tmp->list);
> +		else
> +			list_add(&step->list, &tmp->list);
> +
> +		return 0;
> +	}
> +
> +	/* Shouldn't get here, unless the flag value is wrong. */
> +	WARN_ONCE(1, "%s(): Invalid flag value\n", __func__);
> +	return -EINVAL;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_add_job_step);
> +
> +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
> +			   void *data)
> +{
> +	struct media_job_dep *dep;
> +
> +	if (!ops || !ops->check_dep || !data)
> +		return -EINVAL;
> +
> +	guard(spinlock)(&job->lock);
> +
> +	/* Confirm the same dependency hasn't already been added */
> +	list_for_each_entry(dep, &job->deps, list)
> +		if (dep->ops == ops && dep->data == data)
> +			return -EINVAL;
> +
> +	dep = kzalloc(sizeof(*dep), GFP_KERNEL);
> +	if (!dep)
> +		return -ENOMEM;
> +
> +	dep->ops = ops;
> +	dep->data = data;
> +	list_add(&dep->list, &job->deps);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_add_job_dep);
> +
> +static bool media_jobs_check_pending_job(struct media_job *job,
> +					 enum media_job_types type,
> +					 struct media_job_dep_ops *dep_ops,
> +					 void *data)
> +{
> +	struct media_job_dep *dep;
> +
> +	guard(spinlock)(&job->lock);
> +
> +	if (job->type != type)
> +		return false;

possibily moved outside of the lock as well

> +
> +	list_for_each_entry(dep, &job->deps, list) {
> +		if (dep->ops == dep_ops && dep->data == data) {
> +			if (dep->met)
> +				return false;
> +
> +			break;
> +		}
> +	}
> +
> +	dep->met = true;
> +	return true;
> +}
> +
> +static struct media_job *media_jobs_get_job(struct media_job_scheduler *sched,
> +					    enum media_job_types type,
> +					    struct media_job_dep_ops *dep_ops,
> +					    void *dep_data)
> +{
> +	struct media_job_setup_func *jsf;
> +	struct media_job *job;
> +	int ret;
> +
> +	list_for_each_entry(job, &sched->pending, list)
> +		if (media_jobs_check_pending_job(job, type, dep_ops, dep_data))
> +			return job;

Not sure I got this part.

media_jobs_get_job() has a single caller, and it either returns a
pending job or allocates a new one.

To check if a pending job should be returned you try to math the
dep_ops and dep_data and if you get one that matches you identify the
job and return it, but only once, as once a dependency is met it will
then invalidate the search. What am I missing ?

> +
> +	job = kzalloc(sizeof(*job), GFP_KERNEL);
> +	if (!job)
> +		return ERR_PTR(-ENOMEM);
> +
> +	spin_lock_init(&job->lock);
> +	INIT_LIST_HEAD(&job->deps);
> +	INIT_LIST_HEAD(&job->steps);
> +	job->type = type;
> +	job->sched = sched;
> +
> +	list_for_each_entry(jsf, &sched->setup_funcs, list) {
> +		if (jsf->type != type)
> +			continue;
> +
> +		ret = jsf->job_setup(job, jsf->data);
> +		if (ret) {
> +			kfree(job);

do we need an operation to undo job_setup() or do you think
it's not needed ?

> +			return ERR_PTR(ret);
> +		}
> +	}
> +
> +	list_add_tail(&job->list, &sched->pending);
> +
> +	/* This marks the dependency as met */
> +	media_jobs_check_pending_job(job, type, dep_ops, dep_data);
> +
> +	return job;
> +}
> +
> +static void media_jobs_free_job(struct media_job *job, bool reset)
> +{
> +	struct media_job_step *step, *stmp;
> +	struct media_job_dep *dep, *dtmp;
> +
> +	scoped_guard(spinlock, &job->lock) {
> +		list_for_each_entry_safe(dep, dtmp, &job->deps, list) {
> +			if (reset && dep->ops->reset_dep)
> +				dep->ops->reset_dep(dep->data);
> +
> +			list_del(&dep->list);
> +			kfree(dep);
> +		}
> +
> +		list_for_each_entry_safe(step, stmp, &job->steps, list) {
> +			list_del(&step->list);
> +			kfree(step);
> +		}
> +	}
> +
> +	list_del(&job->list);

Probably better to be done while holding the sched spinlock that
protects the jobs list ?


> +	kfree(job);
> +}
> +
> +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
> +			     enum media_job_types type,
> +			     struct media_job_dep_ops *dep_ops, void *dep_data)
> +{
> +	struct media_job_dep *dep;
> +	struct media_job *job;
> +
> +	if (!sched)

If this happens, I guess something went very wrong. Should we
WARN_ONCE() to catch this during development ?

> +		return 0;
> +
> +	guard(spinlock)(&sched->lock);
> +
> +	job = media_jobs_get_job(sched, type, dep_ops, dep_data);
> +	if (IS_ERR(job))
> +		return PTR_ERR(job);
> +
> +	list_for_each_entry(dep, &job->deps, list)
> +		if (!dep->ops->check_dep(dep->data))

Should you check if dep->ops->check_dep() is valid before calling it ?

> +			return 0; /* Not a failure */
> +
> +	list_for_each_entry(dep, &job->deps, list)
> +		if (dep->ops->clear_dep)
> +			dep->ops->clear_dep(dep->data);
> +
> +	list_move_tail(&job->list, &sched->queue);
> +	queue_work(sched->async_wq, &sched->work);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_try_queue_job);
> +
> +static void __media_jobs_run_jobs(struct work_struct *work)
> +{
> +	struct media_job_scheduler *sched = container_of(work,
> +							 struct media_job_scheduler,
> +							 work);
> +	struct media_job_step *step;
> +	struct media_job *job;
> +
> +	while (true) {
> +		scoped_guard(spinlock, &sched->lock) {
> +			if (list_empty(&sched->queue))
> +				return;
> +
> +			job = list_first_entry(&sched->queue, struct media_job,
> +					       list);
> +		}
> +
> +		list_for_each_entry(step, &job->steps, list)

I take this as step->run_step() has been validated

> +			step->run_step(step->data);
> +
> +		media_jobs_free_job(job, false);
> +	}
> +}
> +
> +void media_jobs_run_jobs(struct media_job_scheduler *sched)
> +{
> +	if (!sched)
> +		return;
> +
> +	queue_work(sched->async_wq, &sched->work);
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_run_jobs);
> +
> +static void __media_jobs_cancel_jobs(struct media_job_scheduler *sched)
> +{
> +	struct media_job *job, *jtmp;
> +

if you want to make sure a lock is held when calling a function you
could use lockdep_assert_held

> +	list_for_each_entry_safe(job, jtmp, &sched->pending, list)
> +		media_jobs_free_job(job, true);
> +
> +	list_for_each_entry_safe(job, jtmp, &sched->queue, list)
> +		media_jobs_free_job(job, true);
> +}
> +
> +void media_jobs_cancel_jobs(struct media_job_scheduler *sched)
> +{
> +	if (!sched)
> +		return;
> +
> +	guard(spinlock)(&sched->lock);
> +	__media_jobs_cancel_jobs(sched);
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_cancel_jobs);
> +
> +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
> +				  int (*job_setup)(struct media_job *job, void *data),
> +				  void *data, enum media_job_types type)
> +{
> +	struct media_job_setup_func *new_setup_func;
> +
> +	guard(spinlock)(&sched->lock);
> +
> +	new_setup_func = kzalloc(sizeof(*new_setup_func), GFP_KERNEL);
> +	if (!new_setup_func)
> +		return -ENOMEM;
> +
> +	new_setup_func->type = type;
> +	new_setup_func->job_setup = job_setup;
> +	new_setup_func->data = data;
> +	list_add_tail(&new_setup_func->list, &sched->setup_funcs);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_add_job_setup_func);
> +
> +static void __media_jobs_put_scheduler(struct kref *kref)
> +{
> +	struct media_job_scheduler *sched =
> +		container_of(kref, struct media_job_scheduler, kref);
> +	struct media_job_setup_func *func, *ftmp;
> +
> +	cancel_work_sync(&sched->work);
> +	destroy_workqueue(sched->async_wq);
> +
> +	scoped_guard(spinlock, &sched->lock) {
> +		__media_jobs_cancel_jobs(sched);
> +
> +		list_for_each_entry_safe(func, ftmp, &sched->setup_funcs, list) {
> +			list_del(&func->list);
> +			kfree(func);
> +		}
> +	}
> +
> +	list_del(&sched->list);
> +	kfree(sched);
> +}
> +
> +void media_jobs_put_scheduler(struct media_job_scheduler *sched)
> +{
> +	kref_put(&sched->kref, __media_jobs_put_scheduler);
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_put_scheduler);
> +
> +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev)
> +{
> +	struct media_job_scheduler *sched;
> +	char workqueue_name[32];
> +	int ret;
> +
> +	guard(mutex)(&media_job_schedulers_lock);
> +
> +	list_for_each_entry(sched, &media_job_schedulers, list) {
> +		if (sched->mdev == mdev) {
> +			kref_get(&sched->kref);
> +			return sched;
> +		}
> +	}

Ok big question here: why are we keeping a static list of schedulers
if a single scheduler can be associated to an mdev ?

Thinking forward a bit here: once we have multiple contexts, each
context should have its own scheduler ? Or are planning to a global
scheduler per single media device ?

> +
> +	ret = snprintf(workqueue_name, sizeof(workqueue_name),
> +		       "mc jobs (%s)", mdev->driver_name);
> +	if (!ret)
> +		return ERR_PTR(-EINVAL);
> +
> +	sched = kzalloc(sizeof(*sched), GFP_KERNEL);
> +	if (!sched)
> +		return ERR_PTR(-ENOMEM);
> +
> +	sched->async_wq = alloc_workqueue(workqueue_name, 0, 0);
> +	if (!sched->async_wq) {
> +		kfree(sched);
> +		return ERR_PTR(-EINVAL);
> +	}
> +
> +	sched->mdev = mdev;
> +	kref_init(&sched->kref);
> +	spin_lock_init(&sched->lock);
> +	INIT_LIST_HEAD(&sched->setup_funcs);
> +	INIT_LIST_HEAD(&sched->pending);
> +	INIT_LIST_HEAD(&sched->queue);
> +	INIT_WORK(&sched->work, __media_jobs_run_jobs);
> +
> +	list_add_tail(&sched->list, &media_job_schedulers);
> +
> +	return sched;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_get_scheduler);
> +
> +LIST_HEAD(media_job_schedulers);
> +
> +/* Synchronise access to the global schedulers list */
> +DEFINE_MUTEX(media_job_schedulers_lock);

Still, even with a single scheduler per media device, I wonder why the
scheduler cannot live in the mdev itself...

I'll have a more detailed look at the users and I'll get back on the
interface.

In the meantime, I wonder if an introductory documentation page that
explains why direct function calls are not always possible in drivers
that interact in a pipeline and what the use case for media_job is,
wouldn't help people getting familiar with it quicker.

Thanks
   j

> diff --git a/include/media/media-jobs.h b/include/media/media-jobs.h
> new file mode 100644
> index 000000000000..a97270861251
> --- /dev/null
> +++ b/include/media/media-jobs.h
> @@ -0,0 +1,354 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Media jobs framework
> + *
> + * Copyright 2025 Ideas on Board Oy
> + *
> + * Author: Daniel Scally <dan.scally@ideasonboard.com>
> + */
> +
> +#include <linux/kref.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/spinlock.h>
> +#include <linux/types.h>
> +#include <linux/workqueue.h>
> +
> +#ifndef _MEDIA_JOBS_H
> +#define _MEDIA_JOBS_H
> +
> +struct media_device;
> +struct media_entity;
> +struct media_job;
> +struct media_job_dep;
> +
> +/**
> + * define MEDIA_JOBS_FL_STEP_ANYWHERE - \
> + *    Flag a media job step as able to run anytime
> + *
> + * This flag informs the framework that a job step does not need a particular
> + * position in the list of job steps and can be placed anywhere.
> + */
> +#define MEDIA_JOBS_FL_STEP_ANYWHERE			BIT(0)
> +
> +/**
> + * define MEDIA_JOBS_FL_STEP_FROM_FRONT - \
> + *    Flag a media job step as needing to be placed near the start of the list
> + *
> + * This flag informs the framework that a job step needs to be placed at a set
> + * position from the start of the list of job steps.
> + */
> +#define MEDIA_JOBS_FL_STEP_FROM_FRONT			BIT(1)
> +
> +/**
> + * define MEDIA_JOBS_FL_STEP_FROM_BACK - \
> + *    Flag a media job step as needing to be placed near the end of the list
> + *
> + * This flag informs the framework that a job step needs to be placed at a set
> + * position from the end of the list of job steps.
> + */
> +#define MEDIA_JOBS_FL_STEP_FROM_BACK			BIT(2)
> +
> +/**
> + * enum media_job_types - Type of media job
> + *
> + * @MEDIA_JOB_TYPE_PIPELINE_PULSE:	A data event moving through the media
> + *					pipeline
> + *
> + * This enumeration details different types of media jobs. The type can be used
> + * to differentiate between which steps and dependencies a driver needs to add
> + * to a job when it is created.
> + */
> +enum media_job_types {
> +	MEDIA_JOB_TYPE_PIPELINE_PULSE,
> +};
> +
> +/**
> + * struct media_job_scheduler - A job scheduler for a particular media device
> + *
> + * @mdev:		Media device this scheduler is for
> + * @list:		List head to attach to the global list of schedulers
> + * @kref:		Reference counter
> + * @lock:		Lock to protect access to the scheduler
> + * @setup_funcs:	List of &struct media_job_setup_func to populate jobs
> + * @pending:		List of &struct media_jobs created but not yet queued
> + * @queue:		List of &struct media_jobs queued to the scheduler
> + * @work:		Work item to run the jobs
> + * @async_wq:		Workqueue to run the work on
> + *
> + * This struct is the main job scheduler struct - drivers wanting to use this
> + * framework should acquire an instance through media_jobs_get_scheduler() and
> + * subsequently populate it with job setup functions.
> + */
> +struct media_job_scheduler {
> +	struct media_device *mdev;
> +	struct list_head list;
> +	struct kref kref;
> +
> +	spinlock_t lock; /* Synchronise access to the struct's lists */
> +	struct list_head setup_funcs;
> +	struct list_head pending;
> +	struct list_head queue;
> +	struct work_struct work;
> +	struct workqueue_struct *async_wq;
> +};
> +
> +/**
> + * struct media_job_setup_func - A function to populate a media job with steps
> + *				 and dependencies
> + *
> + * @list:	The list object to attach to the scheduler
> + * @type:	The &enum media_job_types that this function populates a job for
> + * @job_setup:	Function pointer to the driver's job setup function
> + * @data:	Pointer to the driver data for use with @job_setup
> + *
> + * This struct holds data about the functions a driver registers with the jobs
> + * framework in order to populate a new job with steps and dependencies.
> + */
> +struct media_job_setup_func {
> +	struct list_head list;
> +	enum media_job_types type;
> +	int (*job_setup)(struct media_job *job, void *data);
> +	void *data;
> +};
> +
> +/**
> + * struct media_job - A representation of a job to be run through the pipeline
> + *
> + * @lock:	Lock to protect access to the job's lists
> + * @list:	List head to attach the job to &struct media_job_scheduler in
> + *		either the pending or queue lists
> + * @steps:	List of &struct media_job_step to run the job
> + * @deps:	List of &struct media_job_dep to check that the job can be
> + *		queued
> + * @sched:	Pointer to the media job scheduler
> + * @type:	The type of the job
> + *
> + * This struct holds lists of steps that need to be performed to carry out a
> + * job in the pipeline. A separate list of dependencies allows the queueing of
> + * the job to be delayed until all drivers are ready to carry it out.
> + */
> +struct media_job {
> +	spinlock_t lock; /* Synchronise access to the struct's lists 6*/
> +	struct list_head list;
> +	struct list_head steps;
> +	struct list_head deps;
> +	struct media_job_scheduler *sched;
> +	enum media_job_types type;
> +};
> +
> +/**
> + * struct media_job_step - A holder for a function to run as part of a job
> + *
> + * @list:	List head to attach the job step to a &struct media_job.steps
> + * @run_step:	The function to run to perform the step
> + * @data:	Data to pass to the .run_step() function
> + * @flags:	Flags to control how the step is ordered within the job's list
> + *		of steps
> + * @pos:	Position indicator to control how the step is ordered within the
> + *		job's list of steps
> + *
> + * This struct defines a function that needs to be run as part of the execution
> + * of a job in a media pipeline, along with information that help the scheduler
> + * determine what order it should be ran in in reference to the other steps that
> + * are part of the same job.
> + */
> +struct media_job_step {
> +	struct list_head list;
> +	void (*run_step)(void *data);
> +	void *data;
> +	unsigned int flags;
> +	unsigned int pos;
> +};
> +
> +/**
> + * struct media_job_dep_ops - Operations to manage a media job dependency
> + *
> + * @check_dep:	A function to ask the driver whether the dependency is met
> + * @clear_dep:	A function to tell the driver that the job has been queued
> + * @reset_dep:	A function to tell the driver that the job has been cancelled
> + *
> + * Media jobs have dependencies, such as requiring buffers to be queued. These
> + * operations allow a driver to define how the media jobs framework should check
> + * whether or not those dependencies are met and how it should inform them that
> + * it is taking action based on the state of those dependencies.
> + */
> +struct media_job_dep_ops {
> +	bool (*check_dep)(void *data);
> +	void (*clear_dep)(void *data);
> +	void (*reset_dep)(void *data);
> +};
> +
> +/**
> + * struct media_job_dep - Representation of media job dependency
> + *
> + * @list:	List head to attach to a &struct media_job.deps
> + * @ops:	A pointer to the dependency's operations functions
> + * @met:	A flag to record whether or not the dependency is met
> + * @data:	Data to pass to the dependency's operations
> + *
> + * This struct represents a dependency of a media job. The operations member
> + * holds pointers to functions allowing the framework to interact with the
> + * driver to check whether or not the dependency is met.
> + */
> +struct media_job_dep {
> +	struct list_head list;
> +	struct media_job_dep_ops *ops;
> +	bool met;
> +	void *data;
> +};
> +
> +/**
> + * media_jobs_try_queue_job - Try to queue a &struct media_job
> + *
> + * @sched:	Pointer to the job scheduler
> + * @type:	The type of the media job
> + * @dep_ops:	A pointer to the dependency operations for this job
> + * @dep_data:	A pointer to the dependency data for this job
> + *
> + * Try to queue a media job with the scheduler. This function should be called
> + * by the drivers whenever a dependency for a media job is met - for example
> + * when a buffer is queued to the driver. The framework will check to see if an
> + * existing job on the scheduler's pending list shares the same type, dependency
> + * operations and dependency data. If it does then that existing job will be
> + * considered. If there is no extant job with those same parameters, a new job
> + * is allocated and populated by calling the setup functions registered with
> + * the framework.
> + *
> + * The function iterates over the dependencies that are registered with the job
> + * and checks to see if they are met. If they're all met, they're cleared and
> + * the job is placed onto the scheduler's queue.
> + *
> + * To help reduce conditionals in drivers where a driver supports both the use
> + * of the media jobs framework and operation without it, this function is a no
> + * op if @sched is NULL.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
> +			     enum media_job_types type,
> +			     struct media_job_dep_ops *dep_ops, void *dep_data);
> +
> +/**
> + * media_jobs_add_job_step - Add a step to a media job
> + *
> + * @job:	Pointer to the &struct media_job
> + * @run_step:	Pointer to the function to run to execute the step
> + * @data:	Pointer to the data to pass to @run_ste
> + * @flags:	One of the MEDIA_JOBS_FL_STEP_* flags
> + * @pos:	A position indicator to use with @flags
> + *
> + * This function adds a step to the job and should be called from the drivers'
> + * job setup functions as registered with the framework through
> + * media_jobs_add_job_setup_func(). The @flags and @pos parameters are used
> + * to determine the ordering of the steps within the job:
> + *
> + * If @flags has the MEDIA_JOBS_FL_STEP_ANYWHERE bit set, the step is placed
> + * after all steps with MEDIA_JOBS_FL_STEP_FROM_FRONT and before all steps with
> + * MEDIA_JOBS_FL_STEP_FROM_BACK bit set, but otherwise in whatever order this
> + * function is called.
> + *
> + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_FRONT bit set then the step is
> + * placed @pos steps from the front of the list. Attempting to place multiple
> + * steps in the same position will result in an error.
> + *
> + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_BACK bit set then the step is
> + * placed @pos steps from the back of the list. Attempting to place multiple
> + * steps in the same position will result in an error.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
> +			    void *data, unsigned int flags, unsigned int pos);
> +
> +/**
> + * media_jobs_add_job_dep - Add a dependency to a media job
> + *
> + * @job:	Pointer to the &struct media_job
> + * @ops:	Pointer to the &struct media_job_dep_ops
> + * @data:	Pointer to the data to pass to the dependency's operations
> + *
> + * This function adds a dependency to the job and should be called from the
> + * drivers job setup functions as registered with the framework through the
> + * media_jobs_add_job_setup_func() function.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
> +			   void *data);
> +
> +/**
> + * media_jobs_add_job_setup_func - Add a function that populates a media job
> + *
> + * @sched:	Pointer to the media jobs scheduler
> + * @job_setup:	Pointer to the new job setup function
> + * @data:	Data to pass to the job setup function
> + * @type:	The type of job that this function should be called for
> + *
> + * Drivers that wish to utilise the framework need to use this function to
> + * register a callback that adds job steps and dependencies when one is created.
> + * The function must call media_jobs_add_job_step() and media_jobs_add_job_dep()
> + * to populate the job.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
> +				  int (*job_setup)(struct media_job *job, void *data),
> +				  void *data, enum media_job_types type);
> +
> +/**
> + * media_jobs_put_scheduler - Put a reference to the media jobs scheduler
> + *
> + * @sched:	Pointer to the media jobs scheduler
> + *
> + * This function puts a reference to the media jobs scheduler, and is intended
> + * to be called in error and exit paths for consuming drivers
> + */
> +void media_jobs_put_scheduler(struct media_job_scheduler *sched);
> +
> +/**
> + * media_jobs_get_scheduler - Get a media jobs scheduler
> + *
> + * @mdev:	Pointer to the media device associated with the scheduler
> + *
> + * This function gets a pointer to a &struct media_job_scheduler associated with
> + * the media device passed to @mdev. If one is not available then it is
> + * allocated and returned. This allows multiple drivers sharing a media graph to
> + * work with the same media job scheduler.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev);
> +
> +/**
> + * media_jobs_run_jobs - Run any media jobs that are ready in the queue
> + *
> + * @sched:	Pointer to the media job scheduler
> + *
> + * This function triggers the workqueue that processes any jobs that have been
> + * queued, and should be called whenever the pipeline is ready to do so.
> + *
> + * To help reduce conditionals in drivers where a driver supports both the use
> + * of the media jobs framework and operation without it, this function is a no
> + * op if @sched is NULL.
> + */
> +void media_jobs_run_jobs(struct media_job_scheduler *sched);
> +
> +/**
> + * media_jobs_cancel_jobs - cancel all waiting jobs
> + *
> + * @sched:	Pointer to the media job scheduler
> + *
> + * This function iterates over any pending and queued jobs, resets their
> + * dependencies and frees the job
> + *
> + * To help reduce conditionals in drivers where a driver supports both the use
> + * of the media jobs framework and operation without it, this function is a no
> + * op if @sched is NULL.
> + */
> +void media_jobs_cancel_jobs(struct media_job_scheduler *sched);
> +
> +extern struct list_head media_job_schedulers;
> +extern struct mutex media_job_schedulers_lock;
> +
> +#endif /* _MEDIA_JOBS_H */
> --
> 2.34.1
>
>

Re: [PATCH 2/3] media: mc: Add media jobs framework

[Jacopo Mondi]

Hi Dan, back with more questions :)

On Mon, May 19, 2025 at 03:04:02PM +0100, Daniel Scally wrote:
> Add a new framework to the media subsystem describing media jobs.
> This framework is intended to be able to model the interactions
> between multiple different drivers that need to be run in concert
> to fully control a media pipeline, for example an ISP driver and a
> driver controlling a DMA device that feeds data from memory in to
> that ISP.
>
> The new framework allows all drivers involved to add explicit steps
> that need to be performed, and to control the ordering of those steps
> precisely. Once the job with its steps has been created it's then
> scheduled to be run with a workqueue which executes each step in the
> defined order.
>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
>  drivers/media/mc/Makefile  |   2 +-
>  drivers/media/mc/mc-jobs.c | 446 +++++++++++++++++++++++++++++++++++++
>  include/media/media-jobs.h | 354 +++++++++++++++++++++++++++++
>  3 files changed, 801 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/media/mc/mc-jobs.c
>  create mode 100644 include/media/media-jobs.h
>
> diff --git a/drivers/media/mc/Makefile b/drivers/media/mc/Makefile
> index 2b7af42ba59c..9148bbfd1578 100644
> --- a/drivers/media/mc/Makefile
> +++ b/drivers/media/mc/Makefile
> @@ -1,7 +1,7 @@
>  # SPDX-License-Identifier: GPL-2.0
>
>  mc-objs	:= mc-device.o mc-devnode.o mc-entity.o \
> -	   mc-request.o
> +	   mc-jobs.o mc-request.o
>
>  ifneq ($(CONFIG_USB),)
>  	mc-objs += mc-dev-allocator.o
> diff --git a/drivers/media/mc/mc-jobs.c b/drivers/media/mc/mc-jobs.c
> new file mode 100644
> index 000000000000..1f04cdf63d27
> --- /dev/null
> +++ b/drivers/media/mc/mc-jobs.c
> @@ -0,0 +1,446 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Media jobs framework
> + *
> + * Copyright 2025 Ideas on Board Oy
> + *
> + * Author: Daniel Scally <dan.scally@ideasonboard.com>
> + */
> +
> +#include <linux/cleanup.h>
> +#include <linux/kref.h>
> +#include <linux/list.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +
> +#include <media/media-device.h>
> +#include <media/media-entity.h>
> +#include <media/media-jobs.h>
> +
> +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
> +			    void *data, unsigned int flags, unsigned int pos)
> +{
> +	struct media_job_step *step, *tmp;
> +	unsigned int num = flags;
> +	unsigned int count = 0;
> +
> +	guard(spinlock)(&job->lock);
> +
> +	if (!flags) {
> +		WARN_ONCE(1, "%s(): No flag bits set\n", __func__);
> +		return -EINVAL;
> +	}
> +
> +	/* Count the number of set flags; they're mutually exclusive. */
> +	while (num) {
> +		num &= (num - 1);
> +		count++;
> +	}
> +
> +	if (count > 1) {
> +		WARN_ONCE(1, "%s(): Multiple flag bits set\n", __func__);
> +		return -EINVAL;
> +	}
> +
> +	step = kzalloc(sizeof(*step), GFP_KERNEL);
> +	if (!step)
> +		return -ENOMEM;
> +
> +	step->run_step = run_step;
> +	step->data = data;
> +	step->flags = flags;
> +	step->pos = pos;
> +
> +	/*
> +	 * We need to decide where to place the step. If the list is empty that
> +	 * is really easy (and also the later code is much easier if the code is
> +	 * guaranteed not to be empty...)
> +	 */
> +	if (list_empty(&job->steps)) {
> +		list_add_tail(&step->list, &job->steps);
> +		return 0;
> +	}
> +
> +	/*
> +	 * If we've been asked to place it at a specific position from the end
> +	 * of the list, we cycle back through it until either we exhaust the
> +	 * list or find an entry that needs to go further from the back than the
> +	 * new one.
> +	 */
> +	if ((flags & MEDIA_JOBS_FL_STEP_FROM_BACK)) {
> +		list_for_each_entry_reverse(tmp, &job->steps, list) {
> +			if (tmp->flags == flags && tmp->pos == pos)
> +				return -EINVAL;
> +
> +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_BACK ||
> +			    tmp->pos > pos)
> +				break;
> +		}
> +
> +		/*
> +		 * If the entry we broke on is also one placed from the back and
> +		 * should be closer to the back than the new one, we place the
> +		 * new one in front of it...otherwise place the new one behind
> +		 * it.
> +		 */
> +		if (tmp->flags == flags && tmp->pos < pos)
> +			list_add_tail(&step->list, &tmp->list);
> +		else
> +			list_add(&step->list, &tmp->list);
> +
> +		return 0;
> +	}
> +
> +	/*
> +	 * If we've been asked to place it a specific position from the front of
> +	 * the list we do the same kind of operation, but going from the front
> +	 * instead.
> +	 */
> +	if (flags & MEDIA_JOBS_FL_STEP_FROM_FRONT) {
> +		list_for_each_entry(tmp, &job->steps, list) {
> +			if (tmp->flags == flags && tmp->pos == pos)
> +				return -EINVAL;
> +
> +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_FRONT ||
> +			    tmp->pos > pos)
> +				break;
> +		}
> +
> +		/*
> +		 * If the entry we broke on is also placed from the front and
> +		 * should be closed to the front than the new one, we place the
> +		 * new one behind it, otherwise in front of it.
> +		 */
> +		if (tmp->flags == flags && tmp->pos < pos)
> +			list_add(&step->list, &tmp->list);
> +		else
> +			list_add_tail(&step->list, &tmp->list);
> +
> +		return 0;
> +	}
> +
> +	/*
> +	 * If the step is flagged as "can go anywhere" we just need to try to
> +	 * find the first "from the back" entry and add it immediately before
> +	 * that. If we can't find one, add it after whatever we did find.
> +	 */
> +	if (flags & MEDIA_JOBS_FL_STEP_ANYWHERE) {
> +		list_for_each_entry(tmp, &job->steps, list)
> +			if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK))
> +				break;
> +
> +		if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK) ||
> +		    list_entry_is_head(tmp, &job->steps, list))
> +			list_add_tail(&step->list, &tmp->list);
> +		else
> +			list_add(&step->list, &tmp->list);
> +
> +		return 0;
> +	}
> +
> +	/* Shouldn't get here, unless the flag value is wrong. */
> +	WARN_ONCE(1, "%s(): Invalid flag value\n", __func__);
> +	return -EINVAL;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_add_job_step);
> +
> +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
> +			   void *data)
> +{
> +	struct media_job_dep *dep;
> +
> +	if (!ops || !ops->check_dep || !data)
> +		return -EINVAL;
> +
> +	guard(spinlock)(&job->lock);
> +
> +	/* Confirm the same dependency hasn't already been added */
> +	list_for_each_entry(dep, &job->deps, list)
> +		if (dep->ops == ops && dep->data == data)
> +			return -EINVAL;
> +
> +	dep = kzalloc(sizeof(*dep), GFP_KERNEL);
> +	if (!dep)
> +		return -ENOMEM;
> +
> +	dep->ops = ops;
> +	dep->data = data;
> +	list_add(&dep->list, &job->deps);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_add_job_dep);
> +
> +static bool media_jobs_check_pending_job(struct media_job *job,
> +					 enum media_job_types type,
> +					 struct media_job_dep_ops *dep_ops,
> +					 void *data)
> +{
> +	struct media_job_dep *dep;
> +
> +	guard(spinlock)(&job->lock);
> +
> +	if (job->type != type)
> +		return false;
> +
> +	list_for_each_entry(dep, &job->deps, list) {
> +		if (dep->ops == dep_ops && dep->data == data) {
> +			if (dep->met)
> +				return false;
> +
> +			break;
> +		}
> +	}
> +
> +	dep->met = true;
> +	return true;
> +}
> +
> +static struct media_job *media_jobs_get_job(struct media_job_scheduler *sched,
> +					    enum media_job_types type,
> +					    struct media_job_dep_ops *dep_ops,
> +					    void *dep_data)
> +{
> +	struct media_job_setup_func *jsf;
> +	struct media_job *job;
> +	int ret;
> +
> +	list_for_each_entry(job, &sched->pending, list)
> +		if (media_jobs_check_pending_job(job, type, dep_ops, dep_data))
> +			return job;
> +

Thanks to your offline explanation, I got how this works now, however
some questions here

The basic idea is that each driver that registers a 'setup' function
adds to a job, when its created, its list of dependencies.

When a job is "try_queue" and we get here, to decide if a new job has
to be created or if we have to run one which is already in the pending
queue.

How is this identification performed ? Each entry point (assume it's a
video device op) will populate the job with its own dependencies,
identified by the dep_ops and data address.

We walk the 'pending' queue in the media_jobs_check_pending_job()
function and we search for one job not already visited from the same
entry point, identified by the dep_ops (the 'visited' state is kept by
the deps->met flag).

Let's assume 2 video devices X and Y

qbuf(x) -> try_queue_job() -> new job created on 'pending'
qbuf(x) -> try_queue_job() -> the job in the queue has 'deps->met' set, skip
it and create a new one
qbuf(y) -> try_queue_job() -> the first job in the queue has not
deps->set, so return it

All in all I would describe this as: when requesting a job try to find
the first one not already visited by this entry point, if none is
available create a new one.

Now, we briefly discussed that when moving to multi-context comparing
dep_ops and data to identify an entry point won't be enough: buffers
from the same video device but from different contexts do not have to
be associated together. So we'll need to extend the identification
criteria. Also, I don't find the idea of using dep_ops and data for
this purpose particularly neat, as it makes mandatory to add
dependencies to a job in the setup function, something not all driver
might want to do ?

There might be ways to handle this "track the entry point" thing that
could be separated by deps, making deps do what they actually are
described for: track dependencies to validate if a job can be run or
not. Before exploring options, I would like to know if this only mine
concern or is it shared by others.

> +	job = kzalloc(sizeof(*job), GFP_KERNEL);
> +	if (!job)
> +		return ERR_PTR(-ENOMEM);
> +
> +	spin_lock_init(&job->lock);
> +	INIT_LIST_HEAD(&job->deps);
> +	INIT_LIST_HEAD(&job->steps);
> +	job->type = type;
> +	job->sched = sched;
> +
> +	list_for_each_entry(jsf, &sched->setup_funcs, list) {
> +		if (jsf->type != type)
> +			continue;
> +
> +		ret = jsf->job_setup(job, jsf->data);
> +		if (ret) {
> +			kfree(job);
> +			return ERR_PTR(ret);
> +		}
> +	}
> +
> +	list_add_tail(&job->list, &sched->pending);
> +
> +	/* This marks the dependency as met */
> +	media_jobs_check_pending_job(job, type, dep_ops, dep_data);
> +
> +	return job;
> +}
> +
> +static void media_jobs_free_job(struct media_job *job, bool reset)
> +{
> +	struct media_job_step *step, *stmp;
> +	struct media_job_dep *dep, *dtmp;
> +
> +	scoped_guard(spinlock, &job->lock) {
> +		list_for_each_entry_safe(dep, dtmp, &job->deps, list) {
> +			if (reset && dep->ops->reset_dep)
> +				dep->ops->reset_dep(dep->data);
> +
> +			list_del(&dep->list);
> +			kfree(dep);
> +		}
> +
> +		list_for_each_entry_safe(step, stmp, &job->steps, list) {
> +			list_del(&step->list);
> +			kfree(step);
> +		}
> +	}
> +
> +	list_del(&job->list);
> +	kfree(job);
> +}
> +
> +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
> +			     enum media_job_types type,
> +			     struct media_job_dep_ops *dep_ops, void *dep_data)
> +{
> +	struct media_job_dep *dep;
> +	struct media_job *job;
> +
> +	if (!sched)
> +		return 0;
> +
> +	guard(spinlock)(&sched->lock);
> +
> +	job = media_jobs_get_job(sched, type, dep_ops, dep_data);
> +	if (IS_ERR(job))
> +		return PTR_ERR(job);
> +
> +	list_for_each_entry(dep, &job->deps, list)
> +		if (!dep->ops->check_dep(dep->data))
> +			return 0; /* Not a failure */
> +
> +	list_for_each_entry(dep, &job->deps, list)
> +		if (dep->ops->clear_dep)
> +			dep->ops->clear_dep(dep->data);
> +
> +	list_move_tail(&job->list, &sched->queue);
> +	queue_work(sched->async_wq, &sched->work);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_try_queue_job);
> +
> +static void __media_jobs_run_jobs(struct work_struct *work)
> +{
> +	struct media_job_scheduler *sched = container_of(work,
> +							 struct media_job_scheduler,
> +							 work);
> +	struct media_job_step *step;
> +	struct media_job *job;
> +
> +	while (true) {
> +		scoped_guard(spinlock, &sched->lock) {
> +			if (list_empty(&sched->queue))
> +				return;
> +
> +			job = list_first_entry(&sched->queue, struct media_job,
> +					       list);
> +		}
> +
> +		list_for_each_entry(step, &job->steps, list)
> +			step->run_step(step->data);
> +
> +		media_jobs_free_job(job, false);
> +	}
> +}
> +
> +void media_jobs_run_jobs(struct media_job_scheduler *sched)
> +{
> +	if (!sched)
> +		return;
> +
> +	queue_work(sched->async_wq, &sched->work);
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_run_jobs);
> +
> +static void __media_jobs_cancel_jobs(struct media_job_scheduler *sched)
> +{
> +	struct media_job *job, *jtmp;
> +
> +	list_for_each_entry_safe(job, jtmp, &sched->pending, list)
> +		media_jobs_free_job(job, true);
> +
> +	list_for_each_entry_safe(job, jtmp, &sched->queue, list)
> +		media_jobs_free_job(job, true);
> +}
> +
> +void media_jobs_cancel_jobs(struct media_job_scheduler *sched)
> +{
> +	if (!sched)
> +		return;
> +
> +	guard(spinlock)(&sched->lock);
> +	__media_jobs_cancel_jobs(sched);
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_cancel_jobs);
> +
> +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
> +				  int (*job_setup)(struct media_job *job, void *data),
> +				  void *data, enum media_job_types type)
> +{
> +	struct media_job_setup_func *new_setup_func;
> +
> +	guard(spinlock)(&sched->lock);
> +
> +	new_setup_func = kzalloc(sizeof(*new_setup_func), GFP_KERNEL);
> +	if (!new_setup_func)
> +		return -ENOMEM;
> +
> +	new_setup_func->type = type;
> +	new_setup_func->job_setup = job_setup;
> +	new_setup_func->data = data;
> +	list_add_tail(&new_setup_func->list, &sched->setup_funcs);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_add_job_setup_func);
> +
> +static void __media_jobs_put_scheduler(struct kref *kref)
> +{
> +	struct media_job_scheduler *sched =
> +		container_of(kref, struct media_job_scheduler, kref);
> +	struct media_job_setup_func *func, *ftmp;
> +
> +	cancel_work_sync(&sched->work);
> +	destroy_workqueue(sched->async_wq);
> +
> +	scoped_guard(spinlock, &sched->lock) {
> +		__media_jobs_cancel_jobs(sched);
> +
> +		list_for_each_entry_safe(func, ftmp, &sched->setup_funcs, list) {
> +			list_del(&func->list);
> +			kfree(func);
> +		}
> +	}
> +
> +	list_del(&sched->list);
> +	kfree(sched);
> +}
> +
> +void media_jobs_put_scheduler(struct media_job_scheduler *sched)
> +{
> +	kref_put(&sched->kref, __media_jobs_put_scheduler);
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_put_scheduler);
> +
> +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev)
> +{
> +	struct media_job_scheduler *sched;
> +	char workqueue_name[32];
> +	int ret;
> +
> +	guard(mutex)(&media_job_schedulers_lock);
> +
> +	list_for_each_entry(sched, &media_job_schedulers, list) {
> +		if (sched->mdev == mdev) {
> +			kref_get(&sched->kref);
> +			return sched;
> +		}
> +	}
> +
> +	ret = snprintf(workqueue_name, sizeof(workqueue_name),
> +		       "mc jobs (%s)", mdev->driver_name);
> +	if (!ret)
> +		return ERR_PTR(-EINVAL);
> +
> +	sched = kzalloc(sizeof(*sched), GFP_KERNEL);
> +	if (!sched)
> +		return ERR_PTR(-ENOMEM);
> +
> +	sched->async_wq = alloc_workqueue(workqueue_name, 0, 0);
> +	if (!sched->async_wq) {
> +		kfree(sched);
> +		return ERR_PTR(-EINVAL);
> +	}
> +
> +	sched->mdev = mdev;
> +	kref_init(&sched->kref);
> +	spin_lock_init(&sched->lock);
> +	INIT_LIST_HEAD(&sched->setup_funcs);
> +	INIT_LIST_HEAD(&sched->pending);
> +	INIT_LIST_HEAD(&sched->queue);
> +	INIT_WORK(&sched->work, __media_jobs_run_jobs);
> +
> +	list_add_tail(&sched->list, &media_job_schedulers);
> +
> +	return sched;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_get_scheduler);
> +
> +LIST_HEAD(media_job_schedulers);
> +
> +/* Synchronise access to the global schedulers list */
> +DEFINE_MUTEX(media_job_schedulers_lock);
> diff --git a/include/media/media-jobs.h b/include/media/media-jobs.h
> new file mode 100644
> index 000000000000..a97270861251
> --- /dev/null
> +++ b/include/media/media-jobs.h
> @@ -0,0 +1,354 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Media jobs framework
> + *
> + * Copyright 2025 Ideas on Board Oy
> + *
> + * Author: Daniel Scally <dan.scally@ideasonboard.com>
> + */
> +
> +#include <linux/kref.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/spinlock.h>
> +#include <linux/types.h>
> +#include <linux/workqueue.h>
> +
> +#ifndef _MEDIA_JOBS_H
> +#define _MEDIA_JOBS_H
> +
> +struct media_device;
> +struct media_entity;
> +struct media_job;
> +struct media_job_dep;
> +
> +/**
> + * define MEDIA_JOBS_FL_STEP_ANYWHERE - \
> + *    Flag a media job step as able to run anytime
> + *
> + * This flag informs the framework that a job step does not need a particular
> + * position in the list of job steps and can be placed anywhere.
> + */
> +#define MEDIA_JOBS_FL_STEP_ANYWHERE			BIT(0)
> +
> +/**
> + * define MEDIA_JOBS_FL_STEP_FROM_FRONT - \
> + *    Flag a media job step as needing to be placed near the start of the list
> + *
> + * This flag informs the framework that a job step needs to be placed at a set
> + * position from the start of the list of job steps.
> + */
> +#define MEDIA_JOBS_FL_STEP_FROM_FRONT			BIT(1)
> +
> +/**
> + * define MEDIA_JOBS_FL_STEP_FROM_BACK - \
> + *    Flag a media job step as needing to be placed near the end of the list
> + *
> + * This flag informs the framework that a job step needs to be placed at a set
> + * position from the end of the list of job steps.
> + */
> +#define MEDIA_JOBS_FL_STEP_FROM_BACK			BIT(2)
> +
> +/**
> + * enum media_job_types - Type of media job
> + *
> + * @MEDIA_JOB_TYPE_PIPELINE_PULSE:	A data event moving through the media
> + *					pipeline
> + *
> + * This enumeration details different types of media jobs. The type can be used
> + * to differentiate between which steps and dependencies a driver needs to add
> + * to a job when it is created.
> + */
> +enum media_job_types {
> +	MEDIA_JOB_TYPE_PIPELINE_PULSE,
> +};
> +
> +/**
> + * struct media_job_scheduler - A job scheduler for a particular media device
> + *
> + * @mdev:		Media device this scheduler is for
> + * @list:		List head to attach to the global list of schedulers
> + * @kref:		Reference counter
> + * @lock:		Lock to protect access to the scheduler
> + * @setup_funcs:	List of &struct media_job_setup_func to populate jobs
> + * @pending:		List of &struct media_jobs created but not yet queued
> + * @queue:		List of &struct media_jobs queued to the scheduler
> + * @work:		Work item to run the jobs
> + * @async_wq:		Workqueue to run the work on
> + *
> + * This struct is the main job scheduler struct - drivers wanting to use this
> + * framework should acquire an instance through media_jobs_get_scheduler() and
> + * subsequently populate it with job setup functions.
> + */
> +struct media_job_scheduler {
> +	struct media_device *mdev;
> +	struct list_head list;
> +	struct kref kref;
> +
> +	spinlock_t lock; /* Synchronise access to the struct's lists */
> +	struct list_head setup_funcs;
> +	struct list_head pending;
> +	struct list_head queue;
> +	struct work_struct work;
> +	struct workqueue_struct *async_wq;
> +};
> +
> +/**
> + * struct media_job_setup_func - A function to populate a media job with steps
> + *				 and dependencies
> + *
> + * @list:	The list object to attach to the scheduler
> + * @type:	The &enum media_job_types that this function populates a job for
> + * @job_setup:	Function pointer to the driver's job setup function
> + * @data:	Pointer to the driver data for use with @job_setup
> + *
> + * This struct holds data about the functions a driver registers with the jobs
> + * framework in order to populate a new job with steps and dependencies.
> + */
> +struct media_job_setup_func {
> +	struct list_head list;
> +	enum media_job_types type;
> +	int (*job_setup)(struct media_job *job, void *data);
> +	void *data;
> +};
> +
> +/**
> + * struct media_job - A representation of a job to be run through the pipeline
> + *
> + * @lock:	Lock to protect access to the job's lists
> + * @list:	List head to attach the job to &struct media_job_scheduler in
> + *		either the pending or queue lists
> + * @steps:	List of &struct media_job_step to run the job
> + * @deps:	List of &struct media_job_dep to check that the job can be
> + *		queued
> + * @sched:	Pointer to the media job scheduler
> + * @type:	The type of the job
> + *
> + * This struct holds lists of steps that need to be performed to carry out a
> + * job in the pipeline. A separate list of dependencies allows the queueing of
> + * the job to be delayed until all drivers are ready to carry it out.
> + */
> +struct media_job {
> +	spinlock_t lock; /* Synchronise access to the struct's lists 6*/
> +	struct list_head list;
> +	struct list_head steps;
> +	struct list_head deps;
> +	struct media_job_scheduler *sched;
> +	enum media_job_types type;
> +};
> +
> +/**
> + * struct media_job_step - A holder for a function to run as part of a job
> + *
> + * @list:	List head to attach the job step to a &struct media_job.steps
> + * @run_step:	The function to run to perform the step
> + * @data:	Data to pass to the .run_step() function
> + * @flags:	Flags to control how the step is ordered within the job's list
> + *		of steps
> + * @pos:	Position indicator to control how the step is ordered within the
> + *		job's list of steps
> + *
> + * This struct defines a function that needs to be run as part of the execution
> + * of a job in a media pipeline, along with information that help the scheduler
> + * determine what order it should be ran in in reference to the other steps that
> + * are part of the same job.
> + */
> +struct media_job_step {
> +	struct list_head list;
> +	void (*run_step)(void *data);
> +	void *data;
> +	unsigned int flags;
> +	unsigned int pos;
> +};
> +
> +/**
> + * struct media_job_dep_ops - Operations to manage a media job dependency
> + *
> + * @check_dep:	A function to ask the driver whether the dependency is met
> + * @clear_dep:	A function to tell the driver that the job has been queued
> + * @reset_dep:	A function to tell the driver that the job has been cancelled
> + *
> + * Media jobs have dependencies, such as requiring buffers to be queued. These
> + * operations allow a driver to define how the media jobs framework should check
> + * whether or not those dependencies are met and how it should inform them that
> + * it is taking action based on the state of those dependencies.
> + */
> +struct media_job_dep_ops {
> +	bool (*check_dep)(void *data);
> +	void (*clear_dep)(void *data);
> +	void (*reset_dep)(void *data);
> +};
> +
> +/**
> + * struct media_job_dep - Representation of media job dependency
> + *
> + * @list:	List head to attach to a &struct media_job.deps
> + * @ops:	A pointer to the dependency's operations functions
> + * @met:	A flag to record whether or not the dependency is met
> + * @data:	Data to pass to the dependency's operations
> + *
> + * This struct represents a dependency of a media job. The operations member
> + * holds pointers to functions allowing the framework to interact with the
> + * driver to check whether or not the dependency is met.
> + */
> +struct media_job_dep {
> +	struct list_head list;
> +	struct media_job_dep_ops *ops;
> +	bool met;
> +	void *data;
> +};
> +
> +/**
> + * media_jobs_try_queue_job - Try to queue a &struct media_job
> + *
> + * @sched:	Pointer to the job scheduler
> + * @type:	The type of the media job
> + * @dep_ops:	A pointer to the dependency operations for this job
> + * @dep_data:	A pointer to the dependency data for this job
> + *
> + * Try to queue a media job with the scheduler. This function should be called
> + * by the drivers whenever a dependency for a media job is met - for example
> + * when a buffer is queued to the driver. The framework will check to see if an
> + * existing job on the scheduler's pending list shares the same type, dependency
> + * operations and dependency data. If it does then that existing job will be
> + * considered. If there is no extant job with those same parameters, a new job
> + * is allocated and populated by calling the setup functions registered with
> + * the framework.
> + *
> + * The function iterates over the dependencies that are registered with the job
> + * and checks to see if they are met. If they're all met, they're cleared and
> + * the job is placed onto the scheduler's queue.
> + *
> + * To help reduce conditionals in drivers where a driver supports both the use
> + * of the media jobs framework and operation without it, this function is a no
> + * op if @sched is NULL.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
> +			     enum media_job_types type,
> +			     struct media_job_dep_ops *dep_ops, void *dep_data);
> +
> +/**
> + * media_jobs_add_job_step - Add a step to a media job
> + *
> + * @job:	Pointer to the &struct media_job
> + * @run_step:	Pointer to the function to run to execute the step
> + * @data:	Pointer to the data to pass to @run_ste
> + * @flags:	One of the MEDIA_JOBS_FL_STEP_* flags
> + * @pos:	A position indicator to use with @flags
> + *
> + * This function adds a step to the job and should be called from the drivers'
> + * job setup functions as registered with the framework through
> + * media_jobs_add_job_setup_func(). The @flags and @pos parameters are used
> + * to determine the ordering of the steps within the job:
> + *
> + * If @flags has the MEDIA_JOBS_FL_STEP_ANYWHERE bit set, the step is placed
> + * after all steps with MEDIA_JOBS_FL_STEP_FROM_FRONT and before all steps with
> + * MEDIA_JOBS_FL_STEP_FROM_BACK bit set, but otherwise in whatever order this
> + * function is called.
> + *
> + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_FRONT bit set then the step is
> + * placed @pos steps from the front of the list. Attempting to place multiple
> + * steps in the same position will result in an error.
> + *
> + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_BACK bit set then the step is
> + * placed @pos steps from the back of the list. Attempting to place multiple
> + * steps in the same position will result in an error.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
> +			    void *data, unsigned int flags, unsigned int pos);
> +
> +/**
> + * media_jobs_add_job_dep - Add a dependency to a media job
> + *
> + * @job:	Pointer to the &struct media_job
> + * @ops:	Pointer to the &struct media_job_dep_ops
> + * @data:	Pointer to the data to pass to the dependency's operations
> + *
> + * This function adds a dependency to the job and should be called from the
> + * drivers job setup functions as registered with the framework through the
> + * media_jobs_add_job_setup_func() function.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
> +			   void *data);
> +
> +/**
> + * media_jobs_add_job_setup_func - Add a function that populates a media job
> + *
> + * @sched:	Pointer to the media jobs scheduler
> + * @job_setup:	Pointer to the new job setup function
> + * @data:	Data to pass to the job setup function
> + * @type:	The type of job that this function should be called for
> + *
> + * Drivers that wish to utilise the framework need to use this function to
> + * register a callback that adds job steps and dependencies when one is created.
> + * The function must call media_jobs_add_job_step() and media_jobs_add_job_dep()
> + * to populate the job.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
> +				  int (*job_setup)(struct media_job *job, void *data),
> +				  void *data, enum media_job_types type);
> +
> +/**
> + * media_jobs_put_scheduler - Put a reference to the media jobs scheduler
> + *
> + * @sched:	Pointer to the media jobs scheduler
> + *
> + * This function puts a reference to the media jobs scheduler, and is intended
> + * to be called in error and exit paths for consuming drivers
> + */
> +void media_jobs_put_scheduler(struct media_job_scheduler *sched);
> +
> +/**
> + * media_jobs_get_scheduler - Get a media jobs scheduler
> + *
> + * @mdev:	Pointer to the media device associated with the scheduler
> + *
> + * This function gets a pointer to a &struct media_job_scheduler associated with
> + * the media device passed to @mdev. If one is not available then it is
> + * allocated and returned. This allows multiple drivers sharing a media graph to
> + * work with the same media job scheduler.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev);
> +
> +/**
> + * media_jobs_run_jobs - Run any media jobs that are ready in the queue
> + *
> + * @sched:	Pointer to the media job scheduler
> + *
> + * This function triggers the workqueue that processes any jobs that have been
> + * queued, and should be called whenever the pipeline is ready to do so.
> + *
> + * To help reduce conditionals in drivers where a driver supports both the use
> + * of the media jobs framework and operation without it, this function is a no
> + * op if @sched is NULL.
> + */
> +void media_jobs_run_jobs(struct media_job_scheduler *sched);
> +
> +/**
> + * media_jobs_cancel_jobs - cancel all waiting jobs
> + *
> + * @sched:	Pointer to the media job scheduler
> + *
> + * This function iterates over any pending and queued jobs, resets their
> + * dependencies and frees the job
> + *
> + * To help reduce conditionals in drivers where a driver supports both the use
> + * of the media jobs framework and operation without it, this function is a no
> + * op if @sched is NULL.
> + */
> +void media_jobs_cancel_jobs(struct media_job_scheduler *sched);
> +
> +extern struct list_head media_job_schedulers;
> +extern struct mutex media_job_schedulers_lock;
> +
> +#endif /* _MEDIA_JOBS_H */
> --
> 2.34.1
>
>

Re: [PATCH 2/3] media: mc: Add media jobs framework

[Dan Scally]

Hi Jacopo - thanks a lot for the review

On 21/05/2025 19:10, Jacopo Mondi wrote:
> Hi Dan,
>     just a few notes here and there, I will have to check users in more
> detail to comment on the interface
>
> On Mon, May 19, 2025 at 03:04:02PM +0100, Daniel Scally wrote:
>> Add a new framework to the media subsystem describing media jobs.
>> This framework is intended to be able to model the interactions
>> between multiple different drivers that need to be run in concert
>> to fully control a media pipeline, for example an ISP driver and a
>> driver controlling a DMA device that feeds data from memory in to
>> that ISP.
>>
>> The new framework allows all drivers involved to add explicit steps
>> that need to be performed, and to control the ordering of those steps
>> precisely. Once the job with its steps has been created it's then
>> scheduled to be run with a workqueue which executes each step in the
>> defined order.
>>
>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>> ---
>>   drivers/media/mc/Makefile  |   2 +-
>>   drivers/media/mc/mc-jobs.c | 446 +++++++++++++++++++++++++++++++++++++
>>   include/media/media-jobs.h | 354 +++++++++++++++++++++++++++++
>>   3 files changed, 801 insertions(+), 1 deletion(-)
>>   create mode 100644 drivers/media/mc/mc-jobs.c
>>   create mode 100644 include/media/media-jobs.h
>>
>> diff --git a/drivers/media/mc/Makefile b/drivers/media/mc/Makefile
>> index 2b7af42ba59c..9148bbfd1578 100644
>> --- a/drivers/media/mc/Makefile
>> +++ b/drivers/media/mc/Makefile
>> @@ -1,7 +1,7 @@
>>   # SPDX-License-Identifier: GPL-2.0
>>
>>   mc-objs	:= mc-device.o mc-devnode.o mc-entity.o \
>> -	   mc-request.o
>> +	   mc-jobs.o mc-request.o
>>
>>   ifneq ($(CONFIG_USB),)
>>   	mc-objs += mc-dev-allocator.o
>> diff --git a/drivers/media/mc/mc-jobs.c b/drivers/media/mc/mc-jobs.c
>> new file mode 100644
>> index 000000000000..1f04cdf63d27
>> --- /dev/null
>> +++ b/drivers/media/mc/mc-jobs.c
>> @@ -0,0 +1,446 @@
>> +// SPDX-License-Identifier: GPL-2.0-only
>> +/*
>> + * Media jobs framework
>> + *
>> + * Copyright 2025 Ideas on Board Oy
>> + *
>> + * Author: Daniel Scally <dan.scally@ideasonboard.com>
>> + */
>> +
>> +#include <linux/cleanup.h>
>> +#include <linux/kref.h>
>> +#include <linux/list.h>
>> +#include <linux/slab.h>
>> +#include <linux/spinlock.h>
> Some of these are included by the header file


Ack

>
>> +
>> +#include <media/media-device.h>
>> +#include <media/media-entity.h>
>> +#include <media/media-jobs.h>
>> +
>> +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
>> +			    void *data, unsigned int flags, unsigned int pos)
>> +{
>> +	struct media_job_step *step, *tmp;
>> +	unsigned int num = flags;
>> +	unsigned int count = 0;
>> +
>> +	guard(spinlock)(&job->lock);
>> +
>> +	if (!flags) {
>> +		WARN_ONCE(1, "%s(): No flag bits set\n", __func__);
>> +		return -EINVAL;
>> +	}
> You could check this before taking the lock


Good point

>
>> +
>> +	/* Count the number of set flags; they're mutually exclusive. */
> if you want just to count flags you can use hweightX from bitops.h


I knew there must be a better way of doing this - thanks!

>
>> +	while (num) {
>> +		num &= (num - 1);
>> +		count++;
>> +	}
>> +
>> +	if (count > 1) {
>> +		WARN_ONCE(1, "%s(): Multiple flag bits set\n", __func__);
>> +		return -EINVAL;
>> +	}
>> +
>> +	step = kzalloc(sizeof(*step), GFP_KERNEL);
>> +	if (!step)
>> +		return -ENOMEM;
>> +
>> +	step->run_step = run_step;
>> +	step->data = data;
>> +	step->flags = flags;
>> +	step->pos = pos;
>> +
>> +	/*
>> +	 * We need to decide where to place the step. If the list is empty that
>> +	 * is really easy (and also the later code is much easier if the code is
>> +	 * guaranteed not to be empty...)
>> +	 */
>> +	if (list_empty(&job->steps)) {
>> +		list_add_tail(&step->list, &job->steps);
>> +		return 0;
>> +	}
>> +
>> +	/*
>> +	 * If we've been asked to place it at a specific position from the end
>> +	 * of the list, we cycle back through it until either we exhaust the
>> +	 * list or find an entry that needs to go further from the back than the
>> +	 * new one.
>> +	 */
>> +	if ((flags & MEDIA_JOBS_FL_STEP_FROM_BACK)) {
>> +		list_for_each_entry_reverse(tmp, &job->steps, list) {
>> +			if (tmp->flags == flags && tmp->pos == pos)
>> +				return -EINVAL;
>> +
>> +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_BACK ||
>> +			    tmp->pos > pos)
>> +				break;
>> +		}
>> +
>> +		/*
>> +		 * If the entry we broke on is also one placed from the back and
>> +		 * should be closer to the back than the new one, we place the
>> +		 * new one in front of it...otherwise place the new one behind
>> +		 * it.
>> +		 */
>> +		if (tmp->flags == flags && tmp->pos < pos)
>> +			list_add_tail(&step->list, &tmp->list);
>> +		else
>> +			list_add(&step->list, &tmp->list);
>> +
>> +		return 0;
>> +	}
>> +
>> +	/*
>> +	 * If we've been asked to place it a specific position from the front of
>> +	 * the list we do the same kind of operation, but going from the front
>> +	 * instead.
>> +	 */
>> +	if (flags & MEDIA_JOBS_FL_STEP_FROM_FRONT) {
>> +		list_for_each_entry(tmp, &job->steps, list) {
>> +			if (tmp->flags == flags && tmp->pos == pos)
>> +				return -EINVAL;
>> +
>> +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_FRONT ||
>> +			    tmp->pos > pos)
>> +				break;
>> +		}
>> +
>> +		/*
>> +		 * If the entry we broke on is also placed from the front and
>> +		 * should be closed to the front than the new one, we place the
>> +		 * new one behind it, otherwise in front of it.
>> +		 */
>> +		if (tmp->flags == flags && tmp->pos < pos)
>> +			list_add(&step->list, &tmp->list);
>> +		else
>> +			list_add_tail(&step->list, &tmp->list);
>> +
>> +		return 0;
>> +	}
> There probably is room to factorize out the common code parts from
> here
Agreed, and indeed it was shorter in an earlier iteration, but I found it quite difficult to follow 
what was going on so in the end rewrote it like this to make the code easier to understand and 
debug, which certainly helped during development. I think there's value in that that's worth the 
extra length but I don't feel too strongly about it, so I'm happy to shorten it if the consensus is 
that that's better.
>
>> +
>> +	/*
>> +	 * If the step is flagged as "can go anywhere" we just need to try to
>> +	 * find the first "from the back" entry and add it immediately before
>> +	 * that. If we can't find one, add it after whatever we did find.
>> +	 */
>> +	if (flags & MEDIA_JOBS_FL_STEP_ANYWHERE) {
>> +		list_for_each_entry(tmp, &job->steps, list)
>> +			if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK))
>> +				break;
>> +
>> +		if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK) ||
>> +		    list_entry_is_head(tmp, &job->steps, list))
>> +			list_add_tail(&step->list, &tmp->list);
>> +		else
>> +			list_add(&step->list, &tmp->list);
>> +
>> +		return 0;
>> +	}
>> +
>> +	/* Shouldn't get here, unless the flag value is wrong. */
>> +	WARN_ONCE(1, "%s(): Invalid flag value\n", __func__);
>> +	return -EINVAL;
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_add_job_step);
>> +
>> +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
>> +			   void *data)
>> +{
>> +	struct media_job_dep *dep;
>> +
>> +	if (!ops || !ops->check_dep || !data)
>> +		return -EINVAL;
>> +
>> +	guard(spinlock)(&job->lock);
>> +
>> +	/* Confirm the same dependency hasn't already been added */
>> +	list_for_each_entry(dep, &job->deps, list)
>> +		if (dep->ops == ops && dep->data == data)
>> +			return -EINVAL;
>> +
>> +	dep = kzalloc(sizeof(*dep), GFP_KERNEL);
>> +	if (!dep)
>> +		return -ENOMEM;
>> +
>> +	dep->ops = ops;
>> +	dep->data = data;
>> +	list_add(&dep->list, &job->deps);
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_add_job_dep);
>> +
>> +static bool media_jobs_check_pending_job(struct media_job *job,
>> +					 enum media_job_types type,
>> +					 struct media_job_dep_ops *dep_ops,
>> +					 void *data)
>> +{
>> +	struct media_job_dep *dep;
>> +
>> +	guard(spinlock)(&job->lock);
>> +
>> +	if (job->type != type)
>> +		return false;
> possibily moved outside of the lock as well
Ack
>
>> +
>> +	list_for_each_entry(dep, &job->deps, list) {
>> +		if (dep->ops == dep_ops && dep->data == data) {
>> +			if (dep->met)
>> +				return false;
>> +
>> +			break;
>> +		}
>> +	}
>> +
>> +	dep->met = true;
>> +	return true;
>> +}
>> +
>> +static struct media_job *media_jobs_get_job(struct media_job_scheduler *sched,
>> +					    enum media_job_types type,
>> +					    struct media_job_dep_ops *dep_ops,
>> +					    void *dep_data)
>> +{
>> +	struct media_job_setup_func *jsf;
>> +	struct media_job *job;
>> +	int ret;
>> +
>> +	list_for_each_entry(job, &sched->pending, list)
>> +		if (media_jobs_check_pending_job(job, type, dep_ops, dep_data))
>> +			return job;
> Not sure I got this part.
>
> media_jobs_get_job() has a single caller, and it either returns a
> pending job or allocates a new one.
>
> To check if a pending job should be returned you try to math the
> dep_ops and dep_data and if you get one that matches you identify the
> job and return it, but only once, as once a dependency is met it will
> then invalidate the search. What am I missing ?


Nothing - that's right. So the point is that in the typical use case the dependencies might be 
buffers queued to a video device, and in the mali-c55 case there's at least the stats, params, 
capture and output devices. The drivers call media_job_try_queue_job() in .buf_queue() for the video 
devices, which then checks the list of pending jobs to see if it has one already allocated for which 
this dependency is not yet met, and if so returns that. The next time .buf_queue() is called for the 
same video device though we don't want to return that same pending job, so instead a new one is 
allocated.

>
>> +
>> +	job = kzalloc(sizeof(*job), GFP_KERNEL);
>> +	if (!job)
>> +		return ERR_PTR(-ENOMEM);
>> +
>> +	spin_lock_init(&job->lock);
>> +	INIT_LIST_HEAD(&job->deps);
>> +	INIT_LIST_HEAD(&job->steps);
>> +	job->type = type;
>> +	job->sched = sched;
>> +
>> +	list_for_each_entry(jsf, &sched->setup_funcs, list) {
>> +		if (jsf->type != type)
>> +			continue;
>> +
>> +		ret = jsf->job_setup(job, jsf->data);
>> +		if (ret) {
>> +			kfree(job);
> do we need an operation to undo job_setup() or do you think
> it's not needed ?


It's done by media_jobs_free_job()

>
>> +			return ERR_PTR(ret);
>> +		}
>> +	}
>> +
>> +	list_add_tail(&job->list, &sched->pending);
>> +
>> +	/* This marks the dependency as met */
>> +	media_jobs_check_pending_job(job, type, dep_ops, dep_data);
>> +
>> +	return job;
>> +}
>> +
>> +static void media_jobs_free_job(struct media_job *job, bool reset)
>> +{
>> +	struct media_job_step *step, *stmp;
>> +	struct media_job_dep *dep, *dtmp;
>> +
>> +	scoped_guard(spinlock, &job->lock) {
>> +		list_for_each_entry_safe(dep, dtmp, &job->deps, list) {
>> +			if (reset && dep->ops->reset_dep)
>> +				dep->ops->reset_dep(dep->data);
>> +
>> +			list_del(&dep->list);
>> +			kfree(dep);
>> +		}
>> +
>> +		list_for_each_entry_safe(step, stmp, &job->steps, list) {
>> +			list_del(&step->list);
>> +			kfree(step);
>> +		}
>> +	}
>> +
>> +	list_del(&job->list);
> Probably better to be done while holding the sched spinlock that
> protects the jobs list ?
Hmmmm yes. sched->lock is held for one callsite but not the other...but it probably should be. I'll 
look into why I specifically didn't include it in the locked portion there, thanks.
>
>
>> +	kfree(job);
>> +}
>> +
>> +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
>> +			     enum media_job_types type,
>> +			     struct media_job_dep_ops *dep_ops, void *dep_data)
>> +{
>> +	struct media_job_dep *dep;
>> +	struct media_job *job;
>> +
>> +	if (!sched)
> If this happens, I guess something went very wrong. Should we
> WARN_ONCE() to catch this during development ?


This is intentional actually, because there's situations in which a driver might not want the 
framework to take effect. The C55 driver for example might be working in inline mode rather than 
memory input mode, in which case it's not needed...the point of making them no-ops was to reduce the 
amount of "if (!inline)..." stuff in the drivers...perhaps a debug statement here to flag it?

>
>> +		return 0;
>> +
>> +	guard(spinlock)(&sched->lock);
>> +
>> +	job = media_jobs_get_job(sched, type, dep_ops, dep_data);
>> +	if (IS_ERR(job))
>> +		return PTR_ERR(job);
>> +
>> +	list_for_each_entry(dep, &job->deps, list)
>> +		if (!dep->ops->check_dep(dep->data))
> Should you check if dep->ops->check_dep() is valid before calling it ?
I guess so to be safe, though it is checked in media_jobs_add_job_dep()
>
>> +			return 0; /* Not a failure */
>> +
>> +	list_for_each_entry(dep, &job->deps, list)
>> +		if (dep->ops->clear_dep)
>> +			dep->ops->clear_dep(dep->data);
>> +
>> +	list_move_tail(&job->list, &sched->queue);
>> +	queue_work(sched->async_wq, &sched->work);
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_try_queue_job);
>> +
>> +static void __media_jobs_run_jobs(struct work_struct *work)
>> +{
>> +	struct media_job_scheduler *sched = container_of(work,
>> +							 struct media_job_scheduler,
>> +							 work);
>> +	struct media_job_step *step;
>> +	struct media_job *job;
>> +
>> +	while (true) {
>> +		scoped_guard(spinlock, &sched->lock) {
>> +			if (list_empty(&sched->queue))
>> +				return;
>> +
>> +			job = list_first_entry(&sched->queue, struct media_job,
>> +					       list);
>> +		}
>> +
>> +		list_for_each_entry(step, &job->steps, list)
> I take this as step->run_step() has been validated
Hmm this actually **isn't** validated in media_jobs_add_job_step(). Better fix that, and I can add a 
guard here yes.
>
>> +			step->run_step(step->data);
>> +
>> +		media_jobs_free_job(job, false);
>> +	}
>> +}
>> +
>> +void media_jobs_run_jobs(struct media_job_scheduler *sched)
>> +{
>> +	if (!sched)
>> +		return;
>> +
>> +	queue_work(sched->async_wq, &sched->work);
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_run_jobs);
>> +
>> +static void __media_jobs_cancel_jobs(struct media_job_scheduler *sched)
>> +{
>> +	struct media_job *job, *jtmp;
>> +
> if you want to make sure a lock is held when calling a function you
> could use lockdep_assert_held
>
>> +	list_for_each_entry_safe(job, jtmp, &sched->pending, list)
>> +		media_jobs_free_job(job, true);
>> +
>> +	list_for_each_entry_safe(job, jtmp, &sched->queue, list)
>> +		media_jobs_free_job(job, true);
>> +}
>> +
>> +void media_jobs_cancel_jobs(struct media_job_scheduler *sched)
>> +{
>> +	if (!sched)
>> +		return;
>> +
>> +	guard(spinlock)(&sched->lock);
>> +	__media_jobs_cancel_jobs(sched);
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_cancel_jobs);
>> +
>> +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
>> +				  int (*job_setup)(struct media_job *job, void *data),
>> +				  void *data, enum media_job_types type)
>> +{
>> +	struct media_job_setup_func *new_setup_func;
>> +
>> +	guard(spinlock)(&sched->lock);
>> +
>> +	new_setup_func = kzalloc(sizeof(*new_setup_func), GFP_KERNEL);
>> +	if (!new_setup_func)
>> +		return -ENOMEM;
>> +
>> +	new_setup_func->type = type;
>> +	new_setup_func->job_setup = job_setup;
>> +	new_setup_func->data = data;
>> +	list_add_tail(&new_setup_func->list, &sched->setup_funcs);
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_add_job_setup_func);
>> +
>> +static void __media_jobs_put_scheduler(struct kref *kref)
>> +{
>> +	struct media_job_scheduler *sched =
>> +		container_of(kref, struct media_job_scheduler, kref);
>> +	struct media_job_setup_func *func, *ftmp;
>> +
>> +	cancel_work_sync(&sched->work);
>> +	destroy_workqueue(sched->async_wq);
>> +
>> +	scoped_guard(spinlock, &sched->lock) {
>> +		__media_jobs_cancel_jobs(sched);
>> +
>> +		list_for_each_entry_safe(func, ftmp, &sched->setup_funcs, list) {
>> +			list_del(&func->list);
>> +			kfree(func);
>> +		}
>> +	}
>> +
>> +	list_del(&sched->list);
>> +	kfree(sched);
>> +}
>> +
>> +void media_jobs_put_scheduler(struct media_job_scheduler *sched)
>> +{
>> +	kref_put(&sched->kref, __media_jobs_put_scheduler);
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_put_scheduler);
>> +
>> +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev)
>> +{
>> +	struct media_job_scheduler *sched;
>> +	char workqueue_name[32];
>> +	int ret;
>> +
>> +	guard(mutex)(&media_job_schedulers_lock);
>> +
>> +	list_for_each_entry(sched, &media_job_schedulers, list) {
>> +		if (sched->mdev == mdev) {
>> +			kref_get(&sched->kref);
>> +			return sched;
>> +		}
>> +	}
> Ok big question here: why are we keeping a static list of schedulers
> if a single scheduler can be associated to an mdev ?
>
> Thinking forward a bit here: once we have multiple contexts, each
> context should have its own scheduler ? Or are planning to a global
> scheduler per single media device ?


I would still expect a single scheduler per media device even in a multi context situation - the 
thinking behind the complete separation was that it would be easier to compartmentalise...but 
there's no functional reason, it could just be embedded into a media device if that's a better approach.

>
>> +
>> +	ret = snprintf(workqueue_name, sizeof(workqueue_name),
>> +		       "mc jobs (%s)", mdev->driver_name);
>> +	if (!ret)
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	sched = kzalloc(sizeof(*sched), GFP_KERNEL);
>> +	if (!sched)
>> +		return ERR_PTR(-ENOMEM);
>> +
>> +	sched->async_wq = alloc_workqueue(workqueue_name, 0, 0);
>> +	if (!sched->async_wq) {
>> +		kfree(sched);
>> +		return ERR_PTR(-EINVAL);
>> +	}
>> +
>> +	sched->mdev = mdev;
>> +	kref_init(&sched->kref);
>> +	spin_lock_init(&sched->lock);
>> +	INIT_LIST_HEAD(&sched->setup_funcs);
>> +	INIT_LIST_HEAD(&sched->pending);
>> +	INIT_LIST_HEAD(&sched->queue);
>> +	INIT_WORK(&sched->work, __media_jobs_run_jobs);
>> +
>> +	list_add_tail(&sched->list, &media_job_schedulers);
>> +
>> +	return sched;
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_get_scheduler);
>> +
>> +LIST_HEAD(media_job_schedulers);
>> +
>> +/* Synchronise access to the global schedulers list */
>> +DEFINE_MUTEX(media_job_schedulers_lock);
> Still, even with a single scheduler per media device, I wonder why the
> scheduler cannot live in the mdev itself...
>
> I'll have a more detailed look at the users and I'll get back on the
> interface.
>
> In the meantime, I wonder if an introductory documentation page that
> explains why direct function calls are not always possible in drivers
> that interact in a pipeline and what the use case for media_job is,
> wouldn't help people getting familiar with it quicker.


I hope that the documentation patch will cover that, but perhaps it should come first in the series then

> Thanks
>     j
>
>> diff --git a/include/media/media-jobs.h b/include/media/media-jobs.h
>> new file mode 100644
>> index 000000000000..a97270861251
>> --- /dev/null
>> +++ b/include/media/media-jobs.h
>> @@ -0,0 +1,354 @@
>> +/* SPDX-License-Identifier: GPL-2.0-only */
>> +/*
>> + * Media jobs framework
>> + *
>> + * Copyright 2025 Ideas on Board Oy
>> + *
>> + * Author: Daniel Scally <dan.scally@ideasonboard.com>
>> + */
>> +
>> +#include <linux/kref.h>
>> +#include <linux/list.h>
>> +#include <linux/mutex.h>
>> +#include <linux/spinlock.h>
>> +#include <linux/types.h>
>> +#include <linux/workqueue.h>
>> +
>> +#ifndef _MEDIA_JOBS_H
>> +#define _MEDIA_JOBS_H
>> +
>> +struct media_device;
>> +struct media_entity;
>> +struct media_job;
>> +struct media_job_dep;
>> +
>> +/**
>> + * define MEDIA_JOBS_FL_STEP_ANYWHERE - \
>> + *    Flag a media job step as able to run anytime
>> + *
>> + * This flag informs the framework that a job step does not need a particular
>> + * position in the list of job steps and can be placed anywhere.
>> + */
>> +#define MEDIA_JOBS_FL_STEP_ANYWHERE			BIT(0)
>> +
>> +/**
>> + * define MEDIA_JOBS_FL_STEP_FROM_FRONT - \
>> + *    Flag a media job step as needing to be placed near the start of the list
>> + *
>> + * This flag informs the framework that a job step needs to be placed at a set
>> + * position from the start of the list of job steps.
>> + */
>> +#define MEDIA_JOBS_FL_STEP_FROM_FRONT			BIT(1)
>> +
>> +/**
>> + * define MEDIA_JOBS_FL_STEP_FROM_BACK - \
>> + *    Flag a media job step as needing to be placed near the end of the list
>> + *
>> + * This flag informs the framework that a job step needs to be placed at a set
>> + * position from the end of the list of job steps.
>> + */
>> +#define MEDIA_JOBS_FL_STEP_FROM_BACK			BIT(2)
>> +
>> +/**
>> + * enum media_job_types - Type of media job
>> + *
>> + * @MEDIA_JOB_TYPE_PIPELINE_PULSE:	A data event moving through the media
>> + *					pipeline
>> + *
>> + * This enumeration details different types of media jobs. The type can be used
>> + * to differentiate between which steps and dependencies a driver needs to add
>> + * to a job when it is created.
>> + */
>> +enum media_job_types {
>> +	MEDIA_JOB_TYPE_PIPELINE_PULSE,
>> +};
>> +
>> +/**
>> + * struct media_job_scheduler - A job scheduler for a particular media device
>> + *
>> + * @mdev:		Media device this scheduler is for
>> + * @list:		List head to attach to the global list of schedulers
>> + * @kref:		Reference counter
>> + * @lock:		Lock to protect access to the scheduler
>> + * @setup_funcs:	List of &struct media_job_setup_func to populate jobs
>> + * @pending:		List of &struct media_jobs created but not yet queued
>> + * @queue:		List of &struct media_jobs queued to the scheduler
>> + * @work:		Work item to run the jobs
>> + * @async_wq:		Workqueue to run the work on
>> + *
>> + * This struct is the main job scheduler struct - drivers wanting to use this
>> + * framework should acquire an instance through media_jobs_get_scheduler() and
>> + * subsequently populate it with job setup functions.
>> + */
>> +struct media_job_scheduler {
>> +	struct media_device *mdev;
>> +	struct list_head list;
>> +	struct kref kref;
>> +
>> +	spinlock_t lock; /* Synchronise access to the struct's lists */
>> +	struct list_head setup_funcs;
>> +	struct list_head pending;
>> +	struct list_head queue;
>> +	struct work_struct work;
>> +	struct workqueue_struct *async_wq;
>> +};
>> +
>> +/**
>> + * struct media_job_setup_func - A function to populate a media job with steps
>> + *				 and dependencies
>> + *
>> + * @list:	The list object to attach to the scheduler
>> + * @type:	The &enum media_job_types that this function populates a job for
>> + * @job_setup:	Function pointer to the driver's job setup function
>> + * @data:	Pointer to the driver data for use with @job_setup
>> + *
>> + * This struct holds data about the functions a driver registers with the jobs
>> + * framework in order to populate a new job with steps and dependencies.
>> + */
>> +struct media_job_setup_func {
>> +	struct list_head list;
>> +	enum media_job_types type;
>> +	int (*job_setup)(struct media_job *job, void *data);
>> +	void *data;
>> +};
>> +
>> +/**
>> + * struct media_job - A representation of a job to be run through the pipeline
>> + *
>> + * @lock:	Lock to protect access to the job's lists
>> + * @list:	List head to attach the job to &struct media_job_scheduler in
>> + *		either the pending or queue lists
>> + * @steps:	List of &struct media_job_step to run the job
>> + * @deps:	List of &struct media_job_dep to check that the job can be
>> + *		queued
>> + * @sched:	Pointer to the media job scheduler
>> + * @type:	The type of the job
>> + *
>> + * This struct holds lists of steps that need to be performed to carry out a
>> + * job in the pipeline. A separate list of dependencies allows the queueing of
>> + * the job to be delayed until all drivers are ready to carry it out.
>> + */
>> +struct media_job {
>> +	spinlock_t lock; /* Synchronise access to the struct's lists 6*/
>> +	struct list_head list;
>> +	struct list_head steps;
>> +	struct list_head deps;
>> +	struct media_job_scheduler *sched;
>> +	enum media_job_types type;
>> +};
>> +
>> +/**
>> + * struct media_job_step - A holder for a function to run as part of a job
>> + *
>> + * @list:	List head to attach the job step to a &struct media_job.steps
>> + * @run_step:	The function to run to perform the step
>> + * @data:	Data to pass to the .run_step() function
>> + * @flags:	Flags to control how the step is ordered within the job's list
>> + *		of steps
>> + * @pos:	Position indicator to control how the step is ordered within the
>> + *		job's list of steps
>> + *
>> + * This struct defines a function that needs to be run as part of the execution
>> + * of a job in a media pipeline, along with information that help the scheduler
>> + * determine what order it should be ran in in reference to the other steps that
>> + * are part of the same job.
>> + */
>> +struct media_job_step {
>> +	struct list_head list;
>> +	void (*run_step)(void *data);
>> +	void *data;
>> +	unsigned int flags;
>> +	unsigned int pos;
>> +};
>> +
>> +/**
>> + * struct media_job_dep_ops - Operations to manage a media job dependency
>> + *
>> + * @check_dep:	A function to ask the driver whether the dependency is met
>> + * @clear_dep:	A function to tell the driver that the job has been queued
>> + * @reset_dep:	A function to tell the driver that the job has been cancelled
>> + *
>> + * Media jobs have dependencies, such as requiring buffers to be queued. These
>> + * operations allow a driver to define how the media jobs framework should check
>> + * whether or not those dependencies are met and how it should inform them that
>> + * it is taking action based on the state of those dependencies.
>> + */
>> +struct media_job_dep_ops {
>> +	bool (*check_dep)(void *data);
>> +	void (*clear_dep)(void *data);
>> +	void (*reset_dep)(void *data);
>> +};
>> +
>> +/**
>> + * struct media_job_dep - Representation of media job dependency
>> + *
>> + * @list:	List head to attach to a &struct media_job.deps
>> + * @ops:	A pointer to the dependency's operations functions
>> + * @met:	A flag to record whether or not the dependency is met
>> + * @data:	Data to pass to the dependency's operations
>> + *
>> + * This struct represents a dependency of a media job. The operations member
>> + * holds pointers to functions allowing the framework to interact with the
>> + * driver to check whether or not the dependency is met.
>> + */
>> +struct media_job_dep {
>> +	struct list_head list;
>> +	struct media_job_dep_ops *ops;
>> +	bool met;
>> +	void *data;
>> +};
>> +
>> +/**
>> + * media_jobs_try_queue_job - Try to queue a &struct media_job
>> + *
>> + * @sched:	Pointer to the job scheduler
>> + * @type:	The type of the media job
>> + * @dep_ops:	A pointer to the dependency operations for this job
>> + * @dep_data:	A pointer to the dependency data for this job
>> + *
>> + * Try to queue a media job with the scheduler. This function should be called
>> + * by the drivers whenever a dependency for a media job is met - for example
>> + * when a buffer is queued to the driver. The framework will check to see if an
>> + * existing job on the scheduler's pending list shares the same type, dependency
>> + * operations and dependency data. If it does then that existing job will be
>> + * considered. If there is no extant job with those same parameters, a new job
>> + * is allocated and populated by calling the setup functions registered with
>> + * the framework.
>> + *
>> + * The function iterates over the dependencies that are registered with the job
>> + * and checks to see if they are met. If they're all met, they're cleared and
>> + * the job is placed onto the scheduler's queue.
>> + *
>> + * To help reduce conditionals in drivers where a driver supports both the use
>> + * of the media jobs framework and operation without it, this function is a no
>> + * op if @sched is NULL.
>> + *
>> + * Return: 0 on success or a negative error number
>> + */
>> +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
>> +			     enum media_job_types type,
>> +			     struct media_job_dep_ops *dep_ops, void *dep_data);
>> +
>> +/**
>> + * media_jobs_add_job_step - Add a step to a media job
>> + *
>> + * @job:	Pointer to the &struct media_job
>> + * @run_step:	Pointer to the function to run to execute the step
>> + * @data:	Pointer to the data to pass to @run_ste
>> + * @flags:	One of the MEDIA_JOBS_FL_STEP_* flags
>> + * @pos:	A position indicator to use with @flags
>> + *
>> + * This function adds a step to the job and should be called from the drivers'
>> + * job setup functions as registered with the framework through
>> + * media_jobs_add_job_setup_func(). The @flags and @pos parameters are used
>> + * to determine the ordering of the steps within the job:
>> + *
>> + * If @flags has the MEDIA_JOBS_FL_STEP_ANYWHERE bit set, the step is placed
>> + * after all steps with MEDIA_JOBS_FL_STEP_FROM_FRONT and before all steps with
>> + * MEDIA_JOBS_FL_STEP_FROM_BACK bit set, but otherwise in whatever order this
>> + * function is called.
>> + *
>> + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_FRONT bit set then the step is
>> + * placed @pos steps from the front of the list. Attempting to place multiple
>> + * steps in the same position will result in an error.
>> + *
>> + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_BACK bit set then the step is
>> + * placed @pos steps from the back of the list. Attempting to place multiple
>> + * steps in the same position will result in an error.
>> + *
>> + * Return: 0 on success or a negative error number
>> + */
>> +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
>> +			    void *data, unsigned int flags, unsigned int pos);
>> +
>> +/**
>> + * media_jobs_add_job_dep - Add a dependency to a media job
>> + *
>> + * @job:	Pointer to the &struct media_job
>> + * @ops:	Pointer to the &struct media_job_dep_ops
>> + * @data:	Pointer to the data to pass to the dependency's operations
>> + *
>> + * This function adds a dependency to the job and should be called from the
>> + * drivers job setup functions as registered with the framework through the
>> + * media_jobs_add_job_setup_func() function.
>> + *
>> + * Return: 0 on success or a negative error number
>> + */
>> +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
>> +			   void *data);
>> +
>> +/**
>> + * media_jobs_add_job_setup_func - Add a function that populates a media job
>> + *
>> + * @sched:	Pointer to the media jobs scheduler
>> + * @job_setup:	Pointer to the new job setup function
>> + * @data:	Data to pass to the job setup function
>> + * @type:	The type of job that this function should be called for
>> + *
>> + * Drivers that wish to utilise the framework need to use this function to
>> + * register a callback that adds job steps and dependencies when one is created.
>> + * The function must call media_jobs_add_job_step() and media_jobs_add_job_dep()
>> + * to populate the job.
>> + *
>> + * Return: 0 on success or a negative error number
>> + */
>> +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
>> +				  int (*job_setup)(struct media_job *job, void *data),
>> +				  void *data, enum media_job_types type);
>> +
>> +/**
>> + * media_jobs_put_scheduler - Put a reference to the media jobs scheduler
>> + *
>> + * @sched:	Pointer to the media jobs scheduler
>> + *
>> + * This function puts a reference to the media jobs scheduler, and is intended
>> + * to be called in error and exit paths for consuming drivers
>> + */
>> +void media_jobs_put_scheduler(struct media_job_scheduler *sched);
>> +
>> +/**
>> + * media_jobs_get_scheduler - Get a media jobs scheduler
>> + *
>> + * @mdev:	Pointer to the media device associated with the scheduler
>> + *
>> + * This function gets a pointer to a &struct media_job_scheduler associated with
>> + * the media device passed to @mdev. If one is not available then it is
>> + * allocated and returned. This allows multiple drivers sharing a media graph to
>> + * work with the same media job scheduler.
>> + *
>> + * Return: 0 on success or a negative error number
>> + */
>> +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev);
>> +
>> +/**
>> + * media_jobs_run_jobs - Run any media jobs that are ready in the queue
>> + *
>> + * @sched:	Pointer to the media job scheduler
>> + *
>> + * This function triggers the workqueue that processes any jobs that have been
>> + * queued, and should be called whenever the pipeline is ready to do so.
>> + *
>> + * To help reduce conditionals in drivers where a driver supports both the use
>> + * of the media jobs framework and operation without it, this function is a no
>> + * op if @sched is NULL.
>> + */
>> +void media_jobs_run_jobs(struct media_job_scheduler *sched);
>> +
>> +/**
>> + * media_jobs_cancel_jobs - cancel all waiting jobs
>> + *
>> + * @sched:	Pointer to the media job scheduler
>> + *
>> + * This function iterates over any pending and queued jobs, resets their
>> + * dependencies and frees the job
>> + *
>> + * To help reduce conditionals in drivers where a driver supports both the use
>> + * of the media jobs framework and operation without it, this function is a no
>> + * op if @sched is NULL.
>> + */
>> +void media_jobs_cancel_jobs(struct media_job_scheduler *sched);
>> +
>> +extern struct list_head media_job_schedulers;
>> +extern struct mutex media_job_schedulers_lock;
>> +
>> +#endif /* _MEDIA_JOBS_H */
>> --
>> 2.34.1
>>
>>

Re: [PATCH 2/3] media: mc: Add media jobs framework

[Dan Scally]

Hi Jacopo

On 22/05/2025 12:00, Jacopo Mondi wrote:
> Hi Dan, back with more questions :)
>
> On Mon, May 19, 2025 at 03:04:02PM +0100, Daniel Scally wrote:
>> Add a new framework to the media subsystem describing media jobs.
>> This framework is intended to be able to model the interactions
>> between multiple different drivers that need to be run in concert
>> to fully control a media pipeline, for example an ISP driver and a
>> driver controlling a DMA device that feeds data from memory in to
>> that ISP.
>>
>> The new framework allows all drivers involved to add explicit steps
>> that need to be performed, and to control the ordering of those steps
>> precisely. Once the job with its steps has been created it's then
>> scheduled to be run with a workqueue which executes each step in the
>> defined order.
>>
>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>> ---
>>   drivers/media/mc/Makefile  |   2 +-
>>   drivers/media/mc/mc-jobs.c | 446 +++++++++++++++++++++++++++++++++++++
>>   include/media/media-jobs.h | 354 +++++++++++++++++++++++++++++
>>   3 files changed, 801 insertions(+), 1 deletion(-)
>>   create mode 100644 drivers/media/mc/mc-jobs.c
>>   create mode 100644 include/media/media-jobs.h
>>
>> diff --git a/drivers/media/mc/Makefile b/drivers/media/mc/Makefile
>> index 2b7af42ba59c..9148bbfd1578 100644
>> --- a/drivers/media/mc/Makefile
>> +++ b/drivers/media/mc/Makefile
>> @@ -1,7 +1,7 @@
>>   # SPDX-License-Identifier: GPL-2.0
>>
>>   mc-objs	:= mc-device.o mc-devnode.o mc-entity.o \
>> -	   mc-request.o
>> +	   mc-jobs.o mc-request.o
>>
>>   ifneq ($(CONFIG_USB),)
>>   	mc-objs += mc-dev-allocator.o
>> diff --git a/drivers/media/mc/mc-jobs.c b/drivers/media/mc/mc-jobs.c
>> new file mode 100644
>> index 000000000000..1f04cdf63d27
>> --- /dev/null
>> +++ b/drivers/media/mc/mc-jobs.c
>> @@ -0,0 +1,446 @@
>> +// SPDX-License-Identifier: GPL-2.0-only
>> +/*
>> + * Media jobs framework
>> + *
>> + * Copyright 2025 Ideas on Board Oy
>> + *
>> + * Author: Daniel Scally <dan.scally@ideasonboard.com>
>> + */
>> +
>> +#include <linux/cleanup.h>
>> +#include <linux/kref.h>
>> +#include <linux/list.h>
>> +#include <linux/slab.h>
>> +#include <linux/spinlock.h>
>> +
>> +#include <media/media-device.h>
>> +#include <media/media-entity.h>
>> +#include <media/media-jobs.h>
>> +
>> +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
>> +			    void *data, unsigned int flags, unsigned int pos)
>> +{
>> +	struct media_job_step *step, *tmp;
>> +	unsigned int num = flags;
>> +	unsigned int count = 0;
>> +
>> +	guard(spinlock)(&job->lock);
>> +
>> +	if (!flags) {
>> +		WARN_ONCE(1, "%s(): No flag bits set\n", __func__);
>> +		return -EINVAL;
>> +	}
>> +
>> +	/* Count the number of set flags; they're mutually exclusive. */
>> +	while (num) {
>> +		num &= (num - 1);
>> +		count++;
>> +	}
>> +
>> +	if (count > 1) {
>> +		WARN_ONCE(1, "%s(): Multiple flag bits set\n", __func__);
>> +		return -EINVAL;
>> +	}
>> +
>> +	step = kzalloc(sizeof(*step), GFP_KERNEL);
>> +	if (!step)
>> +		return -ENOMEM;
>> +
>> +	step->run_step = run_step;
>> +	step->data = data;
>> +	step->flags = flags;
>> +	step->pos = pos;
>> +
>> +	/*
>> +	 * We need to decide where to place the step. If the list is empty that
>> +	 * is really easy (and also the later code is much easier if the code is
>> +	 * guaranteed not to be empty...)
>> +	 */
>> +	if (list_empty(&job->steps)) {
>> +		list_add_tail(&step->list, &job->steps);
>> +		return 0;
>> +	}
>> +
>> +	/*
>> +	 * If we've been asked to place it at a specific position from the end
>> +	 * of the list, we cycle back through it until either we exhaust the
>> +	 * list or find an entry that needs to go further from the back than the
>> +	 * new one.
>> +	 */
>> +	if ((flags & MEDIA_JOBS_FL_STEP_FROM_BACK)) {
>> +		list_for_each_entry_reverse(tmp, &job->steps, list) {
>> +			if (tmp->flags == flags && tmp->pos == pos)
>> +				return -EINVAL;
>> +
>> +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_BACK ||
>> +			    tmp->pos > pos)
>> +				break;
>> +		}
>> +
>> +		/*
>> +		 * If the entry we broke on is also one placed from the back and
>> +		 * should be closer to the back than the new one, we place the
>> +		 * new one in front of it...otherwise place the new one behind
>> +		 * it.
>> +		 */
>> +		if (tmp->flags == flags && tmp->pos < pos)
>> +			list_add_tail(&step->list, &tmp->list);
>> +		else
>> +			list_add(&step->list, &tmp->list);
>> +
>> +		return 0;
>> +	}
>> +
>> +	/*
>> +	 * If we've been asked to place it a specific position from the front of
>> +	 * the list we do the same kind of operation, but going from the front
>> +	 * instead.
>> +	 */
>> +	if (flags & MEDIA_JOBS_FL_STEP_FROM_FRONT) {
>> +		list_for_each_entry(tmp, &job->steps, list) {
>> +			if (tmp->flags == flags && tmp->pos == pos)
>> +				return -EINVAL;
>> +
>> +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_FRONT ||
>> +			    tmp->pos > pos)
>> +				break;
>> +		}
>> +
>> +		/*
>> +		 * If the entry we broke on is also placed from the front and
>> +		 * should be closed to the front than the new one, we place the
>> +		 * new one behind it, otherwise in front of it.
>> +		 */
>> +		if (tmp->flags == flags && tmp->pos < pos)
>> +			list_add(&step->list, &tmp->list);
>> +		else
>> +			list_add_tail(&step->list, &tmp->list);
>> +
>> +		return 0;
>> +	}
>> +
>> +	/*
>> +	 * If the step is flagged as "can go anywhere" we just need to try to
>> +	 * find the first "from the back" entry and add it immediately before
>> +	 * that. If we can't find one, add it after whatever we did find.
>> +	 */
>> +	if (flags & MEDIA_JOBS_FL_STEP_ANYWHERE) {
>> +		list_for_each_entry(tmp, &job->steps, list)
>> +			if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK))
>> +				break;
>> +
>> +		if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK) ||
>> +		    list_entry_is_head(tmp, &job->steps, list))
>> +			list_add_tail(&step->list, &tmp->list);
>> +		else
>> +			list_add(&step->list, &tmp->list);
>> +
>> +		return 0;
>> +	}
>> +
>> +	/* Shouldn't get here, unless the flag value is wrong. */
>> +	WARN_ONCE(1, "%s(): Invalid flag value\n", __func__);
>> +	return -EINVAL;
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_add_job_step);
>> +
>> +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
>> +			   void *data)
>> +{
>> +	struct media_job_dep *dep;
>> +
>> +	if (!ops || !ops->check_dep || !data)
>> +		return -EINVAL;
>> +
>> +	guard(spinlock)(&job->lock);
>> +
>> +	/* Confirm the same dependency hasn't already been added */
>> +	list_for_each_entry(dep, &job->deps, list)
>> +		if (dep->ops == ops && dep->data == data)
>> +			return -EINVAL;
>> +
>> +	dep = kzalloc(sizeof(*dep), GFP_KERNEL);
>> +	if (!dep)
>> +		return -ENOMEM;
>> +
>> +	dep->ops = ops;
>> +	dep->data = data;
>> +	list_add(&dep->list, &job->deps);
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_add_job_dep);
>> +
>> +static bool media_jobs_check_pending_job(struct media_job *job,
>> +					 enum media_job_types type,
>> +					 struct media_job_dep_ops *dep_ops,
>> +					 void *data)
>> +{
>> +	struct media_job_dep *dep;
>> +
>> +	guard(spinlock)(&job->lock);
>> +
>> +	if (job->type != type)
>> +		return false;
>> +
>> +	list_for_each_entry(dep, &job->deps, list) {
>> +		if (dep->ops == dep_ops && dep->data == data) {
>> +			if (dep->met)
>> +				return false;
>> +
>> +			break;
>> +		}
>> +	}
>> +
>> +	dep->met = true;
>> +	return true;
>> +}
>> +
>> +static struct media_job *media_jobs_get_job(struct media_job_scheduler *sched,
>> +					    enum media_job_types type,
>> +					    struct media_job_dep_ops *dep_ops,
>> +					    void *dep_data)
>> +{
>> +	struct media_job_setup_func *jsf;
>> +	struct media_job *job;
>> +	int ret;
>> +
>> +	list_for_each_entry(job, &sched->pending, list)
>> +		if (media_jobs_check_pending_job(job, type, dep_ops, dep_data))
>> +			return job;
>> +
> Thanks to your offline explanation, I got how this works now, however
> some questions here
>
> The basic idea is that each driver that registers a 'setup' function
> adds to a job, when its created, its list of dependencies.
>
> When a job is "try_queue" and we get here, to decide if a new job has
> to be created or if we have to run one which is already in the pending
> queue.
>
> How is this identification performed ? Each entry point (assume it's a
> video device op) will populate the job with its own dependencies,
> identified by the dep_ops and data address.
>
> We walk the 'pending' queue in the media_jobs_check_pending_job()
> function and we search for one job not already visited from the same
> entry point, identified by the dep_ops (the 'visited' state is kept by
> the deps->met flag).
>
> Let's assume 2 video devices X and Y
>
> qbuf(x) -> try_queue_job() -> new job created on 'pending'
> qbuf(x) -> try_queue_job() -> the job in the queue has 'deps->met' set, skip
> it and create a new one
> qbuf(y) -> try_queue_job() -> the first job in the queue has not
> deps->set, so return it
>
> All in all I would describe this as: when requesting a job try to find
> the first one not already visited by this entry point, if none is
> available create a new one.


Yep, all seems fine.

> Now, we briefly discussed that when moving to multi-context comparing
> dep_ops and data to identify an entry point won't be enough: buffers
> from the same video device but from different contexts do not have to
> be associated together. So we'll need to extend the identification
> criteria. Also, I don't find the idea of using dep_ops and data for
> this purpose particularly neat, as it makes mandatory to add
> dependencies to a job in the setup function, something not all driver
> might want to do ?

I don't think it's mandatory for a driver to add dependencies to a job; the implication of lacking 
them is that whatever step the driver is running for the job takes no input (no buffers need to be 
available, no parameters need to have been set, no per-requisites need to have been met) in which 
case it can simply be ignored for the purposes of evaluating whether the job can be queued or not, 
because it's always ready by definition...does that make sense?

> There might be ways to handle this "track the entry point" thing that
> could be separated by deps, making deps do what they actually are
> described for: track dependencies to validate if a job can be run or
> not. Before exploring options, I would like to know if this only mine
> concern or is it shared by others.


I do agree that a nicer way of tracking them rather than dep_ops and data would be better...perhaps 
tying it to the entry point as you've conceptualised here is the right thing to do, and the pointer 
to the function calling media_jobs_try_queue_job() should be passed, along with a context ID?


Thanks

Dan

>
>> +	job = kzalloc(sizeof(*job), GFP_KERNEL);
>> +	if (!job)
>> +		return ERR_PTR(-ENOMEM);
>> +
>> +	spin_lock_init(&job->lock);
>> +	INIT_LIST_HEAD(&job->deps);
>> +	INIT_LIST_HEAD(&job->steps);
>> +	job->type = type;
>> +	job->sched = sched;
>> +
>> +	list_for_each_entry(jsf, &sched->setup_funcs, list) {
>> +		if (jsf->type != type)
>> +			continue;
>> +
>> +		ret = jsf->job_setup(job, jsf->data);
>> +		if (ret) {
>> +			kfree(job);
>> +			return ERR_PTR(ret);
>> +		}
>> +	}
>> +
>> +	list_add_tail(&job->list, &sched->pending);
>> +
>> +	/* This marks the dependency as met */
>> +	media_jobs_check_pending_job(job, type, dep_ops, dep_data);
>> +
>> +	return job;
>> +}
>> +
>> +static void media_jobs_free_job(struct media_job *job, bool reset)
>> +{
>> +	struct media_job_step *step, *stmp;
>> +	struct media_job_dep *dep, *dtmp;
>> +
>> +	scoped_guard(spinlock, &job->lock) {
>> +		list_for_each_entry_safe(dep, dtmp, &job->deps, list) {
>> +			if (reset && dep->ops->reset_dep)
>> +				dep->ops->reset_dep(dep->data);
>> +
>> +			list_del(&dep->list);
>> +			kfree(dep);
>> +		}
>> +
>> +		list_for_each_entry_safe(step, stmp, &job->steps, list) {
>> +			list_del(&step->list);
>> +			kfree(step);
>> +		}
>> +	}
>> +
>> +	list_del(&job->list);
>> +	kfree(job);
>> +}
>> +
>> +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
>> +			     enum media_job_types type,
>> +			     struct media_job_dep_ops *dep_ops, void *dep_data)
>> +{
>> +	struct media_job_dep *dep;
>> +	struct media_job *job;
>> +
>> +	if (!sched)
>> +		return 0;
>> +
>> +	guard(spinlock)(&sched->lock);
>> +
>> +	job = media_jobs_get_job(sched, type, dep_ops, dep_data);
>> +	if (IS_ERR(job))
>> +		return PTR_ERR(job);
>> +
>> +	list_for_each_entry(dep, &job->deps, list)
>> +		if (!dep->ops->check_dep(dep->data))
>> +			return 0; /* Not a failure */
>> +
>> +	list_for_each_entry(dep, &job->deps, list)
>> +		if (dep->ops->clear_dep)
>> +			dep->ops->clear_dep(dep->data);
>> +
>> +	list_move_tail(&job->list, &sched->queue);
>> +	queue_work(sched->async_wq, &sched->work);
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_try_queue_job);
>> +
>> +static void __media_jobs_run_jobs(struct work_struct *work)
>> +{
>> +	struct media_job_scheduler *sched = container_of(work,
>> +							 struct media_job_scheduler,
>> +							 work);
>> +	struct media_job_step *step;
>> +	struct media_job *job;
>> +
>> +	while (true) {
>> +		scoped_guard(spinlock, &sched->lock) {
>> +			if (list_empty(&sched->queue))
>> +				return;
>> +
>> +			job = list_first_entry(&sched->queue, struct media_job,
>> +					       list);
>> +		}
>> +
>> +		list_for_each_entry(step, &job->steps, list)
>> +			step->run_step(step->data);
>> +
>> +		media_jobs_free_job(job, false);
>> +	}
>> +}
>> +
>> +void media_jobs_run_jobs(struct media_job_scheduler *sched)
>> +{
>> +	if (!sched)
>> +		return;
>> +
>> +	queue_work(sched->async_wq, &sched->work);
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_run_jobs);
>> +
>> +static void __media_jobs_cancel_jobs(struct media_job_scheduler *sched)
>> +{
>> +	struct media_job *job, *jtmp;
>> +
>> +	list_for_each_entry_safe(job, jtmp, &sched->pending, list)
>> +		media_jobs_free_job(job, true);
>> +
>> +	list_for_each_entry_safe(job, jtmp, &sched->queue, list)
>> +		media_jobs_free_job(job, true);
>> +}
>> +
>> +void media_jobs_cancel_jobs(struct media_job_scheduler *sched)
>> +{
>> +	if (!sched)
>> +		return;
>> +
>> +	guard(spinlock)(&sched->lock);
>> +	__media_jobs_cancel_jobs(sched);
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_cancel_jobs);
>> +
>> +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
>> +				  int (*job_setup)(struct media_job *job, void *data),
>> +				  void *data, enum media_job_types type)
>> +{
>> +	struct media_job_setup_func *new_setup_func;
>> +
>> +	guard(spinlock)(&sched->lock);
>> +
>> +	new_setup_func = kzalloc(sizeof(*new_setup_func), GFP_KERNEL);
>> +	if (!new_setup_func)
>> +		return -ENOMEM;
>> +
>> +	new_setup_func->type = type;
>> +	new_setup_func->job_setup = job_setup;
>> +	new_setup_func->data = data;
>> +	list_add_tail(&new_setup_func->list, &sched->setup_funcs);
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_add_job_setup_func);
>> +
>> +static void __media_jobs_put_scheduler(struct kref *kref)
>> +{
>> +	struct media_job_scheduler *sched =
>> +		container_of(kref, struct media_job_scheduler, kref);
>> +	struct media_job_setup_func *func, *ftmp;
>> +
>> +	cancel_work_sync(&sched->work);
>> +	destroy_workqueue(sched->async_wq);
>> +
>> +	scoped_guard(spinlock, &sched->lock) {
>> +		__media_jobs_cancel_jobs(sched);
>> +
>> +		list_for_each_entry_safe(func, ftmp, &sched->setup_funcs, list) {
>> +			list_del(&func->list);
>> +			kfree(func);
>> +		}
>> +	}
>> +
>> +	list_del(&sched->list);
>> +	kfree(sched);
>> +}
>> +
>> +void media_jobs_put_scheduler(struct media_job_scheduler *sched)
>> +{
>> +	kref_put(&sched->kref, __media_jobs_put_scheduler);
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_put_scheduler);
>> +
>> +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev)
>> +{
>> +	struct media_job_scheduler *sched;
>> +	char workqueue_name[32];
>> +	int ret;
>> +
>> +	guard(mutex)(&media_job_schedulers_lock);
>> +
>> +	list_for_each_entry(sched, &media_job_schedulers, list) {
>> +		if (sched->mdev == mdev) {
>> +			kref_get(&sched->kref);
>> +			return sched;
>> +		}
>> +	}
>> +
>> +	ret = snprintf(workqueue_name, sizeof(workqueue_name),
>> +		       "mc jobs (%s)", mdev->driver_name);
>> +	if (!ret)
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	sched = kzalloc(sizeof(*sched), GFP_KERNEL);
>> +	if (!sched)
>> +		return ERR_PTR(-ENOMEM);
>> +
>> +	sched->async_wq = alloc_workqueue(workqueue_name, 0, 0);
>> +	if (!sched->async_wq) {
>> +		kfree(sched);
>> +		return ERR_PTR(-EINVAL);
>> +	}
>> +
>> +	sched->mdev = mdev;
>> +	kref_init(&sched->kref);
>> +	spin_lock_init(&sched->lock);
>> +	INIT_LIST_HEAD(&sched->setup_funcs);
>> +	INIT_LIST_HEAD(&sched->pending);
>> +	INIT_LIST_HEAD(&sched->queue);
>> +	INIT_WORK(&sched->work, __media_jobs_run_jobs);
>> +
>> +	list_add_tail(&sched->list, &media_job_schedulers);
>> +
>> +	return sched;
>> +}
>> +EXPORT_SYMBOL_GPL(media_jobs_get_scheduler);
>> +
>> +LIST_HEAD(media_job_schedulers);
>> +
>> +/* Synchronise access to the global schedulers list */
>> +DEFINE_MUTEX(media_job_schedulers_lock);
>> diff --git a/include/media/media-jobs.h b/include/media/media-jobs.h
>> new file mode 100644
>> index 000000000000..a97270861251
>> --- /dev/null
>> +++ b/include/media/media-jobs.h
>> @@ -0,0 +1,354 @@
>> +/* SPDX-License-Identifier: GPL-2.0-only */
>> +/*
>> + * Media jobs framework
>> + *
>> + * Copyright 2025 Ideas on Board Oy
>> + *
>> + * Author: Daniel Scally <dan.scally@ideasonboard.com>
>> + */
>> +
>> +#include <linux/kref.h>
>> +#include <linux/list.h>
>> +#include <linux/mutex.h>
>> +#include <linux/spinlock.h>
>> +#include <linux/types.h>
>> +#include <linux/workqueue.h>
>> +
>> +#ifndef _MEDIA_JOBS_H
>> +#define _MEDIA_JOBS_H
>> +
>> +struct media_device;
>> +struct media_entity;
>> +struct media_job;
>> +struct media_job_dep;
>> +
>> +/**
>> + * define MEDIA_JOBS_FL_STEP_ANYWHERE - \
>> + *    Flag a media job step as able to run anytime
>> + *
>> + * This flag informs the framework that a job step does not need a particular
>> + * position in the list of job steps and can be placed anywhere.
>> + */
>> +#define MEDIA_JOBS_FL_STEP_ANYWHERE			BIT(0)
>> +
>> +/**
>> + * define MEDIA_JOBS_FL_STEP_FROM_FRONT - \
>> + *    Flag a media job step as needing to be placed near the start of the list
>> + *
>> + * This flag informs the framework that a job step needs to be placed at a set
>> + * position from the start of the list of job steps.
>> + */
>> +#define MEDIA_JOBS_FL_STEP_FROM_FRONT			BIT(1)
>> +
>> +/**
>> + * define MEDIA_JOBS_FL_STEP_FROM_BACK - \
>> + *    Flag a media job step as needing to be placed near the end of the list
>> + *
>> + * This flag informs the framework that a job step needs to be placed at a set
>> + * position from the end of the list of job steps.
>> + */
>> +#define MEDIA_JOBS_FL_STEP_FROM_BACK			BIT(2)
>> +
>> +/**
>> + * enum media_job_types - Type of media job
>> + *
>> + * @MEDIA_JOB_TYPE_PIPELINE_PULSE:	A data event moving through the media
>> + *					pipeline
>> + *
>> + * This enumeration details different types of media jobs. The type can be used
>> + * to differentiate between which steps and dependencies a driver needs to add
>> + * to a job when it is created.
>> + */
>> +enum media_job_types {
>> +	MEDIA_JOB_TYPE_PIPELINE_PULSE,
>> +};
>> +
>> +/**
>> + * struct media_job_scheduler - A job scheduler for a particular media device
>> + *
>> + * @mdev:		Media device this scheduler is for
>> + * @list:		List head to attach to the global list of schedulers
>> + * @kref:		Reference counter
>> + * @lock:		Lock to protect access to the scheduler
>> + * @setup_funcs:	List of &struct media_job_setup_func to populate jobs
>> + * @pending:		List of &struct media_jobs created but not yet queued
>> + * @queue:		List of &struct media_jobs queued to the scheduler
>> + * @work:		Work item to run the jobs
>> + * @async_wq:		Workqueue to run the work on
>> + *
>> + * This struct is the main job scheduler struct - drivers wanting to use this
>> + * framework should acquire an instance through media_jobs_get_scheduler() and
>> + * subsequently populate it with job setup functions.
>> + */
>> +struct media_job_scheduler {
>> +	struct media_device *mdev;
>> +	struct list_head list;
>> +	struct kref kref;
>> +
>> +	spinlock_t lock; /* Synchronise access to the struct's lists */
>> +	struct list_head setup_funcs;
>> +	struct list_head pending;
>> +	struct list_head queue;
>> +	struct work_struct work;
>> +	struct workqueue_struct *async_wq;
>> +};
>> +
>> +/**
>> + * struct media_job_setup_func - A function to populate a media job with steps
>> + *				 and dependencies
>> + *
>> + * @list:	The list object to attach to the scheduler
>> + * @type:	The &enum media_job_types that this function populates a job for
>> + * @job_setup:	Function pointer to the driver's job setup function
>> + * @data:	Pointer to the driver data for use with @job_setup
>> + *
>> + * This struct holds data about the functions a driver registers with the jobs
>> + * framework in order to populate a new job with steps and dependencies.
>> + */
>> +struct media_job_setup_func {
>> +	struct list_head list;
>> +	enum media_job_types type;
>> +	int (*job_setup)(struct media_job *job, void *data);
>> +	void *data;
>> +};
>> +
>> +/**
>> + * struct media_job - A representation of a job to be run through the pipeline
>> + *
>> + * @lock:	Lock to protect access to the job's lists
>> + * @list:	List head to attach the job to &struct media_job_scheduler in
>> + *		either the pending or queue lists
>> + * @steps:	List of &struct media_job_step to run the job
>> + * @deps:	List of &struct media_job_dep to check that the job can be
>> + *		queued
>> + * @sched:	Pointer to the media job scheduler
>> + * @type:	The type of the job
>> + *
>> + * This struct holds lists of steps that need to be performed to carry out a
>> + * job in the pipeline. A separate list of dependencies allows the queueing of
>> + * the job to be delayed until all drivers are ready to carry it out.
>> + */
>> +struct media_job {
>> +	spinlock_t lock; /* Synchronise access to the struct's lists 6*/
>> +	struct list_head list;
>> +	struct list_head steps;
>> +	struct list_head deps;
>> +	struct media_job_scheduler *sched;
>> +	enum media_job_types type;
>> +};
>> +
>> +/**
>> + * struct media_job_step - A holder for a function to run as part of a job
>> + *
>> + * @list:	List head to attach the job step to a &struct media_job.steps
>> + * @run_step:	The function to run to perform the step
>> + * @data:	Data to pass to the .run_step() function
>> + * @flags:	Flags to control how the step is ordered within the job's list
>> + *		of steps
>> + * @pos:	Position indicator to control how the step is ordered within the
>> + *		job's list of steps
>> + *
>> + * This struct defines a function that needs to be run as part of the execution
>> + * of a job in a media pipeline, along with information that help the scheduler
>> + * determine what order it should be ran in in reference to the other steps that
>> + * are part of the same job.
>> + */
>> +struct media_job_step {
>> +	struct list_head list;
>> +	void (*run_step)(void *data);
>> +	void *data;
>> +	unsigned int flags;
>> +	unsigned int pos;
>> +};
>> +
>> +/**
>> + * struct media_job_dep_ops - Operations to manage a media job dependency
>> + *
>> + * @check_dep:	A function to ask the driver whether the dependency is met
>> + * @clear_dep:	A function to tell the driver that the job has been queued
>> + * @reset_dep:	A function to tell the driver that the job has been cancelled
>> + *
>> + * Media jobs have dependencies, such as requiring buffers to be queued. These
>> + * operations allow a driver to define how the media jobs framework should check
>> + * whether or not those dependencies are met and how it should inform them that
>> + * it is taking action based on the state of those dependencies.
>> + */
>> +struct media_job_dep_ops {
>> +	bool (*check_dep)(void *data);
>> +	void (*clear_dep)(void *data);
>> +	void (*reset_dep)(void *data);
>> +};
>> +
>> +/**
>> + * struct media_job_dep - Representation of media job dependency
>> + *
>> + * @list:	List head to attach to a &struct media_job.deps
>> + * @ops:	A pointer to the dependency's operations functions
>> + * @met:	A flag to record whether or not the dependency is met
>> + * @data:	Data to pass to the dependency's operations
>> + *
>> + * This struct represents a dependency of a media job. The operations member
>> + * holds pointers to functions allowing the framework to interact with the
>> + * driver to check whether or not the dependency is met.
>> + */
>> +struct media_job_dep {
>> +	struct list_head list;
>> +	struct media_job_dep_ops *ops;
>> +	bool met;
>> +	void *data;
>> +};
>> +
>> +/**
>> + * media_jobs_try_queue_job - Try to queue a &struct media_job
>> + *
>> + * @sched:	Pointer to the job scheduler
>> + * @type:	The type of the media job
>> + * @dep_ops:	A pointer to the dependency operations for this job
>> + * @dep_data:	A pointer to the dependency data for this job
>> + *
>> + * Try to queue a media job with the scheduler. This function should be called
>> + * by the drivers whenever a dependency for a media job is met - for example
>> + * when a buffer is queued to the driver. The framework will check to see if an
>> + * existing job on the scheduler's pending list shares the same type, dependency
>> + * operations and dependency data. If it does then that existing job will be
>> + * considered. If there is no extant job with those same parameters, a new job
>> + * is allocated and populated by calling the setup functions registered with
>> + * the framework.
>> + *
>> + * The function iterates over the dependencies that are registered with the job
>> + * and checks to see if they are met. If they're all met, they're cleared and
>> + * the job is placed onto the scheduler's queue.
>> + *
>> + * To help reduce conditionals in drivers where a driver supports both the use
>> + * of the media jobs framework and operation without it, this function is a no
>> + * op if @sched is NULL.
>> + *
>> + * Return: 0 on success or a negative error number
>> + */
>> +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
>> +			     enum media_job_types type,
>> +			     struct media_job_dep_ops *dep_ops, void *dep_data);
>> +
>> +/**
>> + * media_jobs_add_job_step - Add a step to a media job
>> + *
>> + * @job:	Pointer to the &struct media_job
>> + * @run_step:	Pointer to the function to run to execute the step
>> + * @data:	Pointer to the data to pass to @run_ste
>> + * @flags:	One of the MEDIA_JOBS_FL_STEP_* flags
>> + * @pos:	A position indicator to use with @flags
>> + *
>> + * This function adds a step to the job and should be called from the drivers'
>> + * job setup functions as registered with the framework through
>> + * media_jobs_add_job_setup_func(). The @flags and @pos parameters are used
>> + * to determine the ordering of the steps within the job:
>> + *
>> + * If @flags has the MEDIA_JOBS_FL_STEP_ANYWHERE bit set, the step is placed
>> + * after all steps with MEDIA_JOBS_FL_STEP_FROM_FRONT and before all steps with
>> + * MEDIA_JOBS_FL_STEP_FROM_BACK bit set, but otherwise in whatever order this
>> + * function is called.
>> + *
>> + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_FRONT bit set then the step is
>> + * placed @pos steps from the front of the list. Attempting to place multiple
>> + * steps in the same position will result in an error.
>> + *
>> + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_BACK bit set then the step is
>> + * placed @pos steps from the back of the list. Attempting to place multiple
>> + * steps in the same position will result in an error.
>> + *
>> + * Return: 0 on success or a negative error number
>> + */
>> +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
>> +			    void *data, unsigned int flags, unsigned int pos);
>> +
>> +/**
>> + * media_jobs_add_job_dep - Add a dependency to a media job
>> + *
>> + * @job:	Pointer to the &struct media_job
>> + * @ops:	Pointer to the &struct media_job_dep_ops
>> + * @data:	Pointer to the data to pass to the dependency's operations
>> + *
>> + * This function adds a dependency to the job and should be called from the
>> + * drivers job setup functions as registered with the framework through the
>> + * media_jobs_add_job_setup_func() function.
>> + *
>> + * Return: 0 on success or a negative error number
>> + */
>> +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
>> +			   void *data);
>> +
>> +/**
>> + * media_jobs_add_job_setup_func - Add a function that populates a media job
>> + *
>> + * @sched:	Pointer to the media jobs scheduler
>> + * @job_setup:	Pointer to the new job setup function
>> + * @data:	Data to pass to the job setup function
>> + * @type:	The type of job that this function should be called for
>> + *
>> + * Drivers that wish to utilise the framework need to use this function to
>> + * register a callback that adds job steps and dependencies when one is created.
>> + * The function must call media_jobs_add_job_step() and media_jobs_add_job_dep()
>> + * to populate the job.
>> + *
>> + * Return: 0 on success or a negative error number
>> + */
>> +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
>> +				  int (*job_setup)(struct media_job *job, void *data),
>> +				  void *data, enum media_job_types type);
>> +
>> +/**
>> + * media_jobs_put_scheduler - Put a reference to the media jobs scheduler
>> + *
>> + * @sched:	Pointer to the media jobs scheduler
>> + *
>> + * This function puts a reference to the media jobs scheduler, and is intended
>> + * to be called in error and exit paths for consuming drivers
>> + */
>> +void media_jobs_put_scheduler(struct media_job_scheduler *sched);
>> +
>> +/**
>> + * media_jobs_get_scheduler - Get a media jobs scheduler
>> + *
>> + * @mdev:	Pointer to the media device associated with the scheduler
>> + *
>> + * This function gets a pointer to a &struct media_job_scheduler associated with
>> + * the media device passed to @mdev. If one is not available then it is
>> + * allocated and returned. This allows multiple drivers sharing a media graph to
>> + * work with the same media job scheduler.
>> + *
>> + * Return: 0 on success or a negative error number
>> + */
>> +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev);
>> +
>> +/**
>> + * media_jobs_run_jobs - Run any media jobs that are ready in the queue
>> + *
>> + * @sched:	Pointer to the media job scheduler
>> + *
>> + * This function triggers the workqueue that processes any jobs that have been
>> + * queued, and should be called whenever the pipeline is ready to do so.
>> + *
>> + * To help reduce conditionals in drivers where a driver supports both the use
>> + * of the media jobs framework and operation without it, this function is a no
>> + * op if @sched is NULL.
>> + */
>> +void media_jobs_run_jobs(struct media_job_scheduler *sched);
>> +
>> +/**
>> + * media_jobs_cancel_jobs - cancel all waiting jobs
>> + *
>> + * @sched:	Pointer to the media job scheduler
>> + *
>> + * This function iterates over any pending and queued jobs, resets their
>> + * dependencies and frees the job
>> + *
>> + * To help reduce conditionals in drivers where a driver supports both the use
>> + * of the media jobs framework and operation without it, this function is a no
>> + * op if @sched is NULL.
>> + */
>> +void media_jobs_cancel_jobs(struct media_job_scheduler *sched);
>> +
>> +extern struct list_head media_job_schedulers;
>> +extern struct mutex media_job_schedulers_lock;
>> +
>> +#endif /* _MEDIA_JOBS_H */
>> --
>> 2.34.1
>>
>>

Re: [PATCH 1/3] media: mc: entity: Add pipeline_started/stopped ops

[kernel test robot]

Hi Daniel,

kernel test robot noticed the following build warnings:

[auto build test WARNING on linuxtv-media-pending/master]
[also build test WARNING on linus/master sailus-media-tree/master media-tree/master v6.15-rc7 next-20250522]
[cannot apply to sailus-media-tree/streams]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Daniel-Scally/media-mc-entity-Add-pipeline_started-stopped-ops/20250519-222333
base:   https://git.linuxtv.org/media-ci/media-pending.git master
patch link:    https://lore.kernel.org/r/20250519140403.443915-2-dan.scally%40ideasonboard.com
patch subject: [PATCH 1/3] media: mc: entity: Add pipeline_started/stopped ops
config: m68k-randconfig-r111-20250522 (https://download.01.org/0day-ci/archive/20250522/202505222107.vNoAm2e7-lkp@intel.com/config)
compiler: m68k-linux-gcc (GCC) 13.3.0
reproduce: (https://download.01.org/0day-ci/archive/20250522/202505222107.vNoAm2e7-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202505222107.vNoAm2e7-lkp@intel.com/

sparse warnings: (new ones prefixed by >>)
>> drivers/media/mc/mc-entity.c:1093:17: sparse: sparse: incompatible types in conditional expression (different base types):
   drivers/media/mc/mc-entity.c:1093:17: sparse:    void
   drivers/media/mc/mc-entity.c:1093:17: sparse:    int

vim +1093 drivers/media/mc/mc-entity.c

  1081	
  1082	int media_pipeline_stopped(struct media_pipeline *pipe)
  1083	{
  1084		struct media_pipeline_entity_iter iter;
  1085		struct media_entity *entity;
  1086		int ret;
  1087	
  1088		ret = media_pipeline_entity_iter_init(pipe, &iter);
  1089		if (ret)
  1090			return ret;
  1091	
  1092		media_pipeline_for_each_entity(pipe, &iter, entity)
> 1093			media_entity_call(entity, pipeline_stopped);
  1094	
  1095		media_pipeline_entity_iter_cleanup(&iter);
  1096	
  1097		return 0;
  1098	}
  1099	EXPORT_SYMBOL_GPL(media_pipeline_stopped);
  1100	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH 2/3] media: mc: Add media jobs framework

[Jacopo Mondi]

Hi Dan

On Thu, May 22, 2025 at 12:24:46PM +0100, Dan Scally wrote:
> Hi Jacopo
>
> On 22/05/2025 12:00, Jacopo Mondi wrote:
> > Hi Dan, back with more questions :)
> >
> > On Mon, May 19, 2025 at 03:04:02PM +0100, Daniel Scally wrote:
> > > Add a new framework to the media subsystem describing media jobs.
> > > This framework is intended to be able to model the interactions
> > > between multiple different drivers that need to be run in concert
> > > to fully control a media pipeline, for example an ISP driver and a
> > > driver controlling a DMA device that feeds data from memory in to
> > > that ISP.
> > >
> > > The new framework allows all drivers involved to add explicit steps
> > > that need to be performed, and to control the ordering of those steps
> > > precisely. Once the job with its steps has been created it's then
> > > scheduled to be run with a workqueue which executes each step in the
> > > defined order.
> > >
> > > Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> > > ---
> > >   drivers/media/mc/Makefile  |   2 +-
> > >   drivers/media/mc/mc-jobs.c | 446 +++++++++++++++++++++++++++++++++++++
> > >   include/media/media-jobs.h | 354 +++++++++++++++++++++++++++++
> > >   3 files changed, 801 insertions(+), 1 deletion(-)
> > >   create mode 100644 drivers/media/mc/mc-jobs.c
> > >   create mode 100644 include/media/media-jobs.h
> > >
> > > diff --git a/drivers/media/mc/Makefile b/drivers/media/mc/Makefile
> > > index 2b7af42ba59c..9148bbfd1578 100644
> > > --- a/drivers/media/mc/Makefile
> > > +++ b/drivers/media/mc/Makefile
> > > @@ -1,7 +1,7 @@
> > >   # SPDX-License-Identifier: GPL-2.0
> > >
> > >   mc-objs	:= mc-device.o mc-devnode.o mc-entity.o \
> > > -	   mc-request.o
> > > +	   mc-jobs.o mc-request.o
> > >
> > >   ifneq ($(CONFIG_USB),)
> > >   	mc-objs += mc-dev-allocator.o
> > > diff --git a/drivers/media/mc/mc-jobs.c b/drivers/media/mc/mc-jobs.c
> > > new file mode 100644
> > > index 000000000000..1f04cdf63d27
> > > --- /dev/null
> > > +++ b/drivers/media/mc/mc-jobs.c
> > > @@ -0,0 +1,446 @@
> > > +// SPDX-License-Identifier: GPL-2.0-only
> > > +/*
> > > + * Media jobs framework
> > > + *
> > > + * Copyright 2025 Ideas on Board Oy
> > > + *
> > > + * Author: Daniel Scally <dan.scally@ideasonboard.com>
> > > + */
> > > +
> > > +#include <linux/cleanup.h>
> > > +#include <linux/kref.h>
> > > +#include <linux/list.h>
> > > +#include <linux/slab.h>
> > > +#include <linux/spinlock.h>
> > > +
> > > +#include <media/media-device.h>
> > > +#include <media/media-entity.h>
> > > +#include <media/media-jobs.h>
> > > +
> > > +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
> > > +			    void *data, unsigned int flags, unsigned int pos)
> > > +{
> > > +	struct media_job_step *step, *tmp;
> > > +	unsigned int num = flags;
> > > +	unsigned int count = 0;
> > > +
> > > +	guard(spinlock)(&job->lock);
> > > +
> > > +	if (!flags) {
> > > +		WARN_ONCE(1, "%s(): No flag bits set\n", __func__);
> > > +		return -EINVAL;
> > > +	}
> > > +
> > > +	/* Count the number of set flags; they're mutually exclusive. */
> > > +	while (num) {
> > > +		num &= (num - 1);
> > > +		count++;
> > > +	}
> > > +
> > > +	if (count > 1) {
> > > +		WARN_ONCE(1, "%s(): Multiple flag bits set\n", __func__);
> > > +		return -EINVAL;
> > > +	}
> > > +
> > > +	step = kzalloc(sizeof(*step), GFP_KERNEL);
> > > +	if (!step)
> > > +		return -ENOMEM;
> > > +
> > > +	step->run_step = run_step;
> > > +	step->data = data;
> > > +	step->flags = flags;
> > > +	step->pos = pos;
> > > +
> > > +	/*
> > > +	 * We need to decide where to place the step. If the list is empty that
> > > +	 * is really easy (and also the later code is much easier if the code is
> > > +	 * guaranteed not to be empty...)
> > > +	 */
> > > +	if (list_empty(&job->steps)) {
> > > +		list_add_tail(&step->list, &job->steps);
> > > +		return 0;
> > > +	}
> > > +
> > > +	/*
> > > +	 * If we've been asked to place it at a specific position from the end
> > > +	 * of the list, we cycle back through it until either we exhaust the
> > > +	 * list or find an entry that needs to go further from the back than the
> > > +	 * new one.
> > > +	 */
> > > +	if ((flags & MEDIA_JOBS_FL_STEP_FROM_BACK)) {
> > > +		list_for_each_entry_reverse(tmp, &job->steps, list) {
> > > +			if (tmp->flags == flags && tmp->pos == pos)
> > > +				return -EINVAL;
> > > +
> > > +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_BACK ||
> > > +			    tmp->pos > pos)
> > > +				break;
> > > +		}
> > > +
> > > +		/*
> > > +		 * If the entry we broke on is also one placed from the back and
> > > +		 * should be closer to the back than the new one, we place the
> > > +		 * new one in front of it...otherwise place the new one behind
> > > +		 * it.
> > > +		 */
> > > +		if (tmp->flags == flags && tmp->pos < pos)
> > > +			list_add_tail(&step->list, &tmp->list);
> > > +		else
> > > +			list_add(&step->list, &tmp->list);
> > > +
> > > +		return 0;
> > > +	}
> > > +
> > > +	/*
> > > +	 * If we've been asked to place it a specific position from the front of
> > > +	 * the list we do the same kind of operation, but going from the front
> > > +	 * instead.
> > > +	 */
> > > +	if (flags & MEDIA_JOBS_FL_STEP_FROM_FRONT) {
> > > +		list_for_each_entry(tmp, &job->steps, list) {
> > > +			if (tmp->flags == flags && tmp->pos == pos)
> > > +				return -EINVAL;
> > > +
> > > +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_FRONT ||
> > > +			    tmp->pos > pos)
> > > +				break;
> > > +		}
> > > +
> > > +		/*
> > > +		 * If the entry we broke on is also placed from the front and
> > > +		 * should be closed to the front than the new one, we place the
> > > +		 * new one behind it, otherwise in front of it.
> > > +		 */
> > > +		if (tmp->flags == flags && tmp->pos < pos)
> > > +			list_add(&step->list, &tmp->list);
> > > +		else
> > > +			list_add_tail(&step->list, &tmp->list);
> > > +
> > > +		return 0;
> > > +	}
> > > +
> > > +	/*
> > > +	 * If the step is flagged as "can go anywhere" we just need to try to
> > > +	 * find the first "from the back" entry and add it immediately before
> > > +	 * that. If we can't find one, add it after whatever we did find.
> > > +	 */
> > > +	if (flags & MEDIA_JOBS_FL_STEP_ANYWHERE) {
> > > +		list_for_each_entry(tmp, &job->steps, list)
> > > +			if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK))
> > > +				break;
> > > +
> > > +		if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK) ||
> > > +		    list_entry_is_head(tmp, &job->steps, list))
> > > +			list_add_tail(&step->list, &tmp->list);
> > > +		else
> > > +			list_add(&step->list, &tmp->list);
> > > +
> > > +		return 0;
> > > +	}
> > > +
> > > +	/* Shouldn't get here, unless the flag value is wrong. */
> > > +	WARN_ONCE(1, "%s(): Invalid flag value\n", __func__);
> > > +	return -EINVAL;
> > > +}
> > > +EXPORT_SYMBOL_GPL(media_jobs_add_job_step);
> > > +
> > > +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
> > > +			   void *data)
> > > +{
> > > +	struct media_job_dep *dep;
> > > +
> > > +	if (!ops || !ops->check_dep || !data)
> > > +		return -EINVAL;
> > > +
> > > +	guard(spinlock)(&job->lock);
> > > +
> > > +	/* Confirm the same dependency hasn't already been added */
> > > +	list_for_each_entry(dep, &job->deps, list)
> > > +		if (dep->ops == ops && dep->data == data)
> > > +			return -EINVAL;
> > > +
> > > +	dep = kzalloc(sizeof(*dep), GFP_KERNEL);
> > > +	if (!dep)
> > > +		return -ENOMEM;
> > > +
> > > +	dep->ops = ops;
> > > +	dep->data = data;
> > > +	list_add(&dep->list, &job->deps);
> > > +
> > > +	return 0;
> > > +}
> > > +EXPORT_SYMBOL_GPL(media_jobs_add_job_dep);
> > > +
> > > +static bool media_jobs_check_pending_job(struct media_job *job,
> > > +					 enum media_job_types type,
> > > +					 struct media_job_dep_ops *dep_ops,
> > > +					 void *data)
> > > +{
> > > +	struct media_job_dep *dep;
> > > +
> > > +	guard(spinlock)(&job->lock);
> > > +
> > > +	if (job->type != type)
> > > +		return false;
> > > +
> > > +	list_for_each_entry(dep, &job->deps, list) {
> > > +		if (dep->ops == dep_ops && dep->data == data) {
> > > +			if (dep->met)
> > > +				return false;
> > > +
> > > +			break;
> > > +		}
> > > +	}
> > > +
> > > +	dep->met = true;
> > > +	return true;
> > > +}
> > > +
> > > +static struct media_job *media_jobs_get_job(struct media_job_scheduler *sched,
> > > +					    enum media_job_types type,
> > > +					    struct media_job_dep_ops *dep_ops,
> > > +					    void *dep_data)
> > > +{
> > > +	struct media_job_setup_func *jsf;
> > > +	struct media_job *job;
> > > +	int ret;
> > > +
> > > +	list_for_each_entry(job, &sched->pending, list)
> > > +		if (media_jobs_check_pending_job(job, type, dep_ops, dep_data))
> > > +			return job;
> > > +
> > Thanks to your offline explanation, I got how this works now, however
> > some questions here
> >
> > The basic idea is that each driver that registers a 'setup' function
> > adds to a job, when its created, its list of dependencies.
> >
> > When a job is "try_queue" and we get here, to decide if a new job has
> > to be created or if we have to run one which is already in the pending
> > queue.
> >
> > How is this identification performed ? Each entry point (assume it's a
> > video device op) will populate the job with its own dependencies,
> > identified by the dep_ops and data address.
> >
> > We walk the 'pending' queue in the media_jobs_check_pending_job()
> > function and we search for one job not already visited from the same
> > entry point, identified by the dep_ops (the 'visited' state is kept by
> > the deps->met flag).
> >
> > Let's assume 2 video devices X and Y
> >
> > qbuf(x) -> try_queue_job() -> new job created on 'pending'
> > qbuf(x) -> try_queue_job() -> the job in the queue has 'deps->met' set, skip
> > it and create a new one
> > qbuf(y) -> try_queue_job() -> the first job in the queue has not
> > deps->set, so return it
> >
> > All in all I would describe this as: when requesting a job try to find
> > the first one not already visited by this entry point, if none is
> > available create a new one.
>
>
> Yep, all seems fine.
>
> > Now, we briefly discussed that when moving to multi-context comparing
> > dep_ops and data to identify an entry point won't be enough: buffers
> > from the same video device but from different contexts do not have to
> > be associated together. So we'll need to extend the identification
> > criteria. Also, I don't find the idea of using dep_ops and data for
> > this purpose particularly neat, as it makes mandatory to add
> > dependencies to a job in the setup function, something not all driver
> > might want to do ?
>
> I don't think it's mandatory for a driver to add dependencies to a job; the
> implication of lacking them is that whatever step the driver is running for
> the job takes no input (no buffers need to be available, no parameters need
> to have been set, no per-requisites need to have been met) in which case it
> can simply be ignored for the purposes of evaluating whether the job can be
> queued or not, because it's always ready by definition...does that make
> sense?

Yes, but what are the implications of not setting deps on
media_jobs_check_pending_job() ?

If I'm not mistaken

	if (dep->ops == dep_ops && dep->data == data) {

will now always return false, as there won't be any 'dep' that matches
with dep_ops as the driver has never called media_jobs_add_job_dep().

If so, if a video device that doesn't register deps (tbh I don't see
why it would, but..) will always match and we will overwrite the same job over
and over ? Anyway, a corner case I guess

>
> > There might be ways to handle this "track the entry point" thing that
> > could be separated by deps, making deps do what they actually are
> > described for: track dependencies to validate if a job can be run or
> > not. Before exploring options, I would like to know if this only mine
> > concern or is it shared by others.
>
>
> I do agree that a nicer way of tracking them rather than dep_ops and data
> would be better...perhaps tying it to the entry point as you've
> conceptualised here is the right thing to do, and the pointer to the
> function calling media_jobs_try_queue_job() should be passed, along with a
> context ID?
>

I think that's a possible way forward. For the time being I think
identifying a "user" of a job (iow any drivers that registers a
dependency or a step) with a pointer to the video device or to the
driver-specific types should be enough and we can easily
move it to use the video device context later.

I have patches to push if you want to see.

A bit more comments on the API below

>
> Thanks
>
> Dan
>
> >
> > > +	job = kzalloc(sizeof(*job), GFP_KERNEL);
> > > +	if (!job)
> > > +		return ERR_PTR(-ENOMEM);
> > > +
> > > +	spin_lock_init(&job->lock);
> > > +	INIT_LIST_HEAD(&job->deps);
> > > +	INIT_LIST_HEAD(&job->steps);
> > > +	job->type = type;
> > > +	job->sched = sched;
> > > +
> > > +	list_for_each_entry(jsf, &sched->setup_funcs, list) {
> > > +		if (jsf->type != type)
> > > +			continue;
> > > +
> > > +		ret = jsf->job_setup(job, jsf->data);
> > > +		if (ret) {
> > > +			kfree(job);
> > > +			return ERR_PTR(ret);
> > > +		}
> > > +	}
> > > +
> > > +	list_add_tail(&job->list, &sched->pending);
> > > +
> > > +	/* This marks the dependency as met */
> > > +	media_jobs_check_pending_job(job, type, dep_ops, dep_data);
> > > +
> > > +	return job;
> > > +}
> > > +
> > > +static void media_jobs_free_job(struct media_job *job, bool reset)
> > > +{
> > > +	struct media_job_step *step, *stmp;
> > > +	struct media_job_dep *dep, *dtmp;
> > > +
> > > +	scoped_guard(spinlock, &job->lock) {
> > > +		list_for_each_entry_safe(dep, dtmp, &job->deps, list) {
> > > +			if (reset && dep->ops->reset_dep)
> > > +				dep->ops->reset_dep(dep->data);
> > > +
> > > +			list_del(&dep->list);
> > > +			kfree(dep);
> > > +		}
> > > +
> > > +		list_for_each_entry_safe(step, stmp, &job->steps, list) {
> > > +			list_del(&step->list);
> > > +			kfree(step);
> > > +		}
> > > +	}
> > > +
> > > +	list_del(&job->list);
> > > +	kfree(job);
> > > +}
> > > +
> > > +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
> > > +			     enum media_job_types type,
> > > +			     struct media_job_dep_ops *dep_ops, void *dep_data)
> > > +{
> > > +	struct media_job_dep *dep;
> > > +	struct media_job *job;
> > > +
> > > +	if (!sched)
> > > +		return 0;
> > > +
> > > +	guard(spinlock)(&sched->lock);
> > > +
> > > +	job = media_jobs_get_job(sched, type, dep_ops, dep_data);
> > > +	if (IS_ERR(job))
> > > +		return PTR_ERR(job);
> > > +
> > > +	list_for_each_entry(dep, &job->deps, list)
> > > +		if (!dep->ops->check_dep(dep->data))
> > > +			return 0; /* Not a failure */
> > > +
> > > +	list_for_each_entry(dep, &job->deps, list)
> > > +		if (dep->ops->clear_dep)
> > > +			dep->ops->clear_dep(dep->data);
> > > +
> > > +	list_move_tail(&job->list, &sched->queue);
> > > +	queue_work(sched->async_wq, &sched->work);
> > > +
> > > +	return 0;
> > > +}
> > > +EXPORT_SYMBOL_GPL(media_jobs_try_queue_job);
> > > +
> > > +static void __media_jobs_run_jobs(struct work_struct *work)
> > > +{
> > > +	struct media_job_scheduler *sched = container_of(work,
> > > +							 struct media_job_scheduler,
> > > +							 work);
> > > +	struct media_job_step *step;
> > > +	struct media_job *job;
> > > +
> > > +	while (true) {
> > > +		scoped_guard(spinlock, &sched->lock) {
> > > +			if (list_empty(&sched->queue))
> > > +				return;
> > > +
> > > +			job = list_first_entry(&sched->queue, struct media_job,
> > > +					       list);
> > > +		}
> > > +
> > > +		list_for_each_entry(step, &job->steps, list)
> > > +			step->run_step(step->data);
> > > +
> > > +		media_jobs_free_job(job, false);
> > > +	}
> > > +}
> > > +
> > > +void media_jobs_run_jobs(struct media_job_scheduler *sched)
> > > +{
> > > +	if (!sched)
> > > +		return;
> > > +
> > > +	queue_work(sched->async_wq, &sched->work);
> > > +}
> > > +EXPORT_SYMBOL_GPL(media_jobs_run_jobs);
> > > +
> > > +static void __media_jobs_cancel_jobs(struct media_job_scheduler *sched)
> > > +{
> > > +	struct media_job *job, *jtmp;
> > > +
> > > +	list_for_each_entry_safe(job, jtmp, &sched->pending, list)
> > > +		media_jobs_free_job(job, true);
> > > +
> > > +	list_for_each_entry_safe(job, jtmp, &sched->queue, list)
> > > +		media_jobs_free_job(job, true);
> > > +}
> > > +
> > > +void media_jobs_cancel_jobs(struct media_job_scheduler *sched)
> > > +{
> > > +	if (!sched)
> > > +		return;
> > > +
> > > +	guard(spinlock)(&sched->lock);
> > > +	__media_jobs_cancel_jobs(sched);
> > > +}
> > > +EXPORT_SYMBOL_GPL(media_jobs_cancel_jobs);
> > > +
> > > +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
> > > +				  int (*job_setup)(struct media_job *job, void *data),
> > > +				  void *data, enum media_job_types type)
> > > +{
> > > +	struct media_job_setup_func *new_setup_func;
> > > +
> > > +	guard(spinlock)(&sched->lock);
> > > +
> > > +	new_setup_func = kzalloc(sizeof(*new_setup_func), GFP_KERNEL);
> > > +	if (!new_setup_func)
> > > +		return -ENOMEM;
> > > +
> > > +	new_setup_func->type = type;
> > > +	new_setup_func->job_setup = job_setup;
> > > +	new_setup_func->data = data;
> > > +	list_add_tail(&new_setup_func->list, &sched->setup_funcs);
> > > +
> > > +	return 0;
> > > +}
> > > +EXPORT_SYMBOL_GPL(media_jobs_add_job_setup_func);
> > > +
> > > +static void __media_jobs_put_scheduler(struct kref *kref)
> > > +{
> > > +	struct media_job_scheduler *sched =
> > > +		container_of(kref, struct media_job_scheduler, kref);
> > > +	struct media_job_setup_func *func, *ftmp;
> > > +
> > > +	cancel_work_sync(&sched->work);
> > > +	destroy_workqueue(sched->async_wq);
> > > +
> > > +	scoped_guard(spinlock, &sched->lock) {
> > > +		__media_jobs_cancel_jobs(sched);
> > > +
> > > +		list_for_each_entry_safe(func, ftmp, &sched->setup_funcs, list) {
> > > +			list_del(&func->list);
> > > +			kfree(func);
> > > +		}
> > > +	}
> > > +
> > > +	list_del(&sched->list);
> > > +	kfree(sched);
> > > +}
> > > +
> > > +void media_jobs_put_scheduler(struct media_job_scheduler *sched)
> > > +{
> > > +	kref_put(&sched->kref, __media_jobs_put_scheduler);
> > > +}
> > > +EXPORT_SYMBOL_GPL(media_jobs_put_scheduler);
> > > +
> > > +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev)
> > > +{
> > > +	struct media_job_scheduler *sched;
> > > +	char workqueue_name[32];
> > > +	int ret;
> > > +
> > > +	guard(mutex)(&media_job_schedulers_lock);
> > > +
> > > +	list_for_each_entry(sched, &media_job_schedulers, list) {
> > > +		if (sched->mdev == mdev) {
> > > +			kref_get(&sched->kref);
> > > +			return sched;
> > > +		}
> > > +	}
> > > +
> > > +	ret = snprintf(workqueue_name, sizeof(workqueue_name),
> > > +		       "mc jobs (%s)", mdev->driver_name);
> > > +	if (!ret)
> > > +		return ERR_PTR(-EINVAL);
> > > +
> > > +	sched = kzalloc(sizeof(*sched), GFP_KERNEL);
> > > +	if (!sched)
> > > +		return ERR_PTR(-ENOMEM);
> > > +
> > > +	sched->async_wq = alloc_workqueue(workqueue_name, 0, 0);
> > > +	if (!sched->async_wq) {
> > > +		kfree(sched);
> > > +		return ERR_PTR(-EINVAL);
> > > +	}
> > > +
> > > +	sched->mdev = mdev;
> > > +	kref_init(&sched->kref);
> > > +	spin_lock_init(&sched->lock);
> > > +	INIT_LIST_HEAD(&sched->setup_funcs);
> > > +	INIT_LIST_HEAD(&sched->pending);
> > > +	INIT_LIST_HEAD(&sched->queue);
> > > +	INIT_WORK(&sched->work, __media_jobs_run_jobs);
> > > +
> > > +	list_add_tail(&sched->list, &media_job_schedulers);
> > > +
> > > +	return sched;
> > > +}
> > > +EXPORT_SYMBOL_GPL(media_jobs_get_scheduler);
> > > +
> > > +LIST_HEAD(media_job_schedulers);
> > > +
> > > +/* Synchronise access to the global schedulers list */
> > > +DEFINE_MUTEX(media_job_schedulers_lock);
> > > diff --git a/include/media/media-jobs.h b/include/media/media-jobs.h
> > > new file mode 100644
> > > index 000000000000..a97270861251
> > > --- /dev/null
> > > +++ b/include/media/media-jobs.h
> > > @@ -0,0 +1,354 @@
> > > +/* SPDX-License-Identifier: GPL-2.0-only */
> > > +/*
> > > + * Media jobs framework
> > > + *
> > > + * Copyright 2025 Ideas on Board Oy
> > > + *
> > > + * Author: Daniel Scally <dan.scally@ideasonboard.com>
> > > + */
> > > +

Could you include the header in the .c file first to make sure it's
self-contained ?

> > > +#include <linux/kref.h>
> > > +#include <linux/list.h>
> > > +#include <linux/mutex.h>
> > > +#include <linux/spinlock.h>
> > > +#include <linux/types.h>
> > > +#include <linux/workqueue.h>
> > > +
> > > +#ifndef _MEDIA_JOBS_H
> > > +#define _MEDIA_JOBS_H

Place this at the beginning of the file

> > > +
> > > +struct media_device;
> > > +struct media_entity;
> > > +struct media_job;
> > > +struct media_job_dep;
> > > +
> > > +/**
> > > + * define MEDIA_JOBS_FL_STEP_ANYWHERE - \
> > > + *    Flag a media job step as able to run anytime
> > > + *
> > > + * This flag informs the framework that a job step does not need a particular
> > > + * position in the list of job steps and can be placed anywhere.
> > > + */
> > > +#define MEDIA_JOBS_FL_STEP_ANYWHERE			BIT(0)
> > > +
> > > +/**
> > > + * define MEDIA_JOBS_FL_STEP_FROM_FRONT - \
> > > + *    Flag a media job step as needing to be placed near the start of the list
> > > + *
> > > + * This flag informs the framework that a job step needs to be placed at a set
> > > + * position from the start of the list of job steps.
> > > + */
> > > +#define MEDIA_JOBS_FL_STEP_FROM_FRONT			BIT(1)
> > > +
> > > +/**
> > > + * define MEDIA_JOBS_FL_STEP_FROM_BACK - \
> > > + *    Flag a media job step as needing to be placed near the end of the list
> > > + *
> > > + * This flag informs the framework that a job step needs to be placed at a set
> > > + * position from the end of the list of job steps.
> > > + */
> > > +#define MEDIA_JOBS_FL_STEP_FROM_BACK			BIT(2)
> > > +
> > > +/**
> > > + * enum media_job_types - Type of media job
> > > + *
> > > + * @MEDIA_JOB_TYPE_PIPELINE_PULSE:	A data event moving through the media
> > > + *					pipeline
> > > + *
> > > + * This enumeration details different types of media jobs. The type can be used
> > > + * to differentiate between which steps and dependencies a driver needs to add
> > > + * to a job when it is created.
> > > + */
> > > +enum media_job_types {
> > > +	MEDIA_JOB_TYPE_PIPELINE_PULSE,
> > > +};

I'm wondering if we could omit type for the time being.
A little trick would be to use variadic macros to allow users to not
specify the type and default it to MEDIA_JOB_TYPE_PIPELINE_PULSE and
at the same time allow future users of other types to specify them
later on

Something like (not based on this version of the patch sorry)

-int media_jobs_try_queue_job(struct media_job_scheduler *sched,
-                            enum media_job_types type,
-                            void *cookie);
+int __media_jobs_try_queue_job(struct media_job_scheduler *sched,
+                              void *cookie, enum media_job_types type);
+
+#define __media_jobs_try_queue_job_notype(sched, cookie)       \
+       __media_jobs_try_queue_job(sched, cookie, MEDIA_JOB_TYPE_PIPELINE_PULSE)
+#define __media_jobs_try_queue_job_type(sched, cookie, type)   \
+       __media_jobs_try_queue_job(sched, cookie, type)

+#define MEDIA_JOBS_TRY_QUEUE_EXPAND(_1, _2, _3, FUNC, ...) FUNC
+#define media_jobs_try_queue_job(...)                                  \
+       MEDIA_JOBS_TRY_QUEUE_EXPAND(__VA_ARGS__,                        \
+                                   __media_jobs_try_queue_job_type,    \
+                                   __media_jobs_try_queue_job_notype)  \
+                                   (__VA_ARGS__)

Maybe not good looking but allows to call both

media_jobs_try_queue_job(sched, cookie, type) and
media_jobs_try_queue_job(sched, cookie)

(this could be done to all functions that accepts a type, apart the
job setup functions)

> > > +
> > > +/**
> > > + * struct media_job_scheduler - A job scheduler for a particular media device
> > > + *
> > > + * @mdev:		Media device this scheduler is for
> > > + * @list:		List head to attach to the global list of schedulers
> > > + * @kref:		Reference counter
> > > + * @lock:		Lock to protect access to the scheduler
> > > + * @setup_funcs:	List of &struct media_job_setup_func to populate jobs
> > > + * @pending:		List of &struct media_jobs created but not yet queued
> > > + * @queue:		List of &struct media_jobs queued to the scheduler
> > > + * @work:		Work item to run the jobs
> > > + * @async_wq:		Workqueue to run the work on
> > > + *
> > > + * This struct is the main job scheduler struct - drivers wanting to use this
> > > + * framework should acquire an instance through media_jobs_get_scheduler() and
> > > + * subsequently populate it with job setup functions.
> > > + */
> > > +struct media_job_scheduler {
> > > +	struct media_device *mdev;
> > > +	struct list_head list;
> > > +	struct kref kref;
> > > +
> > > +	spinlock_t lock; /* Synchronise access to the struct's lists */
> > > +	struct list_head setup_funcs;
> > > +	struct list_head pending;
> > > +	struct list_head queue;

Empty line please

> > > +	struct work_struct work;
> > > +	struct workqueue_struct *async_wq;
> > > +};

This type should be defined inside the .c file and not exposed ?

> > > +
> > > +/**
> > > + * struct media_job_setup_func - A function to populate a media job with steps
> > > + *				 and dependencies
> > > + *
> > > + * @list:	The list object to attach to the scheduler
> > > + * @type:	The &enum media_job_types that this function populates a job for
> > > + * @job_setup:	Function pointer to the driver's job setup function
> > > + * @data:	Pointer to the driver data for use with @job_setup
> > > + *
> > > + * This struct holds data about the functions a driver registers with the jobs
> > > + * framework in order to populate a new job with steps and dependencies.
> > > + */
> > > +struct media_job_setup_func {
> > > +	struct list_head list;
> > > +	enum media_job_types type;
> > > +	int (*job_setup)(struct media_job *job, void *data);

You could define a type and use it. I would do that for all functions
where it makes sense

> > > +	void *data;
> > > +};
> > > +
> > > +/**
> > > + * struct media_job - A representation of a job to be run through the pipeline
> > > + *
> > > + * @lock:	Lock to protect access to the job's lists
> > > + * @list:	List head to attach the job to &struct media_job_scheduler in
> > > + *		either the pending or queue lists
> > > + * @steps:	List of &struct media_job_step to run the job
> > > + * @deps:	List of &struct media_job_dep to check that the job can be
> > > + *		queued
> > > + * @sched:	Pointer to the media job scheduler
> > > + * @type:	The type of the job
> > > + *
> > > + * This struct holds lists of steps that need to be performed to carry out a
> > > + * job in the pipeline. A separate list of dependencies allows the queueing of
> > > + * the job to be delayed until all drivers are ready to carry it out.
> > > + */
> > > +struct media_job {
> > > +	spinlock_t lock; /* Synchronise access to the struct's lists 6*/
> > > +	struct list_head list;
> > > +	struct list_head steps;
> > > +	struct list_head deps;
> > > +	struct media_job_scheduler *sched;
> > > +	enum media_job_types type;

nit: list_head are usually at the end, at least it seems to to me.
Doesn't

	struct media_job_scheduler *sched;
	enum media_job_types type;

	spinlock_t lock; /* Synchronise access to the struct's lists 6*/
	struct list_head list;
	struct list_head steps;
	struct list_head deps;

look better ?

};
> > > +
> > > +/**
> > > + * struct media_job_step - A holder for a function to run as part of a job
> > > + *
> > > + * @list:	List head to attach the job step to a &struct media_job.steps
> > > + * @run_step:	The function to run to perform the step
> > > + * @data:	Data to pass to the .run_step() function
> > > + * @flags:	Flags to control how the step is ordered within the job's list
> > > + *		of steps
> > > + * @pos:	Position indicator to control how the step is ordered within the
> > > + *		job's list of steps
> > > + *
> > > + * This struct defines a function that needs to be run as part of the execution
> > > + * of a job in a media pipeline, along with information that help the scheduler
> > > + * determine what order it should be ran in in reference to the other steps that
> > > + * are part of the same job.
> > > + */
> > > +struct media_job_step {
> > > +	struct list_head list;
> > > +	void (*run_step)(void *data);
> > > +	void *data;
> > > +	unsigned int flags;
> > > +	unsigned int pos;
> > > +};
> > > +
> > > +/**
> > > + * struct media_job_dep_ops - Operations to manage a media job dependency

mmm are these dep_ops or job_ops ?

> > > + *
> > > + * @check_dep:	A function to ask the driver whether the dependency is met
> > > + * @clear_dep:	A function to tell the driver that the job has been queued

Took me a while to get this. check_dep asks the driver to check,
reset_dep asks the driver to reset, and then clear_dep asks the driver
to ... clear ?

I would call this something like job_ready (which could possibly be
paired with device_run and job_abort and we have an identical replica
of v4l2_m2m_ops. Does this hint something ?)

Anyway, not feeling strong about this naming thing, but yes, if
clear_dep could be changed it would be nice


> > > + * @reset_dep:	A function to tell the driver that the job has been cancelled
> > > + *
> > > + * Media jobs have dependencies, such as requiring buffers to be queued. These
> > > + * operations allow a driver to define how the media jobs framework should check
> > > + * whether or not those dependencies are met and how it should inform them that
> > > + * it is taking action based on the state of those dependencies.
> > > + */
> > > +struct media_job_dep_ops {
> > > +	bool (*check_dep)(void *data);
> > > +	void (*clear_dep)(void *data);
> > > +	void (*reset_dep)(void *data);
> > > +};
> > > +
> > > +/**
> > > + * struct media_job_dep - Representation of media job dependency
> > > + *
> > > + * @list:	List head to attach to a &struct media_job.deps
> > > + * @ops:	A pointer to the dependency's operations functions
> > > + * @met:	A flag to record whether or not the dependency is met
> > > + * @data:	Data to pass to the dependency's operations
> > > + *
> > > + * This struct represents a dependency of a media job. The operations member
> > > + * holds pointers to functions allowing the framework to interact with the
> > > + * driver to check whether or not the dependency is met.
> > > + */
> > > +struct media_job_dep {
> > > +	struct list_head list;
> > > +	struct media_job_dep_ops *ops;
> > > +	bool met;
> > > +	void *data;

	struct media_job_dep_ops *ops;
	void *data;
	bool met;

	struct list_head list;

> > > +};
> > > +
> > > +/**
> > > + * media_jobs_try_queue_job - Try to queue a &struct media_job
> > > + *
> > > + * @sched:	Pointer to the job scheduler
> > > + * @type:	The type of the media job
> > > + * @dep_ops:	A pointer to the dependency operations for this job
> > > + * @dep_data:	A pointer to the dependency data for this job
> > > + *
> > > + * Try to queue a media job with the scheduler. This function should be called
> > > + * by the drivers whenever a dependency for a media job is met - for example
> > > + * when a buffer is queued to the driver. The framework will check to see if an
> > > + * existing job on the scheduler's pending list shares the same type, dependency
> > > + * operations and dependency data. If it does then that existing job will be
> > > + * considered. If there is no extant job with those same parameters, a new job
> > > + * is allocated and populated by calling the setup functions registered with
> > > + * the framework.

This is about the internals, I wouldn't move part of the documentation
to the implementation.

> > > + *
> > > + * The function iterates over the dependencies that are registered with the job
> > > + * and checks to see if they are met. If they're all met, they're cleared and
> > > + * the job is placed onto the scheduler's queue.
> > > + *
> > > + * To help reduce conditionals in drivers where a driver supports both the use
> > > + * of the media jobs framework and operation without it, this function is a no
> > > + * op if @sched is NULL.

Wouldn't a driver written to use the media_jobs framework use the
framework regardless on the presence of other drivers that use the same
scheduler on the media device ? In what case sched would be null ?


> > > + *
> > > + * Return: 0 on success or a negative error number
> > > + */
> > > +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
> > > +			     enum media_job_types type,
> > > +			     struct media_job_dep_ops *dep_ops, void *dep_data);
> > > +
> > > +/**
> > > + * media_jobs_add_job_step - Add a step to a media job
> > > + *
> > > + * @job:	Pointer to the &struct media_job
> > > + * @run_step:	Pointer to the function to run to execute the step
> > > + * @data:	Pointer to the data to pass to @run_ste
> > > + * @flags:	One of the MEDIA_JOBS_FL_STEP_* flags
> > > + * @pos:	A position indicator to use with @flags
> > > + *
> > > + * This function adds a step to the job and should be called from the drivers'
> > > + * job setup functions as registered with the framework through
> > > + * media_jobs_add_job_setup_func(). The @flags and @pos parameters are used
> > > + * to determine the ordering of the steps within the job:
> > > + *
> > > + * If @flags has the MEDIA_JOBS_FL_STEP_ANYWHERE bit set, the step is placed
> > > + * after all steps with MEDIA_JOBS_FL_STEP_FROM_FRONT and before all steps with
> > > + * MEDIA_JOBS_FL_STEP_FROM_BACK bit set, but otherwise in whatever order this
> > > + * function is called.
> > > + *
> > > + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_FRONT bit set then the step is
> > > + * placed @pos steps from the front of the list. Attempting to place multiple
> > > + * steps in the same position will result in an error.
> > > + *
> > > + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_BACK bit set then the step is
> > > + * placed @pos steps from the back of the list. Attempting to place multiple
> > > + * steps in the same position will result in an error.
> > > + *
> > > + * Return: 0 on success or a negative error number
> > > + */
> > > +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
> > > +			    void *data, unsigned int flags, unsigned int pos);
> > > +
> > > +/**
> > > + * media_jobs_add_job_dep - Add a dependency to a media job
> > > + *
> > > + * @job:	Pointer to the &struct media_job
> > > + * @ops:	Pointer to the &struct media_job_dep_ops
> > > + * @data:	Pointer to the data to pass to the dependency's operations
> > > + *
> > > + * This function adds a dependency to the job and should be called from the
> > > + * drivers job setup functions as registered with the framework through the
> > > + * media_jobs_add_job_setup_func() function.
> > > + *
> > > + * Return: 0 on success or a negative error number
> > > + */
> > > +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
> > > +			   void *data);
> > > +

We'll get back to this when discussing how to identify a dependency
user

> > > +/**
> > > + * media_jobs_add_job_setup_func - Add a function that populates a media job
> > > + *
> > > + * @sched:	Pointer to the media jobs scheduler
> > > + * @job_setup:	Pointer to the new job setup function
> > > + * @data:	Data to pass to the job setup function
> > > + * @type:	The type of job that this function should be called for
> > > + *
> > > + * Drivers that wish to utilise the framework need to use this function to
> > > + * register a callback that adds job steps and dependencies when one is created.
> > > + * The function must call media_jobs_add_job_step() and media_jobs_add_job_dep()
> > > + * to populate the job.
> > > + *
> > > + * Return: 0 on success or a negative error number
> > > + */
> > > +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,

If you move the scheduler to the media dev, then all these function
can operate on the media dev and the scheduler would be internal to
the media dev

> > > +				  int (*job_setup)(struct media_job *job, void *data),

define a type for this as well maybe

> > > +				  void *data, enum media_job_types type);
> > > +
> > > +/**
> > > + * media_jobs_put_scheduler - Put a reference to the media jobs scheduler
> > > + *
> > > + * @sched:	Pointer to the media jobs scheduler
> > > + *
> > > + * This function puts a reference to the media jobs scheduler, and is intended
> > > + * to be called in error and exit paths for consuming drivers
> > > + */
> > > +void media_jobs_put_scheduler(struct media_job_scheduler *sched);
> > > +
> > > +/**
> > > + * media_jobs_get_scheduler - Get a media jobs scheduler
> > > + *
> > > + * @mdev:	Pointer to the media device associated with the scheduler
> > > + *
> > > + * This function gets a pointer to a &struct media_job_scheduler associated with
> > > + * the media device passed to @mdev. If one is not available then it is
> > > + * allocated and returned. This allows multiple drivers sharing a media graph to
> > > + * work with the same media job scheduler.
> > > + *
> > > + * Return: 0 on success or a negative error number
> > > + */
> > > +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev);
> > > +
> > > +/**
> > > + * media_jobs_run_jobs - Run any media jobs that are ready in the queue
> > > + *
> > > + * @sched:	Pointer to the media job scheduler
> > > + *
> > > + * This function triggers the workqueue that processes any jobs that have been
> > > + * queued, and should be called whenever the pipeline is ready to do so.
> > > + *
> > > + * To help reduce conditionals in drivers where a driver supports both the use
> > > + * of the media jobs framework and operation without it, this function is a no
> > > + * op if @sched is NULL.
> > > + */
> > > +void media_jobs_run_jobs(struct media_job_scheduler *sched);
> > > +
> > > +/**
> > > + * media_jobs_cancel_jobs - cancel all waiting jobs
> > > + *
> > > + * @sched:	Pointer to the media job scheduler
> > > + *
> > > + * This function iterates over any pending and queued jobs, resets their
> > > + * dependencies and frees the job
> > > + *
> > > + * To help reduce conditionals in drivers where a driver supports both the use
> > > + * of the media jobs framework and operation without it, this function is a no
> > > + * op if @sched is NULL.
> > > + */
> > > +void media_jobs_cancel_jobs(struct media_job_scheduler *sched);
> > > +
> > > +extern struct list_head media_job_schedulers;
> > > +extern struct mutex media_job_schedulers_lock;

These can (and should) be made static internal to mc-jobs.c

I've pushed a few patches for you to look at in the meantime, very
nice overall, I think even drivers without extenal dependencies could
use this framework and simplify the job (I've already seen in at
least 3 drivers) to maintain queues of jobs where to associate buffers
into. One step at the time :)

> > > +
> > > +#endif /* _MEDIA_JOBS_H */
> > > --
> > > 2.34.1
> > >
> > >

Re: [PATCH 1/3] media: mc: entity: Add pipeline_started/stopped ops

[Nicolas Dufresne]

Hi,

Le lundi 19 mai 2025 à 15:04 +0100, Daniel Scally a écrit :
> Add two new members to struct media_entity_operations, along with new
> functions in media-entity.c to traverse a media pipeline and call the
> new operations. The new functions are intended to be used to signal
> to a media pipeline that it has fully started, with the entity ops
> allowing drivers to define some action to be taken when those
> conditions are met.
> 
> The combination of the new functions and operations allows drivers
> which are part of a multi-driver pipeline to delay actually starting
> streaming until all of the conditions for streaming succcessfully are
> met across all drivers.
> 
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
>  drivers/media/mc/mc-entity.c | 45 ++++++++++++++++++++++++++++++++++++
>  include/media/media-entity.h | 24 +++++++++++++++++++
>  2 files changed, 69 insertions(+)
> 
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 045590905582..e36b1710669d 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -1053,6 +1053,51 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
>  }
>  EXPORT_SYMBOL_GPL(__media_pipeline_entity_iter_next);
>  
> +int media_pipeline_started(struct media_pipeline *pipe)
> +{
> +	struct media_pipeline_entity_iter iter;
> +	struct media_entity *entity;
> +	int ret;
> +
> +	ret = media_pipeline_entity_iter_init(pipe, &iter);
> +	if (ret)
> +		return ret;
> +
> +	media_pipeline_for_each_entity(pipe, &iter, entity) {
> +		ret = media_entity_call(entity, pipeline_started);
> +		if (ret && ret != -ENOIOCTLCMD)
> +			goto err_notify_stopped;
> +	}

Would this be more useful if it had a specified traversal order ? Perhaps
sink to source traversal?

Nicolas

> +
> +	media_pipeline_entity_iter_cleanup(&iter);
> +
> +	return ret == -ENOIOCTLCMD ? 0 : ret;
> +
> +err_notify_stopped:
> +	media_pipeline_stopped(pipe);
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(media_pipeline_started);
> +
> +int media_pipeline_stopped(struct media_pipeline *pipe)
> +{
> +	struct media_pipeline_entity_iter iter;
> +	struct media_entity *entity;
> +	int ret;
> +
> +	ret = media_pipeline_entity_iter_init(pipe, &iter);
> +	if (ret)
> +		return ret;
> +
> +	media_pipeline_for_each_entity(pipe, &iter, entity)
> +		media_entity_call(entity, pipeline_stopped);
> +
> +	media_pipeline_entity_iter_cleanup(&iter);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(media_pipeline_stopped);
> +
>  /* -----------------------------------------------------------------------------
>   * Links management
>   */
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 64cf590b1134..e858326b95cb 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -269,6 +269,10 @@ struct media_pad {
>   *			media_entity_has_pad_interdep().
>   *			Optional: If the operation isn't implemented all pads
>   *			will be considered as interdependent.
> + * @pipeline_started:	Notify this entity that the pipeline it is a part of has
> + *			been started
> + * @pipeline_stopped:	Notify this entity that the pipeline it is a part of has
> + *			been stopped
>   *
>   * .. note::
>   *
> @@ -284,6 +288,8 @@ struct media_entity_operations {
>  	int (*link_validate)(struct media_link *link);
>  	bool (*has_pad_interdep)(struct media_entity *entity, unsigned int pad0,
>  				 unsigned int pad1);
> +	int (*pipeline_started)(struct media_entity *entity);
> +	void (*pipeline_stopped)(struct media_entity *entity);
>  };
>  
>  /**
> @@ -1261,6 +1267,24 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
>  	     entity != NULL;							\
>  	     entity = __media_pipeline_entity_iter_next((pipe), iter, entity))
>  
> +/**
> + * media_pipeline_started - Inform entities in a pipeline that it has started
> + * @pipe:	The pipeline
> + *
> + * Iterate on all entities in a media pipeline and call their pipeline_started
> + * member of media_entity_operations.
> + */
> +int media_pipeline_started(struct media_pipeline *pipe);
> +
> +/**
> + * media_pipeline_stopped - Inform entities in a pipeline that it has stopped
> + * @pipe:	The pipeline
> + *
> + * Iterate on all entities in a media pipeline and call their pipeline_stopped
> + * member of media_entity_operations.
> + */
> +int media_pipeline_stopped(struct media_pipeline *pipe);
> +
>  /**
>   * media_pipeline_alloc_start - Mark a pipeline as streaming
>   * @pad: Starting pad

Re: [PATCH 2/3] media: mc: Add media jobs framework

[Nicolas Dufresne]

Hi,

Le lundi 19 mai 2025 à 15:04 +0100, Daniel Scally a écrit :
> Add a new framework to the media subsystem describing media jobs.
> This framework is intended to be able to model the interactions
> between multiple different drivers that need to be run in concert
> to fully control a media pipeline, for example an ISP driver and a
> driver controlling a DMA device that feeds data from memory in to
> that ISP.
> 
> The new framework allows all drivers involved to add explicit steps
> that need to be performed, and to control the ordering of those steps
> precisely. Once the job with its steps has been created it's then
> scheduled to be run with a workqueue which executes each step in the
> defined order.
> 
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
>  drivers/media/mc/Makefile  |   2 +-
>  drivers/media/mc/mc-jobs.c | 446 +++++++++++++++++++++++++++++++++++++
>  include/media/media-jobs.h | 354 +++++++++++++++++++++++++++++
>  3 files changed, 801 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/media/mc/mc-jobs.c
>  create mode 100644 include/media/media-jobs.h
> 
> diff --git a/drivers/media/mc/Makefile b/drivers/media/mc/Makefile
> index 2b7af42ba59c..9148bbfd1578 100644
> --- a/drivers/media/mc/Makefile
> +++ b/drivers/media/mc/Makefile
> @@ -1,7 +1,7 @@
>  # SPDX-License-Identifier: GPL-2.0
>  
>  mc-objs	:= mc-device.o mc-devnode.o mc-entity.o \
> -	   mc-request.o
> +	   mc-jobs.o mc-request.o
>  
>  ifneq ($(CONFIG_USB),)
>  	mc-objs += mc-dev-allocator.o
> diff --git a/drivers/media/mc/mc-jobs.c b/drivers/media/mc/mc-jobs.c
> new file mode 100644
> index 000000000000..1f04cdf63d27
> --- /dev/null
> +++ b/drivers/media/mc/mc-jobs.c
> @@ -0,0 +1,446 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Media jobs framework
> + *
> + * Copyright 2025 Ideas on Board Oy
> + *
> + * Author: Daniel Scally <dan.scally@ideasonboard.com>
> + */
> +
> +#include <linux/cleanup.h>
> +#include <linux/kref.h>
> +#include <linux/list.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +
> +#include <media/media-device.h>
> +#include <media/media-entity.h>
> +#include <media/media-jobs.h>
> +
> +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
> +			    void *data, unsigned int flags, unsigned int pos)
> +{
> +	struct media_job_step *step, *tmp;
> +	unsigned int num = flags;
> +	unsigned int count = 0;
> +
> +	guard(spinlock)(&job->lock);
> +
> +	if (!flags) {
> +		WARN_ONCE(1, "%s(): No flag bits set\n", __func__);
> +		return -EINVAL;
> +	}
> +
> +	/* Count the number of set flags; they're mutually exclusive. */
> +	while (num) {
> +		num &= (num - 1);
> +		count++;
> +	}

Can this be replaced by hweight_long() or hweight32() ?

just a drive by comment,
Nicolas

> +
> +	if (count > 1) {
> +		WARN_ONCE(1, "%s(): Multiple flag bits set\n", __func__);
> +		return -EINVAL;
> +	}
> +
> +	step = kzalloc(sizeof(*step), GFP_KERNEL);
> +	if (!step)
> +		return -ENOMEM;
> +
> +	step->run_step = run_step;
> +	step->data = data;
> +	step->flags = flags;
> +	step->pos = pos;
> +
> +	/*
> +	 * We need to decide where to place the step. If the list is empty that
> +	 * is really easy (and also the later code is much easier if the code is
> +	 * guaranteed not to be empty...)
> +	 */
> +	if (list_empty(&job->steps)) {
> +		list_add_tail(&step->list, &job->steps);
> +		return 0;
> +	}
> +
> +	/*
> +	 * If we've been asked to place it at a specific position from the end
> +	 * of the list, we cycle back through it until either we exhaust the
> +	 * list or find an entry that needs to go further from the back than the
> +	 * new one.
> +	 */
> +	if ((flags & MEDIA_JOBS_FL_STEP_FROM_BACK)) {
> +		list_for_each_entry_reverse(tmp, &job->steps, list) {
> +			if (tmp->flags == flags && tmp->pos == pos)
> +				return -EINVAL;
> +
> +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_BACK ||
> +			    tmp->pos > pos)
> +				break;
> +		}
> +
> +		/*
> +		 * If the entry we broke on is also one placed from the back and
> +		 * should be closer to the back than the new one, we place the
> +		 * new one in front of it...otherwise place the new one behind
> +		 * it.
> +		 */
> +		if (tmp->flags == flags && tmp->pos < pos)
> +			list_add_tail(&step->list, &tmp->list);
> +		else
> +			list_add(&step->list, &tmp->list);
> +
> +		return 0;
> +	}
> +
> +	/*
> +	 * If we've been asked to place it a specific position from the front of
> +	 * the list we do the same kind of operation, but going from the front
> +	 * instead.
> +	 */
> +	if (flags & MEDIA_JOBS_FL_STEP_FROM_FRONT) {
> +		list_for_each_entry(tmp, &job->steps, list) {
> +			if (tmp->flags == flags && tmp->pos == pos)
> +				return -EINVAL;
> +
> +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_FRONT ||
> +			    tmp->pos > pos)
> +				break;
> +		}
> +
> +		/*
> +		 * If the entry we broke on is also placed from the front and
> +		 * should be closed to the front than the new one, we place the
> +		 * new one behind it, otherwise in front of it.
> +		 */
> +		if (tmp->flags == flags && tmp->pos < pos)
> +			list_add(&step->list, &tmp->list);
> +		else
> +			list_add_tail(&step->list, &tmp->list);
> +
> +		return 0;
> +	}
> +
> +	/*
> +	 * If the step is flagged as "can go anywhere" we just need to try to
> +	 * find the first "from the back" entry and add it immediately before
> +	 * that. If we can't find one, add it after whatever we did find.
> +	 */
> +	if (flags & MEDIA_JOBS_FL_STEP_ANYWHERE) {
> +		list_for_each_entry(tmp, &job->steps, list)
> +			if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK))
> +				break;
> +
> +		if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK) ||
> +		    list_entry_is_head(tmp, &job->steps, list))
> +			list_add_tail(&step->list, &tmp->list);
> +		else
> +			list_add(&step->list, &tmp->list);
> +
> +		return 0;
> +	}
> +
> +	/* Shouldn't get here, unless the flag value is wrong. */
> +	WARN_ONCE(1, "%s(): Invalid flag value\n", __func__);
> +	return -EINVAL;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_add_job_step);
> +
> +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
> +			   void *data)
> +{
> +	struct media_job_dep *dep;
> +
> +	if (!ops || !ops->check_dep || !data)
> +		return -EINVAL;
> +
> +	guard(spinlock)(&job->lock);
> +
> +	/* Confirm the same dependency hasn't already been added */
> +	list_for_each_entry(dep, &job->deps, list)
> +		if (dep->ops == ops && dep->data == data)
> +			return -EINVAL;
> +
> +	dep = kzalloc(sizeof(*dep), GFP_KERNEL);
> +	if (!dep)
> +		return -ENOMEM;
> +
> +	dep->ops = ops;
> +	dep->data = data;
> +	list_add(&dep->list, &job->deps);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_add_job_dep);
> +
> +static bool media_jobs_check_pending_job(struct media_job *job,
> +					 enum media_job_types type,
> +					 struct media_job_dep_ops *dep_ops,
> +					 void *data)
> +{
> +	struct media_job_dep *dep;
> +
> +	guard(spinlock)(&job->lock);
> +
> +	if (job->type != type)
> +		return false;
> +
> +	list_for_each_entry(dep, &job->deps, list) {
> +		if (dep->ops == dep_ops && dep->data == data) {
> +			if (dep->met)
> +				return false;
> +
> +			break;
> +		}
> +	}
> +
> +	dep->met = true;
> +	return true;
> +}
> +
> +static struct media_job *media_jobs_get_job(struct media_job_scheduler *sched,
> +					    enum media_job_types type,
> +					    struct media_job_dep_ops *dep_ops,
> +					    void *dep_data)
> +{
> +	struct media_job_setup_func *jsf;
> +	struct media_job *job;
> +	int ret;
> +
> +	list_for_each_entry(job, &sched->pending, list)
> +		if (media_jobs_check_pending_job(job, type, dep_ops, dep_data))
> +			return job;
> +
> +	job = kzalloc(sizeof(*job), GFP_KERNEL);
> +	if (!job)
> +		return ERR_PTR(-ENOMEM);
> +
> +	spin_lock_init(&job->lock);
> +	INIT_LIST_HEAD(&job->deps);
> +	INIT_LIST_HEAD(&job->steps);
> +	job->type = type;
> +	job->sched = sched;
> +
> +	list_for_each_entry(jsf, &sched->setup_funcs, list) {
> +		if (jsf->type != type)
> +			continue;
> +
> +		ret = jsf->job_setup(job, jsf->data);
> +		if (ret) {
> +			kfree(job);
> +			return ERR_PTR(ret);
> +		}
> +	}
> +
> +	list_add_tail(&job->list, &sched->pending);
> +
> +	/* This marks the dependency as met */
> +	media_jobs_check_pending_job(job, type, dep_ops, dep_data);
> +
> +	return job;
> +}
> +
> +static void media_jobs_free_job(struct media_job *job, bool reset)
> +{
> +	struct media_job_step *step, *stmp;
> +	struct media_job_dep *dep, *dtmp;
> +
> +	scoped_guard(spinlock, &job->lock) {
> +		list_for_each_entry_safe(dep, dtmp, &job->deps, list) {
> +			if (reset && dep->ops->reset_dep)
> +				dep->ops->reset_dep(dep->data);
> +
> +			list_del(&dep->list);
> +			kfree(dep);
> +		}
> +
> +		list_for_each_entry_safe(step, stmp, &job->steps, list) {
> +			list_del(&step->list);
> +			kfree(step);
> +		}
> +	}
> +
> +	list_del(&job->list);
> +	kfree(job);
> +}
> +
> +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
> +			     enum media_job_types type,
> +			     struct media_job_dep_ops *dep_ops, void *dep_data)
> +{
> +	struct media_job_dep *dep;
> +	struct media_job *job;
> +
> +	if (!sched)
> +		return 0;
> +
> +	guard(spinlock)(&sched->lock);
> +
> +	job = media_jobs_get_job(sched, type, dep_ops, dep_data);
> +	if (IS_ERR(job))
> +		return PTR_ERR(job);
> +
> +	list_for_each_entry(dep, &job->deps, list)
> +		if (!dep->ops->check_dep(dep->data))
> +			return 0; /* Not a failure */
> +
> +	list_for_each_entry(dep, &job->deps, list)
> +		if (dep->ops->clear_dep)
> +			dep->ops->clear_dep(dep->data);
> +
> +	list_move_tail(&job->list, &sched->queue);
> +	queue_work(sched->async_wq, &sched->work);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_try_queue_job);
> +
> +static void __media_jobs_run_jobs(struct work_struct *work)
> +{
> +	struct media_job_scheduler *sched = container_of(work,
> +							 struct media_job_scheduler,
> +							 work);
> +	struct media_job_step *step;
> +	struct media_job *job;
> +
> +	while (true) {
> +		scoped_guard(spinlock, &sched->lock) {
> +			if (list_empty(&sched->queue))
> +				return;
> +
> +			job = list_first_entry(&sched->queue, struct media_job,
> +					       list);
> +		}
> +
> +		list_for_each_entry(step, &job->steps, list)
> +			step->run_step(step->data);
> +
> +		media_jobs_free_job(job, false);
> +	}
> +}
> +
> +void media_jobs_run_jobs(struct media_job_scheduler *sched)
> +{
> +	if (!sched)
> +		return;
> +
> +	queue_work(sched->async_wq, &sched->work);
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_run_jobs);
> +
> +static void __media_jobs_cancel_jobs(struct media_job_scheduler *sched)
> +{
> +	struct media_job *job, *jtmp;
> +
> +	list_for_each_entry_safe(job, jtmp, &sched->pending, list)
> +		media_jobs_free_job(job, true);
> +
> +	list_for_each_entry_safe(job, jtmp, &sched->queue, list)
> +		media_jobs_free_job(job, true);
> +}
> +
> +void media_jobs_cancel_jobs(struct media_job_scheduler *sched)
> +{
> +	if (!sched)
> +		return;
> +
> +	guard(spinlock)(&sched->lock);
> +	__media_jobs_cancel_jobs(sched);
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_cancel_jobs);
> +
> +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
> +				  int (*job_setup)(struct media_job *job, void *data),
> +				  void *data, enum media_job_types type)
> +{
> +	struct media_job_setup_func *new_setup_func;
> +
> +	guard(spinlock)(&sched->lock);
> +
> +	new_setup_func = kzalloc(sizeof(*new_setup_func), GFP_KERNEL);
> +	if (!new_setup_func)
> +		return -ENOMEM;
> +
> +	new_setup_func->type = type;
> +	new_setup_func->job_setup = job_setup;
> +	new_setup_func->data = data;
> +	list_add_tail(&new_setup_func->list, &sched->setup_funcs);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_add_job_setup_func);
> +
> +static void __media_jobs_put_scheduler(struct kref *kref)
> +{
> +	struct media_job_scheduler *sched =
> +		container_of(kref, struct media_job_scheduler, kref);
> +	struct media_job_setup_func *func, *ftmp;
> +
> +	cancel_work_sync(&sched->work);
> +	destroy_workqueue(sched->async_wq);
> +
> +	scoped_guard(spinlock, &sched->lock) {
> +		__media_jobs_cancel_jobs(sched);
> +
> +		list_for_each_entry_safe(func, ftmp, &sched->setup_funcs, list) {
> +			list_del(&func->list);
> +			kfree(func);
> +		}
> +	}
> +
> +	list_del(&sched->list);
> +	kfree(sched);
> +}
> +
> +void media_jobs_put_scheduler(struct media_job_scheduler *sched)
> +{
> +	kref_put(&sched->kref, __media_jobs_put_scheduler);
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_put_scheduler);
> +
> +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev)
> +{
> +	struct media_job_scheduler *sched;
> +	char workqueue_name[32];
> +	int ret;
> +
> +	guard(mutex)(&media_job_schedulers_lock);
> +
> +	list_for_each_entry(sched, &media_job_schedulers, list) {
> +		if (sched->mdev == mdev) {
> +			kref_get(&sched->kref);
> +			return sched;
> +		}
> +	}
> +
> +	ret = snprintf(workqueue_name, sizeof(workqueue_name),
> +		       "mc jobs (%s)", mdev->driver_name);
> +	if (!ret)
> +		return ERR_PTR(-EINVAL);
> +
> +	sched = kzalloc(sizeof(*sched), GFP_KERNEL);
> +	if (!sched)
> +		return ERR_PTR(-ENOMEM);
> +
> +	sched->async_wq = alloc_workqueue(workqueue_name, 0, 0);
> +	if (!sched->async_wq) {
> +		kfree(sched);
> +		return ERR_PTR(-EINVAL);
> +	}
> +
> +	sched->mdev = mdev;
> +	kref_init(&sched->kref);
> +	spin_lock_init(&sched->lock);
> +	INIT_LIST_HEAD(&sched->setup_funcs);
> +	INIT_LIST_HEAD(&sched->pending);
> +	INIT_LIST_HEAD(&sched->queue);
> +	INIT_WORK(&sched->work, __media_jobs_run_jobs);
> +
> +	list_add_tail(&sched->list, &media_job_schedulers);
> +
> +	return sched;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_get_scheduler);
> +
> +LIST_HEAD(media_job_schedulers);
> +
> +/* Synchronise access to the global schedulers list */
> +DEFINE_MUTEX(media_job_schedulers_lock);
> diff --git a/include/media/media-jobs.h b/include/media/media-jobs.h
> new file mode 100644
> index 000000000000..a97270861251
> --- /dev/null
> +++ b/include/media/media-jobs.h
> @@ -0,0 +1,354 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Media jobs framework
> + *
> + * Copyright 2025 Ideas on Board Oy
> + *
> + * Author: Daniel Scally <dan.scally@ideasonboard.com>
> + */
> +
> +#include <linux/kref.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/spinlock.h>
> +#include <linux/types.h>
> +#include <linux/workqueue.h>
> +
> +#ifndef _MEDIA_JOBS_H
> +#define _MEDIA_JOBS_H
> +
> +struct media_device;
> +struct media_entity;
> +struct media_job;
> +struct media_job_dep;
> +
> +/**
> + * define MEDIA_JOBS_FL_STEP_ANYWHERE - \
> + *    Flag a media job step as able to run anytime
> + *
> + * This flag informs the framework that a job step does not need a particular
> + * position in the list of job steps and can be placed anywhere.
> + */
> +#define MEDIA_JOBS_FL_STEP_ANYWHERE			BIT(0)
> +
> +/**
> + * define MEDIA_JOBS_FL_STEP_FROM_FRONT - \
> + *    Flag a media job step as needing to be placed near the start of the list
> + *
> + * This flag informs the framework that a job step needs to be placed at a set
> + * position from the start of the list of job steps.
> + */
> +#define MEDIA_JOBS_FL_STEP_FROM_FRONT			BIT(1)
> +
> +/**
> + * define MEDIA_JOBS_FL_STEP_FROM_BACK - \
> + *    Flag a media job step as needing to be placed near the end of the list
> + *
> + * This flag informs the framework that a job step needs to be placed at a set
> + * position from the end of the list of job steps.
> + */
> +#define MEDIA_JOBS_FL_STEP_FROM_BACK			BIT(2)
> +
> +/**
> + * enum media_job_types - Type of media job
> + *
> + * @MEDIA_JOB_TYPE_PIPELINE_PULSE:	A data event moving through the media
> + *					pipeline
> + *
> + * This enumeration details different types of media jobs. The type can be used
> + * to differentiate between which steps and dependencies a driver needs to add
> + * to a job when it is created.
> + */
> +enum media_job_types {
> +	MEDIA_JOB_TYPE_PIPELINE_PULSE,
> +};
> +
> +/**
> + * struct media_job_scheduler - A job scheduler for a particular media device
> + *
> + * @mdev:		Media device this scheduler is for
> + * @list:		List head to attach to the global list of schedulers
> + * @kref:		Reference counter
> + * @lock:		Lock to protect access to the scheduler
> + * @setup_funcs:	List of &struct media_job_setup_func to populate jobs
> + * @pending:		List of &struct media_jobs created but not yet queued
> + * @queue:		List of &struct media_jobs queued to the scheduler
> + * @work:		Work item to run the jobs
> + * @async_wq:		Workqueue to run the work on
> + *
> + * This struct is the main job scheduler struct - drivers wanting to use this
> + * framework should acquire an instance through media_jobs_get_scheduler() and
> + * subsequently populate it with job setup functions.
> + */
> +struct media_job_scheduler {
> +	struct media_device *mdev;
> +	struct list_head list;
> +	struct kref kref;
> +
> +	spinlock_t lock; /* Synchronise access to the struct's lists */
> +	struct list_head setup_funcs;
> +	struct list_head pending;
> +	struct list_head queue;
> +	struct work_struct work;
> +	struct workqueue_struct *async_wq;
> +};
> +
> +/**
> + * struct media_job_setup_func - A function to populate a media job with steps
> + *				 and dependencies
> + *
> + * @list:	The list object to attach to the scheduler
> + * @type:	The &enum media_job_types that this function populates a job for
> + * @job_setup:	Function pointer to the driver's job setup function
> + * @data:	Pointer to the driver data for use with @job_setup
> + *
> + * This struct holds data about the functions a driver registers with the jobs
> + * framework in order to populate a new job with steps and dependencies.
> + */
> +struct media_job_setup_func {
> +	struct list_head list;
> +	enum media_job_types type;
> +	int (*job_setup)(struct media_job *job, void *data);
> +	void *data;
> +};
> +
> +/**
> + * struct media_job - A representation of a job to be run through the pipeline
> + *
> + * @lock:	Lock to protect access to the job's lists
> + * @list:	List head to attach the job to &struct media_job_scheduler in
> + *		either the pending or queue lists
> + * @steps:	List of &struct media_job_step to run the job
> + * @deps:	List of &struct media_job_dep to check that the job can be
> + *		queued
> + * @sched:	Pointer to the media job scheduler
> + * @type:	The type of the job
> + *
> + * This struct holds lists of steps that need to be performed to carry out a
> + * job in the pipeline. A separate list of dependencies allows the queueing of
> + * the job to be delayed until all drivers are ready to carry it out.
> + */
> +struct media_job {
> +	spinlock_t lock; /* Synchronise access to the struct's lists 6*/
> +	struct list_head list;
> +	struct list_head steps;
> +	struct list_head deps;
> +	struct media_job_scheduler *sched;
> +	enum media_job_types type;
> +};
> +
> +/**
> + * struct media_job_step - A holder for a function to run as part of a job
> + *
> + * @list:	List head to attach the job step to a &struct media_job.steps
> + * @run_step:	The function to run to perform the step
> + * @data:	Data to pass to the .run_step() function
> + * @flags:	Flags to control how the step is ordered within the job's list
> + *		of steps
> + * @pos:	Position indicator to control how the step is ordered within the
> + *		job's list of steps
> + *
> + * This struct defines a function that needs to be run as part of the execution
> + * of a job in a media pipeline, along with information that help the scheduler
> + * determine what order it should be ran in in reference to the other steps that
> + * are part of the same job.
> + */
> +struct media_job_step {
> +	struct list_head list;
> +	void (*run_step)(void *data);
> +	void *data;
> +	unsigned int flags;
> +	unsigned int pos;
> +};
> +
> +/**
> + * struct media_job_dep_ops - Operations to manage a media job dependency
> + *
> + * @check_dep:	A function to ask the driver whether the dependency is met
> + * @clear_dep:	A function to tell the driver that the job has been queued
> + * @reset_dep:	A function to tell the driver that the job has been cancelled
> + *
> + * Media jobs have dependencies, such as requiring buffers to be queued. These
> + * operations allow a driver to define how the media jobs framework should check
> + * whether or not those dependencies are met and how it should inform them that
> + * it is taking action based on the state of those dependencies.
> + */
> +struct media_job_dep_ops {
> +	bool (*check_dep)(void *data);
> +	void (*clear_dep)(void *data);
> +	void (*reset_dep)(void *data);
> +};
> +
> +/**
> + * struct media_job_dep - Representation of media job dependency
> + *
> + * @list:	List head to attach to a &struct media_job.deps
> + * @ops:	A pointer to the dependency's operations functions
> + * @met:	A flag to record whether or not the dependency is met
> + * @data:	Data to pass to the dependency's operations
> + *
> + * This struct represents a dependency of a media job. The operations member
> + * holds pointers to functions allowing the framework to interact with the
> + * driver to check whether or not the dependency is met.
> + */
> +struct media_job_dep {
> +	struct list_head list;
> +	struct media_job_dep_ops *ops;
> +	bool met;
> +	void *data;
> +};
> +
> +/**
> + * media_jobs_try_queue_job - Try to queue a &struct media_job
> + *
> + * @sched:	Pointer to the job scheduler
> + * @type:	The type of the media job
> + * @dep_ops:	A pointer to the dependency operations for this job
> + * @dep_data:	A pointer to the dependency data for this job
> + *
> + * Try to queue a media job with the scheduler. This function should be called
> + * by the drivers whenever a dependency for a media job is met - for example
> + * when a buffer is queued to the driver. The framework will check to see if an
> + * existing job on the scheduler's pending list shares the same type, dependency
> + * operations and dependency data. If it does then that existing job will be
> + * considered. If there is no extant job with those same parameters, a new job
> + * is allocated and populated by calling the setup functions registered with
> + * the framework.
> + *
> + * The function iterates over the dependencies that are registered with the job
> + * and checks to see if they are met. If they're all met, they're cleared and
> + * the job is placed onto the scheduler's queue.
> + *
> + * To help reduce conditionals in drivers where a driver supports both the use
> + * of the media jobs framework and operation without it, this function is a no
> + * op if @sched is NULL.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
> +			     enum media_job_types type,
> +			     struct media_job_dep_ops *dep_ops, void *dep_data);
> +
> +/**
> + * media_jobs_add_job_step - Add a step to a media job
> + *
> + * @job:	Pointer to the &struct media_job
> + * @run_step:	Pointer to the function to run to execute the step
> + * @data:	Pointer to the data to pass to @run_ste
> + * @flags:	One of the MEDIA_JOBS_FL_STEP_* flags
> + * @pos:	A position indicator to use with @flags
> + *
> + * This function adds a step to the job and should be called from the drivers'
> + * job setup functions as registered with the framework through
> + * media_jobs_add_job_setup_func(). The @flags and @pos parameters are used
> + * to determine the ordering of the steps within the job:
> + *
> + * If @flags has the MEDIA_JOBS_FL_STEP_ANYWHERE bit set, the step is placed
> + * after all steps with MEDIA_JOBS_FL_STEP_FROM_FRONT and before all steps with
> + * MEDIA_JOBS_FL_STEP_FROM_BACK bit set, but otherwise in whatever order this
> + * function is called.
> + *
> + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_FRONT bit set then the step is
> + * placed @pos steps from the front of the list. Attempting to place multiple
> + * steps in the same position will result in an error.
> + *
> + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_BACK bit set then the step is
> + * placed @pos steps from the back of the list. Attempting to place multiple
> + * steps in the same position will result in an error.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
> +			    void *data, unsigned int flags, unsigned int pos);
> +
> +/**
> + * media_jobs_add_job_dep - Add a dependency to a media job
> + *
> + * @job:	Pointer to the &struct media_job
> + * @ops:	Pointer to the &struct media_job_dep_ops
> + * @data:	Pointer to the data to pass to the dependency's operations
> + *
> + * This function adds a dependency to the job and should be called from the
> + * drivers job setup functions as registered with the framework through the
> + * media_jobs_add_job_setup_func() function.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
> +			   void *data);
> +
> +/**
> + * media_jobs_add_job_setup_func - Add a function that populates a media job
> + *
> + * @sched:	Pointer to the media jobs scheduler
> + * @job_setup:	Pointer to the new job setup function
> + * @data:	Data to pass to the job setup function
> + * @type:	The type of job that this function should be called for
> + *
> + * Drivers that wish to utilise the framework need to use this function to
> + * register a callback that adds job steps and dependencies when one is created.
> + * The function must call media_jobs_add_job_step() and media_jobs_add_job_dep()
> + * to populate the job.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
> +				  int (*job_setup)(struct media_job *job, void *data),
> +				  void *data, enum media_job_types type);
> +
> +/**
> + * media_jobs_put_scheduler - Put a reference to the media jobs scheduler
> + *
> + * @sched:	Pointer to the media jobs scheduler
> + *
> + * This function puts a reference to the media jobs scheduler, and is intended
> + * to be called in error and exit paths for consuming drivers
> + */
> +void media_jobs_put_scheduler(struct media_job_scheduler *sched);
> +
> +/**
> + * media_jobs_get_scheduler - Get a media jobs scheduler
> + *
> + * @mdev:	Pointer to the media device associated with the scheduler
> + *
> + * This function gets a pointer to a &struct media_job_scheduler associated with
> + * the media device passed to @mdev. If one is not available then it is
> + * allocated and returned. This allows multiple drivers sharing a media graph to
> + * work with the same media job scheduler.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev);
> +
> +/**
> + * media_jobs_run_jobs - Run any media jobs that are ready in the queue
> + *
> + * @sched:	Pointer to the media job scheduler
> + *
> + * This function triggers the workqueue that processes any jobs that have been
> + * queued, and should be called whenever the pipeline is ready to do so.
> + *
> + * To help reduce conditionals in drivers where a driver supports both the use
> + * of the media jobs framework and operation without it, this function is a no
> + * op if @sched is NULL.
> + */
> +void media_jobs_run_jobs(struct media_job_scheduler *sched);
> +
> +/**
> + * media_jobs_cancel_jobs - cancel all waiting jobs
> + *
> + * @sched:	Pointer to the media job scheduler
> + *
> + * This function iterates over any pending and queued jobs, resets their
> + * dependencies and frees the job
> + *
> + * To help reduce conditionals in drivers where a driver supports both the use
> + * of the media jobs framework and operation without it, this function is a no
> + * op if @sched is NULL.
> + */
> +void media_jobs_cancel_jobs(struct media_job_scheduler *sched);
> +
> +extern struct list_head media_job_schedulers;
> +extern struct mutex media_job_schedulers_lock;
> +
> +#endif /* _MEDIA_JOBS_H */

Re: [PATCH 1/3] media: mc: entity: Add pipeline_started/stopped ops

[Dan Scally]

Hi Jacopo

On 21/05/2025 16:18, Jacopo Mondi wrote:
> Hi Dan
>
> On Mon, May 19, 2025 at 03:04:01PM +0100, Daniel Scally wrote:
>> Add two new members to struct media_entity_operations, along with new
>> functions in media-entity.c to traverse a media pipeline and call the
>> new operations. The new functions are intended to be used to signal
>> to a media pipeline that it has fully started, with the entity ops
>> allowing drivers to define some action to be taken when those
>> conditions are met.
>>
>> The combination of the new functions and operations allows drivers
>> which are part of a multi-driver pipeline to delay actually starting
>> streaming until all of the conditions for streaming succcessfully are
>> met across all drivers.
> Maybe s/succcessfully are/are successfully/
>
> Or was this (the three 'c' apart) intentional ?


It was intentional as-is but I think it reads just fine the other way round too; happy to change to 
that.

>
>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>> ---
>>   drivers/media/mc/mc-entity.c | 45 ++++++++++++++++++++++++++++++++++++
>>   include/media/media-entity.h | 24 +++++++++++++++++++
>>   2 files changed, 69 insertions(+)
>>
>> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
>> index 045590905582..e36b1710669d 100644
>> --- a/drivers/media/mc/mc-entity.c
>> +++ b/drivers/media/mc/mc-entity.c
>> @@ -1053,6 +1053,51 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
>>   }
>>   EXPORT_SYMBOL_GPL(__media_pipeline_entity_iter_next);
>>
>> +int media_pipeline_started(struct media_pipeline *pipe)
>> +{
>> +	struct media_pipeline_entity_iter iter;
>> +	struct media_entity *entity;
>> +	int ret;
>> +
>> +	ret = media_pipeline_entity_iter_init(pipe, &iter);
>> +	if (ret)
>> +		return ret;
>> +
>> +	media_pipeline_for_each_entity(pipe, &iter, entity) {
>> +		ret = media_entity_call(entity, pipeline_started);
>> +		if (ret && ret != -ENOIOCTLCMD)
>> +			goto err_notify_stopped;
>> +	}
>> +
>> +	media_pipeline_entity_iter_cleanup(&iter);
>> +
>> +	return ret == -ENOIOCTLCMD ? 0 : ret;
> Shouldn't you just return 0 ? If a ret < 0 is encoutered in the loop
> we just to the below label
Oh yeah; good point.
>
>> +
>> +err_notify_stopped:
>> +	media_pipeline_stopped(pipe);
> Do you need to media_pipeline_entity_iter_cleanup() ?


Yes - thanks.

>
>> +	return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(media_pipeline_started);
>> +
>> +int media_pipeline_stopped(struct media_pipeline *pipe)
>> +{
>> +	struct media_pipeline_entity_iter iter;
>> +	struct media_entity *entity;
>> +	int ret;
>> +
>> +	ret = media_pipeline_entity_iter_init(pipe, &iter);
>> +	if (ret)
>> +		return ret;
>> +
>> +	media_pipeline_for_each_entity(pipe, &iter, entity)
>> +		media_entity_call(entity, pipeline_stopped);
>> +
>> +	media_pipeline_entity_iter_cleanup(&iter);
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(media_pipeline_stopped);
>> +
>>   /* -----------------------------------------------------------------------------
>>    * Links management
>>    */
>> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
>> index 64cf590b1134..e858326b95cb 100644
>> --- a/include/media/media-entity.h
>> +++ b/include/media/media-entity.h
>> @@ -269,6 +269,10 @@ struct media_pad {
>>    *			media_entity_has_pad_interdep().
>>    *			Optional: If the operation isn't implemented all pads
>>    *			will be considered as interdependent.
>> + * @pipeline_started:	Notify this entity that the pipeline it is a part of has
>> + *			been started
>> + * @pipeline_stopped:	Notify this entity that the pipeline it is a part of has
>> + *			been stopped
>>    *
>>    * .. note::
>>    *
>> @@ -284,6 +288,8 @@ struct media_entity_operations {
>>   	int (*link_validate)(struct media_link *link);
>>   	bool (*has_pad_interdep)(struct media_entity *entity, unsigned int pad0,
>>   				 unsigned int pad1);
>> +	int (*pipeline_started)(struct media_entity *entity);
>> +	void (*pipeline_stopped)(struct media_entity *entity);
>>   };
>>
>>   /**
>> @@ -1261,6 +1267,24 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
>>   	     entity != NULL;							\
>>   	     entity = __media_pipeline_entity_iter_next((pipe), iter, entity))
>>
>> +/**
>> + * media_pipeline_started - Inform entities in a pipeline that it has started
>> + * @pipe:	The pipeline
>> + *
>> + * Iterate on all entities in a media pipeline and call their pipeline_started
>> + * member of media_entity_operations.
>> + */
>> +int media_pipeline_started(struct media_pipeline *pipe);
>> +
>> +/**
>> + * media_pipeline_stopped - Inform entities in a pipeline that it has stopped
>> + * @pipe:	The pipeline
>> + *
>> + * Iterate on all entities in a media pipeline and call their pipeline_stopped
>> + * member of media_entity_operations.
>> + */
>> +int media_pipeline_stopped(struct media_pipeline *pipe);
>> +
> All good, but I don't see these operations being used at all in this
> series ?


Ah, yeah...organisational deficiency on my part; it's used by the ISP and IVC drivers in the series' 
which are based on this one, and it's kinda related in that it's required to get the multiple 
drivers registering video devices into the same media device to play nicely together, so I bundled 
them into the same series. I could send it alone, or update the series cover message to make more 
clear it's encompassing a few things?

>
>>   /**
>>    * media_pipeline_alloc_start - Mark a pipeline as streaming
>>    * @pad: Starting pad
>> --
>> 2.34.1
>>
>>

Re: [PATCH 1/3] media: mc: entity: Add pipeline_started/stopped ops

[Dan Scally]

Hi Nicolas - thanks for the comment

On 22/05/2025 20:43, Nicolas Dufresne wrote:
> Hi,
>
> Le lundi 19 mai 2025 à 15:04 +0100, Daniel Scally a écrit :
>> Add two new members to struct media_entity_operations, along with new
>> functions in media-entity.c to traverse a media pipeline and call the
>> new operations. The new functions are intended to be used to signal
>> to a media pipeline that it has fully started, with the entity ops
>> allowing drivers to define some action to be taken when those
>> conditions are met.
>>
>> The combination of the new functions and operations allows drivers
>> which are part of a multi-driver pipeline to delay actually starting
>> streaming until all of the conditions for streaming succcessfully are
>> met across all drivers.
>>
>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>> ---
>>   drivers/media/mc/mc-entity.c | 45 ++++++++++++++++++++++++++++++++++++
>>   include/media/media-entity.h | 24 +++++++++++++++++++
>>   2 files changed, 69 insertions(+)
>>
>> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
>> index 045590905582..e36b1710669d 100644
>> --- a/drivers/media/mc/mc-entity.c
>> +++ b/drivers/media/mc/mc-entity.c
>> @@ -1053,6 +1053,51 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
>>   }
>>   EXPORT_SYMBOL_GPL(__media_pipeline_entity_iter_next);
>>   
>> +int media_pipeline_started(struct media_pipeline *pipe)
>> +{
>> +	struct media_pipeline_entity_iter iter;
>> +	struct media_entity *entity;
>> +	int ret;
>> +
>> +	ret = media_pipeline_entity_iter_init(pipe, &iter);
>> +	if (ret)
>> +		return ret;
>> +
>> +	media_pipeline_for_each_entity(pipe, &iter, entity) {
>> +		ret = media_entity_call(entity, pipeline_started);
>> +		if (ret && ret != -ENOIOCTLCMD)
>> +			goto err_notify_stopped;
>> +	}
> Would this be more useful if it had a specified traversal order ? Perhaps
> sink to source traversal?


Hmm...maybe? I don't think that it would matter either way for my particular use case, but I can 
have a look at giving it a specific order.

>
> Nicolas
>
>> +
>> +	media_pipeline_entity_iter_cleanup(&iter);
>> +
>> +	return ret == -ENOIOCTLCMD ? 0 : ret;
>> +
>> +err_notify_stopped:
>> +	media_pipeline_stopped(pipe);
>> +	return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(media_pipeline_started);
>> +
>> +int media_pipeline_stopped(struct media_pipeline *pipe)
>> +{
>> +	struct media_pipeline_entity_iter iter;
>> +	struct media_entity *entity;
>> +	int ret;
>> +
>> +	ret = media_pipeline_entity_iter_init(pipe, &iter);
>> +	if (ret)
>> +		return ret;
>> +
>> +	media_pipeline_for_each_entity(pipe, &iter, entity)
>> +		media_entity_call(entity, pipeline_stopped);
>> +
>> +	media_pipeline_entity_iter_cleanup(&iter);
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(media_pipeline_stopped);
>> +
>>   /* -----------------------------------------------------------------------------
>>    * Links management
>>    */
>> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
>> index 64cf590b1134..e858326b95cb 100644
>> --- a/include/media/media-entity.h
>> +++ b/include/media/media-entity.h
>> @@ -269,6 +269,10 @@ struct media_pad {
>>    *			media_entity_has_pad_interdep().
>>    *			Optional: If the operation isn't implemented all pads
>>    *			will be considered as interdependent.
>> + * @pipeline_started:	Notify this entity that the pipeline it is a part of has
>> + *			been started
>> + * @pipeline_stopped:	Notify this entity that the pipeline it is a part of has
>> + *			been stopped
>>    *
>>    * .. note::
>>    *
>> @@ -284,6 +288,8 @@ struct media_entity_operations {
>>   	int (*link_validate)(struct media_link *link);
>>   	bool (*has_pad_interdep)(struct media_entity *entity, unsigned int pad0,
>>   				 unsigned int pad1);
>> +	int (*pipeline_started)(struct media_entity *entity);
>> +	void (*pipeline_stopped)(struct media_entity *entity);
>>   };
>>   
>>   /**
>> @@ -1261,6 +1267,24 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
>>   	     entity != NULL;							\
>>   	     entity = __media_pipeline_entity_iter_next((pipe), iter, entity))
>>   
>> +/**
>> + * media_pipeline_started - Inform entities in a pipeline that it has started
>> + * @pipe:	The pipeline
>> + *
>> + * Iterate on all entities in a media pipeline and call their pipeline_started
>> + * member of media_entity_operations.
>> + */
>> +int media_pipeline_started(struct media_pipeline *pipe);
>> +
>> +/**
>> + * media_pipeline_stopped - Inform entities in a pipeline that it has stopped
>> + * @pipe:	The pipeline
>> + *
>> + * Iterate on all entities in a media pipeline and call their pipeline_stopped
>> + * member of media_entity_operations.
>> + */
>> +int media_pipeline_stopped(struct media_pipeline *pipe);
>> +
>>   /**
>>    * media_pipeline_alloc_start - Mark a pipeline as streaming
>>    * @pad: Starting pad

Re: [PATCH 2/3] media: mc: Add media jobs framework

[Dan Scally]

Hi Jacopo

On 22/05/2025 20:04, Jacopo Mondi wrote:
> Hi Dan
>
> On Thu, May 22, 2025 at 12:24:46PM +0100, Dan Scally wrote:
>> Hi Jacopo
>>
>> On 22/05/2025 12:00, Jacopo Mondi wrote:
>>> Hi Dan, back with more questions :)
>>>
>>> On Mon, May 19, 2025 at 03:04:02PM +0100, Daniel Scally wrote:
>>>> Add a new framework to the media subsystem describing media jobs.
>>>> This framework is intended to be able to model the interactions
>>>> between multiple different drivers that need to be run in concert
>>>> to fully control a media pipeline, for example an ISP driver and a
>>>> driver controlling a DMA device that feeds data from memory in to
>>>> that ISP.
>>>>
>>>> The new framework allows all drivers involved to add explicit steps
>>>> that need to be performed, and to control the ordering of those steps
>>>> precisely. Once the job with its steps has been created it's then
>>>> scheduled to be run with a workqueue which executes each step in the
>>>> defined order.
>>>>
>>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>>>> ---
>>>>    drivers/media/mc/Makefile  |   2 +-
>>>>    drivers/media/mc/mc-jobs.c | 446 +++++++++++++++++++++++++++++++++++++
>>>>    include/media/media-jobs.h | 354 +++++++++++++++++++++++++++++
>>>>    3 files changed, 801 insertions(+), 1 deletion(-)
>>>>    create mode 100644 drivers/media/mc/mc-jobs.c
>>>>    create mode 100644 include/media/media-jobs.h
>>>>
>>>> diff --git a/drivers/media/mc/Makefile b/drivers/media/mc/Makefile
>>>> index 2b7af42ba59c..9148bbfd1578 100644
>>>> --- a/drivers/media/mc/Makefile
>>>> +++ b/drivers/media/mc/Makefile
>>>> @@ -1,7 +1,7 @@
>>>>    # SPDX-License-Identifier: GPL-2.0
>>>>
>>>>    mc-objs	:= mc-device.o mc-devnode.o mc-entity.o \
>>>> -	   mc-request.o
>>>> +	   mc-jobs.o mc-request.o
>>>>
>>>>    ifneq ($(CONFIG_USB),)
>>>>    	mc-objs += mc-dev-allocator.o
>>>> diff --git a/drivers/media/mc/mc-jobs.c b/drivers/media/mc/mc-jobs.c
>>>> new file mode 100644
>>>> index 000000000000..1f04cdf63d27
>>>> --- /dev/null
>>>> +++ b/drivers/media/mc/mc-jobs.c
>>>> @@ -0,0 +1,446 @@
>>>> +// SPDX-License-Identifier: GPL-2.0-only
>>>> +/*
>>>> + * Media jobs framework
>>>> + *
>>>> + * Copyright 2025 Ideas on Board Oy
>>>> + *
>>>> + * Author: Daniel Scally <dan.scally@ideasonboard.com>
>>>> + */
>>>> +
>>>> +#include <linux/cleanup.h>
>>>> +#include <linux/kref.h>
>>>> +#include <linux/list.h>
>>>> +#include <linux/slab.h>
>>>> +#include <linux/spinlock.h>
>>>> +
>>>> +#include <media/media-device.h>
>>>> +#include <media/media-entity.h>
>>>> +#include <media/media-jobs.h>
>>>> +
>>>> +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
>>>> +			    void *data, unsigned int flags, unsigned int pos)
>>>> +{
>>>> +	struct media_job_step *step, *tmp;
>>>> +	unsigned int num = flags;
>>>> +	unsigned int count = 0;
>>>> +
>>>> +	guard(spinlock)(&job->lock);
>>>> +
>>>> +	if (!flags) {
>>>> +		WARN_ONCE(1, "%s(): No flag bits set\n", __func__);
>>>> +		return -EINVAL;
>>>> +	}
>>>> +
>>>> +	/* Count the number of set flags; they're mutually exclusive. */
>>>> +	while (num) {
>>>> +		num &= (num - 1);
>>>> +		count++;
>>>> +	}
>>>> +
>>>> +	if (count > 1) {
>>>> +		WARN_ONCE(1, "%s(): Multiple flag bits set\n", __func__);
>>>> +		return -EINVAL;
>>>> +	}
>>>> +
>>>> +	step = kzalloc(sizeof(*step), GFP_KERNEL);
>>>> +	if (!step)
>>>> +		return -ENOMEM;
>>>> +
>>>> +	step->run_step = run_step;
>>>> +	step->data = data;
>>>> +	step->flags = flags;
>>>> +	step->pos = pos;
>>>> +
>>>> +	/*
>>>> +	 * We need to decide where to place the step. If the list is empty that
>>>> +	 * is really easy (and also the later code is much easier if the code is
>>>> +	 * guaranteed not to be empty...)
>>>> +	 */
>>>> +	if (list_empty(&job->steps)) {
>>>> +		list_add_tail(&step->list, &job->steps);
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	/*
>>>> +	 * If we've been asked to place it at a specific position from the end
>>>> +	 * of the list, we cycle back through it until either we exhaust the
>>>> +	 * list or find an entry that needs to go further from the back than the
>>>> +	 * new one.
>>>> +	 */
>>>> +	if ((flags & MEDIA_JOBS_FL_STEP_FROM_BACK)) {
>>>> +		list_for_each_entry_reverse(tmp, &job->steps, list) {
>>>> +			if (tmp->flags == flags && tmp->pos == pos)
>>>> +				return -EINVAL;
>>>> +
>>>> +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_BACK ||
>>>> +			    tmp->pos > pos)
>>>> +				break;
>>>> +		}
>>>> +
>>>> +		/*
>>>> +		 * If the entry we broke on is also one placed from the back and
>>>> +		 * should be closer to the back than the new one, we place the
>>>> +		 * new one in front of it...otherwise place the new one behind
>>>> +		 * it.
>>>> +		 */
>>>> +		if (tmp->flags == flags && tmp->pos < pos)
>>>> +			list_add_tail(&step->list, &tmp->list);
>>>> +		else
>>>> +			list_add(&step->list, &tmp->list);
>>>> +
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	/*
>>>> +	 * If we've been asked to place it a specific position from the front of
>>>> +	 * the list we do the same kind of operation, but going from the front
>>>> +	 * instead.
>>>> +	 */
>>>> +	if (flags & MEDIA_JOBS_FL_STEP_FROM_FRONT) {
>>>> +		list_for_each_entry(tmp, &job->steps, list) {
>>>> +			if (tmp->flags == flags && tmp->pos == pos)
>>>> +				return -EINVAL;
>>>> +
>>>> +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_FRONT ||
>>>> +			    tmp->pos > pos)
>>>> +				break;
>>>> +		}
>>>> +
>>>> +		/*
>>>> +		 * If the entry we broke on is also placed from the front and
>>>> +		 * should be closed to the front than the new one, we place the
>>>> +		 * new one behind it, otherwise in front of it.
>>>> +		 */
>>>> +		if (tmp->flags == flags && tmp->pos < pos)
>>>> +			list_add(&step->list, &tmp->list);
>>>> +		else
>>>> +			list_add_tail(&step->list, &tmp->list);
>>>> +
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	/*
>>>> +	 * If the step is flagged as "can go anywhere" we just need to try to
>>>> +	 * find the first "from the back" entry and add it immediately before
>>>> +	 * that. If we can't find one, add it after whatever we did find.
>>>> +	 */
>>>> +	if (flags & MEDIA_JOBS_FL_STEP_ANYWHERE) {
>>>> +		list_for_each_entry(tmp, &job->steps, list)
>>>> +			if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK))
>>>> +				break;
>>>> +
>>>> +		if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK) ||
>>>> +		    list_entry_is_head(tmp, &job->steps, list))
>>>> +			list_add_tail(&step->list, &tmp->list);
>>>> +		else
>>>> +			list_add(&step->list, &tmp->list);
>>>> +
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	/* Shouldn't get here, unless the flag value is wrong. */
>>>> +	WARN_ONCE(1, "%s(): Invalid flag value\n", __func__);
>>>> +	return -EINVAL;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(media_jobs_add_job_step);
>>>> +
>>>> +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
>>>> +			   void *data)
>>>> +{
>>>> +	struct media_job_dep *dep;
>>>> +
>>>> +	if (!ops || !ops->check_dep || !data)
>>>> +		return -EINVAL;
>>>> +
>>>> +	guard(spinlock)(&job->lock);
>>>> +
>>>> +	/* Confirm the same dependency hasn't already been added */
>>>> +	list_for_each_entry(dep, &job->deps, list)
>>>> +		if (dep->ops == ops && dep->data == data)
>>>> +			return -EINVAL;
>>>> +
>>>> +	dep = kzalloc(sizeof(*dep), GFP_KERNEL);
>>>> +	if (!dep)
>>>> +		return -ENOMEM;
>>>> +
>>>> +	dep->ops = ops;
>>>> +	dep->data = data;
>>>> +	list_add(&dep->list, &job->deps);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(media_jobs_add_job_dep);
>>>> +
>>>> +static bool media_jobs_check_pending_job(struct media_job *job,
>>>> +					 enum media_job_types type,
>>>> +					 struct media_job_dep_ops *dep_ops,
>>>> +					 void *data)
>>>> +{
>>>> +	struct media_job_dep *dep;
>>>> +
>>>> +	guard(spinlock)(&job->lock);
>>>> +
>>>> +	if (job->type != type)
>>>> +		return false;
>>>> +
>>>> +	list_for_each_entry(dep, &job->deps, list) {
>>>> +		if (dep->ops == dep_ops && dep->data == data) {
>>>> +			if (dep->met)
>>>> +				return false;
>>>> +
>>>> +			break;
>>>> +		}
>>>> +	}
>>>> +
>>>> +	dep->met = true;
>>>> +	return true;
>>>> +}
>>>> +
>>>> +static struct media_job *media_jobs_get_job(struct media_job_scheduler *sched,
>>>> +					    enum media_job_types type,
>>>> +					    struct media_job_dep_ops *dep_ops,
>>>> +					    void *dep_data)
>>>> +{
>>>> +	struct media_job_setup_func *jsf;
>>>> +	struct media_job *job;
>>>> +	int ret;
>>>> +
>>>> +	list_for_each_entry(job, &sched->pending, list)
>>>> +		if (media_jobs_check_pending_job(job, type, dep_ops, dep_data))
>>>> +			return job;
>>>> +
>>> Thanks to your offline explanation, I got how this works now, however
>>> some questions here
>>>
>>> The basic idea is that each driver that registers a 'setup' function
>>> adds to a job, when its created, its list of dependencies.
>>>
>>> When a job is "try_queue" and we get here, to decide if a new job has
>>> to be created or if we have to run one which is already in the pending
>>> queue.
>>>
>>> How is this identification performed ? Each entry point (assume it's a
>>> video device op) will populate the job with its own dependencies,
>>> identified by the dep_ops and data address.
>>>
>>> We walk the 'pending' queue in the media_jobs_check_pending_job()
>>> function and we search for one job not already visited from the same
>>> entry point, identified by the dep_ops (the 'visited' state is kept by
>>> the deps->met flag).
>>>
>>> Let's assume 2 video devices X and Y
>>>
>>> qbuf(x) -> try_queue_job() -> new job created on 'pending'
>>> qbuf(x) -> try_queue_job() -> the job in the queue has 'deps->met' set, skip
>>> it and create a new one
>>> qbuf(y) -> try_queue_job() -> the first job in the queue has not
>>> deps->set, so return it
>>>
>>> All in all I would describe this as: when requesting a job try to find
>>> the first one not already visited by this entry point, if none is
>>> available create a new one.
>>
>> Yep, all seems fine.
>>
>>> Now, we briefly discussed that when moving to multi-context comparing
>>> dep_ops and data to identify an entry point won't be enough: buffers
>>> from the same video device but from different contexts do not have to
>>> be associated together. So we'll need to extend the identification
>>> criteria. Also, I don't find the idea of using dep_ops and data for
>>> this purpose particularly neat, as it makes mandatory to add
>>> dependencies to a job in the setup function, something not all driver
>>> might want to do ?
>> I don't think it's mandatory for a driver to add dependencies to a job; the
>> implication of lacking them is that whatever step the driver is running for
>> the job takes no input (no buffers need to be available, no parameters need
>> to have been set, no per-requisites need to have been met) in which case it
>> can simply be ignored for the purposes of evaluating whether the job can be
>> queued or not, because it's always ready by definition...does that make
>> sense?
> Yes, but what are the implications of not setting deps on
> media_jobs_check_pending_job() ?
>
> If I'm not mistaken
>
> 	if (dep->ops == dep_ops && dep->data == data) {
>
> will now always return false, as there won't be any 'dep' that matches
> with dep_ops as the driver has never called media_jobs_add_job_dep().
>
> If so, if a video device that doesn't register deps (tbh I don't see
> why it would, but..) will always match and we will overwrite the same job over
> and over ? Anyway, a corner case I guess
Ah, sorry, you are right of course...I suppose we can specifically check for NULL in whatever 
arguments we pass to guarantee uniqueness and handle that.
>
>>> There might be ways to handle this "track the entry point" thing that
>>> could be separated by deps, making deps do what they actually are
>>> described for: track dependencies to validate if a job can be run or
>>> not. Before exploring options, I would like to know if this only mine
>>> concern or is it shared by others.
>>
>> I do agree that a nicer way of tracking them rather than dep_ops and data
>> would be better...perhaps tying it to the entry point as you've
>> conceptualised here is the right thing to do, and the pointer to the
>> function calling media_jobs_try_queue_job() should be passed, along with a
>> context ID?
>>
> I think that's a possible way forward. For the time being I think
> identifying a "user" of a job (iow any drivers that registers a
> dependency or a step) with a pointer to the video device or to the
> driver-specific types should be enough and we can easily
> move it to use the video device context later.
>
> I have patches to push if you want to see.


Yes please!

>
> A bit more comments on the API below
>
>> Thanks
>>
>> Dan
>>
>>>> +	job = kzalloc(sizeof(*job), GFP_KERNEL);
>>>> +	if (!job)
>>>> +		return ERR_PTR(-ENOMEM);
>>>> +
>>>> +	spin_lock_init(&job->lock);
>>>> +	INIT_LIST_HEAD(&job->deps);
>>>> +	INIT_LIST_HEAD(&job->steps);
>>>> +	job->type = type;
>>>> +	job->sched = sched;
>>>> +
>>>> +	list_for_each_entry(jsf, &sched->setup_funcs, list) {
>>>> +		if (jsf->type != type)
>>>> +			continue;
>>>> +
>>>> +		ret = jsf->job_setup(job, jsf->data);
>>>> +		if (ret) {
>>>> +			kfree(job);
>>>> +			return ERR_PTR(ret);
>>>> +		}
>>>> +	}
>>>> +
>>>> +	list_add_tail(&job->list, &sched->pending);
>>>> +
>>>> +	/* This marks the dependency as met */
>>>> +	media_jobs_check_pending_job(job, type, dep_ops, dep_data);
>>>> +
>>>> +	return job;
>>>> +}
>>>> +
>>>> +static void media_jobs_free_job(struct media_job *job, bool reset)
>>>> +{
>>>> +	struct media_job_step *step, *stmp;
>>>> +	struct media_job_dep *dep, *dtmp;
>>>> +
>>>> +	scoped_guard(spinlock, &job->lock) {
>>>> +		list_for_each_entry_safe(dep, dtmp, &job->deps, list) {
>>>> +			if (reset && dep->ops->reset_dep)
>>>> +				dep->ops->reset_dep(dep->data);
>>>> +
>>>> +			list_del(&dep->list);
>>>> +			kfree(dep);
>>>> +		}
>>>> +
>>>> +		list_for_each_entry_safe(step, stmp, &job->steps, list) {
>>>> +			list_del(&step->list);
>>>> +			kfree(step);
>>>> +		}
>>>> +	}
>>>> +
>>>> +	list_del(&job->list);
>>>> +	kfree(job);
>>>> +}
>>>> +
>>>> +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
>>>> +			     enum media_job_types type,
>>>> +			     struct media_job_dep_ops *dep_ops, void *dep_data)
>>>> +{
>>>> +	struct media_job_dep *dep;
>>>> +	struct media_job *job;
>>>> +
>>>> +	if (!sched)
>>>> +		return 0;
>>>> +
>>>> +	guard(spinlock)(&sched->lock);
>>>> +
>>>> +	job = media_jobs_get_job(sched, type, dep_ops, dep_data);
>>>> +	if (IS_ERR(job))
>>>> +		return PTR_ERR(job);
>>>> +
>>>> +	list_for_each_entry(dep, &job->deps, list)
>>>> +		if (!dep->ops->check_dep(dep->data))
>>>> +			return 0; /* Not a failure */
>>>> +
>>>> +	list_for_each_entry(dep, &job->deps, list)
>>>> +		if (dep->ops->clear_dep)
>>>> +			dep->ops->clear_dep(dep->data);
>>>> +
>>>> +	list_move_tail(&job->list, &sched->queue);
>>>> +	queue_work(sched->async_wq, &sched->work);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(media_jobs_try_queue_job);
>>>> +
>>>> +static void __media_jobs_run_jobs(struct work_struct *work)
>>>> +{
>>>> +	struct media_job_scheduler *sched = container_of(work,
>>>> +							 struct media_job_scheduler,
>>>> +							 work);
>>>> +	struct media_job_step *step;
>>>> +	struct media_job *job;
>>>> +
>>>> +	while (true) {
>>>> +		scoped_guard(spinlock, &sched->lock) {
>>>> +			if (list_empty(&sched->queue))
>>>> +				return;
>>>> +
>>>> +			job = list_first_entry(&sched->queue, struct media_job,
>>>> +					       list);
>>>> +		}
>>>> +
>>>> +		list_for_each_entry(step, &job->steps, list)
>>>> +			step->run_step(step->data);
>>>> +
>>>> +		media_jobs_free_job(job, false);
>>>> +	}
>>>> +}
>>>> +
>>>> +void media_jobs_run_jobs(struct media_job_scheduler *sched)
>>>> +{
>>>> +	if (!sched)
>>>> +		return;
>>>> +
>>>> +	queue_work(sched->async_wq, &sched->work);
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(media_jobs_run_jobs);
>>>> +
>>>> +static void __media_jobs_cancel_jobs(struct media_job_scheduler *sched)
>>>> +{
>>>> +	struct media_job *job, *jtmp;
>>>> +
>>>> +	list_for_each_entry_safe(job, jtmp, &sched->pending, list)
>>>> +		media_jobs_free_job(job, true);
>>>> +
>>>> +	list_for_each_entry_safe(job, jtmp, &sched->queue, list)
>>>> +		media_jobs_free_job(job, true);
>>>> +}
>>>> +
>>>> +void media_jobs_cancel_jobs(struct media_job_scheduler *sched)
>>>> +{
>>>> +	if (!sched)
>>>> +		return;
>>>> +
>>>> +	guard(spinlock)(&sched->lock);
>>>> +	__media_jobs_cancel_jobs(sched);
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(media_jobs_cancel_jobs);
>>>> +
>>>> +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
>>>> +				  int (*job_setup)(struct media_job *job, void *data),
>>>> +				  void *data, enum media_job_types type)
>>>> +{
>>>> +	struct media_job_setup_func *new_setup_func;
>>>> +
>>>> +	guard(spinlock)(&sched->lock);
>>>> +
>>>> +	new_setup_func = kzalloc(sizeof(*new_setup_func), GFP_KERNEL);
>>>> +	if (!new_setup_func)
>>>> +		return -ENOMEM;
>>>> +
>>>> +	new_setup_func->type = type;
>>>> +	new_setup_func->job_setup = job_setup;
>>>> +	new_setup_func->data = data;
>>>> +	list_add_tail(&new_setup_func->list, &sched->setup_funcs);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(media_jobs_add_job_setup_func);
>>>> +
>>>> +static void __media_jobs_put_scheduler(struct kref *kref)
>>>> +{
>>>> +	struct media_job_scheduler *sched =
>>>> +		container_of(kref, struct media_job_scheduler, kref);
>>>> +	struct media_job_setup_func *func, *ftmp;
>>>> +
>>>> +	cancel_work_sync(&sched->work);
>>>> +	destroy_workqueue(sched->async_wq);
>>>> +
>>>> +	scoped_guard(spinlock, &sched->lock) {
>>>> +		__media_jobs_cancel_jobs(sched);
>>>> +
>>>> +		list_for_each_entry_safe(func, ftmp, &sched->setup_funcs, list) {
>>>> +			list_del(&func->list);
>>>> +			kfree(func);
>>>> +		}
>>>> +	}
>>>> +
>>>> +	list_del(&sched->list);
>>>> +	kfree(sched);
>>>> +}
>>>> +
>>>> +void media_jobs_put_scheduler(struct media_job_scheduler *sched)
>>>> +{
>>>> +	kref_put(&sched->kref, __media_jobs_put_scheduler);
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(media_jobs_put_scheduler);
>>>> +
>>>> +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev)
>>>> +{
>>>> +	struct media_job_scheduler *sched;
>>>> +	char workqueue_name[32];
>>>> +	int ret;
>>>> +
>>>> +	guard(mutex)(&media_job_schedulers_lock);
>>>> +
>>>> +	list_for_each_entry(sched, &media_job_schedulers, list) {
>>>> +		if (sched->mdev == mdev) {
>>>> +			kref_get(&sched->kref);
>>>> +			return sched;
>>>> +		}
>>>> +	}
>>>> +
>>>> +	ret = snprintf(workqueue_name, sizeof(workqueue_name),
>>>> +		       "mc jobs (%s)", mdev->driver_name);
>>>> +	if (!ret)
>>>> +		return ERR_PTR(-EINVAL);
>>>> +
>>>> +	sched = kzalloc(sizeof(*sched), GFP_KERNEL);
>>>> +	if (!sched)
>>>> +		return ERR_PTR(-ENOMEM);
>>>> +
>>>> +	sched->async_wq = alloc_workqueue(workqueue_name, 0, 0);
>>>> +	if (!sched->async_wq) {
>>>> +		kfree(sched);
>>>> +		return ERR_PTR(-EINVAL);
>>>> +	}
>>>> +
>>>> +	sched->mdev = mdev;
>>>> +	kref_init(&sched->kref);
>>>> +	spin_lock_init(&sched->lock);
>>>> +	INIT_LIST_HEAD(&sched->setup_funcs);
>>>> +	INIT_LIST_HEAD(&sched->pending);
>>>> +	INIT_LIST_HEAD(&sched->queue);
>>>> +	INIT_WORK(&sched->work, __media_jobs_run_jobs);
>>>> +
>>>> +	list_add_tail(&sched->list, &media_job_schedulers);
>>>> +
>>>> +	return sched;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(media_jobs_get_scheduler);
>>>> +
>>>> +LIST_HEAD(media_job_schedulers);
>>>> +
>>>> +/* Synchronise access to the global schedulers list */
>>>> +DEFINE_MUTEX(media_job_schedulers_lock);
>>>> diff --git a/include/media/media-jobs.h b/include/media/media-jobs.h
>>>> new file mode 100644
>>>> index 000000000000..a97270861251
>>>> --- /dev/null
>>>> +++ b/include/media/media-jobs.h
>>>> @@ -0,0 +1,354 @@
>>>> +/* SPDX-License-Identifier: GPL-2.0-only */
>>>> +/*
>>>> + * Media jobs framework
>>>> + *
>>>> + * Copyright 2025 Ideas on Board Oy
>>>> + *
>>>> + * Author: Daniel Scally <dan.scally@ideasonboard.com>
>>>> + */
>>>> +
> Could you include the header in the .c file first to make sure it's
> self-contained ?
>
>>>> +#include <linux/kref.h>
>>>> +#include <linux/list.h>
>>>> +#include <linux/mutex.h>
>>>> +#include <linux/spinlock.h>
>>>> +#include <linux/types.h>
>>>> +#include <linux/workqueue.h>
>>>> +
>>>> +#ifndef _MEDIA_JOBS_H
>>>> +#define _MEDIA_JOBS_H
> Place this at the beginning of the file
>
>>>> +
>>>> +struct media_device;
>>>> +struct media_entity;
>>>> +struct media_job;
>>>> +struct media_job_dep;
>>>> +
>>>> +/**
>>>> + * define MEDIA_JOBS_FL_STEP_ANYWHERE - \
>>>> + *    Flag a media job step as able to run anytime
>>>> + *
>>>> + * This flag informs the framework that a job step does not need a particular
>>>> + * position in the list of job steps and can be placed anywhere.
>>>> + */
>>>> +#define MEDIA_JOBS_FL_STEP_ANYWHERE			BIT(0)
>>>> +
>>>> +/**
>>>> + * define MEDIA_JOBS_FL_STEP_FROM_FRONT - \
>>>> + *    Flag a media job step as needing to be placed near the start of the list
>>>> + *
>>>> + * This flag informs the framework that a job step needs to be placed at a set
>>>> + * position from the start of the list of job steps.
>>>> + */
>>>> +#define MEDIA_JOBS_FL_STEP_FROM_FRONT			BIT(1)
>>>> +
>>>> +/**
>>>> + * define MEDIA_JOBS_FL_STEP_FROM_BACK - \
>>>> + *    Flag a media job step as needing to be placed near the end of the list
>>>> + *
>>>> + * This flag informs the framework that a job step needs to be placed at a set
>>>> + * position from the end of the list of job steps.
>>>> + */
>>>> +#define MEDIA_JOBS_FL_STEP_FROM_BACK			BIT(2)
>>>> +
>>>> +/**
>>>> + * enum media_job_types - Type of media job
>>>> + *
>>>> + * @MEDIA_JOB_TYPE_PIPELINE_PULSE:	A data event moving through the media
>>>> + *					pipeline
>>>> + *
>>>> + * This enumeration details different types of media jobs. The type can be used
>>>> + * to differentiate between which steps and dependencies a driver needs to add
>>>> + * to a job when it is created.
>>>> + */
>>>> +enum media_job_types {
>>>> +	MEDIA_JOB_TYPE_PIPELINE_PULSE,
>>>> +};
> I'm wondering if we could omit type for the time being.
> A little trick would be to use variadic macros to allow users to not
> specify the type and default it to MEDIA_JOB_TYPE_PIPELINE_PULSE and
> at the same time allow future users of other types to specify them
> later on
>
> Something like (not based on this version of the patch sorry)
>
> -int media_jobs_try_queue_job(struct media_job_scheduler *sched,
> -                            enum media_job_types type,
> -                            void *cookie);
> +int __media_jobs_try_queue_job(struct media_job_scheduler *sched,
> +                              void *cookie, enum media_job_types type);
> +
> +#define __media_jobs_try_queue_job_notype(sched, cookie)       \
> +       __media_jobs_try_queue_job(sched, cookie, MEDIA_JOB_TYPE_PIPELINE_PULSE)
> +#define __media_jobs_try_queue_job_type(sched, cookie, type)   \
> +       __media_jobs_try_queue_job(sched, cookie, type)
>
> +#define MEDIA_JOBS_TRY_QUEUE_EXPAND(_1, _2, _3, FUNC, ...) FUNC
> +#define media_jobs_try_queue_job(...)                                  \
> +       MEDIA_JOBS_TRY_QUEUE_EXPAND(__VA_ARGS__,                        \
> +                                   __media_jobs_try_queue_job_type,    \
> +                                   __media_jobs_try_queue_job_notype)  \
> +                                   (__VA_ARGS__)
>
> Maybe not good looking but allows to call both
>
> media_jobs_try_queue_job(sched, cookie, type) and
> media_jobs_try_queue_job(sched, cookie)
>
> (this could be done to all functions that accepts a type, apart the
> job setup functions)
>
I'm fine with something like this...I feel like the option to have multiple types of job for a 
single scheduler is worth keeping around, but making things so the users don't have to deal with it 
for now is probably good.
>>>> +
>>>> +/**
>>>> + * struct media_job_scheduler - A job scheduler for a particular media device
>>>> + *
>>>> + * @mdev:		Media device this scheduler is for
>>>> + * @list:		List head to attach to the global list of schedulers
>>>> + * @kref:		Reference counter
>>>> + * @lock:		Lock to protect access to the scheduler
>>>> + * @setup_funcs:	List of &struct media_job_setup_func to populate jobs
>>>> + * @pending:		List of &struct media_jobs created but not yet queued
>>>> + * @queue:		List of &struct media_jobs queued to the scheduler
>>>> + * @work:		Work item to run the jobs
>>>> + * @async_wq:		Workqueue to run the work on
>>>> + *
>>>> + * This struct is the main job scheduler struct - drivers wanting to use this
>>>> + * framework should acquire an instance through media_jobs_get_scheduler() and
>>>> + * subsequently populate it with job setup functions.
>>>> + */
>>>> +struct media_job_scheduler {
>>>> +	struct media_device *mdev;
>>>> +	struct list_head list;
>>>> +	struct kref kref;
>>>> +
>>>> +	spinlock_t lock; /* Synchronise access to the struct's lists */
>>>> +	struct list_head setup_funcs;
>>>> +	struct list_head pending;
>>>> +	struct list_head queue;
> Empty line please
>
>>>> +	struct work_struct work;
>>>> +	struct workqueue_struct *async_wq;
>>>> +};
> This type should be defined inside the .c file and not exposed ?


Ah, yes, thanks.

>
>>>> +
>>>> +/**
>>>> + * struct media_job_setup_func - A function to populate a media job with steps
>>>> + *				 and dependencies
>>>> + *
>>>> + * @list:	The list object to attach to the scheduler
>>>> + * @type:	The &enum media_job_types that this function populates a job for
>>>> + * @job_setup:	Function pointer to the driver's job setup function
>>>> + * @data:	Pointer to the driver data for use with @job_setup
>>>> + *
>>>> + * This struct holds data about the functions a driver registers with the jobs
>>>> + * framework in order to populate a new job with steps and dependencies.
>>>> + */
>>>> +struct media_job_setup_func {
>>>> +	struct list_head list;
>>>> +	enum media_job_types type;
>>>> +	int (*job_setup)(struct media_job *job, void *data);
> You could define a type and use it. I would do that for all functions
> where it makes sense


A typedef for the function signature you mean? If I'm honest I never quite saw a reason to do that; 
what's it in aid of?

>
>>>> +	void *data;
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct media_job - A representation of a job to be run through the pipeline
>>>> + *
>>>> + * @lock:	Lock to protect access to the job's lists
>>>> + * @list:	List head to attach the job to &struct media_job_scheduler in
>>>> + *		either the pending or queue lists
>>>> + * @steps:	List of &struct media_job_step to run the job
>>>> + * @deps:	List of &struct media_job_dep to check that the job can be
>>>> + *		queued
>>>> + * @sched:	Pointer to the media job scheduler
>>>> + * @type:	The type of the job
>>>> + *
>>>> + * This struct holds lists of steps that need to be performed to carry out a
>>>> + * job in the pipeline. A separate list of dependencies allows the queueing of
>>>> + * the job to be delayed until all drivers are ready to carry it out.
>>>> + */
>>>> +struct media_job {
>>>> +	spinlock_t lock; /* Synchronise access to the struct's lists 6*/
>>>> +	struct list_head list;
>>>> +	struct list_head steps;
>>>> +	struct list_head deps;
>>>> +	struct media_job_scheduler *sched;
>>>> +	enum media_job_types type;
> nit: list_head are usually at the end, at least it seems to to me.
> Doesn't
>
> 	struct media_job_scheduler *sched;
> 	enum media_job_types type;
>
> 	spinlock_t lock; /* Synchronise access to the struct's lists 6*/
> 	struct list_head list;
> 	struct list_head steps;
> 	struct list_head deps;
>
> look better ?


Yes :)

>
> };
>>>> +
>>>> +/**
>>>> + * struct media_job_step - A holder for a function to run as part of a job
>>>> + *
>>>> + * @list:	List head to attach the job step to a &struct media_job.steps
>>>> + * @run_step:	The function to run to perform the step
>>>> + * @data:	Data to pass to the .run_step() function
>>>> + * @flags:	Flags to control how the step is ordered within the job's list
>>>> + *		of steps
>>>> + * @pos:	Position indicator to control how the step is ordered within the
>>>> + *		job's list of steps
>>>> + *
>>>> + * This struct defines a function that needs to be run as part of the execution
>>>> + * of a job in a media pipeline, along with information that help the scheduler
>>>> + * determine what order it should be ran in in reference to the other steps that
>>>> + * are part of the same job.
>>>> + */
>>>> +struct media_job_step {
>>>> +	struct list_head list;
>>>> +	void (*run_step)(void *data);
>>>> +	void *data;
>>>> +	unsigned int flags;
>>>> +	unsigned int pos;
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct media_job_dep_ops - Operations to manage a media job dependency
> mmm are these dep_ops or job_ops ?
I think dep ops is the right terminology for this set...though they could probably become function 
pointers against struct media_job_dep if we rework the "try to queue" function to use the calling 
function pointer.
>>>> + *
>>>> + * @check_dep:	A function to ask the driver whether the dependency is met
>>>> + * @clear_dep:	A function to tell the driver that the job has been queued
> Took me a while to get this. check_dep asks the driver to check,
> reset_dep asks the driver to reset, and then clear_dep asks the driver
> to ... clear ?
>
> I would call this something like job_ready (which could possibly be
> paired with device_run and job_abort and we have an identical replica
> of v4l2_m2m_ops. Does this hint something ?)
I did look at v4l2_m2m when writing this and there were some parallels yeah.
>
> Anyway, not feeling strong about this naming thing, but yes, if
> clear_dep could be changed it would be nice


"mark_dep_fulfilled"?

>
>
>>>> + * @reset_dep:	A function to tell the driver that the job has been cancelled
>>>> + *
>>>> + * Media jobs have dependencies, such as requiring buffers to be queued. These
>>>> + * operations allow a driver to define how the media jobs framework should check
>>>> + * whether or not those dependencies are met and how it should inform them that
>>>> + * it is taking action based on the state of those dependencies.
>>>> + */
>>>> +struct media_job_dep_ops {
>>>> +	bool (*check_dep)(void *data);
>>>> +	void (*clear_dep)(void *data);
>>>> +	void (*reset_dep)(void *data);
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct media_job_dep - Representation of media job dependency
>>>> + *
>>>> + * @list:	List head to attach to a &struct media_job.deps
>>>> + * @ops:	A pointer to the dependency's operations functions
>>>> + * @met:	A flag to record whether or not the dependency is met
>>>> + * @data:	Data to pass to the dependency's operations
>>>> + *
>>>> + * This struct represents a dependency of a media job. The operations member
>>>> + * holds pointers to functions allowing the framework to interact with the
>>>> + * driver to check whether or not the dependency is met.
>>>> + */
>>>> +struct media_job_dep {
>>>> +	struct list_head list;
>>>> +	struct media_job_dep_ops *ops;
>>>> +	bool met;
>>>> +	void *data;
> 	struct media_job_dep_ops *ops;
> 	void *data;
> 	bool met;
>
> 	struct list_head list;
>
>>>> +};
>>>> +
>>>> +/**
>>>> + * media_jobs_try_queue_job - Try to queue a &struct media_job
>>>> + *
>>>> + * @sched:	Pointer to the job scheduler
>>>> + * @type:	The type of the media job
>>>> + * @dep_ops:	A pointer to the dependency operations for this job
>>>> + * @dep_data:	A pointer to the dependency data for this job
>>>> + *
>>>> + * Try to queue a media job with the scheduler. This function should be called
>>>> + * by the drivers whenever a dependency for a media job is met - for example
>>>> + * when a buffer is queued to the driver. The framework will check to see if an
>>>> + * existing job on the scheduler's pending list shares the same type, dependency
>>>> + * operations and dependency data. If it does then that existing job will be
>>>> + * considered. If there is no extant job with those same parameters, a new job
>>>> + * is allocated and populated by calling the setup functions registered with
>>>> + * the framework.
> This is about the internals, I wouldn't move part of the documentation
> to the implementation.
Do you mean you **would** move it? As in the details on internal implementation should go to 
media_jobs.c instead?
>
>>>> + *
>>>> + * The function iterates over the dependencies that are registered with the job
>>>> + * and checks to see if they are met. If they're all met, they're cleared and
>>>> + * the job is placed onto the scheduler's queue.
>>>> + *
>>>> + * To help reduce conditionals in drivers where a driver supports both the use
>>>> + * of the media jobs framework and operation without it, this function is a no
>>>> + * op if @sched is NULL.
> Wouldn't a driver written to use the media_jobs framework use the
> framework regardless on the presence of other drivers that use the same
> scheduler on the media device ? In what case sched would be null ?
Well, at the moment I have the C55 driver **not** using the framework if it's in inline mode with 
direct electrical input from the CSI-2 rx...but that was a bit instinctual and now I'm wondering if 
that was the right approach; I think things would continue to work just fine regardless. I'll give 
it a test.
>
>
>>>> + *
>>>> + * Return: 0 on success or a negative error number
>>>> + */
>>>> +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
>>>> +			     enum media_job_types type,
>>>> +			     struct media_job_dep_ops *dep_ops, void *dep_data);
>>>> +
>>>> +/**
>>>> + * media_jobs_add_job_step - Add a step to a media job
>>>> + *
>>>> + * @job:	Pointer to the &struct media_job
>>>> + * @run_step:	Pointer to the function to run to execute the step
>>>> + * @data:	Pointer to the data to pass to @run_ste
>>>> + * @flags:	One of the MEDIA_JOBS_FL_STEP_* flags
>>>> + * @pos:	A position indicator to use with @flags
>>>> + *
>>>> + * This function adds a step to the job and should be called from the drivers'
>>>> + * job setup functions as registered with the framework through
>>>> + * media_jobs_add_job_setup_func(). The @flags and @pos parameters are used
>>>> + * to determine the ordering of the steps within the job:
>>>> + *
>>>> + * If @flags has the MEDIA_JOBS_FL_STEP_ANYWHERE bit set, the step is placed
>>>> + * after all steps with MEDIA_JOBS_FL_STEP_FROM_FRONT and before all steps with
>>>> + * MEDIA_JOBS_FL_STEP_FROM_BACK bit set, but otherwise in whatever order this
>>>> + * function is called.
>>>> + *
>>>> + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_FRONT bit set then the step is
>>>> + * placed @pos steps from the front of the list. Attempting to place multiple
>>>> + * steps in the same position will result in an error.
>>>> + *
>>>> + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_BACK bit set then the step is
>>>> + * placed @pos steps from the back of the list. Attempting to place multiple
>>>> + * steps in the same position will result in an error.
>>>> + *
>>>> + * Return: 0 on success or a negative error number
>>>> + */
>>>> +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
>>>> +			    void *data, unsigned int flags, unsigned int pos);
>>>> +
>>>> +/**
>>>> + * media_jobs_add_job_dep - Add a dependency to a media job
>>>> + *
>>>> + * @job:	Pointer to the &struct media_job
>>>> + * @ops:	Pointer to the &struct media_job_dep_ops
>>>> + * @data:	Pointer to the data to pass to the dependency's operations
>>>> + *
>>>> + * This function adds a dependency to the job and should be called from the
>>>> + * drivers job setup functions as registered with the framework through the
>>>> + * media_jobs_add_job_setup_func() function.
>>>> + *
>>>> + * Return: 0 on success or a negative error number
>>>> + */
>>>> +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
>>>> +			   void *data);
>>>> +
> We'll get back to this when discussing how to identify a dependency
> user
>
>>>> +/**
>>>> + * media_jobs_add_job_setup_func - Add a function that populates a media job
>>>> + *
>>>> + * @sched:	Pointer to the media jobs scheduler
>>>> + * @job_setup:	Pointer to the new job setup function
>>>> + * @data:	Data to pass to the job setup function
>>>> + * @type:	The type of job that this function should be called for
>>>> + *
>>>> + * Drivers that wish to utilise the framework need to use this function to
>>>> + * register a callback that adds job steps and dependencies when one is created.
>>>> + * The function must call media_jobs_add_job_step() and media_jobs_add_job_dep()
>>>> + * to populate the job.
>>>> + *
>>>> + * Return: 0 on success or a negative error number
>>>> + */
>>>> +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
> If you move the scheduler to the media dev, then all these function
> can operate on the media dev and the scheduler would be internal to
> the media dev
>
>>>> +				  int (*job_setup)(struct media_job *job, void *data),
> define a type for this as well maybe
>
>>>> +				  void *data, enum media_job_types type);
>>>> +
>>>> +/**
>>>> + * media_jobs_put_scheduler - Put a reference to the media jobs scheduler
>>>> + *
>>>> + * @sched:	Pointer to the media jobs scheduler
>>>> + *
>>>> + * This function puts a reference to the media jobs scheduler, and is intended
>>>> + * to be called in error and exit paths for consuming drivers
>>>> + */
>>>> +void media_jobs_put_scheduler(struct media_job_scheduler *sched);
>>>> +
>>>> +/**
>>>> + * media_jobs_get_scheduler - Get a media jobs scheduler
>>>> + *
>>>> + * @mdev:	Pointer to the media device associated with the scheduler
>>>> + *
>>>> + * This function gets a pointer to a &struct media_job_scheduler associated with
>>>> + * the media device passed to @mdev. If one is not available then it is
>>>> + * allocated and returned. This allows multiple drivers sharing a media graph to
>>>> + * work with the same media job scheduler.
>>>> + *
>>>> + * Return: 0 on success or a negative error number
>>>> + */
>>>> +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev);
>>>> +
>>>> +/**
>>>> + * media_jobs_run_jobs - Run any media jobs that are ready in the queue
>>>> + *
>>>> + * @sched:	Pointer to the media job scheduler
>>>> + *
>>>> + * This function triggers the workqueue that processes any jobs that have been
>>>> + * queued, and should be called whenever the pipeline is ready to do so.
>>>> + *
>>>> + * To help reduce conditionals in drivers where a driver supports both the use
>>>> + * of the media jobs framework and operation without it, this function is a no
>>>> + * op if @sched is NULL.
>>>> + */
>>>> +void media_jobs_run_jobs(struct media_job_scheduler *sched);
>>>> +
>>>> +/**
>>>> + * media_jobs_cancel_jobs - cancel all waiting jobs
>>>> + *
>>>> + * @sched:	Pointer to the media job scheduler
>>>> + *
>>>> + * This function iterates over any pending and queued jobs, resets their
>>>> + * dependencies and frees the job
>>>> + *
>>>> + * To help reduce conditionals in drivers where a driver supports both the use
>>>> + * of the media jobs framework and operation without it, this function is a no
>>>> + * op if @sched is NULL.
>>>> + */
>>>> +void media_jobs_cancel_jobs(struct media_job_scheduler *sched);
>>>> +
>>>> +extern struct list_head media_job_schedulers;
>>>> +extern struct mutex media_job_schedulers_lock;
> These can (and should) be made static internal to mc-jobs.c


Ack

>
> I've pushed a few patches for you to look at in the meantime


I found them! Thanks, I'll take a look tomorrow.

> very
> nice overall, I think even drivers without extenal dependencies could
> use this framework and simplify the job (I've already seen in at
> least 3 drivers) to maintain queues of jobs where to associate buffers
> into. One step at the time :)


Thanks very much!

>
>>>> +
>>>> +#endif /* _MEDIA_JOBS_H */
>>>> --
>>>> 2.34.1
>>>>
>>>>

Re: [PATCH 2/3] media: mc: Add media jobs framework

[Jacopo Mondi]

Hi Dan

On Thu, May 22, 2025 at 11:36:14PM +0100, Dan Scally wrote:
> Hi Jacopo
>
> On 22/05/2025 20:04, Jacopo Mondi wrote:
> > Hi Dan
> >
> > On Thu, May 22, 2025 at 12:24:46PM +0100, Dan Scally wrote:
> > > Hi Jacopo
> > >
> > > On 22/05/2025 12:00, Jacopo Mondi wrote:
> > > > Hi Dan, back with more questions :)
> > > >
> > > > On Mon, May 19, 2025 at 03:04:02PM +0100, Daniel Scally wrote:
> > > > > Add a new framework to the media subsystem describing media jobs.
> > > > > This framework is intended to be able to model the interactions
> > > > > between multiple different drivers that need to be run in concert
> > > > > to fully control a media pipeline, for example an ISP driver and a
> > > > > driver controlling a DMA device that feeds data from memory in to
> > > > > that ISP.
> > > > >
> > > > > The new framework allows all drivers involved to add explicit steps
> > > > > that need to be performed, and to control the ordering of those steps
> > > > > precisely. Once the job with its steps has been created it's then
> > > > > scheduled to be run with a workqueue which executes each step in the
> > > > > defined order.
> > > > >
> > > > > Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> > > > > ---
> > > > >    drivers/media/mc/Makefile  |   2 +-
> > > > >    drivers/media/mc/mc-jobs.c | 446 +++++++++++++++++++++++++++++++++++++
> > > > >    include/media/media-jobs.h | 354 +++++++++++++++++++++++++++++
> > > > >    3 files changed, 801 insertions(+), 1 deletion(-)
> > > > >    create mode 100644 drivers/media/mc/mc-jobs.c
> > > > >    create mode 100644 include/media/media-jobs.h
> > > > >
> > > > > diff --git a/drivers/media/mc/Makefile b/drivers/media/mc/Makefile
> > > > > index 2b7af42ba59c..9148bbfd1578 100644
> > > > > --- a/drivers/media/mc/Makefile
> > > > > +++ b/drivers/media/mc/Makefile
> > > > > @@ -1,7 +1,7 @@
> > > > >    # SPDX-License-Identifier: GPL-2.0
> > > > >
> > > > >    mc-objs	:= mc-device.o mc-devnode.o mc-entity.o \
> > > > > -	   mc-request.o
> > > > > +	   mc-jobs.o mc-request.o
> > > > >
> > > > >    ifneq ($(CONFIG_USB),)
> > > > >    	mc-objs += mc-dev-allocator.o
> > > > > diff --git a/drivers/media/mc/mc-jobs.c b/drivers/media/mc/mc-jobs.c
> > > > > new file mode 100644
> > > > > index 000000000000..1f04cdf63d27
> > > > > --- /dev/null
> > > > > +++ b/drivers/media/mc/mc-jobs.c
> > > > > @@ -0,0 +1,446 @@
> > > > > +// SPDX-License-Identifier: GPL-2.0-only
> > > > > +/*
> > > > > + * Media jobs framework
> > > > > + *
> > > > > + * Copyright 2025 Ideas on Board Oy
> > > > > + *
> > > > > + * Author: Daniel Scally <dan.scally@ideasonboard.com>
> > > > > + */
> > > > > +
> > > > > +#include <linux/cleanup.h>
> > > > > +#include <linux/kref.h>
> > > > > +#include <linux/list.h>
> > > > > +#include <linux/slab.h>
> > > > > +#include <linux/spinlock.h>
> > > > > +
> > > > > +#include <media/media-device.h>
> > > > > +#include <media/media-entity.h>
> > > > > +#include <media/media-jobs.h>
> > > > > +
> > > > > +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
> > > > > +			    void *data, unsigned int flags, unsigned int pos)
> > > > > +{
> > > > > +	struct media_job_step *step, *tmp;
> > > > > +	unsigned int num = flags;
> > > > > +	unsigned int count = 0;
> > > > > +
> > > > > +	guard(spinlock)(&job->lock);
> > > > > +
> > > > > +	if (!flags) {
> > > > > +		WARN_ONCE(1, "%s(): No flag bits set\n", __func__);
> > > > > +		return -EINVAL;
> > > > > +	}
> > > > > +
> > > > > +	/* Count the number of set flags; they're mutually exclusive. */
> > > > > +	while (num) {
> > > > > +		num &= (num - 1);
> > > > > +		count++;
> > > > > +	}
> > > > > +
> > > > > +	if (count > 1) {
> > > > > +		WARN_ONCE(1, "%s(): Multiple flag bits set\n", __func__);
> > > > > +		return -EINVAL;
> > > > > +	}
> > > > > +
> > > > > +	step = kzalloc(sizeof(*step), GFP_KERNEL);
> > > > > +	if (!step)
> > > > > +		return -ENOMEM;
> > > > > +
> > > > > +	step->run_step = run_step;
> > > > > +	step->data = data;
> > > > > +	step->flags = flags;
> > > > > +	step->pos = pos;
> > > > > +
> > > > > +	/*
> > > > > +	 * We need to decide where to place the step. If the list is empty that
> > > > > +	 * is really easy (and also the later code is much easier if the code is
> > > > > +	 * guaranteed not to be empty...)
> > > > > +	 */
> > > > > +	if (list_empty(&job->steps)) {
> > > > > +		list_add_tail(&step->list, &job->steps);
> > > > > +		return 0;
> > > > > +	}
> > > > > +
> > > > > +	/*
> > > > > +	 * If we've been asked to place it at a specific position from the end
> > > > > +	 * of the list, we cycle back through it until either we exhaust the
> > > > > +	 * list or find an entry that needs to go further from the back than the
> > > > > +	 * new one.
> > > > > +	 */
> > > > > +	if ((flags & MEDIA_JOBS_FL_STEP_FROM_BACK)) {
> > > > > +		list_for_each_entry_reverse(tmp, &job->steps, list) {
> > > > > +			if (tmp->flags == flags && tmp->pos == pos)
> > > > > +				return -EINVAL;
> > > > > +
> > > > > +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_BACK ||
> > > > > +			    tmp->pos > pos)
> > > > > +				break;
> > > > > +		}
> > > > > +
> > > > > +		/*
> > > > > +		 * If the entry we broke on is also one placed from the back and
> > > > > +		 * should be closer to the back than the new one, we place the
> > > > > +		 * new one in front of it...otherwise place the new one behind
> > > > > +		 * it.
> > > > > +		 */
> > > > > +		if (tmp->flags == flags && tmp->pos < pos)
> > > > > +			list_add_tail(&step->list, &tmp->list);
> > > > > +		else
> > > > > +			list_add(&step->list, &tmp->list);
> > > > > +
> > > > > +		return 0;
> > > > > +	}
> > > > > +
> > > > > +	/*
> > > > > +	 * If we've been asked to place it a specific position from the front of
> > > > > +	 * the list we do the same kind of operation, but going from the front
> > > > > +	 * instead.
> > > > > +	 */
> > > > > +	if (flags & MEDIA_JOBS_FL_STEP_FROM_FRONT) {
> > > > > +		list_for_each_entry(tmp, &job->steps, list) {
> > > > > +			if (tmp->flags == flags && tmp->pos == pos)
> > > > > +				return -EINVAL;
> > > > > +
> > > > > +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_FRONT ||
> > > > > +			    tmp->pos > pos)
> > > > > +				break;
> > > > > +		}
> > > > > +
> > > > > +		/*
> > > > > +		 * If the entry we broke on is also placed from the front and
> > > > > +		 * should be closed to the front than the new one, we place the
> > > > > +		 * new one behind it, otherwise in front of it.
> > > > > +		 */
> > > > > +		if (tmp->flags == flags && tmp->pos < pos)
> > > > > +			list_add(&step->list, &tmp->list);
> > > > > +		else
> > > > > +			list_add_tail(&step->list, &tmp->list);
> > > > > +
> > > > > +		return 0;
> > > > > +	}
> > > > > +
> > > > > +	/*
> > > > > +	 * If the step is flagged as "can go anywhere" we just need to try to
> > > > > +	 * find the first "from the back" entry and add it immediately before
> > > > > +	 * that. If we can't find one, add it after whatever we did find.
> > > > > +	 */
> > > > > +	if (flags & MEDIA_JOBS_FL_STEP_ANYWHERE) {
> > > > > +		list_for_each_entry(tmp, &job->steps, list)
> > > > > +			if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK))
> > > > > +				break;
> > > > > +
> > > > > +		if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK) ||
> > > > > +		    list_entry_is_head(tmp, &job->steps, list))
> > > > > +			list_add_tail(&step->list, &tmp->list);
> > > > > +		else
> > > > > +			list_add(&step->list, &tmp->list);
> > > > > +
> > > > > +		return 0;
> > > > > +	}
> > > > > +
> > > > > +	/* Shouldn't get here, unless the flag value is wrong. */
> > > > > +	WARN_ONCE(1, "%s(): Invalid flag value\n", __func__);
> > > > > +	return -EINVAL;
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(media_jobs_add_job_step);
> > > > > +
> > > > > +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
> > > > > +			   void *data)
> > > > > +{
> > > > > +	struct media_job_dep *dep;
> > > > > +
> > > > > +	if (!ops || !ops->check_dep || !data)
> > > > > +		return -EINVAL;
> > > > > +
> > > > > +	guard(spinlock)(&job->lock);
> > > > > +
> > > > > +	/* Confirm the same dependency hasn't already been added */
> > > > > +	list_for_each_entry(dep, &job->deps, list)
> > > > > +		if (dep->ops == ops && dep->data == data)
> > > > > +			return -EINVAL;
> > > > > +
> > > > > +	dep = kzalloc(sizeof(*dep), GFP_KERNEL);
> > > > > +	if (!dep)
> > > > > +		return -ENOMEM;
> > > > > +
> > > > > +	dep->ops = ops;
> > > > > +	dep->data = data;
> > > > > +	list_add(&dep->list, &job->deps);
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(media_jobs_add_job_dep);
> > > > > +
> > > > > +static bool media_jobs_check_pending_job(struct media_job *job,
> > > > > +					 enum media_job_types type,
> > > > > +					 struct media_job_dep_ops *dep_ops,
> > > > > +					 void *data)
> > > > > +{
> > > > > +	struct media_job_dep *dep;
> > > > > +
> > > > > +	guard(spinlock)(&job->lock);
> > > > > +
> > > > > +	if (job->type != type)
> > > > > +		return false;
> > > > > +
> > > > > +	list_for_each_entry(dep, &job->deps, list) {
> > > > > +		if (dep->ops == dep_ops && dep->data == data) {
> > > > > +			if (dep->met)
> > > > > +				return false;
> > > > > +
> > > > > +			break;
> > > > > +		}
> > > > > +	}
> > > > > +
> > > > > +	dep->met = true;
> > > > > +	return true;
> > > > > +}
> > > > > +
> > > > > +static struct media_job *media_jobs_get_job(struct media_job_scheduler *sched,
> > > > > +					    enum media_job_types type,
> > > > > +					    struct media_job_dep_ops *dep_ops,
> > > > > +					    void *dep_data)
> > > > > +{
> > > > > +	struct media_job_setup_func *jsf;
> > > > > +	struct media_job *job;
> > > > > +	int ret;
> > > > > +
> > > > > +	list_for_each_entry(job, &sched->pending, list)
> > > > > +		if (media_jobs_check_pending_job(job, type, dep_ops, dep_data))
> > > > > +			return job;
> > > > > +
> > > > Thanks to your offline explanation, I got how this works now, however
> > > > some questions here
> > > >
> > > > The basic idea is that each driver that registers a 'setup' function
> > > > adds to a job, when its created, its list of dependencies.
> > > >
> > > > When a job is "try_queue" and we get here, to decide if a new job has
> > > > to be created or if we have to run one which is already in the pending
> > > > queue.
> > > >
> > > > How is this identification performed ? Each entry point (assume it's a
> > > > video device op) will populate the job with its own dependencies,
> > > > identified by the dep_ops and data address.
> > > >
> > > > We walk the 'pending' queue in the media_jobs_check_pending_job()
> > > > function and we search for one job not already visited from the same
> > > > entry point, identified by the dep_ops (the 'visited' state is kept by
> > > > the deps->met flag).
> > > >
> > > > Let's assume 2 video devices X and Y
> > > >
> > > > qbuf(x) -> try_queue_job() -> new job created on 'pending'
> > > > qbuf(x) -> try_queue_job() -> the job in the queue has 'deps->met' set, skip
> > > > it and create a new one
> > > > qbuf(y) -> try_queue_job() -> the first job in the queue has not
> > > > deps->set, so return it
> > > >
> > > > All in all I would describe this as: when requesting a job try to find
> > > > the first one not already visited by this entry point, if none is
> > > > available create a new one.
> > >
> > > Yep, all seems fine.
> > >
> > > > Now, we briefly discussed that when moving to multi-context comparing
> > > > dep_ops and data to identify an entry point won't be enough: buffers
> > > > from the same video device but from different contexts do not have to
> > > > be associated together. So we'll need to extend the identification
> > > > criteria. Also, I don't find the idea of using dep_ops and data for
> > > > this purpose particularly neat, as it makes mandatory to add
> > > > dependencies to a job in the setup function, something not all driver
> > > > might want to do ?
> > > I don't think it's mandatory for a driver to add dependencies to a job; the
> > > implication of lacking them is that whatever step the driver is running for
> > > the job takes no input (no buffers need to be available, no parameters need
> > > to have been set, no per-requisites need to have been met) in which case it
> > > can simply be ignored for the purposes of evaluating whether the job can be
> > > queued or not, because it's always ready by definition...does that make
> > > sense?
> > Yes, but what are the implications of not setting deps on
> > media_jobs_check_pending_job() ?
> >
> > If I'm not mistaken
> >
> > 	if (dep->ops == dep_ops && dep->data == data) {
> >
> > will now always return false, as there won't be any 'dep' that matches
> > with dep_ops as the driver has never called media_jobs_add_job_dep().
> >
> > If so, if a video device that doesn't register deps (tbh I don't see
> > why it would, but..) will always match and we will overwrite the same job over
> > and over ? Anyway, a corner case I guess
> Ah, sorry, you are right of course...I suppose we can specifically check for
> NULL in whatever arguments we pass to guarantee uniqueness and handle that.
> >
> > > > There might be ways to handle this "track the entry point" thing that
> > > > could be separated by deps, making deps do what they actually are
> > > > described for: track dependencies to validate if a job can be run or
> > > > not. Before exploring options, I would like to know if this only mine
> > > > concern or is it shared by others.
> > >
> > > I do agree that a nicer way of tracking them rather than dep_ops and data
> > > would be better...perhaps tying it to the entry point as you've
> > > conceptualised here is the right thing to do, and the pointer to the
> > > function calling media_jobs_try_queue_job() should be passed, along with a
> > > context ID?
> > >
> > I think that's a possible way forward. For the time being I think
> > identifying a "user" of a job (iow any drivers that registers a
> > dependency or a step) with a pointer to the video device or to the
> > driver-specific types should be enough and we can easily
> > move it to use the video device context later.
> >
> > I have patches to push if you want to see.
>
>
> Yes please!
>
> >
> > A bit more comments on the API below
> >
> > > Thanks
> > >
> > > Dan
> > >
> > > > > +	job = kzalloc(sizeof(*job), GFP_KERNEL);
> > > > > +	if (!job)
> > > > > +		return ERR_PTR(-ENOMEM);
> > > > > +
> > > > > +	spin_lock_init(&job->lock);
> > > > > +	INIT_LIST_HEAD(&job->deps);
> > > > > +	INIT_LIST_HEAD(&job->steps);
> > > > > +	job->type = type;
> > > > > +	job->sched = sched;
> > > > > +
> > > > > +	list_for_each_entry(jsf, &sched->setup_funcs, list) {
> > > > > +		if (jsf->type != type)
> > > > > +			continue;
> > > > > +
> > > > > +		ret = jsf->job_setup(job, jsf->data);
> > > > > +		if (ret) {
> > > > > +			kfree(job);
> > > > > +			return ERR_PTR(ret);
> > > > > +		}
> > > > > +	}
> > > > > +
> > > > > +	list_add_tail(&job->list, &sched->pending);
> > > > > +
> > > > > +	/* This marks the dependency as met */
> > > > > +	media_jobs_check_pending_job(job, type, dep_ops, dep_data);
> > > > > +
> > > > > +	return job;
> > > > > +}
> > > > > +
> > > > > +static void media_jobs_free_job(struct media_job *job, bool reset)
> > > > > +{
> > > > > +	struct media_job_step *step, *stmp;
> > > > > +	struct media_job_dep *dep, *dtmp;
> > > > > +
> > > > > +	scoped_guard(spinlock, &job->lock) {
> > > > > +		list_for_each_entry_safe(dep, dtmp, &job->deps, list) {
> > > > > +			if (reset && dep->ops->reset_dep)
> > > > > +				dep->ops->reset_dep(dep->data);
> > > > > +
> > > > > +			list_del(&dep->list);
> > > > > +			kfree(dep);
> > > > > +		}
> > > > > +
> > > > > +		list_for_each_entry_safe(step, stmp, &job->steps, list) {
> > > > > +			list_del(&step->list);
> > > > > +			kfree(step);
> > > > > +		}
> > > > > +	}
> > > > > +
> > > > > +	list_del(&job->list);
> > > > > +	kfree(job);
> > > > > +}
> > > > > +
> > > > > +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
> > > > > +			     enum media_job_types type,
> > > > > +			     struct media_job_dep_ops *dep_ops, void *dep_data)
> > > > > +{
> > > > > +	struct media_job_dep *dep;
> > > > > +	struct media_job *job;
> > > > > +
> > > > > +	if (!sched)
> > > > > +		return 0;
> > > > > +
> > > > > +	guard(spinlock)(&sched->lock);
> > > > > +
> > > > > +	job = media_jobs_get_job(sched, type, dep_ops, dep_data);
> > > > > +	if (IS_ERR(job))
> > > > > +		return PTR_ERR(job);
> > > > > +
> > > > > +	list_for_each_entry(dep, &job->deps, list)
> > > > > +		if (!dep->ops->check_dep(dep->data))
> > > > > +			return 0; /* Not a failure */
> > > > > +
> > > > > +	list_for_each_entry(dep, &job->deps, list)
> > > > > +		if (dep->ops->clear_dep)
> > > > > +			dep->ops->clear_dep(dep->data);
> > > > > +
> > > > > +	list_move_tail(&job->list, &sched->queue);
> > > > > +	queue_work(sched->async_wq, &sched->work);
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(media_jobs_try_queue_job);
> > > > > +
> > > > > +static void __media_jobs_run_jobs(struct work_struct *work)
> > > > > +{
> > > > > +	struct media_job_scheduler *sched = container_of(work,
> > > > > +							 struct media_job_scheduler,
> > > > > +							 work);
> > > > > +	struct media_job_step *step;
> > > > > +	struct media_job *job;
> > > > > +
> > > > > +	while (true) {
> > > > > +		scoped_guard(spinlock, &sched->lock) {
> > > > > +			if (list_empty(&sched->queue))
> > > > > +				return;
> > > > > +
> > > > > +			job = list_first_entry(&sched->queue, struct media_job,
> > > > > +					       list);
> > > > > +		}
> > > > > +
> > > > > +		list_for_each_entry(step, &job->steps, list)
> > > > > +			step->run_step(step->data);
> > > > > +
> > > > > +		media_jobs_free_job(job, false);
> > > > > +	}
> > > > > +}
> > > > > +
> > > > > +void media_jobs_run_jobs(struct media_job_scheduler *sched)
> > > > > +{
> > > > > +	if (!sched)
> > > > > +		return;
> > > > > +
> > > > > +	queue_work(sched->async_wq, &sched->work);
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(media_jobs_run_jobs);
> > > > > +
> > > > > +static void __media_jobs_cancel_jobs(struct media_job_scheduler *sched)
> > > > > +{
> > > > > +	struct media_job *job, *jtmp;
> > > > > +
> > > > > +	list_for_each_entry_safe(job, jtmp, &sched->pending, list)
> > > > > +		media_jobs_free_job(job, true);
> > > > > +
> > > > > +	list_for_each_entry_safe(job, jtmp, &sched->queue, list)
> > > > > +		media_jobs_free_job(job, true);
> > > > > +}
> > > > > +
> > > > > +void media_jobs_cancel_jobs(struct media_job_scheduler *sched)
> > > > > +{
> > > > > +	if (!sched)
> > > > > +		return;
> > > > > +
> > > > > +	guard(spinlock)(&sched->lock);
> > > > > +	__media_jobs_cancel_jobs(sched);
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(media_jobs_cancel_jobs);
> > > > > +
> > > > > +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
> > > > > +				  int (*job_setup)(struct media_job *job, void *data),
> > > > > +				  void *data, enum media_job_types type)
> > > > > +{
> > > > > +	struct media_job_setup_func *new_setup_func;
> > > > > +
> > > > > +	guard(spinlock)(&sched->lock);
> > > > > +
> > > > > +	new_setup_func = kzalloc(sizeof(*new_setup_func), GFP_KERNEL);
> > > > > +	if (!new_setup_func)
> > > > > +		return -ENOMEM;
> > > > > +
> > > > > +	new_setup_func->type = type;
> > > > > +	new_setup_func->job_setup = job_setup;
> > > > > +	new_setup_func->data = data;
> > > > > +	list_add_tail(&new_setup_func->list, &sched->setup_funcs);
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(media_jobs_add_job_setup_func);
> > > > > +
> > > > > +static void __media_jobs_put_scheduler(struct kref *kref)
> > > > > +{
> > > > > +	struct media_job_scheduler *sched =
> > > > > +		container_of(kref, struct media_job_scheduler, kref);
> > > > > +	struct media_job_setup_func *func, *ftmp;
> > > > > +
> > > > > +	cancel_work_sync(&sched->work);
> > > > > +	destroy_workqueue(sched->async_wq);
> > > > > +
> > > > > +	scoped_guard(spinlock, &sched->lock) {
> > > > > +		__media_jobs_cancel_jobs(sched);
> > > > > +
> > > > > +		list_for_each_entry_safe(func, ftmp, &sched->setup_funcs, list) {
> > > > > +			list_del(&func->list);
> > > > > +			kfree(func);
> > > > > +		}
> > > > > +	}
> > > > > +
> > > > > +	list_del(&sched->list);
> > > > > +	kfree(sched);
> > > > > +}
> > > > > +
> > > > > +void media_jobs_put_scheduler(struct media_job_scheduler *sched)
> > > > > +{
> > > > > +	kref_put(&sched->kref, __media_jobs_put_scheduler);
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(media_jobs_put_scheduler);
> > > > > +
> > > > > +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev)
> > > > > +{
> > > > > +	struct media_job_scheduler *sched;
> > > > > +	char workqueue_name[32];
> > > > > +	int ret;
> > > > > +
> > > > > +	guard(mutex)(&media_job_schedulers_lock);
> > > > > +
> > > > > +	list_for_each_entry(sched, &media_job_schedulers, list) {
> > > > > +		if (sched->mdev == mdev) {
> > > > > +			kref_get(&sched->kref);
> > > > > +			return sched;
> > > > > +		}
> > > > > +	}
> > > > > +
> > > > > +	ret = snprintf(workqueue_name, sizeof(workqueue_name),
> > > > > +		       "mc jobs (%s)", mdev->driver_name);
> > > > > +	if (!ret)
> > > > > +		return ERR_PTR(-EINVAL);
> > > > > +
> > > > > +	sched = kzalloc(sizeof(*sched), GFP_KERNEL);
> > > > > +	if (!sched)
> > > > > +		return ERR_PTR(-ENOMEM);
> > > > > +
> > > > > +	sched->async_wq = alloc_workqueue(workqueue_name, 0, 0);
> > > > > +	if (!sched->async_wq) {
> > > > > +		kfree(sched);
> > > > > +		return ERR_PTR(-EINVAL);
> > > > > +	}
> > > > > +
> > > > > +	sched->mdev = mdev;
> > > > > +	kref_init(&sched->kref);
> > > > > +	spin_lock_init(&sched->lock);
> > > > > +	INIT_LIST_HEAD(&sched->setup_funcs);
> > > > > +	INIT_LIST_HEAD(&sched->pending);
> > > > > +	INIT_LIST_HEAD(&sched->queue);
> > > > > +	INIT_WORK(&sched->work, __media_jobs_run_jobs);
> > > > > +
> > > > > +	list_add_tail(&sched->list, &media_job_schedulers);
> > > > > +
> > > > > +	return sched;
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(media_jobs_get_scheduler);
> > > > > +
> > > > > +LIST_HEAD(media_job_schedulers);
> > > > > +
> > > > > +/* Synchronise access to the global schedulers list */
> > > > > +DEFINE_MUTEX(media_job_schedulers_lock);
> > > > > diff --git a/include/media/media-jobs.h b/include/media/media-jobs.h
> > > > > new file mode 100644
> > > > > index 000000000000..a97270861251
> > > > > --- /dev/null
> > > > > +++ b/include/media/media-jobs.h
> > > > > @@ -0,0 +1,354 @@
> > > > > +/* SPDX-License-Identifier: GPL-2.0-only */
> > > > > +/*
> > > > > + * Media jobs framework
> > > > > + *
> > > > > + * Copyright 2025 Ideas on Board Oy
> > > > > + *
> > > > > + * Author: Daniel Scally <dan.scally@ideasonboard.com>
> > > > > + */
> > > > > +
> > Could you include the header in the .c file first to make sure it's
> > self-contained ?
> >
> > > > > +#include <linux/kref.h>
> > > > > +#include <linux/list.h>
> > > > > +#include <linux/mutex.h>
> > > > > +#include <linux/spinlock.h>
> > > > > +#include <linux/types.h>
> > > > > +#include <linux/workqueue.h>
> > > > > +
> > > > > +#ifndef _MEDIA_JOBS_H
> > > > > +#define _MEDIA_JOBS_H
> > Place this at the beginning of the file
> >
> > > > > +
> > > > > +struct media_device;
> > > > > +struct media_entity;
> > > > > +struct media_job;
> > > > > +struct media_job_dep;
> > > > > +
> > > > > +/**
> > > > > + * define MEDIA_JOBS_FL_STEP_ANYWHERE - \
> > > > > + *    Flag a media job step as able to run anytime
> > > > > + *
> > > > > + * This flag informs the framework that a job step does not need a particular
> > > > > + * position in the list of job steps and can be placed anywhere.
> > > > > + */
> > > > > +#define MEDIA_JOBS_FL_STEP_ANYWHERE			BIT(0)
> > > > > +
> > > > > +/**
> > > > > + * define MEDIA_JOBS_FL_STEP_FROM_FRONT - \
> > > > > + *    Flag a media job step as needing to be placed near the start of the list
> > > > > + *
> > > > > + * This flag informs the framework that a job step needs to be placed at a set
> > > > > + * position from the start of the list of job steps.
> > > > > + */
> > > > > +#define MEDIA_JOBS_FL_STEP_FROM_FRONT			BIT(1)
> > > > > +
> > > > > +/**
> > > > > + * define MEDIA_JOBS_FL_STEP_FROM_BACK - \
> > > > > + *    Flag a media job step as needing to be placed near the end of the list
> > > > > + *
> > > > > + * This flag informs the framework that a job step needs to be placed at a set
> > > > > + * position from the end of the list of job steps.
> > > > > + */
> > > > > +#define MEDIA_JOBS_FL_STEP_FROM_BACK			BIT(2)
> > > > > +
> > > > > +/**
> > > > > + * enum media_job_types - Type of media job
> > > > > + *
> > > > > + * @MEDIA_JOB_TYPE_PIPELINE_PULSE:	A data event moving through the media
> > > > > + *					pipeline
> > > > > + *
> > > > > + * This enumeration details different types of media jobs. The type can be used
> > > > > + * to differentiate between which steps and dependencies a driver needs to add
> > > > > + * to a job when it is created.
> > > > > + */
> > > > > +enum media_job_types {
> > > > > +	MEDIA_JOB_TYPE_PIPELINE_PULSE,
> > > > > +};
> > I'm wondering if we could omit type for the time being.
> > A little trick would be to use variadic macros to allow users to not
> > specify the type and default it to MEDIA_JOB_TYPE_PIPELINE_PULSE and
> > at the same time allow future users of other types to specify them
> > later on
> >
> > Something like (not based on this version of the patch sorry)
> >
> > -int media_jobs_try_queue_job(struct media_job_scheduler *sched,
> > -                            enum media_job_types type,
> > -                            void *cookie);
> > +int __media_jobs_try_queue_job(struct media_job_scheduler *sched,
> > +                              void *cookie, enum media_job_types type);
> > +
> > +#define __media_jobs_try_queue_job_notype(sched, cookie)       \
> > +       __media_jobs_try_queue_job(sched, cookie, MEDIA_JOB_TYPE_PIPELINE_PULSE)
> > +#define __media_jobs_try_queue_job_type(sched, cookie, type)   \
> > +       __media_jobs_try_queue_job(sched, cookie, type)
> >
> > +#define MEDIA_JOBS_TRY_QUEUE_EXPAND(_1, _2, _3, FUNC, ...) FUNC
> > +#define media_jobs_try_queue_job(...)                                  \
> > +       MEDIA_JOBS_TRY_QUEUE_EXPAND(__VA_ARGS__,                        \
> > +                                   __media_jobs_try_queue_job_type,    \
> > +                                   __media_jobs_try_queue_job_notype)  \
> > +                                   (__VA_ARGS__)
> >
> > Maybe not good looking but allows to call both
> >
> > media_jobs_try_queue_job(sched, cookie, type) and
> > media_jobs_try_queue_job(sched, cookie)
> >
> > (this could be done to all functions that accepts a type, apart the
> > job setup functions)
> >
> I'm fine with something like this...I feel like the option to have multiple
> types of job for a single scheduler is worth keeping around, but making
> things so the users don't have to deal with it for now is probably good.
> > > > > +
> > > > > +/**
> > > > > + * struct media_job_scheduler - A job scheduler for a particular media device
> > > > > + *
> > > > > + * @mdev:		Media device this scheduler is for
> > > > > + * @list:		List head to attach to the global list of schedulers
> > > > > + * @kref:		Reference counter
> > > > > + * @lock:		Lock to protect access to the scheduler
> > > > > + * @setup_funcs:	List of &struct media_job_setup_func to populate jobs
> > > > > + * @pending:		List of &struct media_jobs created but not yet queued
> > > > > + * @queue:		List of &struct media_jobs queued to the scheduler
> > > > > + * @work:		Work item to run the jobs
> > > > > + * @async_wq:		Workqueue to run the work on
> > > > > + *
> > > > > + * This struct is the main job scheduler struct - drivers wanting to use this
> > > > > + * framework should acquire an instance through media_jobs_get_scheduler() and
> > > > > + * subsequently populate it with job setup functions.
> > > > > + */
> > > > > +struct media_job_scheduler {
> > > > > +	struct media_device *mdev;
> > > > > +	struct list_head list;
> > > > > +	struct kref kref;
> > > > > +
> > > > > +	spinlock_t lock; /* Synchronise access to the struct's lists */
> > > > > +	struct list_head setup_funcs;
> > > > > +	struct list_head pending;
> > > > > +	struct list_head queue;
> > Empty line please
> >
> > > > > +	struct work_struct work;
> > > > > +	struct workqueue_struct *async_wq;
> > > > > +};
> > This type should be defined inside the .c file and not exposed ?
>
>
> Ah, yes, thanks.
>
> >
> > > > > +
> > > > > +/**
> > > > > + * struct media_job_setup_func - A function to populate a media job with steps
> > > > > + *				 and dependencies
> > > > > + *
> > > > > + * @list:	The list object to attach to the scheduler
> > > > > + * @type:	The &enum media_job_types that this function populates a job for
> > > > > + * @job_setup:	Function pointer to the driver's job setup function
> > > > > + * @data:	Pointer to the driver data for use with @job_setup
> > > > > + *
> > > > > + * This struct holds data about the functions a driver registers with the jobs
> > > > > + * framework in order to populate a new job with steps and dependencies.
> > > > > + */
> > > > > +struct media_job_setup_func {
> > > > > +	struct list_head list;
> > > > > +	enum media_job_types type;
> > > > > +	int (*job_setup)(struct media_job *job, void *data);
> > You could define a type and use it. I would do that for all functions
> > where it makes sense
>
>
> A typedef for the function signature you mean? If I'm honest I never quite
> saw a reason to do that; what's it in aid of?
>

That you could

#typedef int (*job_setup_fn)(struct media_job *job, void *data);

struct media_job_setup_func {
	struct list_head list;
	enum media_job_types type;
	job_setup_fn job_setup;
	void *data;
};

Similar in the function signature

 int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
-                                 int (*job_setup)(struct media_job *job, void *data),
-                                 void *data, enum media_job_types type);
+                                 job_setup_fn job_setup, void *data,
+                                 enum media_job_types type);

Ayway, not a big deal, I'll make a patch, feel free to ignore it.

> >
> > > > > +	void *data;
> > > > > +};
> > > > > +
> > > > > +/**
> > > > > + * struct media_job - A representation of a job to be run through the pipeline
> > > > > + *
> > > > > + * @lock:	Lock to protect access to the job's lists
> > > > > + * @list:	List head to attach the job to &struct media_job_scheduler in
> > > > > + *		either the pending or queue lists
> > > > > + * @steps:	List of &struct media_job_step to run the job
> > > > > + * @deps:	List of &struct media_job_dep to check that the job can be
> > > > > + *		queued
> > > > > + * @sched:	Pointer to the media job scheduler
> > > > > + * @type:	The type of the job
> > > > > + *
> > > > > + * This struct holds lists of steps that need to be performed to carry out a
> > > > > + * job in the pipeline. A separate list of dependencies allows the queueing of
> > > > > + * the job to be delayed until all drivers are ready to carry it out.
> > > > > + */
> > > > > +struct media_job {
> > > > > +	spinlock_t lock; /* Synchronise access to the struct's lists 6*/
> > > > > +	struct list_head list;
> > > > > +	struct list_head steps;
> > > > > +	struct list_head deps;
> > > > > +	struct media_job_scheduler *sched;
> > > > > +	enum media_job_types type;
> > nit: list_head are usually at the end, at least it seems to to me.
> > Doesn't
> >
> > 	struct media_job_scheduler *sched;
> > 	enum media_job_types type;
> >
> > 	spinlock_t lock; /* Synchronise access to the struct's lists 6*/
> > 	struct list_head list;
> > 	struct list_head steps;
> > 	struct list_head deps;
> >
> > look better ?
>
>
> Yes :)
>
> >
> > };
> > > > > +
> > > > > +/**
> > > > > + * struct media_job_step - A holder for a function to run as part of a job
> > > > > + *
> > > > > + * @list:	List head to attach the job step to a &struct media_job.steps
> > > > > + * @run_step:	The function to run to perform the step
> > > > > + * @data:	Data to pass to the .run_step() function
> > > > > + * @flags:	Flags to control how the step is ordered within the job's list
> > > > > + *		of steps
> > > > > + * @pos:	Position indicator to control how the step is ordered within the
> > > > > + *		job's list of steps
> > > > > + *
> > > > > + * This struct defines a function that needs to be run as part of the execution
> > > > > + * of a job in a media pipeline, along with information that help the scheduler
> > > > > + * determine what order it should be ran in in reference to the other steps that
> > > > > + * are part of the same job.
> > > > > + */
> > > > > +struct media_job_step {
> > > > > +	struct list_head list;
> > > > > +	void (*run_step)(void *data);
> > > > > +	void *data;
> > > > > +	unsigned int flags;
> > > > > +	unsigned int pos;
> > > > > +};
> > > > > +
> > > > > +/**
> > > > > + * struct media_job_dep_ops - Operations to manage a media job dependency
> > mmm are these dep_ops or job_ops ?
> I think dep ops is the right terminology for this set...though they could
> probably become function pointers against struct media_job_dep if we rework
> the "try to queue" function to use the calling function pointer.
> > > > > + *
> > > > > + * @check_dep:	A function to ask the driver whether the dependency is met
> > > > > + * @clear_dep:	A function to tell the driver that the job has been queued
> > Took me a while to get this. check_dep asks the driver to check,
> > reset_dep asks the driver to reset, and then clear_dep asks the driver
> > to ... clear ?
> >
> > I would call this something like job_ready (which could possibly be
> > paired with device_run and job_abort and we have an identical replica
> > of v4l2_m2m_ops. Does this hint something ?)
> I did look at v4l2_m2m when writing this and there were some parallels yeah.
> >
> > Anyway, not feeling strong about this naming thing, but yes, if
> > clear_dep could be changed it would be nice
>
>
> "mark_dep_fulfilled"?
>

mmm, ok, this function is called after -all- dependencies for a job
have been met

	list_for_each_entry(dep, &job->deps, list)
		if (!dep->ops->check_dep(dep->user))
			return 0; /* Not a failure */

	list_for_each_entry(dep, &job->deps, list)
		if (dep->ops->clear_dep)
			dep->ops->clear_dep(dep->user);

Just before queueing the job

	list_move_tail(&job->list, &sched->queue);
	queue_work(sched->async_wq, &sched->work);

To me this is actually a job_run() and that's why I was hinting
about making these media_job_ops and detach them from the concept of
dependencies

struct media_jobs_ops {
	bool (*job_ready)(void *data);
	void (*job_run)(void *data);
	void (*job_abort)(void *data);
};

In this way we can make the "add_dep" interface simply about
registering callbacks (sorry, still based on my patches that add a
user id cookie to the interface)

int media_jobs_add_job_ops(struct media_job *job, void *cookie,
			   struct media_job_ops *ops);

And document that the 'job_ready' one is called to make sure all job
users are ready, once they are the 'job_run' function is called
before running the job's steps and finlly 'job_abort' is called on
cancelled jobs that have not been queued.

I'll make a patch and let you decide if you like it or not.

But I would go one step further: I think the 'dependency' and the
'job_setup' function can be simply provided at
media_jobs_add_job_setup_func() time, where you provide a user id and
a set of operations, the three above plus one "job_prepare" that is
what the job_setup function is today.

The most notable change in the way the media_jobs API have to be used
is in the fact that now, using the mali c55 driver as an example, only
the core.c module registers a 'job_setup' function, and then call into
submodules that do "add_deps" and "add_steps". With the new proposal
each driver has to call a "register" function, but I think most of the
boilerplate code can be hidden in the core framework and we can ask
drivers to only provide a pointer to a set of operations.

Why I would like this re-design, you might ask: I think we can quite
easily move the registration of the job_ops for a video device to the
framework, during the video_device initialization. In this way all
drivers will opt-in to use the media_jobs framework simply by
providing a set of media_jobs_ops at registration time, no other
instrumentation is needed. And when we'll have contexts, whenever we
allocate a new context for each video dev, we allocate one 'user' for
the media jobs framework with its operations associated.

I would like to know from you about the "drivers instrumeted to use
the media_jobs framework but not using it" for which you have provided
a sched == NULL safeguard. Would they work in such scenario ?

> >
> >
> > > > > + * @reset_dep:	A function to tell the driver that the job has been cancelled
> > > > > + *
> > > > > + * Media jobs have dependencies, such as requiring buffers to be queued. These
> > > > > + * operations allow a driver to define how the media jobs framework should check
> > > > > + * whether or not those dependencies are met and how it should inform them that
> > > > > + * it is taking action based on the state of those dependencies.
> > > > > + */
> > > > > +struct media_job_dep_ops {
> > > > > +	bool (*check_dep)(void *data);
> > > > > +	void (*clear_dep)(void *data);
> > > > > +	void (*reset_dep)(void *data);
> > > > > +};
> > > > > +
> > > > > +/**
> > > > > + * struct media_job_dep - Representation of media job dependency
> > > > > + *
> > > > > + * @list:	List head to attach to a &struct media_job.deps
> > > > > + * @ops:	A pointer to the dependency's operations functions
> > > > > + * @met:	A flag to record whether or not the dependency is met
> > > > > + * @data:	Data to pass to the dependency's operations
> > > > > + *
> > > > > + * This struct represents a dependency of a media job. The operations member
> > > > > + * holds pointers to functions allowing the framework to interact with the
> > > > > + * driver to check whether or not the dependency is met.
> > > > > + */
> > > > > +struct media_job_dep {
> > > > > +	struct list_head list;
> > > > > +	struct media_job_dep_ops *ops;
> > > > > +	bool met;
> > > > > +	void *data;
> > 	struct media_job_dep_ops *ops;
> > 	void *data;
> > 	bool met;
> >
> > 	struct list_head list;
> >
> > > > > +};
> > > > > +
> > > > > +/**
> > > > > + * media_jobs_try_queue_job - Try to queue a &struct media_job
> > > > > + *
> > > > > + * @sched:	Pointer to the job scheduler
> > > > > + * @type:	The type of the media job
> > > > > + * @dep_ops:	A pointer to the dependency operations for this job
> > > > > + * @dep_data:	A pointer to the dependency data for this job
> > > > > + *
> > > > > + * Try to queue a media job with the scheduler. This function should be called
> > > > > + * by the drivers whenever a dependency for a media job is met - for example
> > > > > + * when a buffer is queued to the driver. The framework will check to see if an
> > > > > + * existing job on the scheduler's pending list shares the same type, dependency
> > > > > + * operations and dependency data. If it does then that existing job will be
> > > > > + * considered. If there is no extant job with those same parameters, a new job
> > > > > + * is allocated and populated by calling the setup functions registered with
> > > > > + * the framework.
> > This is about the internals, I wouldn't move part of the documentation
> > to the implementation.
> Do you mean you **would** move it? As in the details on internal
> implementation should go to media_jobs.c instead?

eheh yes, sorry

> >
> > > > > + *
> > > > > + * The function iterates over the dependencies that are registered with the job
> > > > > + * and checks to see if they are met. If they're all met, they're cleared and
> > > > > + * the job is placed onto the scheduler's queue.
> > > > > + *
> > > > > + * To help reduce conditionals in drivers where a driver supports both the use
> > > > > + * of the media jobs framework and operation without it, this function is a no
> > > > > + * op if @sched is NULL.
> > Wouldn't a driver written to use the media_jobs framework use the
> > framework regardless on the presence of other drivers that use the same
> > scheduler on the media device ? In what case sched would be null ?
> Well, at the moment I have the C55 driver **not** using the framework if
> it's in inline mode with direct electrical input from the CSI-2 rx...but
> that was a bit instinctual and now I'm wondering if that was the right
> approach; I think things would continue to work just fine regardless. I'll
> give it a test.
> >

Thanks, what I'm missing at the moment is how a driver instrumented to
use media jobs (which means has its routines shaped as 'deps' and
'setup' functions already) could work if the framework ignores
media_jobs requests with sched == NULL.

> >
> > > > > + *
> > > > > + * Return: 0 on success or a negative error number
> > > > > + */
> > > > > +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
> > > > > +			     enum media_job_types type,
> > > > > +			     struct media_job_dep_ops *dep_ops, void *dep_data);
> > > > > +
> > > > > +/**
> > > > > + * media_jobs_add_job_step - Add a step to a media job
> > > > > + *
> > > > > + * @job:	Pointer to the &struct media_job
> > > > > + * @run_step:	Pointer to the function to run to execute the step
> > > > > + * @data:	Pointer to the data to pass to @run_ste
> > > > > + * @flags:	One of the MEDIA_JOBS_FL_STEP_* flags
> > > > > + * @pos:	A position indicator to use with @flags
> > > > > + *
> > > > > + * This function adds a step to the job and should be called from the drivers'
> > > > > + * job setup functions as registered with the framework through
> > > > > + * media_jobs_add_job_setup_func(). The @flags and @pos parameters are used
> > > > > + * to determine the ordering of the steps within the job:
> > > > > + *
> > > > > + * If @flags has the MEDIA_JOBS_FL_STEP_ANYWHERE bit set, the step is placed
> > > > > + * after all steps with MEDIA_JOBS_FL_STEP_FROM_FRONT and before all steps with
> > > > > + * MEDIA_JOBS_FL_STEP_FROM_BACK bit set, but otherwise in whatever order this
> > > > > + * function is called.
> > > > > + *
> > > > > + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_FRONT bit set then the step is
> > > > > + * placed @pos steps from the front of the list. Attempting to place multiple
> > > > > + * steps in the same position will result in an error.
> > > > > + *
> > > > > + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_BACK bit set then the step is
> > > > > + * placed @pos steps from the back of the list. Attempting to place multiple
> > > > > + * steps in the same position will result in an error.
> > > > > + *
> > > > > + * Return: 0 on success or a negative error number
> > > > > + */
> > > > > +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
> > > > > +			    void *data, unsigned int flags, unsigned int pos);
> > > > > +
> > > > > +/**
> > > > > + * media_jobs_add_job_dep - Add a dependency to a media job
> > > > > + *
> > > > > + * @job:	Pointer to the &struct media_job
> > > > > + * @ops:	Pointer to the &struct media_job_dep_ops
> > > > > + * @data:	Pointer to the data to pass to the dependency's operations
> > > > > + *
> > > > > + * This function adds a dependency to the job and should be called from the
> > > > > + * drivers job setup functions as registered with the framework through the
> > > > > + * media_jobs_add_job_setup_func() function.
> > > > > + *
> > > > > + * Return: 0 on success or a negative error number
> > > > > + */
> > > > > +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
> > > > > +			   void *data);
> > > > > +
> > We'll get back to this when discussing how to identify a dependency
> > user
> >
> > > > > +/**
> > > > > + * media_jobs_add_job_setup_func - Add a function that populates a media job
> > > > > + *
> > > > > + * @sched:	Pointer to the media jobs scheduler
> > > > > + * @job_setup:	Pointer to the new job setup function
> > > > > + * @data:	Data to pass to the job setup function
> > > > > + * @type:	The type of job that this function should be called for
> > > > > + *
> > > > > + * Drivers that wish to utilise the framework need to use this function to
> > > > > + * register a callback that adds job steps and dependencies when one is created.
> > > > > + * The function must call media_jobs_add_job_step() and media_jobs_add_job_dep()
> > > > > + * to populate the job.
> > > > > + *
> > > > > + * Return: 0 on success or a negative error number
> > > > > + */
> > > > > +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
> > If you move the scheduler to the media dev, then all these function
> > can operate on the media dev and the scheduler would be internal to
> > the media dev
> >
> > > > > +				  int (*job_setup)(struct media_job *job, void *data),
> > define a type for this as well maybe
> >
> > > > > +				  void *data, enum media_job_types type);
> > > > > +
> > > > > +/**
> > > > > + * media_jobs_put_scheduler - Put a reference to the media jobs scheduler
> > > > > + *
> > > > > + * @sched:	Pointer to the media jobs scheduler
> > > > > + *
> > > > > + * This function puts a reference to the media jobs scheduler, and is intended
> > > > > + * to be called in error and exit paths for consuming drivers
> > > > > + */
> > > > > +void media_jobs_put_scheduler(struct media_job_scheduler *sched);
> > > > > +
> > > > > +/**
> > > > > + * media_jobs_get_scheduler - Get a media jobs scheduler
> > > > > + *
> > > > > + * @mdev:	Pointer to the media device associated with the scheduler
> > > > > + *
> > > > > + * This function gets a pointer to a &struct media_job_scheduler associated with
> > > > > + * the media device passed to @mdev. If one is not available then it is
> > > > > + * allocated and returned. This allows multiple drivers sharing a media graph to
> > > > > + * work with the same media job scheduler.
> > > > > + *
> > > > > + * Return: 0 on success or a negative error number
> > > > > + */
> > > > > +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev);
> > > > > +
> > > > > +/**
> > > > > + * media_jobs_run_jobs - Run any media jobs that are ready in the queue
> > > > > + *
> > > > > + * @sched:	Pointer to the media job scheduler
> > > > > + *
> > > > > + * This function triggers the workqueue that processes any jobs that have been
> > > > > + * queued, and should be called whenever the pipeline is ready to do so.
> > > > > + *
> > > > > + * To help reduce conditionals in drivers where a driver supports both the use
> > > > > + * of the media jobs framework and operation without it, this function is a no
> > > > > + * op if @sched is NULL.
> > > > > + */
> > > > > +void media_jobs_run_jobs(struct media_job_scheduler *sched);
> > > > > +
> > > > > +/**
> > > > > + * media_jobs_cancel_jobs - cancel all waiting jobs
> > > > > + *
> > > > > + * @sched:	Pointer to the media job scheduler
> > > > > + *
> > > > > + * This function iterates over any pending and queued jobs, resets their
> > > > > + * dependencies and frees the job
> > > > > + *
> > > > > + * To help reduce conditionals in drivers where a driver supports both the use
> > > > > + * of the media jobs framework and operation without it, this function is a no
> > > > > + * op if @sched is NULL.
> > > > > + */
> > > > > +void media_jobs_cancel_jobs(struct media_job_scheduler *sched);
> > > > > +
> > > > > +extern struct list_head media_job_schedulers;
> > > > > +extern struct mutex media_job_schedulers_lock;
> > These can (and should) be made static internal to mc-jobs.c
>
>
> Ack
>
> >
> > I've pushed a few patches for you to look at in the meantime
>
>
> I found them! Thanks, I'll take a look tomorrow.
>
> > very
> > nice overall, I think even drivers without extenal dependencies could
> > use this framework and simplify the job (I've already seen in at
> > least 3 drivers) to maintain queues of jobs where to associate buffers
> > into. One step at the time :)
>
>
> Thanks very much!
>
> >
> > > > > +
> > > > > +#endif /* _MEDIA_JOBS_H */
> > > > > --
> > > > > 2.34.1
> > > > >
> > > > >

Re: [PATCH 2/3] media: mc: Add media jobs framework

[Sakari Ailus]

Hi Daniel,

On Mon, May 19, 2025 at 03:04:02PM +0100, Daniel Scally wrote:
> Add a new framework to the media subsystem describing media jobs.
> This framework is intended to be able to model the interactions
> between multiple different drivers that need to be run in concert
> to fully control a media pipeline, for example an ISP driver and a
> driver controlling a DMA device that feeds data from memory in to
> that ISP.
> 
> The new framework allows all drivers involved to add explicit steps
> that need to be performed, and to control the ordering of those steps
> precisely. Once the job with its steps has been created it's then
> scheduled to be run with a workqueue which executes each step in the
> defined order.
> 
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
>  drivers/media/mc/Makefile  |   2 +-
>  drivers/media/mc/mc-jobs.c | 446 +++++++++++++++++++++++++++++++++++++
>  include/media/media-jobs.h | 354 +++++++++++++++++++++++++++++
>  3 files changed, 801 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/media/mc/mc-jobs.c
>  create mode 100644 include/media/media-jobs.h
> 
> diff --git a/drivers/media/mc/Makefile b/drivers/media/mc/Makefile
> index 2b7af42ba59c..9148bbfd1578 100644
> --- a/drivers/media/mc/Makefile
> +++ b/drivers/media/mc/Makefile
> @@ -1,7 +1,7 @@
>  # SPDX-License-Identifier: GPL-2.0
>  
>  mc-objs	:= mc-device.o mc-devnode.o mc-entity.o \
> -	   mc-request.o
> +	   mc-jobs.o mc-request.o
>  
>  ifneq ($(CONFIG_USB),)
>  	mc-objs += mc-dev-allocator.o
> diff --git a/drivers/media/mc/mc-jobs.c b/drivers/media/mc/mc-jobs.c
> new file mode 100644
> index 000000000000..1f04cdf63d27
> --- /dev/null
> +++ b/drivers/media/mc/mc-jobs.c
> @@ -0,0 +1,446 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Media jobs framework
> + *
> + * Copyright 2025 Ideas on Board Oy
> + *
> + * Author: Daniel Scally <dan.scally@ideasonboard.com>
> + */
> +
> +#include <linux/cleanup.h>
> +#include <linux/kref.h>
> +#include <linux/list.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +
> +#include <media/media-device.h>
> +#include <media/media-entity.h>
> +#include <media/media-jobs.h>
> +
> +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
> +			    void *data, unsigned int flags, unsigned int pos)
> +{
> +	struct media_job_step *step, *tmp;
> +	unsigned int num = flags;
> +	unsigned int count = 0;
> +
> +	guard(spinlock)(&job->lock);
> +
> +	if (!flags) {
> +		WARN_ONCE(1, "%s(): No flag bits set\n", __func__);
> +		return -EINVAL;
> +	}
> +
> +	/* Count the number of set flags; they're mutually exclusive. */
> +	while (num) {
> +		num &= (num - 1);

Extra parentheses.

But is this correct? Maybe for the purpose it does the right thing, even if
the naming suggests it does something else it does?

I'd use ffs() / ffz().

> +		count++;
> +	}
> +
> +	if (count > 1) {
> +		WARN_ONCE(1, "%s(): Multiple flag bits set\n", __func__);
> +		return -EINVAL;
> +	}
> +
> +	step = kzalloc(sizeof(*step), GFP_KERNEL);
> +	if (!step)
> +		return -ENOMEM;
> +
> +	step->run_step = run_step;
> +	step->data = data;
> +	step->flags = flags;
> +	step->pos = pos;
> +
> +	/*
> +	 * We need to decide where to place the step. If the list is empty that
> +	 * is really easy (and also the later code is much easier if the code is
> +	 * guaranteed not to be empty...)
> +	 */
> +	if (list_empty(&job->steps)) {
> +		list_add_tail(&step->list, &job->steps);
> +		return 0;
> +	}
> +
> +	/*
> +	 * If we've been asked to place it at a specific position from the end
> +	 * of the list, we cycle back through it until either we exhaust the
> +	 * list or find an entry that needs to go further from the back than the
> +	 * new one.
> +	 */
> +	if ((flags & MEDIA_JOBS_FL_STEP_FROM_BACK)) {

Extra parentheses.

> +		list_for_each_entry_reverse(tmp, &job->steps, list) {
> +			if (tmp->flags == flags && tmp->pos == pos)
> +				return -EINVAL;
> +
> +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_BACK ||
> +			    tmp->pos > pos)
> +				break;
> +		}
> +
> +		/*
> +		 * If the entry we broke on is also one placed from the back and
> +		 * should be closer to the back than the new one, we place the
> +		 * new one in front of it...otherwise place the new one behind
> +		 * it.
> +		 */
> +		if (tmp->flags == flags && tmp->pos < pos)
> +			list_add_tail(&step->list, &tmp->list);
> +		else
> +			list_add(&step->list, &tmp->list);
> +
> +		return 0;
> +	}
> +
> +	/*
> +	 * If we've been asked to place it a specific position from the front of
> +	 * the list we do the same kind of operation, but going from the front
> +	 * instead.
> +	 */
> +	if (flags & MEDIA_JOBS_FL_STEP_FROM_FRONT) {
> +		list_for_each_entry(tmp, &job->steps, list) {
> +			if (tmp->flags == flags && tmp->pos == pos)
> +				return -EINVAL;
> +
> +			if (tmp->flags != MEDIA_JOBS_FL_STEP_FROM_FRONT ||
> +			    tmp->pos > pos)
> +				break;
> +		}
> +
> +		/*
> +		 * If the entry we broke on is also placed from the front and
> +		 * should be closed to the front than the new one, we place the
> +		 * new one behind it, otherwise in front of it.
> +		 */
> +		if (tmp->flags == flags && tmp->pos < pos)
> +			list_add(&step->list, &tmp->list);
> +		else
> +			list_add_tail(&step->list, &tmp->list);
> +
> +		return 0;
> +	}
> +
> +	/*
> +	 * If the step is flagged as "can go anywhere" we just need to try to
> +	 * find the first "from the back" entry and add it immediately before
> +	 * that. If we can't find one, add it after whatever we did find.
> +	 */
> +	if (flags & MEDIA_JOBS_FL_STEP_ANYWHERE) {
> +		list_for_each_entry(tmp, &job->steps, list)
> +			if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK))
> +				break;
> +
> +		if ((tmp->flags & MEDIA_JOBS_FL_STEP_FROM_BACK) ||
> +		    list_entry_is_head(tmp, &job->steps, list))
> +			list_add_tail(&step->list, &tmp->list);
> +		else
> +			list_add(&step->list, &tmp->list);
> +
> +		return 0;
> +	}
> +
> +	/* Shouldn't get here, unless the flag value is wrong. */
> +	WARN_ONCE(1, "%s(): Invalid flag value\n", __func__);
> +	return -EINVAL;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_add_job_step);
> +
> +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
> +			   void *data)
> +{
> +	struct media_job_dep *dep;
> +
> +	if (!ops || !ops->check_dep || !data)
> +		return -EINVAL;
> +
> +	guard(spinlock)(&job->lock);
> +
> +	/* Confirm the same dependency hasn't already been added */
> +	list_for_each_entry(dep, &job->deps, list)
> +		if (dep->ops == ops && dep->data == data)
> +			return -EINVAL;
> +
> +	dep = kzalloc(sizeof(*dep), GFP_KERNEL);
> +	if (!dep)
> +		return -ENOMEM;
> +
> +	dep->ops = ops;
> +	dep->data = data;
> +	list_add(&dep->list, &job->deps);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_add_job_dep);
> +
> +static bool media_jobs_check_pending_job(struct media_job *job,
> +					 enum media_job_types type,
> +					 struct media_job_dep_ops *dep_ops,
> +					 void *data)
> +{
> +	struct media_job_dep *dep;
> +
> +	guard(spinlock)(&job->lock);
> +
> +	if (job->type != type)
> +		return false;
> +
> +	list_for_each_entry(dep, &job->deps, list) {
> +		if (dep->ops == dep_ops && dep->data == data) {
> +			if (dep->met)
> +				return false;
> +
> +			break;
> +		}
> +	}
> +
> +	dep->met = true;

How about a newline here?

> +	return true;
> +}
> +
> +static struct media_job *media_jobs_get_job(struct media_job_scheduler *sched,
> +					    enum media_job_types type,
> +					    struct media_job_dep_ops *dep_ops,
> +					    void *dep_data)
> +{
> +	struct media_job_setup_func *jsf;
> +	struct media_job *job;
> +	int ret;
> +

How about lockdep_assert_held(&sched->lock) here?

> +	list_for_each_entry(job, &sched->pending, list)
> +		if (media_jobs_check_pending_job(job, type, dep_ops, dep_data))
> +			return job;
> +
> +	job = kzalloc(sizeof(*job), GFP_KERNEL);
> +	if (!job)
> +		return ERR_PTR(-ENOMEM);
> +
> +	spin_lock_init(&job->lock);
> +	INIT_LIST_HEAD(&job->deps);
> +	INIT_LIST_HEAD(&job->steps);
> +	job->type = type;
> +	job->sched = sched;
> +
> +	list_for_each_entry(jsf, &sched->setup_funcs, list) {
> +		if (jsf->type != type)
> +			continue;
> +
> +		ret = jsf->job_setup(job, jsf->data);
> +		if (ret) {
> +			kfree(job);
> +			return ERR_PTR(ret);
> +		}
> +	}
> +
> +	list_add_tail(&job->list, &sched->pending);
> +
> +	/* This marks the dependency as met */
> +	media_jobs_check_pending_job(job, type, dep_ops, dep_data);
> +
> +	return job;
> +}
> +
> +static void media_jobs_free_job(struct media_job *job, bool reset)
> +{
> +	struct media_job_step *step, *stmp;
> +	struct media_job_dep *dep, *dtmp;
> +
> +	scoped_guard(spinlock, &job->lock) {
> +		list_for_each_entry_safe(dep, dtmp, &job->deps, list) {
> +			if (reset && dep->ops->reset_dep)
> +				dep->ops->reset_dep(dep->data);
> +
> +			list_del(&dep->list);
> +			kfree(dep);
> +		}
> +
> +		list_for_each_entry_safe(step, stmp, &job->steps, list) {
> +			list_del(&step->list);
> +			kfree(step);
> +		}
> +	}
> +
> +	list_del(&job->list);
> +	kfree(job);
> +}
> +
> +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
> +			     enum media_job_types type,
> +			     struct media_job_dep_ops *dep_ops, void *dep_data)
> +{
> +	struct media_job_dep *dep;
> +	struct media_job *job;
> +
> +	if (!sched)
> +		return 0;
> +
> +	guard(spinlock)(&sched->lock);
> +
> +	job = media_jobs_get_job(sched, type, dep_ops, dep_data);
> +	if (IS_ERR(job))
> +		return PTR_ERR(job);
> +
> +	list_for_each_entry(dep, &job->deps, list)
> +		if (!dep->ops->check_dep(dep->data))
> +			return 0; /* Not a failure */
> +
> +	list_for_each_entry(dep, &job->deps, list)
> +		if (dep->ops->clear_dep)
> +			dep->ops->clear_dep(dep->data);
> +
> +	list_move_tail(&job->list, &sched->queue);
> +	queue_work(sched->async_wq, &sched->work);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_try_queue_job);
> +
> +static void __media_jobs_run_jobs(struct work_struct *work)
> +{
> +	struct media_job_scheduler *sched = container_of(work,
> +							 struct media_job_scheduler,
> +							 work);
> +	struct media_job_step *step;
> +	struct media_job *job;
> +
> +	while (true) {
> +		scoped_guard(spinlock, &sched->lock) {
> +			if (list_empty(&sched->queue))
> +				return;
> +
> +			job = list_first_entry(&sched->queue, struct media_job,
> +					       list);
> +		}
> +
> +		list_for_each_entry(step, &job->steps, list)
> +			step->run_step(step->data);
> +
> +		media_jobs_free_job(job, false);
> +	}
> +}
> +
> +void media_jobs_run_jobs(struct media_job_scheduler *sched)
> +{
> +	if (!sched)
> +		return;
> +
> +	queue_work(sched->async_wq, &sched->work);
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_run_jobs);
> +
> +static void __media_jobs_cancel_jobs(struct media_job_scheduler *sched)
> +{
> +	struct media_job *job, *jtmp;
> +
> +	list_for_each_entry_safe(job, jtmp, &sched->pending, list)
> +		media_jobs_free_job(job, true);
> +
> +	list_for_each_entry_safe(job, jtmp, &sched->queue, list)
> +		media_jobs_free_job(job, true);
> +}
> +
> +void media_jobs_cancel_jobs(struct media_job_scheduler *sched)
> +{
> +	if (!sched)
> +		return;
> +
> +	guard(spinlock)(&sched->lock);
> +	__media_jobs_cancel_jobs(sched);
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_cancel_jobs);
> +
> +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
> +				  int (*job_setup)(struct media_job *job, void *data),
> +				  void *data, enum media_job_types type)
> +{
> +	struct media_job_setup_func *new_setup_func;
> +
> +	guard(spinlock)(&sched->lock);

This can be acquired just before touching the list.

> +
> +	new_setup_func = kzalloc(sizeof(*new_setup_func), GFP_KERNEL);
> +	if (!new_setup_func)
> +		return -ENOMEM;
> +
> +	new_setup_func->type = type;
> +	new_setup_func->job_setup = job_setup;
> +	new_setup_func->data = data;
> +	list_add_tail(&new_setup_func->list, &sched->setup_funcs);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_add_job_setup_func);
> +
> +static void __media_jobs_put_scheduler(struct kref *kref)
> +{
> +	struct media_job_scheduler *sched =
> +		container_of(kref, struct media_job_scheduler, kref);
> +	struct media_job_setup_func *func, *ftmp;
> +
> +	cancel_work_sync(&sched->work);
> +	destroy_workqueue(sched->async_wq);
> +
> +	scoped_guard(spinlock, &sched->lock) {
> +		__media_jobs_cancel_jobs(sched);
> +
> +		list_for_each_entry_safe(func, ftmp, &sched->setup_funcs, list) {
> +			list_del(&func->list);
> +			kfree(func);
> +		}
> +	}
> +
> +	list_del(&sched->list);
> +	kfree(sched);
> +}
> +
> +void media_jobs_put_scheduler(struct media_job_scheduler *sched)
> +{
> +	kref_put(&sched->kref, __media_jobs_put_scheduler);
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_put_scheduler);
> +
> +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev)
> +{
> +	struct media_job_scheduler *sched;
> +	char workqueue_name[32];
> +	int ret;
> +
> +	guard(mutex)(&media_job_schedulers_lock);
> +
> +	list_for_each_entry(sched, &media_job_schedulers, list) {
> +		if (sched->mdev == mdev) {
> +			kref_get(&sched->kref);
> +			return sched;
> +		}
> +	}
> +
> +	ret = snprintf(workqueue_name, sizeof(workqueue_name),
> +		       "mc jobs (%s)", mdev->driver_name);
> +	if (!ret)
> +		return ERR_PTR(-EINVAL);
> +
> +	sched = kzalloc(sizeof(*sched), GFP_KERNEL);
> +	if (!sched)
> +		return ERR_PTR(-ENOMEM);
> +
> +	sched->async_wq = alloc_workqueue(workqueue_name, 0, 0);
> +	if (!sched->async_wq) {
> +		kfree(sched);
> +		return ERR_PTR(-EINVAL);
> +	}
> +
> +	sched->mdev = mdev;
> +	kref_init(&sched->kref);
> +	spin_lock_init(&sched->lock);
> +	INIT_LIST_HEAD(&sched->setup_funcs);
> +	INIT_LIST_HEAD(&sched->pending);
> +	INIT_LIST_HEAD(&sched->queue);
> +	INIT_WORK(&sched->work, __media_jobs_run_jobs);
> +
> +	list_add_tail(&sched->list, &media_job_schedulers);
> +
> +	return sched;
> +}
> +EXPORT_SYMBOL_GPL(media_jobs_get_scheduler);
> +
> +LIST_HEAD(media_job_schedulers);
> +
> +/* Synchronise access to the global schedulers list */
> +DEFINE_MUTEX(media_job_schedulers_lock);
> diff --git a/include/media/media-jobs.h b/include/media/media-jobs.h
> new file mode 100644
> index 000000000000..a97270861251
> --- /dev/null
> +++ b/include/media/media-jobs.h
> @@ -0,0 +1,354 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Media jobs framework
> + *
> + * Copyright 2025 Ideas on Board Oy
> + *
> + * Author: Daniel Scally <dan.scally@ideasonboard.com>
> + */
> +
> +#include <linux/kref.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/spinlock.h>
> +#include <linux/types.h>
> +#include <linux/workqueue.h>
> +
> +#ifndef _MEDIA_JOBS_H
> +#define _MEDIA_JOBS_H
> +
> +struct media_device;
> +struct media_entity;
> +struct media_job;
> +struct media_job_dep;
> +
> +/**
> + * define MEDIA_JOBS_FL_STEP_ANYWHERE - \
> + *    Flag a media job step as able to run anytime
> + *
> + * This flag informs the framework that a job step does not need a particular
> + * position in the list of job steps and can be placed anywhere.
> + */
> +#define MEDIA_JOBS_FL_STEP_ANYWHERE			BIT(0)
> +
> +/**
> + * define MEDIA_JOBS_FL_STEP_FROM_FRONT - \
> + *    Flag a media job step as needing to be placed near the start of the list
> + *
> + * This flag informs the framework that a job step needs to be placed at a set
> + * position from the start of the list of job steps.
> + */
> +#define MEDIA_JOBS_FL_STEP_FROM_FRONT			BIT(1)
> +
> +/**
> + * define MEDIA_JOBS_FL_STEP_FROM_BACK - \
> + *    Flag a media job step as needing to be placed near the end of the list
> + *
> + * This flag informs the framework that a job step needs to be placed at a set
> + * position from the end of the list of job steps.
> + */
> +#define MEDIA_JOBS_FL_STEP_FROM_BACK			BIT(2)
> +
> +/**
> + * enum media_job_types - Type of media job
> + *
> + * @MEDIA_JOB_TYPE_PIPELINE_PULSE:	A data event moving through the media
> + *					pipeline
> + *
> + * This enumeration details different types of media jobs. The type can be used
> + * to differentiate between which steps and dependencies a driver needs to add
> + * to a job when it is created.
> + */
> +enum media_job_types {
> +	MEDIA_JOB_TYPE_PIPELINE_PULSE,
> +};
> +
> +/**
> + * struct media_job_scheduler - A job scheduler for a particular media device
> + *
> + * @mdev:		Media device this scheduler is for
> + * @list:		List head to attach to the global list of schedulers
> + * @kref:		Reference counter
> + * @lock:		Lock to protect access to the scheduler
> + * @setup_funcs:	List of &struct media_job_setup_func to populate jobs
> + * @pending:		List of &struct media_jobs created but not yet queued
> + * @queue:		List of &struct media_jobs queued to the scheduler
> + * @work:		Work item to run the jobs
> + * @async_wq:		Workqueue to run the work on
> + *
> + * This struct is the main job scheduler struct - drivers wanting to use this
> + * framework should acquire an instance through media_jobs_get_scheduler() and
> + * subsequently populate it with job setup functions.
> + */
> +struct media_job_scheduler {
> +	struct media_device *mdev;
> +	struct list_head list;
> +	struct kref kref;
> +
> +	spinlock_t lock; /* Synchronise access to the struct's lists */
> +	struct list_head setup_funcs;
> +	struct list_head pending;
> +	struct list_head queue;
> +	struct work_struct work;
> +	struct workqueue_struct *async_wq;
> +};
> +
> +/**
> + * struct media_job_setup_func - A function to populate a media job with steps
> + *				 and dependencies
> + *
> + * @list:	The list object to attach to the scheduler
> + * @type:	The &enum media_job_types that this function populates a job for
> + * @job_setup:	Function pointer to the driver's job setup function
> + * @data:	Pointer to the driver data for use with @job_setup
> + *
> + * This struct holds data about the functions a driver registers with the jobs
> + * framework in order to populate a new job with steps and dependencies.
> + */
> +struct media_job_setup_func {
> +	struct list_head list;
> +	enum media_job_types type;
> +	int (*job_setup)(struct media_job *job, void *data);
> +	void *data;
> +};
> +
> +/**
> + * struct media_job - A representation of a job to be run through the pipeline
> + *
> + * @lock:	Lock to protect access to the job's lists
> + * @list:	List head to attach the job to &struct media_job_scheduler in
> + *		either the pending or queue lists
> + * @steps:	List of &struct media_job_step to run the job
> + * @deps:	List of &struct media_job_dep to check that the job can be
> + *		queued
> + * @sched:	Pointer to the media job scheduler
> + * @type:	The type of the job
> + *
> + * This struct holds lists of steps that need to be performed to carry out a
> + * job in the pipeline. A separate list of dependencies allows the queueing of
> + * the job to be delayed until all drivers are ready to carry it out.
> + */
> +struct media_job {
> +	spinlock_t lock; /* Synchronise access to the struct's lists 6*/
> +	struct list_head list;
> +	struct list_head steps;
> +	struct list_head deps;
> +	struct media_job_scheduler *sched;
> +	enum media_job_types type;
> +};
> +
> +/**
> + * struct media_job_step - A holder for a function to run as part of a job
> + *
> + * @list:	List head to attach the job step to a &struct media_job.steps
> + * @run_step:	The function to run to perform the step
> + * @data:	Data to pass to the .run_step() function
> + * @flags:	Flags to control how the step is ordered within the job's list
> + *		of steps
> + * @pos:	Position indicator to control how the step is ordered within the
> + *		job's list of steps
> + *
> + * This struct defines a function that needs to be run as part of the execution
> + * of a job in a media pipeline, along with information that help the scheduler
> + * determine what order it should be ran in in reference to the other steps that
> + * are part of the same job.
> + */
> +struct media_job_step {
> +	struct list_head list;
> +	void (*run_step)(void *data);
> +	void *data;
> +	unsigned int flags;
> +	unsigned int pos;
> +};
> +
> +/**
> + * struct media_job_dep_ops - Operations to manage a media job dependency
> + *
> + * @check_dep:	A function to ask the driver whether the dependency is met
> + * @clear_dep:	A function to tell the driver that the job has been queued
> + * @reset_dep:	A function to tell the driver that the job has been cancelled
> + *
> + * Media jobs have dependencies, such as requiring buffers to be queued. These
> + * operations allow a driver to define how the media jobs framework should check
> + * whether or not those dependencies are met and how it should inform them that
> + * it is taking action based on the state of those dependencies.
> + */
> +struct media_job_dep_ops {
> +	bool (*check_dep)(void *data);
> +	void (*clear_dep)(void *data);
> +	void (*reset_dep)(void *data);
> +};
> +
> +/**
> + * struct media_job_dep - Representation of media job dependency
> + *
> + * @list:	List head to attach to a &struct media_job.deps
> + * @ops:	A pointer to the dependency's operations functions
> + * @met:	A flag to record whether or not the dependency is met
> + * @data:	Data to pass to the dependency's operations
> + *
> + * This struct represents a dependency of a media job. The operations member
> + * holds pointers to functions allowing the framework to interact with the
> + * driver to check whether or not the dependency is met.
> + */
> +struct media_job_dep {
> +	struct list_head list;
> +	struct media_job_dep_ops *ops;
> +	bool met;
> +	void *data;
> +};
> +
> +/**
> + * media_jobs_try_queue_job - Try to queue a &struct media_job
> + *
> + * @sched:	Pointer to the job scheduler
> + * @type:	The type of the media job
> + * @dep_ops:	A pointer to the dependency operations for this job
> + * @dep_data:	A pointer to the dependency data for this job
> + *
> + * Try to queue a media job with the scheduler. This function should be called
> + * by the drivers whenever a dependency for a media job is met - for example
> + * when a buffer is queued to the driver. The framework will check to see if an
> + * existing job on the scheduler's pending list shares the same type, dependency
> + * operations and dependency data. If it does then that existing job will be
> + * considered. If there is no extant job with those same parameters, a new job
> + * is allocated and populated by calling the setup functions registered with
> + * the framework.
> + *
> + * The function iterates over the dependencies that are registered with the job
> + * and checks to see if they are met. If they're all met, they're cleared and
> + * the job is placed onto the scheduler's queue.
> + *
> + * To help reduce conditionals in drivers where a driver supports both the use
> + * of the media jobs framework and operation without it, this function is a no
> + * op if @sched is NULL.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_try_queue_job(struct media_job_scheduler *sched,
> +			     enum media_job_types type,
> +			     struct media_job_dep_ops *dep_ops, void *dep_data);
> +
> +/**
> + * media_jobs_add_job_step - Add a step to a media job
> + *
> + * @job:	Pointer to the &struct media_job
> + * @run_step:	Pointer to the function to run to execute the step
> + * @data:	Pointer to the data to pass to @run_ste
> + * @flags:	One of the MEDIA_JOBS_FL_STEP_* flags
> + * @pos:	A position indicator to use with @flags
> + *
> + * This function adds a step to the job and should be called from the drivers'
> + * job setup functions as registered with the framework through
> + * media_jobs_add_job_setup_func(). The @flags and @pos parameters are used
> + * to determine the ordering of the steps within the job:
> + *
> + * If @flags has the MEDIA_JOBS_FL_STEP_ANYWHERE bit set, the step is placed
> + * after all steps with MEDIA_JOBS_FL_STEP_FROM_FRONT and before all steps with
> + * MEDIA_JOBS_FL_STEP_FROM_BACK bit set, but otherwise in whatever order this
> + * function is called.
> + *
> + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_FRONT bit set then the step is
> + * placed @pos steps from the front of the list. Attempting to place multiple
> + * steps in the same position will result in an error.
> + *
> + * If @flags has the MEDIA_JOBS_FL_STEP_FROM_BACK bit set then the step is
> + * placed @pos steps from the back of the list. Attempting to place multiple
> + * steps in the same position will result in an error.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_add_job_step(struct media_job *job, void (*run_step)(void *data),
> +			    void *data, unsigned int flags, unsigned int pos);
> +
> +/**
> + * media_jobs_add_job_dep - Add a dependency to a media job
> + *
> + * @job:	Pointer to the &struct media_job
> + * @ops:	Pointer to the &struct media_job_dep_ops
> + * @data:	Pointer to the data to pass to the dependency's operations
> + *
> + * This function adds a dependency to the job and should be called from the
> + * drivers job setup functions as registered with the framework through the
> + * media_jobs_add_job_setup_func() function.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_add_job_dep(struct media_job *job, struct media_job_dep_ops *ops,
> +			   void *data);
> +
> +/**
> + * media_jobs_add_job_setup_func - Add a function that populates a media job
> + *
> + * @sched:	Pointer to the media jobs scheduler
> + * @job_setup:	Pointer to the new job setup function
> + * @data:	Data to pass to the job setup function
> + * @type:	The type of job that this function should be called for
> + *
> + * Drivers that wish to utilise the framework need to use this function to
> + * register a callback that adds job steps and dependencies when one is created.
> + * The function must call media_jobs_add_job_step() and media_jobs_add_job_dep()
> + * to populate the job.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +int media_jobs_add_job_setup_func(struct media_job_scheduler *sched,
> +				  int (*job_setup)(struct media_job *job, void *data),
> +				  void *data, enum media_job_types type);
> +
> +/**
> + * media_jobs_put_scheduler - Put a reference to the media jobs scheduler
> + *
> + * @sched:	Pointer to the media jobs scheduler
> + *
> + * This function puts a reference to the media jobs scheduler, and is intended
> + * to be called in error and exit paths for consuming drivers
> + */
> +void media_jobs_put_scheduler(struct media_job_scheduler *sched);
> +
> +/**
> + * media_jobs_get_scheduler - Get a media jobs scheduler
> + *
> + * @mdev:	Pointer to the media device associated with the scheduler
> + *
> + * This function gets a pointer to a &struct media_job_scheduler associated with
> + * the media device passed to @mdev. If one is not available then it is
> + * allocated and returned. This allows multiple drivers sharing a media graph to
> + * work with the same media job scheduler.
> + *
> + * Return: 0 on success or a negative error number
> + */
> +struct media_job_scheduler *media_jobs_get_scheduler(struct media_device *mdev);
> +
> +/**
> + * media_jobs_run_jobs - Run any media jobs that are ready in the queue
> + *
> + * @sched:	Pointer to the media job scheduler
> + *
> + * This function triggers the workqueue that processes any jobs that have been
> + * queued, and should be called whenever the pipeline is ready to do so.
> + *
> + * To help reduce conditionals in drivers where a driver supports both the use
> + * of the media jobs framework and operation without it, this function is a no
> + * op if @sched is NULL.
> + */
> +void media_jobs_run_jobs(struct media_job_scheduler *sched);
> +
> +/**
> + * media_jobs_cancel_jobs - cancel all waiting jobs
> + *
> + * @sched:	Pointer to the media job scheduler
> + *
> + * This function iterates over any pending and queued jobs, resets their
> + * dependencies and frees the job
> + *
> + * To help reduce conditionals in drivers where a driver supports both the use
> + * of the media jobs framework and operation without it, this function is a no
> + * op if @sched is NULL.
> + */
> +void media_jobs_cancel_jobs(struct media_job_scheduler *sched);
> +
> +extern struct list_head media_job_schedulers;
> +extern struct mutex media_job_schedulers_lock;
> +
> +#endif /* _MEDIA_JOBS_H */

-- 
Regards,

Sakari Ailus

Re: [PATCH 0/3] Add media jobs framework

[Nicolas Dufresne]

Hi Daniel,

Le lundi 19 mai 2025 à 15:04 +0100, Daniel Scally a écrit :
> Hello all
> 
> This series adds a new API to the media controller framework, which
> I'm calling "media jobs". The framework is intended to facilitate
> communication between separate drivers which need to work together
> to fully operate a media pipeline. For example, the need for the
> framework arose when writing support for the ISP in the RZ/V2H; the
> ISP is fed by a DMA engine which is not part of the same IP block,
> and so is driven by its own driver (though sharing a media graph).
> The ISP driver needs to be able to communicate with the DMA engine
> driver to instruct it to push the next frame. Because the DMA engine
> might be different on a different platform that used the ISP, direct
> calls into functions exported by the DMA engine driver wouldn't be
> scalable, and so this driver agnostic route was adopted. The
> framework allows drivers to define the steps that need to be taken
> (for example writing configuration data, reading statistics data,
> writing buffer addresses and triggering data transmission) to complete
> a "job" (of which the only current example is the processing of a
> frame of data through the pipeline, though I expect that other use
> cases could become apparent too) and to then schedule them into a
> work queue once driver definable dependencies have been met. The
> dependencies might, for example, be the queuing of buffers to V4L2
> capture / output devices.

This is just a first impression, about the naming of course. If you
haven't been there, Request was once called Job, so bringing back
jobs next to request looks like it going to be very confusing.

When I look at your proposal, what I see here is a scheduler, the
jobs being the abstraction of the task being scheduled. What about
going forward with "Media Scheduler" ?

I can't stop finding similarities with the DRM scheduler. I'm not expert,
still learning, but it seems DRM agnostic and possible usable even by us.
It does support dependencies, but it also support priorities. Instead of
ops for deps, uses fences. The fences are not necessarily userspace facing,
they do allow having jobs that depends on foreign job without any sub-system
boundaries. They properly abstract the dependencies too.

I would like to see your impression about this. I would also see a reflection
around non-camera use cases, since MC is no longer just a camera API.

regards,
Nicolas

> 
> The framework allows precise definition of the ordering of the steps
> regardless of the order in which the drivers populate the jobs. Steps
> can be flagged as being placed at a particular position from the front
> or back of the queue (I.E. last, or third from last) or as requiring
> no particular order. This would allow the construction of a job like:
> 
> Step 0 (ISP Driver): Program the hardware with parameters
> Step 1 to N-1 (Both drivers): Program the hardware with buffers
> Step N (DMA Engine Driver): Send a frame of data to the ISP
> ... ISP processes data ...
> Step N + 1 (ISP Driver): Read out statistics data from the last frame
> 
> The mali-c55 ISP driver and the DMA engine feeding it on the RZ/V2H
> (called the rzv2h-ivc driver) both use the framework, and will be
> posted shortly after this series with references back to it. I will
> reply to this message with links to those series for convenience.
> 
> The first patch in this set is not strictly part of the framework,
> but also facilitates multiple drivers with V4L2 Video Devices
> sharing a single media graph. We have a requirement to delay the
> start of streaming until all of a pipeline's video devices have had
> their .start_streaming() operations called; these new entity ops
> provide a mechanism through which each driver can inform the other
> that the last video device in the pipeline has now been started.
> 
> Thanks
> Dan
> 
> Daniel Scally (3):
>   media: mc: entity: Add pipeline_started/stopped ops
>   media: mc: Add media jobs framework
>   media: Documentation: Add documentation for media jobs
> 
>  Documentation/driver-api/media/mc-core.rst | 154 +++++++
>  drivers/media/mc/Makefile                  |   2 +-
>  drivers/media/mc/mc-entity.c               |  45 +++
>  drivers/media/mc/mc-jobs.c                 | 446 +++++++++++++++++++++
>  include/media/media-entity.h               |  24 ++
>  include/media/media-jobs.h                 | 354 ++++++++++++++++
>  6 files changed, 1024 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/media/mc/mc-jobs.c
>  create mode 100644 include/media/media-jobs.h

Re: [PATCH 1/3] media: mc: entity: Add pipeline_started/stopped ops

[Dan Scally]

Hi Jacopo

On 21/05/2025 16:18, Jacopo Mondi wrote:
> Hi Dan
>
> On Mon, May 19, 2025 at 03:04:01PM +0100, Daniel Scally wrote:
>> Add two new members to struct media_entity_operations, along with new
>> functions in media-entity.c to traverse a media pipeline and call the
>> new operations. The new functions are intended to be used to signal
>> to a media pipeline that it has fully started, with the entity ops
>> allowing drivers to define some action to be taken when those
>> conditions are met.
>>
>> The combination of the new functions and operations allows drivers
>> which are part of a multi-driver pipeline to delay actually starting
>> streaming until all of the conditions for streaming succcessfully are
>> met across all drivers.
> Maybe s/succcessfully are/are successfully/
>
> Or was this (the three 'c' apart) intentional ?
>
>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>> ---
>>   drivers/media/mc/mc-entity.c | 45 ++++++++++++++++++++++++++++++++++++
>>   include/media/media-entity.h | 24 +++++++++++++++++++
>>   2 files changed, 69 insertions(+)
>>
>> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
>> index 045590905582..e36b1710669d 100644
>> --- a/drivers/media/mc/mc-entity.c
>> +++ b/drivers/media/mc/mc-entity.c
>> @@ -1053,6 +1053,51 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
>>   }
>>   EXPORT_SYMBOL_GPL(__media_pipeline_entity_iter_next);
>>
>> +int media_pipeline_started(struct media_pipeline *pipe)
>> +{
>> +	struct media_pipeline_entity_iter iter;
>> +	struct media_entity *entity;
>> +	int ret;
>> +
>> +	ret = media_pipeline_entity_iter_init(pipe, &iter);
>> +	if (ret)
>> +		return ret;
>> +
>> +	media_pipeline_for_each_entity(pipe, &iter, entity) {
>> +		ret = media_entity_call(entity, pipeline_started);
>> +		if (ret && ret != -ENOIOCTLCMD)
>> +			goto err_notify_stopped;
>> +	}
>> +
>> +	media_pipeline_entity_iter_cleanup(&iter);
>> +
>> +	return ret == -ENOIOCTLCMD ? 0 : ret;
> Shouldn't you just return 0 ? If a ret < 0 is encoutered in the loop
> we just to the below label


Actually thinking more I think this is right as-is. The intention is to avoid returning -ENOIOCTLCMD 
from this function if the last entity in the iterator doesn't have a .pipeline_started operation.

>
>> +
>> +err_notify_stopped:
>> +	media_pipeline_stopped(pipe);
> Do you need to media_pipeline_entity_iter_cleanup() ?
>
>> +	return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(media_pipeline_started);
>> +
>> +int media_pipeline_stopped(struct media_pipeline *pipe)
>> +{
>> +	struct media_pipeline_entity_iter iter;
>> +	struct media_entity *entity;
>> +	int ret;
>> +
>> +	ret = media_pipeline_entity_iter_init(pipe, &iter);
>> +	if (ret)
>> +		return ret;
>> +
>> +	media_pipeline_for_each_entity(pipe, &iter, entity)
>> +		media_entity_call(entity, pipeline_stopped);
>> +
>> +	media_pipeline_entity_iter_cleanup(&iter);
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(media_pipeline_stopped);
>> +
>>   /* -----------------------------------------------------------------------------
>>    * Links management
>>    */
>> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
>> index 64cf590b1134..e858326b95cb 100644
>> --- a/include/media/media-entity.h
>> +++ b/include/media/media-entity.h
>> @@ -269,6 +269,10 @@ struct media_pad {
>>    *			media_entity_has_pad_interdep().
>>    *			Optional: If the operation isn't implemented all pads
>>    *			will be considered as interdependent.
>> + * @pipeline_started:	Notify this entity that the pipeline it is a part of has
>> + *			been started
>> + * @pipeline_stopped:	Notify this entity that the pipeline it is a part of has
>> + *			been stopped
>>    *
>>    * .. note::
>>    *
>> @@ -284,6 +288,8 @@ struct media_entity_operations {
>>   	int (*link_validate)(struct media_link *link);
>>   	bool (*has_pad_interdep)(struct media_entity *entity, unsigned int pad0,
>>   				 unsigned int pad1);
>> +	int (*pipeline_started)(struct media_entity *entity);
>> +	void (*pipeline_stopped)(struct media_entity *entity);
>>   };
>>
>>   /**
>> @@ -1261,6 +1267,24 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
>>   	     entity != NULL;							\
>>   	     entity = __media_pipeline_entity_iter_next((pipe), iter, entity))
>>
>> +/**
>> + * media_pipeline_started - Inform entities in a pipeline that it has started
>> + * @pipe:	The pipeline
>> + *
>> + * Iterate on all entities in a media pipeline and call their pipeline_started
>> + * member of media_entity_operations.
>> + */
>> +int media_pipeline_started(struct media_pipeline *pipe);
>> +
>> +/**
>> + * media_pipeline_stopped - Inform entities in a pipeline that it has stopped
>> + * @pipe:	The pipeline
>> + *
>> + * Iterate on all entities in a media pipeline and call their pipeline_stopped
>> + * member of media_entity_operations.
>> + */
>> +int media_pipeline_stopped(struct media_pipeline *pipe);
>> +
> All good, but I don't see these operations being used at all in this
> series ?
>
>>   /**
>>    * media_pipeline_alloc_start - Mark a pipeline as streaming
>>    * @pad: Starting pad
>> --
>> 2.34.1
>>
>>

Re: [PATCH 1/3] media: mc: entity: Add pipeline_started/stopped ops

[Jacopo Mondi]

Hi Dan
  sorry missed the last reply

On Mon, Jun 09, 2025 at 03:57:55PM +0100, Dan Scally wrote:
> Hi Jacopo
>
> On 21/05/2025 16:18, Jacopo Mondi wrote:
> > Hi Dan
> >
> > On Mon, May 19, 2025 at 03:04:01PM +0100, Daniel Scally wrote:
> > > Add two new members to struct media_entity_operations, along with new
> > > functions in media-entity.c to traverse a media pipeline and call the
> > > new operations. The new functions are intended to be used to signal
> > > to a media pipeline that it has fully started, with the entity ops
> > > allowing drivers to define some action to be taken when those
> > > conditions are met.
> > >
> > > The combination of the new functions and operations allows drivers
> > > which are part of a multi-driver pipeline to delay actually starting
> > > streaming until all of the conditions for streaming succcessfully are
> > > met across all drivers.
> > Maybe s/succcessfully are/are successfully/
> >
> > Or was this (the three 'c' apart) intentional ?
> >
> > > Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> > > ---
> > >   drivers/media/mc/mc-entity.c | 45 ++++++++++++++++++++++++++++++++++++
> > >   include/media/media-entity.h | 24 +++++++++++++++++++
> > >   2 files changed, 69 insertions(+)
> > >
> > > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > > index 045590905582..e36b1710669d 100644
> > > --- a/drivers/media/mc/mc-entity.c
> > > +++ b/drivers/media/mc/mc-entity.c
> > > @@ -1053,6 +1053,51 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
> > >   }
> > >   EXPORT_SYMBOL_GPL(__media_pipeline_entity_iter_next);
> > >
> > > +int media_pipeline_started(struct media_pipeline *pipe)
> > > +{
> > > +	struct media_pipeline_entity_iter iter;
> > > +	struct media_entity *entity;
> > > +	int ret;
> > > +
> > > +	ret = media_pipeline_entity_iter_init(pipe, &iter);
> > > +	if (ret)
> > > +		return ret;
> > > +
> > > +	media_pipeline_for_each_entity(pipe, &iter, entity) {
> > > +		ret = media_entity_call(entity, pipeline_started);
> > > +		if (ret && ret != -ENOIOCTLCMD)
> > > +			goto err_notify_stopped;
> > > +	}
> > > +
> > > +	media_pipeline_entity_iter_cleanup(&iter);
> > > +
> > > +	return ret == -ENOIOCTLCMD ? 0 : ret;
> > Shouldn't you just return 0 ? If a ret < 0 is encoutered in the loop
> > we just to the below label
>
>
> Actually thinking more I think this is right as-is. The intention is to
> avoid returning -ENOIOCTLCMD from this function if the last entity in the
> iterator doesn't have a .pipeline_started operation.

That was my point actually.

If you encounter a ret != -ENOIOCTLCMD in the above for loop you will
jump to the label below.

If you get here ret is either 0 or -ENOIOCTLCMD, and you can just
return 0. Or do I still have to actually wake up ?

>
> >
> > > +
> > > +err_notify_stopped:
> > > +	media_pipeline_stopped(pipe);
> > Do you need to media_pipeline_entity_iter_cleanup() ?
> >
> > > +	return ret;
> > > +}
> > > +EXPORT_SYMBOL_GPL(media_pipeline_started);
> > > +
> > > +int media_pipeline_stopped(struct media_pipeline *pipe)
> > > +{
> > > +	struct media_pipeline_entity_iter iter;
> > > +	struct media_entity *entity;
> > > +	int ret;
> > > +
> > > +	ret = media_pipeline_entity_iter_init(pipe, &iter);
> > > +	if (ret)
> > > +		return ret;
> > > +
> > > +	media_pipeline_for_each_entity(pipe, &iter, entity)
> > > +		media_entity_call(entity, pipeline_stopped);
> > > +
> > > +	media_pipeline_entity_iter_cleanup(&iter);
> > > +
> > > +	return 0;
> > > +}
> > > +EXPORT_SYMBOL_GPL(media_pipeline_stopped);
> > > +
> > >   /* -----------------------------------------------------------------------------
> > >    * Links management
> > >    */
> > > diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> > > index 64cf590b1134..e858326b95cb 100644
> > > --- a/include/media/media-entity.h
> > > +++ b/include/media/media-entity.h
> > > @@ -269,6 +269,10 @@ struct media_pad {
> > >    *			media_entity_has_pad_interdep().
> > >    *			Optional: If the operation isn't implemented all pads
> > >    *			will be considered as interdependent.
> > > + * @pipeline_started:	Notify this entity that the pipeline it is a part of has
> > > + *			been started
> > > + * @pipeline_stopped:	Notify this entity that the pipeline it is a part of has
> > > + *			been stopped
> > >    *
> > >    * .. note::
> > >    *
> > > @@ -284,6 +288,8 @@ struct media_entity_operations {
> > >   	int (*link_validate)(struct media_link *link);
> > >   	bool (*has_pad_interdep)(struct media_entity *entity, unsigned int pad0,
> > >   				 unsigned int pad1);
> > > +	int (*pipeline_started)(struct media_entity *entity);
> > > +	void (*pipeline_stopped)(struct media_entity *entity);
> > >   };
> > >
> > >   /**
> > > @@ -1261,6 +1267,24 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
> > >   	     entity != NULL;							\
> > >   	     entity = __media_pipeline_entity_iter_next((pipe), iter, entity))
> > >
> > > +/**
> > > + * media_pipeline_started - Inform entities in a pipeline that it has started
> > > + * @pipe:	The pipeline
> > > + *
> > > + * Iterate on all entities in a media pipeline and call their pipeline_started
> > > + * member of media_entity_operations.
> > > + */
> > > +int media_pipeline_started(struct media_pipeline *pipe);
> > > +
> > > +/**
> > > + * media_pipeline_stopped - Inform entities in a pipeline that it has stopped
> > > + * @pipe:	The pipeline
> > > + *
> > > + * Iterate on all entities in a media pipeline and call their pipeline_stopped
> > > + * member of media_entity_operations.
> > > + */
> > > +int media_pipeline_stopped(struct media_pipeline *pipe);
> > > +
> > All good, but I don't see these operations being used at all in this
> > series ?
> >
> > >   /**
> > >    * media_pipeline_alloc_start - Mark a pipeline as streaming
> > >    * @pad: Starting pad
> > > --
> > > 2.34.1
> > >
> > >
>

Re: [PATCH 1/3] media: mc: entity: Add pipeline_started/stopped ops

[Dan Scally]

Hi Jacopo

On 12/06/2025 09:50, Jacopo Mondi wrote:
> Hi Dan
>    sorry missed the last reply
>
> On Mon, Jun 09, 2025 at 03:57:55PM +0100, Dan Scally wrote:
>> Hi Jacopo
>>
>> On 21/05/2025 16:18, Jacopo Mondi wrote:
>>> Hi Dan
>>>
>>> On Mon, May 19, 2025 at 03:04:01PM +0100, Daniel Scally wrote:
>>>> Add two new members to struct media_entity_operations, along with new
>>>> functions in media-entity.c to traverse a media pipeline and call the
>>>> new operations. The new functions are intended to be used to signal
>>>> to a media pipeline that it has fully started, with the entity ops
>>>> allowing drivers to define some action to be taken when those
>>>> conditions are met.
>>>>
>>>> The combination of the new functions and operations allows drivers
>>>> which are part of a multi-driver pipeline to delay actually starting
>>>> streaming until all of the conditions for streaming succcessfully are
>>>> met across all drivers.
>>> Maybe s/succcessfully are/are successfully/
>>>
>>> Or was this (the three 'c' apart) intentional ?
>>>
>>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>>>> ---
>>>>    drivers/media/mc/mc-entity.c | 45 ++++++++++++++++++++++++++++++++++++
>>>>    include/media/media-entity.h | 24 +++++++++++++++++++
>>>>    2 files changed, 69 insertions(+)
>>>>
>>>> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
>>>> index 045590905582..e36b1710669d 100644
>>>> --- a/drivers/media/mc/mc-entity.c
>>>> +++ b/drivers/media/mc/mc-entity.c
>>>> @@ -1053,6 +1053,51 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
>>>>    }
>>>>    EXPORT_SYMBOL_GPL(__media_pipeline_entity_iter_next);
>>>>
>>>> +int media_pipeline_started(struct media_pipeline *pipe)
>>>> +{
>>>> +	struct media_pipeline_entity_iter iter;
>>>> +	struct media_entity *entity;
>>>> +	int ret;
>>>> +
>>>> +	ret = media_pipeline_entity_iter_init(pipe, &iter);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	media_pipeline_for_each_entity(pipe, &iter, entity) {
>>>> +		ret = media_entity_call(entity, pipeline_started);
>>>> +		if (ret && ret != -ENOIOCTLCMD)
>>>> +			goto err_notify_stopped;
>>>> +	}
>>>> +
>>>> +	media_pipeline_entity_iter_cleanup(&iter);
>>>> +
>>>> +	return ret == -ENOIOCTLCMD ? 0 : ret;
>>> Shouldn't you just return 0 ? If a ret < 0 is encoutered in the loop
>>> we just to the below label
>>
>> Actually thinking more I think this is right as-is. The intention is to
>> avoid returning -ENOIOCTLCMD from this function if the last entity in the
>> iterator doesn't have a .pipeline_started operation.
> That was my point actually.
>
> If you encounter a ret != -ENOIOCTLCMD in the above for loop you will
> jump to the label below.
>
> If you get here ret is either 0 or -ENOIOCTLCMD, and you can just
> return 0. Or do I still have to actually wake up ?


Oh I see! Yes sorry, you are right...although in the end I refactored this function a bit so that it 
doesn't have the goto to avoid having to put media_pipeline_entity_iter_cleanup() in there twice so 
it becomes a bit moot.

>
>>>> +
>>>> +err_notify_stopped:
>>>> +	media_pipeline_stopped(pipe);
>>> Do you need to media_pipeline_entity_iter_cleanup() ?
>>>
>>>> +	return ret;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(media_pipeline_started);
>>>> +
>>>> +int media_pipeline_stopped(struct media_pipeline *pipe)
>>>> +{
>>>> +	struct media_pipeline_entity_iter iter;
>>>> +	struct media_entity *entity;
>>>> +	int ret;
>>>> +
>>>> +	ret = media_pipeline_entity_iter_init(pipe, &iter);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	media_pipeline_for_each_entity(pipe, &iter, entity)
>>>> +		media_entity_call(entity, pipeline_stopped);
>>>> +
>>>> +	media_pipeline_entity_iter_cleanup(&iter);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(media_pipeline_stopped);
>>>> +
>>>>    /* -----------------------------------------------------------------------------
>>>>     * Links management
>>>>     */
>>>> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
>>>> index 64cf590b1134..e858326b95cb 100644
>>>> --- a/include/media/media-entity.h
>>>> +++ b/include/media/media-entity.h
>>>> @@ -269,6 +269,10 @@ struct media_pad {
>>>>     *			media_entity_has_pad_interdep().
>>>>     *			Optional: If the operation isn't implemented all pads
>>>>     *			will be considered as interdependent.
>>>> + * @pipeline_started:	Notify this entity that the pipeline it is a part of has
>>>> + *			been started
>>>> + * @pipeline_stopped:	Notify this entity that the pipeline it is a part of has
>>>> + *			been stopped
>>>>     *
>>>>     * .. note::
>>>>     *
>>>> @@ -284,6 +288,8 @@ struct media_entity_operations {
>>>>    	int (*link_validate)(struct media_link *link);
>>>>    	bool (*has_pad_interdep)(struct media_entity *entity, unsigned int pad0,
>>>>    				 unsigned int pad1);
>>>> +	int (*pipeline_started)(struct media_entity *entity);
>>>> +	void (*pipeline_stopped)(struct media_entity *entity);
>>>>    };
>>>>
>>>>    /**
>>>> @@ -1261,6 +1267,24 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe,
>>>>    	     entity != NULL;							\
>>>>    	     entity = __media_pipeline_entity_iter_next((pipe), iter, entity))
>>>>
>>>> +/**
>>>> + * media_pipeline_started - Inform entities in a pipeline that it has started
>>>> + * @pipe:	The pipeline
>>>> + *
>>>> + * Iterate on all entities in a media pipeline and call their pipeline_started
>>>> + * member of media_entity_operations.
>>>> + */
>>>> +int media_pipeline_started(struct media_pipeline *pipe);
>>>> +
>>>> +/**
>>>> + * media_pipeline_stopped - Inform entities in a pipeline that it has stopped
>>>> + * @pipe:	The pipeline
>>>> + *
>>>> + * Iterate on all entities in a media pipeline and call their pipeline_stopped
>>>> + * member of media_entity_operations.
>>>> + */
>>>> +int media_pipeline_stopped(struct media_pipeline *pipe);
>>>> +
>>> All good, but I don't see these operations being used at all in this
>>> series ?
>>>
>>>>    /**
>>>>     * media_pipeline_alloc_start - Mark a pipeline as streaming
>>>>     * @pad: Starting pad
>>>> --
>>>> 2.34.1
>>>>
>>>>

Re: [PATCH 0/3] Add media jobs framework

[Dan Scally]

Hi Nicolas - sorry for the delayed response, I missed this message somehow.

On 30/05/2025 16:11, Nicolas Dufresne wrote:
> Hi Daniel,
>
> Le lundi 19 mai 2025 à 15:04 +0100, Daniel Scally a écrit :
>> Hello all
>>
>> This series adds a new API to the media controller framework, which
>> I'm calling "media jobs". The framework is intended to facilitate
>> communication between separate drivers which need to work together
>> to fully operate a media pipeline. For example, the need for the
>> framework arose when writing support for the ISP in the RZ/V2H; the
>> ISP is fed by a DMA engine which is not part of the same IP block,
>> and so is driven by its own driver (though sharing a media graph).
>> The ISP driver needs to be able to communicate with the DMA engine
>> driver to instruct it to push the next frame. Because the DMA engine
>> might be different on a different platform that used the ISP, direct
>> calls into functions exported by the DMA engine driver wouldn't be
>> scalable, and so this driver agnostic route was adopted. The
>> framework allows drivers to define the steps that need to be taken
>> (for example writing configuration data, reading statistics data,
>> writing buffer addresses and triggering data transmission) to complete
>> a "job" (of which the only current example is the processing of a
>> frame of data through the pipeline, though I expect that other use
>> cases could become apparent too) and to then schedule them into a
>> work queue once driver definable dependencies have been met. The
>> dependencies might, for example, be the queuing of buffers to V4L2
>> capture / output devices.
> This is just a first impression, about the naming of course. If you
> haven't been there, Request was once called Job, so bringing back
> jobs next to request looks like it going to be very confusing.
>
> When I look at your proposal, what I see here is a scheduler, the
> jobs being the abstraction of the task being scheduled. What about
> going forward with "Media Scheduler" ?
Ah; Requests being Jobs was before my time so that potential for confusion didn't twig to me....I'm 
happy to rename it, though I think just "Media Scheduler" begs the question "what is being 
scheduled?" so I feel like something more is needed...but I don't know what else.
> I can't stop finding similarities with the DRM scheduler. I'm not expert,
> still learning, but it seems DRM agnostic and possible usable even by us.
> It does support dependencies, but it also support priorities. Instead of
> ops for deps, uses fences. The fences are not necessarily userspace facing,
> they do allow having jobs that depends on foreign job without any sub-system
> boundaries. They properly abstract the dependencies too.
>
> I would like to see your impression about this. I would also see a reflection
> around non-camera use cases, since MC is no longer just a camera API.

Having had a look I agree that there's a lot of parallels. I don't think it would be usable as-is 
unless I'm missing something, but perhaps it could be adapted to be usable instead. I've revised 
this series so I'll post the v2 in a moment, but I'll delve more into the DRM scheduler and see 
where that leads. Thanks for bringing it up!


Thanks

Dan

>
> regards,
> Nicolas
>
>> The framework allows precise definition of the ordering of the steps
>> regardless of the order in which the drivers populate the jobs. Steps
>> can be flagged as being placed at a particular position from the front
>> or back of the queue (I.E. last, or third from last) or as requiring
>> no particular order. This would allow the construction of a job like:
>>
>> Step 0 (ISP Driver): Program the hardware with parameters
>> Step 1 to N-1 (Both drivers): Program the hardware with buffers
>> Step N (DMA Engine Driver): Send a frame of data to the ISP
>> ... ISP processes data ...
>> Step N + 1 (ISP Driver): Read out statistics data from the last frame
>>
>> The mali-c55 ISP driver and the DMA engine feeding it on the RZ/V2H
>> (called the rzv2h-ivc driver) both use the framework, and will be
>> posted shortly after this series with references back to it. I will
>> reply to this message with links to those series for convenience.
>>
>> The first patch in this set is not strictly part of the framework,
>> but also facilitates multiple drivers with V4L2 Video Devices
>> sharing a single media graph. We have a requirement to delay the
>> start of streaming until all of a pipeline's video devices have had
>> their .start_streaming() operations called; these new entity ops
>> provide a mechanism through which each driver can inform the other
>> that the last video device in the pipeline has now been started.
>>
>> Thanks
>> Dan
>>
>> Daniel Scally (3):
>>    media: mc: entity: Add pipeline_started/stopped ops
>>    media: mc: Add media jobs framework
>>    media: Documentation: Add documentation for media jobs
>>
>>   Documentation/driver-api/media/mc-core.rst | 154 +++++++
>>   drivers/media/mc/Makefile                  |   2 +-
>>   drivers/media/mc/mc-entity.c               |  45 +++
>>   drivers/media/mc/mc-jobs.c                 | 446 +++++++++++++++++++++
>>   include/media/media-entity.h               |  24 ++
>>   include/media/media-jobs.h                 | 354 ++++++++++++++++
>>   6 files changed, 1024 insertions(+), 1 deletion(-)
>>   create mode 100644 drivers/media/mc/mc-jobs.c
>>   create mode 100644 include/media/media-jobs.h