[PATCH v3 1/4] media: v4l: dev-decoder: Add source change V4L2_EVENT_SRC_CH_COLORSPACE

[PATCH v3 1/4] media: v4l: dev-decoder: Add source change V4L2_EVENT_SRC_CH_COLORSPACE

[ming.qian]

From: Ming Qian <ming.qian@oss.nxp.com>

Add a new source change V4L2_EVENT_SRC_CH_COLORSPACE that
indicates colorspace change in the stream.
The change V4L2_EVENT_SRC_CH_RESOLUTION will always affect
the allocation, but V4L2_EVENT_SRC_CH_COLORSPACE won't.

Signed-off-by: Ming Qian <ming.qian@oss.nxp.com>
---
v3
- Improve the description according to comments

 .../userspace-api/media/v4l/vidioc-dqevent.rst       | 12 ++++++++++++
 .../userspace-api/media/videodev2.h.rst.exceptions   |  1 +
 include/uapi/linux/videodev2.h                       |  1 +
 3 files changed, 14 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-dqevent.rst b/Documentation/userspace-api/media/v4l/vidioc-dqevent.rst
index 8db103760930..f317af57a92c 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-dqevent.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-dqevent.rst
@@ -369,6 +369,18 @@ call.
 	loss of signal and so restarting streaming I/O is required in order for
 	the hardware to synchronize to the video signal.
 
+    * - ``V4L2_EVENT_SRC_CH_COLORSPACE``
+      - 0x0002
+      - This event gets triggered when a colorspace change is detected at
+	an input. This can come from a video decoder or a video receiver.
+	Applications will query the new colorspace information
+	(if any, the signal may also have been lost). If the signal is lost,
+	then that is a CH_RESOLUTION change, not CH_COLORSPACE.
+
+	For stateful decoders follow the guidelines in :ref:`decoder`.
+	If CH_COLORSPACE is set, but not CH_RESOLUTION, then only the
+	colorspace changed and there is no need to reallocate buffers.
+
 Return Value
 ============
 
diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
index 35d3456cc812..ac47c6d9448b 100644
--- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions
+++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
@@ -526,6 +526,7 @@ replace define V4L2_EVENT_CTRL_CH_RANGE ctrl-changes-flags
 replace define V4L2_EVENT_CTRL_CH_DIMENSIONS ctrl-changes-flags
 
 replace define V4L2_EVENT_SRC_CH_RESOLUTION src-changes-flags
+replace define V4L2_EVENT_SRC_CH_COLORSPACE src-changes-flags
 
 replace define V4L2_EVENT_MD_FL_HAVE_FRAME_SEQ :c:type:`v4l2_event_motion_det`
 
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index c8cb2796130f..242242c8e57b 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -2559,6 +2559,7 @@ struct v4l2_event_frame_sync {
 };
 
 #define V4L2_EVENT_SRC_CH_RESOLUTION		(1 << 0)
+#define V4L2_EVENT_SRC_CH_COLORSPACE		(1 << 1)
 
 struct v4l2_event_src_change {
 	__u32 changes;
-- 
2.43.0-rc1

[PATCH v3 2/4] media: docs: dev-decoder: Trigger dynamic source change for colorspace

[ming.qian]

From: Ming Qian <ming.qian@oss.nxp.com>

If colorspace changes, the client needs to renegotiate the pipeline,
otherwise the decoded frame may not be displayed correctly.

When a colorspace change in the stream, the decoder sends a
V4L2_EVENT_SOURCE_CHANGE event with changes set to
V4L2_EVENT_SRC_CH_COLORSPACE. After client receive this source change
event, then client can switch to the correct stream setting. And each
frame can be displayed properly.

So add colorspace as a trigger parameter for dynamic resolution change.

Signed-off-by: Ming Qian <ming.qian@oss.nxp.com>
---
v2
- Add V4L2_EVENT_SRC_CH_COLORSPACE for colorspace source change event

 .../userspace-api/media/v4l/dev-decoder.rst     | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/dev-decoder.rst b/Documentation/userspace-api/media/v4l/dev-decoder.rst
index ef8e8cf31f90..51d6da3eea4a 100644
--- a/Documentation/userspace-api/media/v4l/dev-decoder.rst
+++ b/Documentation/userspace-api/media/v4l/dev-decoder.rst
@@ -784,8 +784,8 @@ before the sequence started. Last of the buffers will have the
 must check if there is any pending event and:
 
 * if a ``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to
-  ``V4L2_EVENT_SRC_CH_RESOLUTION`` is pending, the `Dynamic Resolution
-  Change` sequence needs to be followed,
+  ``V4L2_EVENT_SRC_CH_RESOLUTION`` or ``V4L2_EVENT_SRC_CH_COLORSPACE`` is pending,
+  the `Dynamic Resolution Change` sequence needs to be followed,
 
 * if a ``V4L2_EVENT_EOS`` event is pending, the `End of Stream` sequence needs
   to be followed.
@@ -932,13 +932,17 @@ reflected by corresponding queries):
 
 * the minimum number of buffers needed for decoding,
 
-* bit-depth of the bitstream has been changed.
+* bit-depth of the bitstream has been changed,
+
+* colorspace of the bitstream has been changed.
 
 Whenever that happens, the decoder must proceed as follows:
 
 1.  After encountering a resolution change in the stream, the decoder sends a
     ``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to
-    ``V4L2_EVENT_SRC_CH_RESOLUTION``.
+    ``V4L2_EVENT_SRC_CH_RESOLUTION``, or a colorspace change in the stream, the
+    decoder sends a ``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to
+    ``V4L2_EVENT_SRC_CH_COLORSPACE``.
 
     .. important::
 
@@ -946,6 +950,11 @@ Whenever that happens, the decoder must proceed as follows:
        values applying to the stream after the resolution change, including
        queue formats, selection rectangles and controls.
 
+.. note::
+        A ``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to
+        ``V4L2_EVENT_SRC_CH_RESOLUTION`` will affect the allocation, but
+        ``V4L2_EVENT_SRC_CH_COLORSPACE`` won't.
+
 2.  The decoder will then process and decode all remaining buffers from before
     the resolution change point.
 
-- 
2.43.0-rc1

[PATCH v3 3/4] media: amphion: Clear last_buffer_dequeued flag for DEC_CMD_START

[ming.qian]

From: Ming Qian <ming.qian@oss.nxp.com>

The V4L2_DEC_CMD_START command may be used to handle the dynamic source
change, which will triggers an implicit decoder drain.
The last_buffer_dequeued flag is set in the implicit decoder drain,
so driver need to clear it to continue the following decoding flow.

Signed-off-by: Ming Qian <ming.qian@oss.nxp.com>
---
 drivers/media/platform/amphion/vdec.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/media/platform/amphion/vdec.c b/drivers/media/platform/amphion/vdec.c
index d812b86f001e..ee0a7572ed15 100644
--- a/drivers/media/platform/amphion/vdec.c
+++ b/drivers/media/platform/amphion/vdec.c
@@ -728,6 +728,7 @@ static int vdec_decoder_cmd(struct file *file, void *fh, struct v4l2_decoder_cmd
 	switch (cmd->cmd) {
 	case V4L2_DEC_CMD_START:
 		vdec_cmd_start(inst);
+		vb2_clear_last_buffer_dequeued(v4l2_m2m_get_dst_vq(inst->fh.m2m_ctx));
 		break;
 	case V4L2_DEC_CMD_STOP:
 		vdec_cmd_stop(inst);
-- 
2.43.0-rc1

[PATCH v3 4/4] media: amphion: Trigger source change if colorspace chagned

[ming.qian]

From: Ming Qian <ming.qian@oss.nxp.com>

After encountering a colorspace change in the stream, the decoder
sends a V4L2_EVENT_SOURCE_CHANGE event with changes set to
V4L2_EVENT_SRC_CH_COLORSPACE.

Then the client can handle the colorspace change without any buffer
reallocation

Signed-off-by: Ming Qian <ming.qian@oss.nxp.com>
---
 drivers/media/platform/amphion/vdec.c     | 89 +++++++++++++++--------
 drivers/media/platform/amphion/vpu.h      |  1 +
 drivers/media/platform/amphion/vpu_v4l2.c |  7 +-
 3 files changed, 63 insertions(+), 34 deletions(-)

diff --git a/drivers/media/platform/amphion/vdec.c b/drivers/media/platform/amphion/vdec.c
index ee0a7572ed15..f4979d537b97 100644
--- a/drivers/media/platform/amphion/vdec.c
+++ b/drivers/media/platform/amphion/vdec.c
@@ -369,12 +369,16 @@ static void vdec_handle_resolution_change(struct vpu_inst *inst)
 	if (!vdec->source_change)
 		return;
 
+	if (inst->changes) {
+		vpu_notify_source_change(inst);
+		inst->changes = 0;
+	}
+
 	q = v4l2_m2m_get_dst_vq(inst->fh.m2m_ctx);
 	if (!list_empty(&q->done_list))
 		return;
 
 	vdec->source_change--;
-	vpu_notify_source_change(inst);
 	vpu_set_last_buffer_dequeued(inst, false);
 }
 
@@ -954,10 +958,11 @@ static void vdec_stop_done(struct vpu_inst *inst)
 	vpu_inst_unlock(inst);
 }
 
-static bool vdec_check_source_change(struct vpu_inst *inst)
+static bool vdec_check_source_change(struct vpu_inst *inst, struct vpu_dec_codec_info *hdr)
 {
 	struct vdec_t *vdec = inst->priv;
 	const struct vpu_format *sibling;
+	u32 changes = 0;
 
 	if (!inst->fh.m2m_ctx)
 		return false;
@@ -966,27 +971,41 @@ static bool vdec_check_source_change(struct vpu_inst *inst)
 		return false;
 
 	sibling = vpu_helper_find_sibling(inst, inst->cap_format.type, inst->cap_format.pixfmt);
-	if (sibling && vdec->codec_info.pixfmt == sibling->pixfmt)
-		vdec->codec_info.pixfmt = inst->cap_format.pixfmt;
+	if (sibling && hdr->pixfmt == sibling->pixfmt)
+		hdr->pixfmt = inst->cap_format.pixfmt;
 
 	if (!vb2_is_streaming(v4l2_m2m_get_dst_vq(inst->fh.m2m_ctx)))
-		return true;
-	if (inst->cap_format.pixfmt != vdec->codec_info.pixfmt)
-		return true;
-	if (inst->cap_format.width != vdec->codec_info.decoded_width)
-		return true;
-	if (inst->cap_format.height != vdec->codec_info.decoded_height)
-		return true;
+		changes |= V4L2_EVENT_SRC_CH_RESOLUTION;
+	if (inst->cap_format.pixfmt != hdr->pixfmt)
+		changes |= V4L2_EVENT_SRC_CH_RESOLUTION;
+	if (inst->cap_format.width != hdr->decoded_width)
+		changes |= V4L2_EVENT_SRC_CH_RESOLUTION;
+	if (inst->cap_format.height != hdr->decoded_height)
+		changes |= V4L2_EVENT_SRC_CH_RESOLUTION;
 	if (vpu_get_num_buffers(inst, inst->cap_format.type) < inst->min_buffer_cap)
+		changes |= V4L2_EVENT_SRC_CH_RESOLUTION;
+	if (inst->crop.left != hdr->offset_x)
+		changes |= V4L2_EVENT_SRC_CH_RESOLUTION;
+	if (inst->crop.top != hdr->offset_y)
+		changes |= V4L2_EVENT_SRC_CH_RESOLUTION;
+	if (inst->crop.width != hdr->width)
+		changes |= V4L2_EVENT_SRC_CH_RESOLUTION;
+	if (inst->crop.height != hdr->height)
+		changes |= V4L2_EVENT_SRC_CH_RESOLUTION;
+	if (!hdr->progressive)
+		changes |= V4L2_EVENT_SRC_CH_RESOLUTION;
+
+	if (vdec->seq_hdr_found &&
+	    (hdr->color_primaries != vdec->codec_info.color_primaries ||
+	     hdr->transfer_chars != vdec->codec_info.transfer_chars ||
+	     hdr->matrix_coeffs != vdec->codec_info.matrix_coeffs ||
+	     hdr->full_range != vdec->codec_info.full_range))
+		changes |= V4L2_EVENT_SRC_CH_COLORSPACE;
+
+	if (changes) {
+		inst->changes |= changes;
 		return true;
-	if (inst->crop.left != vdec->codec_info.offset_x)
-		return true;
-	if (inst->crop.top != vdec->codec_info.offset_y)
-		return true;
-	if (inst->crop.width != vdec->codec_info.width)
-		return true;
-	if (inst->crop.height != vdec->codec_info.height)
-		return true;
+	}
 
 	return false;
 }
@@ -1337,20 +1356,25 @@ static void vdec_event_seq_hdr(struct vpu_inst *inst, struct vpu_dec_codec_info
 	struct vdec_t *vdec = inst->priv;
 
 	vpu_inst_lock(inst);
-	memcpy(&vdec->codec_info, hdr, sizeof(vdec->codec_info));
 
-	vpu_trace(inst->dev, "[%d] %d x %d, crop : (%d, %d) %d x %d, %d, %d\n",
+	vpu_trace(inst->dev,
+		  "[%d] %d x %d, crop : (%d, %d) %d x %d, %d, %d, colorspace: %d, %d, %d, %d\n",
 		  inst->id,
-		  vdec->codec_info.decoded_width,
-		  vdec->codec_info.decoded_height,
-		  vdec->codec_info.offset_x,
-		  vdec->codec_info.offset_y,
-		  vdec->codec_info.width,
-		  vdec->codec_info.height,
+		  hdr->decoded_width,
+		  hdr->decoded_height,
+		  hdr->offset_x,
+		  hdr->offset_y,
+		  hdr->width,
+		  hdr->height,
 		  hdr->num_ref_frms,
-		  hdr->num_dpb_frms);
+		  hdr->num_dpb_frms,
+		  hdr->color_primaries,
+		  hdr->transfer_chars,
+		  hdr->matrix_coeffs,
+		  hdr->full_range);
 	inst->min_buffer_cap = hdr->num_ref_frms + hdr->num_dpb_frms;
-	vdec->is_source_changed = vdec_check_source_change(inst);
+	vdec->is_source_changed = vdec_check_source_change(inst, hdr);
+	memcpy(&vdec->codec_info, hdr, sizeof(vdec->codec_info));
 	vdec_init_fmt(inst);
 	vdec_init_crop(inst);
 	vdec_init_mbi(inst);
@@ -1379,7 +1403,12 @@ static void vdec_event_resolution_change(struct vpu_inst *inst)
 {
 	struct vdec_t *vdec = inst->priv;
 
-	vpu_trace(inst->dev, "[%d]\n", inst->id);
+	vpu_trace(inst->dev, "[%d] input : %d, decoded : %d, display : %d, sequence : %d\n",
+		  inst->id,
+		  vdec->params.frame_count,
+		  vdec->decoded_frame_count,
+		  vdec->display_frame_count,
+		  vdec->sequence);
 	vpu_inst_lock(inst);
 	vdec->seq_tag = vdec->codec_info.tag;
 	vdec_clear_fs(&vdec->mbi);
diff --git a/drivers/media/platform/amphion/vpu.h b/drivers/media/platform/amphion/vpu.h
index 76bfd6b26170..d8100da160d1 100644
--- a/drivers/media/platform/amphion/vpu.h
+++ b/drivers/media/platform/amphion/vpu.h
@@ -272,6 +272,7 @@ struct vpu_inst {
 	u8 xfer_func;
 	u32 sequence;
 	u32 extra_size;
+	u32 changes;
 
 	u32 flows[16];
 	u32 flow_idx;
diff --git a/drivers/media/platform/amphion/vpu_v4l2.c b/drivers/media/platform/amphion/vpu_v4l2.c
index 74668fa362e2..37ef706c29dd 100644
--- a/drivers/media/platform/amphion/vpu_v4l2.c
+++ b/drivers/media/platform/amphion/vpu_v4l2.c
@@ -96,13 +96,12 @@ int vpu_notify_eos(struct vpu_inst *inst)
 
 int vpu_notify_source_change(struct vpu_inst *inst)
 {
-	static const struct v4l2_event ev = {
-		.id = 0,
+	const struct v4l2_event ev = {
 		.type = V4L2_EVENT_SOURCE_CHANGE,
-		.u.src_change.changes = V4L2_EVENT_SRC_CH_RESOLUTION
+		.u.src_change.changes = inst->changes
 	};
 
-	vpu_trace(inst->dev, "[%d]\n", inst->id);
+	vpu_trace(inst->dev, "[%d] source change 0x%x\n", inst->id, inst->changes);
 	v4l2_event_queue_fh(&inst->fh, &ev);
 	return 0;
 }
-- 
2.43.0-rc1