* [PATCH v7 1/7] media: platform: amd: Introduce amd isp4 capture driver
2025-12-16 9:13 [PATCH v7 0/7] Add AMD ISP4 driver Bin Du
@ 2025-12-16 9:13 ` Bin Du
2025-12-22 9:23 ` Sakari Ailus
2025-12-16 9:13 ` [PATCH v7 2/7] media: platform: amd: low level support for isp4 firmware Bin Du
` (8 subsequent siblings)
9 siblings, 1 reply; 46+ messages in thread
From: Bin Du @ 2025-12-16 9:13 UTC (permalink / raw)
To: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
sultan
Cc: pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, bin.du, Bin Du, Svetoslav Stoilov, Mario Limonciello,
Alexey Zagorodnikov
AMD isp4 capture is a v4l2 media device which implements media controller
interface. It has one sub-device (AMD ISP4 sub-device) endpoint which can
be connected to a remote CSI2 TX endpoint. It supports only one physical
interface for now. Also add ISP4 driver related entry info into the
MAINTAINERS file
Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
Signed-off-by: Bin Du <Bin.Du@amd.com>
Reviewed-by: Mario Limonciello (AMD) <superm1@kernel.org>
Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
---
MAINTAINERS | 13 +++
drivers/media/platform/Kconfig | 1 +
drivers/media/platform/Makefile | 1 +
drivers/media/platform/amd/Kconfig | 3 +
drivers/media/platform/amd/Makefile | 3 +
drivers/media/platform/amd/isp4/Kconfig | 14 +++
drivers/media/platform/amd/isp4/Makefile | 6 ++
drivers/media/platform/amd/isp4/isp4.c | 132 +++++++++++++++++++++++
drivers/media/platform/amd/isp4/isp4.h | 17 +++
9 files changed, 190 insertions(+)
create mode 100644 drivers/media/platform/amd/Kconfig
create mode 100644 drivers/media/platform/amd/Makefile
create mode 100644 drivers/media/platform/amd/isp4/Kconfig
create mode 100644 drivers/media/platform/amd/isp4/Makefile
create mode 100644 drivers/media/platform/amd/isp4/isp4.c
create mode 100644 drivers/media/platform/amd/isp4/isp4.h
diff --git a/MAINTAINERS b/MAINTAINERS
index cd7ff55b5d32..3640a1e3262c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1133,6 +1133,19 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/iommu/linux.git
F: drivers/iommu/amd/
F: include/linux/amd-iommu.h
+AMD ISP4 DRIVER
+M: Bin Du <bin.du@amd.com>
+M: Nirujogi Pratap <pratap.nirujogi@amd.com>
+L: linux-media@vger.kernel.org
+S: Maintained
+T: git git://linuxtv.org/media.git
+F: drivers/media/platform/amd/Kconfig
+F: drivers/media/platform/amd/Makefile
+F: drivers/media/platform/amd/isp4/Kconfig
+F: drivers/media/platform/amd/isp4/Makefile
+F: drivers/media/platform/amd/isp4/isp4.c
+F: drivers/media/platform/amd/isp4/isp4.h
+
AMD KFD
M: Felix Kuehling <Felix.Kuehling@amd.com>
L: amd-gfx@lists.freedesktop.org
diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
index 9287faafdce5..772c70665510 100644
--- a/drivers/media/platform/Kconfig
+++ b/drivers/media/platform/Kconfig
@@ -63,6 +63,7 @@ config VIDEO_MUX
# Platform drivers - Please keep it alphabetically sorted
source "drivers/media/platform/allegro-dvt/Kconfig"
+source "drivers/media/platform/amd/Kconfig"
source "drivers/media/platform/amlogic/Kconfig"
source "drivers/media/platform/amphion/Kconfig"
source "drivers/media/platform/aspeed/Kconfig"
diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
index 6fd7db0541c7..b207bd8d8022 100644
--- a/drivers/media/platform/Makefile
+++ b/drivers/media/platform/Makefile
@@ -6,6 +6,7 @@
# Place here, alphabetically sorted by directory
# (e. g. LC_ALL=C sort Makefile)
obj-y += allegro-dvt/
+obj-y += amd/
obj-y += amlogic/
obj-y += amphion/
obj-y += aspeed/
diff --git a/drivers/media/platform/amd/Kconfig b/drivers/media/platform/amd/Kconfig
new file mode 100644
index 000000000000..25af49f246b2
--- /dev/null
+++ b/drivers/media/platform/amd/Kconfig
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0+
+
+source "drivers/media/platform/amd/isp4/Kconfig"
diff --git a/drivers/media/platform/amd/Makefile b/drivers/media/platform/amd/Makefile
new file mode 100644
index 000000000000..8bfc1955f22e
--- /dev/null
+++ b/drivers/media/platform/amd/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0+
+
+obj-y += isp4/
diff --git a/drivers/media/platform/amd/isp4/Kconfig b/drivers/media/platform/amd/isp4/Kconfig
new file mode 100644
index 000000000000..d4e4ad436600
--- /dev/null
+++ b/drivers/media/platform/amd/isp4/Kconfig
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0+
+
+config AMD_ISP4
+ tristate "AMD ISP4 and camera driver"
+ depends on DRM_AMD_ISP && VIDEO_DEV
+ select VIDEOBUF2_CORE
+ select VIDEOBUF2_MEMOPS
+ select VIDEOBUF2_V4L2
+ select VIDEO_V4L2_SUBDEV_API
+ help
+ This is support for AMD ISP4 and camera subsystem driver.
+ Say Y here to enable the ISP4 and camera device for video capture.
+ To compile this driver as a module, choose M here. The module will
+ be called amd_capture.
diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
new file mode 100644
index 000000000000..de0092dad26f
--- /dev/null
+++ b/drivers/media/platform/amd/isp4/Makefile
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0+
+#
+# Copyright (C) 2025 Advanced Micro Devices, Inc.
+
+obj-$(CONFIG_AMD_ISP4) += amd_capture.o
+amd_capture-objs := isp4.o
diff --git a/drivers/media/platform/amd/isp4/isp4.c b/drivers/media/platform/amd/isp4/isp4.c
new file mode 100644
index 000000000000..ad95e7f89189
--- /dev/null
+++ b/drivers/media/platform/amd/isp4/isp4.c
@@ -0,0 +1,132 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2025 Advanced Micro Devices, Inc.
+ */
+
+#include <linux/pm_runtime.h>
+#include <linux/vmalloc.h>
+#include <media/v4l2-ioctl.h>
+
+#include "isp4.h"
+
+#define VIDEO_BUF_NUM 5
+
+#define ISP4_DRV_NAME "amd_isp_capture"
+
+static const struct {
+ const char *name;
+ u32 status_mask;
+ u32 en_mask;
+ u32 ack_mask;
+ u32 rb_int_num;
+} isp4_irq[] = {
+ /* The IRQ order is aligned with the isp4_subdev.fw_resp_thread order */
+ {
+ .name = "isp_irq_global",
+ .rb_int_num = 4, /* ISP_4_1__SRCID__ISP_RINGBUFFER_WPT12 */
+ },
+ {
+ .name = "isp_irq_stream1",
+ .rb_int_num = 0, /* ISP_4_1__SRCID__ISP_RINGBUFFER_WPT9 */
+ },
+};
+
+static irqreturn_t isp4_irq_handler(int irq, void *arg)
+{
+ return IRQ_HANDLED;
+}
+
+static int isp4_capture_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ int irq[ARRAY_SIZE(isp4_irq)];
+ struct isp4_device *isp_dev;
+ size_t i;
+ int ret;
+
+ isp_dev = devm_kzalloc(dev, sizeof(*isp_dev), GFP_KERNEL);
+ if (!isp_dev)
+ return -ENOMEM;
+
+ dev->init_name = ISP4_DRV_NAME;
+
+ for (i = 0; i < ARRAY_SIZE(isp4_irq); i++) {
+ irq[i] = platform_get_irq(pdev, isp4_irq[i].rb_int_num);
+ if (irq[i] < 0)
+ return dev_err_probe(dev, irq[i], "fail to get irq %d\n",
+ isp4_irq[i].rb_int_num);
+
+ ret = devm_request_irq(dev, irq[i], isp4_irq_handler,
+ IRQF_NO_AUTOEN, isp4_irq[i].name, dev);
+ if (ret)
+ return dev_err_probe(dev, ret, "fail to req irq %d\n", irq[i]);
+ }
+
+ /* Link the media device within the v4l2_device */
+ isp_dev->v4l2_dev.mdev = &isp_dev->mdev;
+
+ /* Initialize media device */
+ strscpy(isp_dev->mdev.model, "amd_isp41_mdev", sizeof(isp_dev->mdev.model));
+ snprintf(isp_dev->mdev.bus_info, sizeof(isp_dev->mdev.bus_info),
+ "platform:%s", ISP4_DRV_NAME);
+ isp_dev->mdev.dev = dev;
+ media_device_init(&isp_dev->mdev);
+
+ /* register v4l2 device */
+ snprintf(isp_dev->v4l2_dev.name, sizeof(isp_dev->v4l2_dev.name),
+ "AMD-V4L2-ROOT");
+ ret = v4l2_device_register(dev, &isp_dev->v4l2_dev);
+ if (ret) {
+ dev_err_probe(dev, ret, "fail register v4l2 device\n");
+ goto err_clean_media;
+ }
+
+ pm_runtime_set_suspended(dev);
+ pm_runtime_enable(dev);
+ ret = media_device_register(&isp_dev->mdev);
+ if (ret) {
+ dev_err_probe(dev, ret, "fail to register media device\n");
+ goto err_isp4_deinit;
+ }
+
+ platform_set_drvdata(pdev, isp_dev);
+
+ return 0;
+
+err_isp4_deinit:
+ pm_runtime_disable(dev);
+ v4l2_device_unregister(&isp_dev->v4l2_dev);
+err_clean_media:
+ media_device_cleanup(&isp_dev->mdev);
+
+ return ret;
+}
+
+static void isp4_capture_remove(struct platform_device *pdev)
+{
+ struct isp4_device *isp_dev = platform_get_drvdata(pdev);
+ struct device *dev = &pdev->dev;
+
+ media_device_unregister(&isp_dev->mdev);
+ pm_runtime_disable(dev);
+ v4l2_device_unregister(&isp_dev->v4l2_dev);
+ media_device_cleanup(&isp_dev->mdev);
+}
+
+static struct platform_driver isp4_capture_drv = {
+ .probe = isp4_capture_probe,
+ .remove = isp4_capture_remove,
+ .driver = {
+ .name = ISP4_DRV_NAME,
+ }
+};
+
+module_platform_driver(isp4_capture_drv);
+
+MODULE_ALIAS("platform:" ISP4_DRV_NAME);
+MODULE_IMPORT_NS("DMA_BUF");
+
+MODULE_DESCRIPTION("AMD ISP4 Driver");
+MODULE_AUTHOR("Bin Du <bin.du@amd.com>");
+MODULE_AUTHOR("Pratap Nirujogi <pratap.nirujogi@amd.com>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/media/platform/amd/isp4/isp4.h b/drivers/media/platform/amd/isp4/isp4.h
new file mode 100644
index 000000000000..7f2db0dfa2d9
--- /dev/null
+++ b/drivers/media/platform/amd/isp4/isp4.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (C) 2025 Advanced Micro Devices, Inc.
+ */
+
+#ifndef _ISP4_H_
+#define _ISP4_H_
+
+#include <media/v4l2-device.h>
+#include <media/videobuf2-memops.h>
+
+struct isp4_device {
+ struct v4l2_device v4l2_dev;
+ struct media_device mdev;
+};
+
+#endif /* _ISP4_H_ */
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread* Re: [PATCH v7 1/7] media: platform: amd: Introduce amd isp4 capture driver
2025-12-16 9:13 ` [PATCH v7 1/7] media: platform: amd: Introduce amd isp4 capture driver Bin Du
@ 2025-12-22 9:23 ` Sakari Ailus
2026-01-06 8:30 ` Du, Bin
0 siblings, 1 reply; 46+ messages in thread
From: Sakari Ailus @ 2025-12-22 9:23 UTC (permalink / raw)
To: Bin Du
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Svetoslav Stoilov, Mario Limonciello,
Alexey Zagorodnikov
Hi Bin,
Thanks for the update. Please see my comments below.
On Tue, Dec 16, 2025 at 05:13:20PM +0800, Bin Du wrote:
> AMD isp4 capture is a v4l2 media device which implements media controller
> interface. It has one sub-device (AMD ISP4 sub-device) endpoint which can
> be connected to a remote CSI2 TX endpoint. It supports only one physical
> interface for now. Also add ISP4 driver related entry info into the
> MAINTAINERS file
>
> Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
> Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
> Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
> Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
> Signed-off-by: Bin Du <Bin.Du@amd.com>
> Reviewed-by: Mario Limonciello (AMD) <superm1@kernel.org>
> Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
> ---
> MAINTAINERS | 13 +++
> drivers/media/platform/Kconfig | 1 +
> drivers/media/platform/Makefile | 1 +
> drivers/media/platform/amd/Kconfig | 3 +
> drivers/media/platform/amd/Makefile | 3 +
> drivers/media/platform/amd/isp4/Kconfig | 14 +++
> drivers/media/platform/amd/isp4/Makefile | 6 ++
> drivers/media/platform/amd/isp4/isp4.c | 132 +++++++++++++++++++++++
> drivers/media/platform/amd/isp4/isp4.h | 17 +++
> 9 files changed, 190 insertions(+)
> create mode 100644 drivers/media/platform/amd/Kconfig
> create mode 100644 drivers/media/platform/amd/Makefile
> create mode 100644 drivers/media/platform/amd/isp4/Kconfig
> create mode 100644 drivers/media/platform/amd/isp4/Makefile
> create mode 100644 drivers/media/platform/amd/isp4/isp4.c
> create mode 100644 drivers/media/platform/amd/isp4/isp4.h
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index cd7ff55b5d32..3640a1e3262c 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1133,6 +1133,19 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/iommu/linux.git
> F: drivers/iommu/amd/
> F: include/linux/amd-iommu.h
>
> +AMD ISP4 DRIVER
> +M: Bin Du <bin.du@amd.com>
> +M: Nirujogi Pratap <pratap.nirujogi@amd.com>
> +L: linux-media@vger.kernel.org
> +S: Maintained
> +T: git git://linuxtv.org/media.git
> +F: drivers/media/platform/amd/Kconfig
> +F: drivers/media/platform/amd/Makefile
> +F: drivers/media/platform/amd/isp4/Kconfig
> +F: drivers/media/platform/amd/isp4/Makefile
> +F: drivers/media/platform/amd/isp4/isp4.c
> +F: drivers/media/platform/amd/isp4/isp4.h
> +
> AMD KFD
> M: Felix Kuehling <Felix.Kuehling@amd.com>
> L: amd-gfx@lists.freedesktop.org
> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> index 9287faafdce5..772c70665510 100644
> --- a/drivers/media/platform/Kconfig
> +++ b/drivers/media/platform/Kconfig
> @@ -63,6 +63,7 @@ config VIDEO_MUX
>
> # Platform drivers - Please keep it alphabetically sorted
> source "drivers/media/platform/allegro-dvt/Kconfig"
> +source "drivers/media/platform/amd/Kconfig"
> source "drivers/media/platform/amlogic/Kconfig"
> source "drivers/media/platform/amphion/Kconfig"
> source "drivers/media/platform/aspeed/Kconfig"
> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> index 6fd7db0541c7..b207bd8d8022 100644
> --- a/drivers/media/platform/Makefile
> +++ b/drivers/media/platform/Makefile
> @@ -6,6 +6,7 @@
> # Place here, alphabetically sorted by directory
> # (e. g. LC_ALL=C sort Makefile)
> obj-y += allegro-dvt/
> +obj-y += amd/
> obj-y += amlogic/
> obj-y += amphion/
> obj-y += aspeed/
> diff --git a/drivers/media/platform/amd/Kconfig b/drivers/media/platform/amd/Kconfig
> new file mode 100644
> index 000000000000..25af49f246b2
> --- /dev/null
> +++ b/drivers/media/platform/amd/Kconfig
> @@ -0,0 +1,3 @@
> +# SPDX-License-Identifier: GPL-2.0+
> +
> +source "drivers/media/platform/amd/isp4/Kconfig"
> diff --git a/drivers/media/platform/amd/Makefile b/drivers/media/platform/amd/Makefile
> new file mode 100644
> index 000000000000..8bfc1955f22e
> --- /dev/null
> +++ b/drivers/media/platform/amd/Makefile
> @@ -0,0 +1,3 @@
> +# SPDX-License-Identifier: GPL-2.0+
> +
> +obj-y += isp4/
> diff --git a/drivers/media/platform/amd/isp4/Kconfig b/drivers/media/platform/amd/isp4/Kconfig
> new file mode 100644
> index 000000000000..d4e4ad436600
> --- /dev/null
> +++ b/drivers/media/platform/amd/isp4/Kconfig
> @@ -0,0 +1,14 @@
> +# SPDX-License-Identifier: GPL-2.0+
> +
> +config AMD_ISP4
We've used "VIDEO_" prefix for V4L2 drivers.
> + tristate "AMD ISP4 and camera driver"
> + depends on DRM_AMD_ISP && VIDEO_DEV
> + select VIDEOBUF2_CORE
> + select VIDEOBUF2_MEMOPS
> + select VIDEOBUF2_V4L2
> + select VIDEO_V4L2_SUBDEV_API
> + help
> + This is support for AMD ISP4 and camera subsystem driver.
> + Say Y here to enable the ISP4 and camera device for video capture.
> + To compile this driver as a module, choose M here. The module will
> + be called amd_capture.
How about "amd_isp4"? That would be aligned with the file names as well as
the Kconfig option.
> diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
> new file mode 100644
> index 000000000000..de0092dad26f
> --- /dev/null
> +++ b/drivers/media/platform/amd/isp4/Makefile
> @@ -0,0 +1,6 @@
> +# SPDX-License-Identifier: GPL-2.0+
> +#
> +# Copyright (C) 2025 Advanced Micro Devices, Inc.
> +
> +obj-$(CONFIG_AMD_ISP4) += amd_capture.o
> +amd_capture-objs := isp4.o
> diff --git a/drivers/media/platform/amd/isp4/isp4.c b/drivers/media/platform/amd/isp4/isp4.c
> new file mode 100644
> index 000000000000..ad95e7f89189
> --- /dev/null
> +++ b/drivers/media/platform/amd/isp4/isp4.c
> @@ -0,0 +1,132 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
> + */
> +
> +#include <linux/pm_runtime.h>
> +#include <linux/vmalloc.h>
> +#include <media/v4l2-ioctl.h>
> +
> +#include "isp4.h"
> +
> +#define VIDEO_BUF_NUM 5
> +
> +#define ISP4_DRV_NAME "amd_isp_capture"
> +
> +static const struct {
> + const char *name;
> + u32 status_mask;
> + u32 en_mask;
> + u32 ack_mask;
> + u32 rb_int_num;
> +} isp4_irq[] = {
> + /* The IRQ order is aligned with the isp4_subdev.fw_resp_thread order */
> + {
> + .name = "isp_irq_global",
> + .rb_int_num = 4, /* ISP_4_1__SRCID__ISP_RINGBUFFER_WPT12 */
> + },
> + {
> + .name = "isp_irq_stream1",
> + .rb_int_num = 0, /* ISP_4_1__SRCID__ISP_RINGBUFFER_WPT9 */
> + },
> +};
> +
> +static irqreturn_t isp4_irq_handler(int irq, void *arg)
> +{
> + return IRQ_HANDLED;
> +}
> +
> +static int isp4_capture_probe(struct platform_device *pdev)
> +{
> + struct device *dev = &pdev->dev;
> + int irq[ARRAY_SIZE(isp4_irq)];
> + struct isp4_device *isp_dev;
> + size_t i;
> + int ret;
> +
> + isp_dev = devm_kzalloc(dev, sizeof(*isp_dev), GFP_KERNEL);
> + if (!isp_dev)
> + return -ENOMEM;
> +
> + dev->init_name = ISP4_DRV_NAME;
> +
> + for (i = 0; i < ARRAY_SIZE(isp4_irq); i++) {
> + irq[i] = platform_get_irq(pdev, isp4_irq[i].rb_int_num);
> + if (irq[i] < 0)
> + return dev_err_probe(dev, irq[i], "fail to get irq %d\n",
> + isp4_irq[i].rb_int_num);
> +
> + ret = devm_request_irq(dev, irq[i], isp4_irq_handler,
> + IRQF_NO_AUTOEN, isp4_irq[i].name, dev);
> + if (ret)
> + return dev_err_probe(dev, ret, "fail to req irq %d\n", irq[i]);
> + }
> +
> + /* Link the media device within the v4l2_device */
> + isp_dev->v4l2_dev.mdev = &isp_dev->mdev;
> +
> + /* Initialize media device */
> + strscpy(isp_dev->mdev.model, "amd_isp41_mdev", sizeof(isp_dev->mdev.model));
> + snprintf(isp_dev->mdev.bus_info, sizeof(isp_dev->mdev.bus_info),
> + "platform:%s", ISP4_DRV_NAME);
No need to do this explicitly.
> + isp_dev->mdev.dev = dev;
> + media_device_init(&isp_dev->mdev);
> +
> + /* register v4l2 device */
Please drop this comment, it's not informational.
> + snprintf(isp_dev->v4l2_dev.name, sizeof(isp_dev->v4l2_dev.name),
> + "AMD-V4L2-ROOT");
> + ret = v4l2_device_register(dev, &isp_dev->v4l2_dev);
> + if (ret) {
> + dev_err_probe(dev, ret, "fail register v4l2 device\n");
> + goto err_clean_media;
> + }
> +
> + pm_runtime_set_suspended(dev);
> + pm_runtime_enable(dev);
> + ret = media_device_register(&isp_dev->mdev);
> + if (ret) {
> + dev_err_probe(dev, ret, "fail to register media device\n");
> + goto err_isp4_deinit;
> + }
> +
> + platform_set_drvdata(pdev, isp_dev);
> +
> + return 0;
> +
> +err_isp4_deinit:
> + pm_runtime_disable(dev);
> + v4l2_device_unregister(&isp_dev->v4l2_dev);
> +err_clean_media:
> + media_device_cleanup(&isp_dev->mdev);
> +
> + return ret;
> +}
> +
> +static void isp4_capture_remove(struct platform_device *pdev)
> +{
> + struct isp4_device *isp_dev = platform_get_drvdata(pdev);
> + struct device *dev = &pdev->dev;
> +
> + media_device_unregister(&isp_dev->mdev);
> + pm_runtime_disable(dev);
> + v4l2_device_unregister(&isp_dev->v4l2_dev);
> + media_device_cleanup(&isp_dev->mdev);
> +}
> +
> +static struct platform_driver isp4_capture_drv = {
> + .probe = isp4_capture_probe,
> + .remove = isp4_capture_remove,
> + .driver = {
> + .name = ISP4_DRV_NAME,
> + }
> +};
> +
> +module_platform_driver(isp4_capture_drv);
> +
> +MODULE_ALIAS("platform:" ISP4_DRV_NAME);
> +MODULE_IMPORT_NS("DMA_BUF");
> +
> +MODULE_DESCRIPTION("AMD ISP4 Driver");
> +MODULE_AUTHOR("Bin Du <bin.du@amd.com>");
> +MODULE_AUTHOR("Pratap Nirujogi <pratap.nirujogi@amd.com>");
> +MODULE_LICENSE("GPL");
> diff --git a/drivers/media/platform/amd/isp4/isp4.h b/drivers/media/platform/amd/isp4/isp4.h
> new file mode 100644
> index 000000000000..7f2db0dfa2d9
> --- /dev/null
> +++ b/drivers/media/platform/amd/isp4/isp4.h
> @@ -0,0 +1,17 @@
> +/* SPDX-License-Identifier: GPL-2.0+ */
> +/*
> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
> + */
> +
> +#ifndef _ISP4_H_
> +#define _ISP4_H_
> +
> +#include <media/v4l2-device.h>
> +#include <media/videobuf2-memops.h>
> +
> +struct isp4_device {
> + struct v4l2_device v4l2_dev;
> + struct media_device mdev;
> +};
> +
> +#endif /* _ISP4_H_ */
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 1/7] media: platform: amd: Introduce amd isp4 capture driver
2025-12-22 9:23 ` Sakari Ailus
@ 2026-01-06 8:30 ` Du, Bin
0 siblings, 0 replies; 46+ messages in thread
From: Du, Bin @ 2026-01-06 8:30 UTC (permalink / raw)
To: Sakari Ailus
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Svetoslav Stoilov, Mario Limonciello,
Alexey Zagorodnikov
Thanks Sarkari for your review, apologies for my delayed response
because I wanted to respect your holiday and avoid any interruption, I
hope this excuse won't appear too lame :)
On 12/22/2025 5:23 PM, Sakari Ailus wrote:
> Hi Bin,
>
> Thanks for the update. Please see my comments below.
>
> On Tue, Dec 16, 2025 at 05:13:20PM +0800, Bin Du wrote:
>> AMD isp4 capture is a v4l2 media device which implements media controller
>> interface. It has one sub-device (AMD ISP4 sub-device) endpoint which can
>> be connected to a remote CSI2 TX endpoint. It supports only one physical
>> interface for now. Also add ISP4 driver related entry info into the
>> MAINTAINERS file
>>
>> Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
>> Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
>> Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
>> Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
>> Signed-off-by: Bin Du <Bin.Du@amd.com>
>> Reviewed-by: Mario Limonciello (AMD) <superm1@kernel.org>
>> Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
>> ---
>> MAINTAINERS | 13 +++
>> drivers/media/platform/Kconfig | 1 +
>> drivers/media/platform/Makefile | 1 +
>> drivers/media/platform/amd/Kconfig | 3 +
>> drivers/media/platform/amd/Makefile | 3 +
>> drivers/media/platform/amd/isp4/Kconfig | 14 +++
>> drivers/media/platform/amd/isp4/Makefile | 6 ++
>> drivers/media/platform/amd/isp4/isp4.c | 132 +++++++++++++++++++++++
>> drivers/media/platform/amd/isp4/isp4.h | 17 +++
>> 9 files changed, 190 insertions(+)
>> create mode 100644 drivers/media/platform/amd/Kconfig
>> create mode 100644 drivers/media/platform/amd/Makefile
>> create mode 100644 drivers/media/platform/amd/isp4/Kconfig
>> create mode 100644 drivers/media/platform/amd/isp4/Makefile
>> create mode 100644 drivers/media/platform/amd/isp4/isp4.c
>> create mode 100644 drivers/media/platform/amd/isp4/isp4.h
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index cd7ff55b5d32..3640a1e3262c 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -1133,6 +1133,19 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/iommu/linux.git
>> F: drivers/iommu/amd/
>> F: include/linux/amd-iommu.h
>>
>> +AMD ISP4 DRIVER
>> +M: Bin Du <bin.du@amd.com>
>> +M: Nirujogi Pratap <pratap.nirujogi@amd.com>
>> +L: linux-media@vger.kernel.org
>> +S: Maintained
>> +T: git git://linuxtv.org/media.git
>> +F: drivers/media/platform/amd/Kconfig
>> +F: drivers/media/platform/amd/Makefile
>> +F: drivers/media/platform/amd/isp4/Kconfig
>> +F: drivers/media/platform/amd/isp4/Makefile
>> +F: drivers/media/platform/amd/isp4/isp4.c
>> +F: drivers/media/platform/amd/isp4/isp4.h
>> +
>> AMD KFD
>> M: Felix Kuehling <Felix.Kuehling@amd.com>
>> L: amd-gfx@lists.freedesktop.org
>> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
>> index 9287faafdce5..772c70665510 100644
>> --- a/drivers/media/platform/Kconfig
>> +++ b/drivers/media/platform/Kconfig
>> @@ -63,6 +63,7 @@ config VIDEO_MUX
>>
>> # Platform drivers - Please keep it alphabetically sorted
>> source "drivers/media/platform/allegro-dvt/Kconfig"
>> +source "drivers/media/platform/amd/Kconfig"
>> source "drivers/media/platform/amlogic/Kconfig"
>> source "drivers/media/platform/amphion/Kconfig"
>> source "drivers/media/platform/aspeed/Kconfig"
>> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
>> index 6fd7db0541c7..b207bd8d8022 100644
>> --- a/drivers/media/platform/Makefile
>> +++ b/drivers/media/platform/Makefile
>> @@ -6,6 +6,7 @@
>> # Place here, alphabetically sorted by directory
>> # (e. g. LC_ALL=C sort Makefile)
>> obj-y += allegro-dvt/
>> +obj-y += amd/
>> obj-y += amlogic/
>> obj-y += amphion/
>> obj-y += aspeed/
>> diff --git a/drivers/media/platform/amd/Kconfig b/drivers/media/platform/amd/Kconfig
>> new file mode 100644
>> index 000000000000..25af49f246b2
>> --- /dev/null
>> +++ b/drivers/media/platform/amd/Kconfig
>> @@ -0,0 +1,3 @@
>> +# SPDX-License-Identifier: GPL-2.0+
>> +
>> +source "drivers/media/platform/amd/isp4/Kconfig"
>> diff --git a/drivers/media/platform/amd/Makefile b/drivers/media/platform/amd/Makefile
>> new file mode 100644
>> index 000000000000..8bfc1955f22e
>> --- /dev/null
>> +++ b/drivers/media/platform/amd/Makefile
>> @@ -0,0 +1,3 @@
>> +# SPDX-License-Identifier: GPL-2.0+
>> +
>> +obj-y += isp4/
>> diff --git a/drivers/media/platform/amd/isp4/Kconfig b/drivers/media/platform/amd/isp4/Kconfig
>> new file mode 100644
>> index 000000000000..d4e4ad436600
>> --- /dev/null
>> +++ b/drivers/media/platform/amd/isp4/Kconfig
>> @@ -0,0 +1,14 @@
>> +# SPDX-License-Identifier: GPL-2.0+
>> +
>> +config AMD_ISP4
>
> We've used "VIDEO_" prefix for V4L2 drivers.
>
Yes, will change it to VIDEO_AMD_ISP4 to meet the convention.
>> + tristate "AMD ISP4 and camera driver"
>> + depends on DRM_AMD_ISP && VIDEO_DEV
>> + select VIDEOBUF2_CORE
>> + select VIDEOBUF2_MEMOPS
>> + select VIDEOBUF2_V4L2
>> + select VIDEO_V4L2_SUBDEV_API
>> + help
>> + This is support for AMD ISP4 and camera subsystem driver.
>> + Say Y here to enable the ISP4 and camera device for video capture.
>> + To compile this driver as a module, choose M here. The module will
>> + be called amd_capture.
>
> How about "amd_isp4"? That would be aligned with the file names as well as
> the Kconfig option.
>
Yes, good suggestion. That makes more sense. Will modify the makefile as
well.
>> diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
>> new file mode 100644
>> index 000000000000..de0092dad26f
>> --- /dev/null
>> +++ b/drivers/media/platform/amd/isp4/Makefile
>> @@ -0,0 +1,6 @@
>> +# SPDX-License-Identifier: GPL-2.0+
>> +#
>> +# Copyright (C) 2025 Advanced Micro Devices, Inc.
>> +
>> +obj-$(CONFIG_AMD_ISP4) += amd_capture.o
>> +amd_capture-objs := isp4.o
>> diff --git a/drivers/media/platform/amd/isp4/isp4.c b/drivers/media/platform/amd/isp4/isp4.c
>> new file mode 100644
>> index 000000000000..ad95e7f89189
>> --- /dev/null
>> +++ b/drivers/media/platform/amd/isp4/isp4.c
>> @@ -0,0 +1,132 @@
>> +// SPDX-License-Identifier: GPL-2.0+
>> +/*
>> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
>> + */
>> +
>> +#include <linux/pm_runtime.h>
>> +#include <linux/vmalloc.h>
>> +#include <media/v4l2-ioctl.h>
>> +
>> +#include "isp4.h"
>> +
>> +#define VIDEO_BUF_NUM 5
>> +
>> +#define ISP4_DRV_NAME "amd_isp_capture"
>> +
>> +static const struct {
>> + const char *name;
>> + u32 status_mask;
>> + u32 en_mask;
>> + u32 ack_mask;
>> + u32 rb_int_num;
>> +} isp4_irq[] = {
>> + /* The IRQ order is aligned with the isp4_subdev.fw_resp_thread order */
>> + {
>> + .name = "isp_irq_global",
>> + .rb_int_num = 4, /* ISP_4_1__SRCID__ISP_RINGBUFFER_WPT12 */
>> + },
>> + {
>> + .name = "isp_irq_stream1",
>> + .rb_int_num = 0, /* ISP_4_1__SRCID__ISP_RINGBUFFER_WPT9 */
>> + },
>> +};
>> +
>> +static irqreturn_t isp4_irq_handler(int irq, void *arg)
>> +{
>> + return IRQ_HANDLED;
>> +}
>> +
>> +static int isp4_capture_probe(struct platform_device *pdev)
>> +{
>> + struct device *dev = &pdev->dev;
>> + int irq[ARRAY_SIZE(isp4_irq)];
>> + struct isp4_device *isp_dev;
>> + size_t i;
>> + int ret;
>> +
>> + isp_dev = devm_kzalloc(dev, sizeof(*isp_dev), GFP_KERNEL);
>> + if (!isp_dev)
>> + return -ENOMEM;
>> +
>> + dev->init_name = ISP4_DRV_NAME;
>> +
>> + for (i = 0; i < ARRAY_SIZE(isp4_irq); i++) {
>> + irq[i] = platform_get_irq(pdev, isp4_irq[i].rb_int_num);
>> + if (irq[i] < 0)
>> + return dev_err_probe(dev, irq[i], "fail to get irq %d\n",
>> + isp4_irq[i].rb_int_num);
>> +
>> + ret = devm_request_irq(dev, irq[i], isp4_irq_handler,
>> + IRQF_NO_AUTOEN, isp4_irq[i].name, dev);
>> + if (ret)
>> + return dev_err_probe(dev, ret, "fail to req irq %d\n", irq[i]);
>> + }
>> +
>> + /* Link the media device within the v4l2_device */
>> + isp_dev->v4l2_dev.mdev = &isp_dev->mdev;
>> +
>> + /* Initialize media device */
>> + strscpy(isp_dev->mdev.model, "amd_isp41_mdev", sizeof(isp_dev->mdev.model));
>> + snprintf(isp_dev->mdev.bus_info, sizeof(isp_dev->mdev.bus_info),
>> + "platform:%s", ISP4_DRV_NAME);
>
> No need to do this explicitly.
>
Yes, it is done inside media_device_init, will remove the redundancy here.
>> + isp_dev->mdev.dev = dev;
>> + media_device_init(&isp_dev->mdev);
>> +
>> + /* register v4l2 device */
>
> Please drop this comment, it's not informational.
>
Sure, will do that.
>> + snprintf(isp_dev->v4l2_dev.name, sizeof(isp_dev->v4l2_dev.name),
>> + "AMD-V4L2-ROOT");
>> + ret = v4l2_device_register(dev, &isp_dev->v4l2_dev);
>> + if (ret) {
>> + dev_err_probe(dev, ret, "fail register v4l2 device\n");
>> + goto err_clean_media;
>> + }
>> +
>> + pm_runtime_set_suspended(dev);
>> + pm_runtime_enable(dev);
>> + ret = media_device_register(&isp_dev->mdev);
>> + if (ret) {
>> + dev_err_probe(dev, ret, "fail to register media device\n");
>> + goto err_isp4_deinit;
>> + }
>> +
>> + platform_set_drvdata(pdev, isp_dev);
>> +
>> + return 0;
>> +
>> +err_isp4_deinit:
>> + pm_runtime_disable(dev);
>> + v4l2_device_unregister(&isp_dev->v4l2_dev);
>> +err_clean_media:
>> + media_device_cleanup(&isp_dev->mdev);
>> +
>> + return ret;
>> +}
>> +
>> +static void isp4_capture_remove(struct platform_device *pdev)
>> +{
>> + struct isp4_device *isp_dev = platform_get_drvdata(pdev);
>> + struct device *dev = &pdev->dev;
>> +
>> + media_device_unregister(&isp_dev->mdev);
>> + pm_runtime_disable(dev);
>> + v4l2_device_unregister(&isp_dev->v4l2_dev);
>> + media_device_cleanup(&isp_dev->mdev);
>> +}
>> +
>> +static struct platform_driver isp4_capture_drv = {
>> + .probe = isp4_capture_probe,
>> + .remove = isp4_capture_remove,
>> + .driver = {
>> + .name = ISP4_DRV_NAME,
>> + }
>> +};
>> +
>> +module_platform_driver(isp4_capture_drv);
>> +
>> +MODULE_ALIAS("platform:" ISP4_DRV_NAME);
>> +MODULE_IMPORT_NS("DMA_BUF");
>> +
>> +MODULE_DESCRIPTION("AMD ISP4 Driver");
>> +MODULE_AUTHOR("Bin Du <bin.du@amd.com>");
>> +MODULE_AUTHOR("Pratap Nirujogi <pratap.nirujogi@amd.com>");
>> +MODULE_LICENSE("GPL");
>> diff --git a/drivers/media/platform/amd/isp4/isp4.h b/drivers/media/platform/amd/isp4/isp4.h
>> new file mode 100644
>> index 000000000000..7f2db0dfa2d9
>> --- /dev/null
>> +++ b/drivers/media/platform/amd/isp4/isp4.h
>> @@ -0,0 +1,17 @@
>> +/* SPDX-License-Identifier: GPL-2.0+ */
>> +/*
>> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
>> + */
>> +
>> +#ifndef _ISP4_H_
>> +#define _ISP4_H_
>> +
>> +#include <media/v4l2-device.h>
>> +#include <media/videobuf2-memops.h>
>> +
>> +struct isp4_device {
>> + struct v4l2_device v4l2_dev;
>> + struct media_device mdev;
>> +};
>> +
>> +#endif /* _ISP4_H_ */
>
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread
* [PATCH v7 2/7] media: platform: amd: low level support for isp4 firmware
2025-12-16 9:13 [PATCH v7 0/7] Add AMD ISP4 driver Bin Du
2025-12-16 9:13 ` [PATCH v7 1/7] media: platform: amd: Introduce amd isp4 capture driver Bin Du
@ 2025-12-16 9:13 ` Bin Du
2025-12-16 9:13 ` [PATCH v7 3/7] media: platform: amd: Add isp4 fw and hw interface Bin Du
` (7 subsequent siblings)
9 siblings, 0 replies; 46+ messages in thread
From: Bin Du @ 2025-12-16 9:13 UTC (permalink / raw)
To: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
sultan
Cc: pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, bin.du, Bin Du, Svetoslav Stoilov,
Alexey Zagorodnikov
Low level functions for accessing the registers and mapping to their
ranges. This change also includes register definitions for ring buffer
used to communicate with ISP Firmware. Ring buffer is the communication
interface between driver and ISP Firmware. Command and responses are
exchanged through the ring buffer.
Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
Signed-off-by: Bin Du <Bin.Du@amd.com>
Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
---
MAINTAINERS | 1 +
drivers/media/platform/amd/isp4/isp4_hw_reg.h | 124 ++++++++++++++++++
2 files changed, 125 insertions(+)
create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 3640a1e3262c..7aa17c7e71d6 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1145,6 +1145,7 @@ F: drivers/media/platform/amd/isp4/Kconfig
F: drivers/media/platform/amd/isp4/Makefile
F: drivers/media/platform/amd/isp4/isp4.c
F: drivers/media/platform/amd/isp4/isp4.h
+F: drivers/media/platform/amd/isp4/isp4_hw_reg.h
AMD KFD
M: Felix Kuehling <Felix.Kuehling@amd.com>
diff --git a/drivers/media/platform/amd/isp4/isp4_hw_reg.h b/drivers/media/platform/amd/isp4/isp4_hw_reg.h
new file mode 100644
index 000000000000..09c76f75c5ee
--- /dev/null
+++ b/drivers/media/platform/amd/isp4/isp4_hw_reg.h
@@ -0,0 +1,124 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (C) 2025 Advanced Micro Devices, Inc.
+ */
+
+#ifndef _ISP4_HW_REG_H_
+#define _ISP4_HW_REG_H_
+
+#include <linux/io.h>
+
+#define ISP_SOFT_RESET 0x62000
+#define ISP_SYS_INT0_EN 0x62010
+#define ISP_SYS_INT0_STATUS 0x62014
+#define ISP_SYS_INT0_ACK 0x62018
+#define ISP_CCPU_CNTL 0x62054
+#define ISP_STATUS 0x62058
+#define ISP_LOG_RB_BASE_LO0 0x62148
+#define ISP_LOG_RB_BASE_HI0 0x6214c
+#define ISP_LOG_RB_SIZE0 0x62150
+#define ISP_LOG_RB_RPTR0 0x62154
+#define ISP_LOG_RB_WPTR0 0x62158
+#define ISP_RB_BASE_LO1 0x62170
+#define ISP_RB_BASE_HI1 0x62174
+#define ISP_RB_SIZE1 0x62178
+#define ISP_RB_RPTR1 0x6217c
+#define ISP_RB_WPTR1 0x62180
+#define ISP_RB_BASE_LO2 0x62184
+#define ISP_RB_BASE_HI2 0x62188
+#define ISP_RB_SIZE2 0x6218c
+#define ISP_RB_RPTR2 0x62190
+#define ISP_RB_WPTR2 0x62194
+#define ISP_RB_BASE_LO3 0x62198
+#define ISP_RB_BASE_HI3 0x6219c
+#define ISP_RB_SIZE3 0x621a0
+#define ISP_RB_RPTR3 0x621a4
+#define ISP_RB_WPTR3 0x621a8
+#define ISP_RB_BASE_LO4 0x621ac
+#define ISP_RB_BASE_HI4 0x621b0
+#define ISP_RB_SIZE4 0x621b4
+#define ISP_RB_RPTR4 0x621b8
+#define ISP_RB_WPTR4 0x621bc
+#define ISP_RB_BASE_LO5 0x621c0
+#define ISP_RB_BASE_HI5 0x621c4
+#define ISP_RB_SIZE5 0x621c8
+#define ISP_RB_RPTR5 0x621cc
+#define ISP_RB_WPTR5 0x621d0
+#define ISP_RB_BASE_LO6 0x621d4
+#define ISP_RB_BASE_HI6 0x621d8
+#define ISP_RB_SIZE6 0x621dc
+#define ISP_RB_RPTR6 0x621e0
+#define ISP_RB_WPTR6 0x621e4
+#define ISP_RB_BASE_LO7 0x621e8
+#define ISP_RB_BASE_HI7 0x621ec
+#define ISP_RB_SIZE7 0x621f0
+#define ISP_RB_RPTR7 0x621f4
+#define ISP_RB_WPTR7 0x621f8
+#define ISP_RB_BASE_LO8 0x621fc
+#define ISP_RB_BASE_HI8 0x62200
+#define ISP_RB_SIZE8 0x62204
+#define ISP_RB_RPTR8 0x62208
+#define ISP_RB_WPTR8 0x6220c
+#define ISP_RB_BASE_LO9 0x62210
+#define ISP_RB_BASE_HI9 0x62214
+#define ISP_RB_SIZE9 0x62218
+#define ISP_RB_RPTR9 0x6221c
+#define ISP_RB_WPTR9 0x62220
+#define ISP_RB_BASE_LO10 0x62224
+#define ISP_RB_BASE_HI10 0x62228
+#define ISP_RB_SIZE10 0x6222c
+#define ISP_RB_RPTR10 0x62230
+#define ISP_RB_WPTR10 0x62234
+#define ISP_RB_BASE_LO11 0x62238
+#define ISP_RB_BASE_HI11 0x6223c
+#define ISP_RB_SIZE11 0x62240
+#define ISP_RB_RPTR11 0x62244
+#define ISP_RB_WPTR11 0x62248
+#define ISP_RB_BASE_LO12 0x6224c
+#define ISP_RB_BASE_HI12 0x62250
+#define ISP_RB_SIZE12 0x62254
+#define ISP_RB_RPTR12 0x62258
+#define ISP_RB_WPTR12 0x6225c
+
+#define ISP_POWER_STATUS 0x60000
+
+/* ISP_SOFT_RESET */
+#define ISP_SOFT_RESET__CCPU_SOFT_RESET_MASK 0x00000001UL
+
+/* ISP_CCPU_CNTL */
+#define ISP_CCPU_CNTL__CCPU_HOST_SOFT_RST_MASK 0x00040000UL
+
+/* ISP_STATUS */
+#define ISP_STATUS__CCPU_REPORT_MASK 0x000000feUL
+
+/* ISP_SYS_INT0_STATUS */
+#define ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT9_INT_MASK 0x00010000UL
+#define ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT10_INT_MASK 0x00040000UL
+#define ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT11_INT_MASK 0x00100000UL
+#define ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT12_INT_MASK 0x00400000UL
+
+/* ISP_SYS_INT0_EN */
+#define ISP_SYS_INT0_EN__SYS_INT_RINGBUFFER_WPT9_EN_MASK 0x00010000UL
+#define ISP_SYS_INT0_EN__SYS_INT_RINGBUFFER_WPT10_EN_MASK 0x00040000UL
+#define ISP_SYS_INT0_EN__SYS_INT_RINGBUFFER_WPT11_EN_MASK 0x00100000UL
+#define ISP_SYS_INT0_EN__SYS_INT_RINGBUFFER_WPT12_EN_MASK 0x00400000UL
+
+/* ISP_SYS_INT0_ACK */
+#define ISP_SYS_INT0_ACK__SYS_INT_RINGBUFFER_WPT9_ACK_MASK 0x00010000UL
+#define ISP_SYS_INT0_ACK__SYS_INT_RINGBUFFER_WPT10_ACK_MASK 0x00040000UL
+#define ISP_SYS_INT0_ACK__SYS_INT_RINGBUFFER_WPT11_ACK_MASK 0x00100000UL
+#define ISP_SYS_INT0_ACK__SYS_INT_RINGBUFFER_WPT12_ACK_MASK 0x00400000UL
+
+/* Helper functions for reading isp registers */
+static inline u32 isp4hw_rreg(void __iomem *base, u32 reg)
+{
+ return readl(base + reg);
+}
+
+/* Helper functions for writing isp registers */
+static inline void isp4hw_wreg(void __iomem *base, u32 reg, u32 val)
+{
+ return writel(val, base + reg);
+}
+
+#endif /* _ISP4_HW_REG_H_ */
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread* [PATCH v7 3/7] media: platform: amd: Add isp4 fw and hw interface
2025-12-16 9:13 [PATCH v7 0/7] Add AMD ISP4 driver Bin Du
2025-12-16 9:13 ` [PATCH v7 1/7] media: platform: amd: Introduce amd isp4 capture driver Bin Du
2025-12-16 9:13 ` [PATCH v7 2/7] media: platform: amd: low level support for isp4 firmware Bin Du
@ 2025-12-16 9:13 ` Bin Du
2025-12-22 9:37 ` Sakari Ailus
2025-12-16 9:13 ` [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added Bin Du
` (6 subsequent siblings)
9 siblings, 1 reply; 46+ messages in thread
From: Bin Du @ 2025-12-16 9:13 UTC (permalink / raw)
To: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
sultan
Cc: pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, bin.du, Bin Du, Svetoslav Stoilov,
Alexey Zagorodnikov
ISP firmware controls ISP HW pipeline using dedicated embedded processor
called ccpu. The communication between ISP FW and driver is using commands
and response messages sent through the ring buffer. Command buffers support
either global setting that is not specific to the steam and support stream
specific parameters. Response buffers contain ISP FW notification
information such as frame buffer done and command done. IRQ is used for
receiving response buffer from ISP firmware, which is handled in the main
isp4 media device. ISP ccpu is booted up through the firmware loading
helper function prior to stream start. Memory used for command buffer and
response buffer needs to be allocated from amdgpu buffer manager because
isp4 is a child device of amdgpu.
Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
Signed-off-by: Bin Du <Bin.Du@amd.com>
Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
---
MAINTAINERS | 3 +
drivers/media/platform/amd/isp4/Makefile | 3 +-
.../platform/amd/isp4/isp4_fw_cmd_resp.h | 314 +++++++
.../media/platform/amd/isp4/isp4_interface.c | 786 ++++++++++++++++++
.../media/platform/amd/isp4/isp4_interface.h | 141 ++++
5 files changed, 1246 insertions(+), 1 deletion(-)
create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c
create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 7aa17c7e71d6..cccae369c876 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1145,7 +1145,10 @@ F: drivers/media/platform/amd/isp4/Kconfig
F: drivers/media/platform/amd/isp4/Makefile
F: drivers/media/platform/amd/isp4/isp4.c
F: drivers/media/platform/amd/isp4/isp4.h
+F: drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
F: drivers/media/platform/amd/isp4/isp4_hw_reg.h
+F: drivers/media/platform/amd/isp4/isp4_interface.c
+F: drivers/media/platform/amd/isp4/isp4_interface.h
AMD KFD
M: Felix Kuehling <Felix.Kuehling@amd.com>
diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
index de0092dad26f..a2a5bf98e912 100644
--- a/drivers/media/platform/amd/isp4/Makefile
+++ b/drivers/media/platform/amd/isp4/Makefile
@@ -3,4 +3,5 @@
# Copyright (C) 2025 Advanced Micro Devices, Inc.
obj-$(CONFIG_AMD_ISP4) += amd_capture.o
-amd_capture-objs := isp4.o
+amd_capture-objs := isp4.o \
+ isp4_interface.o
diff --git a/drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h b/drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
new file mode 100644
index 000000000000..d571b3873edb
--- /dev/null
+++ b/drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
@@ -0,0 +1,314 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (C) 2025 Advanced Micro Devices, Inc.
+ */
+
+#ifndef _ISP4_CMD_RESP_H_
+#define _ISP4_CMD_RESP_H_
+
+/*
+ * @brief Host and Firmware command & response channel.
+ * Two types of command/response channel.
+ * Type Global Command has one command/response channel.
+ * Type Stream Command has one command/response channel.
+ *----------- ------------
+ *| | --------------------------- | |
+ *| | ---->| Global Command |----> | |
+ *| | --------------------------- | |
+ *| | | |
+ *| | | |
+ *| | --------------------------- | |
+ *| | ---->| Stream Command |----> | |
+ *| | --------------------------- | |
+ *| | | |
+ *| | | |
+ *| | | |
+ *| HOST | | Firmware |
+ *| | | |
+ *| | | |
+ *| | -------------------------- | |
+ *| | <----| Global Response |<---- | |
+ *| | -------------------------- | |
+ *| | | |
+ *| | | |
+ *| | -------------------------- | |
+ *| | <----| Stream Response |<---- | |
+ *| | -------------------------- | |
+ *| | | |
+ *| | | |
+ *----------- ------------
+ */
+
+/*
+ * @brief command ID format
+ * cmd_id is in the format of following type:
+ * type: indicate command type, global/stream commands.
+ * group: indicate the command group.
+ * id: A unique command identification in one type and group.
+ * |<-Bit31 ~ Bit24->|<-Bit23 ~ Bit16->|<-Bit15 ~ Bit0->|
+ * | type | group | id |
+ */
+
+#define CMD_TYPE_SHIFT 24
+#define CMD_GROUP_SHIFT 16
+#define CMD_TYPE_STREAM_CTRL (0x2U << CMD_TYPE_SHIFT)
+
+#define CMD_GROUP_STREAM_CTRL (0x1U << CMD_GROUP_SHIFT)
+#define CMD_GROUP_STREAM_BUFFER (0x4U << CMD_GROUP_SHIFT)
+
+/* Stream Command */
+#define CMD_ID_SET_STREAM_CONFIG (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x1)
+#define CMD_ID_SET_OUT_CHAN_PROP (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x3)
+#define CMD_ID_ENABLE_OUT_CHAN (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x5)
+#define CMD_ID_START_STREAM (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x7)
+#define CMD_ID_STOP_STREAM (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x8)
+
+/* Stream Buffer Command */
+#define CMD_ID_SEND_BUFFER (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_BUFFER | 0x1)
+
+/*
+ * @brief response ID format
+ * resp_id is in the format of following type:
+ * type: indicate command type, global/stream commands.
+ * group: indicate the command group.
+ * id: A unique command identification in one type and group.
+ * |<-Bit31 ~ Bit24->|<-Bit23 ~ Bit16->|<-Bit15 ~ Bit0->|
+ * | type | group | id |
+ */
+
+#define RESP_GROUP_SHIFT 16
+#define RESP_GROUP_MASK (0xff << RESP_GROUP_SHIFT)
+
+#define GET_RESP_GROUP_VALUE(resp_id) (((resp_id) & RESP_GROUP_MASK) >> RESP_GROUP_SHIFT)
+#define GET_RESP_ID_VALUE(resp_id) ((resp_id) & 0xffff)
+
+#define RESP_GROUP_GENERAL (0x1 << RESP_GROUP_SHIFT)
+#define RESP_GROUP_NOTIFICATION (0x3 << RESP_GROUP_SHIFT)
+
+/* General Response */
+#define RESP_ID_CMD_DONE (RESP_GROUP_GENERAL | 0x1)
+
+/* Notification */
+#define RESP_ID_NOTI_FRAME_DONE (RESP_GROUP_NOTIFICATION | 0x1)
+
+#define CMD_STATUS_SUCCESS 0
+#define CMD_STATUS_FAIL 1
+#define CMD_STATUS_SKIPPED 2
+
+#define ADDR_SPACE_TYPE_GPU_VA 4
+
+#define FW_MEMORY_POOL_SIZE (100 * 1024 * 1024)
+
+/*
+ * standard ISP mipicsi=>isp
+ */
+#define MIPI0_ISP_PIPELINE_ID 0x5f91
+
+enum isp4fw_sensor_id {
+ SENSOR_ID_ON_MIPI0 = 0, /* Sensor id for ISP input from MIPI port 0 */
+};
+
+enum isp4fw_stream_id {
+ STREAM_ID_INVALID = -1, /* STREAM_ID_INVALID. */
+ STREAM_ID_1 = 0, /* STREAM_ID_1. */
+ STREAM_ID_2 = 1, /* STREAM_ID_2. */
+ STREAM_ID_3 = 2, /* STREAM_ID_3. */
+ STREAM_ID_MAXIMUM /* STREAM_ID_MAXIMUM. */
+};
+
+enum isp4fw_image_format {
+ IMAGE_FORMAT_NV12 = 1, /* 4:2:0,semi-planar, 8-bit */
+ IMAGE_FORMAT_YUV422INTERLEAVED = 7, /* interleave, 4:2:2, 8-bit */
+};
+
+enum isp4fw_pipe_out_ch {
+ ISP_PIPE_OUT_CH_PREVIEW = 0,
+};
+
+enum isp4fw_yuv_range {
+ ISP_YUV_RANGE_FULL = 0, /* YUV value range in 0~255 */
+ ISP_YUV_RANGE_NARROW = 1, /* YUV value range in 16~235 */
+ ISP_YUV_RANGE_MAX
+};
+
+enum isp4fw_buffer_type {
+ BUFFER_TYPE_PREVIEW = 8,
+ BUFFER_TYPE_META_INFO = 10,
+ BUFFER_TYPE_MEM_POOL = 15,
+};
+
+enum isp4fw_buffer_status {
+ BUFFER_STATUS_INVALID, /* The buffer is INVALID */
+ BUFFER_STATUS_SKIPPED, /* The buffer is not filled with image data */
+ BUFFER_STATUS_EXIST, /* The buffer is exist and waiting for filled */
+ BUFFER_STATUS_DONE, /* The buffer is filled with image data */
+ BUFFER_STATUS_LACK, /* The buffer is unavailable */
+ BUFFER_STATUS_DIRTY, /* The buffer is dirty, probably caused by
+ * LMI leakage
+ */
+ BUFFER_STATUS_MAX /* The buffer STATUS_MAX */
+};
+
+enum isp4fw_buffer_source {
+ /* The buffer is from the stream buffer queue */
+ BUFFER_SOURCE_STREAM,
+};
+
+struct isp4fw_error_code {
+ u32 code1;
+ u32 code2;
+ u32 code3;
+ u32 code4;
+ u32 code5;
+};
+
+/*
+ * Command Structure for FW
+ */
+
+struct isp4fw_cmd {
+ u32 cmd_seq_num;
+ u32 cmd_id;
+ u32 cmd_param[12];
+ u16 cmd_stream_id;
+ u8 cmd_silent_resp;
+ u8 reserved;
+ u32 cmd_check_sum;
+};
+
+struct isp4fw_resp_cmd_done {
+ /*
+ * The host2fw command seqNum.
+ * To indicate which command this response refers to.
+ */
+ u32 cmd_seq_num;
+ /* The host2fw command id for host double check. */
+ u32 cmd_id;
+ /*
+ * Indicate the command process status.
+ * 0 means success. 1 means fail. 2 means skipped
+ */
+ u16 cmd_status;
+ /*
+ * If the cmd_status is 1, that means the command is processed fail,
+ * host can check the isp4fw_error_code to get the details
+ * error information
+ */
+ u16 isp4fw_error_code;
+ /* The response payload will be in different struct type */
+ /* according to different cmd done response. */
+ u8 payload[36];
+};
+
+struct isp4fw_resp_param_package {
+ u32 package_addr_lo; /* The low 32 bit addr of the pkg address. */
+ u32 package_addr_hi; /* The high 32 bit addr of the pkg address. */
+ u32 package_size; /* The total pkg size in bytes. */
+ u32 package_check_sum; /* The byte sum of the pkg. */
+};
+
+struct isp4fw_resp {
+ u32 resp_seq_num;
+ u32 resp_id;
+ union {
+ struct isp4fw_resp_cmd_done cmd_done;
+ struct isp4fw_resp_param_package frame_done;
+ u32 resp_param[12];
+ } param;
+ u8 reserved[4];
+ u32 resp_check_sum;
+};
+
+struct isp4fw_mipi_pipe_path_cfg {
+ u32 b_enable;
+ enum isp4fw_sensor_id isp4fw_sensor_id;
+};
+
+struct isp4fw_isp_pipe_path_cfg {
+ u32 isp_pipe_id; /* pipe ids for pipeline construction */
+};
+
+struct isp4fw_isp_stream_cfg {
+ /* Isp mipi path */
+ struct isp4fw_mipi_pipe_path_cfg mipi_pipe_path_cfg;
+ /* Isp pipe path */
+ struct isp4fw_isp_pipe_path_cfg isp_pipe_path_cfg;
+ /* enable TNR */
+ u32 b_enable_tnr;
+ /*
+ * number of frame rta per-processing,
+ * set to 0 to use fw default value
+ */
+ u32 rta_frames_per_proc;
+};
+
+struct isp4fw_image_prop {
+ enum isp4fw_image_format image_format; /* Image format */
+ u32 width; /* Width */
+ u32 height; /* Height */
+ u32 luma_pitch; /* Luma pitch */
+ u32 chroma_pitch; /* Chrom pitch */
+ enum isp4fw_yuv_range yuv_range; /* YUV value range */
+};
+
+struct isp4fw_buffer {
+ /* A check num for debug usage, host need to */
+ /* set the buf_tags to different number */
+ u32 buf_tags;
+ union {
+ u32 value;
+ struct {
+ u32 space : 16;
+ u32 vmid : 16;
+ } bit;
+ } vmid_space;
+ u32 buf_base_a_lo; /* Low address of buffer A */
+ u32 buf_base_a_hi; /* High address of buffer A */
+ u32 buf_size_a; /* Buffer size of buffer A */
+
+ u32 buf_base_b_lo; /* Low address of buffer B */
+ u32 buf_base_b_hi; /* High address of buffer B */
+ u32 buf_size_b; /* Buffer size of buffer B */
+
+ u32 buf_base_c_lo; /* Low address of buffer C */
+ u32 buf_base_c_hi; /* High address of buffer C */
+ u32 buf_size_c; /* Buffer size of buffer C */
+};
+
+struct isp4fw_buffer_meta_info {
+ u32 enabled; /* enabled flag */
+ enum isp4fw_buffer_status status; /* BufferStatus */
+ struct isp4fw_error_code err; /* err code */
+ enum isp4fw_buffer_source source; /* BufferSource */
+ struct isp4fw_image_prop image_prop; /* image_prop */
+ struct isp4fw_buffer buffer; /* buffer */
+};
+
+struct isp4fw_meta_info {
+ u32 poc; /* frame id */
+ u32 fc_id; /* frame ctl id */
+ u32 time_stamp_lo; /* time_stamp_lo */
+ u32 time_stamp_hi; /* time_stamp_hi */
+ struct isp4fw_buffer_meta_info preview; /* preview BufferMetaInfo */
+};
+
+struct isp4fw_cmd_send_buffer {
+ enum isp4fw_buffer_type buffer_type; /* buffer Type */
+ struct isp4fw_buffer buffer; /* buffer info */
+};
+
+struct isp4fw_cmd_set_out_ch_prop {
+ enum isp4fw_pipe_out_ch ch; /* ISP pipe out channel */
+ struct isp4fw_image_prop image_prop; /* image property */
+};
+
+struct isp4fw_cmd_enable_out_ch {
+ enum isp4fw_pipe_out_ch ch; /* ISP pipe out channel */
+ u32 is_enable; /* If enable channel or not */
+};
+
+struct isp4fw_cmd_set_stream_cfg {
+ struct isp4fw_isp_stream_cfg stream_cfg; /* stream path config */
+};
+
+#endif /* _ISP4_CMD_RESP_H_ */
diff --git a/drivers/media/platform/amd/isp4/isp4_interface.c b/drivers/media/platform/amd/isp4/isp4_interface.c
new file mode 100644
index 000000000000..914a93f9652e
--- /dev/null
+++ b/drivers/media/platform/amd/isp4/isp4_interface.c
@@ -0,0 +1,786 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2025 Advanced Micro Devices, Inc.
+ */
+
+#include <linux/iopoll.h>
+
+#include "isp4_fw_cmd_resp.h"
+#include "isp4_hw_reg.h"
+#include "isp4_interface.h"
+
+#define ISP4IF_FW_RESP_RB_IRQ_EN_MASK \
+ (ISP_SYS_INT0_EN__SYS_INT_RINGBUFFER_WPT9_EN_MASK | \
+ ISP_SYS_INT0_EN__SYS_INT_RINGBUFFER_WPT12_EN_MASK)
+
+#define ISP4IF_FW_CMD_TIMEOUT (HZ / 2)
+
+struct isp4if_rb_config {
+ const char *name;
+ u32 index;
+ u32 reg_rptr;
+ u32 reg_wptr;
+ u32 reg_base_lo;
+ u32 reg_base_hi;
+ u32 reg_size;
+ u32 val_size;
+ u64 base_mc_addr;
+ void *base_sys_addr;
+};
+
+/* FW cmd ring buffer configuration */
+static struct isp4if_rb_config
+ isp4if_cmd_rb_config[ISP4IF_STREAM_ID_MAX] = {
+ {
+ .name = "CMD_RB_GBL0",
+ .index = 3,
+ .reg_rptr = ISP_RB_RPTR4,
+ .reg_wptr = ISP_RB_WPTR4,
+ .reg_base_lo = ISP_RB_BASE_LO4,
+ .reg_base_hi = ISP_RB_BASE_HI4,
+ .reg_size = ISP_RB_SIZE4,
+ },
+ {
+ .name = "CMD_RB_STR1",
+ .index = 0,
+ .reg_rptr = ISP_RB_RPTR1,
+ .reg_wptr = ISP_RB_WPTR1,
+ .reg_base_lo = ISP_RB_BASE_LO1,
+ .reg_base_hi = ISP_RB_BASE_HI1,
+ .reg_size = ISP_RB_SIZE1,
+ },
+ {
+ .name = "CMD_RB_STR2",
+ .index = 1,
+ .reg_rptr = ISP_RB_RPTR2,
+ .reg_wptr = ISP_RB_WPTR2,
+ .reg_base_lo = ISP_RB_BASE_LO2,
+ .reg_base_hi = ISP_RB_BASE_HI2,
+ .reg_size = ISP_RB_SIZE2,
+ },
+ {
+ .name = "CMD_RB_STR3",
+ .index = 2,
+ .reg_rptr = ISP_RB_RPTR3,
+ .reg_wptr = ISP_RB_WPTR3,
+ .reg_base_lo = ISP_RB_BASE_LO3,
+ .reg_base_hi = ISP_RB_BASE_HI3,
+ .reg_size = ISP_RB_SIZE3,
+ },
+};
+
+/* FW resp ring buffer configuration */
+static struct isp4if_rb_config
+ isp4if_resp_rb_config[ISP4IF_STREAM_ID_MAX] = {
+ {
+ .name = "RES_RB_GBL0",
+ .index = 3,
+ .reg_rptr = ISP_RB_RPTR12,
+ .reg_wptr = ISP_RB_WPTR12,
+ .reg_base_lo = ISP_RB_BASE_LO12,
+ .reg_base_hi = ISP_RB_BASE_HI12,
+ .reg_size = ISP_RB_SIZE12,
+ },
+ {
+ .name = "RES_RB_STR1",
+ .index = 0,
+ .reg_rptr = ISP_RB_RPTR9,
+ .reg_wptr = ISP_RB_WPTR9,
+ .reg_base_lo = ISP_RB_BASE_LO9,
+ .reg_base_hi = ISP_RB_BASE_HI9,
+ .reg_size = ISP_RB_SIZE9,
+ },
+ {
+ .name = "RES_RB_STR2",
+ .index = 1,
+ .reg_rptr = ISP_RB_RPTR10,
+ .reg_wptr = ISP_RB_WPTR10,
+ .reg_base_lo = ISP_RB_BASE_LO10,
+ .reg_base_hi = ISP_RB_BASE_HI10,
+ .reg_size = ISP_RB_SIZE10,
+ },
+ {
+ .name = "RES_RB_STR3",
+ .index = 2,
+ .reg_rptr = ISP_RB_RPTR11,
+ .reg_wptr = ISP_RB_WPTR11,
+ .reg_base_lo = ISP_RB_BASE_LO11,
+ .reg_base_hi = ISP_RB_BASE_HI11,
+ .reg_size = ISP_RB_SIZE11,
+ },
+};
+
+/* FW log ring buffer configuration */
+static struct isp4if_rb_config isp4if_log_rb_config = {
+ .name = "LOG_RB",
+ .index = 0,
+ .reg_rptr = ISP_LOG_RB_RPTR0,
+ .reg_wptr = ISP_LOG_RB_WPTR0,
+ .reg_base_lo = ISP_LOG_RB_BASE_LO0,
+ .reg_base_hi = ISP_LOG_RB_BASE_HI0,
+ .reg_size = ISP_LOG_RB_SIZE0,
+};
+
+static struct isp4if_gpu_mem_info *isp4if_gpu_mem_alloc(struct isp4_interface *ispif, u32 mem_size)
+{
+ struct isp4if_gpu_mem_info *mem_info;
+ struct device *dev = ispif->dev;
+ int ret;
+
+ mem_info = kmalloc(sizeof(*mem_info), GFP_KERNEL);
+ if (!mem_info)
+ return NULL;
+
+ mem_info->mem_size = mem_size;
+ ret = isp_kernel_buffer_alloc(dev, mem_info->mem_size, &mem_info->mem_handle,
+ &mem_info->gpu_mc_addr, &mem_info->sys_addr);
+ if (ret) {
+ kfree(mem_info);
+ return NULL;
+ }
+
+ return mem_info;
+}
+
+static void isp4if_gpu_mem_free(struct isp4_interface *ispif,
+ struct isp4if_gpu_mem_info **mem_info_ptr)
+{
+ struct isp4if_gpu_mem_info *mem_info = *mem_info_ptr;
+ struct device *dev = ispif->dev;
+
+ if (!mem_info) {
+ dev_err(dev, "invalid mem_info\n");
+ return;
+ }
+
+ *mem_info_ptr = NULL;
+ isp_kernel_buffer_free(&mem_info->mem_handle, &mem_info->gpu_mc_addr, &mem_info->sys_addr);
+ kfree(mem_info);
+}
+
+static void isp4if_dealloc_fw_gpumem(struct isp4_interface *ispif)
+{
+ int i;
+
+ isp4if_gpu_mem_free(ispif, &ispif->fw_mem_pool);
+ isp4if_gpu_mem_free(ispif, &ispif->fw_cmd_resp_buf);
+ isp4if_gpu_mem_free(ispif, &ispif->fw_log_buf);
+
+ for (i = 0; i < ISP4IF_MAX_STREAM_BUF_COUNT; i++)
+ isp4if_gpu_mem_free(ispif, &ispif->meta_info_buf[i]);
+}
+
+static int isp4if_alloc_fw_gpumem(struct isp4_interface *ispif)
+{
+ struct device *dev = ispif->dev;
+ int i;
+
+ ispif->fw_mem_pool = isp4if_gpu_mem_alloc(ispif, FW_MEMORY_POOL_SIZE);
+ if (!ispif->fw_mem_pool)
+ goto error_no_memory;
+
+ ispif->fw_cmd_resp_buf =
+ isp4if_gpu_mem_alloc(ispif, ISP4IF_RB_PMBMAP_MEM_SIZE);
+ if (!ispif->fw_cmd_resp_buf)
+ goto error_no_memory;
+
+ ispif->fw_log_buf =
+ isp4if_gpu_mem_alloc(ispif, ISP4IF_FW_LOG_RINGBUF_SIZE);
+ if (!ispif->fw_log_buf)
+ goto error_no_memory;
+
+ for (i = 0; i < ISP4IF_MAX_STREAM_BUF_COUNT; i++) {
+ ispif->meta_info_buf[i] =
+ isp4if_gpu_mem_alloc(ispif, ISP4IF_META_INFO_BUF_SIZE);
+ if (!ispif->meta_info_buf[i])
+ goto error_no_memory;
+ }
+
+ return 0;
+
+error_no_memory:
+ dev_err(dev, "failed to allocate gpu memory\n");
+ return -ENOMEM;
+}
+
+static u32 isp4if_compute_check_sum(const void *buf, size_t buf_size)
+{
+ const u8 *surplus_ptr;
+ const u32 *buffer;
+ u32 checksum = 0;
+ size_t i;
+
+ buffer = (const u32 *)buf;
+ for (i = 0; i < buf_size / sizeof(u32); i++)
+ checksum += buffer[i];
+
+ surplus_ptr = (const u8 *)&buffer[i];
+ /* add surplus data crc checksum */
+ for (i = 0; i < buf_size % sizeof(u32); i++)
+ checksum += surplus_ptr[i];
+
+ return checksum;
+}
+
+void isp4if_clear_cmdq(struct isp4_interface *ispif)
+{
+ struct isp4if_cmd_element *buf_node, *tmp_node;
+ LIST_HEAD(free_list);
+
+ scoped_guard(spinlock, &ispif->cmdq_lock)
+ list_splice_init(&ispif->cmdq, &free_list);
+
+ list_for_each_entry_safe(buf_node, tmp_node, &free_list, list)
+ kfree(buf_node);
+}
+
+static bool isp4if_is_cmdq_rb_full(struct isp4_interface *ispif, enum isp4if_stream_id stream)
+{
+ struct isp4if_rb_config *rb_config = &isp4if_cmd_rb_config[stream];
+ u32 rreg = rb_config->reg_rptr, wreg = rb_config->reg_wptr;
+ u32 len = rb_config->val_size;
+ u32 rd_ptr, wr_ptr;
+ u32 bytes_free;
+
+ rd_ptr = isp4hw_rreg(ispif->mmio, rreg);
+ wr_ptr = isp4hw_rreg(ispif->mmio, wreg);
+
+ /* Read and write pointers are equal, indicating the ringbuf is empty */
+ if (wr_ptr == rd_ptr)
+ return false;
+
+ if (wr_ptr > rd_ptr)
+ bytes_free = len - (wr_ptr - rd_ptr);
+ else
+ bytes_free = rd_ptr - wr_ptr;
+
+ /*
+ * Ignore one byte from the bytes free to prevent rd_ptr from equaling
+ * wr_ptr when the ringbuf is full, because rd_ptr == wr_ptr is supposed
+ * to indicate that the ringbuf is empty.
+ */
+ return bytes_free <= sizeof(struct isp4fw_cmd);
+}
+
+struct isp4if_cmd_element *isp4if_rm_cmd_from_cmdq(struct isp4_interface *ispif, u32 seq_num,
+ u32 cmd_id)
+{
+ struct isp4if_cmd_element *buf_node;
+
+ guard(spinlock)(&ispif->cmdq_lock);
+
+ list_for_each_entry(buf_node, &ispif->cmdq, list) {
+ if (buf_node->seq_num == seq_num && buf_node->cmd_id == cmd_id) {
+ list_del(&buf_node->list);
+ return buf_node;
+ }
+ }
+
+ return NULL;
+}
+
+/* Must check that isp4if_is_cmdq_rb_full() == false before calling */
+static int isp4if_insert_isp_fw_cmd(struct isp4_interface *ispif, enum isp4if_stream_id stream,
+ const struct isp4fw_cmd *cmd)
+{
+ struct isp4if_rb_config *rb_config = &isp4if_cmd_rb_config[stream];
+ u32 rreg = rb_config->reg_rptr, wreg = rb_config->reg_wptr;
+ void *mem_sys = rb_config->base_sys_addr;
+ const u32 cmd_sz = sizeof(*cmd);
+ struct device *dev = ispif->dev;
+ u32 len = rb_config->val_size;
+ const void *src = cmd;
+ u32 rd_ptr, wr_ptr;
+ u32 bytes_to_end;
+
+ rd_ptr = isp4hw_rreg(ispif->mmio, rreg);
+ wr_ptr = isp4hw_rreg(ispif->mmio, wreg);
+ if (rd_ptr >= len || wr_ptr >= len) {
+ dev_err(dev, "rb invalid: stream=%u, rd=%u, wr=%u, len=%u, cmd_sz=%u\n",
+ stream, rd_ptr, wr_ptr, len, cmd_sz);
+ return -EINVAL;
+ }
+
+ bytes_to_end = len - wr_ptr;
+ if (bytes_to_end >= cmd_sz) {
+ /* FW cmd is just a straight copy to the write pointer */
+ memcpy(mem_sys + wr_ptr, src, cmd_sz);
+ isp4hw_wreg(ispif->mmio, wreg, (wr_ptr + cmd_sz) % len);
+ } else {
+ /* FW cmd is split because the ringbuf needs to wrap around */
+ memcpy(mem_sys + wr_ptr, src, bytes_to_end);
+ memcpy(mem_sys, src + bytes_to_end, cmd_sz - bytes_to_end);
+ isp4hw_wreg(ispif->mmio, wreg, cmd_sz - bytes_to_end);
+ }
+
+ return 0;
+}
+
+static inline enum isp4if_stream_id isp4if_get_fw_stream(u32 cmd_id)
+{
+ return ISP4IF_STREAM_ID_1;
+}
+
+static int isp4if_send_fw_cmd(struct isp4_interface *ispif, u32 cmd_id, const void *package,
+ u32 package_size, bool sync)
+{
+ enum isp4if_stream_id stream = isp4if_get_fw_stream(cmd_id);
+ struct isp4if_cmd_element *ele = NULL;
+ struct device *dev = ispif->dev;
+ struct isp4fw_cmd cmd;
+ u32 seq_num;
+ int ret;
+
+ if (package_size > sizeof(cmd.cmd_param)) {
+ dev_err(dev, "fail pkgsize(%u)>%zu cmd:0x%x,stream %d\n",
+ package_size, sizeof(cmd.cmd_param), cmd_id, stream);
+ return -EINVAL;
+ }
+
+ /*
+ * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
+ * zeroed, since this is not guaranteed on all compilers.
+ */
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.cmd_id = cmd_id;
+ switch (stream) {
+ case ISP4IF_STREAM_ID_GLOBAL:
+ cmd.cmd_stream_id = STREAM_ID_INVALID;
+ break;
+ case ISP4IF_STREAM_ID_1:
+ cmd.cmd_stream_id = STREAM_ID_1;
+ break;
+ default:
+ dev_err(dev, "fail bad stream id %d\n", stream);
+ return -EINVAL;
+ }
+
+ /* Allocate the sync command object early and outside of the lock */
+ if (sync) {
+ ele = kmalloc(sizeof(*ele), GFP_KERNEL);
+ if (!ele)
+ return -ENOMEM;
+
+ /* Get two references: one for the resp thread, one for us */
+ atomic_set(&ele->refcnt, 2);
+ init_completion(&ele->cmd_done);
+ }
+
+ if (package && package_size)
+ memcpy(cmd.cmd_param, package, package_size);
+
+ scoped_guard(mutex, &ispif->isp4if_mutex) {
+ ret = read_poll_timeout(isp4if_is_cmdq_rb_full, ret, !ret, ISP4IF_RB_FULL_SLEEP_US,
+ ISP4IF_RB_FULL_TIMEOUT_US, false, ispif, stream);
+ if (ret) {
+ struct isp4if_rb_config *rb_config = &isp4if_resp_rb_config[stream];
+ u32 rd_ptr = isp4hw_rreg(ispif->mmio, rb_config->reg_rptr);
+ u32 wr_ptr = isp4hw_rreg(ispif->mmio, rb_config->reg_wptr);
+
+ dev_err(dev,
+ "failed to get free cmdq slot, stream (%d),rd %u, wr %u\n",
+ stream, rd_ptr, wr_ptr);
+ ret = -ETIMEDOUT;
+ goto free_ele;
+ }
+
+ seq_num = ispif->host2fw_seq_num++;
+ cmd.cmd_seq_num = seq_num;
+ cmd.cmd_check_sum = isp4if_compute_check_sum(&cmd, sizeof(cmd) - sizeof(u32));
+
+ /*
+ * only append the fw cmd to queue when its response needs to be waited for,
+ * currently there are only two such commands, disable channel and stop stream
+ * which are only sent after close camera
+ */
+ if (ele) {
+ ele->seq_num = seq_num;
+ ele->cmd_id = cmd_id;
+ scoped_guard(spinlock, &ispif->cmdq_lock)
+ list_add_tail(&ele->list, &ispif->cmdq);
+ }
+
+ ret = isp4if_insert_isp_fw_cmd(ispif, stream, &cmd);
+ if (ret) {
+ dev_err(dev, "fail for insert_isp_fw_cmd cmd_id (0x%08x)\n", cmd_id);
+ goto err_dequeue_ele;
+ }
+ }
+
+ if (ele) {
+ ret = wait_for_completion_timeout(&ele->cmd_done, ISP4IF_FW_CMD_TIMEOUT);
+ if (!ret) {
+ ret = -ETIMEDOUT;
+ goto err_dequeue_ele;
+ }
+
+ ret = 0;
+ goto put_ele_ref;
+ }
+
+ return 0;
+
+err_dequeue_ele:
+ /*
+ * Try to remove the command from the queue. If that fails, then it
+ * means the response thread is currently using the object, and we need
+ * to use the refcount to avoid a use-after-free by either side.
+ */
+ if (ele && isp4if_rm_cmd_from_cmdq(ispif, seq_num, cmd_id))
+ goto free_ele;
+
+put_ele_ref:
+ /* Don't free the command if we didn't put the last reference */
+ if (ele && atomic_dec_return(&ele->refcnt))
+ ele = NULL;
+
+free_ele:
+ kfree(ele);
+ return ret;
+}
+
+static int isp4if_send_buffer(struct isp4_interface *ispif, struct isp4if_img_buf_info *buf_info)
+{
+ struct isp4fw_cmd_send_buffer cmd;
+
+ /*
+ * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
+ * zeroed, since this is not guaranteed on all compilers.
+ */
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.buffer_type = BUFFER_TYPE_PREVIEW;
+ cmd.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
+ isp4if_split_addr64(buf_info->planes[0].mc_addr,
+ &cmd.buffer.buf_base_a_lo,
+ &cmd.buffer.buf_base_a_hi);
+ cmd.buffer.buf_size_a = buf_info->planes[0].len;
+
+ isp4if_split_addr64(buf_info->planes[1].mc_addr,
+ &cmd.buffer.buf_base_b_lo,
+ &cmd.buffer.buf_base_b_hi);
+ cmd.buffer.buf_size_b = buf_info->planes[1].len;
+
+ isp4if_split_addr64(buf_info->planes[2].mc_addr,
+ &cmd.buffer.buf_base_c_lo,
+ &cmd.buffer.buf_base_c_hi);
+ cmd.buffer.buf_size_c = buf_info->planes[2].len;
+
+ return isp4if_send_fw_cmd(ispif, CMD_ID_SEND_BUFFER, &cmd, sizeof(cmd), false);
+}
+
+static void isp4if_init_rb_config(struct isp4_interface *ispif, struct isp4if_rb_config *rb_config)
+{
+ isp4hw_wreg(ispif->mmio, rb_config->reg_rptr, 0x0);
+ isp4hw_wreg(ispif->mmio, rb_config->reg_wptr, 0x0);
+ isp4hw_wreg(ispif->mmio, rb_config->reg_base_lo, rb_config->base_mc_addr);
+ isp4hw_wreg(ispif->mmio, rb_config->reg_base_hi, rb_config->base_mc_addr >> 32);
+ isp4hw_wreg(ispif->mmio, rb_config->reg_size, rb_config->val_size);
+}
+
+static int isp4if_fw_init(struct isp4_interface *ispif)
+{
+ u32 aligned_rb_chunk_size = ISP4IF_RB_PMBMAP_MEM_CHUNK & 0xffffffc0;
+ struct isp4if_rb_config *rb_config;
+ u32 offset;
+ int i;
+
+ /* initialize CMD_RB streams */
+ for (i = 0; i < ISP4IF_STREAM_ID_MAX; i++) {
+ rb_config = (isp4if_cmd_rb_config + i);
+ offset = aligned_rb_chunk_size * rb_config->index;
+
+ rb_config->val_size = ISP4IF_FW_CMD_BUF_SIZE;
+ rb_config->base_sys_addr =
+ ispif->fw_cmd_resp_buf->sys_addr + offset;
+ rb_config->base_mc_addr =
+ ispif->fw_cmd_resp_buf->gpu_mc_addr + offset;
+
+ isp4if_init_rb_config(ispif, rb_config);
+ }
+
+ /* initialize RESP_RB streams */
+ for (i = 0; i < ISP4IF_STREAM_ID_MAX; i++) {
+ rb_config = (isp4if_resp_rb_config + i);
+ offset = aligned_rb_chunk_size *
+ (rb_config->index + ISP4IF_RESP_CHAN_TO_RB_OFFSET - 1);
+
+ rb_config->val_size = ISP4IF_FW_CMD_BUF_SIZE;
+ rb_config->base_sys_addr =
+ ispif->fw_cmd_resp_buf->sys_addr + offset;
+ rb_config->base_mc_addr =
+ ispif->fw_cmd_resp_buf->gpu_mc_addr + offset;
+
+ isp4if_init_rb_config(ispif, rb_config);
+ }
+
+ /* initialize LOG_RB stream */
+ rb_config = &isp4if_log_rb_config;
+ rb_config->val_size = ISP4IF_FW_LOG_RINGBUF_SIZE;
+ rb_config->base_mc_addr = ispif->fw_log_buf->gpu_mc_addr;
+ rb_config->base_sys_addr = ispif->fw_log_buf->sys_addr;
+
+ isp4if_init_rb_config(ispif, rb_config);
+
+ return 0;
+}
+
+static int isp4if_wait_fw_ready(struct isp4_interface *ispif, u32 isp_status_addr)
+{
+ struct device *dev = ispif->dev;
+ u32 timeout_ms = 100;
+ u32 interval_ms = 1;
+ u32 reg_val;
+
+ /* wait for FW initialize done! */
+ if (!read_poll_timeout(isp4hw_rreg, reg_val, reg_val & ISP_STATUS__CCPU_REPORT_MASK,
+ interval_ms * 1000, timeout_ms * 1000, false,
+ ispif->mmio, isp_status_addr))
+ return 0;
+
+ dev_err(dev, "ISP CCPU FW boot failed\n");
+
+ return -ETIME;
+}
+
+static void isp4if_enable_ccpu(struct isp4_interface *ispif)
+{
+ u32 reg_val;
+
+ reg_val = isp4hw_rreg(ispif->mmio, ISP_SOFT_RESET);
+ reg_val &= (~ISP_SOFT_RESET__CCPU_SOFT_RESET_MASK);
+ isp4hw_wreg(ispif->mmio, ISP_SOFT_RESET, reg_val);
+
+ usleep_range(100, 150);
+
+ reg_val = isp4hw_rreg(ispif->mmio, ISP_CCPU_CNTL);
+ reg_val &= (~ISP_CCPU_CNTL__CCPU_HOST_SOFT_RST_MASK);
+ isp4hw_wreg(ispif->mmio, ISP_CCPU_CNTL, reg_val);
+}
+
+static void isp4if_disable_ccpu(struct isp4_interface *ispif)
+{
+ u32 reg_val;
+
+ reg_val = isp4hw_rreg(ispif->mmio, ISP_CCPU_CNTL);
+ reg_val |= ISP_CCPU_CNTL__CCPU_HOST_SOFT_RST_MASK;
+ isp4hw_wreg(ispif->mmio, ISP_CCPU_CNTL, reg_val);
+
+ usleep_range(100, 150);
+
+ reg_val = isp4hw_rreg(ispif->mmio, ISP_SOFT_RESET);
+ reg_val |= ISP_SOFT_RESET__CCPU_SOFT_RESET_MASK;
+ isp4hw_wreg(ispif->mmio, ISP_SOFT_RESET, reg_val);
+}
+
+static int isp4if_fw_boot(struct isp4_interface *ispif)
+{
+ struct device *dev = ispif->dev;
+
+ if (ispif->status != ISP4IF_STATUS_PWR_ON) {
+ dev_err(dev, "invalid isp power status %d\n", ispif->status);
+ return -EINVAL;
+ }
+
+ isp4if_disable_ccpu(ispif);
+
+ isp4if_fw_init(ispif);
+
+ /* clear ccpu status */
+ isp4hw_wreg(ispif->mmio, ISP_STATUS, 0x0);
+
+ isp4if_enable_ccpu(ispif);
+
+ if (isp4if_wait_fw_ready(ispif, ISP_STATUS)) {
+ isp4if_disable_ccpu(ispif);
+ return -EINVAL;
+ }
+
+ /* enable interrupts */
+ isp4hw_wreg(ispif->mmio, ISP_SYS_INT0_EN, ISP4IF_FW_RESP_RB_IRQ_EN_MASK);
+
+ ispif->status = ISP4IF_STATUS_FW_RUNNING;
+
+ dev_dbg(dev, "ISP CCPU FW boot success\n");
+
+ return 0;
+}
+
+int isp4if_f2h_resp(struct isp4_interface *ispif, enum isp4if_stream_id stream,
+ struct isp4fw_resp *resp)
+{
+ struct isp4if_rb_config *rb_config = &isp4if_resp_rb_config[stream];
+ u32 rreg = rb_config->reg_rptr, wreg = rb_config->reg_wptr;
+ void *mem_sys = rb_config->base_sys_addr;
+ const u32 resp_sz = sizeof(*resp);
+ struct device *dev = ispif->dev;
+ u32 len = rb_config->val_size;
+ u32 rd_ptr, wr_ptr;
+ u32 bytes_to_end;
+ void *dst = resp;
+ u32 checksum;
+
+ rd_ptr = isp4hw_rreg(ispif->mmio, rreg);
+ wr_ptr = isp4hw_rreg(ispif->mmio, wreg);
+ if (rd_ptr >= len || wr_ptr >= len)
+ goto err_rb_invalid;
+
+ /* Read and write pointers are equal, indicating the ringbuf is empty */
+ if (rd_ptr == wr_ptr)
+ return -ENODATA;
+
+ bytes_to_end = len - rd_ptr;
+ if (bytes_to_end >= resp_sz) {
+ /* FW response is just a straight copy from the read pointer */
+ if (wr_ptr > rd_ptr && wr_ptr - rd_ptr < resp_sz)
+ goto err_rb_invalid;
+
+ memcpy(dst, mem_sys + rd_ptr, resp_sz);
+ isp4hw_wreg(ispif->mmio, rreg, (rd_ptr + resp_sz) % len);
+ } else {
+ /* FW response is split because the ringbuf wrapped around */
+ if (wr_ptr > rd_ptr || wr_ptr < resp_sz - bytes_to_end)
+ goto err_rb_invalid;
+
+ memcpy(dst, mem_sys + rd_ptr, bytes_to_end);
+ memcpy(dst + bytes_to_end, mem_sys, resp_sz - bytes_to_end);
+ isp4hw_wreg(ispif->mmio, rreg, resp_sz - bytes_to_end);
+ }
+
+ checksum = isp4if_compute_check_sum(resp, resp_sz - sizeof(u32));
+ if (checksum != resp->resp_check_sum) {
+ dev_err(dev, "resp checksum 0x%x,should 0x%x,rptr %u,wptr %u\n",
+ checksum, resp->resp_check_sum, rd_ptr, wr_ptr);
+ dev_err(dev, "(%u), seqNo %u, resp_id (0x%x)\n",
+ stream, resp->resp_seq_num,
+ resp->resp_id);
+ return -EINVAL;
+ }
+
+ return 0;
+
+err_rb_invalid:
+ dev_err(dev, "rb invalid: stream=%u, rd=%u, wr=%u, len=%u, resp_sz=%u\n",
+ stream, rd_ptr, wr_ptr, len, resp_sz);
+ return -EINVAL;
+}
+
+int isp4if_send_command(struct isp4_interface *ispif, u32 cmd_id, const void *package,
+ u32 package_size)
+{
+ return isp4if_send_fw_cmd(ispif, cmd_id, package, package_size, false);
+}
+
+int isp4if_send_command_sync(struct isp4_interface *ispif, u32 cmd_id, const void *package,
+ u32 package_size)
+{
+ return isp4if_send_fw_cmd(ispif, cmd_id, package, package_size, true);
+}
+
+void isp4if_clear_bufq(struct isp4_interface *ispif)
+{
+ struct isp4if_img_buf_node *buf_node, *tmp_node;
+ LIST_HEAD(free_list);
+
+ scoped_guard(spinlock, &ispif->bufq_lock)
+ list_splice_init(&ispif->bufq, &free_list);
+
+ list_for_each_entry_safe(buf_node, tmp_node, &free_list, node)
+ kfree(buf_node);
+}
+
+void isp4if_dealloc_buffer_node(struct isp4if_img_buf_node *buf_node)
+{
+ kfree(buf_node);
+}
+
+struct isp4if_img_buf_node *isp4if_alloc_buffer_node(struct isp4if_img_buf_info *buf_info)
+{
+ struct isp4if_img_buf_node *node;
+
+ node = kmalloc(sizeof(*node), GFP_KERNEL);
+ if (node)
+ node->buf_info = *buf_info;
+
+ return node;
+}
+
+struct isp4if_img_buf_node *isp4if_dequeue_buffer(struct isp4_interface *ispif)
+{
+ struct isp4if_img_buf_node *buf_node;
+
+ guard(spinlock)(&ispif->bufq_lock);
+
+ buf_node = list_first_entry_or_null(&ispif->bufq, typeof(*buf_node), node);
+ if (buf_node)
+ list_del(&buf_node->node);
+
+ return buf_node;
+}
+
+int isp4if_queue_buffer(struct isp4_interface *ispif, struct isp4if_img_buf_node *buf_node)
+{
+ int ret;
+
+ ret = isp4if_send_buffer(ispif, &buf_node->buf_info);
+ if (ret)
+ return ret;
+
+ scoped_guard(spinlock, &ispif->bufq_lock)
+ list_add_tail(&buf_node->node, &ispif->bufq);
+
+ return 0;
+}
+
+int isp4if_stop(struct isp4_interface *ispif)
+{
+ isp4if_disable_ccpu(ispif);
+
+ isp4if_dealloc_fw_gpumem(ispif);
+
+ return 0;
+}
+
+int isp4if_start(struct isp4_interface *ispif)
+{
+ int ret;
+
+ ret = isp4if_alloc_fw_gpumem(ispif);
+ if (ret)
+ return ret;
+
+ ret = isp4if_fw_boot(ispif);
+ if (ret)
+ goto failed_fw_boot;
+
+ return 0;
+
+failed_fw_boot:
+ isp4if_dealloc_fw_gpumem(ispif);
+ return ret;
+}
+
+int isp4if_deinit(struct isp4_interface *ispif)
+{
+ isp4if_clear_cmdq(ispif);
+
+ isp4if_clear_bufq(ispif);
+
+ mutex_destroy(&ispif->isp4if_mutex);
+
+ return 0;
+}
+
+int isp4if_init(struct isp4_interface *ispif, struct device *dev, void __iomem *isp_mmio)
+{
+ ispif->dev = dev;
+ ispif->mmio = isp_mmio;
+
+ spin_lock_init(&ispif->cmdq_lock); /* used for cmdq access */
+ spin_lock_init(&ispif->bufq_lock); /* used for bufq access */
+ mutex_init(&ispif->isp4if_mutex); /* used for commands sent to ispfw */
+
+ INIT_LIST_HEAD(&ispif->cmdq);
+ INIT_LIST_HEAD(&ispif->bufq);
+
+ return 0;
+}
diff --git a/drivers/media/platform/amd/isp4/isp4_interface.h b/drivers/media/platform/amd/isp4/isp4_interface.h
new file mode 100644
index 000000000000..01d5268f7d4c
--- /dev/null
+++ b/drivers/media/platform/amd/isp4/isp4_interface.h
@@ -0,0 +1,141 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (C) 2025 Advanced Micro Devices, Inc.
+ */
+
+#ifndef _ISP4_INTERFACE_H_
+#define _ISP4_INTERFACE_H_
+
+#include <drm/amd/isp.h>
+#include <linux/mutex.h>
+#include <linux/platform_device.h>
+#include <linux/spinlock.h>
+
+struct isp4fw_resp;
+
+#define ISP4IF_RB_MAX 25
+#define ISP4IF_RESP_CHAN_TO_RB_OFFSET 9
+#define ISP4IF_RB_PMBMAP_MEM_SIZE (SZ_16M - 1)
+#define ISP4IF_RB_PMBMAP_MEM_CHUNK \
+ (ISP4IF_RB_PMBMAP_MEM_SIZE / (ISP4IF_RB_MAX - 1))
+#define ISP4IF_HOST2FW_COMMAND_SIZE sizeof(struct isp4fw_cmd)
+#define ISP4IF_MAX_NUM_HOST2FW_COMMAND 40
+#define ISP4IF_FW_CMD_BUF_SIZE \
+ (ISP4IF_MAX_NUM_HOST2FW_COMMAND * ISP4IF_HOST2FW_COMMAND_SIZE)
+#define ISP4IF_RB_FULL_SLEEP_US (33 * USEC_PER_MSEC)
+#define ISP4IF_RB_FULL_TIMEOUT_US (10 * ISP4IF_RB_FULL_SLEEP_US)
+
+#define ISP4IF_META_INFO_BUF_SIZE ALIGN(sizeof(struct isp4fw_meta_info), 0x8000)
+#define ISP4IF_MAX_STREAM_BUF_COUNT 8
+
+#define ISP4IF_FW_LOG_RINGBUF_SIZE SZ_2M
+
+enum isp4if_stream_id {
+ ISP4IF_STREAM_ID_GLOBAL = 0,
+ ISP4IF_STREAM_ID_1 = 1,
+ ISP4IF_STREAM_ID_MAX = 4
+};
+
+enum isp4if_status {
+ ISP4IF_STATUS_PWR_OFF,
+ ISP4IF_STATUS_PWR_ON,
+ ISP4IF_STATUS_FW_RUNNING,
+ ISP4IF_FSM_STATUS_MAX
+};
+
+struct isp4if_gpu_mem_info {
+ u64 mem_size;
+ u64 gpu_mc_addr;
+ void *sys_addr;
+ void *mem_handle;
+};
+
+struct isp4if_img_buf_info {
+ struct {
+ void *sys_addr;
+ u64 mc_addr;
+ u32 len;
+ } planes[3];
+};
+
+struct isp4if_img_buf_node {
+ struct list_head node;
+ struct isp4if_img_buf_info buf_info;
+};
+
+struct isp4if_cmd_element {
+ struct list_head list;
+ u32 seq_num;
+ u32 cmd_id;
+ struct completion cmd_done;
+ atomic_t refcnt;
+};
+
+struct isp4_interface {
+ struct device *dev;
+ void __iomem *mmio;
+
+ spinlock_t cmdq_lock; /* used for cmdq access */
+ spinlock_t bufq_lock; /* used for bufq access */
+ struct mutex isp4if_mutex; /* used to send fw cmd and read fw log */
+
+ struct list_head cmdq; /* commands sent to fw */
+ struct list_head bufq; /* buffers sent to fw */
+
+ enum isp4if_status status;
+ u32 host2fw_seq_num;
+
+ /* ISP fw buffers */
+ struct isp4if_gpu_mem_info *fw_log_buf;
+ struct isp4if_gpu_mem_info *fw_cmd_resp_buf;
+ struct isp4if_gpu_mem_info *fw_mem_pool;
+ struct isp4if_gpu_mem_info *meta_info_buf[ISP4IF_MAX_STREAM_BUF_COUNT];
+};
+
+static inline void isp4if_split_addr64(u64 addr, u32 *lo, u32 *hi)
+{
+ if (lo)
+ *lo = addr & 0xffffffff;
+
+ if (hi)
+ *hi = addr >> 32;
+}
+
+static inline u64 isp4if_join_addr64(u32 lo, u32 hi)
+{
+ return (((u64)hi) << 32) | (u64)lo;
+}
+
+int isp4if_f2h_resp(struct isp4_interface *ispif, enum isp4if_stream_id stream,
+ struct isp4fw_resp *resp);
+
+int isp4if_send_command(struct isp4_interface *ispif, u32 cmd_id, const void *package,
+ u32 package_size);
+
+int isp4if_send_command_sync(struct isp4_interface *ispif, u32 cmd_id, const void *package,
+ u32 package_size);
+
+struct isp4if_cmd_element *isp4if_rm_cmd_from_cmdq(struct isp4_interface *ispif, u32 seq_num,
+ u32 cmd_id);
+
+void isp4if_clear_cmdq(struct isp4_interface *ispif);
+
+void isp4if_clear_bufq(struct isp4_interface *ispif);
+
+void isp4if_dealloc_buffer_node(struct isp4if_img_buf_node *buf_node);
+
+struct isp4if_img_buf_node *isp4if_alloc_buffer_node(struct isp4if_img_buf_info *buf_info);
+
+struct isp4if_img_buf_node *isp4if_dequeue_buffer(struct isp4_interface *ispif);
+
+int isp4if_queue_buffer(struct isp4_interface *ispif, struct isp4if_img_buf_node *buf_node);
+
+int isp4if_stop(struct isp4_interface *ispif);
+
+int isp4if_start(struct isp4_interface *ispif);
+
+int isp4if_deinit(struct isp4_interface *ispif);
+
+int isp4if_init(struct isp4_interface *ispif, struct device *dev, void __iomem *isp_mmio);
+
+#endif /* _ISP4_INTERFACE_H_ */
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread* Re: [PATCH v7 3/7] media: platform: amd: Add isp4 fw and hw interface
2025-12-16 9:13 ` [PATCH v7 3/7] media: platform: amd: Add isp4 fw and hw interface Bin Du
@ 2025-12-22 9:37 ` Sakari Ailus
2026-01-07 8:44 ` Du, Bin
0 siblings, 1 reply; 46+ messages in thread
From: Sakari Ailus @ 2025-12-22 9:37 UTC (permalink / raw)
To: Bin Du
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Svetoslav Stoilov, Alexey Zagorodnikov
Hi Bin,
On Tue, Dec 16, 2025 at 05:13:22PM +0800, Bin Du wrote:
> ISP firmware controls ISP HW pipeline using dedicated embedded processor
> called ccpu. The communication between ISP FW and driver is using commands
> and response messages sent through the ring buffer. Command buffers support
> either global setting that is not specific to the steam and support stream
> specific parameters. Response buffers contain ISP FW notification
> information such as frame buffer done and command done. IRQ is used for
> receiving response buffer from ISP firmware, which is handled in the main
> isp4 media device. ISP ccpu is booted up through the firmware loading
> helper function prior to stream start. Memory used for command buffer and
> response buffer needs to be allocated from amdgpu buffer manager because
> isp4 is a child device of amdgpu.
>
> Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
> Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
> Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
> Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
> Signed-off-by: Bin Du <Bin.Du@amd.com>
> Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
> ---
> MAINTAINERS | 3 +
> drivers/media/platform/amd/isp4/Makefile | 3 +-
> .../platform/amd/isp4/isp4_fw_cmd_resp.h | 314 +++++++
> .../media/platform/amd/isp4/isp4_interface.c | 786 ++++++++++++++++++
> .../media/platform/amd/isp4/isp4_interface.h | 141 ++++
> 5 files changed, 1246 insertions(+), 1 deletion(-)
> create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
> create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c
> create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 7aa17c7e71d6..cccae369c876 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1145,7 +1145,10 @@ F: drivers/media/platform/amd/isp4/Kconfig
> F: drivers/media/platform/amd/isp4/Makefile
> F: drivers/media/platform/amd/isp4/isp4.c
> F: drivers/media/platform/amd/isp4/isp4.h
> +F: drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
> F: drivers/media/platform/amd/isp4/isp4_hw_reg.h
> +F: drivers/media/platform/amd/isp4/isp4_interface.c
> +F: drivers/media/platform/amd/isp4/isp4_interface.h
>
> AMD KFD
> M: Felix Kuehling <Felix.Kuehling@amd.com>
> diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
> index de0092dad26f..a2a5bf98e912 100644
> --- a/drivers/media/platform/amd/isp4/Makefile
> +++ b/drivers/media/platform/amd/isp4/Makefile
> @@ -3,4 +3,5 @@
> # Copyright (C) 2025 Advanced Micro Devices, Inc.
>
> obj-$(CONFIG_AMD_ISP4) += amd_capture.o
> -amd_capture-objs := isp4.o
> +amd_capture-objs := isp4.o \
> + isp4_interface.o
> diff --git a/drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h b/drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
> new file mode 100644
> index 000000000000..d571b3873edb
> --- /dev/null
> +++ b/drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
> @@ -0,0 +1,314 @@
> +/* SPDX-License-Identifier: GPL-2.0+ */
> +/*
> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
> + */
> +
> +#ifndef _ISP4_CMD_RESP_H_
> +#define _ISP4_CMD_RESP_H_
> +
> +/*
> + * @brief Host and Firmware command & response channel.
> + * Two types of command/response channel.
> + * Type Global Command has one command/response channel.
> + * Type Stream Command has one command/response channel.
> + *----------- ------------
> + *| | --------------------------- | |
> + *| | ---->| Global Command |----> | |
> + *| | --------------------------- | |
> + *| | | |
> + *| | | |
> + *| | --------------------------- | |
> + *| | ---->| Stream Command |----> | |
> + *| | --------------------------- | |
> + *| | | |
> + *| | | |
> + *| | | |
> + *| HOST | | Firmware |
> + *| | | |
> + *| | | |
> + *| | -------------------------- | |
> + *| | <----| Global Response |<---- | |
> + *| | -------------------------- | |
> + *| | | |
> + *| | | |
> + *| | -------------------------- | |
> + *| | <----| Stream Response |<---- | |
> + *| | -------------------------- | |
> + *| | | |
> + *| | | |
> + *----------- ------------
> + */
> +
> +/*
> + * @brief command ID format
> + * cmd_id is in the format of following type:
> + * type: indicate command type, global/stream commands.
> + * group: indicate the command group.
> + * id: A unique command identification in one type and group.
> + * |<-Bit31 ~ Bit24->|<-Bit23 ~ Bit16->|<-Bit15 ~ Bit0->|
> + * | type | group | id |
> + */
> +
> +#define CMD_TYPE_SHIFT 24
> +#define CMD_GROUP_SHIFT 16
> +#define CMD_TYPE_STREAM_CTRL (0x2U << CMD_TYPE_SHIFT)
> +
> +#define CMD_GROUP_STREAM_CTRL (0x1U << CMD_GROUP_SHIFT)
> +#define CMD_GROUP_STREAM_BUFFER (0x4U << CMD_GROUP_SHIFT)
> +
> +/* Stream Command */
> +#define CMD_ID_SET_STREAM_CONFIG (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x1)
> +#define CMD_ID_SET_OUT_CHAN_PROP (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x3)
> +#define CMD_ID_ENABLE_OUT_CHAN (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x5)
> +#define CMD_ID_START_STREAM (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x7)
> +#define CMD_ID_STOP_STREAM (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x8)
> +
> +/* Stream Buffer Command */
> +#define CMD_ID_SEND_BUFFER (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_BUFFER | 0x1)
> +
> +/*
> + * @brief response ID format
> + * resp_id is in the format of following type:
> + * type: indicate command type, global/stream commands.
> + * group: indicate the command group.
> + * id: A unique command identification in one type and group.
> + * |<-Bit31 ~ Bit24->|<-Bit23 ~ Bit16->|<-Bit15 ~ Bit0->|
> + * | type | group | id |
> + */
> +
> +#define RESP_GROUP_SHIFT 16
> +#define RESP_GROUP_MASK (0xff << RESP_GROUP_SHIFT)
> +
> +#define GET_RESP_GROUP_VALUE(resp_id) (((resp_id) & RESP_GROUP_MASK) >> RESP_GROUP_SHIFT)
> +#define GET_RESP_ID_VALUE(resp_id) ((resp_id) & 0xffff)
> +
> +#define RESP_GROUP_GENERAL (0x1 << RESP_GROUP_SHIFT)
> +#define RESP_GROUP_NOTIFICATION (0x3 << RESP_GROUP_SHIFT)
> +
> +/* General Response */
> +#define RESP_ID_CMD_DONE (RESP_GROUP_GENERAL | 0x1)
> +
> +/* Notification */
> +#define RESP_ID_NOTI_FRAME_DONE (RESP_GROUP_NOTIFICATION | 0x1)
> +
> +#define CMD_STATUS_SUCCESS 0
> +#define CMD_STATUS_FAIL 1
> +#define CMD_STATUS_SKIPPED 2
> +
> +#define ADDR_SPACE_TYPE_GPU_VA 4
> +
> +#define FW_MEMORY_POOL_SIZE (100 * 1024 * 1024)
> +
> +/*
> + * standard ISP mipicsi=>isp
> + */
> +#define MIPI0_ISP_PIPELINE_ID 0x5f91
> +
> +enum isp4fw_sensor_id {
> + SENSOR_ID_ON_MIPI0 = 0, /* Sensor id for ISP input from MIPI port 0 */
> +};
> +
> +enum isp4fw_stream_id {
> + STREAM_ID_INVALID = -1, /* STREAM_ID_INVALID. */
> + STREAM_ID_1 = 0, /* STREAM_ID_1. */
> + STREAM_ID_2 = 1, /* STREAM_ID_2. */
> + STREAM_ID_3 = 2, /* STREAM_ID_3. */
> + STREAM_ID_MAXIMUM /* STREAM_ID_MAXIMUM. */
> +};
> +
> +enum isp4fw_image_format {
> + IMAGE_FORMAT_NV12 = 1, /* 4:2:0,semi-planar, 8-bit */
> + IMAGE_FORMAT_YUV422INTERLEAVED = 7, /* interleave, 4:2:2, 8-bit */
> +};
> +
> +enum isp4fw_pipe_out_ch {
> + ISP_PIPE_OUT_CH_PREVIEW = 0,
> +};
> +
> +enum isp4fw_yuv_range {
> + ISP_YUV_RANGE_FULL = 0, /* YUV value range in 0~255 */
> + ISP_YUV_RANGE_NARROW = 1, /* YUV value range in 16~235 */
> + ISP_YUV_RANGE_MAX
> +};
> +
> +enum isp4fw_buffer_type {
> + BUFFER_TYPE_PREVIEW = 8,
> + BUFFER_TYPE_META_INFO = 10,
> + BUFFER_TYPE_MEM_POOL = 15,
> +};
> +
> +enum isp4fw_buffer_status {
> + BUFFER_STATUS_INVALID, /* The buffer is INVALID */
> + BUFFER_STATUS_SKIPPED, /* The buffer is not filled with image data */
> + BUFFER_STATUS_EXIST, /* The buffer is exist and waiting for filled */
> + BUFFER_STATUS_DONE, /* The buffer is filled with image data */
> + BUFFER_STATUS_LACK, /* The buffer is unavailable */
> + BUFFER_STATUS_DIRTY, /* The buffer is dirty, probably caused by
> + * LMI leakage
> + */
> + BUFFER_STATUS_MAX /* The buffer STATUS_MAX */
> +};
> +
> +enum isp4fw_buffer_source {
> + /* The buffer is from the stream buffer queue */
> + BUFFER_SOURCE_STREAM,
> +};
Could you also use the ISP4 (or ISP4IF) prefix for these, please? Many look
rather generic.
...
> diff --git a/drivers/media/platform/amd/isp4/isp4_interface.h b/drivers/media/platform/amd/isp4/isp4_interface.h
> new file mode 100644
> index 000000000000..01d5268f7d4c
> --- /dev/null
> +++ b/drivers/media/platform/amd/isp4/isp4_interface.h
> @@ -0,0 +1,141 @@
> +/* SPDX-License-Identifier: GPL-2.0+ */
> +/*
> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
> + */
> +
> +#ifndef _ISP4_INTERFACE_H_
> +#define _ISP4_INTERFACE_H_
> +
> +#include <drm/amd/isp.h>
> +#include <linux/mutex.h>
> +#include <linux/platform_device.h>
> +#include <linux/spinlock.h>
> +
> +struct isp4fw_resp;
> +
> +#define ISP4IF_RB_MAX 25
> +#define ISP4IF_RESP_CHAN_TO_RB_OFFSET 9
> +#define ISP4IF_RB_PMBMAP_MEM_SIZE (SZ_16M - 1)
> +#define ISP4IF_RB_PMBMAP_MEM_CHUNK \
> + (ISP4IF_RB_PMBMAP_MEM_SIZE / (ISP4IF_RB_MAX - 1))
> +#define ISP4IF_HOST2FW_COMMAND_SIZE sizeof(struct isp4fw_cmd)
> +#define ISP4IF_MAX_NUM_HOST2FW_COMMAND 40
> +#define ISP4IF_FW_CMD_BUF_SIZE \
> + (ISP4IF_MAX_NUM_HOST2FW_COMMAND * ISP4IF_HOST2FW_COMMAND_SIZE)
> +#define ISP4IF_RB_FULL_SLEEP_US (33 * USEC_PER_MSEC)
> +#define ISP4IF_RB_FULL_TIMEOUT_US (10 * ISP4IF_RB_FULL_SLEEP_US)
> +
> +#define ISP4IF_META_INFO_BUF_SIZE ALIGN(sizeof(struct isp4fw_meta_info), 0x8000)
> +#define ISP4IF_MAX_STREAM_BUF_COUNT 8
> +
> +#define ISP4IF_FW_LOG_RINGBUF_SIZE SZ_2M
> +
> +enum isp4if_stream_id {
> + ISP4IF_STREAM_ID_GLOBAL = 0,
> + ISP4IF_STREAM_ID_1 = 1,
> + ISP4IF_STREAM_ID_MAX = 4
> +};
> +
> +enum isp4if_status {
> + ISP4IF_STATUS_PWR_OFF,
> + ISP4IF_STATUS_PWR_ON,
> + ISP4IF_STATUS_FW_RUNNING,
> + ISP4IF_FSM_STATUS_MAX
> +};
> +
> +struct isp4if_gpu_mem_info {
> + u64 mem_size;
> + u64 gpu_mc_addr;
> + void *sys_addr;
> + void *mem_handle;
> +};
> +
> +struct isp4if_img_buf_info {
> + struct {
> + void *sys_addr;
> + u64 mc_addr;
> + u32 len;
> + } planes[3];
> +};
> +
> +struct isp4if_img_buf_node {
> + struct list_head node;
> + struct isp4if_img_buf_info buf_info;
> +};
> +
> +struct isp4if_cmd_element {
> + struct list_head list;
> + u32 seq_num;
> + u32 cmd_id;
> + struct completion cmd_done;
> + atomic_t refcnt;
> +};
> +
> +struct isp4_interface {
How about just "isp4if"? Up to you.
> + struct device *dev;
> + void __iomem *mmio;
> +
> + spinlock_t cmdq_lock; /* used for cmdq access */
> + spinlock_t bufq_lock; /* used for bufq access */
> + struct mutex isp4if_mutex; /* used to send fw cmd and read fw log */
> +
> + struct list_head cmdq; /* commands sent to fw */
> + struct list_head bufq; /* buffers sent to fw */
> +
> + enum isp4if_status status;
> + u32 host2fw_seq_num;
> +
> + /* ISP fw buffers */
> + struct isp4if_gpu_mem_info *fw_log_buf;
> + struct isp4if_gpu_mem_info *fw_cmd_resp_buf;
> + struct isp4if_gpu_mem_info *fw_mem_pool;
> + struct isp4if_gpu_mem_info *meta_info_buf[ISP4IF_MAX_STREAM_BUF_COUNT];
> +};
> +
> +static inline void isp4if_split_addr64(u64 addr, u32 *lo, u32 *hi)
> +{
> + if (lo)
> + *lo = addr & 0xffffffff;
> +
> + if (hi)
> + *hi = addr >> 32;
> +}
> +
> +static inline u64 isp4if_join_addr64(u32 lo, u32 hi)
> +{
> + return (((u64)hi) << 32) | (u64)lo;
> +}
> +
> +int isp4if_f2h_resp(struct isp4_interface *ispif, enum isp4if_stream_id stream,
> + struct isp4fw_resp *resp);
> +
> +int isp4if_send_command(struct isp4_interface *ispif, u32 cmd_id, const void *package,
> + u32 package_size);
> +
> +int isp4if_send_command_sync(struct isp4_interface *ispif, u32 cmd_id, const void *package,
> + u32 package_size);
> +
> +struct isp4if_cmd_element *isp4if_rm_cmd_from_cmdq(struct isp4_interface *ispif, u32 seq_num,
> + u32 cmd_id);
> +
> +void isp4if_clear_cmdq(struct isp4_interface *ispif);
> +
> +void isp4if_clear_bufq(struct isp4_interface *ispif);
> +
> +void isp4if_dealloc_buffer_node(struct isp4if_img_buf_node *buf_node);
> +
> +struct isp4if_img_buf_node *isp4if_alloc_buffer_node(struct isp4if_img_buf_info *buf_info);
> +
> +struct isp4if_img_buf_node *isp4if_dequeue_buffer(struct isp4_interface *ispif);
> +
> +int isp4if_queue_buffer(struct isp4_interface *ispif, struct isp4if_img_buf_node *buf_node);
> +
> +int isp4if_stop(struct isp4_interface *ispif);
> +
> +int isp4if_start(struct isp4_interface *ispif);
> +
> +int isp4if_deinit(struct isp4_interface *ispif);
> +
> +int isp4if_init(struct isp4_interface *ispif, struct device *dev, void __iomem *isp_mmio);
Could you run
$ ./scripts/checkpatch.pl --strict --max-line-length=80
on the set, please?
> +
> +#endif /* _ISP4_INTERFACE_H_ */
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 3/7] media: platform: amd: Add isp4 fw and hw interface
2025-12-22 9:37 ` Sakari Ailus
@ 2026-01-07 8:44 ` Du, Bin
2026-01-14 20:55 ` Sakari Ailus
2026-01-20 9:22 ` Du, Bin
0 siblings, 2 replies; 46+ messages in thread
From: Du, Bin @ 2026-01-07 8:44 UTC (permalink / raw)
To: Sakari Ailus
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Svetoslav Stoilov, Alexey Zagorodnikov
Thank you, Sakari for the feedback.
On 12/22/2025 5:37 PM, Sakari Ailus wrote:
> Hi Bin,
>
> On Tue, Dec 16, 2025 at 05:13:22PM +0800, Bin Du wrote:
>> ISP firmware controls ISP HW pipeline using dedicated embedded processor
>> called ccpu. The communication between ISP FW and driver is using commands
>> and response messages sent through the ring buffer. Command buffers support
>> either global setting that is not specific to the steam and support stream
>> specific parameters. Response buffers contain ISP FW notification
>> information such as frame buffer done and command done. IRQ is used for
>> receiving response buffer from ISP firmware, which is handled in the main
>> isp4 media device. ISP ccpu is booted up through the firmware loading
>> helper function prior to stream start. Memory used for command buffer and
>> response buffer needs to be allocated from amdgpu buffer manager because
>> isp4 is a child device of amdgpu.
>>
>> Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
>> Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
>> Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
>> Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
>> Signed-off-by: Bin Du <Bin.Du@amd.com>
>> Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
>> ---
>> MAINTAINERS | 3 +
>> drivers/media/platform/amd/isp4/Makefile | 3 +-
>> .../platform/amd/isp4/isp4_fw_cmd_resp.h | 314 +++++++
>> .../media/platform/amd/isp4/isp4_interface.c | 786 ++++++++++++++++++
>> .../media/platform/amd/isp4/isp4_interface.h | 141 ++++
>> 5 files changed, 1246 insertions(+), 1 deletion(-)
>> create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
>> create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c
>> create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 7aa17c7e71d6..cccae369c876 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -1145,7 +1145,10 @@ F: drivers/media/platform/amd/isp4/Kconfig
>> F: drivers/media/platform/amd/isp4/Makefile
>> F: drivers/media/platform/amd/isp4/isp4.c
>> F: drivers/media/platform/amd/isp4/isp4.h
>> +F: drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
>> F: drivers/media/platform/amd/isp4/isp4_hw_reg.h
>> +F: drivers/media/platform/amd/isp4/isp4_interface.c
>> +F: drivers/media/platform/amd/isp4/isp4_interface.h
>>
>> AMD KFD
>> M: Felix Kuehling <Felix.Kuehling@amd.com>
>> diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
>> index de0092dad26f..a2a5bf98e912 100644
>> --- a/drivers/media/platform/amd/isp4/Makefile
>> +++ b/drivers/media/platform/amd/isp4/Makefile
>> @@ -3,4 +3,5 @@
>> # Copyright (C) 2025 Advanced Micro Devices, Inc.
>>
>> obj-$(CONFIG_AMD_ISP4) += amd_capture.o
>> -amd_capture-objs := isp4.o
>> +amd_capture-objs := isp4.o \
>> + isp4_interface.o
>> diff --git a/drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h b/drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
>> new file mode 100644
>> index 000000000000..d571b3873edb
>> --- /dev/null
>> +++ b/drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
>> @@ -0,0 +1,314 @@
>> +/* SPDX-License-Identifier: GPL-2.0+ */
>> +/*
>> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
>> + */
>> +
>> +#ifndef _ISP4_CMD_RESP_H_
>> +#define _ISP4_CMD_RESP_H_
>> +
>> +/*
>> + * @brief Host and Firmware command & response channel.
>> + * Two types of command/response channel.
>> + * Type Global Command has one command/response channel.
>> + * Type Stream Command has one command/response channel.
>> + *----------- ------------
>> + *| | --------------------------- | |
>> + *| | ---->| Global Command |----> | |
>> + *| | --------------------------- | |
>> + *| | | |
>> + *| | | |
>> + *| | --------------------------- | |
>> + *| | ---->| Stream Command |----> | |
>> + *| | --------------------------- | |
>> + *| | | |
>> + *| | | |
>> + *| | | |
>> + *| HOST | | Firmware |
>> + *| | | |
>> + *| | | |
>> + *| | -------------------------- | |
>> + *| | <----| Global Response |<---- | |
>> + *| | -------------------------- | |
>> + *| | | |
>> + *| | | |
>> + *| | -------------------------- | |
>> + *| | <----| Stream Response |<---- | |
>> + *| | -------------------------- | |
>> + *| | | |
>> + *| | | |
>> + *----------- ------------
>> + */
>> +
>> +/*
>> + * @brief command ID format
>> + * cmd_id is in the format of following type:
>> + * type: indicate command type, global/stream commands.
>> + * group: indicate the command group.
>> + * id: A unique command identification in one type and group.
>> + * |<-Bit31 ~ Bit24->|<-Bit23 ~ Bit16->|<-Bit15 ~ Bit0->|
>> + * | type | group | id |
>> + */
>> +
>> +#define CMD_TYPE_SHIFT 24
>> +#define CMD_GROUP_SHIFT 16
>> +#define CMD_TYPE_STREAM_CTRL (0x2U << CMD_TYPE_SHIFT)
>> +
>> +#define CMD_GROUP_STREAM_CTRL (0x1U << CMD_GROUP_SHIFT)
>> +#define CMD_GROUP_STREAM_BUFFER (0x4U << CMD_GROUP_SHIFT)
>> +
>> +/* Stream Command */
>> +#define CMD_ID_SET_STREAM_CONFIG (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x1)
>> +#define CMD_ID_SET_OUT_CHAN_PROP (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x3)
>> +#define CMD_ID_ENABLE_OUT_CHAN (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x5)
>> +#define CMD_ID_START_STREAM (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x7)
>> +#define CMD_ID_STOP_STREAM (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_CTRL | 0x8)
>> +
>> +/* Stream Buffer Command */
>> +#define CMD_ID_SEND_BUFFER (CMD_TYPE_STREAM_CTRL | CMD_GROUP_STREAM_BUFFER | 0x1)
>> +
>> +/*
>> + * @brief response ID format
>> + * resp_id is in the format of following type:
>> + * type: indicate command type, global/stream commands.
>> + * group: indicate the command group.
>> + * id: A unique command identification in one type and group.
>> + * |<-Bit31 ~ Bit24->|<-Bit23 ~ Bit16->|<-Bit15 ~ Bit0->|
>> + * | type | group | id |
>> + */
>> +
>> +#define RESP_GROUP_SHIFT 16
>> +#define RESP_GROUP_MASK (0xff << RESP_GROUP_SHIFT)
>> +
>> +#define GET_RESP_GROUP_VALUE(resp_id) (((resp_id) & RESP_GROUP_MASK) >> RESP_GROUP_SHIFT)
>> +#define GET_RESP_ID_VALUE(resp_id) ((resp_id) & 0xffff)
>> +
>> +#define RESP_GROUP_GENERAL (0x1 << RESP_GROUP_SHIFT)
>> +#define RESP_GROUP_NOTIFICATION (0x3 << RESP_GROUP_SHIFT)
>> +
>> +/* General Response */
>> +#define RESP_ID_CMD_DONE (RESP_GROUP_GENERAL | 0x1)
>> +
>> +/* Notification */
>> +#define RESP_ID_NOTI_FRAME_DONE (RESP_GROUP_NOTIFICATION | 0x1)
>> +
>> +#define CMD_STATUS_SUCCESS 0
>> +#define CMD_STATUS_FAIL 1
>> +#define CMD_STATUS_SKIPPED 2
>> +
>> +#define ADDR_SPACE_TYPE_GPU_VA 4
>> +
>> +#define FW_MEMORY_POOL_SIZE (100 * 1024 * 1024)
>> +
>> +/*
>> + * standard ISP mipicsi=>isp
>> + */
>> +#define MIPI0_ISP_PIPELINE_ID 0x5f91
>> +
>> +enum isp4fw_sensor_id {
>> + SENSOR_ID_ON_MIPI0 = 0, /* Sensor id for ISP input from MIPI port 0 */
>> +};
>> +
>> +enum isp4fw_stream_id {
>> + STREAM_ID_INVALID = -1, /* STREAM_ID_INVALID. */
>> + STREAM_ID_1 = 0, /* STREAM_ID_1. */
>> + STREAM_ID_2 = 1, /* STREAM_ID_2. */
>> + STREAM_ID_3 = 2, /* STREAM_ID_3. */
>> + STREAM_ID_MAXIMUM /* STREAM_ID_MAXIMUM. */
>> +};
>> +
>> +enum isp4fw_image_format {
>> + IMAGE_FORMAT_NV12 = 1, /* 4:2:0,semi-planar, 8-bit */
>> + IMAGE_FORMAT_YUV422INTERLEAVED = 7, /* interleave, 4:2:2, 8-bit */
>> +};
>> +
>> +enum isp4fw_pipe_out_ch {
>> + ISP_PIPE_OUT_CH_PREVIEW = 0,
>> +};
>> +
>> +enum isp4fw_yuv_range {
>> + ISP_YUV_RANGE_FULL = 0, /* YUV value range in 0~255 */
>> + ISP_YUV_RANGE_NARROW = 1, /* YUV value range in 16~235 */
>> + ISP_YUV_RANGE_MAX
>> +};
>> +
>> +enum isp4fw_buffer_type {
>> + BUFFER_TYPE_PREVIEW = 8,
>> + BUFFER_TYPE_META_INFO = 10,
>> + BUFFER_TYPE_MEM_POOL = 15,
>> +};
>> +
>> +enum isp4fw_buffer_status {
>> + BUFFER_STATUS_INVALID, /* The buffer is INVALID */
>> + BUFFER_STATUS_SKIPPED, /* The buffer is not filled with image data */
>> + BUFFER_STATUS_EXIST, /* The buffer is exist and waiting for filled */
>> + BUFFER_STATUS_DONE, /* The buffer is filled with image data */
>> + BUFFER_STATUS_LACK, /* The buffer is unavailable */
>> + BUFFER_STATUS_DIRTY, /* The buffer is dirty, probably caused by
>> + * LMI leakage
>> + */
>> + BUFFER_STATUS_MAX /* The buffer STATUS_MAX */
>> +};
>> +
>> +enum isp4fw_buffer_source {
>> + /* The buffer is from the stream buffer queue */
>> + BUFFER_SOURCE_STREAM,
>> +};
>
> Could you also use the ISP4 (or ISP4IF) prefix for these, please? Many look
> rather generic.
>
Thank you for highlighting this matter, since these definitions are
located in isp4_fw_cmd_resp.h, ISP4_FW may be a more appropriate prefix.
Just to confirm: are you suggesting that we should add this prefix to
all macros and enums? For example, changing CMD_ID_SET_STREAM_CONFIG to
ISP4_FW_CMD_ID_SET_STREAM_CONFIG, and BUFFER_SOURCE_STREAM to ISP4_FW
_BUFFER_SOURCE_STREAM? Our initial thought was that these would only be
used within ISP and shouldn't lead to any confusion.
> ...
>
>> diff --git a/drivers/media/platform/amd/isp4/isp4_interface.h b/drivers/media/platform/amd/isp4/isp4_interface.h
>> new file mode 100644
>> index 000000000000..01d5268f7d4c
>> --- /dev/null
>> +++ b/drivers/media/platform/amd/isp4/isp4_interface.h
>> @@ -0,0 +1,141 @@
>> +/* SPDX-License-Identifier: GPL-2.0+ */
>> +/*
>> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
>> + */
>> +
>> +#ifndef _ISP4_INTERFACE_H_
>> +#define _ISP4_INTERFACE_H_
>> +
>> +#include <drm/amd/isp.h>
>> +#include <linux/mutex.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/spinlock.h>
>> +
>> +struct isp4fw_resp;
>> +
>> +#define ISP4IF_RB_MAX 25
>> +#define ISP4IF_RESP_CHAN_TO_RB_OFFSET 9
>> +#define ISP4IF_RB_PMBMAP_MEM_SIZE (SZ_16M - 1)
>> +#define ISP4IF_RB_PMBMAP_MEM_CHUNK \
>> + (ISP4IF_RB_PMBMAP_MEM_SIZE / (ISP4IF_RB_MAX - 1))
>> +#define ISP4IF_HOST2FW_COMMAND_SIZE sizeof(struct isp4fw_cmd)
>> +#define ISP4IF_MAX_NUM_HOST2FW_COMMAND 40
>> +#define ISP4IF_FW_CMD_BUF_SIZE \
>> + (ISP4IF_MAX_NUM_HOST2FW_COMMAND * ISP4IF_HOST2FW_COMMAND_SIZE)
>> +#define ISP4IF_RB_FULL_SLEEP_US (33 * USEC_PER_MSEC)
>> +#define ISP4IF_RB_FULL_TIMEOUT_US (10 * ISP4IF_RB_FULL_SLEEP_US)
>> +
>> +#define ISP4IF_META_INFO_BUF_SIZE ALIGN(sizeof(struct isp4fw_meta_info), 0x8000)
>> +#define ISP4IF_MAX_STREAM_BUF_COUNT 8
>> +
>> +#define ISP4IF_FW_LOG_RINGBUF_SIZE SZ_2M
>> +
>> +enum isp4if_stream_id {
>> + ISP4IF_STREAM_ID_GLOBAL = 0,
>> + ISP4IF_STREAM_ID_1 = 1,
>> + ISP4IF_STREAM_ID_MAX = 4
>> +};
>> +
>> +enum isp4if_status {
>> + ISP4IF_STATUS_PWR_OFF,
>> + ISP4IF_STATUS_PWR_ON,
>> + ISP4IF_STATUS_FW_RUNNING,
>> + ISP4IF_FSM_STATUS_MAX
>> +};
>> +
>> +struct isp4if_gpu_mem_info {
>> + u64 mem_size;
>> + u64 gpu_mc_addr;
>> + void *sys_addr;
>> + void *mem_handle;
>> +};
>> +
>> +struct isp4if_img_buf_info {
>> + struct {
>> + void *sys_addr;
>> + u64 mc_addr;
>> + u32 len;
>> + } planes[3];
>> +};
>> +
>> +struct isp4if_img_buf_node {
>> + struct list_head node;
>> + struct isp4if_img_buf_info buf_info;
>> +};
>> +
>> +struct isp4if_cmd_element {
>> + struct list_head list;
>> + u32 seq_num;
>> + u32 cmd_id;
>> + struct completion cmd_done;
>> + atomic_t refcnt;
>> +};
>> +
>> +struct isp4_interface {
>
> How about just "isp4if"? Up to you.
>
If that's the case, do you mind if i keep it, so i can save a lot of
conflict and alignment fix efforts in the subsequent patches in the set, LOL
>> + struct device *dev;
>> + void __iomem *mmio;
>> +
>> + spinlock_t cmdq_lock; /* used for cmdq access */
>> + spinlock_t bufq_lock; /* used for bufq access */
>> + struct mutex isp4if_mutex; /* used to send fw cmd and read fw log */
>> +
>> + struct list_head cmdq; /* commands sent to fw */
>> + struct list_head bufq; /* buffers sent to fw */
>> +
>> + enum isp4if_status status;
>> + u32 host2fw_seq_num;
>> +
>> + /* ISP fw buffers */
>> + struct isp4if_gpu_mem_info *fw_log_buf;
>> + struct isp4if_gpu_mem_info *fw_cmd_resp_buf;
>> + struct isp4if_gpu_mem_info *fw_mem_pool;
>> + struct isp4if_gpu_mem_info *meta_info_buf[ISP4IF_MAX_STREAM_BUF_COUNT];
>> +};
>> +
>> +static inline void isp4if_split_addr64(u64 addr, u32 *lo, u32 *hi)
>> +{
>> + if (lo)
>> + *lo = addr & 0xffffffff;
>> +
>> + if (hi)
>> + *hi = addr >> 32;
>> +}
>> +
>> +static inline u64 isp4if_join_addr64(u32 lo, u32 hi)
>> +{
>> + return (((u64)hi) << 32) | (u64)lo;
>> +}
>> +
>> +int isp4if_f2h_resp(struct isp4_interface *ispif, enum isp4if_stream_id stream,
>> + struct isp4fw_resp *resp);
>> +
>> +int isp4if_send_command(struct isp4_interface *ispif, u32 cmd_id, const void *package,
>> + u32 package_size);
>> +
>> +int isp4if_send_command_sync(struct isp4_interface *ispif, u32 cmd_id, const void *package,
>> + u32 package_size);
>> +
>> +struct isp4if_cmd_element *isp4if_rm_cmd_from_cmdq(struct isp4_interface *ispif, u32 seq_num,
>> + u32 cmd_id);
>> +
>> +void isp4if_clear_cmdq(struct isp4_interface *ispif);
>> +
>> +void isp4if_clear_bufq(struct isp4_interface *ispif);
>> +
>> +void isp4if_dealloc_buffer_node(struct isp4if_img_buf_node *buf_node);
>> +
>> +struct isp4if_img_buf_node *isp4if_alloc_buffer_node(struct isp4if_img_buf_info *buf_info);
>> +
>> +struct isp4if_img_buf_node *isp4if_dequeue_buffer(struct isp4_interface *ispif);
>> +
>> +int isp4if_queue_buffer(struct isp4_interface *ispif, struct isp4if_img_buf_node *buf_node);
>> +
>> +int isp4if_stop(struct isp4_interface *ispif);
>> +
>> +int isp4if_start(struct isp4_interface *ispif);
>> +
>> +int isp4if_deinit(struct isp4_interface *ispif);
>> +
>> +int isp4if_init(struct isp4_interface *ispif, struct device *dev, void __iomem *isp_mmio);
>
> Could you run
>
> $ ./scripts/checkpatch.pl --strict --max-line-length=80
>
> on the set, please?
>
Oh, I will change --max-line-length in our checkpatch.pl from 100 to 80
and fix all the violations. I used to think that the requirement has
been relaxed to 100 columns based on
https://www.phoronix.com/news/Linux-Kernel-Deprecates-80-Col
>> +
>> +#endif /* _ISP4_INTERFACE_H_ */
>
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 3/7] media: platform: amd: Add isp4 fw and hw interface
2026-01-07 8:44 ` Du, Bin
@ 2026-01-14 20:55 ` Sakari Ailus
2026-01-15 2:24 ` Du, Bin
2026-01-20 9:22 ` Du, Bin
1 sibling, 1 reply; 46+ messages in thread
From: Sakari Ailus @ 2026-01-14 20:55 UTC (permalink / raw)
To: Du, Bin
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Svetoslav Stoilov, Alexey Zagorodnikov
Hi Bin,
On Wed, Jan 07, 2026 at 04:44:11PM +0800, Du, Bin wrote:
> > > +int isp4if_init(struct isp4_interface *ispif, struct device *dev, void __iomem *isp_mmio);
> >
> > Could you run
> >
> > $ ./scripts/checkpatch.pl --strict --max-line-length=80
> >
> > on the set, please?
> >
>
> Oh, I will change --max-line-length in our checkpatch.pl from 100 to 80 and
> fix all the violations. I used to think that the requirement has been
> relaxed to 100 columns based on
> https://www.phoronix.com/news/Linux-Kernel-Deprecates-80-Col
The default checkpatch.pl warning has been changed but not the coding
style. There are valid reasons why you might want to have longer lines,
still checkpatch.pl not warning about those isn't one. :-)
--
Regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH v7 3/7] media: platform: amd: Add isp4 fw and hw interface
2026-01-14 20:55 ` Sakari Ailus
@ 2026-01-15 2:24 ` Du, Bin
0 siblings, 0 replies; 46+ messages in thread
From: Du, Bin @ 2026-01-15 2:24 UTC (permalink / raw)
To: Sakari Ailus
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Svetoslav Stoilov, Alexey Zagorodnikov
On 1/15/2026 4:55 AM, Sakari Ailus wrote:
> Hi Bin,
>
> On Wed, Jan 07, 2026 at 04:44:11PM +0800, Du, Bin wrote:
>>>> +int isp4if_init(struct isp4_interface *ispif, struct device *dev, void __iomem *isp_mmio);
>>>
>>> Could you run
>>>
>>> $ ./scripts/checkpatch.pl --strict --max-line-length=80
>>>
>>> on the set, please?
>>>
>>
>> Oh, I will change --max-line-length in our checkpatch.pl from 100 to 80 and
>> fix all the violations. I used to think that the requirement has been
>> relaxed to 100 columns based on
>> https://www.phoronix.com/news/Linux-Kernel-Deprecates-80-Col
>
> The default checkpatch.pl warning has been changed but not the coding
> style. There are valid reasons why you might want to have longer lines,
> still checkpatch.pl not warning about those isn't one. :-)
>
Thank you for the clarification. I will adhere to the 80-character limit
as recommended by the kernel coding style.
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH v7 3/7] media: platform: amd: Add isp4 fw and hw interface
2026-01-07 8:44 ` Du, Bin
2026-01-14 20:55 ` Sakari Ailus
@ 2026-01-20 9:22 ` Du, Bin
2026-01-27 8:38 ` Du, Bin
2026-02-02 9:10 ` Sakari Ailus
1 sibling, 2 replies; 46+ messages in thread
From: Du, Bin @ 2026-01-20 9:22 UTC (permalink / raw)
To: Sakari Ailus
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Alexey Zagorodnikov
Hi Sakari, there's still one left. Could you please help check it?
On 1/7/2026 4:44 PM, Du, Bin wrote:
> Thank you, Sakari for the feedback.
>
> On 12/22/2025 5:37 PM, Sakari Ailus wrote:
>> Hi Bin,
>>
>> On Tue, Dec 16, 2025 at 05:13:22PM +0800, Bin Du wrote:
[snip]
>>> +enum isp4fw_buffer_source {
>>> + /* The buffer is from the stream buffer queue */
>>> + BUFFER_SOURCE_STREAM,
>>> +};
>>
>> Could you also use the ISP4 (or ISP4IF) prefix for these, please? Many
>> look
>> rather generic.
>>
>
> Thank you for highlighting this matter, since these definitions are
> located in isp4_fw_cmd_resp.h, ISP4_FW may be a more appropriate prefix.
> Just to confirm: are you suggesting that we should add this prefix to
> all macros and enums? For example, changing CMD_ID_SET_STREAM_CONFIG to
> ISP4_FW_CMD_ID_SET_STREAM_CONFIG, and BUFFER_SOURCE_STREAM to ISP4_FW
> _BUFFER_SOURCE_STREAM? Our initial thought was that these would only be
> used within ISP and shouldn't lead to any confusion.
>
Hi Sakari, would you please help to confirm so we can decide if further
modification is needed.
[snip]
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 3/7] media: platform: amd: Add isp4 fw and hw interface
2026-01-20 9:22 ` Du, Bin
@ 2026-01-27 8:38 ` Du, Bin
2026-02-02 9:10 ` Sakari Ailus
1 sibling, 0 replies; 46+ messages in thread
From: Du, Bin @ 2026-01-27 8:38 UTC (permalink / raw)
To: Sakari Ailus
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Alexey Zagorodnikov
Hi Sakari, apologies for any inconvenience this may cause. Would you
please confirm the following remaining one item in the patch at your
earliest convenience, so that we can ensure its inclusion in version 8?
On 1/20/2026 5:22 PM, Du, Bin wrote:
> Hi Sakari, there's still one left. Could you please help check it?
>
> On 1/7/2026 4:44 PM, Du, Bin wrote:
>> Thank you, Sakari for the feedback.
>>
>> On 12/22/2025 5:37 PM, Sakari Ailus wrote:
>>> Hi Bin,
>>>
>>> On Tue, Dec 16, 2025 at 05:13:22PM +0800, Bin Du wrote:
>
> [snip]
>
>>>> +enum isp4fw_buffer_source {
>>>> + /* The buffer is from the stream buffer queue */
>>>> + BUFFER_SOURCE_STREAM,
>>>> +};
>>>
>>> Could you also use the ISP4 (or ISP4IF) prefix for these, please?
>>> Many look
>>> rather generic.
>>>
>>
>> Thank you for highlighting this matter, since these definitions are
>> located in isp4_fw_cmd_resp.h, ISP4_FW may be a more appropriate
>> prefix. Just to confirm: are you suggesting that we should add this
>> prefix to all macros and enums? For example, changing
>> CMD_ID_SET_STREAM_CONFIG to ISP4_FW_CMD_ID_SET_STREAM_CONFIG, and
>> BUFFER_SOURCE_STREAM to ISP4_FW _BUFFER_SOURCE_STREAM? Our initial
>> thought was that these would only be used within ISP and shouldn't
>> lead to any confusion.
>>
>
> Hi Sakari, would you please help to confirm so we can decide if further
> modification is needed.
>
> [snip]
>
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 3/7] media: platform: amd: Add isp4 fw and hw interface
2026-01-20 9:22 ` Du, Bin
2026-01-27 8:38 ` Du, Bin
@ 2026-02-02 9:10 ` Sakari Ailus
2026-02-03 6:37 ` Du, Bin
1 sibling, 1 reply; 46+ messages in thread
From: Sakari Ailus @ 2026-02-02 9:10 UTC (permalink / raw)
To: Du, Bin
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Alexey Zagorodnikov
Hi Bin,
On Tue, Jan 20, 2026 at 05:22:41PM +0800, Du, Bin wrote:
> Hi Sakari, there's still one left. Could you please help check it?
>
> On 1/7/2026 4:44 PM, Du, Bin wrote:
> > Thank you, Sakari for the feedback.
> >
> > On 12/22/2025 5:37 PM, Sakari Ailus wrote:
> > > Hi Bin,
> > >
> > > On Tue, Dec 16, 2025 at 05:13:22PM +0800, Bin Du wrote:
>
> [snip]
>
> > > > +enum isp4fw_buffer_source {
> > > > + /* The buffer is from the stream buffer queue */
> > > > + BUFFER_SOURCE_STREAM,
> > > > +};
> > >
> > > Could you also use the ISP4 (or ISP4IF) prefix for these, please?
> > > Many look
> > > rather generic.
> > >
> >
> > Thank you for highlighting this matter, since these definitions are
> > located in isp4_fw_cmd_resp.h, ISP4_FW may be a more appropriate prefix.
> > Just to confirm: are you suggesting that we should add this prefix to
> > all macros and enums? For example, changing CMD_ID_SET_STREAM_CONFIG to
> > ISP4_FW_CMD_ID_SET_STREAM_CONFIG, and BUFFER_SOURCE_STREAM to ISP4_FW
> > _BUFFER_SOURCE_STREAM? Our initial thought was that these would only be
> > used within ISP and shouldn't lead to any confusion.
> >
>
> Hi Sakari, would you please help to confirm so we can decide if further
> modification is needed.
I'd prefer to use a specific prefix, indeed. See e.g.
drivers/media/platform/ti/omap3isp/isp.h . This was the first supported ISP
driver so some prefixes are just "isp_". The shorter names you have, the
larger is the probability of clashing with something generic. It also makes
it obvious to the reader this is specific to the driver.
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 3/7] media: platform: amd: Add isp4 fw and hw interface
2026-02-02 9:10 ` Sakari Ailus
@ 2026-02-03 6:37 ` Du, Bin
0 siblings, 0 replies; 46+ messages in thread
From: Du, Bin @ 2026-02-03 6:37 UTC (permalink / raw)
To: Sakari Ailus
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Alexey Zagorodnikov
Hi Sakari,
On 2/2/2026 5:10 PM, Sakari Ailus wrote:
> Hi Bin,
>
> On Tue, Jan 20, 2026 at 05:22:41PM +0800, Du, Bin wrote:
>> Hi Sakari, there's still one left. Could you please help check it?
>>
>> On 1/7/2026 4:44 PM, Du, Bin wrote:
>>> Thank you, Sakari for the feedback.
>>>
>>> On 12/22/2025 5:37 PM, Sakari Ailus wrote:
>>>> Hi Bin,
>>>>
>>>> On Tue, Dec 16, 2025 at 05:13:22PM +0800, Bin Du wrote:
>>
>> [snip]
>>
>>>>> +enum isp4fw_buffer_source {
>>>>> + /* The buffer is from the stream buffer queue */
>>>>> + BUFFER_SOURCE_STREAM,
>>>>> +};
>>>>
>>>> Could you also use the ISP4 (or ISP4IF) prefix for these, please?
>>>> Many look
>>>> rather generic.
>>>>
>>>
>>> Thank you for highlighting this matter, since these definitions are
>>> located in isp4_fw_cmd_resp.h, ISP4_FW may be a more appropriate prefix.
>>> Just to confirm: are you suggesting that we should add this prefix to
>>> all macros and enums? For example, changing CMD_ID_SET_STREAM_CONFIG to
>>> ISP4_FW_CMD_ID_SET_STREAM_CONFIG, and BUFFER_SOURCE_STREAM to ISP4_FW
>>> _BUFFER_SOURCE_STREAM? Our initial thought was that these would only be
>>> used within ISP and shouldn't lead to any confusion.
>>>
>>
>> Hi Sakari, would you please help to confirm so we can decide if further
>> modification is needed.
>
> I'd prefer to use a specific prefix, indeed. See e.g.
> drivers/media/platform/ti/omap3isp/isp.h . This was the first supported ISP
> driver so some prefixes are just "isp_". The shorter names you have, the
> larger is the probability of clashing with something generic. It also makes
> it obvious to the reader this is specific to the driver.
>
Very good consideration and reference, will add prefix ISP4FW_ to make
them specific.
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread
* [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added
2025-12-16 9:13 [PATCH v7 0/7] Add AMD ISP4 driver Bin Du
` (2 preceding siblings ...)
2025-12-16 9:13 ` [PATCH v7 3/7] media: platform: amd: Add isp4 fw and hw interface Bin Du
@ 2025-12-16 9:13 ` Bin Du
2025-12-22 10:11 ` Sakari Ailus
2026-01-14 9:45 ` Markus Elfring
2025-12-16 9:13 ` [PATCH v7 5/7] media: platform: amd: isp4 video node and buffers " Bin Du
` (5 subsequent siblings)
9 siblings, 2 replies; 46+ messages in thread
From: Bin Du @ 2025-12-16 9:13 UTC (permalink / raw)
To: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
sultan
Cc: pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, bin.du, Bin Du, Svetoslav Stoilov,
Alexey Zagorodnikov
Isp4 sub-device is implementing v4l2 sub-device interface. It has one
capture video node, and supports only preview stream. It manages firmware
states, stream configuration. Add interrupt handling and notification for
isp firmware to isp-subdevice.
Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
Signed-off-by: Bin Du <Bin.Du@amd.com>
Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
---
MAINTAINERS | 2 +
drivers/media/platform/amd/isp4/Makefile | 3 +-
drivers/media/platform/amd/isp4/isp4.c | 99 +-
drivers/media/platform/amd/isp4/isp4.h | 7 +-
drivers/media/platform/amd/isp4/isp4_subdev.c | 975 ++++++++++++++++++
drivers/media/platform/amd/isp4/isp4_subdev.h | 124 +++
6 files changed, 1202 insertions(+), 8 deletions(-)
create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
diff --git a/MAINTAINERS b/MAINTAINERS
index cccae369c876..48ffc8bbdcee 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1149,6 +1149,8 @@ F: drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
F: drivers/media/platform/amd/isp4/isp4_hw_reg.h
F: drivers/media/platform/amd/isp4/isp4_interface.c
F: drivers/media/platform/amd/isp4/isp4_interface.h
+F: drivers/media/platform/amd/isp4/isp4_subdev.c
+F: drivers/media/platform/amd/isp4/isp4_subdev.h
AMD KFD
M: Felix Kuehling <Felix.Kuehling@amd.com>
diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
index a2a5bf98e912..6d4e6d6ac7f5 100644
--- a/drivers/media/platform/amd/isp4/Makefile
+++ b/drivers/media/platform/amd/isp4/Makefile
@@ -4,4 +4,5 @@
obj-$(CONFIG_AMD_ISP4) += amd_capture.o
amd_capture-objs := isp4.o \
- isp4_interface.o
+ isp4_interface.o \
+ isp4_subdev.o
diff --git a/drivers/media/platform/amd/isp4/isp4.c b/drivers/media/platform/amd/isp4/isp4.c
index ad95e7f89189..bcd7cad32afd 100644
--- a/drivers/media/platform/amd/isp4/isp4.c
+++ b/drivers/media/platform/amd/isp4/isp4.c
@@ -3,15 +3,19 @@
* Copyright (C) 2025 Advanced Micro Devices, Inc.
*/
+#include <linux/irq.h>
#include <linux/pm_runtime.h>
#include <linux/vmalloc.h>
+#include <media/v4l2-fwnode.h>
#include <media/v4l2-ioctl.h>
#include "isp4.h"
-
-#define VIDEO_BUF_NUM 5
+#include "isp4_hw_reg.h"
#define ISP4_DRV_NAME "amd_isp_capture"
+#define ISP4_FW_RESP_RB_IRQ_STATUS_MASK \
+ (ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT9_INT_MASK | \
+ ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT12_INT_MASK)
static const struct {
const char *name;
@@ -19,27 +23,97 @@ static const struct {
u32 en_mask;
u32 ack_mask;
u32 rb_int_num;
-} isp4_irq[] = {
+} isp4_irq[ISP4SD_MAX_FW_RESP_STREAM_NUM] = {
/* The IRQ order is aligned with the isp4_subdev.fw_resp_thread order */
{
.name = "isp_irq_global",
+ .status_mask = ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT12_INT_MASK,
+ .en_mask = ISP_SYS_INT0_EN__SYS_INT_RINGBUFFER_WPT12_EN_MASK,
+ .ack_mask = ISP_SYS_INT0_ACK__SYS_INT_RINGBUFFER_WPT12_ACK_MASK,
.rb_int_num = 4, /* ISP_4_1__SRCID__ISP_RINGBUFFER_WPT12 */
},
{
.name = "isp_irq_stream1",
+ .status_mask = ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT9_INT_MASK,
+ .en_mask = ISP_SYS_INT0_EN__SYS_INT_RINGBUFFER_WPT9_EN_MASK,
+ .ack_mask = ISP_SYS_INT0_ACK__SYS_INT_RINGBUFFER_WPT9_ACK_MASK,
.rb_int_num = 0, /* ISP_4_1__SRCID__ISP_RINGBUFFER_WPT9 */
},
};
+void isp4_intr_enable(struct isp4_subdev *isp_subdev, u32 index, bool enable)
+{
+ u32 intr_en;
+
+ /* Synchronize ISP_SYS_INT0_EN writes with the IRQ handler's writes */
+ spin_lock_irq(&isp_subdev->irq_lock);
+ intr_en = isp4hw_rreg(isp_subdev->mmio, ISP_SYS_INT0_EN);
+ if (enable)
+ intr_en |= isp4_irq[index].en_mask;
+ else
+ intr_en &= ~isp4_irq[index].en_mask;
+
+ isp4hw_wreg(isp_subdev->mmio, ISP_SYS_INT0_EN, intr_en);
+ spin_unlock_irq(&isp_subdev->irq_lock);
+}
+
+static void isp4_wake_up_resp_thread(struct isp4_subdev *isp_subdev, u32 index)
+{
+ struct isp4sd_thread_handler *thread_ctx = &isp_subdev->fw_resp_thread[index];
+
+ thread_ctx->resp_ready = true;
+ wake_up_interruptible(&thread_ctx->waitq);
+}
+
static irqreturn_t isp4_irq_handler(int irq, void *arg)
{
+ struct isp4_subdev *isp_subdev = arg;
+ u32 intr_ack = 0, intr_en = 0, intr_status;
+ int seen = 0;
+
+ /* Get the ISP_SYS interrupt status */
+ intr_status = isp4hw_rreg(isp_subdev->mmio, ISP_SYS_INT0_STATUS);
+ intr_status &= ISP4_FW_RESP_RB_IRQ_STATUS_MASK;
+
+ /* Find which ISP_SYS interrupts fired */
+ for (size_t i = 0; i < ARRAY_SIZE(isp4_irq); i++) {
+ if (intr_status & isp4_irq[i].status_mask) {
+ intr_ack |= isp4_irq[i].ack_mask;
+ intr_en |= isp4_irq[i].en_mask;
+ seen |= BIT(i);
+ }
+ }
+
+ /*
+ * Disable the ISP_SYS interrupts that fired. Must be done before waking
+ * the response threads, since they re-enable interrupts when finished.
+ * The lock synchronizes RMW of INT0_EN with isp4_enable_interrupt().
+ */
+ spin_lock(&isp_subdev->irq_lock);
+ intr_en = isp4hw_rreg(isp_subdev->mmio, ISP_SYS_INT0_EN) & ~intr_en;
+ isp4hw_wreg(isp_subdev->mmio, ISP_SYS_INT0_EN, intr_en);
+ spin_unlock(&isp_subdev->irq_lock);
+
+ /*
+ * Clear the ISP_SYS interrupts. This must be done after the interrupts
+ * are disabled, so that ISP FW won't flag any new interrupts on these
+ * streams, and thus we don't need to clear interrupts again before
+ * re-enabling them in the response thread.
+ */
+ isp4hw_wreg(isp_subdev->mmio, ISP_SYS_INT0_ACK, intr_ack);
+
+ /* Wake up the response threads */
+ for (int i; (i = ffs(seen)); seen = (seen >> i) << i)
+ isp4_wake_up_resp_thread(isp_subdev, i - 1);
+
return IRQ_HANDLED;
}
static int isp4_capture_probe(struct platform_device *pdev)
{
+ int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM];
struct device *dev = &pdev->dev;
- int irq[ARRAY_SIZE(isp4_irq)];
+ struct isp4_subdev *isp_subdev;
struct isp4_device *isp_dev;
size_t i;
int ret;
@@ -50,6 +124,11 @@ static int isp4_capture_probe(struct platform_device *pdev)
dev->init_name = ISP4_DRV_NAME;
+ isp_subdev = &isp_dev->isp_subdev;
+ isp_subdev->mmio = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(isp_subdev->mmio))
+ return dev_err_probe(dev, PTR_ERR(isp_subdev->mmio), "isp ioremap fail\n");
+
for (i = 0; i < ARRAY_SIZE(isp4_irq); i++) {
irq[i] = platform_get_irq(pdev, isp4_irq[i].rb_int_num);
if (irq[i] < 0)
@@ -57,7 +136,7 @@ static int isp4_capture_probe(struct platform_device *pdev)
isp4_irq[i].rb_int_num);
ret = devm_request_irq(dev, irq[i], isp4_irq_handler,
- IRQF_NO_AUTOEN, isp4_irq[i].name, dev);
+ IRQF_NO_AUTOEN, isp4_irq[i].name, isp_subdev);
if (ret)
return dev_err_probe(dev, ret, "fail to req irq %d\n", irq[i]);
}
@@ -83,6 +162,13 @@ static int isp4_capture_probe(struct platform_device *pdev)
pm_runtime_set_suspended(dev);
pm_runtime_enable(dev);
+ spin_lock_init(&isp_subdev->irq_lock);
+ ret = isp4sd_init(&isp_dev->isp_subdev, &isp_dev->v4l2_dev, irq);
+ if (ret) {
+ dev_err_probe(dev, ret, "fail init isp4 sub dev\n");
+ goto err_pm_disable;
+ }
+
ret = media_device_register(&isp_dev->mdev);
if (ret) {
dev_err_probe(dev, ret, "fail to register media device\n");
@@ -94,6 +180,8 @@ static int isp4_capture_probe(struct platform_device *pdev)
return 0;
err_isp4_deinit:
+ isp4sd_deinit(&isp_dev->isp_subdev);
+err_pm_disable:
pm_runtime_disable(dev);
v4l2_device_unregister(&isp_dev->v4l2_dev);
err_clean_media:
@@ -108,6 +196,7 @@ static void isp4_capture_remove(struct platform_device *pdev)
struct device *dev = &pdev->dev;
media_device_unregister(&isp_dev->mdev);
+ isp4sd_deinit(&isp_dev->isp_subdev);
pm_runtime_disable(dev);
v4l2_device_unregister(&isp_dev->v4l2_dev);
media_device_cleanup(&isp_dev->mdev);
diff --git a/drivers/media/platform/amd/isp4/isp4.h b/drivers/media/platform/amd/isp4/isp4.h
index 7f2db0dfa2d9..2db6683d6d8b 100644
--- a/drivers/media/platform/amd/isp4/isp4.h
+++ b/drivers/media/platform/amd/isp4/isp4.h
@@ -6,12 +6,15 @@
#ifndef _ISP4_H_
#define _ISP4_H_
-#include <media/v4l2-device.h>
-#include <media/videobuf2-memops.h>
+#include <drm/amd/isp.h>
+#include "isp4_subdev.h"
struct isp4_device {
struct v4l2_device v4l2_dev;
+ struct isp4_subdev isp_subdev;
struct media_device mdev;
};
+void isp4_intr_enable(struct isp4_subdev *isp_subdev, u32 index, bool enable);
+
#endif /* _ISP4_H_ */
diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.c b/drivers/media/platform/amd/isp4/isp4_subdev.c
new file mode 100644
index 000000000000..3a25d1fc49ce
--- /dev/null
+++ b/drivers/media/platform/amd/isp4/isp4_subdev.c
@@ -0,0 +1,975 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2025 Advanced Micro Devices, Inc.
+ */
+
+#include <linux/pm_domain.h>
+#include <linux/units.h>
+
+#include "isp4_fw_cmd_resp.h"
+#include "isp4_interface.h"
+#include "isp4.h"
+
+#define ISP4SD_MIN_BUF_CNT_BEF_START_STREAM 4
+
+#define ISP4SD_PERFORMANCE_STATE_LOW 0
+#define ISP4SD_PERFORMANCE_STATE_HIGH 1
+
+/* align 32KB */
+#define ISP4SD_META_BUF_SIZE ALIGN(sizeof(struct isp4fw_meta_info), 0x8000)
+
+#define to_isp4_subdev(v4l2_sdev) \
+ container_of(v4l2_sdev, struct isp4_subdev, sdev)
+
+static const char *isp4sd_entity_name = "amd isp4";
+
+static const char *isp4sd_thread_name[ISP4SD_MAX_FW_RESP_STREAM_NUM] = {
+ "amd_isp4_thread_global",
+ "amd_isp4_thread_stream1",
+};
+
+static void isp4sd_module_enable(struct isp4_subdev *isp_subdev, bool enable)
+{
+ if (isp_subdev->enable_gpio) {
+ gpiod_set_value(isp_subdev->enable_gpio, enable ? 1 : 0);
+ dev_dbg(isp_subdev->dev, "%s isp_subdev module\n",
+ enable ? "enable" : "disable");
+ }
+}
+
+static int isp4sd_setup_fw_mem_pool(struct isp4_subdev *isp_subdev)
+{
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct isp4fw_cmd_send_buffer buf_type;
+ struct device *dev = isp_subdev->dev;
+ int ret;
+
+ if (!ispif->fw_mem_pool) {
+ dev_err(dev, "fail to alloc mem pool\n");
+ return -ENOMEM;
+ }
+
+ /*
+ * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
+ * zeroed, since this is not guaranteed on all compilers.
+ */
+ memset(&buf_type, 0, sizeof(buf_type));
+ buf_type.buffer_type = BUFFER_TYPE_MEM_POOL;
+ buf_type.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
+ isp4if_split_addr64(ispif->fw_mem_pool->gpu_mc_addr,
+ &buf_type.buffer.buf_base_a_lo,
+ &buf_type.buffer.buf_base_a_hi);
+ buf_type.buffer.buf_size_a = ispif->fw_mem_pool->mem_size;
+
+ ret = isp4if_send_command(ispif, CMD_ID_SEND_BUFFER,
+ &buf_type, sizeof(buf_type));
+ if (ret) {
+ dev_err(dev, "send fw mem pool 0x%llx(%u) fail %d\n",
+ ispif->fw_mem_pool->gpu_mc_addr, buf_type.buffer.buf_size_a, ret);
+ return ret;
+ }
+
+ dev_dbg(dev, "send fw mem pool 0x%llx(%u) suc\n",
+ ispif->fw_mem_pool->gpu_mc_addr, buf_type.buffer.buf_size_a);
+
+ return 0;
+}
+
+static int isp4sd_set_stream_path(struct isp4_subdev *isp_subdev)
+{
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct isp4fw_cmd_set_stream_cfg cmd;
+ struct device *dev = isp_subdev->dev;
+
+ /*
+ * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
+ * zeroed, since this is not guaranteed on all compilers.
+ */
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.stream_cfg.mipi_pipe_path_cfg.isp4fw_sensor_id = SENSOR_ID_ON_MIPI0;
+ cmd.stream_cfg.mipi_pipe_path_cfg.b_enable = true;
+ cmd.stream_cfg.isp_pipe_path_cfg.isp_pipe_id = MIPI0_ISP_PIPELINE_ID;
+
+ cmd.stream_cfg.b_enable_tnr = true;
+ dev_dbg(dev, "isp4fw_sensor_id %d, pipeId 0x%x EnableTnr %u\n",
+ cmd.stream_cfg.mipi_pipe_path_cfg.isp4fw_sensor_id,
+ cmd.stream_cfg.isp_pipe_path_cfg.isp_pipe_id,
+ cmd.stream_cfg.b_enable_tnr);
+
+ return isp4if_send_command(ispif, CMD_ID_SET_STREAM_CONFIG,
+ &cmd, sizeof(cmd));
+}
+
+static int isp4sd_send_meta_buf(struct isp4_subdev *isp_subdev)
+{
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct isp4fw_cmd_send_buffer buf_type;
+ struct device *dev = isp_subdev->dev;
+ int i;
+
+ /*
+ * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
+ * zeroed, since this is not guaranteed on all compilers.
+ */
+ memset(&buf_type, 0, sizeof(buf_type));
+ for (i = 0; i < ISP4IF_MAX_STREAM_BUF_COUNT; i++) {
+ struct isp4if_gpu_mem_info *meta_info_buf =
+ isp_subdev->ispif.meta_info_buf[i];
+ int ret;
+
+ if (!meta_info_buf) {
+ dev_err(dev, "fail for no meta info buf(%u)\n", i);
+ return -ENOMEM;
+ }
+
+ buf_type.buffer_type = BUFFER_TYPE_META_INFO;
+ buf_type.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
+ isp4if_split_addr64(meta_info_buf->gpu_mc_addr,
+ &buf_type.buffer.buf_base_a_lo,
+ &buf_type.buffer.buf_base_a_hi);
+ buf_type.buffer.buf_size_a = meta_info_buf->mem_size;
+ ret = isp4if_send_command(ispif, CMD_ID_SEND_BUFFER,
+ &buf_type, sizeof(buf_type));
+ if (ret) {
+ dev_err(dev, "send meta info(%u) fail\n", i);
+ return ret;
+ }
+ }
+
+ dev_dbg(dev, "send meta info suc\n");
+ return 0;
+}
+
+static bool isp4sd_get_str_out_prop(struct isp4_subdev *isp_subdev,
+ struct isp4fw_image_prop *out_prop,
+ struct v4l2_subdev_state *state, u32 pad)
+{
+ struct device *dev = isp_subdev->dev;
+ struct v4l2_mbus_framefmt *format;
+
+ format = v4l2_subdev_state_get_format(state, pad, 0);
+ if (!format) {
+ dev_err(dev, "fail get subdev state format\n");
+ return false;
+ }
+
+ switch (format->code) {
+ case MEDIA_BUS_FMT_YUYV8_1_5X8:
+ out_prop->image_format = IMAGE_FORMAT_NV12;
+ out_prop->width = format->width;
+ out_prop->height = format->height;
+ out_prop->luma_pitch = format->width;
+ out_prop->chroma_pitch = out_prop->width;
+ break;
+ case MEDIA_BUS_FMT_YUYV8_1X16:
+ out_prop->image_format = IMAGE_FORMAT_YUV422INTERLEAVED;
+ out_prop->width = format->width;
+ out_prop->height = format->height;
+ out_prop->luma_pitch = format->width * 2;
+ out_prop->chroma_pitch = 0;
+ break;
+ default:
+ dev_err(dev, "fail for bad image format:0x%x\n",
+ format->code);
+ return false;
+ }
+
+ if (!out_prop->width || !out_prop->height)
+ return false;
+
+ return true;
+}
+
+static int isp4sd_kickoff_stream(struct isp4_subdev *isp_subdev, u32 w, u32 h)
+{
+ struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct device *dev = isp_subdev->dev;
+
+ if (sensor_info->status == ISP4SD_START_STATUS_STARTED)
+ return 0;
+
+ if (sensor_info->status == ISP4SD_START_STATUS_START_FAIL) {
+ dev_err(dev, "fail for previous start fail\n");
+ return -EINVAL;
+ }
+
+ dev_dbg(dev, "w:%u,h:%u\n", w, h);
+
+ if (isp4sd_send_meta_buf(isp_subdev)) {
+ dev_err(dev, "fail to send meta buf\n");
+ sensor_info->status = ISP4SD_START_STATUS_START_FAIL;
+ return -EINVAL;
+ }
+
+ sensor_info->status = ISP4SD_START_STATUS_OFF;
+
+ if (!sensor_info->start_stream_cmd_sent &&
+ sensor_info->buf_sent_cnt >= ISP4SD_MIN_BUF_CNT_BEF_START_STREAM) {
+ int ret = isp4if_send_command(ispif, CMD_ID_START_STREAM,
+ NULL, 0);
+ if (ret) {
+ dev_err(dev, "fail to start stream\n");
+ return ret;
+ }
+
+ sensor_info->start_stream_cmd_sent = true;
+ } else {
+ dev_dbg(dev,
+ "no send START_STREAM, start_sent %u, buf_sent %u\n",
+ sensor_info->start_stream_cmd_sent,
+ sensor_info->buf_sent_cnt);
+ }
+
+ return 0;
+}
+
+static int isp4sd_setup_output(struct isp4_subdev *isp_subdev,
+ struct v4l2_subdev_state *state, u32 pad)
+{
+ struct isp4sd_output_info *output_info = &isp_subdev->sensor_info.output_info;
+ struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct isp4fw_cmd_set_out_ch_prop cmd_ch_prop;
+ struct isp4fw_cmd_enable_out_ch cmd_ch_en;
+ struct device *dev = isp_subdev->dev;
+ int ret;
+
+ if (output_info->start_status == ISP4SD_START_STATUS_STARTED)
+ return 0;
+
+ if (output_info->start_status == ISP4SD_START_STATUS_START_FAIL) {
+ dev_err(dev, "fail for previous start fail\n");
+ return -EINVAL;
+ }
+
+ /*
+ * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
+ * zeroed, since this is not guaranteed on all compilers.
+ */
+ memset(&cmd_ch_prop, 0, sizeof(cmd_ch_prop));
+ cmd_ch_prop.ch = ISP_PIPE_OUT_CH_PREVIEW;
+
+ if (!isp4sd_get_str_out_prop(isp_subdev, &cmd_ch_prop.image_prop, state, pad)) {
+ dev_err(dev, "fail to get out prop\n");
+ return -EINVAL;
+ }
+
+ dev_dbg(dev, "channel:%d,fmt %d,w:h=%u:%u,lp:%u,cp%u\n",
+ cmd_ch_prop.ch,
+ cmd_ch_prop.image_prop.image_format,
+ cmd_ch_prop.image_prop.width, cmd_ch_prop.image_prop.height,
+ cmd_ch_prop.image_prop.luma_pitch,
+ cmd_ch_prop.image_prop.chroma_pitch);
+
+ ret = isp4if_send_command(ispif, CMD_ID_SET_OUT_CHAN_PROP,
+ &cmd_ch_prop, sizeof(cmd_ch_prop));
+ if (ret) {
+ output_info->start_status = ISP4SD_START_STATUS_START_FAIL;
+ dev_err(dev, "fail to set out prop\n");
+ return ret;
+ }
+
+ /*
+ * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
+ * zeroed, since this is not guaranteed on all compilers.
+ */
+ memset(&cmd_ch_en, 0, sizeof(cmd_ch_en));
+ cmd_ch_en.ch = ISP_PIPE_OUT_CH_PREVIEW;
+ cmd_ch_en.is_enable = true;
+ ret = isp4if_send_command(ispif, CMD_ID_ENABLE_OUT_CHAN,
+ &cmd_ch_en, sizeof(cmd_ch_en));
+ if (ret) {
+ output_info->start_status = ISP4SD_START_STATUS_START_FAIL;
+ dev_err(dev, "fail to enable channel\n");
+ return ret;
+ }
+
+ dev_dbg(dev, "enable channel %d\n", cmd_ch_en.ch);
+
+ if (!sensor_info->start_stream_cmd_sent) {
+ ret = isp4sd_kickoff_stream(isp_subdev,
+ cmd_ch_prop.image_prop.width,
+ cmd_ch_prop.image_prop.height);
+ if (ret) {
+ dev_err(dev, "kickoff stream fail %d\n", ret);
+ return ret;
+ }
+ /*
+ * sensor_info->start_stream_cmd_sent will be set to true
+ * 1. in isp4sd_kickoff_stream, if app first send buffer then
+ * start stream
+ * 2. in isp_set_stream_buf, if app first start stream, then
+ * send buffer
+ * because ISP FW has the requirement, host needs to send buffer
+ * before send start stream cmd
+ */
+ if (sensor_info->start_stream_cmd_sent) {
+ sensor_info->status = ISP4SD_START_STATUS_STARTED;
+ output_info->start_status = ISP4SD_START_STATUS_STARTED;
+ dev_dbg(dev, "kickoff stream suc,start cmd sent\n");
+ }
+ } else {
+ dev_dbg(dev, "stream running, no need kickoff\n");
+ output_info->start_status = ISP4SD_START_STATUS_STARTED;
+ }
+
+ dev_dbg(dev, "setup output suc\n");
+ return 0;
+}
+
+static int isp4sd_init_stream(struct isp4_subdev *isp_subdev)
+{
+ struct device *dev = isp_subdev->dev;
+ int ret;
+
+ ret = isp4sd_setup_fw_mem_pool(isp_subdev);
+ if (ret) {
+ dev_err(dev, "fail to setup fw mem pool\n");
+ return ret;
+ }
+
+ ret = isp4sd_set_stream_path(isp_subdev);
+ if (ret) {
+ dev_err(dev, "fail to setup stream path\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static void isp4sd_uninit_stream(struct isp4_subdev *isp_subdev,
+ struct v4l2_subdev_state *state, u32 pad)
+{
+ struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
+ struct isp4sd_output_info *output_info = &sensor_info->output_info;
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct v4l2_mbus_framefmt *format;
+
+ format = v4l2_subdev_state_get_format(state, pad, 0);
+ if (!format) {
+ dev_err(isp_subdev->dev, "fail to get v4l2 format\n");
+ } else {
+ memset(format, 0, sizeof(*format));
+ format->code = MEDIA_BUS_FMT_YUYV8_1_5X8;
+ }
+
+ isp4if_clear_bufq(ispif);
+ isp4if_clear_cmdq(ispif);
+
+ sensor_info->start_stream_cmd_sent = false;
+ sensor_info->buf_sent_cnt = 0;
+
+ sensor_info->status = ISP4SD_START_STATUS_OFF;
+ output_info->start_status = ISP4SD_START_STATUS_OFF;
+}
+
+static void isp4sd_fw_resp_cmd_done(struct isp4_subdev *isp_subdev,
+ enum isp4if_stream_id stream_id,
+ struct isp4fw_resp_cmd_done *para)
+{
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct isp4if_cmd_element *ele =
+ isp4if_rm_cmd_from_cmdq(ispif, para->cmd_seq_num, para->cmd_id);
+ struct device *dev = isp_subdev->dev;
+
+ dev_dbg(dev, "stream %d,cmd (0x%08x)(%d),seq %u, ele %p\n",
+ stream_id,
+ para->cmd_id, para->cmd_status, para->cmd_seq_num,
+ ele);
+
+ if (ele) {
+ complete(&ele->cmd_done);
+ if (atomic_dec_and_test(&ele->refcnt))
+ kfree(ele);
+ }
+}
+
+static struct isp4fw_meta_info *isp4sd_get_meta_by_mc(struct isp4_subdev *isp_subdev,
+ u64 mc)
+{
+ for (int i = 0; i < ISP4IF_MAX_STREAM_BUF_COUNT; i++) {
+ struct isp4if_gpu_mem_info *meta_info_buf =
+ isp_subdev->ispif.meta_info_buf[i];
+
+ if (meta_info_buf->gpu_mc_addr == mc)
+ return meta_info_buf->sys_addr;
+ }
+
+ return NULL;
+}
+
+static void isp4sd_send_meta_info(struct isp4_subdev *isp_subdev,
+ u64 meta_info_mc)
+{
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct isp4fw_cmd_send_buffer buf_type;
+ struct device *dev = isp_subdev->dev;
+
+ if (isp_subdev->sensor_info.status != ISP4SD_START_STATUS_STARTED) {
+ dev_warn(dev, "not working status %i, meta_info 0x%llx\n",
+ isp_subdev->sensor_info.status, meta_info_mc);
+ return;
+ }
+
+ /*
+ * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
+ * zeroed, since this is not guaranteed on all compilers.
+ */
+ memset(&buf_type, 0, sizeof(buf_type));
+ buf_type.buffer_type = BUFFER_TYPE_META_INFO;
+ buf_type.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
+ isp4if_split_addr64(meta_info_mc,
+ &buf_type.buffer.buf_base_a_lo,
+ &buf_type.buffer.buf_base_a_hi);
+ buf_type.buffer.buf_size_a = ISP4SD_META_BUF_SIZE;
+
+ if (isp4if_send_command(ispif, CMD_ID_SEND_BUFFER,
+ &buf_type, sizeof(buf_type)))
+ dev_err(dev, "fail send meta_info 0x%llx\n",
+ meta_info_mc);
+ else
+ dev_dbg(dev, "resend meta_info 0x%llx\n", meta_info_mc);
+}
+
+static void isp4sd_fw_resp_frame_done(struct isp4_subdev *isp_subdev,
+ enum isp4if_stream_id stream_id,
+ struct isp4fw_resp_param_package *para)
+{
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct device *dev = isp_subdev->dev;
+ struct isp4if_img_buf_node *prev;
+ struct isp4fw_meta_info *meta;
+ u64 mc;
+
+ mc = isp4if_join_addr64(para->package_addr_lo, para->package_addr_hi);
+ meta = isp4sd_get_meta_by_mc(isp_subdev, mc);
+ if (!meta) {
+ dev_err(dev, "fail to get meta from mc %llx\n", mc);
+ return;
+ }
+
+ dev_dbg(dev, "ts:%llu,streamId:%d,poc:%u,preview_en:%u(%i)\n",
+ ktime_get_ns(), stream_id, meta->poc,
+ meta->preview.enabled,
+ meta->preview.status);
+
+ if (meta->preview.enabled &&
+ (meta->preview.status == BUFFER_STATUS_SKIPPED ||
+ meta->preview.status == BUFFER_STATUS_DONE ||
+ meta->preview.status == BUFFER_STATUS_DIRTY)) {
+ prev = isp4if_dequeue_buffer(ispif);
+ if (prev)
+ isp4if_dealloc_buffer_node(prev);
+ else
+ dev_err(dev, "fail null prev buf\n");
+
+ } else if (meta->preview.enabled) {
+ dev_err(dev, "fail bad preview status %u\n",
+ meta->preview.status);
+ }
+
+ if (isp_subdev->sensor_info.status == ISP4SD_START_STATUS_STARTED)
+ isp4sd_send_meta_info(isp_subdev, mc);
+
+ dev_dbg(dev, "stream_id:%d, status:%d\n", stream_id,
+ isp_subdev->sensor_info.status);
+}
+
+static void isp4sd_fw_resp_func(struct isp4_subdev *isp_subdev,
+ enum isp4if_stream_id stream_id)
+{
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct device *dev = isp_subdev->dev;
+ struct isp4fw_resp resp;
+
+ while (true) {
+ if (isp4if_f2h_resp(ispif, stream_id, &resp)) {
+ /* Re-enable the interrupt */
+ isp4_intr_enable(isp_subdev, stream_id, true);
+ /*
+ * Recheck to see if there is a new response.
+ * To ensure that an in-flight interrupt is not lost,
+ * enabling the interrupt must occur _before_ checking
+ * for a new response, hence a memory barrier is needed.
+ * Disable the interrupt again if there was a new response.
+ */
+ mb();
+ if (likely(isp4if_f2h_resp(ispif, stream_id, &resp)))
+ break;
+
+ isp4_intr_enable(isp_subdev, stream_id, false);
+ }
+
+ switch (resp.resp_id) {
+ case RESP_ID_CMD_DONE:
+ isp4sd_fw_resp_cmd_done(isp_subdev, stream_id,
+ &resp.param.cmd_done);
+ break;
+ case RESP_ID_NOTI_FRAME_DONE:
+ isp4sd_fw_resp_frame_done(isp_subdev, stream_id,
+ &resp.param.frame_done);
+ break;
+ default:
+ dev_err(dev, "-><- fail respid (0x%x)\n", resp.resp_id);
+ break;
+ }
+ }
+}
+
+static s32 isp4sd_fw_resp_thread(void *context)
+{
+ struct isp4_subdev_thread_param *para = context;
+ struct isp4_subdev *isp_subdev = para->isp_subdev;
+ struct isp4sd_thread_handler *thread_ctx =
+ &isp_subdev->fw_resp_thread[para->idx];
+ struct device *dev = isp_subdev->dev;
+
+ dev_dbg(dev, "[%u] fw resp thread started\n", para->idx);
+ while (true) {
+ wait_event_interruptible(thread_ctx->waitq, thread_ctx->resp_ready);
+ thread_ctx->resp_ready = false;
+
+ if (kthread_should_stop()) {
+ dev_dbg(dev, "[%u] fw resp thread quit\n", para->idx);
+ break;
+ }
+
+ isp4sd_fw_resp_func(isp_subdev, para->idx);
+ }
+
+ return 0;
+}
+
+static int isp4sd_stop_resp_proc_threads(struct isp4_subdev *isp_subdev)
+{
+ int i;
+
+ for (i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++) {
+ struct isp4sd_thread_handler *thread_ctx =
+ &isp_subdev->fw_resp_thread[i];
+
+ if (thread_ctx->thread) {
+ kthread_stop(thread_ctx->thread);
+ thread_ctx->thread = NULL;
+ }
+ }
+
+ return 0;
+}
+
+static int isp4sd_start_resp_proc_threads(struct isp4_subdev *isp_subdev)
+{
+ struct device *dev = isp_subdev->dev;
+ int i;
+
+ for (i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++) {
+ struct isp4sd_thread_handler *thread_ctx = &isp_subdev->fw_resp_thread[i];
+
+ isp_subdev->isp_resp_para[i].idx = i;
+ isp_subdev->isp_resp_para[i].isp_subdev = isp_subdev;
+ init_waitqueue_head(&thread_ctx->waitq);
+ thread_ctx->resp_ready = false;
+
+ thread_ctx->thread = kthread_run(isp4sd_fw_resp_thread,
+ &isp_subdev->isp_resp_para[i],
+ isp4sd_thread_name[i]);
+ if (IS_ERR(thread_ctx->thread)) {
+ dev_err(dev, "create thread [%d] fail\n", i);
+ thread_ctx->thread = NULL;
+ isp4sd_stop_resp_proc_threads(isp_subdev);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+static int isp4sd_pwroff_and_deinit(struct isp4_subdev *isp_subdev)
+{
+ struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
+ unsigned int perf_state = ISP4SD_PERFORMANCE_STATE_LOW;
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct device *dev = isp_subdev->dev;
+ int ret;
+
+ if (sensor_info->status == ISP4SD_START_STATUS_STARTED) {
+ dev_err(dev, "fail for stream still running\n");
+ return -EINVAL;
+ }
+
+ sensor_info->status = ISP4SD_START_STATUS_OFF;
+
+ if (isp_subdev->irq_enabled) {
+ for (int i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++)
+ disable_irq(isp_subdev->irq[i]);
+ isp_subdev->irq_enabled = false;
+ }
+
+ isp4sd_stop_resp_proc_threads(isp_subdev);
+ dev_dbg(dev, "isp_subdev stop resp proc streads suc");
+
+ isp4if_stop(ispif);
+
+ ret = dev_pm_genpd_set_performance_state(dev, perf_state);
+ if (ret)
+ dev_err(dev, "fail to set isp_subdev performance state %u,ret %d\n",
+ perf_state, ret);
+
+ /* hold ccpu reset */
+ isp4hw_wreg(isp_subdev->mmio, ISP_SOFT_RESET, 0);
+ isp4hw_wreg(isp_subdev->mmio, ISP_POWER_STATUS, 0);
+ ret = pm_runtime_put_sync(dev);
+ if (ret)
+ dev_err(dev, "power off isp_subdev fail %d\n", ret);
+ else
+ dev_dbg(dev, "power off isp_subdev suc\n");
+
+ ispif->status = ISP4IF_STATUS_PWR_OFF;
+ isp4if_clear_cmdq(ispif);
+ isp4sd_module_enable(isp_subdev, false);
+
+ /*
+ * When opening the camera, isp4sd_module_enable(isp_subdev, true) is called.
+ * Hardware requires at least a 20ms delay between disabling and enabling the module,
+ * so a sleep is added to ensure ISP stability during quick reopen scenarios.
+ */
+ msleep(20);
+
+ return 0;
+}
+
+static int isp4sd_pwron_and_init(struct isp4_subdev *isp_subdev)
+{
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct device *dev = isp_subdev->dev;
+ int ret;
+
+ if (ispif->status == ISP4IF_STATUS_FW_RUNNING) {
+ dev_dbg(dev, "camera already opened, do nothing\n");
+ return 0;
+ }
+
+ isp4sd_module_enable(isp_subdev, true);
+
+ if (ispif->status < ISP4IF_STATUS_PWR_ON) {
+ unsigned int perf_state = ISP4SD_PERFORMANCE_STATE_HIGH;
+
+ ret = pm_runtime_resume_and_get(dev);
+ if (ret) {
+ dev_err(dev, "fail to power on isp_subdev ret %d\n",
+ ret);
+ goto err_deinit;
+ }
+
+ /* ISPPG ISP Power Status */
+ isp4hw_wreg(isp_subdev->mmio, ISP_POWER_STATUS, 0x7FF);
+ ret = dev_pm_genpd_set_performance_state(dev, perf_state);
+ if (ret) {
+ dev_err(dev,
+ "fail to set performance state %u, ret %d\n",
+ perf_state, ret);
+ goto err_deinit;
+ }
+
+ ispif->status = ISP4IF_STATUS_PWR_ON;
+ }
+
+ isp_subdev->sensor_info.start_stream_cmd_sent = false;
+ isp_subdev->sensor_info.buf_sent_cnt = 0;
+
+ ret = isp4if_start(ispif);
+ if (ret) {
+ dev_err(dev, "fail to start isp_subdev interface\n");
+ goto err_deinit;
+ }
+
+ if (isp4sd_start_resp_proc_threads(isp_subdev)) {
+ dev_err(dev, "isp_start_resp_proc_threads fail");
+ goto err_deinit;
+ }
+
+ dev_dbg(dev, "create resp threads ok");
+
+ for (int i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++)
+ enable_irq(isp_subdev->irq[i]);
+ isp_subdev->irq_enabled = true;
+
+ return 0;
+err_deinit:
+ isp4sd_pwroff_and_deinit(isp_subdev);
+ return -EINVAL;
+}
+
+static int isp4sd_stop_stream(struct isp4_subdev *isp_subdev,
+ struct v4l2_subdev_state *state, u32 pad)
+{
+ struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
+ struct isp4sd_output_info *output_info = &sensor_info->output_info;
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct device *dev = isp_subdev->dev;
+
+ guard(mutex)(&isp_subdev->ops_mutex);
+ dev_dbg(dev, "status %i\n", output_info->start_status);
+
+ if (output_info->start_status == ISP4SD_START_STATUS_STARTED) {
+ struct isp4fw_cmd_enable_out_ch cmd_ch_disable;
+ int ret;
+
+ /*
+ * The struct will be shared with ISP FW, use memset() to guarantee
+ * padding bits are zeroed, since this is not guaranteed on all compilers.
+ */
+ memset(&cmd_ch_disable, 0, sizeof(cmd_ch_disable));
+ cmd_ch_disable.ch = ISP_PIPE_OUT_CH_PREVIEW;
+ /* `cmd_ch_disable.is_enable` is already false */
+ ret = isp4if_send_command_sync(ispif, CMD_ID_ENABLE_OUT_CHAN,
+ &cmd_ch_disable,
+ sizeof(cmd_ch_disable));
+ if (ret)
+ dev_err(dev, "fail to disable stream\n");
+ else
+ dev_dbg(dev, "wait disable stream suc\n");
+
+ ret = isp4if_send_command_sync(ispif, CMD_ID_STOP_STREAM,
+ NULL, 0);
+ if (ret)
+ dev_err(dev, "fail to stop steam\n");
+ else
+ dev_dbg(dev, "wait stop stream suc\n");
+ }
+
+ isp4sd_uninit_stream(isp_subdev, state, pad);
+
+ /*
+ * Return success to ensure the stop process proceeds,
+ * and disregard any errors since they are not fatal.
+ */
+ return 0;
+}
+
+static int isp4sd_start_stream(struct isp4_subdev *isp_subdev,
+ struct v4l2_subdev_state *state, u32 pad)
+{
+ struct isp4sd_output_info *output_info =
+ &isp_subdev->sensor_info.output_info;
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct device *dev = isp_subdev->dev;
+ int ret;
+
+ guard(mutex)(&isp_subdev->ops_mutex);
+
+ if (ispif->status != ISP4IF_STATUS_FW_RUNNING) {
+ dev_err(dev, "fail, bad fsm %d", ispif->status);
+ return -EINVAL;
+ }
+
+ switch (output_info->start_status) {
+ case ISP4SD_START_STATUS_OFF:
+ break;
+ case ISP4SD_START_STATUS_STARTED:
+ dev_dbg(dev, "stream already started, do nothing\n");
+ return 0;
+ case ISP4SD_START_STATUS_START_FAIL:
+ dev_err(dev, "stream previously failed to start\n");
+ return -EINVAL;
+ }
+
+ ret = isp4sd_init_stream(isp_subdev);
+ if (ret) {
+ dev_err(dev, "fail to init isp_subdev stream\n");
+ goto err_stop_stream;
+ }
+
+ ret = isp4sd_setup_output(isp_subdev, state, pad);
+ if (ret) {
+ dev_err(dev, "fail to setup output\n");
+ goto err_stop_stream;
+ }
+
+ return 0;
+
+err_stop_stream:
+ isp4sd_stop_stream(isp_subdev, state, pad);
+ return ret;
+}
+
+static int isp4sd_set_power(struct v4l2_subdev *sd, int on)
+{
+ struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
+
+ guard(mutex)(&isp_subdev->ops_mutex);
+ if (on)
+ return isp4sd_pwron_and_init(isp_subdev);
+ else
+ return isp4sd_pwroff_and_deinit(isp_subdev);
+}
+
+static const struct v4l2_subdev_core_ops isp4sd_core_ops = {
+ .s_power = isp4sd_set_power,
+};
+
+static const struct v4l2_subdev_video_ops isp4sd_video_ops = {
+ .s_stream = v4l2_subdev_s_stream_helper,
+};
+
+static int isp4sd_set_pad_format(struct v4l2_subdev *sd,
+ struct v4l2_subdev_state *sd_state,
+ struct v4l2_subdev_format *fmt)
+{
+ struct isp4sd_output_info *steam_info =
+ &(to_isp4_subdev(sd)->sensor_info.output_info);
+ struct v4l2_mbus_framefmt *format;
+
+ format = v4l2_subdev_state_get_format(sd_state, fmt->pad);
+
+ if (!format) {
+ dev_err(sd->dev, "fail to get state format\n");
+ return -EINVAL;
+ }
+
+ *format = fmt->format;
+ switch (format->code) {
+ case MEDIA_BUS_FMT_YUYV8_1_5X8:
+ steam_info->image_size = format->width * format->height * 3 / 2;
+ break;
+ case MEDIA_BUS_FMT_YUYV8_1X16:
+ steam_info->image_size = format->width * format->height * 2;
+ break;
+ default:
+ steam_info->image_size = 0;
+ break;
+ }
+ if (!steam_info->image_size) {
+ dev_err(sd->dev, "fail set pad format,code 0x%x,width %u, height %u\n",
+ format->code, format->width, format->height);
+ return -EINVAL;
+ }
+
+ dev_dbg(sd->dev, "set pad format suc, code:%x w:%u h:%u size:%u\n",
+ format->code, format->width, format->height, steam_info->image_size);
+
+ return 0;
+}
+
+static int isp4sd_enable_streams(struct v4l2_subdev *sd,
+ struct v4l2_subdev_state *state, u32 pad,
+ u64 streams_mask)
+{
+ struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
+
+ return isp4sd_start_stream(isp_subdev, state, pad);
+}
+
+static int isp4sd_disable_streams(struct v4l2_subdev *sd,
+ struct v4l2_subdev_state *state, u32 pad,
+ u64 streams_mask)
+{
+ struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
+
+ return isp4sd_stop_stream(isp_subdev, state, pad);
+}
+
+static const struct v4l2_subdev_pad_ops isp4sd_pad_ops = {
+ .get_fmt = v4l2_subdev_get_fmt,
+ .set_fmt = isp4sd_set_pad_format,
+ .enable_streams = isp4sd_enable_streams,
+ .disable_streams = isp4sd_disable_streams,
+};
+
+static const struct v4l2_subdev_ops isp4sd_subdev_ops = {
+ .core = &isp4sd_core_ops,
+ .video = &isp4sd_video_ops,
+ .pad = &isp4sd_pad_ops,
+};
+
+static int isp4sd_sdev_link_validate(struct media_link *link)
+{
+ return 0;
+}
+
+static const struct media_entity_operations isp4sd_sdev_ent_ops = {
+ .link_validate = isp4sd_sdev_link_validate,
+};
+
+int isp4sd_init(struct isp4_subdev *isp_subdev, struct v4l2_device *v4l2_dev,
+ int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM])
+{
+ struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct device *dev = v4l2_dev->dev;
+ int ret;
+
+ isp_subdev->dev = dev;
+ v4l2_subdev_init(&isp_subdev->sdev, &isp4sd_subdev_ops);
+ isp_subdev->sdev.owner = THIS_MODULE;
+ isp_subdev->sdev.dev = dev;
+ snprintf(isp_subdev->sdev.name, sizeof(isp_subdev->sdev.name), "%s",
+ dev_name(dev));
+
+ isp_subdev->sdev.entity.name = isp4sd_entity_name;
+ isp_subdev->sdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
+ isp_subdev->sdev.entity.ops = &isp4sd_sdev_ent_ops;
+ isp_subdev->sdev_pad.flags = MEDIA_PAD_FL_SOURCE;
+ ret = media_entity_pads_init(&isp_subdev->sdev.entity, 1,
+ &isp_subdev->sdev_pad);
+ if (ret) {
+ dev_err(dev, "fail to init isp4 subdev entity pad %d\n", ret);
+ return ret;
+ }
+
+ ret = v4l2_subdev_init_finalize(&isp_subdev->sdev);
+ if (ret < 0) {
+ dev_err(dev, "fail to init finalize isp4 subdev %d\n",
+ ret);
+ return ret;
+ }
+
+ ret = v4l2_device_register_subdev(v4l2_dev, &isp_subdev->sdev);
+ if (ret) {
+ dev_err(dev, "fail to register isp4 subdev to V4L2 device %d\n", ret);
+ goto err_media_clean_up;
+ }
+
+ isp4if_init(ispif, dev, isp_subdev->mmio);
+
+ mutex_init(&isp_subdev->ops_mutex);
+ sensor_info->status = ISP4SD_START_STATUS_OFF;
+
+ /* create ISP enable gpio control */
+ isp_subdev->enable_gpio = devm_gpiod_get(isp_subdev->dev,
+ "enable_isp",
+ GPIOD_OUT_LOW);
+ if (IS_ERR(isp_subdev->enable_gpio)) {
+ ret = PTR_ERR(isp_subdev->enable_gpio);
+ dev_err(dev, "fail to get gpiod %d\n", ret);
+ goto err_subdev_unreg;
+ }
+
+ for (int i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++)
+ isp_subdev->irq[i] = irq[i];
+
+ isp_subdev->host2fw_seq_num = 1;
+ ispif->status = ISP4IF_STATUS_PWR_OFF;
+
+ return 0;
+
+err_subdev_unreg:
+ v4l2_device_unregister_subdev(&isp_subdev->sdev);
+err_media_clean_up:
+ v4l2_subdev_cleanup(&isp_subdev->sdev);
+ media_entity_cleanup(&isp_subdev->sdev.entity);
+ return ret;
+}
+
+void isp4sd_deinit(struct isp4_subdev *isp_subdev)
+{
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+
+ v4l2_device_unregister_subdev(&isp_subdev->sdev);
+ media_entity_cleanup(&isp_subdev->sdev.entity);
+ isp4if_deinit(ispif);
+ isp4sd_module_enable(isp_subdev, false);
+
+ ispif->status = ISP4IF_STATUS_PWR_OFF;
+}
diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.h b/drivers/media/platform/amd/isp4/isp4_subdev.h
new file mode 100644
index 000000000000..70007b9de3f3
--- /dev/null
+++ b/drivers/media/platform/amd/isp4/isp4_subdev.h
@@ -0,0 +1,124 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (C) 2025 Advanced Micro Devices, Inc.
+ */
+
+#ifndef _ISP4_SUBDEV_H_
+#define _ISP4_SUBDEV_H_
+
+#include <linux/debugfs.h>
+#include <linux/delay.h>
+#include <linux/firmware.h>
+#include <linux/pm_runtime.h>
+#include <linux/types.h>
+#include <linux/uaccess.h>
+#include <media/v4l2-device.h>
+
+#include "isp4_fw_cmd_resp.h"
+#include "isp4_hw_reg.h"
+#include "isp4_interface.h"
+
+/*
+ * one is for none sesnor specefic response which is not used now
+ * another is for sensor specific response
+ */
+#define ISP4SD_MAX_FW_RESP_STREAM_NUM 2
+
+/*
+ * cmd used to register frame done callback, parameter is
+ * struct isp4sd_register_framedone_cb_param *
+ * when a image buffer is filled by ISP, ISP will call the registered callback.
+ * callback func prototype is isp4sd_framedone_cb, cb_ctx can be anything
+ * provided by caller which will be provided back as the first parameter of the
+ * callback function.
+ * both cb_func and cb_ctx are provide by caller, set cb_func to NULL to
+ * unregister the callback
+ */
+
+/* used to indicate the ISP status */
+enum isp4sd_status {
+ ISP4SD_STATUS_PWR_OFF,
+ ISP4SD_STATUS_PWR_ON,
+ ISP4SD_STATUS_FW_RUNNING,
+ ISP4SD_STATUS_MAX
+};
+
+/* used to indicate the status of sensor, output stream */
+enum isp4sd_start_status {
+ ISP4SD_START_STATUS_OFF,
+ ISP4SD_START_STATUS_STARTED,
+ ISP4SD_START_STATUS_START_FAIL,
+};
+
+struct isp4sd_img_buf_node {
+ struct list_head node;
+ struct isp4if_img_buf_info buf_info;
+};
+
+/* this is isp output after processing bayer raw input from sensor */
+struct isp4sd_output_info {
+ enum isp4sd_start_status start_status;
+ u32 image_size;
+};
+
+/*
+ * This struct represents the sensor info which is input or source of ISP,
+ * status is the sensor status
+ * output_info is the isp output info after ISP processing the sensor input,
+ * start_stream_cmd_sent mean if CMD_ID_START_STREAM has sent to fw.
+ * buf_sent_cnt is buffer count app has sent to receive the images
+ */
+struct isp4sd_sensor_info {
+ struct isp4sd_output_info output_info;
+ enum isp4sd_start_status status;
+ bool start_stream_cmd_sent;
+ u32 buf_sent_cnt;
+};
+
+/*
+ * Thread created by driver to receive fw response
+ * thread will be wakeup by fw to driver response interrupt
+ */
+struct isp4sd_thread_handler {
+ struct task_struct *thread;
+ wait_queue_head_t waitq;
+ bool resp_ready;
+};
+
+struct isp4_subdev_thread_param {
+ u32 idx;
+ struct isp4_subdev *isp_subdev;
+};
+
+struct isp4_subdev {
+ struct v4l2_subdev sdev;
+ struct isp4_interface ispif;
+
+ struct media_pad sdev_pad;
+
+ enum isp4sd_status isp_status;
+ struct mutex ops_mutex; /* ops_mutex */
+
+ /* Used to store fw cmds sent to FW whose response driver needs to wait for */
+ struct isp4sd_thread_handler fw_resp_thread[ISP4SD_MAX_FW_RESP_STREAM_NUM];
+
+ u32 host2fw_seq_num;
+
+ struct isp4sd_sensor_info sensor_info;
+
+ /* gpio descriptor */
+ struct gpio_desc *enable_gpio;
+ struct device *dev;
+ void __iomem *mmio;
+ struct isp4_subdev_thread_param isp_resp_para[ISP4SD_MAX_FW_RESP_STREAM_NUM];
+ int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM];
+ bool irq_enabled;
+ /* spin lock to access ISP_SYS_INT0_EN exclusively */
+ spinlock_t irq_lock;
+};
+
+int isp4sd_init(struct isp4_subdev *isp_subdev, struct v4l2_device *v4l2_dev,
+ int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM]);
+void isp4sd_deinit(struct isp4_subdev *isp_subdev);
+
+#endif /* _ISP4_SUBDEV_H_ */
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread* Re: [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added
2025-12-16 9:13 ` [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added Bin Du
@ 2025-12-22 10:11 ` Sakari Ailus
2026-01-07 7:33 ` Sultan Alsawaf
2026-01-14 10:34 ` Du, Bin
2026-01-14 9:45 ` Markus Elfring
1 sibling, 2 replies; 46+ messages in thread
From: Sakari Ailus @ 2025-12-22 10:11 UTC (permalink / raw)
To: Bin Du
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Svetoslav Stoilov, Alexey Zagorodnikov
Hi Bin,
On Tue, Dec 16, 2025 at 05:13:23PM +0800, Bin Du wrote:
> Isp4 sub-device is implementing v4l2 sub-device interface. It has one
> capture video node, and supports only preview stream. It manages firmware
> states, stream configuration. Add interrupt handling and notification for
> isp firmware to isp-subdevice.
>
> Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
> Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
> Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
> Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
> Signed-off-by: Bin Du <Bin.Du@amd.com>
> Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
> ---
> MAINTAINERS | 2 +
> drivers/media/platform/amd/isp4/Makefile | 3 +-
> drivers/media/platform/amd/isp4/isp4.c | 99 +-
> drivers/media/platform/amd/isp4/isp4.h | 7 +-
> drivers/media/platform/amd/isp4/isp4_subdev.c | 975 ++++++++++++++++++
> drivers/media/platform/amd/isp4/isp4_subdev.h | 124 +++
> 6 files changed, 1202 insertions(+), 8 deletions(-)
> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index cccae369c876..48ffc8bbdcee 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1149,6 +1149,8 @@ F: drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
> F: drivers/media/platform/amd/isp4/isp4_hw_reg.h
> F: drivers/media/platform/amd/isp4/isp4_interface.c
> F: drivers/media/platform/amd/isp4/isp4_interface.h
> +F: drivers/media/platform/amd/isp4/isp4_subdev.c
> +F: drivers/media/platform/amd/isp4/isp4_subdev.h
>
> AMD KFD
> M: Felix Kuehling <Felix.Kuehling@amd.com>
> diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
> index a2a5bf98e912..6d4e6d6ac7f5 100644
> --- a/drivers/media/platform/amd/isp4/Makefile
> +++ b/drivers/media/platform/amd/isp4/Makefile
> @@ -4,4 +4,5 @@
>
> obj-$(CONFIG_AMD_ISP4) += amd_capture.o
> amd_capture-objs := isp4.o \
> - isp4_interface.o
> + isp4_interface.o \
> + isp4_subdev.o
> diff --git a/drivers/media/platform/amd/isp4/isp4.c b/drivers/media/platform/amd/isp4/isp4.c
> index ad95e7f89189..bcd7cad32afd 100644
> --- a/drivers/media/platform/amd/isp4/isp4.c
> +++ b/drivers/media/platform/amd/isp4/isp4.c
> @@ -3,15 +3,19 @@
> * Copyright (C) 2025 Advanced Micro Devices, Inc.
> */
>
> +#include <linux/irq.h>
> #include <linux/pm_runtime.h>
> #include <linux/vmalloc.h>
> +#include <media/v4l2-fwnode.h>
I don't think you need this one.
> #include <media/v4l2-ioctl.h>
>
> #include "isp4.h"
> -
> -#define VIDEO_BUF_NUM 5
What happened with this one? Is it no longer needed?
> +#include "isp4_hw_reg.h"
>
> #define ISP4_DRV_NAME "amd_isp_capture"
> +#define ISP4_FW_RESP_RB_IRQ_STATUS_MASK \
> + (ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT9_INT_MASK | \
> + ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT12_INT_MASK)
>
> static const struct {
> const char *name;
> @@ -19,27 +23,97 @@ static const struct {
> u32 en_mask;
> u32 ack_mask;
> u32 rb_int_num;
> -} isp4_irq[] = {
> +} isp4_irq[ISP4SD_MAX_FW_RESP_STREAM_NUM] = {
> /* The IRQ order is aligned with the isp4_subdev.fw_resp_thread order */
> {
> .name = "isp_irq_global",
> + .status_mask = ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT12_INT_MASK,
> + .en_mask = ISP_SYS_INT0_EN__SYS_INT_RINGBUFFER_WPT12_EN_MASK,
> + .ack_mask = ISP_SYS_INT0_ACK__SYS_INT_RINGBUFFER_WPT12_ACK_MASK,
> .rb_int_num = 4, /* ISP_4_1__SRCID__ISP_RINGBUFFER_WPT12 */
> },
> {
> .name = "isp_irq_stream1",
> + .status_mask = ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT9_INT_MASK,
> + .en_mask = ISP_SYS_INT0_EN__SYS_INT_RINGBUFFER_WPT9_EN_MASK,
> + .ack_mask = ISP_SYS_INT0_ACK__SYS_INT_RINGBUFFER_WPT9_ACK_MASK,
> .rb_int_num = 0, /* ISP_4_1__SRCID__ISP_RINGBUFFER_WPT9 */
> },
> };
>
> +void isp4_intr_enable(struct isp4_subdev *isp_subdev, u32 index, bool enable)
> +{
> + u32 intr_en;
> +
> + /* Synchronize ISP_SYS_INT0_EN writes with the IRQ handler's writes */
> + spin_lock_irq(&isp_subdev->irq_lock);
> + intr_en = isp4hw_rreg(isp_subdev->mmio, ISP_SYS_INT0_EN);
> + if (enable)
> + intr_en |= isp4_irq[index].en_mask;
> + else
> + intr_en &= ~isp4_irq[index].en_mask;
> +
> + isp4hw_wreg(isp_subdev->mmio, ISP_SYS_INT0_EN, intr_en);
> + spin_unlock_irq(&isp_subdev->irq_lock);
> +}
> +
> +static void isp4_wake_up_resp_thread(struct isp4_subdev *isp_subdev, u32 index)
> +{
> + struct isp4sd_thread_handler *thread_ctx = &isp_subdev->fw_resp_thread[index];
> +
> + thread_ctx->resp_ready = true;
> + wake_up_interruptible(&thread_ctx->waitq);
> +}
> +
> static irqreturn_t isp4_irq_handler(int irq, void *arg)
> {
> + struct isp4_subdev *isp_subdev = arg;
> + u32 intr_ack = 0, intr_en = 0, intr_status;
> + int seen = 0;
Is int appropriate here? Should this be u32 or u64?
> +
> + /* Get the ISP_SYS interrupt status */
> + intr_status = isp4hw_rreg(isp_subdev->mmio, ISP_SYS_INT0_STATUS);
> + intr_status &= ISP4_FW_RESP_RB_IRQ_STATUS_MASK;
> +
> + /* Find which ISP_SYS interrupts fired */
> + for (size_t i = 0; i < ARRAY_SIZE(isp4_irq); i++) {
> + if (intr_status & isp4_irq[i].status_mask) {
> + intr_ack |= isp4_irq[i].ack_mask;
> + intr_en |= isp4_irq[i].en_mask;
> + seen |= BIT(i);
> + }
> + }
> +
> + /*
> + * Disable the ISP_SYS interrupts that fired. Must be done before waking
> + * the response threads, since they re-enable interrupts when finished.
> + * The lock synchronizes RMW of INT0_EN with isp4_enable_interrupt().
> + */
> + spin_lock(&isp_subdev->irq_lock);
> + intr_en = isp4hw_rreg(isp_subdev->mmio, ISP_SYS_INT0_EN) & ~intr_en;
> + isp4hw_wreg(isp_subdev->mmio, ISP_SYS_INT0_EN, intr_en);
> + spin_unlock(&isp_subdev->irq_lock);
> +
> + /*
> + * Clear the ISP_SYS interrupts. This must be done after the interrupts
> + * are disabled, so that ISP FW won't flag any new interrupts on these
> + * streams, and thus we don't need to clear interrupts again before
> + * re-enabling them in the response thread.
> + */
> + isp4hw_wreg(isp_subdev->mmio, ISP_SYS_INT0_ACK, intr_ack);
> +
> + /* Wake up the response threads */
> + for (int i; (i = ffs(seen)); seen = (seen >> i) << i)
unsigned int, please. The parentheses around ffs() appear redundant.
The increment could probably be expressed as seen &= ~BIT(i).
> + isp4_wake_up_resp_thread(isp_subdev, i - 1);
> +
> return IRQ_HANDLED;
> }
>
> static int isp4_capture_probe(struct platform_device *pdev)
> {
> + int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM];
> struct device *dev = &pdev->dev;
> - int irq[ARRAY_SIZE(isp4_irq)];
> + struct isp4_subdev *isp_subdev;
> struct isp4_device *isp_dev;
> size_t i;
> int ret;
> @@ -50,6 +124,11 @@ static int isp4_capture_probe(struct platform_device *pdev)
>
> dev->init_name = ISP4_DRV_NAME;
>
> + isp_subdev = &isp_dev->isp_subdev;
> + isp_subdev->mmio = devm_platform_ioremap_resource(pdev, 0);
> + if (IS_ERR(isp_subdev->mmio))
> + return dev_err_probe(dev, PTR_ERR(isp_subdev->mmio), "isp ioremap fail\n");
> +
> for (i = 0; i < ARRAY_SIZE(isp4_irq); i++) {
> irq[i] = platform_get_irq(pdev, isp4_irq[i].rb_int_num);
> if (irq[i] < 0)
> @@ -57,7 +136,7 @@ static int isp4_capture_probe(struct platform_device *pdev)
> isp4_irq[i].rb_int_num);
>
> ret = devm_request_irq(dev, irq[i], isp4_irq_handler,
> - IRQF_NO_AUTOEN, isp4_irq[i].name, dev);
> + IRQF_NO_AUTOEN, isp4_irq[i].name, isp_subdev);
> if (ret)
> return dev_err_probe(dev, ret, "fail to req irq %d\n", irq[i]);
> }
> @@ -83,6 +162,13 @@ static int isp4_capture_probe(struct platform_device *pdev)
>
> pm_runtime_set_suspended(dev);
> pm_runtime_enable(dev);
> + spin_lock_init(&isp_subdev->irq_lock);
> + ret = isp4sd_init(&isp_dev->isp_subdev, &isp_dev->v4l2_dev, irq);
> + if (ret) {
> + dev_err_probe(dev, ret, "fail init isp4 sub dev\n");
> + goto err_pm_disable;
> + }
> +
> ret = media_device_register(&isp_dev->mdev);
> if (ret) {
> dev_err_probe(dev, ret, "fail to register media device\n");
> @@ -94,6 +180,8 @@ static int isp4_capture_probe(struct platform_device *pdev)
> return 0;
>
> err_isp4_deinit:
> + isp4sd_deinit(&isp_dev->isp_subdev);
> +err_pm_disable:
> pm_runtime_disable(dev);
> v4l2_device_unregister(&isp_dev->v4l2_dev);
> err_clean_media:
> @@ -108,6 +196,7 @@ static void isp4_capture_remove(struct platform_device *pdev)
> struct device *dev = &pdev->dev;
>
> media_device_unregister(&isp_dev->mdev);
> + isp4sd_deinit(&isp_dev->isp_subdev);
> pm_runtime_disable(dev);
> v4l2_device_unregister(&isp_dev->v4l2_dev);
> media_device_cleanup(&isp_dev->mdev);
> diff --git a/drivers/media/platform/amd/isp4/isp4.h b/drivers/media/platform/amd/isp4/isp4.h
> index 7f2db0dfa2d9..2db6683d6d8b 100644
> --- a/drivers/media/platform/amd/isp4/isp4.h
> +++ b/drivers/media/platform/amd/isp4/isp4.h
> @@ -6,12 +6,15 @@
> #ifndef _ISP4_H_
> #define _ISP4_H_
>
> -#include <media/v4l2-device.h>
> -#include <media/videobuf2-memops.h>
> +#include <drm/amd/isp.h>
> +#include "isp4_subdev.h"
>
> struct isp4_device {
> struct v4l2_device v4l2_dev;
> + struct isp4_subdev isp_subdev;
> struct media_device mdev;
> };
>
> +void isp4_intr_enable(struct isp4_subdev *isp_subdev, u32 index, bool enable);
> +
> #endif /* _ISP4_H_ */
> diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.c b/drivers/media/platform/amd/isp4/isp4_subdev.c
> new file mode 100644
> index 000000000000..3a25d1fc49ce
> --- /dev/null
> +++ b/drivers/media/platform/amd/isp4/isp4_subdev.c
> @@ -0,0 +1,975 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
> + */
> +
> +#include <linux/pm_domain.h>
> +#include <linux/units.h>
> +
> +#include "isp4_fw_cmd_resp.h"
> +#include "isp4_interface.h"
> +#include "isp4.h"
> +
> +#define ISP4SD_MIN_BUF_CNT_BEF_START_STREAM 4
> +
> +#define ISP4SD_PERFORMANCE_STATE_LOW 0
> +#define ISP4SD_PERFORMANCE_STATE_HIGH 1
> +
> +/* align 32KB */
> +#define ISP4SD_META_BUF_SIZE ALIGN(sizeof(struct isp4fw_meta_info), 0x8000)
> +
> +#define to_isp4_subdev(v4l2_sdev) \
Virtually always variables referring to a sub-device are called either sd
or subdev (or variants of these). Up to you.
> + container_of(v4l2_sdev, struct isp4_subdev, sdev)
> +
> +static const char *isp4sd_entity_name = "amd isp4";
> +
> +static const char *isp4sd_thread_name[ISP4SD_MAX_FW_RESP_STREAM_NUM] = {
> + "amd_isp4_thread_global",
> + "amd_isp4_thread_stream1",
> +};
> +
> +static void isp4sd_module_enable(struct isp4_subdev *isp_subdev, bool enable)
> +{
> + if (isp_subdev->enable_gpio) {
> + gpiod_set_value(isp_subdev->enable_gpio, enable ? 1 : 0);
> + dev_dbg(isp_subdev->dev, "%s isp_subdev module\n",
> + enable ? "enable" : "disable");
> + }
> +}
> +
> +static int isp4sd_setup_fw_mem_pool(struct isp4_subdev *isp_subdev)
> +{
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct isp4fw_cmd_send_buffer buf_type;
> + struct device *dev = isp_subdev->dev;
> + int ret;
> +
> + if (!ispif->fw_mem_pool) {
> + dev_err(dev, "fail to alloc mem pool\n");
> + return -ENOMEM;
> + }
> +
> + /*
> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> + * zeroed, since this is not guaranteed on all compilers.
> + */
> + memset(&buf_type, 0, sizeof(buf_type));
> + buf_type.buffer_type = BUFFER_TYPE_MEM_POOL;
> + buf_type.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
> + isp4if_split_addr64(ispif->fw_mem_pool->gpu_mc_addr,
> + &buf_type.buffer.buf_base_a_lo,
> + &buf_type.buffer.buf_base_a_hi);
> + buf_type.buffer.buf_size_a = ispif->fw_mem_pool->mem_size;
> +
> + ret = isp4if_send_command(ispif, CMD_ID_SEND_BUFFER,
> + &buf_type, sizeof(buf_type));
> + if (ret) {
> + dev_err(dev, "send fw mem pool 0x%llx(%u) fail %d\n",
> + ispif->fw_mem_pool->gpu_mc_addr, buf_type.buffer.buf_size_a, ret);
> + return ret;
> + }
> +
> + dev_dbg(dev, "send fw mem pool 0x%llx(%u) suc\n",
> + ispif->fw_mem_pool->gpu_mc_addr, buf_type.buffer.buf_size_a);
> +
> + return 0;
> +}
> +
> +static int isp4sd_set_stream_path(struct isp4_subdev *isp_subdev)
> +{
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct isp4fw_cmd_set_stream_cfg cmd;
> + struct device *dev = isp_subdev->dev;
> +
> + /*
> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> + * zeroed, since this is not guaranteed on all compilers.
> + */
> + memset(&cmd, 0, sizeof(cmd));
You could assign assign all these in the declaration and avoid zeroing the
memory explicitly at the same time. I presume possibly leaking some
information from memory to the firmware in case there are holes in the
struct isn't an issue.
> + cmd.stream_cfg.mipi_pipe_path_cfg.isp4fw_sensor_id = SENSOR_ID_ON_MIPI0;
> + cmd.stream_cfg.mipi_pipe_path_cfg.b_enable = true;
> + cmd.stream_cfg.isp_pipe_path_cfg.isp_pipe_id = MIPI0_ISP_PIPELINE_ID;
> +
> + cmd.stream_cfg.b_enable_tnr = true;
> + dev_dbg(dev, "isp4fw_sensor_id %d, pipeId 0x%x EnableTnr %u\n",
> + cmd.stream_cfg.mipi_pipe_path_cfg.isp4fw_sensor_id,
> + cmd.stream_cfg.isp_pipe_path_cfg.isp_pipe_id,
> + cmd.stream_cfg.b_enable_tnr);
> +
> + return isp4if_send_command(ispif, CMD_ID_SET_STREAM_CONFIG,
> + &cmd, sizeof(cmd));
> +}
> +
> +static int isp4sd_send_meta_buf(struct isp4_subdev *isp_subdev)
> +{
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct isp4fw_cmd_send_buffer buf_type;
> + struct device *dev = isp_subdev->dev;
> + int i;
unsigned int, please. You can also declare this within the loop as you do
elsewhere. Consistency is nice.
> +
> + /*
> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> + * zeroed, since this is not guaranteed on all compilers.
> + */
> + memset(&buf_type, 0, sizeof(buf_type));
> + for (i = 0; i < ISP4IF_MAX_STREAM_BUF_COUNT; i++) {
> + struct isp4if_gpu_mem_info *meta_info_buf =
> + isp_subdev->ispif.meta_info_buf[i];
> + int ret;
> +
> + if (!meta_info_buf) {
> + dev_err(dev, "fail for no meta info buf(%u)\n", i);
> + return -ENOMEM;
> + }
> +
> + buf_type.buffer_type = BUFFER_TYPE_META_INFO;
> + buf_type.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
> + isp4if_split_addr64(meta_info_buf->gpu_mc_addr,
> + &buf_type.buffer.buf_base_a_lo,
> + &buf_type.buffer.buf_base_a_hi);
> + buf_type.buffer.buf_size_a = meta_info_buf->mem_size;
> + ret = isp4if_send_command(ispif, CMD_ID_SEND_BUFFER,
> + &buf_type, sizeof(buf_type));
> + if (ret) {
> + dev_err(dev, "send meta info(%u) fail\n", i);
> + return ret;
> + }
> + }
> +
> + dev_dbg(dev, "send meta info suc\n");
> + return 0;
> +}
> +
> +static bool isp4sd_get_str_out_prop(struct isp4_subdev *isp_subdev,
> + struct isp4fw_image_prop *out_prop,
> + struct v4l2_subdev_state *state, u32 pad)
> +{
> + struct device *dev = isp_subdev->dev;
> + struct v4l2_mbus_framefmt *format;
> +
> + format = v4l2_subdev_state_get_format(state, pad, 0);
> + if (!format) {
> + dev_err(dev, "fail get subdev state format\n");
> + return false;
> + }
> +
> + switch (format->code) {
> + case MEDIA_BUS_FMT_YUYV8_1_5X8:
> + out_prop->image_format = IMAGE_FORMAT_NV12;
> + out_prop->width = format->width;
> + out_prop->height = format->height;
> + out_prop->luma_pitch = format->width;
> + out_prop->chroma_pitch = out_prop->width;
> + break;
> + case MEDIA_BUS_FMT_YUYV8_1X16:
> + out_prop->image_format = IMAGE_FORMAT_YUV422INTERLEAVED;
> + out_prop->width = format->width;
> + out_prop->height = format->height;
> + out_prop->luma_pitch = format->width * 2;
> + out_prop->chroma_pitch = 0;
> + break;
> + default:
> + dev_err(dev, "fail for bad image format:0x%x\n",
> + format->code);
> + return false;
> + }
> +
> + if (!out_prop->width || !out_prop->height)
> + return false;
> +
> + return true;
> +}
> +
> +static int isp4sd_kickoff_stream(struct isp4_subdev *isp_subdev, u32 w, u32 h)
> +{
> + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct device *dev = isp_subdev->dev;
> +
> + if (sensor_info->status == ISP4SD_START_STATUS_STARTED)
> + return 0;
> +
> + if (sensor_info->status == ISP4SD_START_STATUS_START_FAIL) {
> + dev_err(dev, "fail for previous start fail\n");
> + return -EINVAL;
> + }
> +
> + dev_dbg(dev, "w:%u,h:%u\n", w, h);
> +
> + if (isp4sd_send_meta_buf(isp_subdev)) {
> + dev_err(dev, "fail to send meta buf\n");
> + sensor_info->status = ISP4SD_START_STATUS_START_FAIL;
> + return -EINVAL;
> + }
> +
> + sensor_info->status = ISP4SD_START_STATUS_OFF;
> +
> + if (!sensor_info->start_stream_cmd_sent &&
> + sensor_info->buf_sent_cnt >= ISP4SD_MIN_BUF_CNT_BEF_START_STREAM) {
> + int ret = isp4if_send_command(ispif, CMD_ID_START_STREAM,
> + NULL, 0);
> + if (ret) {
> + dev_err(dev, "fail to start stream\n");
> + return ret;
> + }
> +
> + sensor_info->start_stream_cmd_sent = true;
> + } else {
> + dev_dbg(dev,
> + "no send START_STREAM, start_sent %u, buf_sent %u\n",
> + sensor_info->start_stream_cmd_sent,
> + sensor_info->buf_sent_cnt);
> + }
> +
> + return 0;
> +}
> +
> +static int isp4sd_setup_output(struct isp4_subdev *isp_subdev,
> + struct v4l2_subdev_state *state, u32 pad)
> +{
> + struct isp4sd_output_info *output_info = &isp_subdev->sensor_info.output_info;
> + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct isp4fw_cmd_set_out_ch_prop cmd_ch_prop;
> + struct isp4fw_cmd_enable_out_ch cmd_ch_en;
> + struct device *dev = isp_subdev->dev;
> + int ret;
> +
> + if (output_info->start_status == ISP4SD_START_STATUS_STARTED)
> + return 0;
> +
> + if (output_info->start_status == ISP4SD_START_STATUS_START_FAIL) {
> + dev_err(dev, "fail for previous start fail\n");
> + return -EINVAL;
> + }
> +
> + /*
> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> + * zeroed, since this is not guaranteed on all compilers.
> + */
> + memset(&cmd_ch_prop, 0, sizeof(cmd_ch_prop));
> + cmd_ch_prop.ch = ISP_PIPE_OUT_CH_PREVIEW;
> +
> + if (!isp4sd_get_str_out_prop(isp_subdev, &cmd_ch_prop.image_prop, state, pad)) {
> + dev_err(dev, "fail to get out prop\n");
> + return -EINVAL;
> + }
> +
> + dev_dbg(dev, "channel:%d,fmt %d,w:h=%u:%u,lp:%u,cp%u\n",
> + cmd_ch_prop.ch,
> + cmd_ch_prop.image_prop.image_format,
> + cmd_ch_prop.image_prop.width, cmd_ch_prop.image_prop.height,
> + cmd_ch_prop.image_prop.luma_pitch,
> + cmd_ch_prop.image_prop.chroma_pitch);
> +
> + ret = isp4if_send_command(ispif, CMD_ID_SET_OUT_CHAN_PROP,
> + &cmd_ch_prop, sizeof(cmd_ch_prop));
> + if (ret) {
> + output_info->start_status = ISP4SD_START_STATUS_START_FAIL;
> + dev_err(dev, "fail to set out prop\n");
> + return ret;
> + }
> +
> + /*
> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> + * zeroed, since this is not guaranteed on all compilers.
You should have explicit padding fields in any case and not rely on ABI in
this case.
> + */
> + memset(&cmd_ch_en, 0, sizeof(cmd_ch_en));
> + cmd_ch_en.ch = ISP_PIPE_OUT_CH_PREVIEW;
> + cmd_ch_en.is_enable = true;
> + ret = isp4if_send_command(ispif, CMD_ID_ENABLE_OUT_CHAN,
> + &cmd_ch_en, sizeof(cmd_ch_en));
> + if (ret) {
> + output_info->start_status = ISP4SD_START_STATUS_START_FAIL;
> + dev_err(dev, "fail to enable channel\n");
> + return ret;
> + }
> +
> + dev_dbg(dev, "enable channel %d\n", cmd_ch_en.ch);
> +
> + if (!sensor_info->start_stream_cmd_sent) {
> + ret = isp4sd_kickoff_stream(isp_subdev,
> + cmd_ch_prop.image_prop.width,
> + cmd_ch_prop.image_prop.height);
> + if (ret) {
> + dev_err(dev, "kickoff stream fail %d\n", ret);
> + return ret;
> + }
> + /*
> + * sensor_info->start_stream_cmd_sent will be set to true
> + * 1. in isp4sd_kickoff_stream, if app first send buffer then
> + * start stream
> + * 2. in isp_set_stream_buf, if app first start stream, then
> + * send buffer
> + * because ISP FW has the requirement, host needs to send buffer
> + * before send start stream cmd
> + */
> + if (sensor_info->start_stream_cmd_sent) {
> + sensor_info->status = ISP4SD_START_STATUS_STARTED;
> + output_info->start_status = ISP4SD_START_STATUS_STARTED;
> + dev_dbg(dev, "kickoff stream suc,start cmd sent\n");
> + }
> + } else {
> + dev_dbg(dev, "stream running, no need kickoff\n");
> + output_info->start_status = ISP4SD_START_STATUS_STARTED;
> + }
> +
> + dev_dbg(dev, "setup output suc\n");
> + return 0;
> +}
> +
> +static int isp4sd_init_stream(struct isp4_subdev *isp_subdev)
> +{
> + struct device *dev = isp_subdev->dev;
> + int ret;
> +
> + ret = isp4sd_setup_fw_mem_pool(isp_subdev);
> + if (ret) {
> + dev_err(dev, "fail to setup fw mem pool\n");
> + return ret;
> + }
> +
> + ret = isp4sd_set_stream_path(isp_subdev);
> + if (ret) {
> + dev_err(dev, "fail to setup stream path\n");
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static void isp4sd_uninit_stream(struct isp4_subdev *isp_subdev,
> + struct v4l2_subdev_state *state, u32 pad)
> +{
> + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
> + struct isp4sd_output_info *output_info = &sensor_info->output_info;
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct v4l2_mbus_framefmt *format;
> +
> + format = v4l2_subdev_state_get_format(state, pad, 0);
> + if (!format) {
> + dev_err(isp_subdev->dev, "fail to get v4l2 format\n");
> + } else {
> + memset(format, 0, sizeof(*format));
> + format->code = MEDIA_BUS_FMT_YUYV8_1_5X8;
> + }
> +
> + isp4if_clear_bufq(ispif);
> + isp4if_clear_cmdq(ispif);
> +
> + sensor_info->start_stream_cmd_sent = false;
> + sensor_info->buf_sent_cnt = 0;
> +
> + sensor_info->status = ISP4SD_START_STATUS_OFF;
> + output_info->start_status = ISP4SD_START_STATUS_OFF;
> +}
> +
> +static void isp4sd_fw_resp_cmd_done(struct isp4_subdev *isp_subdev,
> + enum isp4if_stream_id stream_id,
> + struct isp4fw_resp_cmd_done *para)
> +{
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct isp4if_cmd_element *ele =
> + isp4if_rm_cmd_from_cmdq(ispif, para->cmd_seq_num, para->cmd_id);
> + struct device *dev = isp_subdev->dev;
> +
> + dev_dbg(dev, "stream %d,cmd (0x%08x)(%d),seq %u, ele %p\n",
> + stream_id,
> + para->cmd_id, para->cmd_status, para->cmd_seq_num,
> + ele);
> +
> + if (ele) {
> + complete(&ele->cmd_done);
> + if (atomic_dec_and_test(&ele->refcnt))
> + kfree(ele);
> + }
> +}
> +
> +static struct isp4fw_meta_info *isp4sd_get_meta_by_mc(struct isp4_subdev *isp_subdev,
> + u64 mc)
> +{
> + for (int i = 0; i < ISP4IF_MAX_STREAM_BUF_COUNT; i++) {
unsigned int, please. Similarly for the rest.
> + struct isp4if_gpu_mem_info *meta_info_buf =
> + isp_subdev->ispif.meta_info_buf[i];
> +
> + if (meta_info_buf->gpu_mc_addr == mc)
> + return meta_info_buf->sys_addr;
> + }
> +
> + return NULL;
> +}
> +
> +static void isp4sd_send_meta_info(struct isp4_subdev *isp_subdev,
> + u64 meta_info_mc)
> +{
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct isp4fw_cmd_send_buffer buf_type;
> + struct device *dev = isp_subdev->dev;
> +
> + if (isp_subdev->sensor_info.status != ISP4SD_START_STATUS_STARTED) {
> + dev_warn(dev, "not working status %i, meta_info 0x%llx\n",
> + isp_subdev->sensor_info.status, meta_info_mc);
> + return;
> + }
> +
> + /*
> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> + * zeroed, since this is not guaranteed on all compilers.
> + */
> + memset(&buf_type, 0, sizeof(buf_type));
> + buf_type.buffer_type = BUFFER_TYPE_META_INFO;
> + buf_type.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
> + isp4if_split_addr64(meta_info_mc,
> + &buf_type.buffer.buf_base_a_lo,
> + &buf_type.buffer.buf_base_a_hi);
> + buf_type.buffer.buf_size_a = ISP4SD_META_BUF_SIZE;
> +
> + if (isp4if_send_command(ispif, CMD_ID_SEND_BUFFER,
> + &buf_type, sizeof(buf_type)))
> + dev_err(dev, "fail send meta_info 0x%llx\n",
> + meta_info_mc);
> + else
> + dev_dbg(dev, "resend meta_info 0x%llx\n", meta_info_mc);
> +}
> +
> +static void isp4sd_fw_resp_frame_done(struct isp4_subdev *isp_subdev,
> + enum isp4if_stream_id stream_id,
> + struct isp4fw_resp_param_package *para)
> +{
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct device *dev = isp_subdev->dev;
> + struct isp4if_img_buf_node *prev;
> + struct isp4fw_meta_info *meta;
> + u64 mc;
> +
> + mc = isp4if_join_addr64(para->package_addr_lo, para->package_addr_hi);
> + meta = isp4sd_get_meta_by_mc(isp_subdev, mc);
> + if (!meta) {
> + dev_err(dev, "fail to get meta from mc %llx\n", mc);
> + return;
> + }
> +
> + dev_dbg(dev, "ts:%llu,streamId:%d,poc:%u,preview_en:%u(%i)\n",
> + ktime_get_ns(), stream_id, meta->poc,
> + meta->preview.enabled,
> + meta->preview.status);
> +
> + if (meta->preview.enabled &&
> + (meta->preview.status == BUFFER_STATUS_SKIPPED ||
> + meta->preview.status == BUFFER_STATUS_DONE ||
> + meta->preview.status == BUFFER_STATUS_DIRTY)) {
> + prev = isp4if_dequeue_buffer(ispif);
> + if (prev)
> + isp4if_dealloc_buffer_node(prev);
> + else
> + dev_err(dev, "fail null prev buf\n");
> +
> + } else if (meta->preview.enabled) {
> + dev_err(dev, "fail bad preview status %u\n",
> + meta->preview.status);
> + }
> +
> + if (isp_subdev->sensor_info.status == ISP4SD_START_STATUS_STARTED)
> + isp4sd_send_meta_info(isp_subdev, mc);
> +
> + dev_dbg(dev, "stream_id:%d, status:%d\n", stream_id,
> + isp_subdev->sensor_info.status);
> +}
> +
> +static void isp4sd_fw_resp_func(struct isp4_subdev *isp_subdev,
> + enum isp4if_stream_id stream_id)
> +{
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct device *dev = isp_subdev->dev;
> + struct isp4fw_resp resp;
> +
> + while (true) {
> + if (isp4if_f2h_resp(ispif, stream_id, &resp)) {
> + /* Re-enable the interrupt */
> + isp4_intr_enable(isp_subdev, stream_id, true);
> + /*
> + * Recheck to see if there is a new response.
> + * To ensure that an in-flight interrupt is not lost,
> + * enabling the interrupt must occur _before_ checking
> + * for a new response, hence a memory barrier is needed.
> + * Disable the interrupt again if there was a new response.
> + */
> + mb();
> + if (likely(isp4if_f2h_resp(ispif, stream_id, &resp)))
> + break;
> +
> + isp4_intr_enable(isp_subdev, stream_id, false);
> + }
> +
> + switch (resp.resp_id) {
> + case RESP_ID_CMD_DONE:
> + isp4sd_fw_resp_cmd_done(isp_subdev, stream_id,
> + &resp.param.cmd_done);
> + break;
> + case RESP_ID_NOTI_FRAME_DONE:
> + isp4sd_fw_resp_frame_done(isp_subdev, stream_id,
> + &resp.param.frame_done);
> + break;
> + default:
> + dev_err(dev, "-><- fail respid (0x%x)\n", resp.resp_id);
> + break;
> + }
> + }
> +}
> +
> +static s32 isp4sd_fw_resp_thread(void *context)
> +{
> + struct isp4_subdev_thread_param *para = context;
> + struct isp4_subdev *isp_subdev = para->isp_subdev;
> + struct isp4sd_thread_handler *thread_ctx =
> + &isp_subdev->fw_resp_thread[para->idx];
> + struct device *dev = isp_subdev->dev;
> +
> + dev_dbg(dev, "[%u] fw resp thread started\n", para->idx);
> + while (true) {
> + wait_event_interruptible(thread_ctx->waitq, thread_ctx->resp_ready);
> + thread_ctx->resp_ready = false;
> +
> + if (kthread_should_stop()) {
> + dev_dbg(dev, "[%u] fw resp thread quit\n", para->idx);
> + break;
> + }
> +
> + isp4sd_fw_resp_func(isp_subdev, para->idx);
> + }
> +
> + return 0;
> +}
> +
> +static int isp4sd_stop_resp_proc_threads(struct isp4_subdev *isp_subdev)
> +{
> + int i;
> +
> + for (i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++) {
> + struct isp4sd_thread_handler *thread_ctx =
> + &isp_subdev->fw_resp_thread[i];
> +
> + if (thread_ctx->thread) {
> + kthread_stop(thread_ctx->thread);
> + thread_ctx->thread = NULL;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static int isp4sd_start_resp_proc_threads(struct isp4_subdev *isp_subdev)
> +{
> + struct device *dev = isp_subdev->dev;
> + int i;
> +
> + for (i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++) {
> + struct isp4sd_thread_handler *thread_ctx = &isp_subdev->fw_resp_thread[i];
> +
> + isp_subdev->isp_resp_para[i].idx = i;
> + isp_subdev->isp_resp_para[i].isp_subdev = isp_subdev;
> + init_waitqueue_head(&thread_ctx->waitq);
> + thread_ctx->resp_ready = false;
> +
> + thread_ctx->thread = kthread_run(isp4sd_fw_resp_thread,
> + &isp_subdev->isp_resp_para[i],
> + isp4sd_thread_name[i]);
> + if (IS_ERR(thread_ctx->thread)) {
> + dev_err(dev, "create thread [%d] fail\n", i);
> + thread_ctx->thread = NULL;
> + isp4sd_stop_resp_proc_threads(isp_subdev);
> + return -EINVAL;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static int isp4sd_pwroff_and_deinit(struct isp4_subdev *isp_subdev)
> +{
> + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
> + unsigned int perf_state = ISP4SD_PERFORMANCE_STATE_LOW;
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct device *dev = isp_subdev->dev;
> + int ret;
> +
> + if (sensor_info->status == ISP4SD_START_STATUS_STARTED) {
> + dev_err(dev, "fail for stream still running\n");
> + return -EINVAL;
> + }
> +
> + sensor_info->status = ISP4SD_START_STATUS_OFF;
> +
> + if (isp_subdev->irq_enabled) {
> + for (int i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++)
> + disable_irq(isp_subdev->irq[i]);
> + isp_subdev->irq_enabled = false;
> + }
> +
> + isp4sd_stop_resp_proc_threads(isp_subdev);
> + dev_dbg(dev, "isp_subdev stop resp proc streads suc");
> +
> + isp4if_stop(ispif);
> +
> + ret = dev_pm_genpd_set_performance_state(dev, perf_state);
> + if (ret)
> + dev_err(dev, "fail to set isp_subdev performance state %u,ret %d\n",
> + perf_state, ret);
> +
> + /* hold ccpu reset */
> + isp4hw_wreg(isp_subdev->mmio, ISP_SOFT_RESET, 0);
> + isp4hw_wreg(isp_subdev->mmio, ISP_POWER_STATUS, 0);
> + ret = pm_runtime_put_sync(dev);
> + if (ret)
> + dev_err(dev, "power off isp_subdev fail %d\n", ret);
> + else
> + dev_dbg(dev, "power off isp_subdev suc\n");
> +
> + ispif->status = ISP4IF_STATUS_PWR_OFF;
> + isp4if_clear_cmdq(ispif);
> + isp4sd_module_enable(isp_subdev, false);
> +
> + /*
> + * When opening the camera, isp4sd_module_enable(isp_subdev, true) is called.
> + * Hardware requires at least a 20ms delay between disabling and enabling the module,
> + * so a sleep is added to ensure ISP stability during quick reopen scenarios.
> + */
> + msleep(20);
> +
> + return 0;
> +}
> +
> +static int isp4sd_pwron_and_init(struct isp4_subdev *isp_subdev)
> +{
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct device *dev = isp_subdev->dev;
> + int ret;
> +
> + if (ispif->status == ISP4IF_STATUS_FW_RUNNING) {
> + dev_dbg(dev, "camera already opened, do nothing\n");
> + return 0;
> + }
> +
> + isp4sd_module_enable(isp_subdev, true);
> +
> + if (ispif->status < ISP4IF_STATUS_PWR_ON) {
> + unsigned int perf_state = ISP4SD_PERFORMANCE_STATE_HIGH;
> +
> + ret = pm_runtime_resume_and_get(dev);
> + if (ret) {
> + dev_err(dev, "fail to power on isp_subdev ret %d\n",
> + ret);
> + goto err_deinit;
> + }
> +
> + /* ISPPG ISP Power Status */
> + isp4hw_wreg(isp_subdev->mmio, ISP_POWER_STATUS, 0x7FF);
> + ret = dev_pm_genpd_set_performance_state(dev, perf_state);
> + if (ret) {
> + dev_err(dev,
> + "fail to set performance state %u, ret %d\n",
> + perf_state, ret);
> + goto err_deinit;
> + }
> +
> + ispif->status = ISP4IF_STATUS_PWR_ON;
> + }
> +
> + isp_subdev->sensor_info.start_stream_cmd_sent = false;
> + isp_subdev->sensor_info.buf_sent_cnt = 0;
> +
> + ret = isp4if_start(ispif);
> + if (ret) {
> + dev_err(dev, "fail to start isp_subdev interface\n");
> + goto err_deinit;
> + }
> +
> + if (isp4sd_start_resp_proc_threads(isp_subdev)) {
> + dev_err(dev, "isp_start_resp_proc_threads fail");
> + goto err_deinit;
> + }
> +
> + dev_dbg(dev, "create resp threads ok");
> +
> + for (int i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++)
> + enable_irq(isp_subdev->irq[i]);
> + isp_subdev->irq_enabled = true;
> +
> + return 0;
> +err_deinit:
> + isp4sd_pwroff_and_deinit(isp_subdev);
> + return -EINVAL;
> +}
> +
> +static int isp4sd_stop_stream(struct isp4_subdev *isp_subdev,
> + struct v4l2_subdev_state *state, u32 pad)
> +{
> + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
> + struct isp4sd_output_info *output_info = &sensor_info->output_info;
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct device *dev = isp_subdev->dev;
> +
> + guard(mutex)(&isp_subdev->ops_mutex);
> + dev_dbg(dev, "status %i\n", output_info->start_status);
> +
> + if (output_info->start_status == ISP4SD_START_STATUS_STARTED) {
> + struct isp4fw_cmd_enable_out_ch cmd_ch_disable;
> + int ret;
> +
> + /*
> + * The struct will be shared with ISP FW, use memset() to guarantee
> + * padding bits are zeroed, since this is not guaranteed on all compilers.
> + */
> + memset(&cmd_ch_disable, 0, sizeof(cmd_ch_disable));
> + cmd_ch_disable.ch = ISP_PIPE_OUT_CH_PREVIEW;
> + /* `cmd_ch_disable.is_enable` is already false */
> + ret = isp4if_send_command_sync(ispif, CMD_ID_ENABLE_OUT_CHAN,
> + &cmd_ch_disable,
> + sizeof(cmd_ch_disable));
> + if (ret)
> + dev_err(dev, "fail to disable stream\n");
> + else
> + dev_dbg(dev, "wait disable stream suc\n");
> +
> + ret = isp4if_send_command_sync(ispif, CMD_ID_STOP_STREAM,
> + NULL, 0);
> + if (ret)
> + dev_err(dev, "fail to stop steam\n");
> + else
> + dev_dbg(dev, "wait stop stream suc\n");
> + }
> +
> + isp4sd_uninit_stream(isp_subdev, state, pad);
> +
> + /*
> + * Return success to ensure the stop process proceeds,
> + * and disregard any errors since they are not fatal.
> + */
> + return 0;
> +}
> +
> +static int isp4sd_start_stream(struct isp4_subdev *isp_subdev,
> + struct v4l2_subdev_state *state, u32 pad)
> +{
> + struct isp4sd_output_info *output_info =
> + &isp_subdev->sensor_info.output_info;
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct device *dev = isp_subdev->dev;
> + int ret;
> +
> + guard(mutex)(&isp_subdev->ops_mutex);
> +
> + if (ispif->status != ISP4IF_STATUS_FW_RUNNING) {
> + dev_err(dev, "fail, bad fsm %d", ispif->status);
> + return -EINVAL;
> + }
> +
> + switch (output_info->start_status) {
> + case ISP4SD_START_STATUS_OFF:
> + break;
> + case ISP4SD_START_STATUS_STARTED:
> + dev_dbg(dev, "stream already started, do nothing\n");
> + return 0;
> + case ISP4SD_START_STATUS_START_FAIL:
> + dev_err(dev, "stream previously failed to start\n");
> + return -EINVAL;
> + }
> +
> + ret = isp4sd_init_stream(isp_subdev);
> + if (ret) {
> + dev_err(dev, "fail to init isp_subdev stream\n");
> + goto err_stop_stream;
> + }
> +
> + ret = isp4sd_setup_output(isp_subdev, state, pad);
> + if (ret) {
> + dev_err(dev, "fail to setup output\n");
> + goto err_stop_stream;
> + }
> +
> + return 0;
> +
> +err_stop_stream:
> + isp4sd_stop_stream(isp_subdev, state, pad);
> + return ret;
> +}
> +
> +static int isp4sd_set_power(struct v4l2_subdev *sd, int on)
> +{
> + struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
> +
> + guard(mutex)(&isp_subdev->ops_mutex);
> + if (on)
> + return isp4sd_pwron_and_init(isp_subdev);
> + else
> + return isp4sd_pwroff_and_deinit(isp_subdev);
The s_power() callback is deprecated, please rely on runtime PM.
> +}
> +
> +static const struct v4l2_subdev_core_ops isp4sd_core_ops = {
> + .s_power = isp4sd_set_power,
> +};
> +
> +static const struct v4l2_subdev_video_ops isp4sd_video_ops = {
> + .s_stream = v4l2_subdev_s_stream_helper,
> +};
> +
> +static int isp4sd_set_pad_format(struct v4l2_subdev *sd,
> + struct v4l2_subdev_state *sd_state,
> + struct v4l2_subdev_format *fmt)
> +{
> + struct isp4sd_output_info *steam_info =
Steam?
> + &(to_isp4_subdev(sd)->sensor_info.output_info);
> + struct v4l2_mbus_framefmt *format;
> +
> + format = v4l2_subdev_state_get_format(sd_state, fmt->pad);
> +
> + if (!format) {
> + dev_err(sd->dev, "fail to get state format\n");
> + return -EINVAL;
> + }
> +
> + *format = fmt->format;
> + switch (format->code) {
> + case MEDIA_BUS_FMT_YUYV8_1_5X8:
> + steam_info->image_size = format->width * format->height * 3 / 2;
> + break;
> + case MEDIA_BUS_FMT_YUYV8_1X16:
> + steam_info->image_size = format->width * format->height * 2;
> + break;
> + default:
> + steam_info->image_size = 0;
Please stick to formats the driver supports i.e. make either of the above
the default.
> + break;
> + }
> + if (!steam_info->image_size) {
> + dev_err(sd->dev, "fail set pad format,code 0x%x,width %u, height %u\n",
> + format->code, format->width, format->height);
> + return -EINVAL;
> + }
> +
> + dev_dbg(sd->dev, "set pad format suc, code:%x w:%u h:%u size:%u\n",
> + format->code, format->width, format->height, steam_info->image_size);
> +
> + return 0;
> +}
> +
> +static int isp4sd_enable_streams(struct v4l2_subdev *sd,
> + struct v4l2_subdev_state *state, u32 pad,
> + u64 streams_mask)
> +{
> + struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
> +
> + return isp4sd_start_stream(isp_subdev, state, pad);
> +}
> +
> +static int isp4sd_disable_streams(struct v4l2_subdev *sd,
> + struct v4l2_subdev_state *state, u32 pad,
> + u64 streams_mask)
> +{
> + struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
> +
> + return isp4sd_stop_stream(isp_subdev, state, pad);
> +}
> +
> +static const struct v4l2_subdev_pad_ops isp4sd_pad_ops = {
> + .get_fmt = v4l2_subdev_get_fmt,
> + .set_fmt = isp4sd_set_pad_format,
> + .enable_streams = isp4sd_enable_streams,
> + .disable_streams = isp4sd_disable_streams,
> +};
> +
> +static const struct v4l2_subdev_ops isp4sd_subdev_ops = {
> + .core = &isp4sd_core_ops,
> + .video = &isp4sd_video_ops,
> + .pad = &isp4sd_pad_ops,
> +};
> +
> +static int isp4sd_sdev_link_validate(struct media_link *link)
> +{
> + return 0;
Uh-oh.
What is actually being configured via the sub-device? There is no device
node either, is there? Are there plans for future developments, apart from
possibly making the ISP and the sensor controllable by the host?
> +}
> +
> +static const struct media_entity_operations isp4sd_sdev_ent_ops = {
> + .link_validate = isp4sd_sdev_link_validate,
> +};
> +
> +int isp4sd_init(struct isp4_subdev *isp_subdev, struct v4l2_device *v4l2_dev,
> + int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM])
> +{
> + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct device *dev = v4l2_dev->dev;
> + int ret;
> +
> + isp_subdev->dev = dev;
> + v4l2_subdev_init(&isp_subdev->sdev, &isp4sd_subdev_ops);
> + isp_subdev->sdev.owner = THIS_MODULE;
> + isp_subdev->sdev.dev = dev;
> + snprintf(isp_subdev->sdev.name, sizeof(isp_subdev->sdev.name), "%s",
> + dev_name(dev));
> +
> + isp_subdev->sdev.entity.name = isp4sd_entity_name;
> + isp_subdev->sdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
> + isp_subdev->sdev.entity.ops = &isp4sd_sdev_ent_ops;
> + isp_subdev->sdev_pad.flags = MEDIA_PAD_FL_SOURCE;
> + ret = media_entity_pads_init(&isp_subdev->sdev.entity, 1,
> + &isp_subdev->sdev_pad);
> + if (ret) {
> + dev_err(dev, "fail to init isp4 subdev entity pad %d\n", ret);
> + return ret;
> + }
> +
> + ret = v4l2_subdev_init_finalize(&isp_subdev->sdev);
> + if (ret < 0) {
> + dev_err(dev, "fail to init finalize isp4 subdev %d\n",
> + ret);
> + return ret;
> + }
> +
> + ret = v4l2_device_register_subdev(v4l2_dev, &isp_subdev->sdev);
> + if (ret) {
> + dev_err(dev, "fail to register isp4 subdev to V4L2 device %d\n", ret);
> + goto err_media_clean_up;
> + }
> +
> + isp4if_init(ispif, dev, isp_subdev->mmio);
> +
> + mutex_init(&isp_subdev->ops_mutex);
> + sensor_info->status = ISP4SD_START_STATUS_OFF;
> +
> + /* create ISP enable gpio control */
> + isp_subdev->enable_gpio = devm_gpiod_get(isp_subdev->dev,
> + "enable_isp",
> + GPIOD_OUT_LOW);
> + if (IS_ERR(isp_subdev->enable_gpio)) {
> + ret = PTR_ERR(isp_subdev->enable_gpio);
> + dev_err(dev, "fail to get gpiod %d\n", ret);
> + goto err_subdev_unreg;
> + }
> +
> + for (int i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++)
> + isp_subdev->irq[i] = irq[i];
> +
> + isp_subdev->host2fw_seq_num = 1;
> + ispif->status = ISP4IF_STATUS_PWR_OFF;
> +
> + return 0;
> +
> +err_subdev_unreg:
> + v4l2_device_unregister_subdev(&isp_subdev->sdev);
> +err_media_clean_up:
> + v4l2_subdev_cleanup(&isp_subdev->sdev);
> + media_entity_cleanup(&isp_subdev->sdev.entity);
> + return ret;
> +}
> +
> +void isp4sd_deinit(struct isp4_subdev *isp_subdev)
> +{
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> +
> + v4l2_device_unregister_subdev(&isp_subdev->sdev);
> + media_entity_cleanup(&isp_subdev->sdev.entity);
> + isp4if_deinit(ispif);
> + isp4sd_module_enable(isp_subdev, false);
> +
> + ispif->status = ISP4IF_STATUS_PWR_OFF;
> +}
> diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.h b/drivers/media/platform/amd/isp4/isp4_subdev.h
> new file mode 100644
> index 000000000000..70007b9de3f3
> --- /dev/null
> +++ b/drivers/media/platform/amd/isp4/isp4_subdev.h
> @@ -0,0 +1,124 @@
> +/* SPDX-License-Identifier: GPL-2.0+ */
> +/*
> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
> + */
> +
> +#ifndef _ISP4_SUBDEV_H_
> +#define _ISP4_SUBDEV_H_
> +
> +#include <linux/debugfs.h>
> +#include <linux/delay.h>
> +#include <linux/firmware.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/types.h>
> +#include <linux/uaccess.h>
> +#include <media/v4l2-device.h>
> +
> +#include "isp4_fw_cmd_resp.h"
> +#include "isp4_hw_reg.h"
> +#include "isp4_interface.h"
> +
> +/*
> + * one is for none sesnor specefic response which is not used now
> + * another is for sensor specific response
> + */
> +#define ISP4SD_MAX_FW_RESP_STREAM_NUM 2
> +
> +/*
> + * cmd used to register frame done callback, parameter is
> + * struct isp4sd_register_framedone_cb_param *
> + * when a image buffer is filled by ISP, ISP will call the registered callback.
> + * callback func prototype is isp4sd_framedone_cb, cb_ctx can be anything
> + * provided by caller which will be provided back as the first parameter of the
> + * callback function.
> + * both cb_func and cb_ctx are provide by caller, set cb_func to NULL to
> + * unregister the callback
These comments would benefit from grammatical and spelling corrections.
Same below.
> + */
> +
> +/* used to indicate the ISP status */
> +enum isp4sd_status {
> + ISP4SD_STATUS_PWR_OFF,
> + ISP4SD_STATUS_PWR_ON,
> + ISP4SD_STATUS_FW_RUNNING,
> + ISP4SD_STATUS_MAX
> +};
> +
> +/* used to indicate the status of sensor, output stream */
> +enum isp4sd_start_status {
> + ISP4SD_START_STATUS_OFF,
> + ISP4SD_START_STATUS_STARTED,
> + ISP4SD_START_STATUS_START_FAIL,
> +};
> +
> +struct isp4sd_img_buf_node {
> + struct list_head node;
> + struct isp4if_img_buf_info buf_info;
> +};
> +
> +/* this is isp output after processing bayer raw input from sensor */
> +struct isp4sd_output_info {
> + enum isp4sd_start_status start_status;
> + u32 image_size;
> +};
> +
> +/*
> + * This struct represents the sensor info which is input or source of ISP,
> + * status is the sensor status
> + * output_info is the isp output info after ISP processing the sensor input,
> + * start_stream_cmd_sent mean if CMD_ID_START_STREAM has sent to fw.
> + * buf_sent_cnt is buffer count app has sent to receive the images
> + */
> +struct isp4sd_sensor_info {
> + struct isp4sd_output_info output_info;
> + enum isp4sd_start_status status;
> + bool start_stream_cmd_sent;
> + u32 buf_sent_cnt;
> +};
> +
> +/*
> + * Thread created by driver to receive fw response
> + * thread will be wakeup by fw to driver response interrupt
> + */
> +struct isp4sd_thread_handler {
> + struct task_struct *thread;
> + wait_queue_head_t waitq;
> + bool resp_ready;
> +};
> +
> +struct isp4_subdev_thread_param {
> + u32 idx;
> + struct isp4_subdev *isp_subdev;
> +};
> +
> +struct isp4_subdev {
> + struct v4l2_subdev sdev;
> + struct isp4_interface ispif;
> +
> + struct media_pad sdev_pad;
> +
> + enum isp4sd_status isp_status;
> + struct mutex ops_mutex; /* ops_mutex */
> +
> + /* Used to store fw cmds sent to FW whose response driver needs to wait for */
> + struct isp4sd_thread_handler fw_resp_thread[ISP4SD_MAX_FW_RESP_STREAM_NUM];
> +
> + u32 host2fw_seq_num;
> +
> + struct isp4sd_sensor_info sensor_info;
> +
> + /* gpio descriptor */
> + struct gpio_desc *enable_gpio;
> + struct device *dev;
> + void __iomem *mmio;
> + struct isp4_subdev_thread_param isp_resp_para[ISP4SD_MAX_FW_RESP_STREAM_NUM];
> + int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM];
> + bool irq_enabled;
> + /* spin lock to access ISP_SYS_INT0_EN exclusively */
> + spinlock_t irq_lock;
> +};
> +
> +int isp4sd_init(struct isp4_subdev *isp_subdev, struct v4l2_device *v4l2_dev,
> + int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM]);
> +void isp4sd_deinit(struct isp4_subdev *isp_subdev);
> +
> +#endif /* _ISP4_SUBDEV_H_ */
--
Regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added
2025-12-22 10:11 ` Sakari Ailus
@ 2026-01-07 7:33 ` Sultan Alsawaf
2026-01-14 21:03 ` Sakari Ailus
2026-01-14 10:34 ` Du, Bin
1 sibling, 1 reply; 46+ messages in thread
From: Sultan Alsawaf @ 2026-01-07 7:33 UTC (permalink / raw)
To: Sakari Ailus
Cc: Bin Du, mchehab, hverkuil, laurent.pinchart+renesas,
bryan.odonoghue, prabhakar.mahadev-lad.rj, linux-media,
linux-kernel, pratap.nirujogi, benjamin.chan, king.li,
gjorgji.rosikopulos, Phil.Jawich, Dominic.Antony,
mario.limonciello, richard.gong, anson.tsao, Svetoslav Stoilov,
Alexey Zagorodnikov
Hi Sakari,
On Mon, Dec 22, 2025 at 12:11:11PM +0200, Sakari Ailus wrote:
> Hi Bin,
>
> On Tue, Dec 16, 2025 at 05:13:23PM +0800, Bin Du wrote:
> > Isp4 sub-device is implementing v4l2 sub-device interface. It has one
> > capture video node, and supports only preview stream. It manages firmware
> > states, stream configuration. Add interrupt handling and notification for
> > isp firmware to isp-subdevice.
> >
> > Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
> > Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
> > Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
> > Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
> > Signed-off-by: Bin Du <Bin.Du@amd.com>
> > Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
> > ---
> > MAINTAINERS | 2 +
> > drivers/media/platform/amd/isp4/Makefile | 3 +-
> > drivers/media/platform/amd/isp4/isp4.c | 99 +-
> > drivers/media/platform/amd/isp4/isp4.h | 7 +-
> > drivers/media/platform/amd/isp4/isp4_subdev.c | 975 ++++++++++++++++++
> > drivers/media/platform/amd/isp4/isp4_subdev.h | 124 +++
> > 6 files changed, 1202 insertions(+), 8 deletions(-)
> > create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
> > create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index cccae369c876..48ffc8bbdcee 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -1149,6 +1149,8 @@ F: drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
> > F: drivers/media/platform/amd/isp4/isp4_hw_reg.h
> > F: drivers/media/platform/amd/isp4/isp4_interface.c
> > F: drivers/media/platform/amd/isp4/isp4_interface.h
> > +F: drivers/media/platform/amd/isp4/isp4_subdev.c
> > +F: drivers/media/platform/amd/isp4/isp4_subdev.h
> >
> > AMD KFD
> > M: Felix Kuehling <Felix.Kuehling@amd.com>
> > diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
> > index a2a5bf98e912..6d4e6d6ac7f5 100644
> > --- a/drivers/media/platform/amd/isp4/Makefile
> > +++ b/drivers/media/platform/amd/isp4/Makefile
> > @@ -4,4 +4,5 @@
> >
> > obj-$(CONFIG_AMD_ISP4) += amd_capture.o
> > amd_capture-objs := isp4.o \
> > - isp4_interface.o
> > + isp4_interface.o \
> > + isp4_subdev.o
> > diff --git a/drivers/media/platform/amd/isp4/isp4.c b/drivers/media/platform/amd/isp4/isp4.c
> > index ad95e7f89189..bcd7cad32afd 100644
> > --- a/drivers/media/platform/amd/isp4/isp4.c
> > +++ b/drivers/media/platform/amd/isp4/isp4.c
[snip]
> > static irqreturn_t isp4_irq_handler(int irq, void *arg)
> > {
> > + struct isp4_subdev *isp_subdev = arg;
> > + u32 intr_ack = 0, intr_en = 0, intr_status;
> > + int seen = 0;
>
> Is int appropriate here? Should this be u32 or u64?
ffs() is just a macro alias for __builtin_ffs(). The parameter and return value
of __builtin_ffs() are both int.
> > +
> > + /* Get the ISP_SYS interrupt status */
> > + intr_status = isp4hw_rreg(isp_subdev->mmio, ISP_SYS_INT0_STATUS);
> > + intr_status &= ISP4_FW_RESP_RB_IRQ_STATUS_MASK;
> > +
> > + /* Find which ISP_SYS interrupts fired */
> > + for (size_t i = 0; i < ARRAY_SIZE(isp4_irq); i++) {
> > + if (intr_status & isp4_irq[i].status_mask) {
> > + intr_ack |= isp4_irq[i].ack_mask;
> > + intr_en |= isp4_irq[i].en_mask;
> > + seen |= BIT(i);
> > + }
> > + }
> > +
> > + /*
> > + * Disable the ISP_SYS interrupts that fired. Must be done before waking
> > + * the response threads, since they re-enable interrupts when finished.
> > + * The lock synchronizes RMW of INT0_EN with isp4_enable_interrupt().
> > + */
> > + spin_lock(&isp_subdev->irq_lock);
> > + intr_en = isp4hw_rreg(isp_subdev->mmio, ISP_SYS_INT0_EN) & ~intr_en;
> > + isp4hw_wreg(isp_subdev->mmio, ISP_SYS_INT0_EN, intr_en);
> > + spin_unlock(&isp_subdev->irq_lock);
> > +
> > + /*
> > + * Clear the ISP_SYS interrupts. This must be done after the interrupts
> > + * are disabled, so that ISP FW won't flag any new interrupts on these
> > + * streams, and thus we don't need to clear interrupts again before
> > + * re-enabling them in the response thread.
> > + */
> > + isp4hw_wreg(isp_subdev->mmio, ISP_SYS_INT0_ACK, intr_ack);
> > +
> > + /* Wake up the response threads */
> > + for (int i; (i = ffs(seen)); seen = (seen >> i) << i)
>
> unsigned int, please.
As mentioned above, ffs() takes an int and returns an int.
> The parentheses around ffs() appear redundant.
The parentheses are there because it's an assignment. Without them:
drivers/media/platform/amd/isp4/isp4.c: In function ‘isp4_irq_handler’:
drivers/media/platform/amd/isp4/isp4.c:106:21: warning: suggest parentheses around assignment used as truth value [-Wparentheses]
106 | for (int i; i = ffs(seen); seen = (seen >> i) << i)
| ^
> The increment could probably be expressed as seen &= ~BIT(i).
Yes it can be, but it adds several more instructions before the loop body,
without any improvement to the loop body (the sarx in the loop body is replaced
by andn). The right shift trick is faster and this is a hot path (IRQ handler).
Before:
d8: jmp 10d
// loop body
da: lea -0x1(%rax),%eax
dd: xor %ecx,%ecx
df: mov $0x1,%edx
e4: mov $0x1,%esi
e9: lea (%rax,%rax,4),%rax
ed: sarx %ebx,%ebp,%ebp
f2: shlx %ebx,%ebp,%ebp
f7: movb $0x1,0x9b0(%r13,%rax,8)
100: lea 0x998(%r13,%rax,8),%rdi
108: call __wake_up
10d: mov $0xffffffff,%eax
112: bsf %ebp,%eax
115: add $0x1,%eax
118: mov %eax,%ebx
11a: jne da
After (with seen &= ~BIT(i)):
d8: mov $0xffffffff,%eax
dd: bsf %ebp,%eax
e0: add $0x1,%eax
e3: mov %eax,%ebx
e5: je 12f
e7: mov $0x1,%r12d
// loop body
ed: lea -0x1(%rbx),%eax
f0: xor %ecx,%ecx
f2: mov $0x1,%edx
f7: mov $0x1,%esi
fc: lea (%rax,%rax,4),%rax
100: movb $0x1,0x9b0(%r13,%rax,8)
109: lea 0x998(%r13,%rax,8),%rdi
111: call __wake_up
116: shlx %rbx,%r12,%rax
11b: andn %ebp,%eax,%ebp
120: mov $0xffffffff,%eax
125: bsf %ebp,%eax
128: add $0x1,%eax
12b: mov %eax,%ebx
12d: jne ed
> > + isp4_wake_up_resp_thread(isp_subdev, i - 1);
> > +
> > return IRQ_HANDLED;
> > }
> >
[snip]
> > + container_of(v4l2_sdev, struct isp4_subdev, sdev)
> > +
> > +static const char *isp4sd_entity_name = "amd isp4";
> > +
> > +static const char *isp4sd_thread_name[ISP4SD_MAX_FW_RESP_STREAM_NUM] = {
> > + "amd_isp4_thread_global",
> > + "amd_isp4_thread_stream1",
> > +};
> > +
> > +static void isp4sd_module_enable(struct isp4_subdev *isp_subdev, bool enable)
> > +{
> > + if (isp_subdev->enable_gpio) {
> > + gpiod_set_value(isp_subdev->enable_gpio, enable ? 1 : 0);
> > + dev_dbg(isp_subdev->dev, "%s isp_subdev module\n",
> > + enable ? "enable" : "disable");
> > + }
> > +}
> > +
> > +static int isp4sd_setup_fw_mem_pool(struct isp4_subdev *isp_subdev)
> > +{
> > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > + struct isp4fw_cmd_send_buffer buf_type;
> > + struct device *dev = isp_subdev->dev;
> > + int ret;
> > +
> > + if (!ispif->fw_mem_pool) {
> > + dev_err(dev, "fail to alloc mem pool\n");
> > + return -ENOMEM;
> > + }
> > +
> > + /*
> > + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> > + * zeroed, since this is not guaranteed on all compilers.
> > + */
> > + memset(&buf_type, 0, sizeof(buf_type));
> > + buf_type.buffer_type = BUFFER_TYPE_MEM_POOL;
> > + buf_type.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
> > + isp4if_split_addr64(ispif->fw_mem_pool->gpu_mc_addr,
> > + &buf_type.buffer.buf_base_a_lo,
> > + &buf_type.buffer.buf_base_a_hi);
> > + buf_type.buffer.buf_size_a = ispif->fw_mem_pool->mem_size;
> > +
> > + ret = isp4if_send_command(ispif, CMD_ID_SEND_BUFFER,
> > + &buf_type, sizeof(buf_type));
> > + if (ret) {
> > + dev_err(dev, "send fw mem pool 0x%llx(%u) fail %d\n",
> > + ispif->fw_mem_pool->gpu_mc_addr, buf_type.buffer.buf_size_a, ret);
> > + return ret;
> > + }
> > +
> > + dev_dbg(dev, "send fw mem pool 0x%llx(%u) suc\n",
> > + ispif->fw_mem_pool->gpu_mc_addr, buf_type.buffer.buf_size_a);
> > +
> > + return 0;
> > +}
> > +
> > +static int isp4sd_set_stream_path(struct isp4_subdev *isp_subdev)
> > +{
> > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > + struct isp4fw_cmd_set_stream_cfg cmd;
> > + struct device *dev = isp_subdev->dev;
> > +
> > + /*
> > + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> > + * zeroed, since this is not guaranteed on all compilers.
> > + */
> > + memset(&cmd, 0, sizeof(cmd));
>
> You could assign assign all these in the declaration and avoid zeroing the
> memory explicitly at the same time. I presume possibly leaking some
> information from memory to the firmware in case there are holes in the
> struct isn't an issue.
Leaking kernel memory is bad. Also, there is no guarantee that the firmware will
behave as expected with varying values for the padding bytes.
Please see my arguments from v4 on why these structs should be memset [1].
> > + cmd.stream_cfg.mipi_pipe_path_cfg.isp4fw_sensor_id = SENSOR_ID_ON_MIPI0;
> > + cmd.stream_cfg.mipi_pipe_path_cfg.b_enable = true;
> > + cmd.stream_cfg.isp_pipe_path_cfg.isp_pipe_id = MIPI0_ISP_PIPELINE_ID;
> > +
> > + cmd.stream_cfg.b_enable_tnr = true;
> > + dev_dbg(dev, "isp4fw_sensor_id %d, pipeId 0x%x EnableTnr %u\n",
> > + cmd.stream_cfg.mipi_pipe_path_cfg.isp4fw_sensor_id,
> > + cmd.stream_cfg.isp_pipe_path_cfg.isp_pipe_id,
> > + cmd.stream_cfg.b_enable_tnr);
> > +
> > + return isp4if_send_command(ispif, CMD_ID_SET_STREAM_CONFIG,
> > + &cmd, sizeof(cmd));
> > +}
> > +
> > +static int isp4sd_send_meta_buf(struct isp4_subdev *isp_subdev)
> > +{
> > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > + struct isp4fw_cmd_send_buffer buf_type;
> > + struct device *dev = isp_subdev->dev;
> > + int i;
>
> unsigned int, please. You can also declare this within the loop as you do
> elsewhere. Consistency is nice.
Why does this need to be unsigned?
> > +
> > + /*
> > + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> > + * zeroed, since this is not guaranteed on all compilers.
> > + */
> > + memset(&buf_type, 0, sizeof(buf_type));
> > + for (i = 0; i < ISP4IF_MAX_STREAM_BUF_COUNT; i++) {
> > + struct isp4if_gpu_mem_info *meta_info_buf =
> > + isp_subdev->ispif.meta_info_buf[i];
> > + int ret;
> > +
> > + if (!meta_info_buf) {
> > + dev_err(dev, "fail for no meta info buf(%u)\n", i);
> > + return -ENOMEM;
> > + }
> > +
> > + buf_type.buffer_type = BUFFER_TYPE_META_INFO;
> > + buf_type.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
> > + isp4if_split_addr64(meta_info_buf->gpu_mc_addr,
> > + &buf_type.buffer.buf_base_a_lo,
> > + &buf_type.buffer.buf_base_a_hi);
> > + buf_type.buffer.buf_size_a = meta_info_buf->mem_size;
> > + ret = isp4if_send_command(ispif, CMD_ID_SEND_BUFFER,
> > + &buf_type, sizeof(buf_type));
> > + if (ret) {
> > + dev_err(dev, "send meta info(%u) fail\n", i);
> > + return ret;
> > + }
> > + }
> > +
> > + dev_dbg(dev, "send meta info suc\n");
> > + return 0;
> > +}
> > +
> > +static bool isp4sd_get_str_out_prop(struct isp4_subdev *isp_subdev,
> > + struct isp4fw_image_prop *out_prop,
> > + struct v4l2_subdev_state *state, u32 pad)
> > +{
> > + struct device *dev = isp_subdev->dev;
> > + struct v4l2_mbus_framefmt *format;
> > +
> > + format = v4l2_subdev_state_get_format(state, pad, 0);
> > + if (!format) {
> > + dev_err(dev, "fail get subdev state format\n");
> > + return false;
> > + }
> > +
> > + switch (format->code) {
> > + case MEDIA_BUS_FMT_YUYV8_1_5X8:
> > + out_prop->image_format = IMAGE_FORMAT_NV12;
> > + out_prop->width = format->width;
> > + out_prop->height = format->height;
> > + out_prop->luma_pitch = format->width;
> > + out_prop->chroma_pitch = out_prop->width;
> > + break;
> > + case MEDIA_BUS_FMT_YUYV8_1X16:
> > + out_prop->image_format = IMAGE_FORMAT_YUV422INTERLEAVED;
> > + out_prop->width = format->width;
> > + out_prop->height = format->height;
> > + out_prop->luma_pitch = format->width * 2;
> > + out_prop->chroma_pitch = 0;
> > + break;
> > + default:
> > + dev_err(dev, "fail for bad image format:0x%x\n",
> > + format->code);
> > + return false;
> > + }
> > +
> > + if (!out_prop->width || !out_prop->height)
> > + return false;
> > +
> > + return true;
> > +}
> > +
> > +static int isp4sd_kickoff_stream(struct isp4_subdev *isp_subdev, u32 w, u32 h)
> > +{
> > + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
> > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > + struct device *dev = isp_subdev->dev;
> > +
> > + if (sensor_info->status == ISP4SD_START_STATUS_STARTED)
> > + return 0;
> > +
> > + if (sensor_info->status == ISP4SD_START_STATUS_START_FAIL) {
> > + dev_err(dev, "fail for previous start fail\n");
> > + return -EINVAL;
> > + }
> > +
> > + dev_dbg(dev, "w:%u,h:%u\n", w, h);
> > +
> > + if (isp4sd_send_meta_buf(isp_subdev)) {
> > + dev_err(dev, "fail to send meta buf\n");
> > + sensor_info->status = ISP4SD_START_STATUS_START_FAIL;
> > + return -EINVAL;
> > + }
> > +
> > + sensor_info->status = ISP4SD_START_STATUS_OFF;
> > +
> > + if (!sensor_info->start_stream_cmd_sent &&
> > + sensor_info->buf_sent_cnt >= ISP4SD_MIN_BUF_CNT_BEF_START_STREAM) {
> > + int ret = isp4if_send_command(ispif, CMD_ID_START_STREAM,
> > + NULL, 0);
> > + if (ret) {
> > + dev_err(dev, "fail to start stream\n");
> > + return ret;
> > + }
> > +
> > + sensor_info->start_stream_cmd_sent = true;
> > + } else {
> > + dev_dbg(dev,
> > + "no send START_STREAM, start_sent %u, buf_sent %u\n",
> > + sensor_info->start_stream_cmd_sent,
> > + sensor_info->buf_sent_cnt);
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static int isp4sd_setup_output(struct isp4_subdev *isp_subdev,
> > + struct v4l2_subdev_state *state, u32 pad)
> > +{
> > + struct isp4sd_output_info *output_info = &isp_subdev->sensor_info.output_info;
> > + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
> > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > + struct isp4fw_cmd_set_out_ch_prop cmd_ch_prop;
> > + struct isp4fw_cmd_enable_out_ch cmd_ch_en;
> > + struct device *dev = isp_subdev->dev;
> > + int ret;
> > +
> > + if (output_info->start_status == ISP4SD_START_STATUS_STARTED)
> > + return 0;
> > +
> > + if (output_info->start_status == ISP4SD_START_STATUS_START_FAIL) {
> > + dev_err(dev, "fail for previous start fail\n");
> > + return -EINVAL;
> > + }
> > +
> > + /*
> > + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> > + * zeroed, since this is not guaranteed on all compilers.
> > + */
> > + memset(&cmd_ch_prop, 0, sizeof(cmd_ch_prop));
> > + cmd_ch_prop.ch = ISP_PIPE_OUT_CH_PREVIEW;
> > +
> > + if (!isp4sd_get_str_out_prop(isp_subdev, &cmd_ch_prop.image_prop, state, pad)) {
> > + dev_err(dev, "fail to get out prop\n");
> > + return -EINVAL;
> > + }
> > +
> > + dev_dbg(dev, "channel:%d,fmt %d,w:h=%u:%u,lp:%u,cp%u\n",
> > + cmd_ch_prop.ch,
> > + cmd_ch_prop.image_prop.image_format,
> > + cmd_ch_prop.image_prop.width, cmd_ch_prop.image_prop.height,
> > + cmd_ch_prop.image_prop.luma_pitch,
> > + cmd_ch_prop.image_prop.chroma_pitch);
> > +
> > + ret = isp4if_send_command(ispif, CMD_ID_SET_OUT_CHAN_PROP,
> > + &cmd_ch_prop, sizeof(cmd_ch_prop));
> > + if (ret) {
> > + output_info->start_status = ISP4SD_START_STATUS_START_FAIL;
> > + dev_err(dev, "fail to set out prop\n");
> > + return ret;
> > + }
> > +
> > + /*
> > + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> > + * zeroed, since this is not guaranteed on all compilers.
>
> You should have explicit padding fields in any case and not rely on ABI in
> this case.
It is error-prone for a human to make sure that all padding bytes have explicit
struct members. And what about future changes to the firmware API where explicit
padding might be forgotten?
Unless the firmware API structs are all __packed in a future firmware update, I
think the memsets should remain.
> > + */
> > + memset(&cmd_ch_en, 0, sizeof(cmd_ch_en));
> > + cmd_ch_en.ch = ISP_PIPE_OUT_CH_PREVIEW;
> > + cmd_ch_en.is_enable = true;
> > + ret = isp4if_send_command(ispif, CMD_ID_ENABLE_OUT_CHAN,
> > + &cmd_ch_en, sizeof(cmd_ch_en));
> > + if (ret) {
> > + output_info->start_status = ISP4SD_START_STATUS_START_FAIL;
> > + dev_err(dev, "fail to enable channel\n");
> > + return ret;
> > + }
> > +
> > + dev_dbg(dev, "enable channel %d\n", cmd_ch_en.ch);
> > +
> > + if (!sensor_info->start_stream_cmd_sent) {
> > + ret = isp4sd_kickoff_stream(isp_subdev,
> > + cmd_ch_prop.image_prop.width,
> > + cmd_ch_prop.image_prop.height);
> > + if (ret) {
> > + dev_err(dev, "kickoff stream fail %d\n", ret);
> > + return ret;
> > + }
> > + /*
> > + * sensor_info->start_stream_cmd_sent will be set to true
> > + * 1. in isp4sd_kickoff_stream, if app first send buffer then
> > + * start stream
> > + * 2. in isp_set_stream_buf, if app first start stream, then
> > + * send buffer
> > + * because ISP FW has the requirement, host needs to send buffer
> > + * before send start stream cmd
> > + */
> > + if (sensor_info->start_stream_cmd_sent) {
> > + sensor_info->status = ISP4SD_START_STATUS_STARTED;
> > + output_info->start_status = ISP4SD_START_STATUS_STARTED;
> > + dev_dbg(dev, "kickoff stream suc,start cmd sent\n");
> > + }
> > + } else {
> > + dev_dbg(dev, "stream running, no need kickoff\n");
> > + output_info->start_status = ISP4SD_START_STATUS_STARTED;
> > + }
> > +
> > + dev_dbg(dev, "setup output suc\n");
> > + return 0;
> > +}
> > +
> > +static int isp4sd_init_stream(struct isp4_subdev *isp_subdev)
> > +{
> > + struct device *dev = isp_subdev->dev;
> > + int ret;
> > +
> > + ret = isp4sd_setup_fw_mem_pool(isp_subdev);
> > + if (ret) {
> > + dev_err(dev, "fail to setup fw mem pool\n");
> > + return ret;
> > + }
> > +
> > + ret = isp4sd_set_stream_path(isp_subdev);
> > + if (ret) {
> > + dev_err(dev, "fail to setup stream path\n");
> > + return ret;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static void isp4sd_uninit_stream(struct isp4_subdev *isp_subdev,
> > + struct v4l2_subdev_state *state, u32 pad)
> > +{
> > + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
> > + struct isp4sd_output_info *output_info = &sensor_info->output_info;
> > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > + struct v4l2_mbus_framefmt *format;
> > +
> > + format = v4l2_subdev_state_get_format(state, pad, 0);
> > + if (!format) {
> > + dev_err(isp_subdev->dev, "fail to get v4l2 format\n");
> > + } else {
> > + memset(format, 0, sizeof(*format));
> > + format->code = MEDIA_BUS_FMT_YUYV8_1_5X8;
> > + }
> > +
> > + isp4if_clear_bufq(ispif);
> > + isp4if_clear_cmdq(ispif);
> > +
> > + sensor_info->start_stream_cmd_sent = false;
> > + sensor_info->buf_sent_cnt = 0;
> > +
> > + sensor_info->status = ISP4SD_START_STATUS_OFF;
> > + output_info->start_status = ISP4SD_START_STATUS_OFF;
> > +}
> > +
> > +static void isp4sd_fw_resp_cmd_done(struct isp4_subdev *isp_subdev,
> > + enum isp4if_stream_id stream_id,
> > + struct isp4fw_resp_cmd_done *para)
> > +{
> > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > + struct isp4if_cmd_element *ele =
> > + isp4if_rm_cmd_from_cmdq(ispif, para->cmd_seq_num, para->cmd_id);
> > + struct device *dev = isp_subdev->dev;
> > +
> > + dev_dbg(dev, "stream %d,cmd (0x%08x)(%d),seq %u, ele %p\n",
> > + stream_id,
> > + para->cmd_id, para->cmd_status, para->cmd_seq_num,
> > + ele);
> > +
> > + if (ele) {
> > + complete(&ele->cmd_done);
> > + if (atomic_dec_and_test(&ele->refcnt))
> > + kfree(ele);
> > + }
> > +}
> > +
> > +static struct isp4fw_meta_info *isp4sd_get_meta_by_mc(struct isp4_subdev *isp_subdev,
> > + u64 mc)
> > +{
> > + for (int i = 0; i < ISP4IF_MAX_STREAM_BUF_COUNT; i++) {
>
> unsigned int, please. Similarly for the rest.
Why an unsigned int?
> > + struct isp4if_gpu_mem_info *meta_info_buf =
> > + isp_subdev->ispif.meta_info_buf[i];
> > +
> > + if (meta_info_buf->gpu_mc_addr == mc)
> > + return meta_info_buf->sys_addr;
> > + }
> > +
> > + return NULL;
> > +}
[1] https://lore.kernel.org/all/aNTtLHDHf_ozenC-@sultan-box/
Sultan
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added
2026-01-07 7:33 ` Sultan Alsawaf
@ 2026-01-14 21:03 ` Sakari Ailus
2026-01-14 21:08 ` Mario Limonciello
` (2 more replies)
0 siblings, 3 replies; 46+ messages in thread
From: Sakari Ailus @ 2026-01-14 21:03 UTC (permalink / raw)
To: Sultan Alsawaf
Cc: Bin Du, mchehab, hverkuil, laurent.pinchart+renesas,
bryan.odonoghue, prabhakar.mahadev-lad.rj, linux-media,
linux-kernel, pratap.nirujogi, benjamin.chan, king.li,
gjorgji.rosikopulos, Phil.Jawich, Dominic.Antony,
mario.limonciello, richard.gong, anson.tsao, Svetoslav Stoilov,
Alexey Zagorodnikov
Hi Sultan,
On Tue, Jan 06, 2026 at 11:33:53PM -0800, Sultan Alsawaf wrote:
> Hi Sakari,
>
> On Mon, Dec 22, 2025 at 12:11:11PM +0200, Sakari Ailus wrote:
> > Hi Bin,
> >
> > On Tue, Dec 16, 2025 at 05:13:23PM +0800, Bin Du wrote:
> > > Isp4 sub-device is implementing v4l2 sub-device interface. It has one
> > > capture video node, and supports only preview stream. It manages firmware
> > > states, stream configuration. Add interrupt handling and notification for
> > > isp firmware to isp-subdevice.
> > >
> > > Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
> > > Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
> > > Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
> > > Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
> > > Signed-off-by: Bin Du <Bin.Du@amd.com>
> > > Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
> > > ---
> > > MAINTAINERS | 2 +
> > > drivers/media/platform/amd/isp4/Makefile | 3 +-
> > > drivers/media/platform/amd/isp4/isp4.c | 99 +-
> > > drivers/media/platform/amd/isp4/isp4.h | 7 +-
> > > drivers/media/platform/amd/isp4/isp4_subdev.c | 975 ++++++++++++++++++
> > > drivers/media/platform/amd/isp4/isp4_subdev.h | 124 +++
> > > 6 files changed, 1202 insertions(+), 8 deletions(-)
> > > create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
> > > create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
> > >
> > > diff --git a/MAINTAINERS b/MAINTAINERS
> > > index cccae369c876..48ffc8bbdcee 100644
> > > --- a/MAINTAINERS
> > > +++ b/MAINTAINERS
> > > @@ -1149,6 +1149,8 @@ F: drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
> > > F: drivers/media/platform/amd/isp4/isp4_hw_reg.h
> > > F: drivers/media/platform/amd/isp4/isp4_interface.c
> > > F: drivers/media/platform/amd/isp4/isp4_interface.h
> > > +F: drivers/media/platform/amd/isp4/isp4_subdev.c
> > > +F: drivers/media/platform/amd/isp4/isp4_subdev.h
> > >
> > > AMD KFD
> > > M: Felix Kuehling <Felix.Kuehling@amd.com>
> > > diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
> > > index a2a5bf98e912..6d4e6d6ac7f5 100644
> > > --- a/drivers/media/platform/amd/isp4/Makefile
> > > +++ b/drivers/media/platform/amd/isp4/Makefile
> > > @@ -4,4 +4,5 @@
> > >
> > > obj-$(CONFIG_AMD_ISP4) += amd_capture.o
> > > amd_capture-objs := isp4.o \
> > > - isp4_interface.o
> > > + isp4_interface.o \
> > > + isp4_subdev.o
> > > diff --git a/drivers/media/platform/amd/isp4/isp4.c b/drivers/media/platform/amd/isp4/isp4.c
> > > index ad95e7f89189..bcd7cad32afd 100644
> > > --- a/drivers/media/platform/amd/isp4/isp4.c
> > > +++ b/drivers/media/platform/amd/isp4/isp4.c
>
> [snip]
>
> > > static irqreturn_t isp4_irq_handler(int irq, void *arg)
> > > {
> > > + struct isp4_subdev *isp_subdev = arg;
> > > + u32 intr_ack = 0, intr_en = 0, intr_status;
> > > + int seen = 0;
> >
> > Is int appropriate here? Should this be u32 or u64?
>
> ffs() is just a macro alias for __builtin_ffs(). The parameter and return value
> of __builtin_ffs() are both int.
Ack, sounds reasonable.
>
> > > +
> > > + /* Get the ISP_SYS interrupt status */
> > > + intr_status = isp4hw_rreg(isp_subdev->mmio, ISP_SYS_INT0_STATUS);
> > > + intr_status &= ISP4_FW_RESP_RB_IRQ_STATUS_MASK;
> > > +
> > > + /* Find which ISP_SYS interrupts fired */
> > > + for (size_t i = 0; i < ARRAY_SIZE(isp4_irq); i++) {
> > > + if (intr_status & isp4_irq[i].status_mask) {
> > > + intr_ack |= isp4_irq[i].ack_mask;
> > > + intr_en |= isp4_irq[i].en_mask;
> > > + seen |= BIT(i);
> > > + }
> > > + }
> > > +
> > > + /*
> > > + * Disable the ISP_SYS interrupts that fired. Must be done before waking
> > > + * the response threads, since they re-enable interrupts when finished.
> > > + * The lock synchronizes RMW of INT0_EN with isp4_enable_interrupt().
> > > + */
> > > + spin_lock(&isp_subdev->irq_lock);
> > > + intr_en = isp4hw_rreg(isp_subdev->mmio, ISP_SYS_INT0_EN) & ~intr_en;
> > > + isp4hw_wreg(isp_subdev->mmio, ISP_SYS_INT0_EN, intr_en);
> > > + spin_unlock(&isp_subdev->irq_lock);
> > > +
> > > + /*
> > > + * Clear the ISP_SYS interrupts. This must be done after the interrupts
> > > + * are disabled, so that ISP FW won't flag any new interrupts on these
> > > + * streams, and thus we don't need to clear interrupts again before
> > > + * re-enabling them in the response thread.
> > > + */
> > > + isp4hw_wreg(isp_subdev->mmio, ISP_SYS_INT0_ACK, intr_ack);
> > > +
> > > + /* Wake up the response threads */
> > > + for (int i; (i = ffs(seen)); seen = (seen >> i) << i)
> >
> > unsigned int, please.
>
> As mentioned above, ffs() takes an int and returns an int.
>
> > The parentheses around ffs() appear redundant.
>
> The parentheses are there because it's an assignment. Without them:
>
> drivers/media/platform/amd/isp4/isp4.c: In function ‘isp4_irq_handler’:
> drivers/media/platform/amd/isp4/isp4.c:106:21: warning: suggest parentheses around assignment used as truth value [-Wparentheses]
> 106 | for (int i; i = ffs(seen); seen = (seen >> i) << i)
> | ^
>
> > The increment could probably be expressed as seen &= ~BIT(i).
>
> Yes it can be, but it adds several more instructions before the loop body,
> without any improvement to the loop body (the sarx in the loop body is replaced
> by andn). The right shift trick is faster and this is a hot path (IRQ handler).
Fine by me, it won't make much difference in practice either way.
...
> > > +static int isp4sd_set_stream_path(struct isp4_subdev *isp_subdev)
> > > +{
> > > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > > + struct isp4fw_cmd_set_stream_cfg cmd;
> > > + struct device *dev = isp_subdev->dev;
> > > +
> > > + /*
> > > + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> > > + * zeroed, since this is not guaranteed on all compilers.
> > > + */
> > > + memset(&cmd, 0, sizeof(cmd));
> >
> > You could assign assign all these in the declaration and avoid zeroing the
> > memory explicitly at the same time. I presume possibly leaking some
> > information from memory to the firmware in case there are holes in the
> > struct isn't an issue.
>
> Leaking kernel memory is bad. Also, there is no guarantee that the firmware will
> behave as expected with varying values for the padding bytes.
>
> Please see my arguments from v4 on why these structs should be memset [1].
There should be no host CPU related ABI introduced padding in structs defining
firmware interfaces. Just use reserved fields in that case instead. In
other words, the above memset() is equivalent to zeroing the memory using
an assignment.
>
> > > + cmd.stream_cfg.mipi_pipe_path_cfg.isp4fw_sensor_id = SENSOR_ID_ON_MIPI0;
> > > + cmd.stream_cfg.mipi_pipe_path_cfg.b_enable = true;
> > > + cmd.stream_cfg.isp_pipe_path_cfg.isp_pipe_id = MIPI0_ISP_PIPELINE_ID;
> > > +
> > > + cmd.stream_cfg.b_enable_tnr = true;
> > > + dev_dbg(dev, "isp4fw_sensor_id %d, pipeId 0x%x EnableTnr %u\n",
> > > + cmd.stream_cfg.mipi_pipe_path_cfg.isp4fw_sensor_id,
> > > + cmd.stream_cfg.isp_pipe_path_cfg.isp_pipe_id,
> > > + cmd.stream_cfg.b_enable_tnr);
> > > +
> > > + return isp4if_send_command(ispif, CMD_ID_SET_STREAM_CONFIG,
> > > + &cmd, sizeof(cmd));
> > > +}
> > > +
> > > +static int isp4sd_send_meta_buf(struct isp4_subdev *isp_subdev)
> > > +{
> > > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > > + struct isp4fw_cmd_send_buffer buf_type;
> > > + struct device *dev = isp_subdev->dev;
> > > + int i;
> >
> > unsigned int, please. You can also declare this within the loop as you do
> > elsewhere. Consistency is nice.
>
> Why does this need to be unsigned?
Do you need negative numbers there?
>
> > > +
> > > + /*
> > > + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> > > + * zeroed, since this is not guaranteed on all compilers.
> > > + */
> > > + memset(&buf_type, 0, sizeof(buf_type));
> > > + for (i = 0; i < ISP4IF_MAX_STREAM_BUF_COUNT; i++) {
> > > + struct isp4if_gpu_mem_info *meta_info_buf =
> > > + isp_subdev->ispif.meta_info_buf[i];
> > > + int ret;
> > > +
> > > + if (!meta_info_buf) {
> > > + dev_err(dev, "fail for no meta info buf(%u)\n", i);
> > > + return -ENOMEM;
> > > + }
> > > +
> > > + buf_type.buffer_type = BUFFER_TYPE_META_INFO;
> > > + buf_type.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
> > > + isp4if_split_addr64(meta_info_buf->gpu_mc_addr,
> > > + &buf_type.buffer.buf_base_a_lo,
> > > + &buf_type.buffer.buf_base_a_hi);
> > > + buf_type.buffer.buf_size_a = meta_info_buf->mem_size;
> > > + ret = isp4if_send_command(ispif, CMD_ID_SEND_BUFFER,
> > > + &buf_type, sizeof(buf_type));
> > > + if (ret) {
> > > + dev_err(dev, "send meta info(%u) fail\n", i);
> > > + return ret;
> > > + }
> > > + }
> > > +
> > > + dev_dbg(dev, "send meta info suc\n");
> > > + return 0;
> > > +}
> > > +
> > > +static bool isp4sd_get_str_out_prop(struct isp4_subdev *isp_subdev,
> > > + struct isp4fw_image_prop *out_prop,
> > > + struct v4l2_subdev_state *state, u32 pad)
> > > +{
> > > + struct device *dev = isp_subdev->dev;
> > > + struct v4l2_mbus_framefmt *format;
> > > +
> > > + format = v4l2_subdev_state_get_format(state, pad, 0);
> > > + if (!format) {
> > > + dev_err(dev, "fail get subdev state format\n");
> > > + return false;
> > > + }
> > > +
> > > + switch (format->code) {
> > > + case MEDIA_BUS_FMT_YUYV8_1_5X8:
> > > + out_prop->image_format = IMAGE_FORMAT_NV12;
> > > + out_prop->width = format->width;
> > > + out_prop->height = format->height;
> > > + out_prop->luma_pitch = format->width;
> > > + out_prop->chroma_pitch = out_prop->width;
> > > + break;
> > > + case MEDIA_BUS_FMT_YUYV8_1X16:
> > > + out_prop->image_format = IMAGE_FORMAT_YUV422INTERLEAVED;
> > > + out_prop->width = format->width;
> > > + out_prop->height = format->height;
> > > + out_prop->luma_pitch = format->width * 2;
> > > + out_prop->chroma_pitch = 0;
> > > + break;
> > > + default:
> > > + dev_err(dev, "fail for bad image format:0x%x\n",
> > > + format->code);
> > > + return false;
> > > + }
> > > +
> > > + if (!out_prop->width || !out_prop->height)
> > > + return false;
> > > +
> > > + return true;
> > > +}
> > > +
> > > +static int isp4sd_kickoff_stream(struct isp4_subdev *isp_subdev, u32 w, u32 h)
> > > +{
> > > + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
> > > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > > + struct device *dev = isp_subdev->dev;
> > > +
> > > + if (sensor_info->status == ISP4SD_START_STATUS_STARTED)
> > > + return 0;
> > > +
> > > + if (sensor_info->status == ISP4SD_START_STATUS_START_FAIL) {
> > > + dev_err(dev, "fail for previous start fail\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + dev_dbg(dev, "w:%u,h:%u\n", w, h);
> > > +
> > > + if (isp4sd_send_meta_buf(isp_subdev)) {
> > > + dev_err(dev, "fail to send meta buf\n");
> > > + sensor_info->status = ISP4SD_START_STATUS_START_FAIL;
> > > + return -EINVAL;
> > > + }
> > > +
> > > + sensor_info->status = ISP4SD_START_STATUS_OFF;
> > > +
> > > + if (!sensor_info->start_stream_cmd_sent &&
> > > + sensor_info->buf_sent_cnt >= ISP4SD_MIN_BUF_CNT_BEF_START_STREAM) {
> > > + int ret = isp4if_send_command(ispif, CMD_ID_START_STREAM,
> > > + NULL, 0);
> > > + if (ret) {
> > > + dev_err(dev, "fail to start stream\n");
> > > + return ret;
> > > + }
> > > +
> > > + sensor_info->start_stream_cmd_sent = true;
> > > + } else {
> > > + dev_dbg(dev,
> > > + "no send START_STREAM, start_sent %u, buf_sent %u\n",
> > > + sensor_info->start_stream_cmd_sent,
> > > + sensor_info->buf_sent_cnt);
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int isp4sd_setup_output(struct isp4_subdev *isp_subdev,
> > > + struct v4l2_subdev_state *state, u32 pad)
> > > +{
> > > + struct isp4sd_output_info *output_info = &isp_subdev->sensor_info.output_info;
> > > + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
> > > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > > + struct isp4fw_cmd_set_out_ch_prop cmd_ch_prop;
> > > + struct isp4fw_cmd_enable_out_ch cmd_ch_en;
> > > + struct device *dev = isp_subdev->dev;
> > > + int ret;
> > > +
> > > + if (output_info->start_status == ISP4SD_START_STATUS_STARTED)
> > > + return 0;
> > > +
> > > + if (output_info->start_status == ISP4SD_START_STATUS_START_FAIL) {
> > > + dev_err(dev, "fail for previous start fail\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + /*
> > > + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> > > + * zeroed, since this is not guaranteed on all compilers.
> > > + */
> > > + memset(&cmd_ch_prop, 0, sizeof(cmd_ch_prop));
> > > + cmd_ch_prop.ch = ISP_PIPE_OUT_CH_PREVIEW;
> > > +
> > > + if (!isp4sd_get_str_out_prop(isp_subdev, &cmd_ch_prop.image_prop, state, pad)) {
> > > + dev_err(dev, "fail to get out prop\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + dev_dbg(dev, "channel:%d,fmt %d,w:h=%u:%u,lp:%u,cp%u\n",
> > > + cmd_ch_prop.ch,
> > > + cmd_ch_prop.image_prop.image_format,
> > > + cmd_ch_prop.image_prop.width, cmd_ch_prop.image_prop.height,
> > > + cmd_ch_prop.image_prop.luma_pitch,
> > > + cmd_ch_prop.image_prop.chroma_pitch);
> > > +
> > > + ret = isp4if_send_command(ispif, CMD_ID_SET_OUT_CHAN_PROP,
> > > + &cmd_ch_prop, sizeof(cmd_ch_prop));
> > > + if (ret) {
> > > + output_info->start_status = ISP4SD_START_STATUS_START_FAIL;
> > > + dev_err(dev, "fail to set out prop\n");
> > > + return ret;
> > > + }
> > > +
> > > + /*
> > > + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> > > + * zeroed, since this is not guaranteed on all compilers.
> >
> > You should have explicit padding fields in any case and not rely on ABI in
> > this case.
>
> It is error-prone for a human to make sure that all padding bytes have
> explicit struct members. And what about future changes to the firmware
> API where explicit padding might be forgotten?
Just don't do that. Use pahole to verify the result when making changes to
the structs.
>
> Unless the firmware API structs are all __packed in a future firmware update, I
> think the memsets should remain.
If you want to be certain of the size of the structs, use BUILD_BUG_ON().
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added
2026-01-14 21:03 ` Sakari Ailus
@ 2026-01-14 21:08 ` Mario Limonciello
2026-01-15 5:56 ` Du, Bin
2026-01-15 7:21 ` Sultan Alsawaf
2026-01-15 10:36 ` Du, Bin
2 siblings, 1 reply; 46+ messages in thread
From: Mario Limonciello @ 2026-01-14 21:08 UTC (permalink / raw)
To: Sakari Ailus, Sultan Alsawaf
Cc: Bin Du, mchehab, hverkuil, laurent.pinchart+renesas,
bryan.odonoghue, prabhakar.mahadev-lad.rj, linux-media,
linux-kernel, pratap.nirujogi, benjamin.chan, king.li,
gjorgji.rosikopulos, Phil.Jawich, Dominic.Antony, richard.gong,
anson.tsao, Svetoslav Stoilov, Alexey Zagorodnikov
>>
>> Unless the firmware API structs are all __packed in a future firmware update, I
>> think the memsets should remain.
>
> If you want to be certain of the size of the structs, use BUILD_BUG_ON().
>
static_assert is another option here too. I did something like that in
drivers/acpi/platform_profile.c to make sure that a structure got
updated from new members.
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added
2026-01-14 21:08 ` Mario Limonciello
@ 2026-01-15 5:56 ` Du, Bin
0 siblings, 0 replies; 46+ messages in thread
From: Du, Bin @ 2026-01-15 5:56 UTC (permalink / raw)
To: Mario Limonciello, Sakari Ailus, Sultan Alsawaf
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, richard.gong, anson.tsao,
Alexey Zagorodnikov
On 1/15/2026 5:08 AM, Mario Limonciello wrote:
>>>
>>> Unless the firmware API structs are all __packed in a future firmware
>>> update, I
>>> think the memsets should remain.
>>
>> If you want to be certain of the size of the structs, use BUILD_BUG_ON().
>>
>
> static_assert is another option here too. I did something like that in
> drivers/acpi/platform_profile.c to make sure that a structure got
> updated from new members.
Thank you, Sakari and Mario, for your informative sharing.
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added
2026-01-14 21:03 ` Sakari Ailus
2026-01-14 21:08 ` Mario Limonciello
@ 2026-01-15 7:21 ` Sultan Alsawaf
2026-01-20 9:29 ` Du, Bin
2026-01-15 10:36 ` Du, Bin
2 siblings, 1 reply; 46+ messages in thread
From: Sultan Alsawaf @ 2026-01-15 7:21 UTC (permalink / raw)
To: Sakari Ailus
Cc: Bin Du, mchehab, hverkuil, laurent.pinchart+renesas,
bryan.odonoghue, prabhakar.mahadev-lad.rj, linux-media,
linux-kernel, pratap.nirujogi, benjamin.chan, king.li,
gjorgji.rosikopulos, Phil.Jawich, Dominic.Antony,
mario.limonciello, richard.gong, anson.tsao, Svetoslav Stoilov,
Alexey Zagorodnikov
On Wed, Jan 14, 2026 at 11:03:49PM +0200, Sakari Ailus wrote:
> Hi Sultan,
>
> On Tue, Jan 06, 2026 at 11:33:53PM -0800, Sultan Alsawaf wrote:
> > Hi Sakari,
> >
> > On Mon, Dec 22, 2025 at 12:11:11PM +0200, Sakari Ailus wrote:
> > > Hi Bin,
> > >
> > > On Tue, Dec 16, 2025 at 05:13:23PM +0800, Bin Du wrote:
> > > > +static int isp4sd_set_stream_path(struct isp4_subdev *isp_subdev)
> > > > +{
> > > > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > > > + struct isp4fw_cmd_set_stream_cfg cmd;
> > > > + struct device *dev = isp_subdev->dev;
> > > > +
> > > > + /*
> > > > + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> > > > + * zeroed, since this is not guaranteed on all compilers.
> > > > + */
> > > > + memset(&cmd, 0, sizeof(cmd));
> > >
> > > You could assign assign all these in the declaration and avoid zeroing the
> > > memory explicitly at the same time. I presume possibly leaking some
> > > information from memory to the firmware in case there are holes in the
> > > struct isn't an issue.
> >
> > Leaking kernel memory is bad. Also, there is no guarantee that the firmware will
> > behave as expected with varying values for the padding bytes.
> >
> > Please see my arguments from v4 on why these structs should be memset [1].
>
> There should be no host CPU related ABI introduced padding in structs defining
> firmware interfaces. Just use reserved fields in that case instead. In
> other words, the above memset() is equivalent to zeroing the memory using
> an assignment.
I understand that, but I don't work for AMD and don't have firmware source. :)
My personal preference is to pack firmware interface structs since relying on
reserved fields is subject to human error, especially for future changes to the
firmware interface.
That being said, please see what Bin said about the firmware interface [1]:
"Quoted below is Sultan's reply regarding this, does that make sense? On
the other hand, these definitions are shared between the ISP driver and
firmware and have been verified. I prefer not to add extra padding
fields to the driver as it would affect consistency. Is it acceptable to
leave the definitions as they are?"
> >
> > > > + cmd.stream_cfg.mipi_pipe_path_cfg.isp4fw_sensor_id = SENSOR_ID_ON_MIPI0;
> > > > + cmd.stream_cfg.mipi_pipe_path_cfg.b_enable = true;
> > > > + cmd.stream_cfg.isp_pipe_path_cfg.isp_pipe_id = MIPI0_ISP_PIPELINE_ID;
> > > > +
> > > > + cmd.stream_cfg.b_enable_tnr = true;
> > > > + dev_dbg(dev, "isp4fw_sensor_id %d, pipeId 0x%x EnableTnr %u\n",
> > > > + cmd.stream_cfg.mipi_pipe_path_cfg.isp4fw_sensor_id,
> > > > + cmd.stream_cfg.isp_pipe_path_cfg.isp_pipe_id,
> > > > + cmd.stream_cfg.b_enable_tnr);
> > > > +
> > > > + return isp4if_send_command(ispif, CMD_ID_SET_STREAM_CONFIG,
> > > > + &cmd, sizeof(cmd));
> > > > +}
> > > > +
> > > > +static int isp4sd_send_meta_buf(struct isp4_subdev *isp_subdev)
> > > > +{
> > > > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > > > + struct isp4fw_cmd_send_buffer buf_type;
> > > > + struct device *dev = isp_subdev->dev;
> > > > + int i;
> > >
> > > unsigned int, please. You can also declare this within the loop as you do
> > > elsewhere. Consistency is nice.
> >
> > Why does this need to be unsigned?
>
> Do you need negative numbers there?
No, but we don't need to make the variable explicitly unsigned either. It's more
common to see `int i;` than `unsigned int i;` too:
$ rg 'unsigned int i;' drivers/media/ | wc -l
904
$ rg 'int i;' drivers/media/ | rg -v unsigned | wc -l
1208
Making trivial loop iterators unsigned without reason inflates the code and adds
some confusion to future readers trying to understand if the 'unsigned' was
added because it was *required*, IMO.
> >
> > > > +
> > > > + /*
> > > > + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> > > > + * zeroed, since this is not guaranteed on all compilers.
> > > > + */
> > > > + memset(&buf_type, 0, sizeof(buf_type));
> > > > + for (i = 0; i < ISP4IF_MAX_STREAM_BUF_COUNT; i++) {
> > > > + struct isp4if_gpu_mem_info *meta_info_buf =
> > > > + isp_subdev->ispif.meta_info_buf[i];
> > > > + int ret;
> > > > +
> > > > + if (!meta_info_buf) {
> > > > + dev_err(dev, "fail for no meta info buf(%u)\n", i);
> > > > + return -ENOMEM;
> > > > + }
> > > > +
> > > > + buf_type.buffer_type = BUFFER_TYPE_META_INFO;
> > > > + buf_type.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
> > > > + isp4if_split_addr64(meta_info_buf->gpu_mc_addr,
> > > > + &buf_type.buffer.buf_base_a_lo,
> > > > + &buf_type.buffer.buf_base_a_hi);
> > > > + buf_type.buffer.buf_size_a = meta_info_buf->mem_size;
> > > > + ret = isp4if_send_command(ispif, CMD_ID_SEND_BUFFER,
> > > > + &buf_type, sizeof(buf_type));
> > > > + if (ret) {
> > > > + dev_err(dev, "send meta info(%u) fail\n", i);
> > > > + return ret;
> > > > + }
> > > > + }
> > > > +
> > > > + dev_dbg(dev, "send meta info suc\n");
> > > > + return 0;
> > > > +}
> > > > +
> > > > +static bool isp4sd_get_str_out_prop(struct isp4_subdev *isp_subdev,
> > > > + struct isp4fw_image_prop *out_prop,
> > > > + struct v4l2_subdev_state *state, u32 pad)
> > > > +{
> > > > + struct device *dev = isp_subdev->dev;
> > > > + struct v4l2_mbus_framefmt *format;
> > > > +
> > > > + format = v4l2_subdev_state_get_format(state, pad, 0);
> > > > + if (!format) {
> > > > + dev_err(dev, "fail get subdev state format\n");
> > > > + return false;
> > > > + }
> > > > +
> > > > + switch (format->code) {
> > > > + case MEDIA_BUS_FMT_YUYV8_1_5X8:
> > > > + out_prop->image_format = IMAGE_FORMAT_NV12;
> > > > + out_prop->width = format->width;
> > > > + out_prop->height = format->height;
> > > > + out_prop->luma_pitch = format->width;
> > > > + out_prop->chroma_pitch = out_prop->width;
> > > > + break;
> > > > + case MEDIA_BUS_FMT_YUYV8_1X16:
> > > > + out_prop->image_format = IMAGE_FORMAT_YUV422INTERLEAVED;
> > > > + out_prop->width = format->width;
> > > > + out_prop->height = format->height;
> > > > + out_prop->luma_pitch = format->width * 2;
> > > > + out_prop->chroma_pitch = 0;
> > > > + break;
> > > > + default:
> > > > + dev_err(dev, "fail for bad image format:0x%x\n",
> > > > + format->code);
> > > > + return false;
> > > > + }
> > > > +
> > > > + if (!out_prop->width || !out_prop->height)
> > > > + return false;
> > > > +
> > > > + return true;
> > > > +}
> > > > +
> > > > +static int isp4sd_kickoff_stream(struct isp4_subdev *isp_subdev, u32 w, u32 h)
> > > > +{
> > > > + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
> > > > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > > > + struct device *dev = isp_subdev->dev;
> > > > +
> > > > + if (sensor_info->status == ISP4SD_START_STATUS_STARTED)
> > > > + return 0;
> > > > +
> > > > + if (sensor_info->status == ISP4SD_START_STATUS_START_FAIL) {
> > > > + dev_err(dev, "fail for previous start fail\n");
> > > > + return -EINVAL;
> > > > + }
> > > > +
> > > > + dev_dbg(dev, "w:%u,h:%u\n", w, h);
> > > > +
> > > > + if (isp4sd_send_meta_buf(isp_subdev)) {
> > > > + dev_err(dev, "fail to send meta buf\n");
> > > > + sensor_info->status = ISP4SD_START_STATUS_START_FAIL;
> > > > + return -EINVAL;
> > > > + }
> > > > +
> > > > + sensor_info->status = ISP4SD_START_STATUS_OFF;
> > > > +
> > > > + if (!sensor_info->start_stream_cmd_sent &&
> > > > + sensor_info->buf_sent_cnt >= ISP4SD_MIN_BUF_CNT_BEF_START_STREAM) {
> > > > + int ret = isp4if_send_command(ispif, CMD_ID_START_STREAM,
> > > > + NULL, 0);
> > > > + if (ret) {
> > > > + dev_err(dev, "fail to start stream\n");
> > > > + return ret;
> > > > + }
> > > > +
> > > > + sensor_info->start_stream_cmd_sent = true;
> > > > + } else {
> > > > + dev_dbg(dev,
> > > > + "no send START_STREAM, start_sent %u, buf_sent %u\n",
> > > > + sensor_info->start_stream_cmd_sent,
> > > > + sensor_info->buf_sent_cnt);
> > > > + }
> > > > +
> > > > + return 0;
> > > > +}
> > > > +
> > > > +static int isp4sd_setup_output(struct isp4_subdev *isp_subdev,
> > > > + struct v4l2_subdev_state *state, u32 pad)
> > > > +{
> > > > + struct isp4sd_output_info *output_info = &isp_subdev->sensor_info.output_info;
> > > > + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
> > > > + struct isp4_interface *ispif = &isp_subdev->ispif;
> > > > + struct isp4fw_cmd_set_out_ch_prop cmd_ch_prop;
> > > > + struct isp4fw_cmd_enable_out_ch cmd_ch_en;
> > > > + struct device *dev = isp_subdev->dev;
> > > > + int ret;
> > > > +
> > > > + if (output_info->start_status == ISP4SD_START_STATUS_STARTED)
> > > > + return 0;
> > > > +
> > > > + if (output_info->start_status == ISP4SD_START_STATUS_START_FAIL) {
> > > > + dev_err(dev, "fail for previous start fail\n");
> > > > + return -EINVAL;
> > > > + }
> > > > +
> > > > + /*
> > > > + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> > > > + * zeroed, since this is not guaranteed on all compilers.
> > > > + */
> > > > + memset(&cmd_ch_prop, 0, sizeof(cmd_ch_prop));
> > > > + cmd_ch_prop.ch = ISP_PIPE_OUT_CH_PREVIEW;
> > > > +
> > > > + if (!isp4sd_get_str_out_prop(isp_subdev, &cmd_ch_prop.image_prop, state, pad)) {
> > > > + dev_err(dev, "fail to get out prop\n");
> > > > + return -EINVAL;
> > > > + }
> > > > +
> > > > + dev_dbg(dev, "channel:%d,fmt %d,w:h=%u:%u,lp:%u,cp%u\n",
> > > > + cmd_ch_prop.ch,
> > > > + cmd_ch_prop.image_prop.image_format,
> > > > + cmd_ch_prop.image_prop.width, cmd_ch_prop.image_prop.height,
> > > > + cmd_ch_prop.image_prop.luma_pitch,
> > > > + cmd_ch_prop.image_prop.chroma_pitch);
> > > > +
> > > > + ret = isp4if_send_command(ispif, CMD_ID_SET_OUT_CHAN_PROP,
> > > > + &cmd_ch_prop, sizeof(cmd_ch_prop));
> > > > + if (ret) {
> > > > + output_info->start_status = ISP4SD_START_STATUS_START_FAIL;
> > > > + dev_err(dev, "fail to set out prop\n");
> > > > + return ret;
> > > > + }
> > > > +
> > > > + /*
> > > > + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
> > > > + * zeroed, since this is not guaranteed on all compilers.
> > >
> > > You should have explicit padding fields in any case and not rely on ABI in
> > > this case.
> >
> > It is error-prone for a human to make sure that all padding bytes have
> > explicit struct members. And what about future changes to the firmware
> > API where explicit padding might be forgotten?
>
> Just don't do that. Use pahole to verify the result when making changes to
> the structs.
Humans are fallible. Someone will undoubtedly make this mistake in the future
without some way in place to either automatically run pahole and scrape the
output for holes in firmware API structs or memset the whole struct at runtime
so it never matters. OR slap __packed onto all those structs.
> >
> > Unless the firmware API structs are all __packed in a future firmware update, I
> > think the memsets should remain.
>
> If you want to be certain of the size of the structs, use BUILD_BUG_ON().
This won't help for the addition of new structs and still requires a human to
"do the right thing" and make sure there aren't any holes when they hardcode the
struct size into a compile-time assert.
> --
> Kind regards,
>
> Sakari Ailus
[1] https://lore.kernel.org/all/62bd8248-dd8a-4d51-8a85-ad13d3a03180@amd.com/
Sultan
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added
2026-01-15 7:21 ` Sultan Alsawaf
@ 2026-01-20 9:29 ` Du, Bin
2026-01-27 8:39 ` Du, Bin
0 siblings, 1 reply; 46+ messages in thread
From: Du, Bin @ 2026-01-20 9:29 UTC (permalink / raw)
To: Sultan Alsawaf, Sakari Ailus
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Alexey Zagorodnikov
Hi Skari, since this is a mature product, the ISP driver/FW interface is
fixed and should not change. The only remaining question is how to
initialize them—by declaration or with memset. Would you please share
your preference?
On 1/15/2026 3:21 PM, Sultan Alsawaf wrote:
> On Wed, Jan 14, 2026 at 11:03:49PM +0200, Sakari Ailus wrote:
>> Hi Sultan,
>>
>> On Tue, Jan 06, 2026 at 11:33:53PM -0800, Sultan Alsawaf wrote:
>>> Hi Sakari,
>>>
>>> On Mon, Dec 22, 2025 at 12:11:11PM +0200, Sakari Ailus wrote:
>>>> Hi Bin,
>>>>
>>>> On Tue, Dec 16, 2025 at 05:13:23PM +0800, Bin Du wrote:
>>>>> +static int isp4sd_set_stream_path(struct isp4_subdev *isp_subdev)
>>>>> +{
>>>>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>>>>> + struct isp4fw_cmd_set_stream_cfg cmd;
>>>>> + struct device *dev = isp_subdev->dev;
>>>>> +
>>>>> + /*
>>>>> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
>>>>> + * zeroed, since this is not guaranteed on all compilers.
>>>>> + */
>>>>> + memset(&cmd, 0, sizeof(cmd));
>>>>
>>>> You could assign assign all these in the declaration and avoid zeroing the
>>>> memory explicitly at the same time. I presume possibly leaking some
>>>> information from memory to the firmware in case there are holes in the
>>>> struct isn't an issue.
>>>
>>> Leaking kernel memory is bad. Also, there is no guarantee that the firmware will
>>> behave as expected with varying values for the padding bytes.
>>>
>>> Please see my arguments from v4 on why these structs should be memset [1].
>>
>> There should be no host CPU related ABI introduced padding in structs defining
>> firmware interfaces. Just use reserved fields in that case instead. In
>> other words, the above memset() is equivalent to zeroing the memory using
>> an assignment.
>
> I understand that, but I don't work for AMD and don't have firmware source. :)
>
> My personal preference is to pack firmware interface structs since relying on
> reserved fields is subject to human error, especially for future changes to the
> firmware interface.
>
> That being said, please see what Bin said about the firmware interface [1]:
> "Quoted below is Sultan's reply regarding this, does that make sense? On
> the other hand, these definitions are shared between the ISP driver and
> firmware and have been verified. I prefer not to add extra padding
> fields to the driver as it would affect consistency. Is it acceptable to
> leave the definitions as they are?"
>
>>>
>>>>> + cmd.stream_cfg.mipi_pipe_path_cfg.isp4fw_sensor_id = SENSOR_ID_ON_MIPI0;
>>>>> + cmd.stream_cfg.mipi_pipe_path_cfg.b_enable = true;
>>>>> + cmd.stream_cfg.isp_pipe_path_cfg.isp_pipe_id = MIPI0_ISP_PIPELINE_ID;
>>>>> +
>>>>> + cmd.stream_cfg.b_enable_tnr = true;
>>>>> + dev_dbg(dev, "isp4fw_sensor_id %d, pipeId 0x%x EnableTnr %u\n",
>>>>> + cmd.stream_cfg.mipi_pipe_path_cfg.isp4fw_sensor_id,
>>>>> + cmd.stream_cfg.isp_pipe_path_cfg.isp_pipe_id,
>>>>> + cmd.stream_cfg.b_enable_tnr);
>>>>> +
>>>>> + return isp4if_send_command(ispif, CMD_ID_SET_STREAM_CONFIG,
>>>>> + &cmd, sizeof(cmd));
>>>>> +}
>>>>> +
>>>>> +static int isp4sd_send_meta_buf(struct isp4_subdev *isp_subdev)
>>>>> +{
>>>>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>>>>> + struct isp4fw_cmd_send_buffer buf_type;
>>>>> + struct device *dev = isp_subdev->dev;
>>>>> + int i;
>>>>
>>>> unsigned int, please. You can also declare this within the loop as you do
>>>> elsewhere. Consistency is nice.
>>>
>>> Why does this need to be unsigned?
>>
>> Do you need negative numbers there?
>
> No, but we don't need to make the variable explicitly unsigned either. It's more
> common to see `int i;` than `unsigned int i;` too:
>
> $ rg 'unsigned int i;' drivers/media/ | wc -l
> 904
> $ rg 'int i;' drivers/media/ | rg -v unsigned | wc -l
> 1208
>
> Making trivial loop iterators unsigned without reason inflates the code and adds
> some confusion to future readers trying to understand if the 'unsigned' was
> added because it was *required*, IMO.
>
>>>
>>>>> +
>>>>> + /*
>>>>> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
>>>>> + * zeroed, since this is not guaranteed on all compilers.
>>>>> + */
>>>>> + memset(&buf_type, 0, sizeof(buf_type));
>>>>> + for (i = 0; i < ISP4IF_MAX_STREAM_BUF_COUNT; i++) {
>>>>> + struct isp4if_gpu_mem_info *meta_info_buf =
>>>>> + isp_subdev->ispif.meta_info_buf[i];
>>>>> + int ret;
>>>>> +
>>>>> + if (!meta_info_buf) {
>>>>> + dev_err(dev, "fail for no meta info buf(%u)\n", i);
>>>>> + return -ENOMEM;
>>>>> + }
>>>>> +
>>>>> + buf_type.buffer_type = BUFFER_TYPE_META_INFO;
>>>>> + buf_type.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
>>>>> + isp4if_split_addr64(meta_info_buf->gpu_mc_addr,
>>>>> + &buf_type.buffer.buf_base_a_lo,
>>>>> + &buf_type.buffer.buf_base_a_hi);
>>>>> + buf_type.buffer.buf_size_a = meta_info_buf->mem_size;
>>>>> + ret = isp4if_send_command(ispif, CMD_ID_SEND_BUFFER,
>>>>> + &buf_type, sizeof(buf_type));
>>>>> + if (ret) {
>>>>> + dev_err(dev, "send meta info(%u) fail\n", i);
>>>>> + return ret;
>>>>> + }
>>>>> + }
>>>>> +
>>>>> + dev_dbg(dev, "send meta info suc\n");
>>>>> + return 0;
>>>>> +}
>>>>> +
>>>>> +static bool isp4sd_get_str_out_prop(struct isp4_subdev *isp_subdev,
>>>>> + struct isp4fw_image_prop *out_prop,
>>>>> + struct v4l2_subdev_state *state, u32 pad)
>>>>> +{
>>>>> + struct device *dev = isp_subdev->dev;
>>>>> + struct v4l2_mbus_framefmt *format;
>>>>> +
>>>>> + format = v4l2_subdev_state_get_format(state, pad, 0);
>>>>> + if (!format) {
>>>>> + dev_err(dev, "fail get subdev state format\n");
>>>>> + return false;
>>>>> + }
>>>>> +
>>>>> + switch (format->code) {
>>>>> + case MEDIA_BUS_FMT_YUYV8_1_5X8:
>>>>> + out_prop->image_format = IMAGE_FORMAT_NV12;
>>>>> + out_prop->width = format->width;
>>>>> + out_prop->height = format->height;
>>>>> + out_prop->luma_pitch = format->width;
>>>>> + out_prop->chroma_pitch = out_prop->width;
>>>>> + break;
>>>>> + case MEDIA_BUS_FMT_YUYV8_1X16:
>>>>> + out_prop->image_format = IMAGE_FORMAT_YUV422INTERLEAVED;
>>>>> + out_prop->width = format->width;
>>>>> + out_prop->height = format->height;
>>>>> + out_prop->luma_pitch = format->width * 2;
>>>>> + out_prop->chroma_pitch = 0;
>>>>> + break;
>>>>> + default:
>>>>> + dev_err(dev, "fail for bad image format:0x%x\n",
>>>>> + format->code);
>>>>> + return false;
>>>>> + }
>>>>> +
>>>>> + if (!out_prop->width || !out_prop->height)
>>>>> + return false;
>>>>> +
>>>>> + return true;
>>>>> +}
>>>>> +
>>>>> +static int isp4sd_kickoff_stream(struct isp4_subdev *isp_subdev, u32 w, u32 h)
>>>>> +{
>>>>> + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
>>>>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>>>>> + struct device *dev = isp_subdev->dev;
>>>>> +
>>>>> + if (sensor_info->status == ISP4SD_START_STATUS_STARTED)
>>>>> + return 0;
>>>>> +
>>>>> + if (sensor_info->status == ISP4SD_START_STATUS_START_FAIL) {
>>>>> + dev_err(dev, "fail for previous start fail\n");
>>>>> + return -EINVAL;
>>>>> + }
>>>>> +
>>>>> + dev_dbg(dev, "w:%u,h:%u\n", w, h);
>>>>> +
>>>>> + if (isp4sd_send_meta_buf(isp_subdev)) {
>>>>> + dev_err(dev, "fail to send meta buf\n");
>>>>> + sensor_info->status = ISP4SD_START_STATUS_START_FAIL;
>>>>> + return -EINVAL;
>>>>> + }
>>>>> +
>>>>> + sensor_info->status = ISP4SD_START_STATUS_OFF;
>>>>> +
>>>>> + if (!sensor_info->start_stream_cmd_sent &&
>>>>> + sensor_info->buf_sent_cnt >= ISP4SD_MIN_BUF_CNT_BEF_START_STREAM) {
>>>>> + int ret = isp4if_send_command(ispif, CMD_ID_START_STREAM,
>>>>> + NULL, 0);
>>>>> + if (ret) {
>>>>> + dev_err(dev, "fail to start stream\n");
>>>>> + return ret;
>>>>> + }
>>>>> +
>>>>> + sensor_info->start_stream_cmd_sent = true;
>>>>> + } else {
>>>>> + dev_dbg(dev,
>>>>> + "no send START_STREAM, start_sent %u, buf_sent %u\n",
>>>>> + sensor_info->start_stream_cmd_sent,
>>>>> + sensor_info->buf_sent_cnt);
>>>>> + }
>>>>> +
>>>>> + return 0;
>>>>> +}
>>>>> +
>>>>> +static int isp4sd_setup_output(struct isp4_subdev *isp_subdev,
>>>>> + struct v4l2_subdev_state *state, u32 pad)
>>>>> +{
>>>>> + struct isp4sd_output_info *output_info = &isp_subdev->sensor_info.output_info;
>>>>> + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
>>>>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>>>>> + struct isp4fw_cmd_set_out_ch_prop cmd_ch_prop;
>>>>> + struct isp4fw_cmd_enable_out_ch cmd_ch_en;
>>>>> + struct device *dev = isp_subdev->dev;
>>>>> + int ret;
>>>>> +
>>>>> + if (output_info->start_status == ISP4SD_START_STATUS_STARTED)
>>>>> + return 0;
>>>>> +
>>>>> + if (output_info->start_status == ISP4SD_START_STATUS_START_FAIL) {
>>>>> + dev_err(dev, "fail for previous start fail\n");
>>>>> + return -EINVAL;
>>>>> + }
>>>>> +
>>>>> + /*
>>>>> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
>>>>> + * zeroed, since this is not guaranteed on all compilers.
>>>>> + */
>>>>> + memset(&cmd_ch_prop, 0, sizeof(cmd_ch_prop));
>>>>> + cmd_ch_prop.ch = ISP_PIPE_OUT_CH_PREVIEW;
>>>>> +
>>>>> + if (!isp4sd_get_str_out_prop(isp_subdev, &cmd_ch_prop.image_prop, state, pad)) {
>>>>> + dev_err(dev, "fail to get out prop\n");
>>>>> + return -EINVAL;
>>>>> + }
>>>>> +
>>>>> + dev_dbg(dev, "channel:%d,fmt %d,w:h=%u:%u,lp:%u,cp%u\n",
>>>>> + cmd_ch_prop.ch,
>>>>> + cmd_ch_prop.image_prop.image_format,
>>>>> + cmd_ch_prop.image_prop.width, cmd_ch_prop.image_prop.height,
>>>>> + cmd_ch_prop.image_prop.luma_pitch,
>>>>> + cmd_ch_prop.image_prop.chroma_pitch);
>>>>> +
>>>>> + ret = isp4if_send_command(ispif, CMD_ID_SET_OUT_CHAN_PROP,
>>>>> + &cmd_ch_prop, sizeof(cmd_ch_prop));
>>>>> + if (ret) {
>>>>> + output_info->start_status = ISP4SD_START_STATUS_START_FAIL;
>>>>> + dev_err(dev, "fail to set out prop\n");
>>>>> + return ret;
>>>>> + }
>>>>> +
>>>>> + /*
>>>>> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
>>>>> + * zeroed, since this is not guaranteed on all compilers.
>>>>
>>>> You should have explicit padding fields in any case and not rely on ABI in
>>>> this case.
>>>
>>> It is error-prone for a human to make sure that all padding bytes have
>>> explicit struct members. And what about future changes to the firmware
>>> API where explicit padding might be forgotten?
>>
>> Just don't do that. Use pahole to verify the result when making changes to
>> the structs.
>
> Humans are fallible. Someone will undoubtedly make this mistake in the future
> without some way in place to either automatically run pahole and scrape the
> output for holes in firmware API structs or memset the whole struct at runtime
> so it never matters. OR slap __packed onto all those structs.
>
>>>
>>> Unless the firmware API structs are all __packed in a future firmware update, I
>>> think the memsets should remain.
>>
>> If you want to be certain of the size of the structs, use BUILD_BUG_ON().
>
> This won't help for the addition of new structs and still requires a human to
> "do the right thing" and make sure there aren't any holes when they hardcode the
> struct size into a compile-time assert.
>
>> --
>> Kind regards,
>>
>> Sakari Ailus
>
> [1] https://lore.kernel.org/all/62bd8248-dd8a-4d51-8a85-ad13d3a03180@amd.com/
>
> Sultan
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added
2026-01-20 9:29 ` Du, Bin
@ 2026-01-27 8:39 ` Du, Bin
0 siblings, 0 replies; 46+ messages in thread
From: Du, Bin @ 2026-01-27 8:39 UTC (permalink / raw)
To: Sultan Alsawaf, Sakari Ailus
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Alexey Zagorodnikov
Hi Sakari, apologies for any inconvenience this may cause. Would you
please confirm the following remaining one item in the patch at your
earliest convenience, so that we can ensure its inclusion in version 8?
On 1/20/2026 5:29 PM, Du, Bin wrote:
> Hi Skari, since this is a mature product, the ISP driver/FW interface is
> fixed and should not change. The only remaining question is how to
> initialize them—by declaration or with memset. Would you please share
> your preference?
>
> On 1/15/2026 3:21 PM, Sultan Alsawaf wrote:
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added
2026-01-14 21:03 ` Sakari Ailus
2026-01-14 21:08 ` Mario Limonciello
2026-01-15 7:21 ` Sultan Alsawaf
@ 2026-01-15 10:36 ` Du, Bin
2 siblings, 0 replies; 46+ messages in thread
From: Du, Bin @ 2026-01-15 10:36 UTC (permalink / raw)
To: Sakari Ailus, Sultan Alsawaf
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Alexey Zagorodnikov
On 1/15/2026 5:03 AM, Sakari Ailus wrote:
> Hi Sultan,
>
> On Tue, Jan 06, 2026 at 11:33:53PM -0800, Sultan Alsawaf wrote:
>> Hi Sakari,
>>
>> On Mon, Dec 22, 2025 at 12:11:11PM +0200, Sakari Ailus wrote:
>>> Hi Bin,
>>>
>>> On Tue, Dec 16, 2025 at 05:13:23PM +0800, Bin Du wrote:
>>>> Isp4 sub-device is implementing v4l2 sub-device interface. It has one
>>>> capture video node, and supports only preview stream. It manages firmware
>>>> states, stream configuration. Add interrupt handling and notification for
>>>> isp firmware to isp-subdevice.
>>>>
>>>> Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
>>>> Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
>>>> Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
>>>> Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
>>>> Signed-off-by: Bin Du <Bin.Du@amd.com>
>>>> Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
>>>> ---
>>>> MAINTAINERS | 2 +
>>>> drivers/media/platform/amd/isp4/Makefile | 3 +-
>>>> drivers/media/platform/amd/isp4/isp4.c | 99 +-
>>>> drivers/media/platform/amd/isp4/isp4.h | 7 +-
>>>> drivers/media/platform/amd/isp4/isp4_subdev.c | 975 ++++++++++++++++++
>>>> drivers/media/platform/amd/isp4/isp4_subdev.h | 124 +++
>>>> 6 files changed, 1202 insertions(+), 8 deletions(-)
>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
>>>>
>>>> diff --git a/MAINTAINERS b/MAINTAINERS
>>>> index cccae369c876..48ffc8bbdcee 100644
>>>> --- a/MAINTAINERS
>>>> +++ b/MAINTAINERS
>>>> @@ -1149,6 +1149,8 @@ F: drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
>>>> F: drivers/media/platform/amd/isp4/isp4_hw_reg.h
>>>> F: drivers/media/platform/amd/isp4/isp4_interface.c
>>>> F: drivers/media/platform/amd/isp4/isp4_interface.h
>>>> +F: drivers/media/platform/amd/isp4/isp4_subdev.c
>>>> +F: drivers/media/platform/amd/isp4/isp4_subdev.h
>>>>
>>>> AMD KFD
>>>> M: Felix Kuehling <Felix.Kuehling@amd.com>
>>>> diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
>>>> index a2a5bf98e912..6d4e6d6ac7f5 100644
>>>> --- a/drivers/media/platform/amd/isp4/Makefile
>>>> +++ b/drivers/media/platform/amd/isp4/Makefile
>>>> @@ -4,4 +4,5 @@
>>>>
>>>> obj-$(CONFIG_AMD_ISP4) += amd_capture.o
>>>> amd_capture-objs := isp4.o \
>>>> - isp4_interface.o
>>>> + isp4_interface.o \
>>>> + isp4_subdev.o
>>>> diff --git a/drivers/media/platform/amd/isp4/isp4.c b/drivers/media/platform/amd/isp4/isp4.c
>>>> index ad95e7f89189..bcd7cad32afd 100644
>>>> --- a/drivers/media/platform/amd/isp4/isp4.c
>>>> +++ b/drivers/media/platform/amd/isp4/isp4.c
>>
>> [snip]
>>
>>>> static irqreturn_t isp4_irq_handler(int irq, void *arg)
>>>> {
>>>> + struct isp4_subdev *isp_subdev = arg;
>>>> + u32 intr_ack = 0, intr_en = 0, intr_status;
>>>> + int seen = 0;
>>>
>>> Is int appropriate here? Should this be u32 or u64?
>>
>> ffs() is just a macro alias for __builtin_ffs(). The parameter and return value
>> of __builtin_ffs() are both int.
>
> Ack, sounds reasonable.
>
Thanks for confirming, Sakari.
>>
[snip]
>>> unsigned int, please.
>>
>> As mentioned above, ffs() takes an int and returns an int.
>>
>>> The parentheses around ffs() appear redundant.
>>
>> The parentheses are there because it's an assignment. Without them:
>>
>> drivers/media/platform/amd/isp4/isp4.c: In function ‘isp4_irq_handler’:
>> drivers/media/platform/amd/isp4/isp4.c:106:21: warning: suggest parentheses around assignment used as truth value [-Wparentheses]
>> 106 | for (int i; i = ffs(seen); seen = (seen >> i) << i)
>> | ^
>>
>>> The increment could probably be expressed as seen &= ~BIT(i).
>>
>> Yes it can be, but it adds several more instructions before the loop body,
>> without any improvement to the loop body (the sarx in the loop body is replaced
>> by andn). The right shift trick is faster and this is a hot path (IRQ handler).
>
> Fine by me, it won't make much difference in practice either way.
>
> ...
>
Yes, both options work for me either, my idea is to keep it as is and
add the following comment.
/*
* The operation `(seen >> i) << i` is logically equivalent to
* `seen &= ~BIT(i)`, with fewer instructions after compilation.
*/
[snip]
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added
2025-12-22 10:11 ` Sakari Ailus
2026-01-07 7:33 ` Sultan Alsawaf
@ 2026-01-14 10:34 ` Du, Bin
2026-01-14 21:07 ` Sakari Ailus
1 sibling, 1 reply; 46+ messages in thread
From: Du, Bin @ 2026-01-14 10:34 UTC (permalink / raw)
To: Sakari Ailus
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Alexey Zagorodnikov
Thank you, Sakari, for your review.
On 12/22/2025 6:11 PM, Sakari Ailus wrote:
> Hi Bin,
>
> On Tue, Dec 16, 2025 at 05:13:23PM +0800, Bin Du wrote:
>> Isp4 sub-device is implementing v4l2 sub-device interface. It has one
>> capture video node, and supports only preview stream. It manages firmware
>> states, stream configuration. Add interrupt handling and notification for
>> isp firmware to isp-subdevice.
>>
>> Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
>> Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
>> Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
>> Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
>> Signed-off-by: Bin Du <Bin.Du@amd.com>
>> Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
>> ---
>> MAINTAINERS | 2 +
>> drivers/media/platform/amd/isp4/Makefile | 3 +-
>> drivers/media/platform/amd/isp4/isp4.c | 99 +-
>> drivers/media/platform/amd/isp4/isp4.h | 7 +-
>> drivers/media/platform/amd/isp4/isp4_subdev.c | 975 ++++++++++++++++++
>> drivers/media/platform/amd/isp4/isp4_subdev.h | 124 +++
>> 6 files changed, 1202 insertions(+), 8 deletions(-)
>> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
>> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index cccae369c876..48ffc8bbdcee 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -1149,6 +1149,8 @@ F: drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
>> F: drivers/media/platform/amd/isp4/isp4_hw_reg.h
>> F: drivers/media/platform/amd/isp4/isp4_interface.c
>> F: drivers/media/platform/amd/isp4/isp4_interface.h
>> +F: drivers/media/platform/amd/isp4/isp4_subdev.c
>> +F: drivers/media/platform/amd/isp4/isp4_subdev.h
>>
>> AMD KFD
>> M: Felix Kuehling <Felix.Kuehling@amd.com>
>> diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
>> index a2a5bf98e912..6d4e6d6ac7f5 100644
>> --- a/drivers/media/platform/amd/isp4/Makefile
>> +++ b/drivers/media/platform/amd/isp4/Makefile
>> @@ -4,4 +4,5 @@
>>
>> obj-$(CONFIG_AMD_ISP4) += amd_capture.o
>> amd_capture-objs := isp4.o \
>> - isp4_interface.o
>> + isp4_interface.o \
>> + isp4_subdev.o
>> diff --git a/drivers/media/platform/amd/isp4/isp4.c b/drivers/media/platform/amd/isp4/isp4.c
>> index ad95e7f89189..bcd7cad32afd 100644
>> --- a/drivers/media/platform/amd/isp4/isp4.c
>> +++ b/drivers/media/platform/amd/isp4/isp4.c
>> @@ -3,15 +3,19 @@
>> * Copyright (C) 2025 Advanced Micro Devices, Inc.
>> */
>>
>> +#include <linux/irq.h>
>> #include <linux/pm_runtime.h>
>> #include <linux/vmalloc.h>
>> +#include <media/v4l2-fwnode.h>
>
> I don't think you need this one.
>
Yes, will remove it.
>> #include <media/v4l2-ioctl.h>
>>
>> #include "isp4.h"
>> -
>> -#define VIDEO_BUF_NUM 5
>
> What happened with this one? Is it no longer needed?
Yes, it is not used. It was added in the first patch and removed by this
one in the set, so, will not include it in the first patch.
>
>> +#include "isp4_hw_reg.h"
>>
>> #define ISP4_DRV_NAME "amd_isp_capture"
>> +#define ISP4_FW_RESP_RB_IRQ_STATUS_MASK \
>> + (ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT9_INT_MASK | \
>> + ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT12_INT_MASK)
>>
>> static const struct {
>> const char *name;
>> @@ -19,27 +23,97 @@ static const struct {
>> u32 en_mask;
>> u32 ack_mask;
>> u32 rb_int_num;
>> -} isp4_irq[] = {
>> +} isp4_irq[ISP4SD_MAX_FW_RESP_STREAM_NUM] = {
>> /* The IRQ order is aligned with the isp4_subdev.fw_resp_thread order */
>> {
>> .name = "isp_irq_global",
>> + .status_mask = ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT12_INT_MASK,
>> + .en_mask = ISP_SYS_INT0_EN__SYS_INT_RINGBUFFER_WPT12_EN_MASK,
>> + .ack_mask = ISP_SYS_INT0_ACK__SYS_INT_RINGBUFFER_WPT12_ACK_MASK,
>> .rb_int_num = 4, /* ISP_4_1__SRCID__ISP_RINGBUFFER_WPT12 */
>> },
>> {
>> .name = "isp_irq_stream1",
>> + .status_mask = ISP_SYS_INT0_STATUS__SYS_INT_RINGBUFFER_WPT9_INT_MASK,
>> + .en_mask = ISP_SYS_INT0_EN__SYS_INT_RINGBUFFER_WPT9_EN_MASK,
>> + .ack_mask = ISP_SYS_INT0_ACK__SYS_INT_RINGBUFFER_WPT9_ACK_MASK,
>> .rb_int_num = 0, /* ISP_4_1__SRCID__ISP_RINGBUFFER_WPT9 */
>> },
>> };
>>
>> +void isp4_intr_enable(struct isp4_subdev *isp_subdev, u32 index, bool enable)
>> +{
>> + u32 intr_en;
>> +
>> + /* Synchronize ISP_SYS_INT0_EN writes with the IRQ handler's writes */
>> + spin_lock_irq(&isp_subdev->irq_lock);
>> + intr_en = isp4hw_rreg(isp_subdev->mmio, ISP_SYS_INT0_EN);
>> + if (enable)
>> + intr_en |= isp4_irq[index].en_mask;
>> + else
>> + intr_en &= ~isp4_irq[index].en_mask;
>> +
>> + isp4hw_wreg(isp_subdev->mmio, ISP_SYS_INT0_EN, intr_en);
>> + spin_unlock_irq(&isp_subdev->irq_lock);
>> +}
>> +
>> +static void isp4_wake_up_resp_thread(struct isp4_subdev *isp_subdev, u32 index)
>> +{
>> + struct isp4sd_thread_handler *thread_ctx = &isp_subdev->fw_resp_thread[index];
>> +
>> + thread_ctx->resp_ready = true;
>> + wake_up_interruptible(&thread_ctx->waitq);
>> +}
>> +
>> static irqreturn_t isp4_irq_handler(int irq, void *arg)
>> {
>> + struct isp4_subdev *isp_subdev = arg;
>> + u32 intr_ack = 0, intr_en = 0, intr_status;
>> + int seen = 0;
>
> Is int appropriate here? Should this be u32 or u64?
>
Quoted below is Sultan's reply regarding this,
"ffs() is just a macro alias for __builtin_ffs(). The parameter and
return value
of __builtin_ffs() are both int."
>> +
>> + /* Get the ISP_SYS interrupt status */
>> + intr_status = isp4hw_rreg(isp_subdev->mmio, ISP_SYS_INT0_STATUS);
>> + intr_status &= ISP4_FW_RESP_RB_IRQ_STATUS_MASK;
>> +
>> + /* Find which ISP_SYS interrupts fired */
>> + for (size_t i = 0; i < ARRAY_SIZE(isp4_irq); i++) {
>> + if (intr_status & isp4_irq[i].status_mask) {
>> + intr_ack |= isp4_irq[i].ack_mask;
>> + intr_en |= isp4_irq[i].en_mask;
>> + seen |= BIT(i);
>> + }
>> + }
>> +
>> + /*
>> + * Disable the ISP_SYS interrupts that fired. Must be done before waking
>> + * the response threads, since they re-enable interrupts when finished.
>> + * The lock synchronizes RMW of INT0_EN with isp4_enable_interrupt().
>> + */
>> + spin_lock(&isp_subdev->irq_lock);
>> + intr_en = isp4hw_rreg(isp_subdev->mmio, ISP_SYS_INT0_EN) & ~intr_en;
>> + isp4hw_wreg(isp_subdev->mmio, ISP_SYS_INT0_EN, intr_en);
>> + spin_unlock(&isp_subdev->irq_lock);
>> +
>> + /*
>> + * Clear the ISP_SYS interrupts. This must be done after the interrupts
>> + * are disabled, so that ISP FW won't flag any new interrupts on these
>> + * streams, and thus we don't need to clear interrupts again before
>> + * re-enabling them in the response thread.
>> + */
>> + isp4hw_wreg(isp_subdev->mmio, ISP_SYS_INT0_ACK, intr_ack);
>> +
>> + /* Wake up the response threads */
>> + for (int i; (i = ffs(seen)); seen = (seen >> i) << i)
>
> unsigned int, please. The parentheses around ffs() appear redundant.
>
Quoted below is Sultan's reply regarding this,
"As mentioned above, ffs() takes an int and returns an int.
The parentheses are there because it's an assignment. Without them:
drivers/media/platform/amd/isp4/isp4.c: In function ‘isp4_irq_handler’:
drivers/media/platform/amd/isp4/isp4.c:106:21: warning: suggest
parentheses around assignment used as truth value [-Wparentheses]
106 | for (int i; i = ffs(seen); seen = (seen >> i) << i)
| ^"
> The increment could probably be expressed as seen &= ~BIT(i).
>
Quoted below is Sultan's reply regarding this, Would it be acceptable to
leave it unchanged?
"Yes it can be, but it adds several more instructions before the loop body,
without any improvement to the loop body (the sarx in the loop body is
replaced
by andn). The right shift trick is faster and this is a hot path (IRQ
handler).
Before:
d8: jmp 10d
// loop body
da: lea -0x1(%rax),%eax
dd: xor %ecx,%ecx
df: mov $0x1,%edx
e4: mov $0x1,%esi
e9: lea (%rax,%rax,4),%rax
ed: sarx %ebx,%ebp,%ebp
f2: shlx %ebx,%ebp,%ebp
f7: movb $0x1,0x9b0(%r13,%rax,8)
100: lea 0x998(%r13,%rax,8),%rdi
108: call __wake_up
10d: mov $0xffffffff,%eax
112: bsf %ebp,%eax
115: add $0x1,%eax
118: mov %eax,%ebx
11a: jne da
After (with seen &= ~BIT(i)):
d8: mov $0xffffffff,%eax
dd: bsf %ebp,%eax
e0: add $0x1,%eax
e3: mov %eax,%ebx
e5: je 12f
e7: mov $0x1,%r12d
// loop body
ed: lea -0x1(%rbx),%eax
f0: xor %ecx,%ecx
f2: mov $0x1,%edx
f7: mov $0x1,%esi
fc: lea (%rax,%rax,4),%rax
100: movb $0x1,0x9b0(%r13,%rax,8)
109: lea 0x998(%r13,%rax,8),%rdi
111: call __wake_up
116: shlx %rbx,%r12,%rax
11b: andn %ebp,%eax,%ebp
120: mov $0xffffffff,%eax
125: bsf %ebp,%eax
128: add $0x1,%eax
12b: mov %eax,%ebx
12d: jne ed"
>> + isp4_wake_up_resp_thread(isp_subdev, i - 1);
>> +
>> return IRQ_HANDLED;
>> }
>>
>> static int isp4_capture_probe(struct platform_device *pdev)
>> {
>> + int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM];
>> struct device *dev = &pdev->dev;
>> - int irq[ARRAY_SIZE(isp4_irq)];
>> + struct isp4_subdev *isp_subdev;
>> struct isp4_device *isp_dev;
>> size_t i;
>> int ret;
>> @@ -50,6 +124,11 @@ static int isp4_capture_probe(struct platform_device *pdev)
>>
>> dev->init_name = ISP4_DRV_NAME;
>>
>> + isp_subdev = &isp_dev->isp_subdev;
>> + isp_subdev->mmio = devm_platform_ioremap_resource(pdev, 0);
>> + if (IS_ERR(isp_subdev->mmio))
>> + return dev_err_probe(dev, PTR_ERR(isp_subdev->mmio), "isp ioremap fail\n");
>> +
>> for (i = 0; i < ARRAY_SIZE(isp4_irq); i++) {
>> irq[i] = platform_get_irq(pdev, isp4_irq[i].rb_int_num);
>> if (irq[i] < 0)
>> @@ -57,7 +136,7 @@ static int isp4_capture_probe(struct platform_device *pdev)
>> isp4_irq[i].rb_int_num);
>>
>> ret = devm_request_irq(dev, irq[i], isp4_irq_handler,
>> - IRQF_NO_AUTOEN, isp4_irq[i].name, dev);
>> + IRQF_NO_AUTOEN, isp4_irq[i].name, isp_subdev);
>> if (ret)
>> return dev_err_probe(dev, ret, "fail to req irq %d\n", irq[i]);
>> }
>> @@ -83,6 +162,13 @@ static int isp4_capture_probe(struct platform_device *pdev)
>>
>> pm_runtime_set_suspended(dev);
>> pm_runtime_enable(dev);
>> + spin_lock_init(&isp_subdev->irq_lock);
>> + ret = isp4sd_init(&isp_dev->isp_subdev, &isp_dev->v4l2_dev, irq);
>> + if (ret) {
>> + dev_err_probe(dev, ret, "fail init isp4 sub dev\n");
>> + goto err_pm_disable;
>> + }
>> +
>> ret = media_device_register(&isp_dev->mdev);
>> if (ret) {
>> dev_err_probe(dev, ret, "fail to register media device\n");
>> @@ -94,6 +180,8 @@ static int isp4_capture_probe(struct platform_device *pdev)
>> return 0;
>>
>> err_isp4_deinit:
>> + isp4sd_deinit(&isp_dev->isp_subdev);
>> +err_pm_disable:
>> pm_runtime_disable(dev);
>> v4l2_device_unregister(&isp_dev->v4l2_dev);
>> err_clean_media:
>> @@ -108,6 +196,7 @@ static void isp4_capture_remove(struct platform_device *pdev)
>> struct device *dev = &pdev->dev;
>>
>> media_device_unregister(&isp_dev->mdev);
>> + isp4sd_deinit(&isp_dev->isp_subdev);
>> pm_runtime_disable(dev);
>> v4l2_device_unregister(&isp_dev->v4l2_dev);
>> media_device_cleanup(&isp_dev->mdev);
>> diff --git a/drivers/media/platform/amd/isp4/isp4.h b/drivers/media/platform/amd/isp4/isp4.h
>> index 7f2db0dfa2d9..2db6683d6d8b 100644
>> --- a/drivers/media/platform/amd/isp4/isp4.h
>> +++ b/drivers/media/platform/amd/isp4/isp4.h
>> @@ -6,12 +6,15 @@
>> #ifndef _ISP4_H_
>> #define _ISP4_H_
>>
>> -#include <media/v4l2-device.h>
>> -#include <media/videobuf2-memops.h>
>> +#include <drm/amd/isp.h>
>> +#include "isp4_subdev.h"
>>
>> struct isp4_device {
>> struct v4l2_device v4l2_dev;
>> + struct isp4_subdev isp_subdev;
>> struct media_device mdev;
>> };
>>
>> +void isp4_intr_enable(struct isp4_subdev *isp_subdev, u32 index, bool enable);
>> +
>> #endif /* _ISP4_H_ */
>> diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.c b/drivers/media/platform/amd/isp4/isp4_subdev.c
>> new file mode 100644
>> index 000000000000..3a25d1fc49ce
>> --- /dev/null
>> +++ b/drivers/media/platform/amd/isp4/isp4_subdev.c
>> @@ -0,0 +1,975 @@
>> +// SPDX-License-Identifier: GPL-2.0+
>> +/*
>> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
>> + */
>> +
>> +#include <linux/pm_domain.h>
>> +#include <linux/units.h>
>> +
>> +#include "isp4_fw_cmd_resp.h"
>> +#include "isp4_interface.h"
>> +#include "isp4.h"
>> +
>> +#define ISP4SD_MIN_BUF_CNT_BEF_START_STREAM 4
>> +
>> +#define ISP4SD_PERFORMANCE_STATE_LOW 0
>> +#define ISP4SD_PERFORMANCE_STATE_HIGH 1
>> +
>> +/* align 32KB */
>> +#define ISP4SD_META_BUF_SIZE ALIGN(sizeof(struct isp4fw_meta_info), 0x8000)
>> +
>> +#define to_isp4_subdev(v4l2_sdev) \
>
> Virtually always variables referring to a sub-device are called either sd
> or subdev (or variants of these). Up to you.
>
Sure, will change it to sd to meet the convension.
>> + container_of(v4l2_sdev, struct isp4_subdev, sdev)
>> +
>> +static const char *isp4sd_entity_name = "amd isp4";
>> +
>> +static const char *isp4sd_thread_name[ISP4SD_MAX_FW_RESP_STREAM_NUM] = {
>> + "amd_isp4_thread_global",
>> + "amd_isp4_thread_stream1",
>> +};
>> +
>> +static void isp4sd_module_enable(struct isp4_subdev *isp_subdev, bool enable)
>> +{
>> + if (isp_subdev->enable_gpio) {
>> + gpiod_set_value(isp_subdev->enable_gpio, enable ? 1 : 0);
>> + dev_dbg(isp_subdev->dev, "%s isp_subdev module\n",
>> + enable ? "enable" : "disable");
>> + }
>> +}
>> +
>> +static int isp4sd_setup_fw_mem_pool(struct isp4_subdev *isp_subdev)
>> +{
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct isp4fw_cmd_send_buffer buf_type;
>> + struct device *dev = isp_subdev->dev;
>> + int ret;
>> +
>> + if (!ispif->fw_mem_pool) {
>> + dev_err(dev, "fail to alloc mem pool\n");
>> + return -ENOMEM;
>> + }
>> +
>> + /*
>> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
>> + * zeroed, since this is not guaranteed on all compilers.
>> + */
>> + memset(&buf_type, 0, sizeof(buf_type));
>> + buf_type.buffer_type = BUFFER_TYPE_MEM_POOL;
>> + buf_type.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
>> + isp4if_split_addr64(ispif->fw_mem_pool->gpu_mc_addr,
>> + &buf_type.buffer.buf_base_a_lo,
>> + &buf_type.buffer.buf_base_a_hi);
>> + buf_type.buffer.buf_size_a = ispif->fw_mem_pool->mem_size;
>> +
>> + ret = isp4if_send_command(ispif, CMD_ID_SEND_BUFFER,
>> + &buf_type, sizeof(buf_type));
>> + if (ret) {
>> + dev_err(dev, "send fw mem pool 0x%llx(%u) fail %d\n",
>> + ispif->fw_mem_pool->gpu_mc_addr, buf_type.buffer.buf_size_a, ret);
>> + return ret;
>> + }
>> +
>> + dev_dbg(dev, "send fw mem pool 0x%llx(%u) suc\n",
>> + ispif->fw_mem_pool->gpu_mc_addr, buf_type.buffer.buf_size_a);
>> +
>> + return 0;
>> +}
>> +
>> +static int isp4sd_set_stream_path(struct isp4_subdev *isp_subdev)
>> +{
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct isp4fw_cmd_set_stream_cfg cmd;
>> + struct device *dev = isp_subdev->dev;
>> +
>> + /*
>> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
>> + * zeroed, since this is not guaranteed on all compilers.
>> + */
>> + memset(&cmd, 0, sizeof(cmd));
>
> You could assign assign all these in the declaration and avoid zeroing the
> memory explicitly at the same time. I presume possibly leaking some
> information from memory to the firmware in case there are holes in the
> struct isn't an issue.
>
Quoted below is Sultan's reply regarding this, does that make sense?
Could you please make a decision so that I may continue?
"Leaking kernel memory is bad. Also, there is no guarantee that the
firmware will
behave as expected with varying values for the padding bytes.
Please see my arguments from v4 on why these structs should be memset [1]."
[1] https://lore.kernel.org/all/aNTtLHDHf_ozenC-@sultan-box/
>> + cmd.stream_cfg.mipi_pipe_path_cfg.isp4fw_sensor_id = SENSOR_ID_ON_MIPI0;
>> + cmd.stream_cfg.mipi_pipe_path_cfg.b_enable = true;
>> + cmd.stream_cfg.isp_pipe_path_cfg.isp_pipe_id = MIPI0_ISP_PIPELINE_ID;
>> +
>> + cmd.stream_cfg.b_enable_tnr = true;
>> + dev_dbg(dev, "isp4fw_sensor_id %d, pipeId 0x%x EnableTnr %u\n",
>> + cmd.stream_cfg.mipi_pipe_path_cfg.isp4fw_sensor_id,
>> + cmd.stream_cfg.isp_pipe_path_cfg.isp_pipe_id,
>> + cmd.stream_cfg.b_enable_tnr);
>> +
>> + return isp4if_send_command(ispif, CMD_ID_SET_STREAM_CONFIG,
>> + &cmd, sizeof(cmd));
>> +}
>> +
>> +static int isp4sd_send_meta_buf(struct isp4_subdev *isp_subdev)
>> +{
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct isp4fw_cmd_send_buffer buf_type;
>> + struct device *dev = isp_subdev->dev;
>> + int i;
>
> unsigned int, please. You can also declare this within the loop as you do
> elsewhere. Consistency is nice.
>
Ok, will declare it in the loop by unsigned int for the consistency and
check all the other for loop in the patch set.
>> +
>> + /*
>> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
>> + * zeroed, since this is not guaranteed on all compilers.
>> + */
>> + memset(&buf_type, 0, sizeof(buf_type));
>> + for (i = 0; i < ISP4IF_MAX_STREAM_BUF_COUNT; i++) {
>> + struct isp4if_gpu_mem_info *meta_info_buf =
>> + isp_subdev->ispif.meta_info_buf[i];
>> + int ret;
>> +
>> + if (!meta_info_buf) {
>> + dev_err(dev, "fail for no meta info buf(%u)\n", i);
>> + return -ENOMEM;
>> + }
>> +
>> + buf_type.buffer_type = BUFFER_TYPE_META_INFO;
>> + buf_type.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
>> + isp4if_split_addr64(meta_info_buf->gpu_mc_addr,
>> + &buf_type.buffer.buf_base_a_lo,
>> + &buf_type.buffer.buf_base_a_hi);
>> + buf_type.buffer.buf_size_a = meta_info_buf->mem_size;
>> + ret = isp4if_send_command(ispif, CMD_ID_SEND_BUFFER,
>> + &buf_type, sizeof(buf_type));
>> + if (ret) {
>> + dev_err(dev, "send meta info(%u) fail\n", i);
>> + return ret;
>> + }
>> + }
>> +
>> + dev_dbg(dev, "send meta info suc\n");
>> + return 0;
>> +}
>> +
>> +static bool isp4sd_get_str_out_prop(struct isp4_subdev *isp_subdev,
>> + struct isp4fw_image_prop *out_prop,
>> + struct v4l2_subdev_state *state, u32 pad)
>> +{
>> + struct device *dev = isp_subdev->dev;
>> + struct v4l2_mbus_framefmt *format;
>> +
>> + format = v4l2_subdev_state_get_format(state, pad, 0);
>> + if (!format) {
>> + dev_err(dev, "fail get subdev state format\n");
>> + return false;
>> + }
>> +
>> + switch (format->code) {
>> + case MEDIA_BUS_FMT_YUYV8_1_5X8:
>> + out_prop->image_format = IMAGE_FORMAT_NV12;
>> + out_prop->width = format->width;
>> + out_prop->height = format->height;
>> + out_prop->luma_pitch = format->width;
>> + out_prop->chroma_pitch = out_prop->width;
>> + break;
>> + case MEDIA_BUS_FMT_YUYV8_1X16:
>> + out_prop->image_format = IMAGE_FORMAT_YUV422INTERLEAVED;
>> + out_prop->width = format->width;
>> + out_prop->height = format->height;
>> + out_prop->luma_pitch = format->width * 2;
>> + out_prop->chroma_pitch = 0;
>> + break;
>> + default:
>> + dev_err(dev, "fail for bad image format:0x%x\n",
>> + format->code);
>> + return false;
>> + }
>> +
>> + if (!out_prop->width || !out_prop->height)
>> + return false;
>> +
>> + return true;
>> +}
>> +
>> +static int isp4sd_kickoff_stream(struct isp4_subdev *isp_subdev, u32 w, u32 h)
>> +{
>> + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct device *dev = isp_subdev->dev;
>> +
>> + if (sensor_info->status == ISP4SD_START_STATUS_STARTED)
>> + return 0;
>> +
>> + if (sensor_info->status == ISP4SD_START_STATUS_START_FAIL) {
>> + dev_err(dev, "fail for previous start fail\n");
>> + return -EINVAL;
>> + }
>> +
>> + dev_dbg(dev, "w:%u,h:%u\n", w, h);
>> +
>> + if (isp4sd_send_meta_buf(isp_subdev)) {
>> + dev_err(dev, "fail to send meta buf\n");
>> + sensor_info->status = ISP4SD_START_STATUS_START_FAIL;
>> + return -EINVAL;
>> + }
>> +
>> + sensor_info->status = ISP4SD_START_STATUS_OFF;
>> +
>> + if (!sensor_info->start_stream_cmd_sent &&
>> + sensor_info->buf_sent_cnt >= ISP4SD_MIN_BUF_CNT_BEF_START_STREAM) {
>> + int ret = isp4if_send_command(ispif, CMD_ID_START_STREAM,
>> + NULL, 0);
>> + if (ret) {
>> + dev_err(dev, "fail to start stream\n");
>> + return ret;
>> + }
>> +
>> + sensor_info->start_stream_cmd_sent = true;
>> + } else {
>> + dev_dbg(dev,
>> + "no send START_STREAM, start_sent %u, buf_sent %u\n",
>> + sensor_info->start_stream_cmd_sent,
>> + sensor_info->buf_sent_cnt);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int isp4sd_setup_output(struct isp4_subdev *isp_subdev,
>> + struct v4l2_subdev_state *state, u32 pad)
>> +{
>> + struct isp4sd_output_info *output_info = &isp_subdev->sensor_info.output_info;
>> + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct isp4fw_cmd_set_out_ch_prop cmd_ch_prop;
>> + struct isp4fw_cmd_enable_out_ch cmd_ch_en;
>> + struct device *dev = isp_subdev->dev;
>> + int ret;
>> +
>> + if (output_info->start_status == ISP4SD_START_STATUS_STARTED)
>> + return 0;
>> +
>> + if (output_info->start_status == ISP4SD_START_STATUS_START_FAIL) {
>> + dev_err(dev, "fail for previous start fail\n");
>> + return -EINVAL;
>> + }
>> +
>> + /*
>> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
>> + * zeroed, since this is not guaranteed on all compilers.
>> + */
>> + memset(&cmd_ch_prop, 0, sizeof(cmd_ch_prop));
>> + cmd_ch_prop.ch = ISP_PIPE_OUT_CH_PREVIEW;
>> +
>> + if (!isp4sd_get_str_out_prop(isp_subdev, &cmd_ch_prop.image_prop, state, pad)) {
>> + dev_err(dev, "fail to get out prop\n");
>> + return -EINVAL;
>> + }
>> +
>> + dev_dbg(dev, "channel:%d,fmt %d,w:h=%u:%u,lp:%u,cp%u\n",
>> + cmd_ch_prop.ch,
>> + cmd_ch_prop.image_prop.image_format,
>> + cmd_ch_prop.image_prop.width, cmd_ch_prop.image_prop.height,
>> + cmd_ch_prop.image_prop.luma_pitch,
>> + cmd_ch_prop.image_prop.chroma_pitch);
>> +
>> + ret = isp4if_send_command(ispif, CMD_ID_SET_OUT_CHAN_PROP,
>> + &cmd_ch_prop, sizeof(cmd_ch_prop));
>> + if (ret) {
>> + output_info->start_status = ISP4SD_START_STATUS_START_FAIL;
>> + dev_err(dev, "fail to set out prop\n");
>> + return ret;
>> + }
>> +
>> + /*
>> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
>> + * zeroed, since this is not guaranteed on all compilers.
>
> You should have explicit padding fields in any case and not rely on ABI in
> this case.
>
Quoted below is Sultan's reply regarding this, does that make sense? On
the other hand, these definitions are shared between the ISP driver and
firmware and have been verified. I prefer not to add extra padding
fields to the driver as it would affect consistency. Is it acceptable to
leave the definitions as they are?
"It is error-prone for a human to make sure that all padding bytes have
explicit
struct members. And what about future changes to the firmware API where
explicit
padding might be forgotten?
Unless the firmware API structs are all __packed in a future firmware
update, I
think the memsets should remain."
>> + */
>> + memset(&cmd_ch_en, 0, sizeof(cmd_ch_en));
>> + cmd_ch_en.ch = ISP_PIPE_OUT_CH_PREVIEW;
>> + cmd_ch_en.is_enable = true;
>> + ret = isp4if_send_command(ispif, CMD_ID_ENABLE_OUT_CHAN,
>> + &cmd_ch_en, sizeof(cmd_ch_en));
>> + if (ret) {
>> + output_info->start_status = ISP4SD_START_STATUS_START_FAIL;
>> + dev_err(dev, "fail to enable channel\n");
>> + return ret;
>> + }
>> +
>> + dev_dbg(dev, "enable channel %d\n", cmd_ch_en.ch);
>> +
>> + if (!sensor_info->start_stream_cmd_sent) {
>> + ret = isp4sd_kickoff_stream(isp_subdev,
>> + cmd_ch_prop.image_prop.width,
>> + cmd_ch_prop.image_prop.height);
>> + if (ret) {
>> + dev_err(dev, "kickoff stream fail %d\n", ret);
>> + return ret;
>> + }
>> + /*
>> + * sensor_info->start_stream_cmd_sent will be set to true
>> + * 1. in isp4sd_kickoff_stream, if app first send buffer then
>> + * start stream
>> + * 2. in isp_set_stream_buf, if app first start stream, then
>> + * send buffer
>> + * because ISP FW has the requirement, host needs to send buffer
>> + * before send start stream cmd
>> + */
>> + if (sensor_info->start_stream_cmd_sent) {
>> + sensor_info->status = ISP4SD_START_STATUS_STARTED;
>> + output_info->start_status = ISP4SD_START_STATUS_STARTED;
>> + dev_dbg(dev, "kickoff stream suc,start cmd sent\n");
>> + }
>> + } else {
>> + dev_dbg(dev, "stream running, no need kickoff\n");
>> + output_info->start_status = ISP4SD_START_STATUS_STARTED;
>> + }
>> +
>> + dev_dbg(dev, "setup output suc\n");
>> + return 0;
>> +}
>> +
>> +static int isp4sd_init_stream(struct isp4_subdev *isp_subdev)
>> +{
>> + struct device *dev = isp_subdev->dev;
>> + int ret;
>> +
>> + ret = isp4sd_setup_fw_mem_pool(isp_subdev);
>> + if (ret) {
>> + dev_err(dev, "fail to setup fw mem pool\n");
>> + return ret;
>> + }
>> +
>> + ret = isp4sd_set_stream_path(isp_subdev);
>> + if (ret) {
>> + dev_err(dev, "fail to setup stream path\n");
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static void isp4sd_uninit_stream(struct isp4_subdev *isp_subdev,
>> + struct v4l2_subdev_state *state, u32 pad)
>> +{
>> + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
>> + struct isp4sd_output_info *output_info = &sensor_info->output_info;
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct v4l2_mbus_framefmt *format;
>> +
>> + format = v4l2_subdev_state_get_format(state, pad, 0);
>> + if (!format) {
>> + dev_err(isp_subdev->dev, "fail to get v4l2 format\n");
>> + } else {
>> + memset(format, 0, sizeof(*format));
>> + format->code = MEDIA_BUS_FMT_YUYV8_1_5X8;
>> + }
>> +
>> + isp4if_clear_bufq(ispif);
>> + isp4if_clear_cmdq(ispif);
>> +
>> + sensor_info->start_stream_cmd_sent = false;
>> + sensor_info->buf_sent_cnt = 0;
>> +
>> + sensor_info->status = ISP4SD_START_STATUS_OFF;
>> + output_info->start_status = ISP4SD_START_STATUS_OFF;
>> +}
>> +
>> +static void isp4sd_fw_resp_cmd_done(struct isp4_subdev *isp_subdev,
>> + enum isp4if_stream_id stream_id,
>> + struct isp4fw_resp_cmd_done *para)
>> +{
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct isp4if_cmd_element *ele =
>> + isp4if_rm_cmd_from_cmdq(ispif, para->cmd_seq_num, para->cmd_id);
>> + struct device *dev = isp_subdev->dev;
>> +
>> + dev_dbg(dev, "stream %d,cmd (0x%08x)(%d),seq %u, ele %p\n",
>> + stream_id,
>> + para->cmd_id, para->cmd_status, para->cmd_seq_num,
>> + ele);
>> +
>> + if (ele) {
>> + complete(&ele->cmd_done);
>> + if (atomic_dec_and_test(&ele->refcnt))
>> + kfree(ele);
>> + }
>> +}
>> +
>> +static struct isp4fw_meta_info *isp4sd_get_meta_by_mc(struct isp4_subdev *isp_subdev,
>> + u64 mc)
>> +{
>> + for (int i = 0; i < ISP4IF_MAX_STREAM_BUF_COUNT; i++) {
>
> unsigned int, please. Similarly for the rest.
>
Yes, will do that as mentioned above.
>> + struct isp4if_gpu_mem_info *meta_info_buf =
>> + isp_subdev->ispif.meta_info_buf[i];
>> +
>> + if (meta_info_buf->gpu_mc_addr == mc)
>> + return meta_info_buf->sys_addr;
>> + }
>> +
>> + return NULL;
>> +}
>> +
>> +static void isp4sd_send_meta_info(struct isp4_subdev *isp_subdev,
>> + u64 meta_info_mc)
>> +{
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct isp4fw_cmd_send_buffer buf_type;
>> + struct device *dev = isp_subdev->dev;
>> +
>> + if (isp_subdev->sensor_info.status != ISP4SD_START_STATUS_STARTED) {
>> + dev_warn(dev, "not working status %i, meta_info 0x%llx\n",
>> + isp_subdev->sensor_info.status, meta_info_mc);
>> + return;
>> + }
>> +
>> + /*
>> + * The struct will be shared with ISP FW, use memset() to guarantee padding bits are
>> + * zeroed, since this is not guaranteed on all compilers.
>> + */
>> + memset(&buf_type, 0, sizeof(buf_type));
>> + buf_type.buffer_type = BUFFER_TYPE_META_INFO;
>> + buf_type.buffer.vmid_space.bit.space = ADDR_SPACE_TYPE_GPU_VA;
>> + isp4if_split_addr64(meta_info_mc,
>> + &buf_type.buffer.buf_base_a_lo,
>> + &buf_type.buffer.buf_base_a_hi);
>> + buf_type.buffer.buf_size_a = ISP4SD_META_BUF_SIZE;
>> +
>> + if (isp4if_send_command(ispif, CMD_ID_SEND_BUFFER,
>> + &buf_type, sizeof(buf_type)))
>> + dev_err(dev, "fail send meta_info 0x%llx\n",
>> + meta_info_mc);
>> + else
>> + dev_dbg(dev, "resend meta_info 0x%llx\n", meta_info_mc);
>> +}
>> +
>> +static void isp4sd_fw_resp_frame_done(struct isp4_subdev *isp_subdev,
>> + enum isp4if_stream_id stream_id,
>> + struct isp4fw_resp_param_package *para)
>> +{
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct device *dev = isp_subdev->dev;
>> + struct isp4if_img_buf_node *prev;
>> + struct isp4fw_meta_info *meta;
>> + u64 mc;
>> +
>> + mc = isp4if_join_addr64(para->package_addr_lo, para->package_addr_hi);
>> + meta = isp4sd_get_meta_by_mc(isp_subdev, mc);
>> + if (!meta) {
>> + dev_err(dev, "fail to get meta from mc %llx\n", mc);
>> + return;
>> + }
>> +
>> + dev_dbg(dev, "ts:%llu,streamId:%d,poc:%u,preview_en:%u(%i)\n",
>> + ktime_get_ns(), stream_id, meta->poc,
>> + meta->preview.enabled,
>> + meta->preview.status);
>> +
>> + if (meta->preview.enabled &&
>> + (meta->preview.status == BUFFER_STATUS_SKIPPED ||
>> + meta->preview.status == BUFFER_STATUS_DONE ||
>> + meta->preview.status == BUFFER_STATUS_DIRTY)) {
>> + prev = isp4if_dequeue_buffer(ispif);
>> + if (prev)
>> + isp4if_dealloc_buffer_node(prev);
>> + else
>> + dev_err(dev, "fail null prev buf\n");
>> +
>> + } else if (meta->preview.enabled) {
>> + dev_err(dev, "fail bad preview status %u\n",
>> + meta->preview.status);
>> + }
>> +
>> + if (isp_subdev->sensor_info.status == ISP4SD_START_STATUS_STARTED)
>> + isp4sd_send_meta_info(isp_subdev, mc);
>> +
>> + dev_dbg(dev, "stream_id:%d, status:%d\n", stream_id,
>> + isp_subdev->sensor_info.status);
>> +}
>> +
>> +static void isp4sd_fw_resp_func(struct isp4_subdev *isp_subdev,
>> + enum isp4if_stream_id stream_id)
>> +{
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct device *dev = isp_subdev->dev;
>> + struct isp4fw_resp resp;
>> +
>> + while (true) {
>> + if (isp4if_f2h_resp(ispif, stream_id, &resp)) {
>> + /* Re-enable the interrupt */
>> + isp4_intr_enable(isp_subdev, stream_id, true);
>> + /*
>> + * Recheck to see if there is a new response.
>> + * To ensure that an in-flight interrupt is not lost,
>> + * enabling the interrupt must occur _before_ checking
>> + * for a new response, hence a memory barrier is needed.
>> + * Disable the interrupt again if there was a new response.
>> + */
>> + mb();
>> + if (likely(isp4if_f2h_resp(ispif, stream_id, &resp)))
>> + break;
>> +
>> + isp4_intr_enable(isp_subdev, stream_id, false);
>> + }
>> +
>> + switch (resp.resp_id) {
>> + case RESP_ID_CMD_DONE:
>> + isp4sd_fw_resp_cmd_done(isp_subdev, stream_id,
>> + &resp.param.cmd_done);
>> + break;
>> + case RESP_ID_NOTI_FRAME_DONE:
>> + isp4sd_fw_resp_frame_done(isp_subdev, stream_id,
>> + &resp.param.frame_done);
>> + break;
>> + default:
>> + dev_err(dev, "-><- fail respid (0x%x)\n", resp.resp_id);
>> + break;
>> + }
>> + }
>> +}
>> +
>> +static s32 isp4sd_fw_resp_thread(void *context)
>> +{
>> + struct isp4_subdev_thread_param *para = context;
>> + struct isp4_subdev *isp_subdev = para->isp_subdev;
>> + struct isp4sd_thread_handler *thread_ctx =
>> + &isp_subdev->fw_resp_thread[para->idx];
>> + struct device *dev = isp_subdev->dev;
>> +
>> + dev_dbg(dev, "[%u] fw resp thread started\n", para->idx);
>> + while (true) {
>> + wait_event_interruptible(thread_ctx->waitq, thread_ctx->resp_ready);
>> + thread_ctx->resp_ready = false;
>> +
>> + if (kthread_should_stop()) {
>> + dev_dbg(dev, "[%u] fw resp thread quit\n", para->idx);
>> + break;
>> + }
>> +
>> + isp4sd_fw_resp_func(isp_subdev, para->idx);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int isp4sd_stop_resp_proc_threads(struct isp4_subdev *isp_subdev)
>> +{
>> + int i;
>> +
>> + for (i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++) {
>> + struct isp4sd_thread_handler *thread_ctx =
>> + &isp_subdev->fw_resp_thread[i];
>> +
>> + if (thread_ctx->thread) {
>> + kthread_stop(thread_ctx->thread);
>> + thread_ctx->thread = NULL;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int isp4sd_start_resp_proc_threads(struct isp4_subdev *isp_subdev)
>> +{
>> + struct device *dev = isp_subdev->dev;
>> + int i;
>> +
>> + for (i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++) {
>> + struct isp4sd_thread_handler *thread_ctx = &isp_subdev->fw_resp_thread[i];
>> +
>> + isp_subdev->isp_resp_para[i].idx = i;
>> + isp_subdev->isp_resp_para[i].isp_subdev = isp_subdev;
>> + init_waitqueue_head(&thread_ctx->waitq);
>> + thread_ctx->resp_ready = false;
>> +
>> + thread_ctx->thread = kthread_run(isp4sd_fw_resp_thread,
>> + &isp_subdev->isp_resp_para[i],
>> + isp4sd_thread_name[i]);
>> + if (IS_ERR(thread_ctx->thread)) {
>> + dev_err(dev, "create thread [%d] fail\n", i);
>> + thread_ctx->thread = NULL;
>> + isp4sd_stop_resp_proc_threads(isp_subdev);
>> + return -EINVAL;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int isp4sd_pwroff_and_deinit(struct isp4_subdev *isp_subdev)
>> +{
>> + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
>> + unsigned int perf_state = ISP4SD_PERFORMANCE_STATE_LOW;
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct device *dev = isp_subdev->dev;
>> + int ret;
>> +
>> + if (sensor_info->status == ISP4SD_START_STATUS_STARTED) {
>> + dev_err(dev, "fail for stream still running\n");
>> + return -EINVAL;
>> + }
>> +
>> + sensor_info->status = ISP4SD_START_STATUS_OFF;
>> +
>> + if (isp_subdev->irq_enabled) {
>> + for (int i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++)
>> + disable_irq(isp_subdev->irq[i]);
>> + isp_subdev->irq_enabled = false;
>> + }
>> +
>> + isp4sd_stop_resp_proc_threads(isp_subdev);
>> + dev_dbg(dev, "isp_subdev stop resp proc streads suc");
>> +
>> + isp4if_stop(ispif);
>> +
>> + ret = dev_pm_genpd_set_performance_state(dev, perf_state);
>> + if (ret)
>> + dev_err(dev, "fail to set isp_subdev performance state %u,ret %d\n",
>> + perf_state, ret);
>> +
>> + /* hold ccpu reset */
>> + isp4hw_wreg(isp_subdev->mmio, ISP_SOFT_RESET, 0);
>> + isp4hw_wreg(isp_subdev->mmio, ISP_POWER_STATUS, 0);
>> + ret = pm_runtime_put_sync(dev);
>> + if (ret)
>> + dev_err(dev, "power off isp_subdev fail %d\n", ret);
>> + else
>> + dev_dbg(dev, "power off isp_subdev suc\n");
>> +
>> + ispif->status = ISP4IF_STATUS_PWR_OFF;
>> + isp4if_clear_cmdq(ispif);
>> + isp4sd_module_enable(isp_subdev, false);
>> +
>> + /*
>> + * When opening the camera, isp4sd_module_enable(isp_subdev, true) is called.
>> + * Hardware requires at least a 20ms delay between disabling and enabling the module,
>> + * so a sleep is added to ensure ISP stability during quick reopen scenarios.
>> + */
>> + msleep(20);
>> +
>> + return 0;
>> +}
>> +
>> +static int isp4sd_pwron_and_init(struct isp4_subdev *isp_subdev)
>> +{
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct device *dev = isp_subdev->dev;
>> + int ret;
>> +
>> + if (ispif->status == ISP4IF_STATUS_FW_RUNNING) {
>> + dev_dbg(dev, "camera already opened, do nothing\n");
>> + return 0;
>> + }
>> +
>> + isp4sd_module_enable(isp_subdev, true);
>> +
>> + if (ispif->status < ISP4IF_STATUS_PWR_ON) {
>> + unsigned int perf_state = ISP4SD_PERFORMANCE_STATE_HIGH;
>> +
>> + ret = pm_runtime_resume_and_get(dev);
>> + if (ret) {
>> + dev_err(dev, "fail to power on isp_subdev ret %d\n",
>> + ret);
>> + goto err_deinit;
>> + }
>> +
>> + /* ISPPG ISP Power Status */
>> + isp4hw_wreg(isp_subdev->mmio, ISP_POWER_STATUS, 0x7FF);
>> + ret = dev_pm_genpd_set_performance_state(dev, perf_state);
>> + if (ret) {
>> + dev_err(dev,
>> + "fail to set performance state %u, ret %d\n",
>> + perf_state, ret);
>> + goto err_deinit;
>> + }
>> +
>> + ispif->status = ISP4IF_STATUS_PWR_ON;
>> + }
>> +
>> + isp_subdev->sensor_info.start_stream_cmd_sent = false;
>> + isp_subdev->sensor_info.buf_sent_cnt = 0;
>> +
>> + ret = isp4if_start(ispif);
>> + if (ret) {
>> + dev_err(dev, "fail to start isp_subdev interface\n");
>> + goto err_deinit;
>> + }
>> +
>> + if (isp4sd_start_resp_proc_threads(isp_subdev)) {
>> + dev_err(dev, "isp_start_resp_proc_threads fail");
>> + goto err_deinit;
>> + }
>> +
>> + dev_dbg(dev, "create resp threads ok");
>> +
>> + for (int i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++)
>> + enable_irq(isp_subdev->irq[i]);
>> + isp_subdev->irq_enabled = true;
>> +
>> + return 0;
>> +err_deinit:
>> + isp4sd_pwroff_and_deinit(isp_subdev);
>> + return -EINVAL;
>> +}
>> +
>> +static int isp4sd_stop_stream(struct isp4_subdev *isp_subdev,
>> + struct v4l2_subdev_state *state, u32 pad)
>> +{
>> + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
>> + struct isp4sd_output_info *output_info = &sensor_info->output_info;
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct device *dev = isp_subdev->dev;
>> +
>> + guard(mutex)(&isp_subdev->ops_mutex);
>> + dev_dbg(dev, "status %i\n", output_info->start_status);
>> +
>> + if (output_info->start_status == ISP4SD_START_STATUS_STARTED) {
>> + struct isp4fw_cmd_enable_out_ch cmd_ch_disable;
>> + int ret;
>> +
>> + /*
>> + * The struct will be shared with ISP FW, use memset() to guarantee
>> + * padding bits are zeroed, since this is not guaranteed on all compilers.
>> + */
>> + memset(&cmd_ch_disable, 0, sizeof(cmd_ch_disable));
>> + cmd_ch_disable.ch = ISP_PIPE_OUT_CH_PREVIEW;
>> + /* `cmd_ch_disable.is_enable` is already false */
>> + ret = isp4if_send_command_sync(ispif, CMD_ID_ENABLE_OUT_CHAN,
>> + &cmd_ch_disable,
>> + sizeof(cmd_ch_disable));
>> + if (ret)
>> + dev_err(dev, "fail to disable stream\n");
>> + else
>> + dev_dbg(dev, "wait disable stream suc\n");
>> +
>> + ret = isp4if_send_command_sync(ispif, CMD_ID_STOP_STREAM,
>> + NULL, 0);
>> + if (ret)
>> + dev_err(dev, "fail to stop steam\n");
>> + else
>> + dev_dbg(dev, "wait stop stream suc\n");
>> + }
>> +
>> + isp4sd_uninit_stream(isp_subdev, state, pad);
>> +
>> + /*
>> + * Return success to ensure the stop process proceeds,
>> + * and disregard any errors since they are not fatal.
>> + */
>> + return 0;
>> +}
>> +
>> +static int isp4sd_start_stream(struct isp4_subdev *isp_subdev,
>> + struct v4l2_subdev_state *state, u32 pad)
>> +{
>> + struct isp4sd_output_info *output_info =
>> + &isp_subdev->sensor_info.output_info;
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct device *dev = isp_subdev->dev;
>> + int ret;
>> +
>> + guard(mutex)(&isp_subdev->ops_mutex);
>> +
>> + if (ispif->status != ISP4IF_STATUS_FW_RUNNING) {
>> + dev_err(dev, "fail, bad fsm %d", ispif->status);
>> + return -EINVAL;
>> + }
>> +
>> + switch (output_info->start_status) {
>> + case ISP4SD_START_STATUS_OFF:
>> + break;
>> + case ISP4SD_START_STATUS_STARTED:
>> + dev_dbg(dev, "stream already started, do nothing\n");
>> + return 0;
>> + case ISP4SD_START_STATUS_START_FAIL:
>> + dev_err(dev, "stream previously failed to start\n");
>> + return -EINVAL;
>> + }
>> +
>> + ret = isp4sd_init_stream(isp_subdev);
>> + if (ret) {
>> + dev_err(dev, "fail to init isp_subdev stream\n");
>> + goto err_stop_stream;
>> + }
>> +
>> + ret = isp4sd_setup_output(isp_subdev, state, pad);
>> + if (ret) {
>> + dev_err(dev, "fail to setup output\n");
>> + goto err_stop_stream;
>> + }
>> +
>> + return 0;
>> +
>> +err_stop_stream:
>> + isp4sd_stop_stream(isp_subdev, state, pad);
>> + return ret;
>> +}
>> +
>> +static int isp4sd_set_power(struct v4l2_subdev *sd, int on)
>> +{
>> + struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
>> +
>> + guard(mutex)(&isp_subdev->ops_mutex);
>> + if (on)
>> + return isp4sd_pwron_and_init(isp_subdev);
>> + else
>> + return isp4sd_pwroff_and_deinit(isp_subdev);
>
> The s_power() callback is deprecated, please rely on runtime PM.
>
Thanks, I'll switch from deprecated s_power() to runtime PM callbacks.
>> +}
>> +
>> +static const struct v4l2_subdev_core_ops isp4sd_core_ops = {
>> + .s_power = isp4sd_set_power,
>> +};
>> +
>> +static const struct v4l2_subdev_video_ops isp4sd_video_ops = {
>> + .s_stream = v4l2_subdev_s_stream_helper,
>> +};
>> +
>> +static int isp4sd_set_pad_format(struct v4l2_subdev *sd,
>> + struct v4l2_subdev_state *sd_state,
>> + struct v4l2_subdev_format *fmt)
>> +{
>> + struct isp4sd_output_info *steam_info =
>
> Steam?
>
yes, typo here, should be stream.
>> + &(to_isp4_subdev(sd)->sensor_info.output_info);
>> + struct v4l2_mbus_framefmt *format;
>> +
>> + format = v4l2_subdev_state_get_format(sd_state, fmt->pad);
>> +
>> + if (!format) {
>> + dev_err(sd->dev, "fail to get state format\n");
>> + return -EINVAL;
>> + }
>> +
>> + *format = fmt->format;
>> + switch (format->code) {
>> + case MEDIA_BUS_FMT_YUYV8_1_5X8:
>> + steam_info->image_size = format->width * format->height * 3 / 2;
>> + break;
>> + case MEDIA_BUS_FMT_YUYV8_1X16:
>> + steam_info->image_size = format->width * format->height * 2;
>> + break;
>> + default:
>> + steam_info->image_size = 0;
>
> Please stick to formats the driver supports i.e. make either of the above
> the default.
>
Sure, will make MEDIA_BUS_FMT_YUYV8_1_5X8 the default.
>> + break;
>> + }
>> + if (!steam_info->image_size) {
>> + dev_err(sd->dev, "fail set pad format,code 0x%x,width %u, height %u\n",
>> + format->code, format->width, format->height);
>> + return -EINVAL;
>> + }
>> +
>> + dev_dbg(sd->dev, "set pad format suc, code:%x w:%u h:%u size:%u\n",
>> + format->code, format->width, format->height, steam_info->image_size);
>> +
>> + return 0;
>> +}
>> +
>> +static int isp4sd_enable_streams(struct v4l2_subdev *sd,
>> + struct v4l2_subdev_state *state, u32 pad,
>> + u64 streams_mask)
>> +{
>> + struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
>> +
>> + return isp4sd_start_stream(isp_subdev, state, pad);
>> +}
>> +
>> +static int isp4sd_disable_streams(struct v4l2_subdev *sd,
>> + struct v4l2_subdev_state *state, u32 pad,
>> + u64 streams_mask)
>> +{
>> + struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
>> +
>> + return isp4sd_stop_stream(isp_subdev, state, pad);
>> +}
>> +
>> +static const struct v4l2_subdev_pad_ops isp4sd_pad_ops = {
>> + .get_fmt = v4l2_subdev_get_fmt,
>> + .set_fmt = isp4sd_set_pad_format,
>> + .enable_streams = isp4sd_enable_streams,
>> + .disable_streams = isp4sd_disable_streams,
>> +};
>> +
>> +static const struct v4l2_subdev_ops isp4sd_subdev_ops = {
>> + .core = &isp4sd_core_ops,
>> + .video = &isp4sd_video_ops,
>> + .pad = &isp4sd_pad_ops,
>> +};
>> +
>> +static int isp4sd_sdev_link_validate(struct media_link *link)
>> +{
>> + return 0;
>
> Uh-oh.
>
> What is actually being configured via the sub-device? There is no device
> node either, is there? Are there plans for future developments, apart from
> possibly making the ISP and the sensor controllable by the host?
>
Yes, you are correct. For the first version, no device node and
configuration for the sub-device now. Possible future development plan
is under internal discussion.
>> +}
>> +
>> +static const struct media_entity_operations isp4sd_sdev_ent_ops = {
>> + .link_validate = isp4sd_sdev_link_validate,
>> +};
>> +
>> +int isp4sd_init(struct isp4_subdev *isp_subdev, struct v4l2_device *v4l2_dev,
>> + int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM])
>> +{
>> + struct isp4sd_sensor_info *sensor_info = &isp_subdev->sensor_info;
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct device *dev = v4l2_dev->dev;
>> + int ret;
>> +
>> + isp_subdev->dev = dev;
>> + v4l2_subdev_init(&isp_subdev->sdev, &isp4sd_subdev_ops);
>> + isp_subdev->sdev.owner = THIS_MODULE;
>> + isp_subdev->sdev.dev = dev;
>> + snprintf(isp_subdev->sdev.name, sizeof(isp_subdev->sdev.name), "%s",
>> + dev_name(dev));
>> +
>> + isp_subdev->sdev.entity.name = isp4sd_entity_name;
>> + isp_subdev->sdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
>> + isp_subdev->sdev.entity.ops = &isp4sd_sdev_ent_ops;
>> + isp_subdev->sdev_pad.flags = MEDIA_PAD_FL_SOURCE;
>> + ret = media_entity_pads_init(&isp_subdev->sdev.entity, 1,
>> + &isp_subdev->sdev_pad);
>> + if (ret) {
>> + dev_err(dev, "fail to init isp4 subdev entity pad %d\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = v4l2_subdev_init_finalize(&isp_subdev->sdev);
>> + if (ret < 0) {
>> + dev_err(dev, "fail to init finalize isp4 subdev %d\n",
>> + ret);
>> + return ret;
>> + }
>> +
>> + ret = v4l2_device_register_subdev(v4l2_dev, &isp_subdev->sdev);
>> + if (ret) {
>> + dev_err(dev, "fail to register isp4 subdev to V4L2 device %d\n", ret);
>> + goto err_media_clean_up;
>> + }
>> +
>> + isp4if_init(ispif, dev, isp_subdev->mmio);
>> +
>> + mutex_init(&isp_subdev->ops_mutex);
>> + sensor_info->status = ISP4SD_START_STATUS_OFF;
>> +
>> + /* create ISP enable gpio control */
>> + isp_subdev->enable_gpio = devm_gpiod_get(isp_subdev->dev,
>> + "enable_isp",
>> + GPIOD_OUT_LOW);
>> + if (IS_ERR(isp_subdev->enable_gpio)) {
>> + ret = PTR_ERR(isp_subdev->enable_gpio);
>> + dev_err(dev, "fail to get gpiod %d\n", ret);
>> + goto err_subdev_unreg;
>> + }
>> +
>> + for (int i = 0; i < ISP4SD_MAX_FW_RESP_STREAM_NUM; i++)
>> + isp_subdev->irq[i] = irq[i];
>> +
>> + isp_subdev->host2fw_seq_num = 1;
>> + ispif->status = ISP4IF_STATUS_PWR_OFF;
>> +
>> + return 0;
>> +
>> +err_subdev_unreg:
>> + v4l2_device_unregister_subdev(&isp_subdev->sdev);
>> +err_media_clean_up:
>> + v4l2_subdev_cleanup(&isp_subdev->sdev);
>> + media_entity_cleanup(&isp_subdev->sdev.entity);
>> + return ret;
>> +}
>> +
>> +void isp4sd_deinit(struct isp4_subdev *isp_subdev)
>> +{
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> +
>> + v4l2_device_unregister_subdev(&isp_subdev->sdev);
>> + media_entity_cleanup(&isp_subdev->sdev.entity);
>> + isp4if_deinit(ispif);
>> + isp4sd_module_enable(isp_subdev, false);
>> +
>> + ispif->status = ISP4IF_STATUS_PWR_OFF;
>> +}
>> diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.h b/drivers/media/platform/amd/isp4/isp4_subdev.h
>> new file mode 100644
>> index 000000000000..70007b9de3f3
>> --- /dev/null
>> +++ b/drivers/media/platform/amd/isp4/isp4_subdev.h
>> @@ -0,0 +1,124 @@
>> +/* SPDX-License-Identifier: GPL-2.0+ */
>> +/*
>> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
>> + */
>> +
>> +#ifndef _ISP4_SUBDEV_H_
>> +#define _ISP4_SUBDEV_H_
>> +
>> +#include <linux/debugfs.h>
>> +#include <linux/delay.h>
>> +#include <linux/firmware.h>
>> +#include <linux/pm_runtime.h>
>> +#include <linux/types.h>
>> +#include <linux/uaccess.h>
>> +#include <media/v4l2-device.h>
>> +
>> +#include "isp4_fw_cmd_resp.h"
>> +#include "isp4_hw_reg.h"
>> +#include "isp4_interface.h"
>> +
>> +/*
>> + * one is for none sesnor specefic response which is not used now
>> + * another is for sensor specific response
>> + */
>> +#define ISP4SD_MAX_FW_RESP_STREAM_NUM 2
>> +
>> +/*
>> + * cmd used to register frame done callback, parameter is
>> + * struct isp4sd_register_framedone_cb_param *
>> + * when a image buffer is filled by ISP, ISP will call the registered callback.
>> + * callback func prototype is isp4sd_framedone_cb, cb_ctx can be anything
>> + * provided by caller which will be provided back as the first parameter of the
>> + * callback function.
>> + * both cb_func and cb_ctx are provide by caller, set cb_func to NULL to
>> + * unregister the callback
>
> These comments would benefit from grammatical and spelling corrections.
> Same below.
>
Thanks for noting this; I'll check all comments in the set.
>> + */
>> +
>> +/* used to indicate the ISP status */
>> +enum isp4sd_status {
>> + ISP4SD_STATUS_PWR_OFF,
>> + ISP4SD_STATUS_PWR_ON,
>> + ISP4SD_STATUS_FW_RUNNING,
>> + ISP4SD_STATUS_MAX
>> +};
>> +
>> +/* used to indicate the status of sensor, output stream */
>> +enum isp4sd_start_status {
>> + ISP4SD_START_STATUS_OFF,
>> + ISP4SD_START_STATUS_STARTED,
>> + ISP4SD_START_STATUS_START_FAIL,
>> +};
>> +
>> +struct isp4sd_img_buf_node {
>> + struct list_head node;
>> + struct isp4if_img_buf_info buf_info;
>> +};
>> +
>> +/* this is isp output after processing bayer raw input from sensor */
>> +struct isp4sd_output_info {
>> + enum isp4sd_start_status start_status;
>> + u32 image_size;
>> +};
>> +
>> +/*
>> + * This struct represents the sensor info which is input or source of ISP,
>> + * status is the sensor status
>> + * output_info is the isp output info after ISP processing the sensor input,
>> + * start_stream_cmd_sent mean if CMD_ID_START_STREAM has sent to fw.
>> + * buf_sent_cnt is buffer count app has sent to receive the images
>> + */
>> +struct isp4sd_sensor_info {
>> + struct isp4sd_output_info output_info;
>> + enum isp4sd_start_status status;
>> + bool start_stream_cmd_sent;
>> + u32 buf_sent_cnt;
>> +};
>> +
>> +/*
>> + * Thread created by driver to receive fw response
>> + * thread will be wakeup by fw to driver response interrupt
>> + */
>> +struct isp4sd_thread_handler {
>> + struct task_struct *thread;
>> + wait_queue_head_t waitq;
>> + bool resp_ready;
>> +};
>> +
>> +struct isp4_subdev_thread_param {
>> + u32 idx;
>> + struct isp4_subdev *isp_subdev;
>> +};
>> +
>> +struct isp4_subdev {
>> + struct v4l2_subdev sdev;
>> + struct isp4_interface ispif;
>> +
>> + struct media_pad sdev_pad;
>> +
>> + enum isp4sd_status isp_status;
>> + struct mutex ops_mutex; /* ops_mutex */
>> +
>> + /* Used to store fw cmds sent to FW whose response driver needs to wait for */
>> + struct isp4sd_thread_handler fw_resp_thread[ISP4SD_MAX_FW_RESP_STREAM_NUM];
>> +
>> + u32 host2fw_seq_num;
>> +
>> + struct isp4sd_sensor_info sensor_info;
>> +
>> + /* gpio descriptor */
>> + struct gpio_desc *enable_gpio;
>> + struct device *dev;
>> + void __iomem *mmio;
>> + struct isp4_subdev_thread_param isp_resp_para[ISP4SD_MAX_FW_RESP_STREAM_NUM];
>> + int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM];
>> + bool irq_enabled;
>> + /* spin lock to access ISP_SYS_INT0_EN exclusively */
>> + spinlock_t irq_lock;
>> +};
>> +
>> +int isp4sd_init(struct isp4_subdev *isp_subdev, struct v4l2_device *v4l2_dev,
>> + int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM]);
>> +void isp4sd_deinit(struct isp4_subdev *isp_subdev);
>> +
>> +#endif /* _ISP4_SUBDEV_H_ */
>
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added
2026-01-14 10:34 ` Du, Bin
@ 2026-01-14 21:07 ` Sakari Ailus
2026-01-15 7:51 ` Du, Bin
0 siblings, 1 reply; 46+ messages in thread
From: Sakari Ailus @ 2026-01-14 21:07 UTC (permalink / raw)
To: Du, Bin
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Alexey Zagorodnikov
Hi Bin,
On Wed, Jan 14, 2026 at 06:34:34PM +0800, Du, Bin wrote:
> > The increment could probably be expressed as seen &= ~BIT(i).
> >
>
> Quoted below is Sultan's reply regarding this, Would it be acceptable to
> leave it unchanged?
>
> "Yes it can be, but it adds several more instructions before the loop body,
> without any improvement to the loop body (the sarx in the loop body is
> replaced
> by andn). The right shift trick is faster and this is a hot path (IRQ
> handler).
Please see my reply to Sultan.
> > > +static int isp4sd_sdev_link_validate(struct media_link *link)
> > > +{
> > > + return 0;
> >
> > Uh-oh.
> >
> > What is actually being configured via the sub-device? There is no device
> > node either, is there? Are there plans for future developments, apart from
> > possibly making the ISP and the sensor controllable by the host?
> >
>
> Yes, you are correct. For the first version, no device node and
> configuration for the sub-device now. Possible future development plan is
> under internal discussion.
You can drop these for now.
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added
2026-01-14 21:07 ` Sakari Ailus
@ 2026-01-15 7:51 ` Du, Bin
0 siblings, 0 replies; 46+ messages in thread
From: Du, Bin @ 2026-01-15 7:51 UTC (permalink / raw)
To: Sakari Ailus
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Alexey Zagorodnikov
On 1/15/2026 5:07 AM, Sakari Ailus wrote:
> Hi Bin,
>
> On Wed, Jan 14, 2026 at 06:34:34PM +0800, Du, Bin wrote:
>>> The increment could probably be expressed as seen &= ~BIT(i).
>>>
>>
>> Quoted below is Sultan's reply regarding this, Would it be acceptable to
>> leave it unchanged?
>>
>> "Yes it can be, but it adds several more instructions before the loop body,
>> without any improvement to the loop body (the sarx in the loop body is
>> replaced
>> by andn). The right shift trick is faster and this is a hot path (IRQ
>> handler).
>
> Please see my reply to Sultan.
>
Yes, I appreciate the instructive ideas from you and Sultan.
>>>> +static int isp4sd_sdev_link_validate(struct media_link *link)
>>>> +{
>>>> + return 0;
>>>
>>> Uh-oh.
>>>
>>> What is actually being configured via the sub-device? There is no device
>>> node either, is there? Are there plans for future developments, apart from
>>> possibly making the ISP and the sensor controllable by the host?
>>>
>>
>> Yes, you are correct. For the first version, no device node and
>> configuration for the sub-device now. Possible future development plan is
>> under internal discussion.
>
> You can drop these for now.
>
Sure, will remove these now and reintroduce them when they're needed in
the future.
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added
2025-12-16 9:13 ` [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added Bin Du
2025-12-22 10:11 ` Sakari Ailus
@ 2026-01-14 9:45 ` Markus Elfring
1 sibling, 0 replies; 46+ messages in thread
From: Markus Elfring @ 2026-01-14 9:45 UTC (permalink / raw)
To: Bin Du, Sultan Alsawaf, Svetoslav Stoilov, linux-media,
Bryan O'Donoghue, Hans Verkuil, Lad Prabhakar,
Laurent Pinchart, Mauro Carvalho Chehab, Sakari Ailus
Cc: LKML, kernel-janitors, Alexey Zagorodnikov, Anson Tsao,
Benjamin Chan, Dominic Antony, Gjorgji Rosikopulos, King Li,
Mario Limonciello, Phil Jawich, Pratap Nirujogi, Richard Gong
…
> +++ b/drivers/media/platform/amd/isp4/isp4.c
…
> +void isp4_intr_enable(struct isp4_subdev *isp_subdev, u32 index, bool enable)
> +{
…
> + spin_lock_irq(&isp_subdev->irq_lock);
> + intr_en = isp4hw_rreg(isp_subdev->mmio, ISP_SYS_INT0_EN);
…
> + isp4hw_wreg(isp_subdev->mmio, ISP_SYS_INT0_EN, intr_en);
> + spin_unlock_irq(&isp_subdev->irq_lock);
> +}
…
Under which circumstances would you become interested to apply a statement
like “guard(spinlock_irq)(&isp_subdev->irq_lock);”?
https://elixir.bootlin.com/linux/v6.19-rc5/source/include/linux/spinlock.h#L571-L573
Regards,
Markus
^ permalink raw reply [flat|nested] 46+ messages in thread
* [PATCH v7 5/7] media: platform: amd: isp4 video node and buffers handling added
2025-12-16 9:13 [PATCH v7 0/7] Add AMD ISP4 driver Bin Du
` (3 preceding siblings ...)
2025-12-16 9:13 ` [PATCH v7 4/7] media: platform: amd: isp4 subdev and firmware loading handling added Bin Du
@ 2025-12-16 9:13 ` Bin Du
2025-12-22 17:07 ` Sakari Ailus
2025-12-16 9:13 ` [PATCH v7 6/7] media: platform: amd: isp4 debug fs logging and more descriptive errors Bin Du
` (4 subsequent siblings)
9 siblings, 1 reply; 46+ messages in thread
From: Bin Du @ 2025-12-16 9:13 UTC (permalink / raw)
To: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
sultan
Cc: pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, bin.du, Bin Du, Svetoslav Stoilov,
Alexey Zagorodnikov
Isp video implements v4l2 video interface and supports NV12 and YUYV. It
manages buffers, pipeline power and state. Cherry-picked Sultan's DMA
buffer related fix from branch v6.16-drm-tip-isp4-for-amd on
https://github.com/kerneltoast/kernel_x86_laptop.git
Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
Signed-off-by: Bin Du <Bin.Du@amd.com>
Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
---
MAINTAINERS | 2 +
drivers/media/platform/amd/isp4/Makefile | 3 +-
drivers/media/platform/amd/isp4/isp4.c | 10 +
drivers/media/platform/amd/isp4/isp4_subdev.c | 79 +-
drivers/media/platform/amd/isp4/isp4_subdev.h | 2 +
drivers/media/platform/amd/isp4/isp4_video.c | 1165 +++++++++++++++++
drivers/media/platform/amd/isp4/isp4_video.h | 65 +
7 files changed, 1321 insertions(+), 5 deletions(-)
create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c
create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 48ffc8bbdcee..80c966fde0b4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1151,6 +1151,8 @@ F: drivers/media/platform/amd/isp4/isp4_interface.c
F: drivers/media/platform/amd/isp4/isp4_interface.h
F: drivers/media/platform/amd/isp4/isp4_subdev.c
F: drivers/media/platform/amd/isp4/isp4_subdev.h
+F: drivers/media/platform/amd/isp4/isp4_video.c
+F: drivers/media/platform/amd/isp4/isp4_video.h
AMD KFD
M: Felix Kuehling <Felix.Kuehling@amd.com>
diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
index 6d4e6d6ac7f5..398c20ea7866 100644
--- a/drivers/media/platform/amd/isp4/Makefile
+++ b/drivers/media/platform/amd/isp4/Makefile
@@ -5,4 +5,5 @@
obj-$(CONFIG_AMD_ISP4) += amd_capture.o
amd_capture-objs := isp4.o \
isp4_interface.o \
- isp4_subdev.o
+ isp4_subdev.o \
+ isp4_video.o
diff --git a/drivers/media/platform/amd/isp4/isp4.c b/drivers/media/platform/amd/isp4/isp4.c
index bcd7cad32afd..b5470f6dc93e 100644
--- a/drivers/media/platform/amd/isp4/isp4.c
+++ b/drivers/media/platform/amd/isp4/isp4.c
@@ -169,6 +169,16 @@ static int isp4_capture_probe(struct platform_device *pdev)
goto err_pm_disable;
}
+ ret = media_create_pad_link(&isp_dev->isp_subdev.sdev.entity,
+ 0, &isp_dev->isp_subdev.isp_vdev.vdev.entity,
+ 0,
+ MEDIA_LNK_FL_ENABLED |
+ MEDIA_LNK_FL_IMMUTABLE);
+ if (ret) {
+ dev_err_probe(dev, ret, "fail to create pad link\n");
+ goto err_isp4_deinit;
+ }
+
ret = media_device_register(&isp_dev->mdev);
if (ret) {
dev_err_probe(dev, ret, "fail to register media device\n");
diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.c b/drivers/media/platform/amd/isp4/isp4_subdev.c
index 3a25d1fc49ce..8c33187efc54 100644
--- a/drivers/media/platform/amd/isp4/isp4_subdev.c
+++ b/drivers/media/platform/amd/isp4/isp4_subdev.c
@@ -449,7 +449,7 @@ static void isp4sd_fw_resp_frame_done(struct isp4_subdev *isp_subdev,
return;
}
- dev_dbg(dev, "ts:%llu,streamId:%d,poc:%u,preview_en:%u(%i)\n",
+ dev_dbg(dev, "ts:%llu,streamId:%d,poc:%u,preview_en:%u,(%i)\n",
ktime_get_ns(), stream_id, meta->poc,
meta->preview.enabled,
meta->preview.status);
@@ -459,11 +459,13 @@ static void isp4sd_fw_resp_frame_done(struct isp4_subdev *isp_subdev,
meta->preview.status == BUFFER_STATUS_DONE ||
meta->preview.status == BUFFER_STATUS_DIRTY)) {
prev = isp4if_dequeue_buffer(ispif);
- if (prev)
+ if (prev) {
+ isp4vid_handle_frame_done(&isp_subdev->isp_vdev,
+ &prev->buf_info);
isp4if_dealloc_buffer_node(prev);
- else
+ } else {
dev_err(dev, "fail null prev buf\n");
-
+ }
} else if (meta->preview.enabled) {
dev_err(dev, "fail bad preview status %u\n",
meta->preview.status);
@@ -794,6 +796,65 @@ static int isp4sd_start_stream(struct isp4_subdev *isp_subdev,
return ret;
}
+static int isp4sd_ioc_send_img_buf(struct v4l2_subdev *sd,
+ struct isp4if_img_buf_info *buf_info)
+{
+ struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
+ struct isp4_interface *ispif = &isp_subdev->ispif;
+ struct isp4if_img_buf_node *buf_node;
+ struct device *dev = isp_subdev->dev;
+ int ret;
+
+ guard(mutex)(&isp_subdev->ops_mutex);
+
+ if (ispif->status != ISP4IF_STATUS_FW_RUNNING) {
+ dev_err(dev, "fail send img buf for bad fsm %d\n",
+ ispif->status);
+ return -EINVAL;
+ }
+
+ buf_node = isp4if_alloc_buffer_node(buf_info);
+ if (!buf_node) {
+ dev_err(dev, "fail alloc sys img buf info node\n");
+ return -ENOMEM;
+ }
+
+ ret = isp4if_queue_buffer(ispif, buf_node);
+ if (ret) {
+ dev_err(dev, "fail to queue image buf, %d\n", ret);
+ goto error_release_buf_node;
+ }
+
+ if (!isp_subdev->sensor_info.start_stream_cmd_sent) {
+ isp_subdev->sensor_info.buf_sent_cnt++;
+
+ if (isp_subdev->sensor_info.buf_sent_cnt >=
+ ISP4SD_MIN_BUF_CNT_BEF_START_STREAM) {
+ ret = isp4if_send_command(ispif, CMD_ID_START_STREAM,
+ NULL, 0);
+ if (ret) {
+ dev_err(dev, "fail to START_STREAM");
+ goto error_release_buf_node;
+ }
+ isp_subdev->sensor_info.start_stream_cmd_sent = true;
+ isp_subdev->sensor_info.output_info.start_status =
+ ISP4SD_START_STATUS_STARTED;
+ isp_subdev->sensor_info.status =
+ ISP4SD_START_STATUS_STARTED;
+ } else {
+ dev_dbg(dev, "no send start, required %u, buf sent %u\n",
+ ISP4SD_MIN_BUF_CNT_BEF_START_STREAM,
+ isp_subdev->sensor_info.buf_sent_cnt);
+ }
+ }
+
+ return 0;
+
+error_release_buf_node:
+ isp4if_dealloc_buffer_node(buf_node);
+ return ret;
+}
+
static int isp4sd_set_power(struct v4l2_subdev *sd, int on)
{
struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
@@ -892,6 +953,10 @@ static const struct media_entity_operations isp4sd_sdev_ent_ops = {
.link_validate = isp4sd_sdev_link_validate,
};
+static const struct isp4vid_ops isp4sd_isp4vid_ops = {
+ .send_buffer = isp4sd_ioc_send_img_buf,
+};
+
int isp4sd_init(struct isp4_subdev *isp_subdev, struct v4l2_device *v4l2_dev,
int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM])
{
@@ -952,6 +1017,11 @@ int isp4sd_init(struct isp4_subdev *isp_subdev, struct v4l2_device *v4l2_dev,
isp_subdev->host2fw_seq_num = 1;
ispif->status = ISP4IF_STATUS_PWR_OFF;
+ ret = isp4vid_dev_init(&isp_subdev->isp_vdev, &isp_subdev->sdev,
+ &isp4sd_isp4vid_ops);
+ if (ret)
+ goto err_subdev_unreg;
+
return 0;
err_subdev_unreg:
@@ -966,6 +1036,7 @@ void isp4sd_deinit(struct isp4_subdev *isp_subdev)
{
struct isp4_interface *ispif = &isp_subdev->ispif;
+ isp4vid_dev_deinit(&isp_subdev->isp_vdev);
v4l2_device_unregister_subdev(&isp_subdev->sdev);
media_entity_cleanup(&isp_subdev->sdev.entity);
isp4if_deinit(ispif);
diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.h b/drivers/media/platform/amd/isp4/isp4_subdev.h
index 70007b9de3f3..386e4e68f3fa 100644
--- a/drivers/media/platform/amd/isp4/isp4_subdev.h
+++ b/drivers/media/platform/amd/isp4/isp4_subdev.h
@@ -17,6 +17,7 @@
#include "isp4_fw_cmd_resp.h"
#include "isp4_hw_reg.h"
#include "isp4_interface.h"
+#include "isp4_video.h"
/*
* one is for none sesnor specefic response which is not used now
@@ -93,6 +94,7 @@ struct isp4_subdev_thread_param {
struct isp4_subdev {
struct v4l2_subdev sdev;
struct isp4_interface ispif;
+ struct isp4vid_dev isp_vdev;
struct media_pad sdev_pad;
diff --git a/drivers/media/platform/amd/isp4/isp4_video.c b/drivers/media/platform/amd/isp4/isp4_video.c
new file mode 100644
index 000000000000..5006e5021155
--- /dev/null
+++ b/drivers/media/platform/amd/isp4/isp4_video.c
@@ -0,0 +1,1165 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2025 Advanced Micro Devices, Inc.
+ */
+
+#include <linux/vmalloc.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-mc.h>
+
+#include "isp4_interface.h"
+#include "isp4_subdev.h"
+#include "isp4_video.h"
+
+#define ISP4VID_ISP_DRV_NAME "amd_isp_capture"
+#define ISP4VID_MAX_PREVIEW_FPS 30
+#define ISP4VID_DEFAULT_FMT isp4vid_formats[0]
+
+#define ISP4VID_PAD_VIDEO_OUTPUT 0
+
+/* timeperframe default */
+#define ISP4VID_ISP_TPF_DEFAULT isp4vid_tpfs[0]
+
+/* amdisp buffer for vb2 operations */
+struct isp4vid_vb2_buf {
+ struct device *dev;
+ void *vaddr;
+ unsigned long size;
+ refcount_t refcount;
+ struct dma_buf *dbuf;
+ void *bo;
+ u64 gpu_addr;
+ struct vb2_vmarea_handler handler;
+};
+
+static void isp4vid_vb2_put(void *buf_priv);
+
+static const char *const isp4vid_video_dev_name = "Preview";
+
+/* Sizes must be in increasing order */
+static const struct v4l2_frmsize_discrete isp4vid_frmsize[] = {
+ {640, 360},
+ {640, 480},
+ {1280, 720},
+ {1280, 960},
+ {1920, 1080},
+ {1920, 1440},
+ {2560, 1440},
+ {2880, 1620},
+ {2880, 1624},
+ {2888, 1808},
+};
+
+static const u32 isp4vid_formats[] = {
+ V4L2_PIX_FMT_NV12,
+ V4L2_PIX_FMT_YUYV
+};
+
+/* timeperframe list */
+static const struct v4l2_fract isp4vid_tpfs[] = {
+ { 1, ISP4VID_MAX_PREVIEW_FPS }
+};
+
+void isp4vid_handle_frame_done(struct isp4vid_dev *isp_vdev,
+ const struct isp4if_img_buf_info *img_buf)
+{
+ struct isp4vid_capture_buffer *isp4vid_buf;
+ void *vbuf;
+
+ scoped_guard(mutex, &isp_vdev->buf_list_lock) {
+ /* Get the first entry of the list */
+ isp4vid_buf = list_first_entry_or_null(&isp_vdev->buf_list, typeof(*isp4vid_buf),
+ list);
+ if (!isp4vid_buf)
+ return;
+
+ vbuf = vb2_plane_vaddr(&isp4vid_buf->vb2.vb2_buf, 0);
+
+ if (vbuf != img_buf->planes[0].sys_addr) {
+ dev_err(isp_vdev->dev, "Invalid vbuf");
+ return;
+ }
+
+ /* Remove this entry from the list */
+ list_del(&isp4vid_buf->list);
+ }
+
+ /* Fill the buffer */
+ isp4vid_buf->vb2.vb2_buf.timestamp = ktime_get_ns();
+ isp4vid_buf->vb2.sequence = isp_vdev->sequence++;
+ isp4vid_buf->vb2.field = V4L2_FIELD_ANY;
+
+ /* at most 2 planes */
+ vb2_set_plane_payload(&isp4vid_buf->vb2.vb2_buf,
+ 0, isp_vdev->format.sizeimage);
+
+ vb2_buffer_done(&isp4vid_buf->vb2.vb2_buf, VB2_BUF_STATE_DONE);
+
+ dev_dbg(isp_vdev->dev, "call vb2_buffer_done(size=%u)\n",
+ isp_vdev->format.sizeimage);
+}
+
+static unsigned int isp4vid_vb2_num_users(void *buf_priv)
+{
+ struct isp4vid_vb2_buf *buf = buf_priv;
+
+ return refcount_read(&buf->refcount);
+}
+
+static int isp4vid_vb2_mmap(void *buf_priv, struct vm_area_struct *vma)
+{
+ struct isp4vid_vb2_buf *buf = buf_priv;
+ int ret;
+
+ if (!buf) {
+ pr_err("fail no memory to map\n");
+ return -EINVAL;
+ }
+
+ ret = remap_vmalloc_range(vma, buf->vaddr, 0);
+ if (ret) {
+ dev_err(buf->dev, "fail remap vmalloc mem, %d\n", ret);
+ return ret;
+ }
+
+ /*
+ * Make sure that vm_areas for 2 buffers won't be merged together
+ */
+ vm_flags_set(vma, VM_DONTEXPAND);
+
+ /*
+ * Use common vm_area operations to track buffer refcount.
+ */
+ vma->vm_private_data = &buf->handler;
+ vma->vm_ops = &vb2_common_vm_ops;
+
+ vma->vm_ops->open(vma);
+
+ dev_dbg(buf->dev, "mmap isp user bo 0x%llx size %ld refcount %d\n",
+ buf->gpu_addr, buf->size, refcount_read(&buf->refcount));
+
+ return 0;
+}
+
+static void *isp4vid_vb2_vaddr(struct vb2_buffer *vb, void *buf_priv)
+{
+ struct isp4vid_vb2_buf *buf = buf_priv;
+
+ if (!buf->vaddr) {
+ dev_err(buf->dev,
+ "fail for buf vaddr is null\n");
+ return NULL;
+ }
+
+ return buf->vaddr;
+}
+
+static void isp4vid_vb2_detach_dmabuf(void *mem_priv)
+{
+ struct isp4vid_vb2_buf *buf = mem_priv;
+
+ dev_dbg(buf->dev, "detach dmabuf of isp user bo 0x%llx size %ld",
+ buf->gpu_addr, buf->size);
+
+ kfree(buf);
+}
+
+static void *isp4vid_vb2_attach_dmabuf(struct vb2_buffer *vb, struct device *dev,
+ struct dma_buf *dbuf,
+ unsigned long size)
+{
+ struct isp4vid_vb2_buf *dbg_buf = dbuf->priv;
+ struct isp4vid_vb2_buf *buf;
+
+ if (dbuf->size < size) {
+ dev_err(dev, "Invalid dmabuf size %zu %lu", dbuf->size, size);
+ return ERR_PTR(-EFAULT);
+ }
+
+ buf = kzalloc(sizeof(*buf), GFP_KERNEL);
+ if (!buf)
+ return ERR_PTR(-ENOMEM);
+
+ buf->dev = dev;
+ buf->dbuf = dbuf;
+ buf->size = size;
+
+ dev_dbg(dev, "attach dmabuf of isp user bo 0x%llx size %ld",
+ dbg_buf->gpu_addr, dbg_buf->size);
+
+ return buf;
+}
+
+static void isp4vid_vb2_unmap_dmabuf(void *mem_priv)
+{
+ struct isp4vid_vb2_buf *buf = mem_priv;
+ struct iosys_map map = IOSYS_MAP_INIT_VADDR(buf->vaddr);
+
+ dev_dbg(buf->dev, "unmap dmabuf of isp user bo 0x%llx size %ld",
+ buf->gpu_addr, buf->size);
+
+ dma_buf_vunmap_unlocked(buf->dbuf, &map);
+ buf->vaddr = NULL;
+}
+
+static int isp4vid_vb2_map_dmabuf(void *mem_priv)
+{
+ struct isp4vid_vb2_buf *buf = mem_priv, *mmap_buf;
+ struct iosys_map map;
+ int ret;
+
+ ret = dma_buf_vmap_unlocked(buf->dbuf, &map);
+ if (ret) {
+ dev_err(buf->dev, "vmap_unlocked fail");
+ return -EFAULT;
+ }
+
+ buf->vaddr = map.vaddr;
+ mmap_buf = buf->dbuf->priv;
+ buf->gpu_addr = mmap_buf->gpu_addr;
+
+ dev_dbg(buf->dev, "map dmabuf of isp user bo 0x%llx size %ld",
+ buf->gpu_addr, buf->size);
+
+ return 0;
+}
+
+#ifdef CONFIG_HAS_DMA
+struct isp4vid_vb2_amdgpu_attachment {
+ struct sg_table sgt;
+ enum dma_data_direction dma_dir;
+};
+
+static int isp4vid_dmabuf_ops_attach(struct dma_buf *dbuf,
+ struct dma_buf_attachment *dbuf_attach)
+{
+ struct isp4vid_vb2_buf *buf = dbuf->priv;
+ int num_pages = PAGE_ALIGN(buf->size) / PAGE_SIZE;
+ struct isp4vid_vb2_amdgpu_attachment *attach;
+ void *vaddr = buf->vaddr;
+ struct scatterlist *sg;
+ struct sg_table *sgt;
+ int ret;
+ int i;
+
+ attach = kzalloc(sizeof(*attach), GFP_KERNEL);
+ if (!attach)
+ return -ENOMEM;
+
+ sgt = &attach->sgt;
+ ret = sg_alloc_table(sgt, num_pages, GFP_KERNEL);
+ if (ret) {
+ kfree(attach);
+ return ret;
+ }
+
+ for_each_sgtable_sg(sgt, sg, i) {
+ struct page *page = vmalloc_to_page(vaddr);
+
+ if (!page) {
+ sg_free_table(sgt);
+ kfree(attach);
+ return -ENOMEM;
+ }
+
+ sg_set_page(sg, page, PAGE_SIZE, 0);
+ vaddr = ((char *)vaddr) + PAGE_SIZE;
+ }
+
+ attach->dma_dir = DMA_NONE;
+ dbuf_attach->priv = attach;
+
+ return 0;
+}
+
+static void isp4vid_dmabuf_ops_detach(struct dma_buf *dbuf,
+ struct dma_buf_attachment *dbuf_attach)
+{
+ struct isp4vid_vb2_amdgpu_attachment *attach = dbuf_attach->priv;
+ struct sg_table *sgt;
+
+ if (!attach) {
+ pr_err("fail invalid attach handler\n");
+ return;
+ }
+
+ sgt = &attach->sgt;
+
+ /* release the scatterlist cache */
+ if (attach->dma_dir != DMA_NONE)
+ dma_unmap_sgtable(dbuf_attach->dev, sgt, attach->dma_dir, 0);
+
+ sg_free_table(sgt);
+ kfree(attach);
+ dbuf_attach->priv = NULL;
+}
+
+static struct sg_table *isp4vid_dmabuf_ops_map(struct dma_buf_attachment *dbuf_attach,
+ enum dma_data_direction dma_dir)
+{
+ struct isp4vid_vb2_amdgpu_attachment *attach = dbuf_attach->priv;
+ struct sg_table *sgt;
+
+ sgt = &attach->sgt;
+ /* return previously mapped sg table */
+ if (attach->dma_dir == dma_dir)
+ return sgt;
+
+ /* release any previous cache */
+ if (attach->dma_dir != DMA_NONE) {
+ dma_unmap_sgtable(dbuf_attach->dev, sgt, attach->dma_dir, 0);
+ attach->dma_dir = DMA_NONE;
+ }
+
+ /* mapping to the client with new direction */
+ if (dma_map_sgtable(dbuf_attach->dev, sgt, dma_dir, 0)) {
+ dev_err(dbuf_attach->dev, "fail to map scatterlist");
+ return ERR_PTR(-EIO);
+ }
+
+ attach->dma_dir = dma_dir;
+
+ return sgt;
+}
+
+static void isp4vid_dmabuf_ops_unmap(struct dma_buf_attachment *dbuf_attach,
+ struct sg_table *sgt,
+ enum dma_data_direction dma_dir)
+{
+ /* nothing to be done here */
+}
+
+static int isp4vid_dmabuf_ops_vmap(struct dma_buf *dbuf,
+ struct iosys_map *map)
+{
+ struct isp4vid_vb2_buf *buf = dbuf->priv;
+
+ iosys_map_set_vaddr(map, buf->vaddr);
+
+ return 0;
+}
+
+static void isp4vid_dmabuf_ops_release(struct dma_buf *dbuf)
+{
+ struct isp4vid_vb2_buf *buf = dbuf->priv;
+
+ /* drop reference obtained in isp4vid_vb2_get_dmabuf */
+ if (dbuf != buf->dbuf)
+ isp4vid_vb2_put(buf);
+ else
+ kfree(buf);
+}
+
+static int isp4vid_dmabuf_ops_mmap(struct dma_buf *dbuf, struct vm_area_struct *vma)
+{
+ return isp4vid_vb2_mmap(dbuf->priv, vma);
+}
+
+static const struct dma_buf_ops isp4vid_dmabuf_ops = {
+ .attach = isp4vid_dmabuf_ops_attach,
+ .detach = isp4vid_dmabuf_ops_detach,
+ .map_dma_buf = isp4vid_dmabuf_ops_map,
+ .unmap_dma_buf = isp4vid_dmabuf_ops_unmap,
+ .vmap = isp4vid_dmabuf_ops_vmap,
+ .mmap = isp4vid_dmabuf_ops_mmap,
+ .release = isp4vid_dmabuf_ops_release,
+};
+
+static struct dma_buf *isp4vid_get_dmabuf(struct isp4vid_vb2_buf *buf, unsigned long flags)
+{
+ DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
+ struct dma_buf *dbuf;
+
+ if (WARN_ON(!buf->vaddr))
+ return NULL;
+
+ exp_info.ops = &isp4vid_dmabuf_ops;
+ exp_info.size = buf->size;
+ exp_info.flags = flags;
+ exp_info.priv = buf;
+
+ dbuf = dma_buf_export(&exp_info);
+ if (IS_ERR(dbuf))
+ return NULL;
+
+ return dbuf;
+}
+
+static struct dma_buf *isp4vid_vb2_get_dmabuf(struct vb2_buffer *vb, void *buf_priv,
+ unsigned long flags)
+{
+ struct isp4vid_vb2_buf *buf = buf_priv;
+ struct dma_buf *dbuf;
+
+ dbuf = isp4vid_get_dmabuf(buf, flags);
+ if (!dbuf) {
+ dev_err(buf->dev, "fail to get isp dma buffer\n");
+ return NULL;
+ }
+
+ refcount_inc(&buf->refcount);
+
+ dev_dbg(buf->dev, "buf exported, refcount %d\n",
+ refcount_read(&buf->refcount));
+
+ return dbuf;
+}
+#endif /* CONFIG_HAS_DMA */
+
+static void isp4vid_vb2_put(void *buf_priv)
+{
+ struct isp4vid_vb2_buf *buf = buf_priv;
+
+ dev_dbg(buf->dev,
+ "release isp user bo 0x%llx size %ld refcount %d",
+ buf->gpu_addr, buf->size,
+ refcount_read(&buf->refcount));
+
+ if (refcount_dec_and_test(&buf->refcount)) {
+ isp_user_buffer_free(buf->bo);
+ vfree(buf->vaddr);
+ /*
+ * Putting the implicit dmabuf frees `buf`. Freeing `buf` must
+ * be done from the dmabuf release callback since dma_buf_put()
+ * isn't always synchronous; it's just an fput(), which may be
+ * deferred. Since the dmabuf release callback needs to access
+ * `buf`, this means `buf` cannot be freed until then.
+ */
+ dma_buf_put(buf->dbuf);
+ }
+}
+
+static void *isp4vid_vb2_alloc(struct vb2_buffer *vb, struct device *dev, unsigned long size)
+{
+ struct isp4vid_vb2_buf *buf;
+ u64 gpu_addr;
+ void *bo;
+ int ret;
+
+ buf = kzalloc(sizeof(*buf), GFP_KERNEL | vb->vb2_queue->gfp_flags);
+ if (!buf)
+ return ERR_PTR(-ENOMEM);
+
+ buf->dev = dev;
+ buf->size = size;
+ buf->vaddr = vmalloc_user(buf->size);
+ if (!buf->vaddr) {
+ dev_err(dev, "fail to vmalloc buffer\n");
+ goto free_buf;
+ }
+
+ buf->handler.refcount = &buf->refcount;
+ buf->handler.put = isp4vid_vb2_put;
+ buf->handler.arg = buf;
+
+ /* get implicit dmabuf */
+ buf->dbuf = isp4vid_get_dmabuf(buf, 0);
+ if (!buf->dbuf) {
+ dev_err(dev, "fail to get implicit dmabuf\n");
+ goto free_user_vmem;
+ }
+
+ /* create isp user BO and obtain gpu_addr */
+ ret = isp_user_buffer_alloc(dev, buf->dbuf, &bo, &gpu_addr);
+ if (ret) {
+ dev_err(dev, "fail to create isp user BO\n");
+ goto put_dmabuf;
+ }
+
+ buf->bo = bo;
+ buf->gpu_addr = gpu_addr;
+
+ refcount_set(&buf->refcount, 1);
+
+ dev_dbg(dev, "allocated isp user bo 0x%llx size %ld refcount %d\n",
+ buf->gpu_addr, buf->size, refcount_read(&buf->refcount));
+
+ return buf;
+
+put_dmabuf:
+ dma_buf_put(buf->dbuf);
+free_user_vmem:
+ vfree(buf->vaddr);
+free_buf:
+ ret = buf->vaddr ? -EINVAL : -ENOMEM;
+ kfree(buf);
+ return ERR_PTR(ret);
+}
+
+static const struct vb2_mem_ops isp4vid_vb2_memops = {
+ .alloc = isp4vid_vb2_alloc,
+ .put = isp4vid_vb2_put,
+#ifdef CONFIG_HAS_DMA
+ .get_dmabuf = isp4vid_vb2_get_dmabuf,
+#endif
+ .map_dmabuf = isp4vid_vb2_map_dmabuf,
+ .unmap_dmabuf = isp4vid_vb2_unmap_dmabuf,
+ .attach_dmabuf = isp4vid_vb2_attach_dmabuf,
+ .detach_dmabuf = isp4vid_vb2_detach_dmabuf,
+ .vaddr = isp4vid_vb2_vaddr,
+ .mmap = isp4vid_vb2_mmap,
+ .num_users = isp4vid_vb2_num_users,
+};
+
+static const struct v4l2_pix_format isp4vid_fmt_default = {
+ .width = 1920,
+ .height = 1080,
+ .pixelformat = ISP4VID_DEFAULT_FMT,
+ .field = V4L2_FIELD_NONE,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+};
+
+static void isp4vid_capture_return_all_buffers(struct isp4vid_dev *isp_vdev,
+ enum vb2_buffer_state state)
+{
+ struct isp4vid_capture_buffer *vbuf, *node;
+
+ scoped_guard(mutex, &isp_vdev->buf_list_lock) {
+ list_for_each_entry_safe(vbuf, node, &isp_vdev->buf_list, list)
+ vb2_buffer_done(&vbuf->vb2.vb2_buf, state);
+ INIT_LIST_HEAD(&isp_vdev->buf_list);
+ }
+
+ dev_dbg(isp_vdev->dev, "call vb2_buffer_done(%d)\n", state);
+}
+
+static int isp4vid_vdev_link_validate(struct media_link *link)
+{
+ return 0;
+}
+
+static const struct media_entity_operations isp4vid_vdev_ent_ops = {
+ .link_validate = isp4vid_vdev_link_validate,
+};
+
+static const struct v4l2_file_operations isp4vid_vdev_fops = {
+ .owner = THIS_MODULE,
+ .open = v4l2_fh_open,
+ .release = vb2_fop_release,
+ .read = vb2_fop_read,
+ .poll = vb2_fop_poll,
+ .unlocked_ioctl = video_ioctl2,
+ .mmap = vb2_fop_mmap,
+};
+
+static int isp4vid_ioctl_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
+{
+ struct isp4vid_dev *isp_vdev = video_drvdata(file);
+
+ strscpy(cap->driver, ISP4VID_ISP_DRV_NAME, sizeof(cap->driver));
+ snprintf(cap->card, sizeof(cap->card), "%s", ISP4VID_ISP_DRV_NAME);
+ cap->capabilities |= V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_CAPTURE;
+
+ dev_dbg(isp_vdev->dev, "%s|capabilities=0x%X", isp_vdev->vdev.name, cap->capabilities);
+
+ return 0;
+}
+
+static int isp4vid_g_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f)
+{
+ struct isp4vid_dev *isp_vdev = video_drvdata(file);
+
+ f->fmt.pix = isp_vdev->format;
+
+ return 0;
+}
+
+static int isp4vid_fill_buffer_size(struct v4l2_pix_format *fmt)
+{
+ int ret = 0;
+
+ switch (fmt->pixelformat) {
+ case V4L2_PIX_FMT_NV12:
+ fmt->bytesperline = fmt->width;
+ fmt->sizeimage = fmt->bytesperline * fmt->height * 3 / 2;
+ break;
+ case V4L2_PIX_FMT_YUYV:
+ fmt->bytesperline = fmt->width * 2;
+ fmt->sizeimage = fmt->bytesperline * fmt->height;
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static int isp4vid_try_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f)
+{
+ struct isp4vid_dev *isp_vdev = video_drvdata(file);
+ struct v4l2_pix_format *format = &f->fmt.pix;
+ const struct v4l2_frmsize_discrete *fsz;
+ int i;
+
+ if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+ return -EINVAL;
+
+ /*
+ * Check if the hardware supports the requested format, use the default
+ * format otherwise.
+ */
+ for (i = 0; i < ARRAY_SIZE(isp4vid_formats); i++)
+ if (isp4vid_formats[i] == format->pixelformat)
+ break;
+
+ if (i == ARRAY_SIZE(isp4vid_formats))
+ format->pixelformat = ISP4VID_DEFAULT_FMT;
+
+ switch (format->pixelformat) {
+ case V4L2_PIX_FMT_NV12:
+ case V4L2_PIX_FMT_YUYV:
+ fsz = v4l2_find_nearest_size(isp4vid_frmsize, ARRAY_SIZE(isp4vid_frmsize),
+ width, height, format->width, format->height);
+ format->width = fsz->width;
+ format->height = fsz->height;
+ break;
+ default:
+ dev_err(isp_vdev->dev, "%s|unsupported fmt=%u", isp_vdev->vdev.name,
+ format->pixelformat);
+ return -EINVAL;
+ }
+
+ /* There is no need to check the return value, as failure will never happen here */
+ isp4vid_fill_buffer_size(format);
+
+ if (format->field == V4L2_FIELD_ANY)
+ format->field = isp4vid_fmt_default.field;
+
+ if (format->colorspace == V4L2_COLORSPACE_DEFAULT)
+ format->colorspace = isp4vid_fmt_default.colorspace;
+
+ return 0;
+}
+
+static int isp4vid_set_fmt_2_isp(struct v4l2_subdev *sdev, struct v4l2_pix_format *pix_fmt)
+{
+ struct v4l2_subdev_format fmt = {};
+
+ switch (pix_fmt->pixelformat) {
+ case V4L2_PIX_FMT_NV12:
+ fmt.format.code = MEDIA_BUS_FMT_YUYV8_1_5X8;
+ break;
+ case V4L2_PIX_FMT_YUYV:
+ fmt.format.code = MEDIA_BUS_FMT_YUYV8_1X16;
+ break;
+ default:
+ return -EINVAL;
+ }
+ fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
+ fmt.pad = ISP4VID_PAD_VIDEO_OUTPUT;
+ fmt.format.width = pix_fmt->width;
+ fmt.format.height = pix_fmt->height;
+ return v4l2_subdev_call(sdev, pad, set_fmt, NULL, &fmt);
+}
+
+static int isp4vid_s_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f)
+{
+ struct isp4vid_dev *isp_vdev = video_drvdata(file);
+ int ret;
+
+ /* Do not change the format while stream is on */
+ if (vb2_is_busy(&isp_vdev->vbq))
+ return -EBUSY;
+
+ if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+ return -EINVAL;
+
+ ret = isp4vid_try_fmt_vid_cap(file, priv, f);
+ if (ret)
+ return ret;
+
+ dev_dbg(isp_vdev->dev, "%s|width height:%ux%u->%ux%u",
+ isp_vdev->vdev.name,
+ isp_vdev->format.width, isp_vdev->format.height,
+ f->fmt.pix.width, f->fmt.pix.height);
+ dev_dbg(isp_vdev->dev, "%s|pixelformat:0x%x-0x%x",
+ isp_vdev->vdev.name, isp_vdev->format.pixelformat,
+ f->fmt.pix.pixelformat);
+ dev_dbg(isp_vdev->dev, "%s|bytesperline:%u->%u",
+ isp_vdev->vdev.name, isp_vdev->format.bytesperline,
+ f->fmt.pix.bytesperline);
+ dev_dbg(isp_vdev->dev, "%s|sizeimage:%u->%u",
+ isp_vdev->vdev.name, isp_vdev->format.sizeimage,
+ f->fmt.pix.sizeimage);
+
+ isp_vdev->format = f->fmt.pix;
+ ret = isp4vid_set_fmt_2_isp(isp_vdev->isp_sdev, &isp_vdev->format);
+
+ return ret;
+}
+
+static int isp4vid_enum_fmt_vid_cap(struct file *file, void *priv, struct v4l2_fmtdesc *f)
+{
+ struct isp4vid_dev *isp_vdev = video_drvdata(file);
+
+ switch (f->index) {
+ case 0:
+ f->pixelformat = V4L2_PIX_FMT_NV12;
+ break;
+ case 1:
+ f->pixelformat = V4L2_PIX_FMT_YUYV;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ dev_dbg(isp_vdev->dev, "%s|index=%d, pixelformat=0x%X",
+ isp_vdev->vdev.name, f->index, f->pixelformat);
+
+ return 0;
+}
+
+static int isp4vid_enum_framesizes(struct file *file, void *fh, struct v4l2_frmsizeenum *fsize)
+{
+ struct isp4vid_dev *isp_vdev = video_drvdata(file);
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(isp4vid_formats); i++) {
+ if (isp4vid_formats[i] == fsize->pixel_format)
+ break;
+ }
+ if (i == ARRAY_SIZE(isp4vid_formats))
+ return -EINVAL;
+
+ if (fsize->index < ARRAY_SIZE(isp4vid_frmsize)) {
+ fsize->type = V4L2_FRMSIZE_TYPE_DISCRETE;
+ fsize->discrete = isp4vid_frmsize[fsize->index];
+ dev_dbg(isp_vdev->dev, "%s|size[%d]=%dx%d",
+ isp_vdev->vdev.name, fsize->index,
+ fsize->discrete.width, fsize->discrete.height);
+ } else {
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int isp4vid_ioctl_enum_frameintervals(struct file *file, void *priv,
+ struct v4l2_frmivalenum *fival)
+{
+ struct isp4vid_dev *isp_vdev = video_drvdata(file);
+ int i;
+
+ if (fival->index >= ARRAY_SIZE(isp4vid_tpfs))
+ return -EINVAL;
+
+ for (i = 0; i < ARRAY_SIZE(isp4vid_formats); i++)
+ if (isp4vid_formats[i] == fival->pixel_format)
+ break;
+
+ if (i == ARRAY_SIZE(isp4vid_formats))
+ return -EINVAL;
+
+ for (i = 0; i < ARRAY_SIZE(isp4vid_frmsize); i++)
+ if (isp4vid_frmsize[i].width == fival->width &&
+ isp4vid_frmsize[i].height == fival->height)
+ break;
+
+ if (i == ARRAY_SIZE(isp4vid_frmsize))
+ return -EINVAL;
+
+ fival->type = V4L2_FRMIVAL_TYPE_DISCRETE;
+ fival->discrete = isp4vid_tpfs[fival->index];
+ v4l2_simplify_fraction(&fival->discrete.numerator,
+ &fival->discrete.denominator, 8, 333);
+
+ dev_dbg(isp_vdev->dev, "%s|interval[%d]=%d/%d",
+ isp_vdev->vdev.name, fival->index,
+ fival->discrete.numerator,
+ fival->discrete.denominator);
+
+ return 0;
+}
+
+static int isp4vid_ioctl_g_param(struct file *file, void *priv, struct v4l2_streamparm *param)
+{
+ struct v4l2_captureparm *capture = ¶m->parm.capture;
+ struct isp4vid_dev *isp_vdev = video_drvdata(file);
+
+ if (param->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+ return -EINVAL;
+
+ capture->capability = V4L2_CAP_TIMEPERFRAME;
+ capture->timeperframe = isp_vdev->timeperframe;
+ capture->readbuffers = 0;
+
+ dev_dbg(isp_vdev->dev, "%s|timeperframe=%d/%d", isp_vdev->vdev.name,
+ capture->timeperframe.numerator,
+ capture->timeperframe.denominator);
+
+ return 0;
+}
+
+static const struct v4l2_ioctl_ops isp4vid_vdev_ioctl_ops = {
+ /* VIDIOC_QUERYCAP handler */
+ .vidioc_querycap = isp4vid_ioctl_querycap,
+
+ /* VIDIOC_ENUM_FMT handlers */
+ .vidioc_enum_fmt_vid_cap = isp4vid_enum_fmt_vid_cap,
+
+ /* VIDIOC_G_FMT handlers */
+ .vidioc_g_fmt_vid_cap = isp4vid_g_fmt_vid_cap,
+
+ /* VIDIOC_S_FMT handlers */
+ .vidioc_s_fmt_vid_cap = isp4vid_s_fmt_vid_cap,
+
+ /* VIDIOC_TRY_FMT handlers */
+ .vidioc_try_fmt_vid_cap = isp4vid_try_fmt_vid_cap,
+
+ /* Buffer handlers */
+ .vidioc_reqbufs = vb2_ioctl_reqbufs,
+ .vidioc_querybuf = vb2_ioctl_querybuf,
+ .vidioc_qbuf = vb2_ioctl_qbuf,
+ .vidioc_expbuf = vb2_ioctl_expbuf,
+ .vidioc_dqbuf = vb2_ioctl_dqbuf,
+ .vidioc_create_bufs = vb2_ioctl_create_bufs,
+ .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+
+ /* Stream on/off */
+ .vidioc_streamon = vb2_ioctl_streamon,
+ .vidioc_streamoff = vb2_ioctl_streamoff,
+
+ /* Stream type-dependent parameter ioctls */
+ .vidioc_g_parm = isp4vid_ioctl_g_param,
+ .vidioc_s_parm = isp4vid_ioctl_g_param,
+
+ .vidioc_enum_framesizes = isp4vid_enum_framesizes,
+ .vidioc_enum_frameintervals = isp4vid_ioctl_enum_frameintervals,
+
+};
+
+static unsigned int isp4vid_get_image_size(struct v4l2_pix_format *fmt)
+{
+ switch (fmt->pixelformat) {
+ case V4L2_PIX_FMT_NV12:
+ return fmt->width * fmt->height * 3 / 2;
+ case V4L2_PIX_FMT_YUYV:
+ return fmt->width * fmt->height * 2;
+ default:
+ return 0;
+ }
+}
+
+static int isp4vid_qops_queue_setup(struct vb2_queue *vq, unsigned int *nbuffers,
+ unsigned int *nplanes, unsigned int sizes[],
+ struct device *alloc_devs[])
+{
+ struct isp4vid_dev *isp_vdev = vb2_get_drv_priv(vq);
+ unsigned int q_num_bufs = vb2_get_num_buffers(vq);
+
+ if (*nplanes > 1) {
+ dev_err(isp_vdev->dev,
+ "fail to setup queue, no mplane supported %u\n",
+ *nplanes);
+ return -EINVAL;
+ }
+
+ if (*nplanes == 1) {
+ unsigned int size;
+
+ size = isp4vid_get_image_size(&isp_vdev->format);
+ if (sizes[0] < size) {
+ dev_err(isp_vdev->dev,
+ "fail for small plane size %u, %u expected\n",
+ sizes[0], size);
+ return -EINVAL;
+ }
+ }
+
+ if (q_num_bufs + *nbuffers < ISP4IF_MAX_STREAM_BUF_COUNT)
+ *nbuffers = ISP4IF_MAX_STREAM_BUF_COUNT - q_num_bufs;
+
+ switch (isp_vdev->format.pixelformat) {
+ case V4L2_PIX_FMT_NV12:
+ case V4L2_PIX_FMT_YUYV: {
+ *nplanes = 1;
+ sizes[0] = max(sizes[0], isp_vdev->format.sizeimage);
+ isp_vdev->format.sizeimage = sizes[0];
+ }
+ break;
+ default:
+ dev_err(isp_vdev->dev, "%s|unsupported fmt=%u\n",
+ isp_vdev->vdev.name, isp_vdev->format.pixelformat);
+ return -EINVAL;
+ }
+
+ dev_dbg(isp_vdev->dev, "%s|*nbuffers=%u *nplanes=%u sizes[0]=%u\n",
+ isp_vdev->vdev.name,
+ *nbuffers, *nplanes, sizes[0]);
+
+ return 0;
+}
+
+static void isp4vid_qops_buffer_queue(struct vb2_buffer *vb)
+{
+ struct isp4vid_capture_buffer *buf =
+ container_of(vb, struct isp4vid_capture_buffer, vb2.vb2_buf);
+ struct isp4vid_dev *isp_vdev = vb2_get_drv_priv(vb->vb2_queue);
+
+ struct isp4vid_vb2_buf *priv_buf = vb->planes[0].mem_priv;
+ struct isp4if_img_buf_info *img_buf = &buf->img_buf;
+
+ dev_dbg(isp_vdev->dev, "%s|index=%u", isp_vdev->vdev.name, vb->index);
+
+ dev_dbg(isp_vdev->dev, "queue isp user bo 0x%llx size=%lu",
+ priv_buf->gpu_addr,
+ priv_buf->size);
+
+ switch (isp_vdev->format.pixelformat) {
+ case V4L2_PIX_FMT_NV12: {
+ u32 y_size = isp_vdev->format.sizeimage / 3 * 2;
+ u32 uv_size = isp_vdev->format.sizeimage / 3;
+
+ img_buf->planes[0].len = y_size;
+ img_buf->planes[0].sys_addr = priv_buf->vaddr;
+ img_buf->planes[0].mc_addr = priv_buf->gpu_addr;
+
+ dev_dbg(isp_vdev->dev, "img_buf[0]: mc=0x%llx size=%u",
+ img_buf->planes[0].mc_addr,
+ img_buf->planes[0].len);
+
+ img_buf->planes[1].len = uv_size;
+ img_buf->planes[1].sys_addr = priv_buf->vaddr + y_size;
+ img_buf->planes[1].mc_addr = priv_buf->gpu_addr + y_size;
+
+ dev_dbg(isp_vdev->dev, "img_buf[1]: mc=0x%llx size=%u",
+ img_buf->planes[1].mc_addr,
+ img_buf->planes[1].len);
+
+ img_buf->planes[2].len = 0;
+ }
+ break;
+ case V4L2_PIX_FMT_YUYV: {
+ img_buf->planes[0].len = isp_vdev->format.sizeimage;
+ img_buf->planes[0].sys_addr = priv_buf->vaddr;
+ img_buf->planes[0].mc_addr = priv_buf->gpu_addr;
+
+ dev_dbg(isp_vdev->dev, "img_buf[0]: mc=0x%llx size=%u",
+ img_buf->planes[0].mc_addr,
+ img_buf->planes[0].len);
+
+ img_buf->planes[1].len = 0;
+ img_buf->planes[2].len = 0;
+ }
+ break;
+ default:
+ dev_err(isp_vdev->dev, "%s|unsupported fmt=%u",
+ isp_vdev->vdev.name, isp_vdev->format.pixelformat);
+ return;
+ }
+
+ if (isp_vdev->stream_started)
+ isp_vdev->ops->send_buffer(isp_vdev->isp_sdev, img_buf);
+
+ scoped_guard(mutex, &isp_vdev->buf_list_lock)
+ list_add_tail(&buf->list, &isp_vdev->buf_list);
+}
+
+static int isp4vid_qops_start_streaming(struct vb2_queue *vq, unsigned int count)
+{
+ struct isp4vid_dev *isp_vdev = vb2_get_drv_priv(vq);
+ struct isp4vid_capture_buffer *isp4vid_buf;
+ struct media_entity *entity;
+ struct v4l2_subdev *subdev;
+ struct media_pad *pad;
+ int ret = 0;
+
+ isp_vdev->sequence = 0;
+ ret = v4l2_pipeline_pm_get(&isp_vdev->vdev.entity);
+ if (ret) {
+ dev_err(isp_vdev->dev, "power up isp fail %d\n", ret);
+ goto release_buffers;
+ }
+
+ entity = &isp_vdev->vdev.entity;
+ while (1) {
+ pad = &entity->pads[0];
+ if (!(pad->flags & MEDIA_PAD_FL_SINK))
+ break;
+
+ pad = media_pad_remote_pad_first(pad);
+ if (!pad || !is_media_entity_v4l2_subdev(pad->entity))
+ break;
+
+ entity = pad->entity;
+ subdev = media_entity_to_v4l2_subdev(entity);
+
+ ret = v4l2_subdev_call(subdev, video, s_stream, 1);
+ if (ret < 0 && ret != -ENOIOCTLCMD) {
+ dev_dbg(isp_vdev->dev, "fail start streaming: %s %d\n",
+ subdev->name, ret);
+ goto release_buffers;
+ }
+ }
+
+ list_for_each_entry(isp4vid_buf, &isp_vdev->buf_list, list)
+ isp_vdev->ops->send_buffer(isp_vdev->isp_sdev, &isp4vid_buf->img_buf);
+
+ /* Start the media pipeline */
+ ret = video_device_pipeline_start(&isp_vdev->vdev, &isp_vdev->pipe);
+ if (ret) {
+ dev_err(isp_vdev->dev, "video_device_pipeline_start fail:%d",
+ ret);
+ goto release_buffers;
+ }
+
+ isp_vdev->stream_started = true;
+
+ return 0;
+
+release_buffers:
+ isp4vid_capture_return_all_buffers(isp_vdev, VB2_BUF_STATE_QUEUED);
+ return ret;
+}
+
+static void isp4vid_qops_stop_streaming(struct vb2_queue *vq)
+{
+ struct isp4vid_dev *isp_vdev = vb2_get_drv_priv(vq);
+ struct v4l2_subdev *subdev;
+ struct media_entity *entity;
+ struct media_pad *pad;
+ int ret;
+
+ entity = &isp_vdev->vdev.entity;
+ while (1) {
+ pad = &entity->pads[0];
+ if (!(pad->flags & MEDIA_PAD_FL_SINK))
+ break;
+
+ pad = media_pad_remote_pad_first(pad);
+ if (!pad || !is_media_entity_v4l2_subdev(pad->entity))
+ break;
+
+ entity = pad->entity;
+ subdev = media_entity_to_v4l2_subdev(entity);
+
+ ret = v4l2_subdev_call(subdev, video, s_stream, 0);
+
+ if (ret < 0 && ret != -ENOIOCTLCMD)
+ dev_dbg(isp_vdev->dev, "fail stop streaming: %s %d\n",
+ subdev->name, ret);
+ }
+
+ isp_vdev->stream_started = false;
+ v4l2_pipeline_pm_put(&isp_vdev->vdev.entity);
+
+ /* Stop the media pipeline */
+ video_device_pipeline_stop(&isp_vdev->vdev);
+
+ /* Release all active buffers */
+ isp4vid_capture_return_all_buffers(isp_vdev, VB2_BUF_STATE_ERROR);
+}
+
+static const struct vb2_ops isp4vid_qops = {
+ .queue_setup = isp4vid_qops_queue_setup,
+ .buf_queue = isp4vid_qops_buffer_queue,
+ .start_streaming = isp4vid_qops_start_streaming,
+ .stop_streaming = isp4vid_qops_stop_streaming,
+ .wait_prepare = vb2_ops_wait_prepare,
+ .wait_finish = vb2_ops_wait_finish,
+};
+
+int isp4vid_dev_init(struct isp4vid_dev *isp_vdev, struct v4l2_subdev *isp_sdev,
+ const struct isp4vid_ops *ops)
+{
+ const char *vdev_name = isp4vid_video_dev_name;
+ struct v4l2_device *v4l2_dev;
+ struct video_device *vdev;
+ struct vb2_queue *q;
+ int ret;
+
+ if (!isp_vdev || !isp_sdev || !isp_sdev->v4l2_dev)
+ return -EINVAL;
+
+ v4l2_dev = isp_sdev->v4l2_dev;
+ vdev = &isp_vdev->vdev;
+
+ isp_vdev->isp_sdev = isp_sdev;
+ isp_vdev->dev = v4l2_dev->dev;
+ isp_vdev->ops = ops;
+
+ /* Initialize the vb2_queue struct */
+ mutex_init(&isp_vdev->vbq_lock);
+ q = &isp_vdev->vbq;
+ q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+ q->io_modes = VB2_MMAP | VB2_DMABUF;
+ q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+ q->buf_struct_size = sizeof(struct isp4vid_capture_buffer);
+ q->min_queued_buffers = 2;
+ q->ops = &isp4vid_qops;
+ q->drv_priv = isp_vdev;
+ q->mem_ops = &isp4vid_vb2_memops;
+ q->lock = &isp_vdev->vbq_lock;
+ q->dev = v4l2_dev->dev;
+ ret = vb2_queue_init(q);
+ if (ret) {
+ dev_err(v4l2_dev->dev, "vb2_queue_init error:%d", ret);
+ return ret;
+ }
+ /* Initialize buffer list and its lock */
+ mutex_init(&isp_vdev->buf_list_lock);
+ INIT_LIST_HEAD(&isp_vdev->buf_list);
+
+ /* Set default frame format */
+ isp_vdev->format = isp4vid_fmt_default;
+ isp_vdev->timeperframe = ISP4VID_ISP_TPF_DEFAULT;
+ v4l2_simplify_fraction(&isp_vdev->timeperframe.numerator,
+ &isp_vdev->timeperframe.denominator, 8, 333);
+
+ ret = isp4vid_fill_buffer_size(&isp_vdev->format);
+ if (ret) {
+ dev_err(v4l2_dev->dev, "fail to fill buffer size: %d\n", ret);
+ goto err_release_vb2_queue;
+ }
+
+ ret = isp4vid_set_fmt_2_isp(isp_sdev, &isp_vdev->format);
+ if (ret) {
+ dev_err(v4l2_dev->dev, "fail init format :%d\n", ret);
+ goto err_release_vb2_queue;
+ }
+
+ /* Initialize the video_device struct */
+ isp_vdev->vdev.entity.name = vdev_name;
+ isp_vdev->vdev.entity.function = MEDIA_ENT_F_IO_V4L;
+ isp_vdev->vdev_pad.flags = MEDIA_PAD_FL_SINK;
+ ret = media_entity_pads_init(&isp_vdev->vdev.entity, 1,
+ &isp_vdev->vdev_pad);
+
+ if (ret) {
+ dev_err(v4l2_dev->dev, "init media entity pad fail:%d\n", ret);
+ goto err_release_vb2_queue;
+ }
+
+ vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE |
+ V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
+ vdev->entity.ops = &isp4vid_vdev_ent_ops;
+ vdev->release = video_device_release_empty;
+ vdev->fops = &isp4vid_vdev_fops;
+ vdev->ioctl_ops = &isp4vid_vdev_ioctl_ops;
+ vdev->lock = NULL;
+ vdev->queue = q;
+ vdev->v4l2_dev = v4l2_dev;
+ vdev->vfl_dir = VFL_DIR_RX;
+ strscpy(vdev->name, vdev_name, sizeof(vdev->name));
+ video_set_drvdata(vdev, isp_vdev);
+
+ ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
+ if (ret) {
+ dev_err(v4l2_dev->dev, "register video device fail:%d\n", ret);
+ goto err_entity_cleanup;
+ }
+
+ return 0;
+
+err_entity_cleanup:
+ media_entity_cleanup(&isp_vdev->vdev.entity);
+err_release_vb2_queue:
+ vb2_queue_release(q);
+ return ret;
+}
+
+void isp4vid_dev_deinit(struct isp4vid_dev *isp_vdev)
+{
+ vb2_video_unregister_device(&isp_vdev->vdev);
+}
diff --git a/drivers/media/platform/amd/isp4/isp4_video.h b/drivers/media/platform/amd/isp4/isp4_video.h
new file mode 100644
index 000000000000..b87316d2a2e5
--- /dev/null
+++ b/drivers/media/platform/amd/isp4/isp4_video.h
@@ -0,0 +1,65 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (C) 2025 Advanced Micro Devices, Inc.
+ */
+
+#ifndef _ISP4_VIDEO_H_
+#define _ISP4_VIDEO_H_
+
+#include <media/videobuf2-memops.h>
+#include <media/v4l2-dev.h>
+
+#include "isp4_interface.h"
+
+struct isp4vid_capture_buffer {
+ /*
+ * struct vb2_v4l2_buffer must be the first element
+ * the videobuf2 framework will allocate this struct based on
+ * buf_struct_size and use the first sizeof(struct vb2_buffer) bytes of
+ * memory as a vb2_buffer
+ */
+ struct vb2_v4l2_buffer vb2;
+ struct isp4if_img_buf_info img_buf;
+ struct list_head list;
+};
+
+struct isp4vid_ops {
+ int (*send_buffer)(struct v4l2_subdev *sd,
+ struct isp4if_img_buf_info *img_buf);
+};
+
+struct isp4vid_dev {
+ struct video_device vdev;
+ struct media_pad vdev_pad;
+ struct v4l2_pix_format format;
+
+ /* mutex that protects vbq */
+ struct mutex vbq_lock;
+ struct vb2_queue vbq;
+
+ /* mutex that protects buf_list */
+ struct mutex buf_list_lock;
+ struct list_head buf_list;
+
+ u32 sequence;
+ bool stream_started;
+
+ struct media_pipeline pipe;
+ struct device *dev;
+ struct v4l2_subdev *isp_sdev;
+ struct v4l2_fract timeperframe;
+
+ /* Callback operations */
+ const struct isp4vid_ops *ops;
+};
+
+int isp4vid_dev_init(struct isp4vid_dev *isp_vdev,
+ struct v4l2_subdev *isp_sdev,
+ const struct isp4vid_ops *ops);
+
+void isp4vid_dev_deinit(struct isp4vid_dev *isp_vdev);
+
+void isp4vid_handle_frame_done(struct isp4vid_dev *isp_vdev,
+ const struct isp4if_img_buf_info *img_buf);
+
+#endif /* _ISP4_VIDEO_H_ */
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread* Re: [PATCH v7 5/7] media: platform: amd: isp4 video node and buffers handling added
2025-12-16 9:13 ` [PATCH v7 5/7] media: platform: amd: isp4 video node and buffers " Bin Du
@ 2025-12-22 17:07 ` Sakari Ailus
2026-01-09 10:08 ` Du, Bin
0 siblings, 1 reply; 46+ messages in thread
From: Sakari Ailus @ 2025-12-22 17:07 UTC (permalink / raw)
To: Bin Du
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, Svetoslav Stoilov, Alexey Zagorodnikov
Hi Bin,
On Tue, Dec 16, 2025 at 05:13:24PM +0800, Bin Du wrote:
> Isp video implements v4l2 video interface and supports NV12 and YUYV. It
> manages buffers, pipeline power and state. Cherry-picked Sultan's DMA
> buffer related fix from branch v6.16-drm-tip-isp4-for-amd on
> https://github.com/kerneltoast/kernel_x86_laptop.git
>
> Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
> Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
> Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
> Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
> Signed-off-by: Bin Du <Bin.Du@amd.com>
> Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
> ---
> MAINTAINERS | 2 +
> drivers/media/platform/amd/isp4/Makefile | 3 +-
> drivers/media/platform/amd/isp4/isp4.c | 10 +
> drivers/media/platform/amd/isp4/isp4_subdev.c | 79 +-
> drivers/media/platform/amd/isp4/isp4_subdev.h | 2 +
> drivers/media/platform/amd/isp4/isp4_video.c | 1165 +++++++++++++++++
> drivers/media/platform/amd/isp4/isp4_video.h | 65 +
> 7 files changed, 1321 insertions(+), 5 deletions(-)
> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c
> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 48ffc8bbdcee..80c966fde0b4 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1151,6 +1151,8 @@ F: drivers/media/platform/amd/isp4/isp4_interface.c
> F: drivers/media/platform/amd/isp4/isp4_interface.h
> F: drivers/media/platform/amd/isp4/isp4_subdev.c
> F: drivers/media/platform/amd/isp4/isp4_subdev.h
> +F: drivers/media/platform/amd/isp4/isp4_video.c
> +F: drivers/media/platform/amd/isp4/isp4_video.h
>
> AMD KFD
> M: Felix Kuehling <Felix.Kuehling@amd.com>
> diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
> index 6d4e6d6ac7f5..398c20ea7866 100644
> --- a/drivers/media/platform/amd/isp4/Makefile
> +++ b/drivers/media/platform/amd/isp4/Makefile
> @@ -5,4 +5,5 @@
> obj-$(CONFIG_AMD_ISP4) += amd_capture.o
> amd_capture-objs := isp4.o \
> isp4_interface.o \
> - isp4_subdev.o
> + isp4_subdev.o \
> + isp4_video.o
> diff --git a/drivers/media/platform/amd/isp4/isp4.c b/drivers/media/platform/amd/isp4/isp4.c
> index bcd7cad32afd..b5470f6dc93e 100644
> --- a/drivers/media/platform/amd/isp4/isp4.c
> +++ b/drivers/media/platform/amd/isp4/isp4.c
> @@ -169,6 +169,16 @@ static int isp4_capture_probe(struct platform_device *pdev)
> goto err_pm_disable;
> }
>
> + ret = media_create_pad_link(&isp_dev->isp_subdev.sdev.entity,
> + 0, &isp_dev->isp_subdev.isp_vdev.vdev.entity,
> + 0,
> + MEDIA_LNK_FL_ENABLED |
> + MEDIA_LNK_FL_IMMUTABLE);
> + if (ret) {
> + dev_err_probe(dev, ret, "fail to create pad link\n");
> + goto err_isp4_deinit;
> + }
> +
> ret = media_device_register(&isp_dev->mdev);
> if (ret) {
> dev_err_probe(dev, ret, "fail to register media device\n");
> diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.c b/drivers/media/platform/amd/isp4/isp4_subdev.c
> index 3a25d1fc49ce..8c33187efc54 100644
> --- a/drivers/media/platform/amd/isp4/isp4_subdev.c
> +++ b/drivers/media/platform/amd/isp4/isp4_subdev.c
> @@ -449,7 +449,7 @@ static void isp4sd_fw_resp_frame_done(struct isp4_subdev *isp_subdev,
> return;
> }
>
> - dev_dbg(dev, "ts:%llu,streamId:%d,poc:%u,preview_en:%u(%i)\n",
> + dev_dbg(dev, "ts:%llu,streamId:%d,poc:%u,preview_en:%u,(%i)\n",
Please make the change to the patch adding the line.
> ktime_get_ns(), stream_id, meta->poc,
> meta->preview.enabled,
> meta->preview.status);
> @@ -459,11 +459,13 @@ static void isp4sd_fw_resp_frame_done(struct isp4_subdev *isp_subdev,
> meta->preview.status == BUFFER_STATUS_DONE ||
> meta->preview.status == BUFFER_STATUS_DIRTY)) {
> prev = isp4if_dequeue_buffer(ispif);
> - if (prev)
> + if (prev) {
> + isp4vid_handle_frame_done(&isp_subdev->isp_vdev,
> + &prev->buf_info);
> isp4if_dealloc_buffer_node(prev);
> - else
> + } else {
> dev_err(dev, "fail null prev buf\n");
> -
> + }
> } else if (meta->preview.enabled) {
> dev_err(dev, "fail bad preview status %u\n",
> meta->preview.status);
> @@ -794,6 +796,65 @@ static int isp4sd_start_stream(struct isp4_subdev *isp_subdev,
> return ret;
> }
>
> +static int isp4sd_ioc_send_img_buf(struct v4l2_subdev *sd,
> + struct isp4if_img_buf_info *buf_info)
> +{
> + struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
> + struct isp4_interface *ispif = &isp_subdev->ispif;
> + struct isp4if_img_buf_node *buf_node;
> + struct device *dev = isp_subdev->dev;
> + int ret;
> +
> + guard(mutex)(&isp_subdev->ops_mutex);
> +
> + if (ispif->status != ISP4IF_STATUS_FW_RUNNING) {
> + dev_err(dev, "fail send img buf for bad fsm %d\n",
> + ispif->status);
> + return -EINVAL;
> + }
> +
> + buf_node = isp4if_alloc_buffer_node(buf_info);
> + if (!buf_node) {
> + dev_err(dev, "fail alloc sys img buf info node\n");
> + return -ENOMEM;
> + }
> +
> + ret = isp4if_queue_buffer(ispif, buf_node);
> + if (ret) {
> + dev_err(dev, "fail to queue image buf, %d\n", ret);
> + goto error_release_buf_node;
> + }
> +
> + if (!isp_subdev->sensor_info.start_stream_cmd_sent) {
> + isp_subdev->sensor_info.buf_sent_cnt++;
> +
> + if (isp_subdev->sensor_info.buf_sent_cnt >=
> + ISP4SD_MIN_BUF_CNT_BEF_START_STREAM) {
> + ret = isp4if_send_command(ispif, CMD_ID_START_STREAM,
> + NULL, 0);
> + if (ret) {
> + dev_err(dev, "fail to START_STREAM");
> + goto error_release_buf_node;
> + }
> + isp_subdev->sensor_info.start_stream_cmd_sent = true;
> + isp_subdev->sensor_info.output_info.start_status =
> + ISP4SD_START_STATUS_STARTED;
> + isp_subdev->sensor_info.status =
> + ISP4SD_START_STATUS_STARTED;
> + } else {
> + dev_dbg(dev, "no send start, required %u, buf sent %u\n",
> + ISP4SD_MIN_BUF_CNT_BEF_START_STREAM,
> + isp_subdev->sensor_info.buf_sent_cnt);
> + }
> + }
> +
> + return 0;
> +
> +error_release_buf_node:
> + isp4if_dealloc_buffer_node(buf_node);
> + return ret;
> +}
> +
> static int isp4sd_set_power(struct v4l2_subdev *sd, int on)
> {
> struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
> @@ -892,6 +953,10 @@ static const struct media_entity_operations isp4sd_sdev_ent_ops = {
> .link_validate = isp4sd_sdev_link_validate,
> };
>
> +static const struct isp4vid_ops isp4sd_isp4vid_ops = {
> + .send_buffer = isp4sd_ioc_send_img_buf,
> +};
> +
> int isp4sd_init(struct isp4_subdev *isp_subdev, struct v4l2_device *v4l2_dev,
> int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM])
> {
> @@ -952,6 +1017,11 @@ int isp4sd_init(struct isp4_subdev *isp_subdev, struct v4l2_device *v4l2_dev,
> isp_subdev->host2fw_seq_num = 1;
> ispif->status = ISP4IF_STATUS_PWR_OFF;
>
> + ret = isp4vid_dev_init(&isp_subdev->isp_vdev, &isp_subdev->sdev,
> + &isp4sd_isp4vid_ops);
> + if (ret)
> + goto err_subdev_unreg;
> +
> return 0;
>
> err_subdev_unreg:
> @@ -966,6 +1036,7 @@ void isp4sd_deinit(struct isp4_subdev *isp_subdev)
> {
> struct isp4_interface *ispif = &isp_subdev->ispif;
>
> + isp4vid_dev_deinit(&isp_subdev->isp_vdev);
> v4l2_device_unregister_subdev(&isp_subdev->sdev);
> media_entity_cleanup(&isp_subdev->sdev.entity);
> isp4if_deinit(ispif);
> diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.h b/drivers/media/platform/amd/isp4/isp4_subdev.h
> index 70007b9de3f3..386e4e68f3fa 100644
> --- a/drivers/media/platform/amd/isp4/isp4_subdev.h
> +++ b/drivers/media/platform/amd/isp4/isp4_subdev.h
> @@ -17,6 +17,7 @@
> #include "isp4_fw_cmd_resp.h"
> #include "isp4_hw_reg.h"
> #include "isp4_interface.h"
> +#include "isp4_video.h"
>
> /*
> * one is for none sesnor specefic response which is not used now
> @@ -93,6 +94,7 @@ struct isp4_subdev_thread_param {
> struct isp4_subdev {
> struct v4l2_subdev sdev;
> struct isp4_interface ispif;
> + struct isp4vid_dev isp_vdev;
>
> struct media_pad sdev_pad;
>
> diff --git a/drivers/media/platform/amd/isp4/isp4_video.c b/drivers/media/platform/amd/isp4/isp4_video.c
> new file mode 100644
> index 000000000000..5006e5021155
> --- /dev/null
> +++ b/drivers/media/platform/amd/isp4/isp4_video.c
> @@ -0,0 +1,1165 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
> + */
> +
> +#include <linux/vmalloc.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/v4l2-mc.h>
> +
> +#include "isp4_interface.h"
> +#include "isp4_subdev.h"
> +#include "isp4_video.h"
> +
> +#define ISP4VID_ISP_DRV_NAME "amd_isp_capture"
> +#define ISP4VID_MAX_PREVIEW_FPS 30
> +#define ISP4VID_DEFAULT_FMT isp4vid_formats[0]
> +
> +#define ISP4VID_PAD_VIDEO_OUTPUT 0
> +
> +/* timeperframe default */
> +#define ISP4VID_ISP_TPF_DEFAULT isp4vid_tpfs[0]
> +
> +/* amdisp buffer for vb2 operations */
> +struct isp4vid_vb2_buf {
> + struct device *dev;
> + void *vaddr;
> + unsigned long size;
> + refcount_t refcount;
> + struct dma_buf *dbuf;
> + void *bo;
> + u64 gpu_addr;
> + struct vb2_vmarea_handler handler;
> +};
> +
> +static void isp4vid_vb2_put(void *buf_priv);
> +
> +static const char *const isp4vid_video_dev_name = "Preview";
> +
> +/* Sizes must be in increasing order */
> +static const struct v4l2_frmsize_discrete isp4vid_frmsize[] = {
> + {640, 360},
> + {640, 480},
> + {1280, 720},
> + {1280, 960},
> + {1920, 1080},
> + {1920, 1440},
> + {2560, 1440},
> + {2880, 1620},
> + {2880, 1624},
> + {2888, 1808},
> +};
> +
> +static const u32 isp4vid_formats[] = {
> + V4L2_PIX_FMT_NV12,
> + V4L2_PIX_FMT_YUYV
> +};
> +
> +/* timeperframe list */
> +static const struct v4l2_fract isp4vid_tpfs[] = {
> + { 1, ISP4VID_MAX_PREVIEW_FPS }
> +};
> +
> +void isp4vid_handle_frame_done(struct isp4vid_dev *isp_vdev,
> + const struct isp4if_img_buf_info *img_buf)
> +{
> + struct isp4vid_capture_buffer *isp4vid_buf;
> + void *vbuf;
> +
> + scoped_guard(mutex, &isp_vdev->buf_list_lock) {
> + /* Get the first entry of the list */
> + isp4vid_buf = list_first_entry_or_null(&isp_vdev->buf_list, typeof(*isp4vid_buf),
> + list);
> + if (!isp4vid_buf)
> + return;
> +
> + vbuf = vb2_plane_vaddr(&isp4vid_buf->vb2.vb2_buf, 0);
> +
> + if (vbuf != img_buf->planes[0].sys_addr) {
> + dev_err(isp_vdev->dev, "Invalid vbuf");
> + return;
> + }
> +
> + /* Remove this entry from the list */
> + list_del(&isp4vid_buf->list);
> + }
> +
> + /* Fill the buffer */
> + isp4vid_buf->vb2.vb2_buf.timestamp = ktime_get_ns();
> + isp4vid_buf->vb2.sequence = isp_vdev->sequence++;
> + isp4vid_buf->vb2.field = V4L2_FIELD_ANY;
> +
> + /* at most 2 planes */
> + vb2_set_plane_payload(&isp4vid_buf->vb2.vb2_buf,
> + 0, isp_vdev->format.sizeimage);
> +
> + vb2_buffer_done(&isp4vid_buf->vb2.vb2_buf, VB2_BUF_STATE_DONE);
> +
> + dev_dbg(isp_vdev->dev, "call vb2_buffer_done(size=%u)\n",
> + isp_vdev->format.sizeimage);
> +}
> +
> +static unsigned int isp4vid_vb2_num_users(void *buf_priv)
> +{
> + struct isp4vid_vb2_buf *buf = buf_priv;
> +
> + return refcount_read(&buf->refcount);
> +}
> +
> +static int isp4vid_vb2_mmap(void *buf_priv, struct vm_area_struct *vma)
> +{
> + struct isp4vid_vb2_buf *buf = buf_priv;
> + int ret;
> +
> + if (!buf) {
> + pr_err("fail no memory to map\n");
> + return -EINVAL;
> + }
> +
> + ret = remap_vmalloc_range(vma, buf->vaddr, 0);
> + if (ret) {
> + dev_err(buf->dev, "fail remap vmalloc mem, %d\n", ret);
> + return ret;
> + }
> +
> + /*
> + * Make sure that vm_areas for 2 buffers won't be merged together
> + */
> + vm_flags_set(vma, VM_DONTEXPAND);
> +
> + /*
> + * Use common vm_area operations to track buffer refcount.
> + */
> + vma->vm_private_data = &buf->handler;
> + vma->vm_ops = &vb2_common_vm_ops;
> +
> + vma->vm_ops->open(vma);
> +
> + dev_dbg(buf->dev, "mmap isp user bo 0x%llx size %ld refcount %d\n",
> + buf->gpu_addr, buf->size, refcount_read(&buf->refcount));
> +
> + return 0;
> +}
> +
> +static void *isp4vid_vb2_vaddr(struct vb2_buffer *vb, void *buf_priv)
> +{
> + struct isp4vid_vb2_buf *buf = buf_priv;
> +
> + if (!buf->vaddr) {
> + dev_err(buf->dev,
> + "fail for buf vaddr is null\n");
Fits on the prefious line.
Is this a driver bug or something that can be triggered by the user?
> + return NULL;
> + }
> +
> + return buf->vaddr;
> +}
> +
> +static void isp4vid_vb2_detach_dmabuf(void *mem_priv)
> +{
> + struct isp4vid_vb2_buf *buf = mem_priv;
> +
> + dev_dbg(buf->dev, "detach dmabuf of isp user bo 0x%llx size %ld",
> + buf->gpu_addr, buf->size);
> +
> + kfree(buf);
> +}
> +
> +static void *isp4vid_vb2_attach_dmabuf(struct vb2_buffer *vb, struct device *dev,
> + struct dma_buf *dbuf,
> + unsigned long size)
> +{
> + struct isp4vid_vb2_buf *dbg_buf = dbuf->priv;
> + struct isp4vid_vb2_buf *buf;
> +
> + if (dbuf->size < size) {
> + dev_err(dev, "Invalid dmabuf size %zu %lu", dbuf->size, size);
> + return ERR_PTR(-EFAULT);
> + }
> +
> + buf = kzalloc(sizeof(*buf), GFP_KERNEL);
> + if (!buf)
> + return ERR_PTR(-ENOMEM);
> +
> + buf->dev = dev;
> + buf->dbuf = dbuf;
> + buf->size = size;
> +
> + dev_dbg(dev, "attach dmabuf of isp user bo 0x%llx size %ld",
> + dbg_buf->gpu_addr, dbg_buf->size);
> +
> + return buf;
> +}
> +
> +static void isp4vid_vb2_unmap_dmabuf(void *mem_priv)
> +{
> + struct isp4vid_vb2_buf *buf = mem_priv;
> + struct iosys_map map = IOSYS_MAP_INIT_VADDR(buf->vaddr);
> +
> + dev_dbg(buf->dev, "unmap dmabuf of isp user bo 0x%llx size %ld",
> + buf->gpu_addr, buf->size);
> +
> + dma_buf_vunmap_unlocked(buf->dbuf, &map);
> + buf->vaddr = NULL;
> +}
> +
> +static int isp4vid_vb2_map_dmabuf(void *mem_priv)
> +{
> + struct isp4vid_vb2_buf *buf = mem_priv, *mmap_buf;
> + struct iosys_map map;
> + int ret;
> +
> + ret = dma_buf_vmap_unlocked(buf->dbuf, &map);
> + if (ret) {
> + dev_err(buf->dev, "vmap_unlocked fail");
> + return -EFAULT;
> + }
> +
> + buf->vaddr = map.vaddr;
> + mmap_buf = buf->dbuf->priv;
> + buf->gpu_addr = mmap_buf->gpu_addr;
> +
> + dev_dbg(buf->dev, "map dmabuf of isp user bo 0x%llx size %ld",
> + buf->gpu_addr, buf->size);
> +
> + return 0;
> +}
> +
> +#ifdef CONFIG_HAS_DMA
> +struct isp4vid_vb2_amdgpu_attachment {
> + struct sg_table sgt;
> + enum dma_data_direction dma_dir;
> +};
> +
> +static int isp4vid_dmabuf_ops_attach(struct dma_buf *dbuf,
> + struct dma_buf_attachment *dbuf_attach)
> +{
> + struct isp4vid_vb2_buf *buf = dbuf->priv;
> + int num_pages = PAGE_ALIGN(buf->size) / PAGE_SIZE;
> + struct isp4vid_vb2_amdgpu_attachment *attach;
> + void *vaddr = buf->vaddr;
> + struct scatterlist *sg;
> + struct sg_table *sgt;
> + int ret;
> + int i;
> +
> + attach = kzalloc(sizeof(*attach), GFP_KERNEL);
> + if (!attach)
> + return -ENOMEM;
> +
> + sgt = &attach->sgt;
> + ret = sg_alloc_table(sgt, num_pages, GFP_KERNEL);
> + if (ret) {
> + kfree(attach);
> + return ret;
> + }
> +
> + for_each_sgtable_sg(sgt, sg, i) {
> + struct page *page = vmalloc_to_page(vaddr);
> +
> + if (!page) {
> + sg_free_table(sgt);
> + kfree(attach);
> + return -ENOMEM;
> + }
> +
> + sg_set_page(sg, page, PAGE_SIZE, 0);
> + vaddr = ((char *)vaddr) + PAGE_SIZE;
> + }
> +
> + attach->dma_dir = DMA_NONE;
> + dbuf_attach->priv = attach;
> +
> + return 0;
> +}
> +
> +static void isp4vid_dmabuf_ops_detach(struct dma_buf *dbuf,
> + struct dma_buf_attachment *dbuf_attach)
> +{
> + struct isp4vid_vb2_amdgpu_attachment *attach = dbuf_attach->priv;
> + struct sg_table *sgt;
> +
> + if (!attach) {
> + pr_err("fail invalid attach handler\n");
> + return;
> + }
> +
> + sgt = &attach->sgt;
> +
> + /* release the scatterlist cache */
> + if (attach->dma_dir != DMA_NONE)
> + dma_unmap_sgtable(dbuf_attach->dev, sgt, attach->dma_dir, 0);
> +
> + sg_free_table(sgt);
> + kfree(attach);
> + dbuf_attach->priv = NULL;
> +}
> +
> +static struct sg_table *isp4vid_dmabuf_ops_map(struct dma_buf_attachment *dbuf_attach,
> + enum dma_data_direction dma_dir)
> +{
> + struct isp4vid_vb2_amdgpu_attachment *attach = dbuf_attach->priv;
> + struct sg_table *sgt;
> +
> + sgt = &attach->sgt;
> + /* return previously mapped sg table */
> + if (attach->dma_dir == dma_dir)
> + return sgt;
> +
> + /* release any previous cache */
> + if (attach->dma_dir != DMA_NONE) {
> + dma_unmap_sgtable(dbuf_attach->dev, sgt, attach->dma_dir, 0);
> + attach->dma_dir = DMA_NONE;
> + }
> +
> + /* mapping to the client with new direction */
> + if (dma_map_sgtable(dbuf_attach->dev, sgt, dma_dir, 0)) {
> + dev_err(dbuf_attach->dev, "fail to map scatterlist");
> + return ERR_PTR(-EIO);
> + }
> +
> + attach->dma_dir = dma_dir;
> +
> + return sgt;
> +}
> +
> +static void isp4vid_dmabuf_ops_unmap(struct dma_buf_attachment *dbuf_attach,
> + struct sg_table *sgt,
> + enum dma_data_direction dma_dir)
> +{
> + /* nothing to be done here */
> +}
> +
> +static int isp4vid_dmabuf_ops_vmap(struct dma_buf *dbuf,
> + struct iosys_map *map)
> +{
> + struct isp4vid_vb2_buf *buf = dbuf->priv;
> +
> + iosys_map_set_vaddr(map, buf->vaddr);
> +
> + return 0;
> +}
> +
> +static void isp4vid_dmabuf_ops_release(struct dma_buf *dbuf)
> +{
> + struct isp4vid_vb2_buf *buf = dbuf->priv;
> +
> + /* drop reference obtained in isp4vid_vb2_get_dmabuf */
> + if (dbuf != buf->dbuf)
> + isp4vid_vb2_put(buf);
> + else
> + kfree(buf);
> +}
> +
> +static int isp4vid_dmabuf_ops_mmap(struct dma_buf *dbuf, struct vm_area_struct *vma)
> +{
> + return isp4vid_vb2_mmap(dbuf->priv, vma);
> +}
> +
> +static const struct dma_buf_ops isp4vid_dmabuf_ops = {
> + .attach = isp4vid_dmabuf_ops_attach,
> + .detach = isp4vid_dmabuf_ops_detach,
> + .map_dma_buf = isp4vid_dmabuf_ops_map,
> + .unmap_dma_buf = isp4vid_dmabuf_ops_unmap,
> + .vmap = isp4vid_dmabuf_ops_vmap,
> + .mmap = isp4vid_dmabuf_ops_mmap,
> + .release = isp4vid_dmabuf_ops_release,
> +};
> +
> +static struct dma_buf *isp4vid_get_dmabuf(struct isp4vid_vb2_buf *buf, unsigned long flags)
> +{
> + DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
> + struct dma_buf *dbuf;
> +
> + if (WARN_ON(!buf->vaddr))
> + return NULL;
> +
> + exp_info.ops = &isp4vid_dmabuf_ops;
> + exp_info.size = buf->size;
> + exp_info.flags = flags;
> + exp_info.priv = buf;
> +
> + dbuf = dma_buf_export(&exp_info);
> + if (IS_ERR(dbuf))
> + return NULL;
> +
> + return dbuf;
> +}
> +
> +static struct dma_buf *isp4vid_vb2_get_dmabuf(struct vb2_buffer *vb, void *buf_priv,
> + unsigned long flags)
> +{
> + struct isp4vid_vb2_buf *buf = buf_priv;
> + struct dma_buf *dbuf;
> +
> + dbuf = isp4vid_get_dmabuf(buf, flags);
> + if (!dbuf) {
> + dev_err(buf->dev, "fail to get isp dma buffer\n");
> + return NULL;
> + }
> +
> + refcount_inc(&buf->refcount);
> +
> + dev_dbg(buf->dev, "buf exported, refcount %d\n",
> + refcount_read(&buf->refcount));
> +
> + return dbuf;
> +}
> +#endif /* CONFIG_HAS_DMA */
> +
> +static void isp4vid_vb2_put(void *buf_priv)
> +{
> + struct isp4vid_vb2_buf *buf = buf_priv;
> +
> + dev_dbg(buf->dev,
> + "release isp user bo 0x%llx size %ld refcount %d",
> + buf->gpu_addr, buf->size,
> + refcount_read(&buf->refcount));
> +
> + if (refcount_dec_and_test(&buf->refcount)) {
> + isp_user_buffer_free(buf->bo);
> + vfree(buf->vaddr);
> + /*
> + * Putting the implicit dmabuf frees `buf`. Freeing `buf` must
> + * be done from the dmabuf release callback since dma_buf_put()
> + * isn't always synchronous; it's just an fput(), which may be
> + * deferred. Since the dmabuf release callback needs to access
> + * `buf`, this means `buf` cannot be freed until then.
> + */
> + dma_buf_put(buf->dbuf);
> + }
> +}
> +
> +static void *isp4vid_vb2_alloc(struct vb2_buffer *vb, struct device *dev, unsigned long size)
> +{
> + struct isp4vid_vb2_buf *buf;
> + u64 gpu_addr;
> + void *bo;
> + int ret;
> +
> + buf = kzalloc(sizeof(*buf), GFP_KERNEL | vb->vb2_queue->gfp_flags);
> + if (!buf)
> + return ERR_PTR(-ENOMEM);
> +
> + buf->dev = dev;
> + buf->size = size;
> + buf->vaddr = vmalloc_user(buf->size);
> + if (!buf->vaddr) {
> + dev_err(dev, "fail to vmalloc buffer\n");
> + goto free_buf;
> + }
> +
> + buf->handler.refcount = &buf->refcount;
> + buf->handler.put = isp4vid_vb2_put;
> + buf->handler.arg = buf;
> +
> + /* get implicit dmabuf */
> + buf->dbuf = isp4vid_get_dmabuf(buf, 0);
> + if (!buf->dbuf) {
> + dev_err(dev, "fail to get implicit dmabuf\n");
> + goto free_user_vmem;
> + }
> +
> + /* create isp user BO and obtain gpu_addr */
> + ret = isp_user_buffer_alloc(dev, buf->dbuf, &bo, &gpu_addr);
> + if (ret) {
> + dev_err(dev, "fail to create isp user BO\n");
> + goto put_dmabuf;
> + }
> +
> + buf->bo = bo;
> + buf->gpu_addr = gpu_addr;
> +
> + refcount_set(&buf->refcount, 1);
> +
> + dev_dbg(dev, "allocated isp user bo 0x%llx size %ld refcount %d\n",
> + buf->gpu_addr, buf->size, refcount_read(&buf->refcount));
> +
> + return buf;
> +
> +put_dmabuf:
> + dma_buf_put(buf->dbuf);
> +free_user_vmem:
> + vfree(buf->vaddr);
> +free_buf:
> + ret = buf->vaddr ? -EINVAL : -ENOMEM;
> + kfree(buf);
> + return ERR_PTR(ret);
> +}
> +
> +static const struct vb2_mem_ops isp4vid_vb2_memops = {
> + .alloc = isp4vid_vb2_alloc,
> + .put = isp4vid_vb2_put,
> +#ifdef CONFIG_HAS_DMA
> + .get_dmabuf = isp4vid_vb2_get_dmabuf,
> +#endif
> + .map_dmabuf = isp4vid_vb2_map_dmabuf,
> + .unmap_dmabuf = isp4vid_vb2_unmap_dmabuf,
> + .attach_dmabuf = isp4vid_vb2_attach_dmabuf,
> + .detach_dmabuf = isp4vid_vb2_detach_dmabuf,
> + .vaddr = isp4vid_vb2_vaddr,
> + .mmap = isp4vid_vb2_mmap,
> + .num_users = isp4vid_vb2_num_users,
> +};
Could you elaborate a bit why do you need your own videobuf mem ops?
> +
> +static const struct v4l2_pix_format isp4vid_fmt_default = {
> + .width = 1920,
> + .height = 1080,
> + .pixelformat = ISP4VID_DEFAULT_FMT,
> + .field = V4L2_FIELD_NONE,
> + .colorspace = V4L2_COLORSPACE_SRGB,
> +};
> +
> +static void isp4vid_capture_return_all_buffers(struct isp4vid_dev *isp_vdev,
> + enum vb2_buffer_state state)
> +{
> + struct isp4vid_capture_buffer *vbuf, *node;
> +
> + scoped_guard(mutex, &isp_vdev->buf_list_lock) {
> + list_for_each_entry_safe(vbuf, node, &isp_vdev->buf_list, list)
> + vb2_buffer_done(&vbuf->vb2.vb2_buf, state);
> + INIT_LIST_HEAD(&isp_vdev->buf_list);
> + }
> +
> + dev_dbg(isp_vdev->dev, "call vb2_buffer_done(%d)\n", state);
> +}
> +
> +static int isp4vid_vdev_link_validate(struct media_link *link)
> +{
> + return 0;
> +}
> +
> +static const struct media_entity_operations isp4vid_vdev_ent_ops = {
> + .link_validate = isp4vid_vdev_link_validate,
> +};
> +
> +static const struct v4l2_file_operations isp4vid_vdev_fops = {
> + .owner = THIS_MODULE,
> + .open = v4l2_fh_open,
> + .release = vb2_fop_release,
> + .read = vb2_fop_read,
> + .poll = vb2_fop_poll,
> + .unlocked_ioctl = video_ioctl2,
> + .mmap = vb2_fop_mmap,
> +};
> +
> +static int isp4vid_ioctl_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
> +{
> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
> +
> + strscpy(cap->driver, ISP4VID_ISP_DRV_NAME, sizeof(cap->driver));
> + snprintf(cap->card, sizeof(cap->card), "%s", ISP4VID_ISP_DRV_NAME);
> + cap->capabilities |= V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_CAPTURE;
> +
> + dev_dbg(isp_vdev->dev, "%s|capabilities=0x%X", isp_vdev->vdev.name, cap->capabilities);
> +
> + return 0;
> +}
> +
> +static int isp4vid_g_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f)
> +{
> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
> +
> + f->fmt.pix = isp_vdev->format;
> +
> + return 0;
> +}
> +
> +static int isp4vid_fill_buffer_size(struct v4l2_pix_format *fmt)
> +{
> + int ret = 0;
> +
> + switch (fmt->pixelformat) {
> + case V4L2_PIX_FMT_NV12:
> + fmt->bytesperline = fmt->width;
> + fmt->sizeimage = fmt->bytesperline * fmt->height * 3 / 2;
> + break;
> + case V4L2_PIX_FMT_YUYV:
> + fmt->bytesperline = fmt->width * 2;
> + fmt->sizeimage = fmt->bytesperline * fmt->height;
> + break;
> + default:
> + ret = -EINVAL;
> + break;
> + }
> +
> + return ret;
> +}
> +
> +static int isp4vid_try_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f)
> +{
> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
> + struct v4l2_pix_format *format = &f->fmt.pix;
> + const struct v4l2_frmsize_discrete *fsz;
> + int i;
> +
> + if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> + return -EINVAL;
The framework already does the check.
> +
> + /*
> + * Check if the hardware supports the requested format, use the default
> + * format otherwise.
> + */
> + for (i = 0; i < ARRAY_SIZE(isp4vid_formats); i++)
> + if (isp4vid_formats[i] == format->pixelformat)
> + break;
> +
> + if (i == ARRAY_SIZE(isp4vid_formats))
> + format->pixelformat = ISP4VID_DEFAULT_FMT;
> +
> + switch (format->pixelformat) {
> + case V4L2_PIX_FMT_NV12:
> + case V4L2_PIX_FMT_YUYV:
> + fsz = v4l2_find_nearest_size(isp4vid_frmsize, ARRAY_SIZE(isp4vid_frmsize),
> + width, height, format->width, format->height);
> + format->width = fsz->width;
> + format->height = fsz->height;
> + break;
> + default:
> + dev_err(isp_vdev->dev, "%s|unsupported fmt=%u", isp_vdev->vdev.name,
> + format->pixelformat);
> + return -EINVAL;
> + }
> +
> + /* There is no need to check the return value, as failure will never happen here */
> + isp4vid_fill_buffer_size(format);
> +
> + if (format->field == V4L2_FIELD_ANY)
> + format->field = isp4vid_fmt_default.field;
> +
> + if (format->colorspace == V4L2_COLORSPACE_DEFAULT)
> + format->colorspace = isp4vid_fmt_default.colorspace;
> +
> + return 0;
> +}
> +
> +static int isp4vid_set_fmt_2_isp(struct v4l2_subdev *sdev, struct v4l2_pix_format *pix_fmt)
> +{
> + struct v4l2_subdev_format fmt = {};
> +
> + switch (pix_fmt->pixelformat) {
> + case V4L2_PIX_FMT_NV12:
> + fmt.format.code = MEDIA_BUS_FMT_YUYV8_1_5X8;
> + break;
> + case V4L2_PIX_FMT_YUYV:
> + fmt.format.code = MEDIA_BUS_FMT_YUYV8_1X16;
> + break;
> + default:
> + return -EINVAL;
> + }
> + fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
> + fmt.pad = ISP4VID_PAD_VIDEO_OUTPUT;
> + fmt.format.width = pix_fmt->width;
> + fmt.format.height = pix_fmt->height;
> + return v4l2_subdev_call(sdev, pad, set_fmt, NULL, &fmt);
> +}
> +
> +static int isp4vid_s_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f)
> +{
> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
> + int ret;
> +
> + /* Do not change the format while stream is on */
> + if (vb2_is_busy(&isp_vdev->vbq))
> + return -EBUSY;
> +
> + if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> + return -EINVAL;
Ditto.
> +
> + ret = isp4vid_try_fmt_vid_cap(file, priv, f);
> + if (ret)
> + return ret;
> +
> + dev_dbg(isp_vdev->dev, "%s|width height:%ux%u->%ux%u",
> + isp_vdev->vdev.name,
> + isp_vdev->format.width, isp_vdev->format.height,
> + f->fmt.pix.width, f->fmt.pix.height);
> + dev_dbg(isp_vdev->dev, "%s|pixelformat:0x%x-0x%x",
> + isp_vdev->vdev.name, isp_vdev->format.pixelformat,
> + f->fmt.pix.pixelformat);
> + dev_dbg(isp_vdev->dev, "%s|bytesperline:%u->%u",
> + isp_vdev->vdev.name, isp_vdev->format.bytesperline,
> + f->fmt.pix.bytesperline);
> + dev_dbg(isp_vdev->dev, "%s|sizeimage:%u->%u",
> + isp_vdev->vdev.name, isp_vdev->format.sizeimage,
> + f->fmt.pix.sizeimage);
> +
> + isp_vdev->format = f->fmt.pix;
> + ret = isp4vid_set_fmt_2_isp(isp_vdev->isp_sdev, &isp_vdev->format);
> +
> + return ret;
> +}
> +
> +static int isp4vid_enum_fmt_vid_cap(struct file *file, void *priv, struct v4l2_fmtdesc *f)
> +{
> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
> +
> + switch (f->index) {
> + case 0:
> + f->pixelformat = V4L2_PIX_FMT_NV12;
> + break;
> + case 1:
> + f->pixelformat = V4L2_PIX_FMT_YUYV;
> + break;
> + default:
> + return -EINVAL;
> + }
> +
> + dev_dbg(isp_vdev->dev, "%s|index=%d, pixelformat=0x%X",
> + isp_vdev->vdev.name, f->index, f->pixelformat);
> +
> + return 0;
> +}
> +
> +static int isp4vid_enum_framesizes(struct file *file, void *fh, struct v4l2_frmsizeenum *fsize)
> +{
> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
> + unsigned int i;
> +
> + for (i = 0; i < ARRAY_SIZE(isp4vid_formats); i++) {
> + if (isp4vid_formats[i] == fsize->pixel_format)
> + break;
> + }
> + if (i == ARRAY_SIZE(isp4vid_formats))
> + return -EINVAL;
> +
> + if (fsize->index < ARRAY_SIZE(isp4vid_frmsize)) {
> + fsize->type = V4L2_FRMSIZE_TYPE_DISCRETE;
> + fsize->discrete = isp4vid_frmsize[fsize->index];
> + dev_dbg(isp_vdev->dev, "%s|size[%d]=%dx%d",
> + isp_vdev->vdev.name, fsize->index,
> + fsize->discrete.width, fsize->discrete.height);
> + } else {
> + return -EINVAL;
> + }
> +
> + return 0;
> +}
> +
> +static int isp4vid_ioctl_enum_frameintervals(struct file *file, void *priv,
> + struct v4l2_frmivalenum *fival)
> +{
> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
> + int i;
> +
> + if (fival->index >= ARRAY_SIZE(isp4vid_tpfs))
> + return -EINVAL;
> +
> + for (i = 0; i < ARRAY_SIZE(isp4vid_formats); i++)
> + if (isp4vid_formats[i] == fival->pixel_format)
> + break;
> +
> + if (i == ARRAY_SIZE(isp4vid_formats))
> + return -EINVAL;
> +
> + for (i = 0; i < ARRAY_SIZE(isp4vid_frmsize); i++)
> + if (isp4vid_frmsize[i].width == fival->width &&
> + isp4vid_frmsize[i].height == fival->height)
> + break;
> +
> + if (i == ARRAY_SIZE(isp4vid_frmsize))
> + return -EINVAL;
> +
> + fival->type = V4L2_FRMIVAL_TYPE_DISCRETE;
> + fival->discrete = isp4vid_tpfs[fival->index];
> + v4l2_simplify_fraction(&fival->discrete.numerator,
> + &fival->discrete.denominator, 8, 333);
> +
> + dev_dbg(isp_vdev->dev, "%s|interval[%d]=%d/%d",
> + isp_vdev->vdev.name, fival->index,
> + fival->discrete.numerator,
> + fival->discrete.denominator);
> +
> + return 0;
> +}
> +
> +static int isp4vid_ioctl_g_param(struct file *file, void *priv, struct v4l2_streamparm *param)
> +{
> + struct v4l2_captureparm *capture = ¶m->parm.capture;
> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
> +
> + if (param->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> + return -EINVAL;
> +
> + capture->capability = V4L2_CAP_TIMEPERFRAME;
> + capture->timeperframe = isp_vdev->timeperframe;
> + capture->readbuffers = 0;
> +
> + dev_dbg(isp_vdev->dev, "%s|timeperframe=%d/%d", isp_vdev->vdev.name,
> + capture->timeperframe.numerator,
> + capture->timeperframe.denominator);
> +
> + return 0;
> +}
> +
> +static const struct v4l2_ioctl_ops isp4vid_vdev_ioctl_ops = {
> + /* VIDIOC_QUERYCAP handler */
> + .vidioc_querycap = isp4vid_ioctl_querycap,
> +
> + /* VIDIOC_ENUM_FMT handlers */
> + .vidioc_enum_fmt_vid_cap = isp4vid_enum_fmt_vid_cap,
> +
> + /* VIDIOC_G_FMT handlers */
> + .vidioc_g_fmt_vid_cap = isp4vid_g_fmt_vid_cap,
> +
> + /* VIDIOC_S_FMT handlers */
> + .vidioc_s_fmt_vid_cap = isp4vid_s_fmt_vid_cap,
> +
> + /* VIDIOC_TRY_FMT handlers */
> + .vidioc_try_fmt_vid_cap = isp4vid_try_fmt_vid_cap,
> +
> + /* Buffer handlers */
> + .vidioc_reqbufs = vb2_ioctl_reqbufs,
> + .vidioc_querybuf = vb2_ioctl_querybuf,
> + .vidioc_qbuf = vb2_ioctl_qbuf,
> + .vidioc_expbuf = vb2_ioctl_expbuf,
> + .vidioc_dqbuf = vb2_ioctl_dqbuf,
> + .vidioc_create_bufs = vb2_ioctl_create_bufs,
> + .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> +
> + /* Stream on/off */
> + .vidioc_streamon = vb2_ioctl_streamon,
> + .vidioc_streamoff = vb2_ioctl_streamoff,
> +
> + /* Stream type-dependent parameter ioctls */
> + .vidioc_g_parm = isp4vid_ioctl_g_param,
> + .vidioc_s_parm = isp4vid_ioctl_g_param,
> +
> + .vidioc_enum_framesizes = isp4vid_enum_framesizes,
> + .vidioc_enum_frameintervals = isp4vid_ioctl_enum_frameintervals,
> +
I believe you can drop the above comments; they provide no additional
information whatsoever. Extra newline above, too.
> +};
> +
> +static unsigned int isp4vid_get_image_size(struct v4l2_pix_format *fmt)
> +{
> + switch (fmt->pixelformat) {
> + case V4L2_PIX_FMT_NV12:
> + return fmt->width * fmt->height * 3 / 2;
> + case V4L2_PIX_FMT_YUYV:
> + return fmt->width * fmt->height * 2;
> + default:
> + return 0;
> + }
> +}
> +
> +static int isp4vid_qops_queue_setup(struct vb2_queue *vq, unsigned int *nbuffers,
> + unsigned int *nplanes, unsigned int sizes[],
> + struct device *alloc_devs[])
> +{
> + struct isp4vid_dev *isp_vdev = vb2_get_drv_priv(vq);
> + unsigned int q_num_bufs = vb2_get_num_buffers(vq);
> +
> + if (*nplanes > 1) {
> + dev_err(isp_vdev->dev,
> + "fail to setup queue, no mplane supported %u\n",
> + *nplanes);
> + return -EINVAL;
> + }
> +
> + if (*nplanes == 1) {
> + unsigned int size;
> +
> + size = isp4vid_get_image_size(&isp_vdev->format);
> + if (sizes[0] < size) {
> + dev_err(isp_vdev->dev,
> + "fail for small plane size %u, %u expected\n",
> + sizes[0], size);
> + return -EINVAL;
> + }
> + }
> +
> + if (q_num_bufs + *nbuffers < ISP4IF_MAX_STREAM_BUF_COUNT)
> + *nbuffers = ISP4IF_MAX_STREAM_BUF_COUNT - q_num_bufs;
> +
> + switch (isp_vdev->format.pixelformat) {
> + case V4L2_PIX_FMT_NV12:
> + case V4L2_PIX_FMT_YUYV: {
> + *nplanes = 1;
> + sizes[0] = max(sizes[0], isp_vdev->format.sizeimage);
> + isp_vdev->format.sizeimage = sizes[0];
> + }
> + break;
> + default:
> + dev_err(isp_vdev->dev, "%s|unsupported fmt=%u\n",
> + isp_vdev->vdev.name, isp_vdev->format.pixelformat);
> + return -EINVAL;
> + }
> +
> + dev_dbg(isp_vdev->dev, "%s|*nbuffers=%u *nplanes=%u sizes[0]=%u\n",
> + isp_vdev->vdev.name,
> + *nbuffers, *nplanes, sizes[0]);
> +
> + return 0;
> +}
> +
> +static void isp4vid_qops_buffer_queue(struct vb2_buffer *vb)
> +{
> + struct isp4vid_capture_buffer *buf =
> + container_of(vb, struct isp4vid_capture_buffer, vb2.vb2_buf);
> + struct isp4vid_dev *isp_vdev = vb2_get_drv_priv(vb->vb2_queue);
> +
> + struct isp4vid_vb2_buf *priv_buf = vb->planes[0].mem_priv;
> + struct isp4if_img_buf_info *img_buf = &buf->img_buf;
> +
> + dev_dbg(isp_vdev->dev, "%s|index=%u", isp_vdev->vdev.name, vb->index);
> +
> + dev_dbg(isp_vdev->dev, "queue isp user bo 0x%llx size=%lu",
> + priv_buf->gpu_addr,
> + priv_buf->size);
> +
> + switch (isp_vdev->format.pixelformat) {
> + case V4L2_PIX_FMT_NV12: {
> + u32 y_size = isp_vdev->format.sizeimage / 3 * 2;
> + u32 uv_size = isp_vdev->format.sizeimage / 3;
> +
> + img_buf->planes[0].len = y_size;
> + img_buf->planes[0].sys_addr = priv_buf->vaddr;
> + img_buf->planes[0].mc_addr = priv_buf->gpu_addr;
> +
> + dev_dbg(isp_vdev->dev, "img_buf[0]: mc=0x%llx size=%u",
> + img_buf->planes[0].mc_addr,
> + img_buf->planes[0].len);
> +
> + img_buf->planes[1].len = uv_size;
> + img_buf->planes[1].sys_addr = priv_buf->vaddr + y_size;
> + img_buf->planes[1].mc_addr = priv_buf->gpu_addr + y_size;
> +
> + dev_dbg(isp_vdev->dev, "img_buf[1]: mc=0x%llx size=%u",
> + img_buf->planes[1].mc_addr,
> + img_buf->planes[1].len);
> +
> + img_buf->planes[2].len = 0;
> + }
> + break;
> + case V4L2_PIX_FMT_YUYV: {
> + img_buf->planes[0].len = isp_vdev->format.sizeimage;
> + img_buf->planes[0].sys_addr = priv_buf->vaddr;
> + img_buf->planes[0].mc_addr = priv_buf->gpu_addr;
> +
> + dev_dbg(isp_vdev->dev, "img_buf[0]: mc=0x%llx size=%u",
> + img_buf->planes[0].mc_addr,
> + img_buf->planes[0].len);
> +
> + img_buf->planes[1].len = 0;
> + img_buf->planes[2].len = 0;
> + }
> + break;
> + default:
> + dev_err(isp_vdev->dev, "%s|unsupported fmt=%u",
> + isp_vdev->vdev.name, isp_vdev->format.pixelformat);
> + return;
> + }
> +
> + if (isp_vdev->stream_started)
> + isp_vdev->ops->send_buffer(isp_vdev->isp_sdev, img_buf);
> +
> + scoped_guard(mutex, &isp_vdev->buf_list_lock)
> + list_add_tail(&buf->list, &isp_vdev->buf_list);
> +}
> +
> +static int isp4vid_qops_start_streaming(struct vb2_queue *vq, unsigned int count)
> +{
> + struct isp4vid_dev *isp_vdev = vb2_get_drv_priv(vq);
> + struct isp4vid_capture_buffer *isp4vid_buf;
> + struct media_entity *entity;
> + struct v4l2_subdev *subdev;
> + struct media_pad *pad;
> + int ret = 0;
> +
> + isp_vdev->sequence = 0;
> + ret = v4l2_pipeline_pm_get(&isp_vdev->vdev.entity);
> + if (ret) {
> + dev_err(isp_vdev->dev, "power up isp fail %d\n", ret);
> + goto release_buffers;
> + }
> +
> + entity = &isp_vdev->vdev.entity;
> + while (1) {
> + pad = &entity->pads[0];
> + if (!(pad->flags & MEDIA_PAD_FL_SINK))
> + break;
> +
> + pad = media_pad_remote_pad_first(pad);
> + if (!pad || !is_media_entity_v4l2_subdev(pad->entity))
> + break;
> +
> + entity = pad->entity;
> + subdev = media_entity_to_v4l2_subdev(entity);
> +
> + ret = v4l2_subdev_call(subdev, video, s_stream, 1);
> + if (ret < 0 && ret != -ENOIOCTLCMD) {
> + dev_dbg(isp_vdev->dev, "fail start streaming: %s %d\n",
> + subdev->name, ret);
> + goto release_buffers;
> + }
> + }
> +
> + list_for_each_entry(isp4vid_buf, &isp_vdev->buf_list, list)
> + isp_vdev->ops->send_buffer(isp_vdev->isp_sdev, &isp4vid_buf->img_buf);
> +
> + /* Start the media pipeline */
> + ret = video_device_pipeline_start(&isp_vdev->vdev, &isp_vdev->pipe);
> + if (ret) {
> + dev_err(isp_vdev->dev, "video_device_pipeline_start fail:%d",
> + ret);
> + goto release_buffers;
> + }
> +
> + isp_vdev->stream_started = true;
> +
> + return 0;
> +
> +release_buffers:
> + isp4vid_capture_return_all_buffers(isp_vdev, VB2_BUF_STATE_QUEUED);
> + return ret;
> +}
> +
> +static void isp4vid_qops_stop_streaming(struct vb2_queue *vq)
> +{
> + struct isp4vid_dev *isp_vdev = vb2_get_drv_priv(vq);
> + struct v4l2_subdev *subdev;
> + struct media_entity *entity;
> + struct media_pad *pad;
> + int ret;
> +
> + entity = &isp_vdev->vdev.entity;
> + while (1) {
> + pad = &entity->pads[0];
> + if (!(pad->flags & MEDIA_PAD_FL_SINK))
> + break;
> +
> + pad = media_pad_remote_pad_first(pad);
> + if (!pad || !is_media_entity_v4l2_subdev(pad->entity))
> + break;
> +
> + entity = pad->entity;
> + subdev = media_entity_to_v4l2_subdev(entity);
> +
> + ret = v4l2_subdev_call(subdev, video, s_stream, 0);
> +
> + if (ret < 0 && ret != -ENOIOCTLCMD)
> + dev_dbg(isp_vdev->dev, "fail stop streaming: %s %d\n",
> + subdev->name, ret);
> + }
> +
> + isp_vdev->stream_started = false;
> + v4l2_pipeline_pm_put(&isp_vdev->vdev.entity);
Please don't add new users of these functions. (Just omit them.)
> +
> + /* Stop the media pipeline */
> + video_device_pipeline_stop(&isp_vdev->vdev);
> +
> + /* Release all active buffers */
> + isp4vid_capture_return_all_buffers(isp_vdev, VB2_BUF_STATE_ERROR);
> +}
> +
> +static const struct vb2_ops isp4vid_qops = {
> + .queue_setup = isp4vid_qops_queue_setup,
> + .buf_queue = isp4vid_qops_buffer_queue,
> + .start_streaming = isp4vid_qops_start_streaming,
> + .stop_streaming = isp4vid_qops_stop_streaming,
> + .wait_prepare = vb2_ops_wait_prepare,
> + .wait_finish = vb2_ops_wait_finish,
> +};
> +
> +int isp4vid_dev_init(struct isp4vid_dev *isp_vdev, struct v4l2_subdev *isp_sdev,
> + const struct isp4vid_ops *ops)
> +{
> + const char *vdev_name = isp4vid_video_dev_name;
> + struct v4l2_device *v4l2_dev;
> + struct video_device *vdev;
> + struct vb2_queue *q;
> + int ret;
> +
> + if (!isp_vdev || !isp_sdev || !isp_sdev->v4l2_dev)
> + return -EINVAL;
> +
> + v4l2_dev = isp_sdev->v4l2_dev;
> + vdev = &isp_vdev->vdev;
> +
> + isp_vdev->isp_sdev = isp_sdev;
> + isp_vdev->dev = v4l2_dev->dev;
> + isp_vdev->ops = ops;
> +
> + /* Initialize the vb2_queue struct */
> + mutex_init(&isp_vdev->vbq_lock);
> + q = &isp_vdev->vbq;
> + q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> + q->io_modes = VB2_MMAP | VB2_DMABUF;
> + q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> + q->buf_struct_size = sizeof(struct isp4vid_capture_buffer);
> + q->min_queued_buffers = 2;
> + q->ops = &isp4vid_qops;
> + q->drv_priv = isp_vdev;
> + q->mem_ops = &isp4vid_vb2_memops;
> + q->lock = &isp_vdev->vbq_lock;
> + q->dev = v4l2_dev->dev;
> + ret = vb2_queue_init(q);
> + if (ret) {
> + dev_err(v4l2_dev->dev, "vb2_queue_init error:%d", ret);
> + return ret;
> + }
> + /* Initialize buffer list and its lock */
> + mutex_init(&isp_vdev->buf_list_lock);
> + INIT_LIST_HEAD(&isp_vdev->buf_list);
> +
> + /* Set default frame format */
> + isp_vdev->format = isp4vid_fmt_default;
> + isp_vdev->timeperframe = ISP4VID_ISP_TPF_DEFAULT;
> + v4l2_simplify_fraction(&isp_vdev->timeperframe.numerator,
> + &isp_vdev->timeperframe.denominator, 8, 333);
> +
> + ret = isp4vid_fill_buffer_size(&isp_vdev->format);
> + if (ret) {
> + dev_err(v4l2_dev->dev, "fail to fill buffer size: %d\n", ret);
> + goto err_release_vb2_queue;
> + }
> +
> + ret = isp4vid_set_fmt_2_isp(isp_sdev, &isp_vdev->format);
> + if (ret) {
> + dev_err(v4l2_dev->dev, "fail init format :%d\n", ret);
> + goto err_release_vb2_queue;
> + }
> +
> + /* Initialize the video_device struct */
> + isp_vdev->vdev.entity.name = vdev_name;
> + isp_vdev->vdev.entity.function = MEDIA_ENT_F_IO_V4L;
> + isp_vdev->vdev_pad.flags = MEDIA_PAD_FL_SINK;
> + ret = media_entity_pads_init(&isp_vdev->vdev.entity, 1,
> + &isp_vdev->vdev_pad);
> +
> + if (ret) {
> + dev_err(v4l2_dev->dev, "init media entity pad fail:%d\n", ret);
> + goto err_release_vb2_queue;
> + }
> +
> + vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE |
> + V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
> + vdev->entity.ops = &isp4vid_vdev_ent_ops;
> + vdev->release = video_device_release_empty;
> + vdev->fops = &isp4vid_vdev_fops;
> + vdev->ioctl_ops = &isp4vid_vdev_ioctl_ops;
> + vdev->lock = NULL;
> + vdev->queue = q;
> + vdev->v4l2_dev = v4l2_dev;
> + vdev->vfl_dir = VFL_DIR_RX;
> + strscpy(vdev->name, vdev_name, sizeof(vdev->name));
> + video_set_drvdata(vdev, isp_vdev);
> +
> + ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
> + if (ret) {
> + dev_err(v4l2_dev->dev, "register video device fail:%d\n", ret);
> + goto err_entity_cleanup;
> + }
> +
> + return 0;
> +
> +err_entity_cleanup:
> + media_entity_cleanup(&isp_vdev->vdev.entity);
> +err_release_vb2_queue:
> + vb2_queue_release(q);
> + return ret;
> +}
> +
> +void isp4vid_dev_deinit(struct isp4vid_dev *isp_vdev)
> +{
> + vb2_video_unregister_device(&isp_vdev->vdev);
> +}
> diff --git a/drivers/media/platform/amd/isp4/isp4_video.h b/drivers/media/platform/amd/isp4/isp4_video.h
> new file mode 100644
> index 000000000000..b87316d2a2e5
> --- /dev/null
> +++ b/drivers/media/platform/amd/isp4/isp4_video.h
> @@ -0,0 +1,65 @@
> +/* SPDX-License-Identifier: GPL-2.0+ */
> +/*
> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
> + */
> +
> +#ifndef _ISP4_VIDEO_H_
> +#define _ISP4_VIDEO_H_
> +
> +#include <media/videobuf2-memops.h>
> +#include <media/v4l2-dev.h>
> +
> +#include "isp4_interface.h"
> +
> +struct isp4vid_capture_buffer {
> + /*
> + * struct vb2_v4l2_buffer must be the first element
> + * the videobuf2 framework will allocate this struct based on
> + * buf_struct_size and use the first sizeof(struct vb2_buffer) bytes of
> + * memory as a vb2_buffer
> + */
> + struct vb2_v4l2_buffer vb2;
> + struct isp4if_img_buf_info img_buf;
> + struct list_head list;
> +};
> +
> +struct isp4vid_ops {
> + int (*send_buffer)(struct v4l2_subdev *sd,
> + struct isp4if_img_buf_info *img_buf);
Is there a reason why isp4sd_ioc_send_img_buf() isn't called directly?
> +};
> +
> +struct isp4vid_dev {
> + struct video_device vdev;
> + struct media_pad vdev_pad;
> + struct v4l2_pix_format format;
> +
> + /* mutex that protects vbq */
> + struct mutex vbq_lock;
> + struct vb2_queue vbq;
> +
> + /* mutex that protects buf_list */
> + struct mutex buf_list_lock;
> + struct list_head buf_list;
> +
> + u32 sequence;
> + bool stream_started;
> +
> + struct media_pipeline pipe;
> + struct device *dev;
> + struct v4l2_subdev *isp_sdev;
> + struct v4l2_fract timeperframe;
> +
> + /* Callback operations */
> + const struct isp4vid_ops *ops;
> +};
> +
> +int isp4vid_dev_init(struct isp4vid_dev *isp_vdev,
> + struct v4l2_subdev *isp_sdev,
> + const struct isp4vid_ops *ops);
> +
> +void isp4vid_dev_deinit(struct isp4vid_dev *isp_vdev);
> +
> +void isp4vid_handle_frame_done(struct isp4vid_dev *isp_vdev,
> + const struct isp4if_img_buf_info *img_buf);
> +
> +#endif /* _ISP4_VIDEO_H_ */
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 5/7] media: platform: amd: isp4 video node and buffers handling added
2025-12-22 17:07 ` Sakari Ailus
@ 2026-01-09 10:08 ` Du, Bin
2026-01-29 21:17 ` Sakari Ailus
0 siblings, 1 reply; 46+ messages in thread
From: Du, Bin @ 2026-01-09 10:08 UTC (permalink / raw)
To: Sakari Ailus
Cc: mchehab@kernel.org, hverkuil@xs4all.nl,
laurent.pinchart+renesas@ideasonboard.com,
bryan.odonoghue@linaro.org,
prabhakar.mahadev-lad.rj@bp.renesas.com,
linux-media@vger.kernel.org, linux-kernel@vger.kernel.org,
sultan@kerneltoast.com, Nirujogi, Pratap,
Chan, Benjamin (Koon Pan), Li, King, Rosikopulos, Gjorgji,
Jawich, Phil, Antony, Dominic, Limonciello, Mario, Gong, Richard,
Tsao, Anson, Svetoslav Stoilov, Alexey Zagorodnikov
Thanks Sakari for the review.
On 12/23/2025 1:07 AM, Sakari Ailus wrote:
> Hi Bin,
>
> On Tue, Dec 16, 2025 at 05:13:24PM +0800, Bin Du wrote:
>> Isp video implements v4l2 video interface and supports NV12 and YUYV. It
>> manages buffers, pipeline power and state. Cherry-picked Sultan's DMA
>> buffer related fix from branch v6.16-drm-tip-isp4-for-amd on
>> https://github.com/kerneltoast/kernel_x86_laptop.git
>>
>> Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
>> Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
>> Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
>> Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
>> Signed-off-by: Bin Du <Bin.Du@amd.com>
>> Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
>> ---
>> MAINTAINERS | 2 +
>> drivers/media/platform/amd/isp4/Makefile | 3 +-
>> drivers/media/platform/amd/isp4/isp4.c | 10 +
>> drivers/media/platform/amd/isp4/isp4_subdev.c | 79 +-
>> drivers/media/platform/amd/isp4/isp4_subdev.h | 2 +
>> drivers/media/platform/amd/isp4/isp4_video.c | 1165 +++++++++++++++++
>> drivers/media/platform/amd/isp4/isp4_video.h | 65 +
>> 7 files changed, 1321 insertions(+), 5 deletions(-)
>> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c
>> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 48ffc8bbdcee..80c966fde0b4 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -1151,6 +1151,8 @@ F: drivers/media/platform/amd/isp4/isp4_interface.c
>> F: drivers/media/platform/amd/isp4/isp4_interface.h
>> F: drivers/media/platform/amd/isp4/isp4_subdev.c
>> F: drivers/media/platform/amd/isp4/isp4_subdev.h
>> +F: drivers/media/platform/amd/isp4/isp4_video.c
>> +F: drivers/media/platform/amd/isp4/isp4_video.h
>>
>> AMD KFD
>> M: Felix Kuehling <Felix.Kuehling@amd.com>
>> diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
>> index 6d4e6d6ac7f5..398c20ea7866 100644
>> --- a/drivers/media/platform/amd/isp4/Makefile
>> +++ b/drivers/media/platform/amd/isp4/Makefile
>> @@ -5,4 +5,5 @@
>> obj-$(CONFIG_AMD_ISP4) += amd_capture.o
>> amd_capture-objs := isp4.o \
>> isp4_interface.o \
>> - isp4_subdev.o
>> + isp4_subdev.o \
>> + isp4_video.o
>> diff --git a/drivers/media/platform/amd/isp4/isp4.c b/drivers/media/platform/amd/isp4/isp4.c
>> index bcd7cad32afd..b5470f6dc93e 100644
>> --- a/drivers/media/platform/amd/isp4/isp4.c
>> +++ b/drivers/media/platform/amd/isp4/isp4.c
>> @@ -169,6 +169,16 @@ static int isp4_capture_probe(struct platform_device *pdev)
>> goto err_pm_disable;
>> }
>>
>> + ret = media_create_pad_link(&isp_dev->isp_subdev.sdev.entity,
>> + 0, &isp_dev->isp_subdev.isp_vdev.vdev.entity,
>> + 0,
>> + MEDIA_LNK_FL_ENABLED |
>> + MEDIA_LNK_FL_IMMUTABLE);
>> + if (ret) {
>> + dev_err_probe(dev, ret, "fail to create pad link\n");
>> + goto err_isp4_deinit;
>> + }
>> +
>> ret = media_device_register(&isp_dev->mdev);
>> if (ret) {
>> dev_err_probe(dev, ret, "fail to register media device\n");
>> diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.c b/drivers/media/platform/amd/isp4/isp4_subdev.c
>> index 3a25d1fc49ce..8c33187efc54 100644
>> --- a/drivers/media/platform/amd/isp4/isp4_subdev.c
>> +++ b/drivers/media/platform/amd/isp4/isp4_subdev.c
>> @@ -449,7 +449,7 @@ static void isp4sd_fw_resp_frame_done(struct isp4_subdev *isp_subdev,
>> return;
>> }
>>
>> - dev_dbg(dev, "ts:%llu,streamId:%d,poc:%u,preview_en:%u(%i)\n",
>> + dev_dbg(dev, "ts:%llu,streamId:%d,poc:%u,preview_en:%u,(%i)\n",
>
> Please make the change to the patch adding the line.
>
Sure, will do that.
>> ktime_get_ns(), stream_id, meta->poc,
>> meta->preview.enabled,
>> meta->preview.status);
>> @@ -459,11 +459,13 @@ static void isp4sd_fw_resp_frame_done(struct isp4_subdev *isp_subdev,
>> meta->preview.status == BUFFER_STATUS_DONE ||
>> meta->preview.status == BUFFER_STATUS_DIRTY)) {
>> prev = isp4if_dequeue_buffer(ispif);
>> - if (prev)
>> + if (prev) {
>> + isp4vid_handle_frame_done(&isp_subdev->isp_vdev,
>> + &prev->buf_info);
>> isp4if_dealloc_buffer_node(prev);
>> - else
>> + } else {
>> dev_err(dev, "fail null prev buf\n");
>> -
>> + }
>> } else if (meta->preview.enabled) {
>> dev_err(dev, "fail bad preview status %u\n",
>> meta->preview.status);
>> @@ -794,6 +796,65 @@ static int isp4sd_start_stream(struct isp4_subdev *isp_subdev,
>> return ret;
>> }
>>
>> +static int isp4sd_ioc_send_img_buf(struct v4l2_subdev *sd,
>> + struct isp4if_img_buf_info *buf_info)
>> +{
>> + struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
>> + struct isp4_interface *ispif = &isp_subdev->ispif;
>> + struct isp4if_img_buf_node *buf_node;
>> + struct device *dev = isp_subdev->dev;
>> + int ret;
>> +
>> + guard(mutex)(&isp_subdev->ops_mutex);
>> +
>> + if (ispif->status != ISP4IF_STATUS_FW_RUNNING) {
>> + dev_err(dev, "fail send img buf for bad fsm %d\n",
>> + ispif->status);
>> + return -EINVAL;
>> + }
>> +
>> + buf_node = isp4if_alloc_buffer_node(buf_info);
>> + if (!buf_node) {
>> + dev_err(dev, "fail alloc sys img buf info node\n");
>> + return -ENOMEM;
>> + }
>> +
>> + ret = isp4if_queue_buffer(ispif, buf_node);
>> + if (ret) {
>> + dev_err(dev, "fail to queue image buf, %d\n", ret);
>> + goto error_release_buf_node;
>> + }
>> +
>> + if (!isp_subdev->sensor_info.start_stream_cmd_sent) {
>> + isp_subdev->sensor_info.buf_sent_cnt++;
>> +
>> + if (isp_subdev->sensor_info.buf_sent_cnt >=
>> + ISP4SD_MIN_BUF_CNT_BEF_START_STREAM) {
>> + ret = isp4if_send_command(ispif, CMD_ID_START_STREAM,
>> + NULL, 0);
>> + if (ret) {
>> + dev_err(dev, "fail to START_STREAM");
>> + goto error_release_buf_node;
>> + }
>> + isp_subdev->sensor_info.start_stream_cmd_sent = true;
>> + isp_subdev->sensor_info.output_info.start_status =
>> + ISP4SD_START_STATUS_STARTED;
>> + isp_subdev->sensor_info.status =
>> + ISP4SD_START_STATUS_STARTED;
>> + } else {
>> + dev_dbg(dev, "no send start, required %u, buf sent %u\n",
>> + ISP4SD_MIN_BUF_CNT_BEF_START_STREAM,
>> + isp_subdev->sensor_info.buf_sent_cnt);
>> + }
>> + }
>> +
>> + return 0;
>> +
>> +error_release_buf_node:
>> + isp4if_dealloc_buffer_node(buf_node);
>> + return ret;
>> +}
>> +
>> static int isp4sd_set_power(struct v4l2_subdev *sd, int on)
>> {
>> struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
>> @@ -892,6 +953,10 @@ static const struct media_entity_operations isp4sd_sdev_ent_ops = {
>> .link_validate = isp4sd_sdev_link_validate,
>> };
>>
>> +static const struct isp4vid_ops isp4sd_isp4vid_ops = {
>> + .send_buffer = isp4sd_ioc_send_img_buf,
>> +};
>> +
>> int isp4sd_init(struct isp4_subdev *isp_subdev, struct v4l2_device *v4l2_dev,
>> int irq[ISP4SD_MAX_FW_RESP_STREAM_NUM])
>> {
>> @@ -952,6 +1017,11 @@ int isp4sd_init(struct isp4_subdev *isp_subdev, struct v4l2_device *v4l2_dev,
>> isp_subdev->host2fw_seq_num = 1;
>> ispif->status = ISP4IF_STATUS_PWR_OFF;
>>
>> + ret = isp4vid_dev_init(&isp_subdev->isp_vdev, &isp_subdev->sdev,
>> + &isp4sd_isp4vid_ops);
>> + if (ret)
>> + goto err_subdev_unreg;
>> +
>> return 0;
>>
>> err_subdev_unreg:
>> @@ -966,6 +1036,7 @@ void isp4sd_deinit(struct isp4_subdev *isp_subdev)
>> {
>> struct isp4_interface *ispif = &isp_subdev->ispif;
>>
>> + isp4vid_dev_deinit(&isp_subdev->isp_vdev);
>> v4l2_device_unregister_subdev(&isp_subdev->sdev);
>> media_entity_cleanup(&isp_subdev->sdev.entity);
>> isp4if_deinit(ispif);
>> diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.h b/drivers/media/platform/amd/isp4/isp4_subdev.h
>> index 70007b9de3f3..386e4e68f3fa 100644
>> --- a/drivers/media/platform/amd/isp4/isp4_subdev.h
>> +++ b/drivers/media/platform/amd/isp4/isp4_subdev.h
>> @@ -17,6 +17,7 @@
>> #include "isp4_fw_cmd_resp.h"
>> #include "isp4_hw_reg.h"
>> #include "isp4_interface.h"
>> +#include "isp4_video.h"
>>
>> /*
>> * one is for none sesnor specefic response which is not used now
>> @@ -93,6 +94,7 @@ struct isp4_subdev_thread_param {
>> struct isp4_subdev {
>> struct v4l2_subdev sdev;
>> struct isp4_interface ispif;
>> + struct isp4vid_dev isp_vdev;
>>
>> struct media_pad sdev_pad;
>>
>> diff --git a/drivers/media/platform/amd/isp4/isp4_video.c b/drivers/media/platform/amd/isp4/isp4_video.c
>> new file mode 100644
>> index 000000000000..5006e5021155
>> --- /dev/null
>> +++ b/drivers/media/platform/amd/isp4/isp4_video.c
>> @@ -0,0 +1,1165 @@
>> +// SPDX-License-Identifier: GPL-2.0+
>> +/*
>> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
>> + */
>> +
>> +#include <linux/vmalloc.h>
>> +#include <media/v4l2-ioctl.h>
>> +#include <media/v4l2-mc.h>
>> +
>> +#include "isp4_interface.h"
>> +#include "isp4_subdev.h"
>> +#include "isp4_video.h"
>> +
>> +#define ISP4VID_ISP_DRV_NAME "amd_isp_capture"
>> +#define ISP4VID_MAX_PREVIEW_FPS 30
>> +#define ISP4VID_DEFAULT_FMT isp4vid_formats[0]
>> +
>> +#define ISP4VID_PAD_VIDEO_OUTPUT 0
>> +
>> +/* timeperframe default */
>> +#define ISP4VID_ISP_TPF_DEFAULT isp4vid_tpfs[0]
>> +
>> +/* amdisp buffer for vb2 operations */
>> +struct isp4vid_vb2_buf {
>> + struct device *dev;
>> + void *vaddr;
>> + unsigned long size;
>> + refcount_t refcount;
>> + struct dma_buf *dbuf;
>> + void *bo;
>> + u64 gpu_addr;
>> + struct vb2_vmarea_handler handler;
>> +};
>> +
>> +static void isp4vid_vb2_put(void *buf_priv);
>> +
>> +static const char *const isp4vid_video_dev_name = "Preview";
>> +
>> +/* Sizes must be in increasing order */
>> +static const struct v4l2_frmsize_discrete isp4vid_frmsize[] = {
>> + {640, 360},
>> + {640, 480},
>> + {1280, 720},
>> + {1280, 960},
>> + {1920, 1080},
>> + {1920, 1440},
>> + {2560, 1440},
>> + {2880, 1620},
>> + {2880, 1624},
>> + {2888, 1808},
>> +};
>> +
>> +static const u32 isp4vid_formats[] = {
>> + V4L2_PIX_FMT_NV12,
>> + V4L2_PIX_FMT_YUYV
>> +};
>> +
>> +/* timeperframe list */
>> +static const struct v4l2_fract isp4vid_tpfs[] = {
>> + { 1, ISP4VID_MAX_PREVIEW_FPS }
>> +};
>> +
>> +void isp4vid_handle_frame_done(struct isp4vid_dev *isp_vdev,
>> + const struct isp4if_img_buf_info *img_buf)
>> +{
>> + struct isp4vid_capture_buffer *isp4vid_buf;
>> + void *vbuf;
>> +
>> + scoped_guard(mutex, &isp_vdev->buf_list_lock) {
>> + /* Get the first entry of the list */
>> + isp4vid_buf = list_first_entry_or_null(&isp_vdev->buf_list, typeof(*isp4vid_buf),
>> + list);
>> + if (!isp4vid_buf)
>> + return;
>> +
>> + vbuf = vb2_plane_vaddr(&isp4vid_buf->vb2.vb2_buf, 0);
>> +
>> + if (vbuf != img_buf->planes[0].sys_addr) {
>> + dev_err(isp_vdev->dev, "Invalid vbuf");
>> + return;
>> + }
>> +
>> + /* Remove this entry from the list */
>> + list_del(&isp4vid_buf->list);
>> + }
>> +
>> + /* Fill the buffer */
>> + isp4vid_buf->vb2.vb2_buf.timestamp = ktime_get_ns();
>> + isp4vid_buf->vb2.sequence = isp_vdev->sequence++;
>> + isp4vid_buf->vb2.field = V4L2_FIELD_ANY;
>> +
>> + /* at most 2 planes */
>> + vb2_set_plane_payload(&isp4vid_buf->vb2.vb2_buf,
>> + 0, isp_vdev->format.sizeimage);
>> +
>> + vb2_buffer_done(&isp4vid_buf->vb2.vb2_buf, VB2_BUF_STATE_DONE);
>> +
>> + dev_dbg(isp_vdev->dev, "call vb2_buffer_done(size=%u)\n",
>> + isp_vdev->format.sizeimage);
>> +}
>> +
>> +static unsigned int isp4vid_vb2_num_users(void *buf_priv)
>> +{
>> + struct isp4vid_vb2_buf *buf = buf_priv;
>> +
>> + return refcount_read(&buf->refcount);
>> +}
>> +
>> +static int isp4vid_vb2_mmap(void *buf_priv, struct vm_area_struct *vma)
>> +{
>> + struct isp4vid_vb2_buf *buf = buf_priv;
>> + int ret;
>> +
>> + if (!buf) {
>> + pr_err("fail no memory to map\n");
>> + return -EINVAL;
>> + }
>> +
>> + ret = remap_vmalloc_range(vma, buf->vaddr, 0);
>> + if (ret) {
>> + dev_err(buf->dev, "fail remap vmalloc mem, %d\n", ret);
>> + return ret;
>> + }
>> +
>> + /*
>> + * Make sure that vm_areas for 2 buffers won't be merged together
>> + */
>> + vm_flags_set(vma, VM_DONTEXPAND);
>> +
>> + /*
>> + * Use common vm_area operations to track buffer refcount.
>> + */
>> + vma->vm_private_data = &buf->handler;
>> + vma->vm_ops = &vb2_common_vm_ops;
>> +
>> + vma->vm_ops->open(vma);
>> +
>> + dev_dbg(buf->dev, "mmap isp user bo 0x%llx size %ld refcount %d\n",
>> + buf->gpu_addr, buf->size, refcount_read(&buf->refcount));
>> +
>> + return 0;
>> +}
>> +
>> +static void *isp4vid_vb2_vaddr(struct vb2_buffer *vb, void *buf_priv)
>> +{
>> + struct isp4vid_vb2_buf *buf = buf_priv;
>> +
>> + if (!buf->vaddr) {
>> + dev_err(buf->dev,
>> + "fail for buf vaddr is null\n");
>
> Fits on the prefious line.
>
Thanks for catching this,
> Is this a driver bug or something that can be triggered by the user?
No, our implementation follows the same check as vb2_vmalloc_vaddr().
>
>> + return NULL;
>> + }
>> +
>> + return buf->vaddr;
>> +}
>> +
>> +static void isp4vid_vb2_detach_dmabuf(void *mem_priv)
>> +{
>> + struct isp4vid_vb2_buf *buf = mem_priv;
>> +
>> + dev_dbg(buf->dev, "detach dmabuf of isp user bo 0x%llx size %ld",
>> + buf->gpu_addr, buf->size);
>> +
>> + kfree(buf);
>> +}
>> +
>> +static void *isp4vid_vb2_attach_dmabuf(struct vb2_buffer *vb, struct device *dev,
>> + struct dma_buf *dbuf,
>> + unsigned long size)
>> +{
>> + struct isp4vid_vb2_buf *dbg_buf = dbuf->priv;
>> + struct isp4vid_vb2_buf *buf;
>> +
>> + if (dbuf->size < size) {
>> + dev_err(dev, "Invalid dmabuf size %zu %lu", dbuf->size, size);
>> + return ERR_PTR(-EFAULT);
>> + }
>> +
>> + buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>> + if (!buf)
>> + return ERR_PTR(-ENOMEM);
>> +
>> + buf->dev = dev;
>> + buf->dbuf = dbuf;
>> + buf->size = size;
>> +
>> + dev_dbg(dev, "attach dmabuf of isp user bo 0x%llx size %ld",
>> + dbg_buf->gpu_addr, dbg_buf->size);
>> +
>> + return buf;
>> +}
>> +
>> +static void isp4vid_vb2_unmap_dmabuf(void *mem_priv)
>> +{
>> + struct isp4vid_vb2_buf *buf = mem_priv;
>> + struct iosys_map map = IOSYS_MAP_INIT_VADDR(buf->vaddr);
>> +
>> + dev_dbg(buf->dev, "unmap dmabuf of isp user bo 0x%llx size %ld",
>> + buf->gpu_addr, buf->size);
>> +
>> + dma_buf_vunmap_unlocked(buf->dbuf, &map);
>> + buf->vaddr = NULL;
>> +}
>> +
>> +static int isp4vid_vb2_map_dmabuf(void *mem_priv)
>> +{
>> + struct isp4vid_vb2_buf *buf = mem_priv, *mmap_buf;
>> + struct iosys_map map;
>> + int ret;
>> +
>> + ret = dma_buf_vmap_unlocked(buf->dbuf, &map);
>> + if (ret) {
>> + dev_err(buf->dev, "vmap_unlocked fail");
>> + return -EFAULT;
>> + }
>> +
>> + buf->vaddr = map.vaddr;
>> + mmap_buf = buf->dbuf->priv;
>> + buf->gpu_addr = mmap_buf->gpu_addr;
>> +
>> + dev_dbg(buf->dev, "map dmabuf of isp user bo 0x%llx size %ld",
>> + buf->gpu_addr, buf->size);
>> +
>> + return 0;
>> +}
>> +
>> +#ifdef CONFIG_HAS_DMA
>> +struct isp4vid_vb2_amdgpu_attachment {
>> + struct sg_table sgt;
>> + enum dma_data_direction dma_dir;
>> +};
>> +
>> +static int isp4vid_dmabuf_ops_attach(struct dma_buf *dbuf,
>> + struct dma_buf_attachment *dbuf_attach)
>> +{
>> + struct isp4vid_vb2_buf *buf = dbuf->priv;
>> + int num_pages = PAGE_ALIGN(buf->size) / PAGE_SIZE;
>> + struct isp4vid_vb2_amdgpu_attachment *attach;
>> + void *vaddr = buf->vaddr;
>> + struct scatterlist *sg;
>> + struct sg_table *sgt;
>> + int ret;
>> + int i;
>> +
>> + attach = kzalloc(sizeof(*attach), GFP_KERNEL);
>> + if (!attach)
>> + return -ENOMEM;
>> +
>> + sgt = &attach->sgt;
>> + ret = sg_alloc_table(sgt, num_pages, GFP_KERNEL);
>> + if (ret) {
>> + kfree(attach);
>> + return ret;
>> + }
>> +
>> + for_each_sgtable_sg(sgt, sg, i) {
>> + struct page *page = vmalloc_to_page(vaddr);
>> +
>> + if (!page) {
>> + sg_free_table(sgt);
>> + kfree(attach);
>> + return -ENOMEM;
>> + }
>> +
>> + sg_set_page(sg, page, PAGE_SIZE, 0);
>> + vaddr = ((char *)vaddr) + PAGE_SIZE;
>> + }
>> +
>> + attach->dma_dir = DMA_NONE;
>> + dbuf_attach->priv = attach;
>> +
>> + return 0;
>> +}
>> +
>> +static void isp4vid_dmabuf_ops_detach(struct dma_buf *dbuf,
>> + struct dma_buf_attachment *dbuf_attach)
>> +{
>> + struct isp4vid_vb2_amdgpu_attachment *attach = dbuf_attach->priv;
>> + struct sg_table *sgt;
>> +
>> + if (!attach) {
>> + pr_err("fail invalid attach handler\n");
>> + return;
>> + }
>> +
>> + sgt = &attach->sgt;
>> +
>> + /* release the scatterlist cache */
>> + if (attach->dma_dir != DMA_NONE)
>> + dma_unmap_sgtable(dbuf_attach->dev, sgt, attach->dma_dir, 0);
>> +
>> + sg_free_table(sgt);
>> + kfree(attach);
>> + dbuf_attach->priv = NULL;
>> +}
>> +
>> +static struct sg_table *isp4vid_dmabuf_ops_map(struct dma_buf_attachment *dbuf_attach,
>> + enum dma_data_direction dma_dir)
>> +{
>> + struct isp4vid_vb2_amdgpu_attachment *attach = dbuf_attach->priv;
>> + struct sg_table *sgt;
>> +
>> + sgt = &attach->sgt;
>> + /* return previously mapped sg table */
>> + if (attach->dma_dir == dma_dir)
>> + return sgt;
>> +
>> + /* release any previous cache */
>> + if (attach->dma_dir != DMA_NONE) {
>> + dma_unmap_sgtable(dbuf_attach->dev, sgt, attach->dma_dir, 0);
>> + attach->dma_dir = DMA_NONE;
>> + }
>> +
>> + /* mapping to the client with new direction */
>> + if (dma_map_sgtable(dbuf_attach->dev, sgt, dma_dir, 0)) {
>> + dev_err(dbuf_attach->dev, "fail to map scatterlist");
>> + return ERR_PTR(-EIO);
>> + }
>> +
>> + attach->dma_dir = dma_dir;
>> +
>> + return sgt;
>> +}
>> +
>> +static void isp4vid_dmabuf_ops_unmap(struct dma_buf_attachment *dbuf_attach,
>> + struct sg_table *sgt,
>> + enum dma_data_direction dma_dir)
>> +{
>> + /* nothing to be done here */
>> +}
>> +
>> +static int isp4vid_dmabuf_ops_vmap(struct dma_buf *dbuf,
>> + struct iosys_map *map)
>> +{
>> + struct isp4vid_vb2_buf *buf = dbuf->priv;
>> +
>> + iosys_map_set_vaddr(map, buf->vaddr);
>> +
>> + return 0;
>> +}
>> +
>> +static void isp4vid_dmabuf_ops_release(struct dma_buf *dbuf)
>> +{
>> + struct isp4vid_vb2_buf *buf = dbuf->priv;
>> +
>> + /* drop reference obtained in isp4vid_vb2_get_dmabuf */
>> + if (dbuf != buf->dbuf)
>> + isp4vid_vb2_put(buf);
>> + else
>> + kfree(buf);
>> +}
>> +
>> +static int isp4vid_dmabuf_ops_mmap(struct dma_buf *dbuf, struct vm_area_struct *vma)
>> +{
>> + return isp4vid_vb2_mmap(dbuf->priv, vma);
>> +}
>> +
>> +static const struct dma_buf_ops isp4vid_dmabuf_ops = {
>> + .attach = isp4vid_dmabuf_ops_attach,
>> + .detach = isp4vid_dmabuf_ops_detach,
>> + .map_dma_buf = isp4vid_dmabuf_ops_map,
>> + .unmap_dma_buf = isp4vid_dmabuf_ops_unmap,
>> + .vmap = isp4vid_dmabuf_ops_vmap,
>> + .mmap = isp4vid_dmabuf_ops_mmap,
>> + .release = isp4vid_dmabuf_ops_release,
>> +};
>> +
>> +static struct dma_buf *isp4vid_get_dmabuf(struct isp4vid_vb2_buf *buf, unsigned long flags)
>> +{
>> + DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
>> + struct dma_buf *dbuf;
>> +
>> + if (WARN_ON(!buf->vaddr))
>> + return NULL;
>> +
>> + exp_info.ops = &isp4vid_dmabuf_ops;
>> + exp_info.size = buf->size;
>> + exp_info.flags = flags;
>> + exp_info.priv = buf;
>> +
>> + dbuf = dma_buf_export(&exp_info);
>> + if (IS_ERR(dbuf))
>> + return NULL;
>> +
>> + return dbuf;
>> +}
>> +
>> +static struct dma_buf *isp4vid_vb2_get_dmabuf(struct vb2_buffer *vb, void *buf_priv,
>> + unsigned long flags)
>> +{
>> + struct isp4vid_vb2_buf *buf = buf_priv;
>> + struct dma_buf *dbuf;
>> +
>> + dbuf = isp4vid_get_dmabuf(buf, flags);
>> + if (!dbuf) {
>> + dev_err(buf->dev, "fail to get isp dma buffer\n");
>> + return NULL;
>> + }
>> +
>> + refcount_inc(&buf->refcount);
>> +
>> + dev_dbg(buf->dev, "buf exported, refcount %d\n",
>> + refcount_read(&buf->refcount));
>> +
>> + return dbuf;
>> +}
>> +#endif /* CONFIG_HAS_DMA */
>> +
>> +static void isp4vid_vb2_put(void *buf_priv)
>> +{
>> + struct isp4vid_vb2_buf *buf = buf_priv;
>> +
>> + dev_dbg(buf->dev,
>> + "release isp user bo 0x%llx size %ld refcount %d",
>> + buf->gpu_addr, buf->size,
>> + refcount_read(&buf->refcount));
>> +
>> + if (refcount_dec_and_test(&buf->refcount)) {
>> + isp_user_buffer_free(buf->bo);
>> + vfree(buf->vaddr);
>> + /*
>> + * Putting the implicit dmabuf frees `buf`. Freeing `buf` must
>> + * be done from the dmabuf release callback since dma_buf_put()
>> + * isn't always synchronous; it's just an fput(), which may be
>> + * deferred. Since the dmabuf release callback needs to access
>> + * `buf`, this means `buf` cannot be freed until then.
>> + */
>> + dma_buf_put(buf->dbuf);
>> + }
>> +}
>> +
>> +static void *isp4vid_vb2_alloc(struct vb2_buffer *vb, struct device *dev, unsigned long size)
>> +{
>> + struct isp4vid_vb2_buf *buf;
>> + u64 gpu_addr;
>> + void *bo;
>> + int ret;
>> +
>> + buf = kzalloc(sizeof(*buf), GFP_KERNEL | vb->vb2_queue->gfp_flags);
>> + if (!buf)
>> + return ERR_PTR(-ENOMEM);
>> +
>> + buf->dev = dev;
>> + buf->size = size;
>> + buf->vaddr = vmalloc_user(buf->size);
>> + if (!buf->vaddr) {
>> + dev_err(dev, "fail to vmalloc buffer\n");
>> + goto free_buf;
>> + }
>> +
>> + buf->handler.refcount = &buf->refcount;
>> + buf->handler.put = isp4vid_vb2_put;
>> + buf->handler.arg = buf;
>> +
>> + /* get implicit dmabuf */
>> + buf->dbuf = isp4vid_get_dmabuf(buf, 0);
>> + if (!buf->dbuf) {
>> + dev_err(dev, "fail to get implicit dmabuf\n");
>> + goto free_user_vmem;
>> + }
>> +
>> + /* create isp user BO and obtain gpu_addr */
>> + ret = isp_user_buffer_alloc(dev, buf->dbuf, &bo, &gpu_addr);
>> + if (ret) {
>> + dev_err(dev, "fail to create isp user BO\n");
>> + goto put_dmabuf;
>> + }
>> +
>> + buf->bo = bo;
>> + buf->gpu_addr = gpu_addr;
>> +
>> + refcount_set(&buf->refcount, 1);
>> +
>> + dev_dbg(dev, "allocated isp user bo 0x%llx size %ld refcount %d\n",
>> + buf->gpu_addr, buf->size, refcount_read(&buf->refcount));
>> +
>> + return buf;
>> +
>> +put_dmabuf:
>> + dma_buf_put(buf->dbuf);
>> +free_user_vmem:
>> + vfree(buf->vaddr);
>> +free_buf:
>> + ret = buf->vaddr ? -EINVAL : -ENOMEM;
>> + kfree(buf);
>> + return ERR_PTR(ret);
>> +}
>> +
>> +static const struct vb2_mem_ops isp4vid_vb2_memops = {
>> + .alloc = isp4vid_vb2_alloc,
>> + .put = isp4vid_vb2_put,
>> +#ifdef CONFIG_HAS_DMA
>> + .get_dmabuf = isp4vid_vb2_get_dmabuf,
>> +#endif
>> + .map_dmabuf = isp4vid_vb2_map_dmabuf,
>> + .unmap_dmabuf = isp4vid_vb2_unmap_dmabuf,
>> + .attach_dmabuf = isp4vid_vb2_attach_dmabuf,
>> + .detach_dmabuf = isp4vid_vb2_detach_dmabuf,
>> + .vaddr = isp4vid_vb2_vaddr,
>> + .mmap = isp4vid_vb2_mmap,
>> + .num_users = isp4vid_vb2_num_users,
>> +};
>
> Could you elaborate a bit why do you need your own videobuf mem ops?
>
Sure, ISP FW/HW can access system memory only via GPU VA, not system VA,
so vb2_vmalloc_memops can't be used directly, we need to base on it to
implement our own videobuf mem ops to support both GPU VA and system VA.
>> +
>> +static const struct v4l2_pix_format isp4vid_fmt_default = {
>> + .width = 1920,
>> + .height = 1080,
>> + .pixelformat = ISP4VID_DEFAULT_FMT,
>> + .field = V4L2_FIELD_NONE,
>> + .colorspace = V4L2_COLORSPACE_SRGB,
>> +};
>> +
>> +static void isp4vid_capture_return_all_buffers(struct isp4vid_dev *isp_vdev,
>> + enum vb2_buffer_state state)
>> +{
>> + struct isp4vid_capture_buffer *vbuf, *node;
>> +
>> + scoped_guard(mutex, &isp_vdev->buf_list_lock) {
>> + list_for_each_entry_safe(vbuf, node, &isp_vdev->buf_list, list)
>> + vb2_buffer_done(&vbuf->vb2.vb2_buf, state);
>> + INIT_LIST_HEAD(&isp_vdev->buf_list);
>> + }
>> +
>> + dev_dbg(isp_vdev->dev, "call vb2_buffer_done(%d)\n", state);
>> +}
>> +
>> +static int isp4vid_vdev_link_validate(struct media_link *link)
>> +{
>> + return 0;
>> +}
>> +
>> +static const struct media_entity_operations isp4vid_vdev_ent_ops = {
>> + .link_validate = isp4vid_vdev_link_validate,
>> +};
>> +
>> +static const struct v4l2_file_operations isp4vid_vdev_fops = {
>> + .owner = THIS_MODULE,
>> + .open = v4l2_fh_open,
>> + .release = vb2_fop_release,
>> + .read = vb2_fop_read,
>> + .poll = vb2_fop_poll,
>> + .unlocked_ioctl = video_ioctl2,
>> + .mmap = vb2_fop_mmap,
>> +};
>> +
>> +static int isp4vid_ioctl_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
>> +{
>> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
>> +
>> + strscpy(cap->driver, ISP4VID_ISP_DRV_NAME, sizeof(cap->driver));
>> + snprintf(cap->card, sizeof(cap->card), "%s", ISP4VID_ISP_DRV_NAME);
>> + cap->capabilities |= V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_CAPTURE;
>> +
>> + dev_dbg(isp_vdev->dev, "%s|capabilities=0x%X", isp_vdev->vdev.name, cap->capabilities);
>> +
>> + return 0;
>> +}
>> +
>> +static int isp4vid_g_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f)
>> +{
>> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
>> +
>> + f->fmt.pix = isp_vdev->format;
>> +
>> + return 0;
>> +}
>> +
>> +static int isp4vid_fill_buffer_size(struct v4l2_pix_format *fmt)
>> +{
>> + int ret = 0;
>> +
>> + switch (fmt->pixelformat) {
>> + case V4L2_PIX_FMT_NV12:
>> + fmt->bytesperline = fmt->width;
>> + fmt->sizeimage = fmt->bytesperline * fmt->height * 3 / 2;
>> + break;
>> + case V4L2_PIX_FMT_YUYV:
>> + fmt->bytesperline = fmt->width * 2;
>> + fmt->sizeimage = fmt->bytesperline * fmt->height;
>> + break;
>> + default:
>> + ret = -EINVAL;
>> + break;
>> + }
>> +
>> + return ret;
>> +}
>> +
>> +static int isp4vid_try_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f)
>> +{
>> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
>> + struct v4l2_pix_format *format = &f->fmt.pix;
>> + const struct v4l2_frmsize_discrete *fsz;
>> + int i;
>> +
>> + if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
>> + return -EINVAL;
>
> The framework already does the check.
>
Thanks for catching this, will remove the redundant check.
>> +
>> + /*
>> + * Check if the hardware supports the requested format, use the default
>> + * format otherwise.
>> + */
>> + for (i = 0; i < ARRAY_SIZE(isp4vid_formats); i++)
>> + if (isp4vid_formats[i] == format->pixelformat)
>> + break;
>> +
>> + if (i == ARRAY_SIZE(isp4vid_formats))
>> + format->pixelformat = ISP4VID_DEFAULT_FMT;
>> +
>> + switch (format->pixelformat) {
>> + case V4L2_PIX_FMT_NV12:
>> + case V4L2_PIX_FMT_YUYV:
>> + fsz = v4l2_find_nearest_size(isp4vid_frmsize, ARRAY_SIZE(isp4vid_frmsize),
>> + width, height, format->width, format->height);
>> + format->width = fsz->width;
>> + format->height = fsz->height;
>> + break;
>> + default:
>> + dev_err(isp_vdev->dev, "%s|unsupported fmt=%u", isp_vdev->vdev.name,
>> + format->pixelformat);
>> + return -EINVAL;
>> + }
>> +
>> + /* There is no need to check the return value, as failure will never happen here */
>> + isp4vid_fill_buffer_size(format);
>> +
>> + if (format->field == V4L2_FIELD_ANY)
>> + format->field = isp4vid_fmt_default.field;
>> +
>> + if (format->colorspace == V4L2_COLORSPACE_DEFAULT)
>> + format->colorspace = isp4vid_fmt_default.colorspace;
>> +
>> + return 0;
>> +}
>> +
>> +static int isp4vid_set_fmt_2_isp(struct v4l2_subdev *sdev, struct v4l2_pix_format *pix_fmt)
>> +{
>> + struct v4l2_subdev_format fmt = {};
>> +
>> + switch (pix_fmt->pixelformat) {
>> + case V4L2_PIX_FMT_NV12:
>> + fmt.format.code = MEDIA_BUS_FMT_YUYV8_1_5X8;
>> + break;
>> + case V4L2_PIX_FMT_YUYV:
>> + fmt.format.code = MEDIA_BUS_FMT_YUYV8_1X16;
>> + break;
>> + default:
>> + return -EINVAL;
>> + }
>> + fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
>> + fmt.pad = ISP4VID_PAD_VIDEO_OUTPUT;
>> + fmt.format.width = pix_fmt->width;
>> + fmt.format.height = pix_fmt->height;
>> + return v4l2_subdev_call(sdev, pad, set_fmt, NULL, &fmt);
>> +}
>> +
>> +static int isp4vid_s_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f)
>> +{
>> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
>> + int ret;
>> +
>> + /* Do not change the format while stream is on */
>> + if (vb2_is_busy(&isp_vdev->vbq))
>> + return -EBUSY;
>> +
>> + if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
>> + return -EINVAL;
>
> Ditto.
>
Yes, will remove it.
>> +
>> + ret = isp4vid_try_fmt_vid_cap(file, priv, f);
>> + if (ret)
>> + return ret;
>> +
>> + dev_dbg(isp_vdev->dev, "%s|width height:%ux%u->%ux%u",
>> + isp_vdev->vdev.name,
>> + isp_vdev->format.width, isp_vdev->format.height,
>> + f->fmt.pix.width, f->fmt.pix.height);
>> + dev_dbg(isp_vdev->dev, "%s|pixelformat:0x%x-0x%x",
>> + isp_vdev->vdev.name, isp_vdev->format.pixelformat,
>> + f->fmt.pix.pixelformat);
>> + dev_dbg(isp_vdev->dev, "%s|bytesperline:%u->%u",
>> + isp_vdev->vdev.name, isp_vdev->format.bytesperline,
>> + f->fmt.pix.bytesperline);
>> + dev_dbg(isp_vdev->dev, "%s|sizeimage:%u->%u",
>> + isp_vdev->vdev.name, isp_vdev->format.sizeimage,
>> + f->fmt.pix.sizeimage);
>> +
>> + isp_vdev->format = f->fmt.pix;
>> + ret = isp4vid_set_fmt_2_isp(isp_vdev->isp_sdev, &isp_vdev->format);
>> +
>> + return ret;
>> +}
>> +
>> +static int isp4vid_enum_fmt_vid_cap(struct file *file, void *priv, struct v4l2_fmtdesc *f)
>> +{
>> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
>> +
>> + switch (f->index) {
>> + case 0:
>> + f->pixelformat = V4L2_PIX_FMT_NV12;
>> + break;
>> + case 1:
>> + f->pixelformat = V4L2_PIX_FMT_YUYV;
>> + break;
>> + default:
>> + return -EINVAL;
>> + }
>> +
>> + dev_dbg(isp_vdev->dev, "%s|index=%d, pixelformat=0x%X",
>> + isp_vdev->vdev.name, f->index, f->pixelformat);
>> +
>> + return 0;
>> +}
>> +
>> +static int isp4vid_enum_framesizes(struct file *file, void *fh, struct v4l2_frmsizeenum *fsize)
>> +{
>> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
>> + unsigned int i;
>> +
>> + for (i = 0; i < ARRAY_SIZE(isp4vid_formats); i++) {
>> + if (isp4vid_formats[i] == fsize->pixel_format)
>> + break;
>> + }
>> + if (i == ARRAY_SIZE(isp4vid_formats))
>> + return -EINVAL;
>> +
>> + if (fsize->index < ARRAY_SIZE(isp4vid_frmsize)) {
>> + fsize->type = V4L2_FRMSIZE_TYPE_DISCRETE;
>> + fsize->discrete = isp4vid_frmsize[fsize->index];
>> + dev_dbg(isp_vdev->dev, "%s|size[%d]=%dx%d",
>> + isp_vdev->vdev.name, fsize->index,
>> + fsize->discrete.width, fsize->discrete.height);
>> + } else {
>> + return -EINVAL;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int isp4vid_ioctl_enum_frameintervals(struct file *file, void *priv,
>> + struct v4l2_frmivalenum *fival)
>> +{
>> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
>> + int i;
>> +
>> + if (fival->index >= ARRAY_SIZE(isp4vid_tpfs))
>> + return -EINVAL;
>> +
>> + for (i = 0; i < ARRAY_SIZE(isp4vid_formats); i++)
>> + if (isp4vid_formats[i] == fival->pixel_format)
>> + break;
>> +
>> + if (i == ARRAY_SIZE(isp4vid_formats))
>> + return -EINVAL;
>> +
>> + for (i = 0; i < ARRAY_SIZE(isp4vid_frmsize); i++)
>> + if (isp4vid_frmsize[i].width == fival->width &&
>> + isp4vid_frmsize[i].height == fival->height)
>> + break;
>> +
>> + if (i == ARRAY_SIZE(isp4vid_frmsize))
>> + return -EINVAL;
>> +
>> + fival->type = V4L2_FRMIVAL_TYPE_DISCRETE;
>> + fival->discrete = isp4vid_tpfs[fival->index];
>> + v4l2_simplify_fraction(&fival->discrete.numerator,
>> + &fival->discrete.denominator, 8, 333);
>> +
>> + dev_dbg(isp_vdev->dev, "%s|interval[%d]=%d/%d",
>> + isp_vdev->vdev.name, fival->index,
>> + fival->discrete.numerator,
>> + fival->discrete.denominator);
>> +
>> + return 0;
>> +}
>> +
>> +static int isp4vid_ioctl_g_param(struct file *file, void *priv, struct v4l2_streamparm *param)
>> +{
>> + struct v4l2_captureparm *capture = ¶m->parm.capture;
>> + struct isp4vid_dev *isp_vdev = video_drvdata(file);
>> +
>> + if (param->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
>> + return -EINVAL;
>> +
>> + capture->capability = V4L2_CAP_TIMEPERFRAME;
>> + capture->timeperframe = isp_vdev->timeperframe;
>> + capture->readbuffers = 0;
>> +
>> + dev_dbg(isp_vdev->dev, "%s|timeperframe=%d/%d", isp_vdev->vdev.name,
>> + capture->timeperframe.numerator,
>> + capture->timeperframe.denominator);
>> +
>> + return 0;
>> +}
>> +
>> +static const struct v4l2_ioctl_ops isp4vid_vdev_ioctl_ops = {
>> + /* VIDIOC_QUERYCAP handler */
>> + .vidioc_querycap = isp4vid_ioctl_querycap,
>> +
>> + /* VIDIOC_ENUM_FMT handlers */
>> + .vidioc_enum_fmt_vid_cap = isp4vid_enum_fmt_vid_cap,
>> +
>> + /* VIDIOC_G_FMT handlers */
>> + .vidioc_g_fmt_vid_cap = isp4vid_g_fmt_vid_cap,
>> +
>> + /* VIDIOC_S_FMT handlers */
>> + .vidioc_s_fmt_vid_cap = isp4vid_s_fmt_vid_cap,
>> +
>> + /* VIDIOC_TRY_FMT handlers */
>> + .vidioc_try_fmt_vid_cap = isp4vid_try_fmt_vid_cap,
>> +
>> + /* Buffer handlers */
>> + .vidioc_reqbufs = vb2_ioctl_reqbufs,
>> + .vidioc_querybuf = vb2_ioctl_querybuf,
>> + .vidioc_qbuf = vb2_ioctl_qbuf,
>> + .vidioc_expbuf = vb2_ioctl_expbuf,
>> + .vidioc_dqbuf = vb2_ioctl_dqbuf,
>> + .vidioc_create_bufs = vb2_ioctl_create_bufs,
>> + .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
>> +
>> + /* Stream on/off */
>> + .vidioc_streamon = vb2_ioctl_streamon,
>> + .vidioc_streamoff = vb2_ioctl_streamoff,
>> +
>> + /* Stream type-dependent parameter ioctls */
>> + .vidioc_g_parm = isp4vid_ioctl_g_param,
>> + .vidioc_s_parm = isp4vid_ioctl_g_param,
>> +
>> + .vidioc_enum_framesizes = isp4vid_enum_framesizes,
>> + .vidioc_enum_frameintervals = isp4vid_ioctl_enum_frameintervals,
>> +
>
> I believe you can drop the above comments; they provide no additional
> information whatsoever. Extra newline above, too.
>
Yes, excatly, will do that.
>> +};
>> +
>> +static unsigned int isp4vid_get_image_size(struct v4l2_pix_format *fmt)
>> +{
>> + switch (fmt->pixelformat) {
>> + case V4L2_PIX_FMT_NV12:
>> + return fmt->width * fmt->height * 3 / 2;
>> + case V4L2_PIX_FMT_YUYV:
>> + return fmt->width * fmt->height * 2;
>> + default:
>> + return 0;
>> + }
>> +}
>> +
>> +static int isp4vid_qops_queue_setup(struct vb2_queue *vq, unsigned int *nbuffers,
>> + unsigned int *nplanes, unsigned int sizes[],
>> + struct device *alloc_devs[])
>> +{
>> + struct isp4vid_dev *isp_vdev = vb2_get_drv_priv(vq);
>> + unsigned int q_num_bufs = vb2_get_num_buffers(vq);
>> +
>> + if (*nplanes > 1) {
>> + dev_err(isp_vdev->dev,
>> + "fail to setup queue, no mplane supported %u\n",
>> + *nplanes);
>> + return -EINVAL;
>> + }
>> +
>> + if (*nplanes == 1) {
>> + unsigned int size;
>> +
>> + size = isp4vid_get_image_size(&isp_vdev->format);
>> + if (sizes[0] < size) {
>> + dev_err(isp_vdev->dev,
>> + "fail for small plane size %u, %u expected\n",
>> + sizes[0], size);
>> + return -EINVAL;
>> + }
>> + }
>> +
>> + if (q_num_bufs + *nbuffers < ISP4IF_MAX_STREAM_BUF_COUNT)
>> + *nbuffers = ISP4IF_MAX_STREAM_BUF_COUNT - q_num_bufs;
>> +
>> + switch (isp_vdev->format.pixelformat) {
>> + case V4L2_PIX_FMT_NV12:
>> + case V4L2_PIX_FMT_YUYV: {
>> + *nplanes = 1;
>> + sizes[0] = max(sizes[0], isp_vdev->format.sizeimage);
>> + isp_vdev->format.sizeimage = sizes[0];
>> + }
>> + break;
>> + default:
>> + dev_err(isp_vdev->dev, "%s|unsupported fmt=%u\n",
>> + isp_vdev->vdev.name, isp_vdev->format.pixelformat);
>> + return -EINVAL;
>> + }
>> +
>> + dev_dbg(isp_vdev->dev, "%s|*nbuffers=%u *nplanes=%u sizes[0]=%u\n",
>> + isp_vdev->vdev.name,
>> + *nbuffers, *nplanes, sizes[0]);
>> +
>> + return 0;
>> +}
>> +
>> +static void isp4vid_qops_buffer_queue(struct vb2_buffer *vb)
>> +{
>> + struct isp4vid_capture_buffer *buf =
>> + container_of(vb, struct isp4vid_capture_buffer, vb2.vb2_buf);
>> + struct isp4vid_dev *isp_vdev = vb2_get_drv_priv(vb->vb2_queue);
>> +
>> + struct isp4vid_vb2_buf *priv_buf = vb->planes[0].mem_priv;
>> + struct isp4if_img_buf_info *img_buf = &buf->img_buf;
>> +
>> + dev_dbg(isp_vdev->dev, "%s|index=%u", isp_vdev->vdev.name, vb->index);
>> +
>> + dev_dbg(isp_vdev->dev, "queue isp user bo 0x%llx size=%lu",
>> + priv_buf->gpu_addr,
>> + priv_buf->size);
>> +
>> + switch (isp_vdev->format.pixelformat) {
>> + case V4L2_PIX_FMT_NV12: {
>> + u32 y_size = isp_vdev->format.sizeimage / 3 * 2;
>> + u32 uv_size = isp_vdev->format.sizeimage / 3;
>> +
>> + img_buf->planes[0].len = y_size;
>> + img_buf->planes[0].sys_addr = priv_buf->vaddr;
>> + img_buf->planes[0].mc_addr = priv_buf->gpu_addr;
>> +
>> + dev_dbg(isp_vdev->dev, "img_buf[0]: mc=0x%llx size=%u",
>> + img_buf->planes[0].mc_addr,
>> + img_buf->planes[0].len);
>> +
>> + img_buf->planes[1].len = uv_size;
>> + img_buf->planes[1].sys_addr = priv_buf->vaddr + y_size;
>> + img_buf->planes[1].mc_addr = priv_buf->gpu_addr + y_size;
>> +
>> + dev_dbg(isp_vdev->dev, "img_buf[1]: mc=0x%llx size=%u",
>> + img_buf->planes[1].mc_addr,
>> + img_buf->planes[1].len);
>> +
>> + img_buf->planes[2].len = 0;
>> + }
>> + break;
>> + case V4L2_PIX_FMT_YUYV: {
>> + img_buf->planes[0].len = isp_vdev->format.sizeimage;
>> + img_buf->planes[0].sys_addr = priv_buf->vaddr;
>> + img_buf->planes[0].mc_addr = priv_buf->gpu_addr;
>> +
>> + dev_dbg(isp_vdev->dev, "img_buf[0]: mc=0x%llx size=%u",
>> + img_buf->planes[0].mc_addr,
>> + img_buf->planes[0].len);
>> +
>> + img_buf->planes[1].len = 0;
>> + img_buf->planes[2].len = 0;
>> + }
>> + break;
>> + default:
>> + dev_err(isp_vdev->dev, "%s|unsupported fmt=%u",
>> + isp_vdev->vdev.name, isp_vdev->format.pixelformat);
>> + return;
>> + }
>> +
>> + if (isp_vdev->stream_started)
>> + isp_vdev->ops->send_buffer(isp_vdev->isp_sdev, img_buf);
>> +
>> + scoped_guard(mutex, &isp_vdev->buf_list_lock)
>> + list_add_tail(&buf->list, &isp_vdev->buf_list);
>> +}
>> +
>> +static int isp4vid_qops_start_streaming(struct vb2_queue *vq, unsigned int count)
>> +{
>> + struct isp4vid_dev *isp_vdev = vb2_get_drv_priv(vq);
>> + struct isp4vid_capture_buffer *isp4vid_buf;
>> + struct media_entity *entity;
>> + struct v4l2_subdev *subdev;
>> + struct media_pad *pad;
>> + int ret = 0;
>> +
>> + isp_vdev->sequence = 0;
>> + ret = v4l2_pipeline_pm_get(&isp_vdev->vdev.entity);
>> + if (ret) {
>> + dev_err(isp_vdev->dev, "power up isp fail %d\n", ret);
>> + goto release_buffers;
>> + }
>> +
>> + entity = &isp_vdev->vdev.entity;
>> + while (1) {
>> + pad = &entity->pads[0];
>> + if (!(pad->flags & MEDIA_PAD_FL_SINK))
>> + break;
>> +
>> + pad = media_pad_remote_pad_first(pad);
>> + if (!pad || !is_media_entity_v4l2_subdev(pad->entity))
>> + break;
>> +
>> + entity = pad->entity;
>> + subdev = media_entity_to_v4l2_subdev(entity);
>> +
>> + ret = v4l2_subdev_call(subdev, video, s_stream, 1);
>> + if (ret < 0 && ret != -ENOIOCTLCMD) {
>> + dev_dbg(isp_vdev->dev, "fail start streaming: %s %d\n",
>> + subdev->name, ret);
>> + goto release_buffers;
>> + }
>> + }
>> +
>> + list_for_each_entry(isp4vid_buf, &isp_vdev->buf_list, list)
>> + isp_vdev->ops->send_buffer(isp_vdev->isp_sdev, &isp4vid_buf->img_buf);
>> +
>> + /* Start the media pipeline */
>> + ret = video_device_pipeline_start(&isp_vdev->vdev, &isp_vdev->pipe);
>> + if (ret) {
>> + dev_err(isp_vdev->dev, "video_device_pipeline_start fail:%d",
>> + ret);
>> + goto release_buffers;
>> + }
>> +
>> + isp_vdev->stream_started = true;
>> +
>> + return 0;
>> +
>> +release_buffers:
>> + isp4vid_capture_return_all_buffers(isp_vdev, VB2_BUF_STATE_QUEUED);
>> + return ret;
>> +}
>> +
>> +static void isp4vid_qops_stop_streaming(struct vb2_queue *vq)
>> +{
>> + struct isp4vid_dev *isp_vdev = vb2_get_drv_priv(vq);
>> + struct v4l2_subdev *subdev;
>> + struct media_entity *entity;
>> + struct media_pad *pad;
>> + int ret;
>> +
>> + entity = &isp_vdev->vdev.entity;
>> + while (1) {
>> + pad = &entity->pads[0];
>> + if (!(pad->flags & MEDIA_PAD_FL_SINK))
>> + break;
>> +
>> + pad = media_pad_remote_pad_first(pad);
>> + if (!pad || !is_media_entity_v4l2_subdev(pad->entity))
>> + break;
>> +
>> + entity = pad->entity;
>> + subdev = media_entity_to_v4l2_subdev(entity);
>> +
>> + ret = v4l2_subdev_call(subdev, video, s_stream, 0);
>> +
>> + if (ret < 0 && ret != -ENOIOCTLCMD)
>> + dev_dbg(isp_vdev->dev, "fail stop streaming: %s %d\n",
>> + subdev->name, ret);
>> + }
>> +
>> + isp_vdev->stream_started = false;
>> + v4l2_pipeline_pm_put(&isp_vdev->vdev.entity);
>
> Please don't add new users of these functions. (Just omit them.)
>
Thanks, will remove calls to these deprecated functions including
v4l2_pipeline_pm_get();
>> +
>> + /* Stop the media pipeline */
>> + video_device_pipeline_stop(&isp_vdev->vdev);
>> +
>> + /* Release all active buffers */
>> + isp4vid_capture_return_all_buffers(isp_vdev, VB2_BUF_STATE_ERROR);
>> +}
>> +
>> +static const struct vb2_ops isp4vid_qops = {
>> + .queue_setup = isp4vid_qops_queue_setup,
>> + .buf_queue = isp4vid_qops_buffer_queue,
>> + .start_streaming = isp4vid_qops_start_streaming,
>> + .stop_streaming = isp4vid_qops_stop_streaming,
>> + .wait_prepare = vb2_ops_wait_prepare,
>> + .wait_finish = vb2_ops_wait_finish,
>> +};
>> +
>> +int isp4vid_dev_init(struct isp4vid_dev *isp_vdev, struct v4l2_subdev *isp_sdev,
>> + const struct isp4vid_ops *ops)
>> +{
>> + const char *vdev_name = isp4vid_video_dev_name;
>> + struct v4l2_device *v4l2_dev;
>> + struct video_device *vdev;
>> + struct vb2_queue *q;
>> + int ret;
>> +
>> + if (!isp_vdev || !isp_sdev || !isp_sdev->v4l2_dev)
>> + return -EINVAL;
>> +
>> + v4l2_dev = isp_sdev->v4l2_dev;
>> + vdev = &isp_vdev->vdev;
>> +
>> + isp_vdev->isp_sdev = isp_sdev;
>> + isp_vdev->dev = v4l2_dev->dev;
>> + isp_vdev->ops = ops;
>> +
>> + /* Initialize the vb2_queue struct */
>> + mutex_init(&isp_vdev->vbq_lock);
>> + q = &isp_vdev->vbq;
>> + q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> + q->io_modes = VB2_MMAP | VB2_DMABUF;
>> + q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
>> + q->buf_struct_size = sizeof(struct isp4vid_capture_buffer);
>> + q->min_queued_buffers = 2;
>> + q->ops = &isp4vid_qops;
>> + q->drv_priv = isp_vdev;
>> + q->mem_ops = &isp4vid_vb2_memops;
>> + q->lock = &isp_vdev->vbq_lock;
>> + q->dev = v4l2_dev->dev;
>> + ret = vb2_queue_init(q);
>> + if (ret) {
>> + dev_err(v4l2_dev->dev, "vb2_queue_init error:%d", ret);
>> + return ret;
>> + }
>> + /* Initialize buffer list and its lock */
>> + mutex_init(&isp_vdev->buf_list_lock);
>> + INIT_LIST_HEAD(&isp_vdev->buf_list);
>> +
>> + /* Set default frame format */
>> + isp_vdev->format = isp4vid_fmt_default;
>> + isp_vdev->timeperframe = ISP4VID_ISP_TPF_DEFAULT;
>> + v4l2_simplify_fraction(&isp_vdev->timeperframe.numerator,
>> + &isp_vdev->timeperframe.denominator, 8, 333);
>> +
>> + ret = isp4vid_fill_buffer_size(&isp_vdev->format);
>> + if (ret) {
>> + dev_err(v4l2_dev->dev, "fail to fill buffer size: %d\n", ret);
>> + goto err_release_vb2_queue;
>> + }
>> +
>> + ret = isp4vid_set_fmt_2_isp(isp_sdev, &isp_vdev->format);
>> + if (ret) {
>> + dev_err(v4l2_dev->dev, "fail init format :%d\n", ret);
>> + goto err_release_vb2_queue;
>> + }
>> +
>> + /* Initialize the video_device struct */
>> + isp_vdev->vdev.entity.name = vdev_name;
>> + isp_vdev->vdev.entity.function = MEDIA_ENT_F_IO_V4L;
>> + isp_vdev->vdev_pad.flags = MEDIA_PAD_FL_SINK;
>> + ret = media_entity_pads_init(&isp_vdev->vdev.entity, 1,
>> + &isp_vdev->vdev_pad);
>> +
>> + if (ret) {
>> + dev_err(v4l2_dev->dev, "init media entity pad fail:%d\n", ret);
>> + goto err_release_vb2_queue;
>> + }
>> +
>> + vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE |
>> + V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
>> + vdev->entity.ops = &isp4vid_vdev_ent_ops;
>> + vdev->release = video_device_release_empty;
>> + vdev->fops = &isp4vid_vdev_fops;
>> + vdev->ioctl_ops = &isp4vid_vdev_ioctl_ops;
>> + vdev->lock = NULL;
>> + vdev->queue = q;
>> + vdev->v4l2_dev = v4l2_dev;
>> + vdev->vfl_dir = VFL_DIR_RX;
>> + strscpy(vdev->name, vdev_name, sizeof(vdev->name));
>> + video_set_drvdata(vdev, isp_vdev);
>> +
>> + ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
>> + if (ret) {
>> + dev_err(v4l2_dev->dev, "register video device fail:%d\n", ret);
>> + goto err_entity_cleanup;
>> + }
>> +
>> + return 0;
>> +
>> +err_entity_cleanup:
>> + media_entity_cleanup(&isp_vdev->vdev.entity);
>> +err_release_vb2_queue:
>> + vb2_queue_release(q);
>> + return ret;
>> +}
>> +
>> +void isp4vid_dev_deinit(struct isp4vid_dev *isp_vdev)
>> +{
>> + vb2_video_unregister_device(&isp_vdev->vdev);
>> +}
>> diff --git a/drivers/media/platform/amd/isp4/isp4_video.h b/drivers/media/platform/amd/isp4/isp4_video.h
>> new file mode 100644
>> index 000000000000..b87316d2a2e5
>> --- /dev/null
>> +++ b/drivers/media/platform/amd/isp4/isp4_video.h
>> @@ -0,0 +1,65 @@
>> +/* SPDX-License-Identifier: GPL-2.0+ */
>> +/*
>> + * Copyright (C) 2025 Advanced Micro Devices, Inc.
>> + */
>> +
>> +#ifndef _ISP4_VIDEO_H_
>> +#define _ISP4_VIDEO_H_
>> +
>> +#include <media/videobuf2-memops.h>
>> +#include <media/v4l2-dev.h>
>> +
>> +#include "isp4_interface.h"
>> +
>> +struct isp4vid_capture_buffer {
>> + /*
>> + * struct vb2_v4l2_buffer must be the first element
>> + * the videobuf2 framework will allocate this struct based on
>> + * buf_struct_size and use the first sizeof(struct vb2_buffer) bytes of
>> + * memory as a vb2_buffer
>> + */
>> + struct vb2_v4l2_buffer vb2;
>> + struct isp4if_img_buf_info img_buf;
>> + struct list_head list;
>> +};
>> +
>> +struct isp4vid_ops {
>> + int (*send_buffer)(struct v4l2_subdev *sd,
>> + struct isp4if_img_buf_info *img_buf);
>
> Is there a reason why isp4sd_ioc_send_img_buf() isn't called directly?
>
In our design, isp4sd serves as the upper layer of video. Therefore,
isp4sd can directly call functions within video, but not vice versa,
This callback mechanism is implemented to support the call from video to
isp4sd by indirect way.
>> +};
>> +
>> +struct isp4vid_dev {
>> + struct video_device vdev;
>> + struct media_pad vdev_pad;
>> + struct v4l2_pix_format format;
>> +
>> + /* mutex that protects vbq */
>> + struct mutex vbq_lock;
>> + struct vb2_queue vbq;
>> +
>> + /* mutex that protects buf_list */
>> + struct mutex buf_list_lock;
>> + struct list_head buf_list;
>> +
>> + u32 sequence;
>> + bool stream_started;
>> +
>> + struct media_pipeline pipe;
>> + struct device *dev;
>> + struct v4l2_subdev *isp_sdev;
>> + struct v4l2_fract timeperframe;
>> +
>> + /* Callback operations */
>> + const struct isp4vid_ops *ops;
>> +};
>> +
>> +int isp4vid_dev_init(struct isp4vid_dev *isp_vdev,
>> + struct v4l2_subdev *isp_sdev,
>> + const struct isp4vid_ops *ops);
>> +
>> +void isp4vid_dev_deinit(struct isp4vid_dev *isp_vdev);
>> +
>> +void isp4vid_handle_frame_done(struct isp4vid_dev *isp_vdev,
>> + const struct isp4if_img_buf_info *img_buf);
>> +
>> +#endif /* _ISP4_VIDEO_H_ */
>
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 5/7] media: platform: amd: isp4 video node and buffers handling added
2026-01-09 10:08 ` Du, Bin
@ 2026-01-29 21:17 ` Sakari Ailus
2026-02-03 6:32 ` Du, Bin
0 siblings, 1 reply; 46+ messages in thread
From: Sakari Ailus @ 2026-01-29 21:17 UTC (permalink / raw)
To: Du, Bin
Cc: Sakari Ailus, mchehab@kernel.org, hverkuil@xs4all.nl,
laurent.pinchart+renesas@ideasonboard.com,
bryan.odonoghue@linaro.org,
prabhakar.mahadev-lad.rj@bp.renesas.com,
linux-media@vger.kernel.org, linux-kernel@vger.kernel.org,
sultan@kerneltoast.com, Nirujogi, Pratap,
Chan, Benjamin (Koon Pan), Li, King, Rosikopulos, Gjorgji,
Jawich, Phil, Antony, Dominic, Limonciello, Mario, Gong, Richard,
Tsao, Anson, Svetoslav Stoilov, Alexey Zagorodnikov
Hi Bin,
On Fri, Jan 09, 2026 at 06:08:00PM +0800, Du, Bin wrote:
> > > +static const struct vb2_mem_ops isp4vid_vb2_memops = {
> > > + .alloc = isp4vid_vb2_alloc,
> > > + .put = isp4vid_vb2_put,
> > > +#ifdef CONFIG_HAS_DMA
> > > + .get_dmabuf = isp4vid_vb2_get_dmabuf,
> > > +#endif
> > > + .map_dmabuf = isp4vid_vb2_map_dmabuf,
> > > + .unmap_dmabuf = isp4vid_vb2_unmap_dmabuf,
> > > + .attach_dmabuf = isp4vid_vb2_attach_dmabuf,
> > > + .detach_dmabuf = isp4vid_vb2_detach_dmabuf,
> > > + .vaddr = isp4vid_vb2_vaddr,
> > > + .mmap = isp4vid_vb2_mmap,
> > > + .num_users = isp4vid_vb2_num_users,
> > > +};
> >
> > Could you elaborate a bit why do you need your own videobuf mem ops?
> >
>
> Sure, ISP FW/HW can access system memory only via GPU VA, not system VA, so
> vb2_vmalloc_memops can't be used directly, we need to base on it to
> implement our own videobuf mem ops to support both GPU VA and system VA.
It's fine to use other virtual addresses than system ones (a lot of other
drivers do in fact, e.g. IPU6), you generally don't need to add new memory
types for this.
...
> > > +struct isp4vid_capture_buffer {
> > > + /*
> > > + * struct vb2_v4l2_buffer must be the first element
> > > + * the videobuf2 framework will allocate this struct based on
> > > + * buf_struct_size and use the first sizeof(struct vb2_buffer) bytes of
> > > + * memory as a vb2_buffer
> > > + */
> > > + struct vb2_v4l2_buffer vb2;
> > > + struct isp4if_img_buf_info img_buf;
> > > + struct list_head list;
> > > +};
> > > +
> > > +struct isp4vid_ops {
> > > + int (*send_buffer)(struct v4l2_subdev *sd,
> > > + struct isp4if_img_buf_info *img_buf);
> >
> > Is there a reason why isp4sd_ioc_send_img_buf() isn't called directly?
> >
>
> In our design, isp4sd serves as the upper layer of video. Therefore, isp4sd
> can directly call functions within video, but not vice versa, This callback
> mechanism is implemented to support the call from video to isp4sd by
> indirect way.
You still have a single module, don't you? Thus you can make a direct
function call. Only use a callback pointer when you actually need one.
>
> > > +};
> > > +
> > > +struct isp4vid_dev {
> > > + struct video_device vdev;
> > > + struct media_pad vdev_pad;
> > > + struct v4l2_pix_format format;
> > > +
> > > + /* mutex that protects vbq */
> > > + struct mutex vbq_lock;
> > > + struct vb2_queue vbq;
> > > +
> > > + /* mutex that protects buf_list */
> > > + struct mutex buf_list_lock;
> > > + struct list_head buf_list;
> > > +
> > > + u32 sequence;
> > > + bool stream_started;
> > > +
> > > + struct media_pipeline pipe;
You might not need this for the time being at least.
> > > + struct device *dev;
> > > + struct v4l2_subdev *isp_sdev;
> > > + struct v4l2_fract timeperframe;
> > > +
> > > + /* Callback operations */
> > > + const struct isp4vid_ops *ops;
> > > +};
> > > +
> > > +int isp4vid_dev_init(struct isp4vid_dev *isp_vdev,
> > > + struct v4l2_subdev *isp_sdev,
> > > + const struct isp4vid_ops *ops);
> > > +
> > > +void isp4vid_dev_deinit(struct isp4vid_dev *isp_vdev);
> > > +
> > > +void isp4vid_handle_frame_done(struct isp4vid_dev *isp_vdev,
> > > + const struct isp4if_img_buf_info *img_buf);
> > > +
> > > +#endif /* _ISP4_VIDEO_H_ */
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 5/7] media: platform: amd: isp4 video node and buffers handling added
2026-01-29 21:17 ` Sakari Ailus
@ 2026-02-03 6:32 ` Du, Bin
0 siblings, 0 replies; 46+ messages in thread
From: Du, Bin @ 2026-02-03 6:32 UTC (permalink / raw)
To: Sakari Ailus
Cc: Sakari Ailus, mchehab@kernel.org, hverkuil@xs4all.nl,
laurent.pinchart+renesas@ideasonboard.com,
bryan.odonoghue@linaro.org,
prabhakar.mahadev-lad.rj@bp.renesas.com,
linux-media@vger.kernel.org, linux-kernel@vger.kernel.org,
sultan@kerneltoast.com, Nirujogi, Pratap,
Chan, Benjamin (Koon Pan), Li, King, Rosikopulos, Gjorgji,
Jawich, Phil, Antony, Dominic, Limonciello, Mario, Gong, Richard,
Tsao, Anson, Svetoslav Stoilov, Alexey Zagorodnikov
Thank you, Sakari. I sincerely appreciate you taking the time from your
busy schedule to provide the valuable feedback.
On 1/30/2026 5:17 AM, Sakari Ailus wrote:
> Hi Bin,
>
> On Fri, Jan 09, 2026 at 06:08:00PM +0800, Du, Bin wrote:
>>>> +static const struct vb2_mem_ops isp4vid_vb2_memops = {
>>>> + .alloc = isp4vid_vb2_alloc,
>>>> + .put = isp4vid_vb2_put,
>>>> +#ifdef CONFIG_HAS_DMA
>>>> + .get_dmabuf = isp4vid_vb2_get_dmabuf,
>>>> +#endif
>>>> + .map_dmabuf = isp4vid_vb2_map_dmabuf,
>>>> + .unmap_dmabuf = isp4vid_vb2_unmap_dmabuf,
>>>> + .attach_dmabuf = isp4vid_vb2_attach_dmabuf,
>>>> + .detach_dmabuf = isp4vid_vb2_detach_dmabuf,
>>>> + .vaddr = isp4vid_vb2_vaddr,
>>>> + .mmap = isp4vid_vb2_mmap,
>>>> + .num_users = isp4vid_vb2_num_users,
>>>> +};
>>>
>>> Could you elaborate a bit why do you need your own videobuf mem ops?
>>>
>>
>> Sure, ISP FW/HW can access system memory only via GPU VA, not system VA, so
>> vb2_vmalloc_memops can't be used directly, we need to base on it to
>> implement our own videobuf mem ops to support both GPU VA and system VA.
>
> It's fine to use other virtual addresses than system ones (a lot of other
> drivers do in fact, e.g. IPU6), you generally don't need to add new memory
> types for this.
>
> ...
>
Thanks for the guidance, will work on it.
>>>> +struct isp4vid_capture_buffer {
>>>> + /*
>>>> + * struct vb2_v4l2_buffer must be the first element
>>>> + * the videobuf2 framework will allocate this struct based on
>>>> + * buf_struct_size and use the first sizeof(struct vb2_buffer) bytes of
>>>> + * memory as a vb2_buffer
>>>> + */
>>>> + struct vb2_v4l2_buffer vb2;
>>>> + struct isp4if_img_buf_info img_buf;
>>>> + struct list_head list;
>>>> +};
>>>> +
>>>> +struct isp4vid_ops {
>>>> + int (*send_buffer)(struct v4l2_subdev *sd,
>>>> + struct isp4if_img_buf_info *img_buf);
>>>
>>> Is there a reason why isp4sd_ioc_send_img_buf() isn't called directly?
>>>
>>
>> In our design, isp4sd serves as the upper layer of video. Therefore, isp4sd
>> can directly call functions within video, but not vice versa, This callback
>> mechanism is implemented to support the call from video to isp4sd by
>> indirect way.
>
> You still have a single module, don't you? Thus you can make a direct
> function call. Only use a callback pointer when you actually need one.
>
Yes, exactly, will switch to direct function call which is more
straightforward.
>>
>>>> +};
>>>> +
>>>> +struct isp4vid_dev {
>>>> + struct video_device vdev;
>>>> + struct media_pad vdev_pad;
>>>> + struct v4l2_pix_format format;
>>>> +
>>>> + /* mutex that protects vbq */
>>>> + struct mutex vbq_lock;
>>>> + struct vb2_queue vbq;
>>>> +
>>>> + /* mutex that protects buf_list */
>>>> + struct mutex buf_list_lock;
>>>> + struct list_head buf_list;
>>>> +
>>>> + u32 sequence;
>>>> + bool stream_started;
>>>> +
>>>> + struct media_pipeline pipe;
>
> You might not need this for the time being at least.
>
Sure, will drop it.
>>>> + struct device *dev;
>>>> + struct v4l2_subdev *isp_sdev;
>>>> + struct v4l2_fract timeperframe;
>>>> +
>>>> + /* Callback operations */
>>>> + const struct isp4vid_ops *ops;
>>>> +};
>>>> +
>>>> +int isp4vid_dev_init(struct isp4vid_dev *isp_vdev,
>>>> + struct v4l2_subdev *isp_sdev,
>>>> + const struct isp4vid_ops *ops);
>>>> +
>>>> +void isp4vid_dev_deinit(struct isp4vid_dev *isp_vdev);
>>>> +
>>>> +void isp4vid_handle_frame_done(struct isp4vid_dev *isp_vdev,
>>>> + const struct isp4if_img_buf_info *img_buf);
>>>> +
>>>> +#endif /* _ISP4_VIDEO_H_ */
>
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread
* [PATCH v7 6/7] media: platform: amd: isp4 debug fs logging and more descriptive errors
2025-12-16 9:13 [PATCH v7 0/7] Add AMD ISP4 driver Bin Du
` (4 preceding siblings ...)
2025-12-16 9:13 ` [PATCH v7 5/7] media: platform: amd: isp4 video node and buffers " Bin Du
@ 2025-12-16 9:13 ` Bin Du
2025-12-16 9:13 ` [PATCH v7 7/7] Documentation: add documentation of AMD isp 4 driver Bin Du
` (3 subsequent siblings)
9 siblings, 0 replies; 46+ messages in thread
From: Bin Du @ 2025-12-16 9:13 UTC (permalink / raw)
To: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
sultan
Cc: pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, bin.du, Bin Du, Svetoslav Stoilov,
Alexey Zagorodnikov
Add debug fs for isp4 driver and add more detailed descriptive error info
to some of the log message
Co-developed-by: Sultan Alsawaf <sultan@kerneltoast.com>
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
Signed-off-by: Bin Du <Bin.Du@amd.com>
Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
---
MAINTAINERS | 2 +
drivers/media/platform/amd/isp4/Makefile | 1 +
drivers/media/platform/amd/isp4/isp4.c | 4 +
drivers/media/platform/amd/isp4/isp4_debug.c | 271 ++++++++++++++++++
drivers/media/platform/amd/isp4/isp4_debug.h | 41 +++
.../media/platform/amd/isp4/isp4_interface.c | 21 +-
drivers/media/platform/amd/isp4/isp4_subdev.c | 29 +-
drivers/media/platform/amd/isp4/isp4_subdev.h | 5 +
8 files changed, 356 insertions(+), 18 deletions(-)
create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c
create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 80c966fde0b4..8478789ac265 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1145,6 +1145,8 @@ F: drivers/media/platform/amd/isp4/Kconfig
F: drivers/media/platform/amd/isp4/Makefile
F: drivers/media/platform/amd/isp4/isp4.c
F: drivers/media/platform/amd/isp4/isp4.h
+F: drivers/media/platform/amd/isp4/isp4_debug.c
+F: drivers/media/platform/amd/isp4/isp4_debug.h
F: drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
F: drivers/media/platform/amd/isp4/isp4_hw_reg.h
F: drivers/media/platform/amd/isp4/isp4_interface.c
diff --git a/drivers/media/platform/amd/isp4/Makefile b/drivers/media/platform/amd/isp4/Makefile
index 398c20ea7866..607151c0a2be 100644
--- a/drivers/media/platform/amd/isp4/Makefile
+++ b/drivers/media/platform/amd/isp4/Makefile
@@ -4,6 +4,7 @@
obj-$(CONFIG_AMD_ISP4) += amd_capture.o
amd_capture-objs := isp4.o \
+ isp4_debug.o \
isp4_interface.o \
isp4_subdev.o \
isp4_video.o
diff --git a/drivers/media/platform/amd/isp4/isp4.c b/drivers/media/platform/amd/isp4/isp4.c
index b5470f6dc93e..93cb2d41320e 100644
--- a/drivers/media/platform/amd/isp4/isp4.c
+++ b/drivers/media/platform/amd/isp4/isp4.c
@@ -10,6 +10,7 @@
#include <media/v4l2-ioctl.h>
#include "isp4.h"
+#include "isp4_debug.h"
#include "isp4_hw_reg.h"
#define ISP4_DRV_NAME "amd_isp_capture"
@@ -186,6 +187,7 @@ static int isp4_capture_probe(struct platform_device *pdev)
}
platform_set_drvdata(pdev, isp_dev);
+ isp_debugfs_create(isp_dev);
return 0;
@@ -205,6 +207,8 @@ static void isp4_capture_remove(struct platform_device *pdev)
struct isp4_device *isp_dev = platform_get_drvdata(pdev);
struct device *dev = &pdev->dev;
+ isp_debugfs_remove(isp_dev);
+
media_device_unregister(&isp_dev->mdev);
isp4sd_deinit(&isp_dev->isp_subdev);
pm_runtime_disable(dev);
diff --git a/drivers/media/platform/amd/isp4/isp4_debug.c b/drivers/media/platform/amd/isp4/isp4_debug.c
new file mode 100644
index 000000000000..c6d957ea9132
--- /dev/null
+++ b/drivers/media/platform/amd/isp4/isp4_debug.c
@@ -0,0 +1,271 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2025 Advanced Micro Devices, Inc.
+ */
+
+#include "isp4.h"
+#include "isp4_debug.h"
+#include "isp4_hw_reg.h"
+#include "isp4_interface.h"
+
+#define ISP4DBG_FW_LOG_RINGBUF_SIZE (2 * 1024 * 1024)
+#define ISP4DBG_MACRO_2_STR(X) #X
+#define ISP4DBG_MAX_ONE_TIME_LOG_LEN 510
+
+#ifdef CONFIG_DEBUG_FS
+
+void isp_debugfs_create(struct isp4_device *isp_dev)
+{
+ isp_dev->isp_subdev.debugfs_dir = debugfs_create_dir("amd_isp", NULL);
+ debugfs_create_bool("fw_log_enable", 0644,
+ isp_dev->isp_subdev.debugfs_dir,
+ &isp_dev->isp_subdev.enable_fw_log);
+ isp_dev->isp_subdev.fw_log_output =
+ devm_kzalloc(isp_dev->isp_subdev.dev,
+ ISP4DBG_FW_LOG_RINGBUF_SIZE + 32,
+ GFP_KERNEL);
+}
+
+void isp_debugfs_remove(struct isp4_device *isp_dev)
+{
+ debugfs_remove_recursive(isp_dev->isp_subdev.debugfs_dir);
+ isp_dev->isp_subdev.debugfs_dir = NULL;
+}
+
+static u32 isp_fw_fill_rb_log(struct isp4_subdev *isp, void *sys, u32 rb_size)
+{
+ struct isp4_interface *ispif = &isp->ispif;
+ char *buf = isp->fw_log_output;
+ struct device *dev = isp->dev;
+ u32 rd_ptr, wr_ptr;
+ u32 total_cnt = 0;
+ u32 offset = 0;
+ u32 cnt;
+
+ if (!sys || !rb_size)
+ return 0;
+
+ guard(mutex)(&ispif->isp4if_mutex);
+
+ rd_ptr = isp4hw_rreg(isp->mmio, ISP_LOG_RB_RPTR0);
+ wr_ptr = isp4hw_rreg(isp->mmio, ISP_LOG_RB_WPTR0);
+
+ do {
+ if (wr_ptr > rd_ptr)
+ cnt = wr_ptr - rd_ptr;
+ else if (wr_ptr < rd_ptr)
+ cnt = rb_size - rd_ptr;
+ else
+ goto quit;
+
+ if (cnt > rb_size) {
+ dev_err(dev, "fail bad fw log size %u\n", cnt);
+ goto quit;
+ }
+
+ memcpy(buf + offset, sys + rd_ptr, cnt);
+
+ offset += cnt;
+ total_cnt += cnt;
+ rd_ptr = (rd_ptr + cnt) % rb_size;
+ } while (rd_ptr < wr_ptr);
+
+ isp4hw_wreg(isp->mmio, ISP_LOG_RB_RPTR0, rd_ptr);
+
+quit:
+ return total_cnt;
+}
+
+void isp_fw_log_print(struct isp4_subdev *isp)
+{
+ struct isp4_interface *ispif = &isp->ispif;
+ char *fw_log_buf = isp->fw_log_output;
+ u32 cnt;
+
+ if (!isp->enable_fw_log || !fw_log_buf)
+ return;
+
+ cnt = isp_fw_fill_rb_log(isp, ispif->fw_log_buf->sys_addr,
+ ispif->fw_log_buf->mem_size);
+
+ if (cnt) {
+ char *line_end;
+ char temp_ch;
+ char *str;
+ char *end;
+
+ str = (char *)fw_log_buf;
+ end = ((char *)fw_log_buf + cnt);
+ fw_log_buf[cnt] = 0;
+
+ while (str < end) {
+ line_end = strchr(str, 0x0A);
+ if ((line_end && (str + ISP4DBG_MAX_ONE_TIME_LOG_LEN) >= line_end) ||
+ (!line_end && (str + ISP4DBG_MAX_ONE_TIME_LOG_LEN) >= end)) {
+ if (line_end)
+ *line_end = 0;
+
+ if (*str != '\0')
+ dev_dbg(isp->dev,
+ "%s", str);
+
+ if (line_end) {
+ *line_end = 0x0A;
+ str = line_end + 1;
+ } else {
+ break;
+ }
+ } else {
+ u32 tmp_len = ISP4DBG_MAX_ONE_TIME_LOG_LEN;
+
+ temp_ch = str[tmp_len];
+ str[tmp_len] = 0;
+ dev_dbg(isp->dev, "%s", str);
+ str[tmp_len] = temp_ch;
+ str = &str[tmp_len];
+ }
+ }
+ }
+}
+#endif
+
+char *isp4dbg_get_buf_src_str(u32 src)
+{
+ switch (src) {
+ case BUFFER_SOURCE_STREAM:
+ return ISP4DBG_MACRO_2_STR(BUFFER_SOURCE_STREAM);
+ default:
+ return "Unknown buf source";
+ }
+}
+
+char *isp4dbg_get_buf_done_str(u32 status)
+{
+ switch (status) {
+ case BUFFER_STATUS_INVALID:
+ return ISP4DBG_MACRO_2_STR(BUFFER_STATUS_INVALID);
+ case BUFFER_STATUS_SKIPPED:
+ return ISP4DBG_MACRO_2_STR(BUFFER_STATUS_SKIPPED);
+ case BUFFER_STATUS_EXIST:
+ return ISP4DBG_MACRO_2_STR(BUFFER_STATUS_EXIST);
+ case BUFFER_STATUS_DONE:
+ return ISP4DBG_MACRO_2_STR(BUFFER_STATUS_DONE);
+ case BUFFER_STATUS_LACK:
+ return ISP4DBG_MACRO_2_STR(BUFFER_STATUS_LACK);
+ case BUFFER_STATUS_DIRTY:
+ return ISP4DBG_MACRO_2_STR(BUFFER_STATUS_DIRTY);
+ case BUFFER_STATUS_MAX:
+ return ISP4DBG_MACRO_2_STR(BUFFER_STATUS_MAX);
+ default:
+ return "Unknown Buf Done Status";
+ }
+}
+
+char *isp4dbg_get_img_fmt_str(int fmt /* enum isp4fw_image_format * */)
+{
+ switch (fmt) {
+ case IMAGE_FORMAT_NV12:
+ return "NV12";
+ case IMAGE_FORMAT_YUV422INTERLEAVED:
+ return "YUV422INTERLEAVED";
+ default:
+ return "unknown fmt";
+ }
+}
+
+void isp4dbg_show_bufmeta_info(struct device *dev, char *pre,
+ void *in, void *orig_buf)
+{
+ struct isp4fw_buffer_meta_info *p;
+ struct isp4if_img_buf_info *orig;
+
+ if (!in)
+ return;
+
+ if (!pre)
+ pre = "";
+
+ p = in;
+ orig = orig_buf;
+
+ dev_dbg(dev, "%s(%s) en:%d,stat:%s(%u),src:%s\n", pre,
+ isp4dbg_get_img_fmt_str(p->image_prop.image_format),
+ p->enabled, isp4dbg_get_buf_done_str(p->status), p->status,
+ isp4dbg_get_buf_src_str(p->source));
+
+ dev_dbg(dev, "%p,0x%llx(%u) %p,0x%llx(%u) %p,0x%llx(%u)\n",
+ orig->planes[0].sys_addr, orig->planes[0].mc_addr,
+ orig->planes[0].len, orig->planes[1].sys_addr,
+ orig->planes[1].mc_addr, orig->planes[1].len,
+ orig->planes[2].sys_addr, orig->planes[2].mc_addr,
+ orig->planes[2].len);
+}
+
+char *isp4dbg_get_buf_type(u32 type)
+{
+ /* enum isp4fw_buffer_type */
+ switch (type) {
+ case BUFFER_TYPE_PREVIEW:
+ return ISP4DBG_MACRO_2_STR(BUFFER_TYPE_PREVIEW);
+ case BUFFER_TYPE_META_INFO:
+ return ISP4DBG_MACRO_2_STR(BUFFER_TYPE_META_INFO);
+ case BUFFER_TYPE_MEM_POOL:
+ return ISP4DBG_MACRO_2_STR(BUFFER_TYPE_MEM_POOL);
+ default:
+ return "unknown type";
+ }
+}
+
+char *isp4dbg_get_cmd_str(u32 cmd)
+{
+ switch (cmd) {
+ case CMD_ID_START_STREAM:
+ return ISP4DBG_MACRO_2_STR(CMD_ID_START_STREAM);
+ case CMD_ID_STOP_STREAM:
+ return ISP4DBG_MACRO_2_STR(CMD_ID_STOP_STREAM);
+ case CMD_ID_SEND_BUFFER:
+ return ISP4DBG_MACRO_2_STR(CMD_ID_SEND_BUFFER);
+ case CMD_ID_SET_STREAM_CONFIG:
+ return ISP4DBG_MACRO_2_STR(CMD_ID_SET_STREAM_CONFIG);
+ case CMD_ID_SET_OUT_CHAN_PROP:
+ return ISP4DBG_MACRO_2_STR(CMD_ID_SET_OUT_CHAN_PROP);
+ case CMD_ID_ENABLE_OUT_CHAN:
+ return ISP4DBG_MACRO_2_STR(CMD_ID_ENABLE_OUT_CHAN);
+ default:
+ return "unknown cmd";
+ }
+}
+
+char *isp4dbg_get_resp_str(u32 cmd)
+{
+ switch (cmd) {
+ case RESP_ID_CMD_DONE:
+ return ISP4DBG_MACRO_2_STR(RESP_ID_CMD_DONE);
+ case RESP_ID_NOTI_FRAME_DONE:
+ return ISP4DBG_MACRO_2_STR(RESP_ID_NOTI_FRAME_DONE);
+ default:
+ return "unknown respid";
+ }
+}
+
+char *isp4dbg_get_if_stream_str(u32 stream /* enum fw_cmd_resp_stream_id */)
+{
+ switch (stream) {
+ case ISP4IF_STREAM_ID_GLOBAL:
+ return "STREAM_GLOBAL";
+ case ISP4IF_STREAM_ID_1:
+ return "STREAM1";
+ default:
+ return "unknown streamID";
+ }
+}
+
+char *isp4dbg_get_out_ch_str(int ch /* enum isp4fw_pipe_out_ch */)
+{
+ switch ((enum isp4fw_pipe_out_ch)ch) {
+ case ISP_PIPE_OUT_CH_PREVIEW:
+ return "prev";
+ default:
+ return "unknown channel";
+ }
+}
diff --git a/drivers/media/platform/amd/isp4/isp4_debug.h b/drivers/media/platform/amd/isp4/isp4_debug.h
new file mode 100644
index 000000000000..1a13762af502
--- /dev/null
+++ b/drivers/media/platform/amd/isp4/isp4_debug.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (C) 2025 Advanced Micro Devices, Inc.
+ */
+
+#ifndef _ISP4_DEBUG_H_
+#define _ISP4_DEBUG_H_
+
+#include <linux/printk.h>
+#include <linux/dev_printk.h>
+
+#include "isp4_subdev.h"
+
+#ifdef CONFIG_DEBUG_FS
+struct isp4_device;
+
+void isp_debugfs_create(struct isp4_device *isp_dev);
+void isp_debugfs_remove(struct isp4_device *isp_dev);
+void isp_fw_log_print(struct isp4_subdev *isp);
+
+#else
+
+/* to avoid checkpatch warning */
+#define isp_debugfs_create(cam) ((void)(cam))
+#define isp_debugfs_remove(cam) ((void)(cam))
+#define isp_fw_log_print(isp) ((void)(isp))
+
+#endif /* CONFIG_DEBUG_FS */
+
+void isp4dbg_show_bufmeta_info(struct device *dev, char *pre, void *p,
+ void *orig_buf /* struct sys_img_buf_handle */);
+char *isp4dbg_get_img_fmt_str(int fmt /* enum _image_format_t */);
+char *isp4dbg_get_out_ch_str(int ch /* enum _isp_pipe_out_ch_t */);
+char *isp4dbg_get_cmd_str(u32 cmd);
+char *isp4dbg_get_buf_type(u32 type);/* enum _buffer_type_t */
+char *isp4dbg_get_resp_str(u32 resp);
+char *isp4dbg_get_buf_src_str(u32 src);
+char *isp4dbg_get_buf_done_str(u32 status);
+char *isp4dbg_get_if_stream_str(u32 stream);
+
+#endif /* _ISP4_DEBUG_H_ */
diff --git a/drivers/media/platform/amd/isp4/isp4_interface.c b/drivers/media/platform/amd/isp4/isp4_interface.c
index 914a93f9652e..c3b19bab7a02 100644
--- a/drivers/media/platform/amd/isp4/isp4_interface.c
+++ b/drivers/media/platform/amd/isp4/isp4_interface.c
@@ -5,6 +5,7 @@
#include <linux/iopoll.h>
+#include "isp4_debug.h"
#include "isp4_fw_cmd_resp.h"
#include "isp4_hw_reg.h"
#include "isp4_interface.h"
@@ -296,8 +297,8 @@ static int isp4if_insert_isp_fw_cmd(struct isp4_interface *ispif, enum isp4if_st
rd_ptr = isp4hw_rreg(ispif->mmio, rreg);
wr_ptr = isp4hw_rreg(ispif->mmio, wreg);
if (rd_ptr >= len || wr_ptr >= len) {
- dev_err(dev, "rb invalid: stream=%u, rd=%u, wr=%u, len=%u, cmd_sz=%u\n",
- stream, rd_ptr, wr_ptr, len, cmd_sz);
+ dev_err(dev, "rb invalid: stream=%u(%s), rd=%u, wr=%u, len=%u, cmd_sz=%u\n",
+ stream, isp4dbg_get_if_stream_str(stream), rd_ptr, wr_ptr, len, cmd_sz);
return -EINVAL;
}
@@ -378,7 +379,8 @@ static int isp4if_send_fw_cmd(struct isp4_interface *ispif, u32 cmd_id, const vo
u32 wr_ptr = isp4hw_rreg(ispif->mmio, rb_config->reg_wptr);
dev_err(dev,
- "failed to get free cmdq slot, stream (%d),rd %u, wr %u\n",
+ "failed to get free cmdq slot, stream %s(%d),rd %u, wr %u\n",
+ isp4dbg_get_if_stream_str(stream),
stream, rd_ptr, wr_ptr);
ret = -ETIMEDOUT;
goto free_ele;
@@ -402,7 +404,8 @@ static int isp4if_send_fw_cmd(struct isp4_interface *ispif, u32 cmd_id, const vo
ret = isp4if_insert_isp_fw_cmd(ispif, stream, &cmd);
if (ret) {
- dev_err(dev, "fail for insert_isp_fw_cmd cmd_id (0x%08x)\n", cmd_id);
+ dev_err(dev, "fail for insert_isp_fw_cmd cmd_id %s(0x%08x)\n",
+ isp4dbg_get_cmd_str(cmd_id), cmd_id);
goto err_dequeue_ele;
}
}
@@ -650,17 +653,17 @@ int isp4if_f2h_resp(struct isp4_interface *ispif, enum isp4if_stream_id stream,
if (checksum != resp->resp_check_sum) {
dev_err(dev, "resp checksum 0x%x,should 0x%x,rptr %u,wptr %u\n",
checksum, resp->resp_check_sum, rd_ptr, wr_ptr);
- dev_err(dev, "(%u), seqNo %u, resp_id (0x%x)\n",
- stream, resp->resp_seq_num,
- resp->resp_id);
+ dev_err(dev, "%s(%u), seqNo %u, resp_id %s(0x%x)\n",
+ isp4dbg_get_if_stream_str(stream), stream, resp->resp_seq_num,
+ isp4dbg_get_resp_str(resp->resp_id), resp->resp_id);
return -EINVAL;
}
return 0;
err_rb_invalid:
- dev_err(dev, "rb invalid: stream=%u, rd=%u, wr=%u, len=%u, resp_sz=%u\n",
- stream, rd_ptr, wr_ptr, len, resp_sz);
+ dev_err(dev, "rb invalid: stream=%u(%s), rd=%u, wr=%u, len=%u, resp_sz=%u\n",
+ stream, isp4dbg_get_if_stream_str(stream), rd_ptr, wr_ptr, len, resp_sz);
return -EINVAL;
}
diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.c b/drivers/media/platform/amd/isp4/isp4_subdev.c
index 8c33187efc54..21de6bc7fce0 100644
--- a/drivers/media/platform/amd/isp4/isp4_subdev.c
+++ b/drivers/media/platform/amd/isp4/isp4_subdev.c
@@ -6,6 +6,7 @@
#include <linux/pm_domain.h>
#include <linux/units.h>
+#include "isp4_debug.h"
#include "isp4_fw_cmd_resp.h"
#include "isp4_interface.h"
#include "isp4.h"
@@ -255,9 +256,9 @@ static int isp4sd_setup_output(struct isp4_subdev *isp_subdev,
return -EINVAL;
}
- dev_dbg(dev, "channel:%d,fmt %d,w:h=%u:%u,lp:%u,cp%u\n",
- cmd_ch_prop.ch,
- cmd_ch_prop.image_prop.image_format,
+ dev_dbg(dev, "channel:%s,fmt %s,w:h=%u:%u,lp:%u,cp%u\n",
+ isp4dbg_get_out_ch_str(cmd_ch_prop.ch),
+ isp4dbg_get_img_fmt_str(cmd_ch_prop.image_prop.image_format),
cmd_ch_prop.image_prop.width, cmd_ch_prop.image_prop.height,
cmd_ch_prop.image_prop.luma_pitch,
cmd_ch_prop.image_prop.chroma_pitch);
@@ -285,7 +286,7 @@ static int isp4sd_setup_output(struct isp4_subdev *isp_subdev,
return ret;
}
- dev_dbg(dev, "enable channel %d\n", cmd_ch_en.ch);
+ dev_dbg(dev, "enable channel %s\n", isp4dbg_get_out_ch_str(cmd_ch_en.ch));
if (!sensor_info->start_stream_cmd_sent) {
ret = isp4sd_kickoff_stream(isp_subdev,
@@ -373,8 +374,9 @@ static void isp4sd_fw_resp_cmd_done(struct isp4_subdev *isp_subdev,
isp4if_rm_cmd_from_cmdq(ispif, para->cmd_seq_num, para->cmd_id);
struct device *dev = isp_subdev->dev;
- dev_dbg(dev, "stream %d,cmd (0x%08x)(%d),seq %u, ele %p\n",
+ dev_dbg(dev, "stream %d,cmd %s(0x%08x)(%d),seq %u, ele %p\n",
stream_id,
+ isp4dbg_get_cmd_str(para->cmd_id),
para->cmd_id, para->cmd_status, para->cmd_seq_num,
ele);
@@ -449,9 +451,10 @@ static void isp4sd_fw_resp_frame_done(struct isp4_subdev *isp_subdev,
return;
}
- dev_dbg(dev, "ts:%llu,streamId:%d,poc:%u,preview_en:%u,(%i)\n",
+ dev_dbg(dev, "ts:%llu,streamId:%d,poc:%u,preview_en:%u,%s(%i)\n",
ktime_get_ns(), stream_id, meta->poc,
meta->preview.enabled,
+ isp4dbg_get_buf_done_str(meta->preview.status),
meta->preview.status);
if (meta->preview.enabled &&
@@ -460,6 +463,8 @@ static void isp4sd_fw_resp_frame_done(struct isp4_subdev *isp_subdev,
meta->preview.status == BUFFER_STATUS_DIRTY)) {
prev = isp4if_dequeue_buffer(ispif);
if (prev) {
+ isp4dbg_show_bufmeta_info(dev, "prev", &meta->preview,
+ &prev->buf_info);
isp4vid_handle_frame_done(&isp_subdev->isp_vdev,
&prev->buf_info);
isp4if_dealloc_buffer_node(prev);
@@ -467,8 +472,9 @@ static void isp4sd_fw_resp_frame_done(struct isp4_subdev *isp_subdev,
dev_err(dev, "fail null prev buf\n");
}
} else if (meta->preview.enabled) {
- dev_err(dev, "fail bad preview status %u\n",
- meta->preview.status);
+ dev_err(dev, "fail bad preview status %u(%s)\n",
+ meta->preview.status,
+ isp4dbg_get_buf_done_str(meta->preview.status));
}
if (isp_subdev->sensor_info.status == ISP4SD_START_STATUS_STARTED)
@@ -485,6 +491,9 @@ static void isp4sd_fw_resp_func(struct isp4_subdev *isp_subdev,
struct device *dev = isp_subdev->dev;
struct isp4fw_resp resp;
+ if (stream_id == ISP4IF_STREAM_ID_1)
+ isp_fw_log_print(isp_subdev);
+
while (true) {
if (isp4if_f2h_resp(ispif, stream_id, &resp)) {
/* Re-enable the interrupt */
@@ -513,7 +522,9 @@ static void isp4sd_fw_resp_func(struct isp4_subdev *isp_subdev,
&resp.param.frame_done);
break;
default:
- dev_err(dev, "-><- fail respid (0x%x)\n", resp.resp_id);
+ dev_err(dev, "-><- fail respid %s(0x%x)\n",
+ isp4dbg_get_resp_str(resp.resp_id),
+ resp.resp_id);
break;
}
}
diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.h b/drivers/media/platform/amd/isp4/isp4_subdev.h
index 386e4e68f3fa..942245a6f356 100644
--- a/drivers/media/platform/amd/isp4/isp4_subdev.h
+++ b/drivers/media/platform/amd/isp4/isp4_subdev.h
@@ -117,6 +117,11 @@ struct isp4_subdev {
bool irq_enabled;
/* spin lock to access ISP_SYS_INT0_EN exclusively */
spinlock_t irq_lock;
+#ifdef CONFIG_DEBUG_FS
+ bool enable_fw_log;
+ struct dentry *debugfs_dir;
+ char *fw_log_output;
+#endif
};
int isp4sd_init(struct isp4_subdev *isp_subdev, struct v4l2_device *v4l2_dev,
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread* [PATCH v7 7/7] Documentation: add documentation of AMD isp 4 driver
2025-12-16 9:13 [PATCH v7 0/7] Add AMD ISP4 driver Bin Du
` (5 preceding siblings ...)
2025-12-16 9:13 ` [PATCH v7 6/7] media: platform: amd: isp4 debug fs logging and more descriptive errors Bin Du
@ 2025-12-16 9:13 ` Bin Du
2025-12-17 7:36 ` [PATCH v7 0/7] Add AMD ISP4 driver Sultan Alsawaf
` (2 subsequent siblings)
9 siblings, 0 replies; 46+ messages in thread
From: Bin Du @ 2025-12-16 9:13 UTC (permalink / raw)
To: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
sultan
Cc: pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao, bin.du, Bin Du, Svetoslav Stoilov, Mario Limonciello,
Alexey Zagorodnikov
Add documentation for AMD ISP 4 and describe the main components
Co-developed-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
Signed-off-by: Svetoslav Stoilov <Svetoslav.Stoilov@amd.com>
Signed-off-by: Bin Du <Bin.Du@amd.com>
Reviewed-by: Mario Limonciello (AMD) <superm1@kernel.org>
Tested-by: Alexey Zagorodnikov <xglooom@gmail.com>
---
Documentation/admin-guide/media/amdisp4-1.rst | 63 +++++++++++++++++++
Documentation/admin-guide/media/amdisp4.dot | 6 ++
.../admin-guide/media/v4l-drivers.rst | 1 +
MAINTAINERS | 2 +
4 files changed, 72 insertions(+)
create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst
create mode 100644 Documentation/admin-guide/media/amdisp4.dot
diff --git a/Documentation/admin-guide/media/amdisp4-1.rst b/Documentation/admin-guide/media/amdisp4-1.rst
new file mode 100644
index 000000000000..878141154f96
--- /dev/null
+++ b/Documentation/admin-guide/media/amdisp4-1.rst
@@ -0,0 +1,63 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+.. include:: <isonum.txt>
+
+====================================
+AMD Image Signal Processor (amdisp4)
+====================================
+
+Introduction
+============
+
+This file documents the driver for the AMD ISP4 that is part of
+AMD Ryzen AI Max 300 Series.
+
+The driver is located under drivers/media/platform/amd/isp4 and uses
+the Media-Controller API.
+
+The driver exposes one video capture device to userspace and provide
+web camera like interface. Internally the video device is connected
+to the isp4 sub-device responsible for communication with the CCPU FW.
+
+Topology
+========
+
+.. _amdisp4_topology_graph:
+
+.. kernel-figure:: amdisp4.dot
+ :alt: Diagram of the media pipeline topology
+ :align: center
+
+
+
+The driver has 1 sub-device: Representing isp4 image signal processor.
+The driver has 1 video device: Capture device for retrieving images.
+
+- ISP4 Image Signal Processing Subdevice Node
+
+---------------------------------------------
+
+The isp4 is represented as a single V4L2 subdev, the sub-device does not
+provide interface to the user space. The sub-device is connected to one video node
+(isp4_capture) with immutable active link. The sub-device represents ISP with
+connected sensor similar to smart cameras (sensors with integrated ISP).
+sub-device has only one link to the video device for capturing the frames.
+The sub-device communicates with CCPU FW for streaming configuration and
+buffer management.
+
+
+- isp4_capture - Frames Capture Video Node
+
+------------------------------------------
+
+Isp4_capture is a capture device to capture frames to memory.
+The entity is connected to isp4 sub-device. The video device
+provides web camera like interface to userspace. It supports
+mmap and dma buf types of memory.
+
+Capturing Video Frames Example
+==============================
+
+.. code-block:: bash
+
+ v4l2-ctl "-d" "/dev/video0" "--set-fmt-video=width=1920,height=1080,pixelformat=NV12" "--stream-mmap" "--stream-count=10"
diff --git a/Documentation/admin-guide/media/amdisp4.dot b/Documentation/admin-guide/media/amdisp4.dot
new file mode 100644
index 000000000000..978f30c1a31a
--- /dev/null
+++ b/Documentation/admin-guide/media/amdisp4.dot
@@ -0,0 +1,6 @@
+digraph board {
+ rankdir=TB
+ n00000001 [label="{{} | amd isp4\n | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
+ n00000001:port0 -> n00000003 [style=bold]
+ n00000003 [label="Preview\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
+}
diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst
index 3bac5165b134..6027416e5373 100644
--- a/Documentation/admin-guide/media/v4l-drivers.rst
+++ b/Documentation/admin-guide/media/v4l-drivers.rst
@@ -9,6 +9,7 @@ Video4Linux (V4L) driver-specific documentation
.. toctree::
:maxdepth: 2
+ amdisp4-1
bttv
c3-isp
cafe_ccic
diff --git a/MAINTAINERS b/MAINTAINERS
index 8478789ac265..c34137e27b55 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1139,6 +1139,8 @@ M: Nirujogi Pratap <pratap.nirujogi@amd.com>
L: linux-media@vger.kernel.org
S: Maintained
T: git git://linuxtv.org/media.git
+F: Documentation/admin-guide/media/amdisp4-1.rst
+F: Documentation/admin-guide/media/amdisp4.dot
F: drivers/media/platform/amd/Kconfig
F: drivers/media/platform/amd/Makefile
F: drivers/media/platform/amd/isp4/Kconfig
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread* Re: [PATCH v7 0/7] Add AMD ISP4 driver
2025-12-16 9:13 [PATCH v7 0/7] Add AMD ISP4 driver Bin Du
` (6 preceding siblings ...)
2025-12-16 9:13 ` [PATCH v7 7/7] Documentation: add documentation of AMD isp 4 driver Bin Du
@ 2025-12-17 7:36 ` Sultan Alsawaf
2025-12-17 7:47 ` Du, Bin
2025-12-17 10:29 ` Du, Bin
2025-12-31 9:03 ` Kate Hsuan
9 siblings, 1 reply; 46+ messages in thread
From: Sultan Alsawaf @ 2025-12-17 7:36 UTC (permalink / raw)
To: Bin Du
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao
On Tue, Dec 16, 2025 at 05:13:19PM +0800, Bin Du wrote:
> Hello,
>
> AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC (Ryzen AI Max 300 Series)
> (https://ubuntu.com/certified/202411-36043)
> This patch series introduces the initial driver support for the AMD ISP4.
>
> Patch summary:
> - Powers up/off and initializes ISP HW
> - Configures and kicks off ISP FW
> - Interacts with APP using standard V4l2 interface by video node
> - Controls ISP HW and interacts with ISP FW to do image processing
> - Supports enum/set output image format and resolution
> - Supports queueing buffer from app and dequeuing ISP filled buffer to App
> - It is verified on qv4l2, cheese and qcam
> - It is verified together with following patches
> platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/20250514215623.522746-1-pratap.nirujogi@amd.com/)
> pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
> drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)
> drm/amd/amdgpu: Declare isp firmware binary file (https://gitlab.freedesktop.org/agd5f/linux/-/commit/35345917bc9f7c86152b270d9d93c220230b667f)
>
> AMD ISP4 Key features:
> - Processes bayer raw data from the connected sensor and output them to different YUV formats
> - Downscale input image to different output image resolution
> - Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc.
>
> ----------
>
> Changes v6 -> v7:
>
> - Added missed blank line after some if statements.
> - Changed the pm_runtime_disable() order in isp4_capture_remove() to align with initialization.
> - Reset buf_sent_cnt and start_stream_cmd_sent on stream stop.
> - Removed duplicate buf_sent_cnt and start_stream_cmd_sent reset in isp4sd_pwron_and_init().
> - Combined isp4sd_reset_stream_info() and isp4sd_reset_camera_info() into isp4sd_uninit_stream() to eliminate redundant stream info reset.
> - Removed always-false status check in isp4sd_uninit_stream().
> - Minor style improvements.
> [snip]
Hi Bin,
v7 looks great to me! :)
For the whole series,
Reviewed-by: Sultan Alsawaf <sultan@kerneltoast.com>
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 0/7] Add AMD ISP4 driver
2025-12-17 7:36 ` [PATCH v7 0/7] Add AMD ISP4 driver Sultan Alsawaf
@ 2025-12-17 7:47 ` Du, Bin
0 siblings, 0 replies; 46+ messages in thread
From: Du, Bin @ 2025-12-17 7:47 UTC (permalink / raw)
To: Sultan Alsawaf
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao
Thank you, Sultan, for your confirmation, contributions, and ongoing
support！
On 12/17/2025 3:36 PM, Sultan Alsawaf wrote:
> On Tue, Dec 16, 2025 at 05:13:19PM +0800, Bin Du wrote:
>> Hello,
>>
>> AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC (Ryzen AI Max 300 Series)
>> (https://ubuntu.com/certified/202411-36043)
>> This patch series introduces the initial driver support for the AMD ISP4.
>>
>> Patch summary:
>> - Powers up/off and initializes ISP HW
>> - Configures and kicks off ISP FW
>> - Interacts with APP using standard V4l2 interface by video node
>> - Controls ISP HW and interacts with ISP FW to do image processing
>> - Supports enum/set output image format and resolution
>> - Supports queueing buffer from app and dequeuing ISP filled buffer to App
>> - It is verified on qv4l2, cheese and qcam
>> - It is verified together with following patches
>> platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/20250514215623.522746-1-pratap.nirujogi@amd.com/)
>> pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
>> drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)
>> drm/amd/amdgpu: Declare isp firmware binary file (https://gitlab.freedesktop.org/agd5f/linux/-/commit/35345917bc9f7c86152b270d9d93c220230b667f)
>>
>> AMD ISP4 Key features:
>> - Processes bayer raw data from the connected sensor and output them to different YUV formats
>> - Downscale input image to different output image resolution
>> - Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc.
>>
>> ----------
>>
>> Changes v6 -> v7:
>>
>> - Added missed blank line after some if statements.
>> - Changed the pm_runtime_disable() order in isp4_capture_remove() to align with initialization.
>> - Reset buf_sent_cnt and start_stream_cmd_sent on stream stop.
>> - Removed duplicate buf_sent_cnt and start_stream_cmd_sent reset in isp4sd_pwron_and_init().
>> - Combined isp4sd_reset_stream_info() and isp4sd_reset_camera_info() into isp4sd_uninit_stream() to eliminate redundant stream info reset.
>> - Removed always-false status check in isp4sd_uninit_stream().
>> - Minor style improvements.
>> [snip]
>
> Hi Bin,
>
> v7 looks great to me! :)
>
> For the whole series,
>
> Reviewed-by: Sultan Alsawaf <sultan@kerneltoast.com>
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH v7 0/7] Add AMD ISP4 driver
2025-12-16 9:13 [PATCH v7 0/7] Add AMD ISP4 driver Bin Du
` (7 preceding siblings ...)
2025-12-17 7:36 ` [PATCH v7 0/7] Add AMD ISP4 driver Sultan Alsawaf
@ 2025-12-17 10:29 ` Du, Bin
2025-12-17 11:04 ` Sakari Ailus
2025-12-31 9:03 ` Kate Hsuan
9 siblings, 1 reply; 46+ messages in thread
From: Du, Bin @ 2025-12-17 10:29 UTC (permalink / raw)
To: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
sultan
Cc: pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao
Hi Laurent and Sakari,
Apologies for the gentle reminder, especially during the holiday season.
If you have a moment, could you please take a look at version 7? It’s
been significantly optimized and improved, and I’d appreciate any feedback.
On 12/16/2025 5:13 PM, Bin Du wrote:
> Hello,
>
> AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC (Ryzen AI Max 300 Series)
> (https://ubuntu.com/certified/202411-36043)
> This patch series introduces the initial driver support for the AMD ISP4.
>
[snip]
> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c
> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h
>
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 0/7] Add AMD ISP4 driver
2025-12-17 10:29 ` Du, Bin
@ 2025-12-17 11:04 ` Sakari Ailus
0 siblings, 0 replies; 46+ messages in thread
From: Sakari Ailus @ 2025-12-17 11:04 UTC (permalink / raw)
To: Du, Bin
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
prabhakar.mahadev-lad.rj, linux-media, linux-kernel, sultan,
pratap.nirujogi, benjamin.chan, king.li, gjorgji.rosikopulos,
Phil.Jawich, Dominic.Antony, mario.limonciello, richard.gong,
anson.tsao
Hi Bin,
On Wed, Dec 17, 2025 at 06:29:18PM +0800, Du, Bin wrote:
> Hi Laurent and Sakari,
>
> Apologies for the gentle reminder, especially during the holiday season. If
> you have a moment, could you please take a look at version 7? It’s been
> significantly optimized and improved, and I’d appreciate any feedback.
Thanks for the ping. I'll try to take a look this Friday.
--
Regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH v7 0/7] Add AMD ISP4 driver
2025-12-16 9:13 [PATCH v7 0/7] Add AMD ISP4 driver Bin Du
` (8 preceding siblings ...)
2025-12-17 10:29 ` Du, Bin
@ 2025-12-31 9:03 ` Kate Hsuan
2026-01-06 5:49 ` Kate Hsuan
9 siblings, 1 reply; 46+ messages in thread
From: Kate Hsuan @ 2025-12-31 9:03 UTC (permalink / raw)
To: Bin Du
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
sultan, pratap.nirujogi, benjamin.chan, king.li,
gjorgji.rosikopulos, Phil.Jawich, Dominic.Antony,
mario.limonciello, richard.gong, anson.tsao
On Tue, Dec 16, 2025 at 5:14 PM Bin Du <Bin.Du@amd.com> wrote:
>
> Hello,
>
> AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC (Ryzen AI Max 300 Series)
> (https://ubuntu.com/certified/202411-36043)
> This patch series introduces the initial driver support for the AMD ISP4.
>
> Patch summary:
> - Powers up/off and initializes ISP HW
> - Configures and kicks off ISP FW
> - Interacts with APP using standard V4l2 interface by video node
> - Controls ISP HW and interacts with ISP FW to do image processing
> - Supports enum/set output image format and resolution
> - Supports queueing buffer from app and dequeuing ISP filled buffer to App
> - It is verified on qv4l2, cheese and qcam
> - It is verified together with following patches
> platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/20250514215623.522746-1-pratap.nirujogi@amd.com/)
> pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
> drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)
> drm/amd/amdgpu: Declare isp firmware binary file (https://gitlab.freedesktop.org/agd5f/linux/-/commit/35345917bc9f7c86152b270d9d93c220230b667f)
>
> AMD ISP4 Key features:
> - Processes bayer raw data from the connected sensor and output them to different YUV formats
> - Downscale input image to different output image resolution
> - Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc.
>
> ----------
>
> Changes v6 -> v7:
>
> - Added missed blank line after some if statements.
> - Changed the pm_runtime_disable() order in isp4_capture_remove() to align with initialization.
> - Reset buf_sent_cnt and start_stream_cmd_sent on stream stop.
> - Removed duplicate buf_sent_cnt and start_stream_cmd_sent reset in isp4sd_pwron_and_init().
> - Combined isp4sd_reset_stream_info() and isp4sd_reset_camera_info() into isp4sd_uninit_stream() to eliminate redundant stream info reset.
> - Removed always-false status check in isp4sd_uninit_stream().
> - Minor style improvements.
>
>
> Changes v5 -> v6:
>
> - Lowered FW mempool buffer size from 200M to 100M (actual usage).
> - Added an irq_enabled member to the ISP subdev for proper IRQ disable handling in both normal and error cases.
> - Removed unnecessary .owner assignment from isp4_capture_drv definition
> - Updated IRQ handling to enable and disable interrupts via the ISP register for improved performance.
> - Revised ring buffer management in isp4if_f2h_resp(), addressing safety checks to ensure the read pointer is validated before memcpy operations, reducing the risk of out-of-bounds access. The ring buffer logic was also streamlined significantly.
> - Modified ring buffer handling in isp4if_is_cmdq_rb_full(), correcting an off-by-one error in safety checks that previously allowed rd_ptr to equal wr_ptr when the buffer was full.
> - Refactored ring buffer handling in isp4if_insert_isp_fw_cmd(), simplifying overall logic.
> - Resolved a regression from v4 to v5 where isp4if_dequeue_buffer() did not protect list_del with bufq_lock.
> - Addressed a subtle use-after-free issue that could occur if a timeout on a synchronous command coincided with completion.
> - Added missing pm_runtime_disable() calls to isp4_capture_remove() and to the error path in isp4_capture_probe().
> - Removed stray semicolons following closing curly braces.
> - Improved and clarified macro definitions in isp4_interface.h.
> - Eliminated unnecessary (u8 *) casts.
> - Added missing memset for firmware command structures in isp4sd_stop_stream().
> - Excluded streams 2 and 3 from ISP4IF_FW_RESP_RB_IRQ_EN_MASK, preventing their activation in the interrupt enable register.
> - Enhanced error handling to clean up kthreads in the event of startup failure.
> - Corrected a race condition during kthread creation where waitqueue head initialization could be delayed, as it was performed by the kthread itself.
> - Removed status checks in isp4sd_pwroff_and_deinit() that were always false.
> - Ensured isp4sd_init_stream() is only invoked once per stream start and reordered corresponding status checks in isp4sd_start_stream().
> - Improved error handling in isp4sd_start_stream() to propagate errors from failed functions.
> - Relocated debugging messages in isp4sd_stop_stream() to execute under lock protection due to access to output_info->start_status.
> - Eliminated redundant GET_REG_BASE() macros.
> - Removed isp4sd_is_stream_running() function.
> - Corrected error message in isp4sd_init_stream() caused by copy/paste.
> - Refined struct isp4_interface to remove firmware ring buffer configurations.
> - Removed obsolete isp4sd_is_stream_running function.
> - Removed pdev member from struct isp4_device, as it is unnecessary.
> - Fixed typo in 'isp_mmip' parameter name within isp4if_init().
> - Removed gap in struct isp4_subdev definition.
> - Performed extensive dead code removal and minor style improvements throughout the codebase.
>
>
> Changes v4 -> v5:
>
> - Transitioned VIDEOBUF2_V4L2 from 'depends' to 'select' within Kconfig.
> - Standardized object file naming conventions in the Makefile and sorted entries alphabetically.
> - Removed the unused macro definition to_isp4_device.
> - Eliminated unused members mem_domain and mem_align from struct isp4if_gpu_mem_info.
> - Deleted unused fields mc_addr and gpu_pkg from struct isp4if_cmd_element.
> - Removed obsolete pltf_data, i2c_nb, and notifier elements from struct isp4_device.
> - Updated platform_get_irq failure handling to return its actual result rather than -ENODEV.
> - Refined inclusion of header files for clarity and efficiency.
> - Appended comments following #endif statements in header files.
> - Improved implementation of isp4if_gpu_mem_free and isp4if_dealloc_fw_gpumem.
> - Removed isp4if_append_cmd_2_cmdq and revised isp4if_send_fw_cmd accordingly.
> - Enhanced isp4if_clear_cmdq and isp4if_clear_bufq by eliminating unnecessary list_del operations.
> - Adopted completion mechanism instead of wait queue and condition for command completion notifications.
> - Employed memset to ensure proper zeroing of padding bits in structures shared between ISP driver and firmware.
> - Streamlined IRQs, reducing total from four to two, retaining only essential ones.
> - Optimized IRQ handler logic using a while loop for greater efficiency.
> - Introduced dynamic IRQ enable/disable functionality based on camera status (open/close).
> - Applied distinct identifiers to differentiate multiple threads and IRQs.
> - Removed unnecessary initialization of local variables.
> - Refined camera start/stop workflow to mitigate potential synchronization concerns.
> - Replaced all remaining mutex with guard mutex.
> - Enhanced command and buffer queue performance by substituting mutexes with spinlocks.
> - Removed redundant isp4sd_init_meta_buf function and its references.
> - Limited firmware logging activities to the stream1 thread.
> - Relocated v4l2_device_unregister_subdev() and media_entity_cleanup() calls from isp4_capture_remove to isp4sd_deinit.
> - Resolved media device registration sequence issues.
> - Modified stream processing thread behavior to await IRQ without a timeout.
> - Addressed cleanup procedures in video device initialization and deinitialization routines.
> - Corrected typos and made other cosmetic improvements.
>
>
> Changes v3 -> v4:
>
> - Replace one mutex with guard mutex.
> - Remove unnecessary bus_info initialization of v4l2_capability.
> - Drop V4L2_CAP_IO_MC from capabilities of v4l2_capability.
> - Modify document with better SOC description.
> - Fix Test x86 failure in Media CI test https://linux-media.pages.freedesktop.org/-/users/patchwork/-/jobs/83470456/artifacts/report.htm
> - Modify some commit messages by describing changes in imperative mood.
> - Add media-ctl output in cover letter.
> - Create separated dedicated amdgpu patch to add declaration MODULE_FIRMWARE("amdgpu/isp_4_1_1.bin");
> - Fix typo errors and other cosmetic issues.
> - Add DRM_AMD_ISP dependency in Kconfig.
>
>
> Changes v2 -> v3:
>
> - All the dependent patches in other modules (drm/amd/amdgpu, platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
> - Removed usage of amdgpu structs in ISP driver. Added helper functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
> - Moved sensor and MIPI phy control entirely into ISP FW instead of the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
> - Removed phy configuration and sensor binding as x86 (sensor driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
> - New FW with built-in sensor driver is submitted on upstream linux-firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
> - Please note the new FW submitted is not directly compatible with OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
> - If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
> - Included critical fixes from Sultan Alsawaf branch (https://github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
> media: amd: isp4: Add missing refcount tracking to mmap memop
> media: amd: isp4: Don't put or unmap the dmabuf when detaching
> media: amd: isp4: Don't increment refcount when dmabuf export fails
> media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
> media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
> media: amd: isp4: Fix implicit dmabuf lifetime tracking
> media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
> media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
> media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
> media: amd: isp4: Remove unused userptr memops
> media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
> media: amd: isp4: Release queued buffers on error in start_streaming
> - Addressed all code related upstream comments
> - Fix typo errors and other cosmetic issues.
>
>
> Changes v1 -> v2:
>
> - Fix media CI test errors and valid warnings
> - Reduce patch number in the series from 9 to 8 by merging MAINTAINERS adding patch to the first patch
> - In patch 5
> - do modification to use remote endpoint instead of local endpoint
> - use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint
>
> ----------
>
> It passes v4l2 compliance test, the test reports for:
>
> (a) amd_isp_capture device /dev/video0
>
> Compliance test for amd_isp_capture device /dev/video0:
> -------------------------------------------------------
>
> atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
> v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
> v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17
>
> Compliance test for amd_isp_capture device /dev/video0:
>
> Driver Info:
> Driver name : amd_isp_capture
> Card type : amd_isp_capture
> Bus info : platform:amd_isp_capture
> Driver version : 6.14.0
> Capabilities : 0xa4200001
> Video Capture
> I/O MC
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x24200001
> Video Capture
> I/O MC
> Streaming
> Extended Pix Format
> Media Driver Info:
> Driver name : amd_isp_capture
> Model : amd_isp41_mdev
> Serial :
> Bus info : platform:amd_isp_capture
> Media version : 6.14.0
> Hardware revision: 0x00000000 (0)
> Driver version : 6.14.0
> Interface Info:
> ID : 0x03000005
> Type : V4L Video
> Entity Info:
> ID : 0x00000003 (3)
> Name : Preview
> Function : V4L2 I/O
> Pad 0x01000004 : 0: Sink
> Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable
>
> Required ioctls:
> test MC information (see 'Media Driver Info' above): OK
> test VIDIOC_QUERYCAP: OK
> test invalid ioctls: OK
>
> Allow for multiple opens:
> test second /dev/video0 open: OK
> test VIDIOC_QUERYCAP: OK
> test VIDIOC_G/S_PRIORITY: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 1 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 0 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Control ioctls (Input 0):
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> test VIDIOC_QUERYCTRL: OK (Not Supported)
> test VIDIOC_G/S_CTRL: OK (Not Supported)
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 0 Private Controls: 0
>
> Format ioctls (Input 0):
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> test VIDIOC_G/S_PARM: OK
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK
> test VIDIOC_TRY_FMT: OK
> test VIDIOC_S_FMT: OK
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK (Not Supported)
>
> Codec ioctls (Input 0):
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls (Input 0):
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> test CREATE_BUFS maximum buffers: OK
> test VIDIOC_REMOVE_BUFS: OK
> test VIDIOC_EXPBUF: OK
> test Requests: OK (Not Supported)
> test blocking wait: OK
>
> Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0
>
> The media-ctl output of media device /dev/media0:
> -------------------------------------------------------
>
> atg@atg-HP-PV:~$ media-ctl -p -d /dev/media0
> Media controller API version 6.17.0
>
> Media device information
> ------------------------
> driver amd_isp_capture
> model amd_isp41_mdev
> serial
> bus info platform:amd_isp_capture
> hw revision 0x0
> driver version 6.17.0
>
> Device topology
> - entity 1: amd isp4 (1 pad, 1 link, 0 routes)
> type V4L2 subdev subtype Unknown flags 0
> pad0: Source
> -> "Preview":0 [ENABLED,IMMUTABLE]
>
> - entity 3: Preview (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video0
> pad0: Sink
> <- "amd isp4":0 [ENABLED,IMMUTABLE]
>
> Please review and provide feedback.
>
> Many thanks,
>
> Bin Du (7):
> media: platform: amd: Introduce amd isp4 capture driver
> media: platform: amd: low level support for isp4 firmware
> media: platform: amd: Add isp4 fw and hw interface
> media: platform: amd: isp4 subdev and firmware loading handling added
> media: platform: amd: isp4 video node and buffers handling added
> media: platform: amd: isp4 debug fs logging and more descriptive
> errors
> Documentation: add documentation of AMD isp 4 driver
>
> Documentation/admin-guide/media/amdisp4-1.rst | 63 +
> Documentation/admin-guide/media/amdisp4.dot | 6 +
> .../admin-guide/media/v4l-drivers.rst | 1 +
> MAINTAINERS | 25 +
> drivers/media/platform/Kconfig | 1 +
> drivers/media/platform/Makefile | 1 +
> drivers/media/platform/amd/Kconfig | 3 +
> drivers/media/platform/amd/Makefile | 3 +
> drivers/media/platform/amd/isp4/Kconfig | 14 +
> drivers/media/platform/amd/isp4/Makefile | 10 +
> drivers/media/platform/amd/isp4/isp4.c | 235 ++++
> drivers/media/platform/amd/isp4/isp4.h | 20 +
> drivers/media/platform/amd/isp4/isp4_debug.c | 271 ++++
> drivers/media/platform/amd/isp4/isp4_debug.h | 41 +
> .../platform/amd/isp4/isp4_fw_cmd_resp.h | 314 +++++
> drivers/media/platform/amd/isp4/isp4_hw_reg.h | 124 ++
> .../media/platform/amd/isp4/isp4_interface.c | 789 +++++++++++
> .../media/platform/amd/isp4/isp4_interface.h | 141 ++
> drivers/media/platform/amd/isp4/isp4_subdev.c | 1057 +++++++++++++++
> drivers/media/platform/amd/isp4/isp4_subdev.h | 131 ++
> drivers/media/platform/amd/isp4/isp4_video.c | 1165 +++++++++++++++++
> drivers/media/platform/amd/isp4/isp4_video.h | 65 +
> 22 files changed, 4480 insertions(+)
> create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst
> create mode 100644 Documentation/admin-guide/media/amdisp4.dot
> create mode 100644 drivers/media/platform/amd/Kconfig
> create mode 100644 drivers/media/platform/amd/Makefile
> create mode 100644 drivers/media/platform/amd/isp4/Kconfig
> create mode 100644 drivers/media/platform/amd/isp4/Makefile
> create mode 100644 drivers/media/platform/amd/isp4/isp4.c
> create mode 100644 drivers/media/platform/amd/isp4/isp4.h
> create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c
> create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h
> create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
> create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h
> create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c
> create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h
> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c
> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h
>
> --
> 2.34.1
>
>
v7 worked on my HP ZBook Ultra G1a 14 inch Mobile Workstation PC and
the camera worked fine.
It was tested with the latest upstream firmware [1] and 6.19-rc3 kernel.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/amdgpu/isp_4_1_1.bin
Thank you for your work :)
--
BR,
Kate
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 0/7] Add AMD ISP4 driver
2025-12-31 9:03 ` Kate Hsuan
@ 2026-01-06 5:49 ` Kate Hsuan
2026-01-06 8:35 ` Du, Bin
0 siblings, 1 reply; 46+ messages in thread
From: Kate Hsuan @ 2026-01-06 5:49 UTC (permalink / raw)
To: Bin Du
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
sultan, pratap.nirujogi, benjamin.chan, king.li,
gjorgji.rosikopulos, Phil.Jawich, Dominic.Antony,
mario.limonciello, richard.gong, anson.tsao
Hi
On Wed, Dec 31, 2025 at 5:03 PM Kate Hsuan <hpa@redhat.com> wrote:
>
> On Tue, Dec 16, 2025 at 5:14 PM Bin Du <Bin.Du@amd.com> wrote:
> >
> > Hello,
> >
> > AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC (Ryzen AI Max 300 Series)
> > (https://ubuntu.com/certified/202411-36043)
> > This patch series introduces the initial driver support for the AMD ISP4.
> >
> > Patch summary:
> > - Powers up/off and initializes ISP HW
> > - Configures and kicks off ISP FW
> > - Interacts with APP using standard V4l2 interface by video node
> > - Controls ISP HW and interacts with ISP FW to do image processing
> > - Supports enum/set output image format and resolution
> > - Supports queueing buffer from app and dequeuing ISP filled buffer to App
> > - It is verified on qv4l2, cheese and qcam
> > - It is verified together with following patches
> > platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/20250514215623.522746-1-pratap.nirujogi@amd.com/)
> > pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
> > drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)
> > drm/amd/amdgpu: Declare isp firmware binary file (https://gitlab.freedesktop.org/agd5f/linux/-/commit/35345917bc9f7c86152b270d9d93c220230b667f)
> >
> > AMD ISP4 Key features:
> > - Processes bayer raw data from the connected sensor and output them to different YUV formats
> > - Downscale input image to different output image resolution
> > - Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc.
> >
> > ----------
> >
> > Changes v6 -> v7:
> >
> > - Added missed blank line after some if statements.
> > - Changed the pm_runtime_disable() order in isp4_capture_remove() to align with initialization.
> > - Reset buf_sent_cnt and start_stream_cmd_sent on stream stop.
> > - Removed duplicate buf_sent_cnt and start_stream_cmd_sent reset in isp4sd_pwron_and_init().
> > - Combined isp4sd_reset_stream_info() and isp4sd_reset_camera_info() into isp4sd_uninit_stream() to eliminate redundant stream info reset.
> > - Removed always-false status check in isp4sd_uninit_stream().
> > - Minor style improvements.
> >
> >
> > Changes v5 -> v6:
> >
> > - Lowered FW mempool buffer size from 200M to 100M (actual usage).
> > - Added an irq_enabled member to the ISP subdev for proper IRQ disable handling in both normal and error cases.
> > - Removed unnecessary .owner assignment from isp4_capture_drv definition
> > - Updated IRQ handling to enable and disable interrupts via the ISP register for improved performance.
> > - Revised ring buffer management in isp4if_f2h_resp(), addressing safety checks to ensure the read pointer is validated before memcpy operations, reducing the risk of out-of-bounds access. The ring buffer logic was also streamlined significantly.
> > - Modified ring buffer handling in isp4if_is_cmdq_rb_full(), correcting an off-by-one error in safety checks that previously allowed rd_ptr to equal wr_ptr when the buffer was full.
> > - Refactored ring buffer handling in isp4if_insert_isp_fw_cmd(), simplifying overall logic.
> > - Resolved a regression from v4 to v5 where isp4if_dequeue_buffer() did not protect list_del with bufq_lock.
> > - Addressed a subtle use-after-free issue that could occur if a timeout on a synchronous command coincided with completion.
> > - Added missing pm_runtime_disable() calls to isp4_capture_remove() and to the error path in isp4_capture_probe().
> > - Removed stray semicolons following closing curly braces.
> > - Improved and clarified macro definitions in isp4_interface.h.
> > - Eliminated unnecessary (u8 *) casts.
> > - Added missing memset for firmware command structures in isp4sd_stop_stream().
> > - Excluded streams 2 and 3 from ISP4IF_FW_RESP_RB_IRQ_EN_MASK, preventing their activation in the interrupt enable register.
> > - Enhanced error handling to clean up kthreads in the event of startup failure.
> > - Corrected a race condition during kthread creation where waitqueue head initialization could be delayed, as it was performed by the kthread itself.
> > - Removed status checks in isp4sd_pwroff_and_deinit() that were always false.
> > - Ensured isp4sd_init_stream() is only invoked once per stream start and reordered corresponding status checks in isp4sd_start_stream().
> > - Improved error handling in isp4sd_start_stream() to propagate errors from failed functions.
> > - Relocated debugging messages in isp4sd_stop_stream() to execute under lock protection due to access to output_info->start_status.
> > - Eliminated redundant GET_REG_BASE() macros.
> > - Removed isp4sd_is_stream_running() function.
> > - Corrected error message in isp4sd_init_stream() caused by copy/paste.
> > - Refined struct isp4_interface to remove firmware ring buffer configurations.
> > - Removed obsolete isp4sd_is_stream_running function.
> > - Removed pdev member from struct isp4_device, as it is unnecessary.
> > - Fixed typo in 'isp_mmip' parameter name within isp4if_init().
> > - Removed gap in struct isp4_subdev definition.
> > - Performed extensive dead code removal and minor style improvements throughout the codebase.
> >
> >
> > Changes v4 -> v5:
> >
> > - Transitioned VIDEOBUF2_V4L2 from 'depends' to 'select' within Kconfig.
> > - Standardized object file naming conventions in the Makefile and sorted entries alphabetically.
> > - Removed the unused macro definition to_isp4_device.
> > - Eliminated unused members mem_domain and mem_align from struct isp4if_gpu_mem_info.
> > - Deleted unused fields mc_addr and gpu_pkg from struct isp4if_cmd_element.
> > - Removed obsolete pltf_data, i2c_nb, and notifier elements from struct isp4_device.
> > - Updated platform_get_irq failure handling to return its actual result rather than -ENODEV.
> > - Refined inclusion of header files for clarity and efficiency.
> > - Appended comments following #endif statements in header files.
> > - Improved implementation of isp4if_gpu_mem_free and isp4if_dealloc_fw_gpumem.
> > - Removed isp4if_append_cmd_2_cmdq and revised isp4if_send_fw_cmd accordingly.
> > - Enhanced isp4if_clear_cmdq and isp4if_clear_bufq by eliminating unnecessary list_del operations.
> > - Adopted completion mechanism instead of wait queue and condition for command completion notifications.
> > - Employed memset to ensure proper zeroing of padding bits in structures shared between ISP driver and firmware.
> > - Streamlined IRQs, reducing total from four to two, retaining only essential ones.
> > - Optimized IRQ handler logic using a while loop for greater efficiency.
> > - Introduced dynamic IRQ enable/disable functionality based on camera status (open/close).
> > - Applied distinct identifiers to differentiate multiple threads and IRQs.
> > - Removed unnecessary initialization of local variables.
> > - Refined camera start/stop workflow to mitigate potential synchronization concerns.
> > - Replaced all remaining mutex with guard mutex.
> > - Enhanced command and buffer queue performance by substituting mutexes with spinlocks.
> > - Removed redundant isp4sd_init_meta_buf function and its references.
> > - Limited firmware logging activities to the stream1 thread.
> > - Relocated v4l2_device_unregister_subdev() and media_entity_cleanup() calls from isp4_capture_remove to isp4sd_deinit.
> > - Resolved media device registration sequence issues.
> > - Modified stream processing thread behavior to await IRQ without a timeout.
> > - Addressed cleanup procedures in video device initialization and deinitialization routines.
> > - Corrected typos and made other cosmetic improvements.
> >
> >
> > Changes v3 -> v4:
> >
> > - Replace one mutex with guard mutex.
> > - Remove unnecessary bus_info initialization of v4l2_capability.
> > - Drop V4L2_CAP_IO_MC from capabilities of v4l2_capability.
> > - Modify document with better SOC description.
> > - Fix Test x86 failure in Media CI test https://linux-media.pages.freedesktop.org/-/users/patchwork/-/jobs/83470456/artifacts/report.htm
> > - Modify some commit messages by describing changes in imperative mood.
> > - Add media-ctl output in cover letter.
> > - Create separated dedicated amdgpu patch to add declaration MODULE_FIRMWARE("amdgpu/isp_4_1_1.bin");
> > - Fix typo errors and other cosmetic issues.
> > - Add DRM_AMD_ISP dependency in Kconfig.
> >
> >
> > Changes v2 -> v3:
> >
> > - All the dependent patches in other modules (drm/amd/amdgpu, platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
> > - Removed usage of amdgpu structs in ISP driver. Added helper functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
> > - Moved sensor and MIPI phy control entirely into ISP FW instead of the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
> > - Removed phy configuration and sensor binding as x86 (sensor driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
> > - New FW with built-in sensor driver is submitted on upstream linux-firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
> > - Please note the new FW submitted is not directly compatible with OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
> > - If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
> > - Included critical fixes from Sultan Alsawaf branch (https://github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
> > media: amd: isp4: Add missing refcount tracking to mmap memop
> > media: amd: isp4: Don't put or unmap the dmabuf when detaching
> > media: amd: isp4: Don't increment refcount when dmabuf export fails
> > media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
> > media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
> > media: amd: isp4: Fix implicit dmabuf lifetime tracking
> > media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
> > media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
> > media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
> > media: amd: isp4: Remove unused userptr memops
> > media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
> > media: amd: isp4: Release queued buffers on error in start_streaming
> > - Addressed all code related upstream comments
> > - Fix typo errors and other cosmetic issues.
> >
> >
> > Changes v1 -> v2:
> >
> > - Fix media CI test errors and valid warnings
> > - Reduce patch number in the series from 9 to 8 by merging MAINTAINERS adding patch to the first patch
> > - In patch 5
> > - do modification to use remote endpoint instead of local endpoint
> > - use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint
> >
> > ----------
> >
> > It passes v4l2 compliance test, the test reports for:
> >
> > (a) amd_isp_capture device /dev/video0
> >
> > Compliance test for amd_isp_capture device /dev/video0:
> > -------------------------------------------------------
> >
> > atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
> > v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
> > v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17
> >
> > Compliance test for amd_isp_capture device /dev/video0:
> >
> > Driver Info:
> > Driver name : amd_isp_capture
> > Card type : amd_isp_capture
> > Bus info : platform:amd_isp_capture
> > Driver version : 6.14.0
> > Capabilities : 0xa4200001
> > Video Capture
> > I/O MC
> > Streaming
> > Extended Pix Format
> > Device Capabilities
> > Device Caps : 0x24200001
> > Video Capture
> > I/O MC
> > Streaming
> > Extended Pix Format
> > Media Driver Info:
> > Driver name : amd_isp_capture
> > Model : amd_isp41_mdev
> > Serial :
> > Bus info : platform:amd_isp_capture
> > Media version : 6.14.0
> > Hardware revision: 0x00000000 (0)
> > Driver version : 6.14.0
> > Interface Info:
> > ID : 0x03000005
> > Type : V4L Video
> > Entity Info:
> > ID : 0x00000003 (3)
> > Name : Preview
> > Function : V4L2 I/O
> > Pad 0x01000004 : 0: Sink
> > Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable
> >
> > Required ioctls:
> > test MC information (see 'Media Driver Info' above): OK
> > test VIDIOC_QUERYCAP: OK
> > test invalid ioctls: OK
> >
> > Allow for multiple opens:
> > test second /dev/video0 open: OK
> > test VIDIOC_QUERYCAP: OK
> > test VIDIOC_G/S_PRIORITY: OK
> > test for unlimited opens: OK
> >
> > Debug ioctls:
> > test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> > test VIDIOC_LOG_STATUS: OK (Not Supported)
> >
> > Input ioctls:
> > test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> > test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> > test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> > test VIDIOC_ENUMAUDIO: OK (Not Supported)
> > test VIDIOC_G/S/ENUMINPUT: OK
> > test VIDIOC_G/S_AUDIO: OK (Not Supported)
> > Inputs: 1 Audio Inputs: 0 Tuners: 0
> >
> > Output ioctls:
> > test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> > test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> > test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> > test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> > test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> > Outputs: 0 Audio Outputs: 0 Modulators: 0
> >
> > Input/Output configuration ioctls:
> > test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> > test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> > test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> > test VIDIOC_G/S_EDID: OK (Not Supported)
> >
> > Control ioctls (Input 0):
> > test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> > test VIDIOC_QUERYCTRL: OK (Not Supported)
> > test VIDIOC_G/S_CTRL: OK (Not Supported)
> > test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> > test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> > test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> > Standard Controls: 0 Private Controls: 0
> >
> > Format ioctls (Input 0):
> > test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> > test VIDIOC_G/S_PARM: OK
> > test VIDIOC_G_FBUF: OK (Not Supported)
> > test VIDIOC_G_FMT: OK
> > test VIDIOC_TRY_FMT: OK
> > test VIDIOC_S_FMT: OK
> > test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> > test Cropping: OK (Not Supported)
> > test Composing: OK (Not Supported)
> > test Scaling: OK (Not Supported)
> >
> > Codec ioctls (Input 0):
> > test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> > test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> > test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
> >
> > Buffer ioctls (Input 0):
> > test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> > test CREATE_BUFS maximum buffers: OK
> > test VIDIOC_REMOVE_BUFS: OK
> > test VIDIOC_EXPBUF: OK
> > test Requests: OK (Not Supported)
> > test blocking wait: OK
> >
> > Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0
> >
> > The media-ctl output of media device /dev/media0:
> > -------------------------------------------------------
> >
> > atg@atg-HP-PV:~$ media-ctl -p -d /dev/media0
> > Media controller API version 6.17.0
> >
> > Media device information
> > ------------------------
> > driver amd_isp_capture
> > model amd_isp41_mdev
> > serial
> > bus info platform:amd_isp_capture
> > hw revision 0x0
> > driver version 6.17.0
> >
> > Device topology
> > - entity 1: amd isp4 (1 pad, 1 link, 0 routes)
> > type V4L2 subdev subtype Unknown flags 0
> > pad0: Source
> > -> "Preview":0 [ENABLED,IMMUTABLE]
> >
> > - entity 3: Preview (1 pad, 1 link)
> > type Node subtype V4L flags 0
> > device node name /dev/video0
> > pad0: Sink
> > <- "amd isp4":0 [ENABLED,IMMUTABLE]
> >
> > Please review and provide feedback.
> >
> > Many thanks,
> >
> > Bin Du (7):
> > media: platform: amd: Introduce amd isp4 capture driver
> > media: platform: amd: low level support for isp4 firmware
> > media: platform: amd: Add isp4 fw and hw interface
> > media: platform: amd: isp4 subdev and firmware loading handling added
> > media: platform: amd: isp4 video node and buffers handling added
> > media: platform: amd: isp4 debug fs logging and more descriptive
> > errors
> > Documentation: add documentation of AMD isp 4 driver
> >
> > Documentation/admin-guide/media/amdisp4-1.rst | 63 +
> > Documentation/admin-guide/media/amdisp4.dot | 6 +
> > .../admin-guide/media/v4l-drivers.rst | 1 +
> > MAINTAINERS | 25 +
> > drivers/media/platform/Kconfig | 1 +
> > drivers/media/platform/Makefile | 1 +
> > drivers/media/platform/amd/Kconfig | 3 +
> > drivers/media/platform/amd/Makefile | 3 +
> > drivers/media/platform/amd/isp4/Kconfig | 14 +
> > drivers/media/platform/amd/isp4/Makefile | 10 +
> > drivers/media/platform/amd/isp4/isp4.c | 235 ++++
> > drivers/media/platform/amd/isp4/isp4.h | 20 +
> > drivers/media/platform/amd/isp4/isp4_debug.c | 271 ++++
> > drivers/media/platform/amd/isp4/isp4_debug.h | 41 +
> > .../platform/amd/isp4/isp4_fw_cmd_resp.h | 314 +++++
> > drivers/media/platform/amd/isp4/isp4_hw_reg.h | 124 ++
> > .../media/platform/amd/isp4/isp4_interface.c | 789 +++++++++++
> > .../media/platform/amd/isp4/isp4_interface.h | 141 ++
> > drivers/media/platform/amd/isp4/isp4_subdev.c | 1057 +++++++++++++++
> > drivers/media/platform/amd/isp4/isp4_subdev.h | 131 ++
> > drivers/media/platform/amd/isp4/isp4_video.c | 1165 +++++++++++++++++
> > drivers/media/platform/amd/isp4/isp4_video.h | 65 +
> > 22 files changed, 4480 insertions(+)
> > create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst
> > create mode 100644 Documentation/admin-guide/media/amdisp4.dot
> > create mode 100644 drivers/media/platform/amd/Kconfig
> > create mode 100644 drivers/media/platform/amd/Makefile
> > create mode 100644 drivers/media/platform/amd/isp4/Kconfig
> > create mode 100644 drivers/media/platform/amd/isp4/Makefile
> > create mode 100644 drivers/media/platform/amd/isp4/isp4.c
> > create mode 100644 drivers/media/platform/amd/isp4/isp4.h
> > create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c
> > create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h
> > create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
> > create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h
> > create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c
> > create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h
> > create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
> > create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
> > create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c
> > create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h
> >
> > --
> > 2.34.1
> >
> >
>
> v7 worked on my HP ZBook Ultra G1a 14 inch Mobile Workstation PC and
> the camera worked fine.
> It was tested with the latest upstream firmware [1] and 6.19-rc3 kernel.
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/amdgpu/isp_4_1_1.bin
>
> Thank you for your work :)
>
> --
> BR,
> Kate
Leave a "Tested-by" tag for my previous testing assertion.
Tested-by: Kate Hsuan <hpa@redhat.com>
--
BR,
Kate
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 0/7] Add AMD ISP4 driver
2026-01-06 5:49 ` Kate Hsuan
@ 2026-01-06 8:35 ` Du, Bin
2026-01-13 14:11 ` Kate Hsuan
0 siblings, 1 reply; 46+ messages in thread
From: Du, Bin @ 2026-01-06 8:35 UTC (permalink / raw)
To: Kate Hsuan
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
sultan, pratap.nirujogi, benjamin.chan, king.li,
gjorgji.rosikopulos, Phil.Jawich, Dominic.Antony,
mario.limonciello, richard.gong, anson.tsao
Thank you, Kate, for the verification, it gives us more confidence.
On 1/6/2026 1:49 PM, Kate Hsuan wrote:
> Hi
>
> On Wed, Dec 31, 2025 at 5:03 PM Kate Hsuan <hpa@redhat.com> wrote:
>>
>> On Tue, Dec 16, 2025 at 5:14 PM Bin Du <Bin.Du@amd.com> wrote:
>>>
>>> Hello,
>>>
>>> AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC (Ryzen AI Max 300 Series)
>>> (https://ubuntu.com/certified/202411-36043)
>>> This patch series introduces the initial driver support for the AMD ISP4.
>>>
>>> Patch summary:
>>> - Powers up/off and initializes ISP HW
>>> - Configures and kicks off ISP FW
>>> - Interacts with APP using standard V4l2 interface by video node
>>> - Controls ISP HW and interacts with ISP FW to do image processing
>>> - Supports enum/set output image format and resolution
>>> - Supports queueing buffer from app and dequeuing ISP filled buffer to App
>>> - It is verified on qv4l2, cheese and qcam
>>> - It is verified together with following patches
>>> platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/20250514215623.522746-1-pratap.nirujogi@amd.com/)
>>> pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
>>> drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)
>>> drm/amd/amdgpu: Declare isp firmware binary file (https://gitlab.freedesktop.org/agd5f/linux/-/commit/35345917bc9f7c86152b270d9d93c220230b667f)
>>>
>>> AMD ISP4 Key features:
>>> - Processes bayer raw data from the connected sensor and output them to different YUV formats
>>> - Downscale input image to different output image resolution
>>> - Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc.
>>>
>>> ----------
>>>
>>> Changes v6 -> v7:
>>>
>>> - Added missed blank line after some if statements.
>>> - Changed the pm_runtime_disable() order in isp4_capture_remove() to align with initialization.
>>> - Reset buf_sent_cnt and start_stream_cmd_sent on stream stop.
>>> - Removed duplicate buf_sent_cnt and start_stream_cmd_sent reset in isp4sd_pwron_and_init().
>>> - Combined isp4sd_reset_stream_info() and isp4sd_reset_camera_info() into isp4sd_uninit_stream() to eliminate redundant stream info reset.
>>> - Removed always-false status check in isp4sd_uninit_stream().
>>> - Minor style improvements.
>>>
>>>
>>> Changes v5 -> v6:
>>>
>>> - Lowered FW mempool buffer size from 200M to 100M (actual usage).
>>> - Added an irq_enabled member to the ISP subdev for proper IRQ disable handling in both normal and error cases.
>>> - Removed unnecessary .owner assignment from isp4_capture_drv definition
>>> - Updated IRQ handling to enable and disable interrupts via the ISP register for improved performance.
>>> - Revised ring buffer management in isp4if_f2h_resp(), addressing safety checks to ensure the read pointer is validated before memcpy operations, reducing the risk of out-of-bounds access. The ring buffer logic was also streamlined significantly.
>>> - Modified ring buffer handling in isp4if_is_cmdq_rb_full(), correcting an off-by-one error in safety checks that previously allowed rd_ptr to equal wr_ptr when the buffer was full.
>>> - Refactored ring buffer handling in isp4if_insert_isp_fw_cmd(), simplifying overall logic.
>>> - Resolved a regression from v4 to v5 where isp4if_dequeue_buffer() did not protect list_del with bufq_lock.
>>> - Addressed a subtle use-after-free issue that could occur if a timeout on a synchronous command coincided with completion.
>>> - Added missing pm_runtime_disable() calls to isp4_capture_remove() and to the error path in isp4_capture_probe().
>>> - Removed stray semicolons following closing curly braces.
>>> - Improved and clarified macro definitions in isp4_interface.h.
>>> - Eliminated unnecessary (u8 *) casts.
>>> - Added missing memset for firmware command structures in isp4sd_stop_stream().
>>> - Excluded streams 2 and 3 from ISP4IF_FW_RESP_RB_IRQ_EN_MASK, preventing their activation in the interrupt enable register.
>>> - Enhanced error handling to clean up kthreads in the event of startup failure.
>>> - Corrected a race condition during kthread creation where waitqueue head initialization could be delayed, as it was performed by the kthread itself.
>>> - Removed status checks in isp4sd_pwroff_and_deinit() that were always false.
>>> - Ensured isp4sd_init_stream() is only invoked once per stream start and reordered corresponding status checks in isp4sd_start_stream().
>>> - Improved error handling in isp4sd_start_stream() to propagate errors from failed functions.
>>> - Relocated debugging messages in isp4sd_stop_stream() to execute under lock protection due to access to output_info->start_status.
>>> - Eliminated redundant GET_REG_BASE() macros.
>>> - Removed isp4sd_is_stream_running() function.
>>> - Corrected error message in isp4sd_init_stream() caused by copy/paste.
>>> - Refined struct isp4_interface to remove firmware ring buffer configurations.
>>> - Removed obsolete isp4sd_is_stream_running function.
>>> - Removed pdev member from struct isp4_device, as it is unnecessary.
>>> - Fixed typo in 'isp_mmip' parameter name within isp4if_init().
>>> - Removed gap in struct isp4_subdev definition.
>>> - Performed extensive dead code removal and minor style improvements throughout the codebase.
>>>
>>>
>>> Changes v4 -> v5:
>>>
>>> - Transitioned VIDEOBUF2_V4L2 from 'depends' to 'select' within Kconfig.
>>> - Standardized object file naming conventions in the Makefile and sorted entries alphabetically.
>>> - Removed the unused macro definition to_isp4_device.
>>> - Eliminated unused members mem_domain and mem_align from struct isp4if_gpu_mem_info.
>>> - Deleted unused fields mc_addr and gpu_pkg from struct isp4if_cmd_element.
>>> - Removed obsolete pltf_data, i2c_nb, and notifier elements from struct isp4_device.
>>> - Updated platform_get_irq failure handling to return its actual result rather than -ENODEV.
>>> - Refined inclusion of header files for clarity and efficiency.
>>> - Appended comments following #endif statements in header files.
>>> - Improved implementation of isp4if_gpu_mem_free and isp4if_dealloc_fw_gpumem.
>>> - Removed isp4if_append_cmd_2_cmdq and revised isp4if_send_fw_cmd accordingly.
>>> - Enhanced isp4if_clear_cmdq and isp4if_clear_bufq by eliminating unnecessary list_del operations.
>>> - Adopted completion mechanism instead of wait queue and condition for command completion notifications.
>>> - Employed memset to ensure proper zeroing of padding bits in structures shared between ISP driver and firmware.
>>> - Streamlined IRQs, reducing total from four to two, retaining only essential ones.
>>> - Optimized IRQ handler logic using a while loop for greater efficiency.
>>> - Introduced dynamic IRQ enable/disable functionality based on camera status (open/close).
>>> - Applied distinct identifiers to differentiate multiple threads and IRQs.
>>> - Removed unnecessary initialization of local variables.
>>> - Refined camera start/stop workflow to mitigate potential synchronization concerns.
>>> - Replaced all remaining mutex with guard mutex.
>>> - Enhanced command and buffer queue performance by substituting mutexes with spinlocks.
>>> - Removed redundant isp4sd_init_meta_buf function and its references.
>>> - Limited firmware logging activities to the stream1 thread.
>>> - Relocated v4l2_device_unregister_subdev() and media_entity_cleanup() calls from isp4_capture_remove to isp4sd_deinit.
>>> - Resolved media device registration sequence issues.
>>> - Modified stream processing thread behavior to await IRQ without a timeout.
>>> - Addressed cleanup procedures in video device initialization and deinitialization routines.
>>> - Corrected typos and made other cosmetic improvements.
>>>
>>>
>>> Changes v3 -> v4:
>>>
>>> - Replace one mutex with guard mutex.
>>> - Remove unnecessary bus_info initialization of v4l2_capability.
>>> - Drop V4L2_CAP_IO_MC from capabilities of v4l2_capability.
>>> - Modify document with better SOC description.
>>> - Fix Test x86 failure in Media CI test https://linux-media.pages.freedesktop.org/-/users/patchwork/-/jobs/83470456/artifacts/report.htm
>>> - Modify some commit messages by describing changes in imperative mood.
>>> - Add media-ctl output in cover letter.
>>> - Create separated dedicated amdgpu patch to add declaration MODULE_FIRMWARE("amdgpu/isp_4_1_1.bin");
>>> - Fix typo errors and other cosmetic issues.
>>> - Add DRM_AMD_ISP dependency in Kconfig.
>>>
>>>
>>> Changes v2 -> v3:
>>>
>>> - All the dependent patches in other modules (drm/amd/amdgpu, platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
>>> - Removed usage of amdgpu structs in ISP driver. Added helper functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
>>> - Moved sensor and MIPI phy control entirely into ISP FW instead of the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
>>> - Removed phy configuration and sensor binding as x86 (sensor driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
>>> - New FW with built-in sensor driver is submitted on upstream linux-firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
>>> - Please note the new FW submitted is not directly compatible with OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
>>> - If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
>>> - Included critical fixes from Sultan Alsawaf branch (https://github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
>>> media: amd: isp4: Add missing refcount tracking to mmap memop
>>> media: amd: isp4: Don't put or unmap the dmabuf when detaching
>>> media: amd: isp4: Don't increment refcount when dmabuf export fails
>>> media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
>>> media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
>>> media: amd: isp4: Fix implicit dmabuf lifetime tracking
>>> media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
>>> media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
>>> media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
>>> media: amd: isp4: Remove unused userptr memops
>>> media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
>>> media: amd: isp4: Release queued buffers on error in start_streaming
>>> - Addressed all code related upstream comments
>>> - Fix typo errors and other cosmetic issues.
>>>
>>>
>>> Changes v1 -> v2:
>>>
>>> - Fix media CI test errors and valid warnings
>>> - Reduce patch number in the series from 9 to 8 by merging MAINTAINERS adding patch to the first patch
>>> - In patch 5
>>> - do modification to use remote endpoint instead of local endpoint
>>> - use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint
>>>
>>> ----------
>>>
>>> It passes v4l2 compliance test, the test reports for:
>>>
>>> (a) amd_isp_capture device /dev/video0
>>>
>>> Compliance test for amd_isp_capture device /dev/video0:
>>> -------------------------------------------------------
>>>
>>> atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
>>> v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
>>> v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17
>>>
>>> Compliance test for amd_isp_capture device /dev/video0:
>>>
>>> Driver Info:
>>> Driver name : amd_isp_capture
>>> Card type : amd_isp_capture
>>> Bus info : platform:amd_isp_capture
>>> Driver version : 6.14.0
>>> Capabilities : 0xa4200001
>>> Video Capture
>>> I/O MC
>>> Streaming
>>> Extended Pix Format
>>> Device Capabilities
>>> Device Caps : 0x24200001
>>> Video Capture
>>> I/O MC
>>> Streaming
>>> Extended Pix Format
>>> Media Driver Info:
>>> Driver name : amd_isp_capture
>>> Model : amd_isp41_mdev
>>> Serial :
>>> Bus info : platform:amd_isp_capture
>>> Media version : 6.14.0
>>> Hardware revision: 0x00000000 (0)
>>> Driver version : 6.14.0
>>> Interface Info:
>>> ID : 0x03000005
>>> Type : V4L Video
>>> Entity Info:
>>> ID : 0x00000003 (3)
>>> Name : Preview
>>> Function : V4L2 I/O
>>> Pad 0x01000004 : 0: Sink
>>> Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable
>>>
>>> Required ioctls:
>>> test MC information (see 'Media Driver Info' above): OK
>>> test VIDIOC_QUERYCAP: OK
>>> test invalid ioctls: OK
>>>
>>> Allow for multiple opens:
>>> test second /dev/video0 open: OK
>>> test VIDIOC_QUERYCAP: OK
>>> test VIDIOC_G/S_PRIORITY: OK
>>> test for unlimited opens: OK
>>>
>>> Debug ioctls:
>>> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
>>> test VIDIOC_LOG_STATUS: OK (Not Supported)
>>>
>>> Input ioctls:
>>> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
>>> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>>> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
>>> test VIDIOC_ENUMAUDIO: OK (Not Supported)
>>> test VIDIOC_G/S/ENUMINPUT: OK
>>> test VIDIOC_G/S_AUDIO: OK (Not Supported)
>>> Inputs: 1 Audio Inputs: 0 Tuners: 0
>>>
>>> Output ioctls:
>>> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
>>> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>>> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
>>> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
>>> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
>>> Outputs: 0 Audio Outputs: 0 Modulators: 0
>>>
>>> Input/Output configuration ioctls:
>>> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
>>> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
>>> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
>>> test VIDIOC_G/S_EDID: OK (Not Supported)
>>>
>>> Control ioctls (Input 0):
>>> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
>>> test VIDIOC_QUERYCTRL: OK (Not Supported)
>>> test VIDIOC_G/S_CTRL: OK (Not Supported)
>>> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
>>> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
>>> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
>>> Standard Controls: 0 Private Controls: 0
>>>
>>> Format ioctls (Input 0):
>>> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
>>> test VIDIOC_G/S_PARM: OK
>>> test VIDIOC_G_FBUF: OK (Not Supported)
>>> test VIDIOC_G_FMT: OK
>>> test VIDIOC_TRY_FMT: OK
>>> test VIDIOC_S_FMT: OK
>>> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
>>> test Cropping: OK (Not Supported)
>>> test Composing: OK (Not Supported)
>>> test Scaling: OK (Not Supported)
>>>
>>> Codec ioctls (Input 0):
>>> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
>>> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
>>> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>>>
>>> Buffer ioctls (Input 0):
>>> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
>>> test CREATE_BUFS maximum buffers: OK
>>> test VIDIOC_REMOVE_BUFS: OK
>>> test VIDIOC_EXPBUF: OK
>>> test Requests: OK (Not Supported)
>>> test blocking wait: OK
>>>
>>> Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0
>>>
>>> The media-ctl output of media device /dev/media0:
>>> -------------------------------------------------------
>>>
>>> atg@atg-HP-PV:~$ media-ctl -p -d /dev/media0
>>> Media controller API version 6.17.0
>>>
>>> Media device information
>>> ------------------------
>>> driver amd_isp_capture
>>> model amd_isp41_mdev
>>> serial
>>> bus info platform:amd_isp_capture
>>> hw revision 0x0
>>> driver version 6.17.0
>>>
>>> Device topology
>>> - entity 1: amd isp4 (1 pad, 1 link, 0 routes)
>>> type V4L2 subdev subtype Unknown flags 0
>>> pad0: Source
>>> -> "Preview":0 [ENABLED,IMMUTABLE]
>>>
>>> - entity 3: Preview (1 pad, 1 link)
>>> type Node subtype V4L flags 0
>>> device node name /dev/video0
>>> pad0: Sink
>>> <- "amd isp4":0 [ENABLED,IMMUTABLE]
>>>
>>> Please review and provide feedback.
>>>
>>> Many thanks,
>>>
>>> Bin Du (7):
>>> media: platform: amd: Introduce amd isp4 capture driver
>>> media: platform: amd: low level support for isp4 firmware
>>> media: platform: amd: Add isp4 fw and hw interface
>>> media: platform: amd: isp4 subdev and firmware loading handling added
>>> media: platform: amd: isp4 video node and buffers handling added
>>> media: platform: amd: isp4 debug fs logging and more descriptive
>>> errors
>>> Documentation: add documentation of AMD isp 4 driver
>>>
>>> Documentation/admin-guide/media/amdisp4-1.rst | 63 +
>>> Documentation/admin-guide/media/amdisp4.dot | 6 +
>>> .../admin-guide/media/v4l-drivers.rst | 1 +
>>> MAINTAINERS | 25 +
>>> drivers/media/platform/Kconfig | 1 +
>>> drivers/media/platform/Makefile | 1 +
>>> drivers/media/platform/amd/Kconfig | 3 +
>>> drivers/media/platform/amd/Makefile | 3 +
>>> drivers/media/platform/amd/isp4/Kconfig | 14 +
>>> drivers/media/platform/amd/isp4/Makefile | 10 +
>>> drivers/media/platform/amd/isp4/isp4.c | 235 ++++
>>> drivers/media/platform/amd/isp4/isp4.h | 20 +
>>> drivers/media/platform/amd/isp4/isp4_debug.c | 271 ++++
>>> drivers/media/platform/amd/isp4/isp4_debug.h | 41 +
>>> .../platform/amd/isp4/isp4_fw_cmd_resp.h | 314 +++++
>>> drivers/media/platform/amd/isp4/isp4_hw_reg.h | 124 ++
>>> .../media/platform/amd/isp4/isp4_interface.c | 789 +++++++++++
>>> .../media/platform/amd/isp4/isp4_interface.h | 141 ++
>>> drivers/media/platform/amd/isp4/isp4_subdev.c | 1057 +++++++++++++++
>>> drivers/media/platform/amd/isp4/isp4_subdev.h | 131 ++
>>> drivers/media/platform/amd/isp4/isp4_video.c | 1165 +++++++++++++++++
>>> drivers/media/platform/amd/isp4/isp4_video.h | 65 +
>>> 22 files changed, 4480 insertions(+)
>>> create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst
>>> create mode 100644 Documentation/admin-guide/media/amdisp4.dot
>>> create mode 100644 drivers/media/platform/amd/Kconfig
>>> create mode 100644 drivers/media/platform/amd/Makefile
>>> create mode 100644 drivers/media/platform/amd/isp4/Kconfig
>>> create mode 100644 drivers/media/platform/amd/isp4/Makefile
>>> create mode 100644 drivers/media/platform/amd/isp4/isp4.c
>>> create mode 100644 drivers/media/platform/amd/isp4/isp4.h
>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c
>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h
>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h
>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c
>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h
>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c
>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h
>>>
>>> --
>>> 2.34.1
>>>
>>>
>>
>> v7 worked on my HP ZBook Ultra G1a 14 inch Mobile Workstation PC and
>> the camera worked fine.
>> It was tested with the latest upstream firmware [1] and 6.19-rc3 kernel.
>>
>> [1] https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/amdgpu/isp_4_1_1.bin
>>
>> Thank you for your work :)
>>
>> --
>> BR,
>> Kate
> Leave a "Tested-by" tag for my previous testing assertion.
>
> Tested-by: Kate Hsuan <hpa@redhat.com>
>
>
>
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 0/7] Add AMD ISP4 driver
2026-01-06 8:35 ` Du, Bin
@ 2026-01-13 14:11 ` Kate Hsuan
2026-01-13 17:14 ` Nirujogi, Pratap
0 siblings, 1 reply; 46+ messages in thread
From: Kate Hsuan @ 2026-01-13 14:11 UTC (permalink / raw)
To: Du, Bin
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
sultan, pratap.nirujogi, benjamin.chan, king.li,
gjorgji.rosikopulos, Phil.Jawich, Dominic.Antony,
mario.limonciello, richard.gong, anson.tsao
[-- Attachment #1: Type: text/plain, Size: 23483 bytes --]
Hi Bin,
On Tue, Jan 6, 2026 at 4:35 PM Du, Bin <bin.du@amd.com> wrote:
>
> Thank you, Kate, for the verification, it gives us more confidence.
>
> On 1/6/2026 1:49 PM, Kate Hsuan wrote:
> > Hi
> >
> > On Wed, Dec 31, 2025 at 5:03 PM Kate Hsuan <hpa@redhat.com> wrote:
> >>
> >> On Tue, Dec 16, 2025 at 5:14 PM Bin Du <Bin.Du@amd.com> wrote:
> >>>
> >>> Hello,
> >>>
> >>> AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC (Ryzen AI Max 300 Series)
> >>> (https://ubuntu.com/certified/202411-36043)
> >>> This patch series introduces the initial driver support for the AMD ISP4.
> >>>
> >>> Patch summary:
> >>> - Powers up/off and initializes ISP HW
> >>> - Configures and kicks off ISP FW
> >>> - Interacts with APP using standard V4l2 interface by video node
> >>> - Controls ISP HW and interacts with ISP FW to do image processing
> >>> - Supports enum/set output image format and resolution
> >>> - Supports queueing buffer from app and dequeuing ISP filled buffer to App
> >>> - It is verified on qv4l2, cheese and qcam
> >>> - It is verified together with following patches
> >>> platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/20250514215623.522746-1-pratap.nirujogi@amd.com/)
> >>> pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
> >>> drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)
> >>> drm/amd/amdgpu: Declare isp firmware binary file (https://gitlab.freedesktop.org/agd5f/linux/-/commit/35345917bc9f7c86152b270d9d93c220230b667f)
> >>>
> >>> AMD ISP4 Key features:
> >>> - Processes bayer raw data from the connected sensor and output them to different YUV formats
> >>> - Downscale input image to different output image resolution
> >>> - Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc.
> >>>
> >>> ----------
> >>>
> >>> Changes v6 -> v7:
> >>>
> >>> - Added missed blank line after some if statements.
> >>> - Changed the pm_runtime_disable() order in isp4_capture_remove() to align with initialization.
> >>> - Reset buf_sent_cnt and start_stream_cmd_sent on stream stop.
> >>> - Removed duplicate buf_sent_cnt and start_stream_cmd_sent reset in isp4sd_pwron_and_init().
> >>> - Combined isp4sd_reset_stream_info() and isp4sd_reset_camera_info() into isp4sd_uninit_stream() to eliminate redundant stream info reset.
> >>> - Removed always-false status check in isp4sd_uninit_stream().
> >>> - Minor style improvements.
> >>>
> >>>
> >>> Changes v5 -> v6:
> >>>
> >>> - Lowered FW mempool buffer size from 200M to 100M (actual usage).
> >>> - Added an irq_enabled member to the ISP subdev for proper IRQ disable handling in both normal and error cases.
> >>> - Removed unnecessary .owner assignment from isp4_capture_drv definition
> >>> - Updated IRQ handling to enable and disable interrupts via the ISP register for improved performance.
> >>> - Revised ring buffer management in isp4if_f2h_resp(), addressing safety checks to ensure the read pointer is validated before memcpy operations, reducing the risk of out-of-bounds access. The ring buffer logic was also streamlined significantly.
> >>> - Modified ring buffer handling in isp4if_is_cmdq_rb_full(), correcting an off-by-one error in safety checks that previously allowed rd_ptr to equal wr_ptr when the buffer was full.
> >>> - Refactored ring buffer handling in isp4if_insert_isp_fw_cmd(), simplifying overall logic.
> >>> - Resolved a regression from v4 to v5 where isp4if_dequeue_buffer() did not protect list_del with bufq_lock.
> >>> - Addressed a subtle use-after-free issue that could occur if a timeout on a synchronous command coincided with completion.
> >>> - Added missing pm_runtime_disable() calls to isp4_capture_remove() and to the error path in isp4_capture_probe().
> >>> - Removed stray semicolons following closing curly braces.
> >>> - Improved and clarified macro definitions in isp4_interface.h.
> >>> - Eliminated unnecessary (u8 *) casts.
> >>> - Added missing memset for firmware command structures in isp4sd_stop_stream().
> >>> - Excluded streams 2 and 3 from ISP4IF_FW_RESP_RB_IRQ_EN_MASK, preventing their activation in the interrupt enable register.
> >>> - Enhanced error handling to clean up kthreads in the event of startup failure.
> >>> - Corrected a race condition during kthread creation where waitqueue head initialization could be delayed, as it was performed by the kthread itself.
> >>> - Removed status checks in isp4sd_pwroff_and_deinit() that were always false.
> >>> - Ensured isp4sd_init_stream() is only invoked once per stream start and reordered corresponding status checks in isp4sd_start_stream().
> >>> - Improved error handling in isp4sd_start_stream() to propagate errors from failed functions.
> >>> - Relocated debugging messages in isp4sd_stop_stream() to execute under lock protection due to access to output_info->start_status.
> >>> - Eliminated redundant GET_REG_BASE() macros.
> >>> - Removed isp4sd_is_stream_running() function.
> >>> - Corrected error message in isp4sd_init_stream() caused by copy/paste.
> >>> - Refined struct isp4_interface to remove firmware ring buffer configurations.
> >>> - Removed obsolete isp4sd_is_stream_running function.
> >>> - Removed pdev member from struct isp4_device, as it is unnecessary.
> >>> - Fixed typo in 'isp_mmip' parameter name within isp4if_init().
> >>> - Removed gap in struct isp4_subdev definition.
> >>> - Performed extensive dead code removal and minor style improvements throughout the codebase.
> >>>
> >>>
> >>> Changes v4 -> v5:
> >>>
> >>> - Transitioned VIDEOBUF2_V4L2 from 'depends' to 'select' within Kconfig.
> >>> - Standardized object file naming conventions in the Makefile and sorted entries alphabetically.
> >>> - Removed the unused macro definition to_isp4_device.
> >>> - Eliminated unused members mem_domain and mem_align from struct isp4if_gpu_mem_info.
> >>> - Deleted unused fields mc_addr and gpu_pkg from struct isp4if_cmd_element.
> >>> - Removed obsolete pltf_data, i2c_nb, and notifier elements from struct isp4_device.
> >>> - Updated platform_get_irq failure handling to return its actual result rather than -ENODEV.
> >>> - Refined inclusion of header files for clarity and efficiency.
> >>> - Appended comments following #endif statements in header files.
> >>> - Improved implementation of isp4if_gpu_mem_free and isp4if_dealloc_fw_gpumem.
> >>> - Removed isp4if_append_cmd_2_cmdq and revised isp4if_send_fw_cmd accordingly.
> >>> - Enhanced isp4if_clear_cmdq and isp4if_clear_bufq by eliminating unnecessary list_del operations.
> >>> - Adopted completion mechanism instead of wait queue and condition for command completion notifications.
> >>> - Employed memset to ensure proper zeroing of padding bits in structures shared between ISP driver and firmware.
> >>> - Streamlined IRQs, reducing total from four to two, retaining only essential ones.
> >>> - Optimized IRQ handler logic using a while loop for greater efficiency.
> >>> - Introduced dynamic IRQ enable/disable functionality based on camera status (open/close).
> >>> - Applied distinct identifiers to differentiate multiple threads and IRQs.
> >>> - Removed unnecessary initialization of local variables.
> >>> - Refined camera start/stop workflow to mitigate potential synchronization concerns.
> >>> - Replaced all remaining mutex with guard mutex.
> >>> - Enhanced command and buffer queue performance by substituting mutexes with spinlocks.
> >>> - Removed redundant isp4sd_init_meta_buf function and its references.
> >>> - Limited firmware logging activities to the stream1 thread.
> >>> - Relocated v4l2_device_unregister_subdev() and media_entity_cleanup() calls from isp4_capture_remove to isp4sd_deinit.
> >>> - Resolved media device registration sequence issues.
> >>> - Modified stream processing thread behavior to await IRQ without a timeout.
> >>> - Addressed cleanup procedures in video device initialization and deinitialization routines.
> >>> - Corrected typos and made other cosmetic improvements.
> >>>
> >>>
> >>> Changes v3 -> v4:
> >>>
> >>> - Replace one mutex with guard mutex.
> >>> - Remove unnecessary bus_info initialization of v4l2_capability.
> >>> - Drop V4L2_CAP_IO_MC from capabilities of v4l2_capability.
> >>> - Modify document with better SOC description.
> >>> - Fix Test x86 failure in Media CI test https://linux-media.pages.freedesktop.org/-/users/patchwork/-/jobs/83470456/artifacts/report.htm
> >>> - Modify some commit messages by describing changes in imperative mood.
> >>> - Add media-ctl output in cover letter.
> >>> - Create separated dedicated amdgpu patch to add declaration MODULE_FIRMWARE("amdgpu/isp_4_1_1.bin");
> >>> - Fix typo errors and other cosmetic issues.
> >>> - Add DRM_AMD_ISP dependency in Kconfig.
> >>>
> >>>
> >>> Changes v2 -> v3:
> >>>
> >>> - All the dependent patches in other modules (drm/amd/amdgpu, platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
> >>> - Removed usage of amdgpu structs in ISP driver. Added helper functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
> >>> - Moved sensor and MIPI phy control entirely into ISP FW instead of the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
> >>> - Removed phy configuration and sensor binding as x86 (sensor driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
> >>> - New FW with built-in sensor driver is submitted on upstream linux-firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
> >>> - Please note the new FW submitted is not directly compatible with OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
> >>> - If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
> >>> - Included critical fixes from Sultan Alsawaf branch (https://github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
> >>> media: amd: isp4: Add missing refcount tracking to mmap memop
> >>> media: amd: isp4: Don't put or unmap the dmabuf when detaching
> >>> media: amd: isp4: Don't increment refcount when dmabuf export fails
> >>> media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
> >>> media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
> >>> media: amd: isp4: Fix implicit dmabuf lifetime tracking
> >>> media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
> >>> media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
> >>> media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
> >>> media: amd: isp4: Remove unused userptr memops
> >>> media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
> >>> media: amd: isp4: Release queued buffers on error in start_streaming
> >>> - Addressed all code related upstream comments
> >>> - Fix typo errors and other cosmetic issues.
> >>>
> >>>
> >>> Changes v1 -> v2:
> >>>
> >>> - Fix media CI test errors and valid warnings
> >>> - Reduce patch number in the series from 9 to 8 by merging MAINTAINERS adding patch to the first patch
> >>> - In patch 5
> >>> - do modification to use remote endpoint instead of local endpoint
> >>> - use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint
> >>>
> >>> ----------
> >>>
> >>> It passes v4l2 compliance test, the test reports for:
> >>>
> >>> (a) amd_isp_capture device /dev/video0
> >>>
> >>> Compliance test for amd_isp_capture device /dev/video0:
> >>> -------------------------------------------------------
> >>>
> >>> atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
> >>> v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
> >>> v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17
> >>>
> >>> Compliance test for amd_isp_capture device /dev/video0:
> >>>
> >>> Driver Info:
> >>> Driver name : amd_isp_capture
> >>> Card type : amd_isp_capture
> >>> Bus info : platform:amd_isp_capture
> >>> Driver version : 6.14.0
> >>> Capabilities : 0xa4200001
> >>> Video Capture
> >>> I/O MC
> >>> Streaming
> >>> Extended Pix Format
> >>> Device Capabilities
> >>> Device Caps : 0x24200001
> >>> Video Capture
> >>> I/O MC
> >>> Streaming
> >>> Extended Pix Format
> >>> Media Driver Info:
> >>> Driver name : amd_isp_capture
> >>> Model : amd_isp41_mdev
> >>> Serial :
> >>> Bus info : platform:amd_isp_capture
> >>> Media version : 6.14.0
> >>> Hardware revision: 0x00000000 (0)
> >>> Driver version : 6.14.0
> >>> Interface Info:
> >>> ID : 0x03000005
> >>> Type : V4L Video
> >>> Entity Info:
> >>> ID : 0x00000003 (3)
> >>> Name : Preview
> >>> Function : V4L2 I/O
> >>> Pad 0x01000004 : 0: Sink
> >>> Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable
> >>>
> >>> Required ioctls:
> >>> test MC information (see 'Media Driver Info' above): OK
> >>> test VIDIOC_QUERYCAP: OK
> >>> test invalid ioctls: OK
> >>>
> >>> Allow for multiple opens:
> >>> test second /dev/video0 open: OK
> >>> test VIDIOC_QUERYCAP: OK
> >>> test VIDIOC_G/S_PRIORITY: OK
> >>> test for unlimited opens: OK
> >>>
> >>> Debug ioctls:
> >>> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> >>> test VIDIOC_LOG_STATUS: OK (Not Supported)
> >>>
> >>> Input ioctls:
> >>> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> >>> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> >>> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> >>> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> >>> test VIDIOC_G/S/ENUMINPUT: OK
> >>> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> >>> Inputs: 1 Audio Inputs: 0 Tuners: 0
> >>>
> >>> Output ioctls:
> >>> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> >>> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> >>> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> >>> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> >>> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> >>> Outputs: 0 Audio Outputs: 0 Modulators: 0
> >>>
> >>> Input/Output configuration ioctls:
> >>> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> >>> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> >>> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> >>> test VIDIOC_G/S_EDID: OK (Not Supported)
> >>>
> >>> Control ioctls (Input 0):
> >>> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> >>> test VIDIOC_QUERYCTRL: OK (Not Supported)
> >>> test VIDIOC_G/S_CTRL: OK (Not Supported)
> >>> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> >>> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> >>> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> >>> Standard Controls: 0 Private Controls: 0
> >>>
> >>> Format ioctls (Input 0):
> >>> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> >>> test VIDIOC_G/S_PARM: OK
> >>> test VIDIOC_G_FBUF: OK (Not Supported)
> >>> test VIDIOC_G_FMT: OK
> >>> test VIDIOC_TRY_FMT: OK
> >>> test VIDIOC_S_FMT: OK
> >>> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> >>> test Cropping: OK (Not Supported)
> >>> test Composing: OK (Not Supported)
> >>> test Scaling: OK (Not Supported)
> >>>
> >>> Codec ioctls (Input 0):
> >>> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> >>> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> >>> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
> >>>
> >>> Buffer ioctls (Input 0):
> >>> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> >>> test CREATE_BUFS maximum buffers: OK
> >>> test VIDIOC_REMOVE_BUFS: OK
> >>> test VIDIOC_EXPBUF: OK
> >>> test Requests: OK (Not Supported)
> >>> test blocking wait: OK
> >>>
> >>> Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0
> >>>
> >>> The media-ctl output of media device /dev/media0:
> >>> -------------------------------------------------------
> >>>
> >>> atg@atg-HP-PV:~$ media-ctl -p -d /dev/media0
> >>> Media controller API version 6.17.0
> >>>
> >>> Media device information
> >>> ------------------------
> >>> driver amd_isp_capture
> >>> model amd_isp41_mdev
> >>> serial
> >>> bus info platform:amd_isp_capture
> >>> hw revision 0x0
> >>> driver version 6.17.0
> >>>
> >>> Device topology
> >>> - entity 1: amd isp4 (1 pad, 1 link, 0 routes)
> >>> type V4L2 subdev subtype Unknown flags 0
> >>> pad0: Source
> >>> -> "Preview":0 [ENABLED,IMMUTABLE]
> >>>
> >>> - entity 3: Preview (1 pad, 1 link)
> >>> type Node subtype V4L flags 0
> >>> device node name /dev/video0
> >>> pad0: Sink
> >>> <- "amd isp4":0 [ENABLED,IMMUTABLE]
> >>>
> >>> Please review and provide feedback.
> >>>
> >>> Many thanks,
> >>>
> >>> Bin Du (7):
> >>> media: platform: amd: Introduce amd isp4 capture driver
> >>> media: platform: amd: low level support for isp4 firmware
> >>> media: platform: amd: Add isp4 fw and hw interface
> >>> media: platform: amd: isp4 subdev and firmware loading handling added
> >>> media: platform: amd: isp4 video node and buffers handling added
> >>> media: platform: amd: isp4 debug fs logging and more descriptive
> >>> errors
> >>> Documentation: add documentation of AMD isp 4 driver
> >>>
> >>> Documentation/admin-guide/media/amdisp4-1.rst | 63 +
> >>> Documentation/admin-guide/media/amdisp4.dot | 6 +
> >>> .../admin-guide/media/v4l-drivers.rst | 1 +
> >>> MAINTAINERS | 25 +
> >>> drivers/media/platform/Kconfig | 1 +
> >>> drivers/media/platform/Makefile | 1 +
> >>> drivers/media/platform/amd/Kconfig | 3 +
> >>> drivers/media/platform/amd/Makefile | 3 +
> >>> drivers/media/platform/amd/isp4/Kconfig | 14 +
> >>> drivers/media/platform/amd/isp4/Makefile | 10 +
> >>> drivers/media/platform/amd/isp4/isp4.c | 235 ++++
> >>> drivers/media/platform/amd/isp4/isp4.h | 20 +
> >>> drivers/media/platform/amd/isp4/isp4_debug.c | 271 ++++
> >>> drivers/media/platform/amd/isp4/isp4_debug.h | 41 +
> >>> .../platform/amd/isp4/isp4_fw_cmd_resp.h | 314 +++++
> >>> drivers/media/platform/amd/isp4/isp4_hw_reg.h | 124 ++
> >>> .../media/platform/amd/isp4/isp4_interface.c | 789 +++++++++++
> >>> .../media/platform/amd/isp4/isp4_interface.h | 141 ++
> >>> drivers/media/platform/amd/isp4/isp4_subdev.c | 1057 +++++++++++++++
> >>> drivers/media/platform/amd/isp4/isp4_subdev.h | 131 ++
> >>> drivers/media/platform/amd/isp4/isp4_video.c | 1165 +++++++++++++++++
> >>> drivers/media/platform/amd/isp4/isp4_video.h | 65 +
> >>> 22 files changed, 4480 insertions(+)
> >>> create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst
> >>> create mode 100644 Documentation/admin-guide/media/amdisp4.dot
> >>> create mode 100644 drivers/media/platform/amd/Kconfig
> >>> create mode 100644 drivers/media/platform/amd/Makefile
> >>> create mode 100644 drivers/media/platform/amd/isp4/Kconfig
> >>> create mode 100644 drivers/media/platform/amd/isp4/Makefile
> >>> create mode 100644 drivers/media/platform/amd/isp4/isp4.c
> >>> create mode 100644 drivers/media/platform/amd/isp4/isp4.h
> >>> create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c
> >>> create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h
> >>> create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
> >>> create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h
> >>> create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c
> >>> create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h
> >>> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
> >>> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
> >>> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c
> >>> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h
> >>>
> >>> --
> >>> 2.34.1
> >>>
> >>>
> >>
> >> v7 worked on my HP ZBook Ultra G1a 14 inch Mobile Workstation PC and
> >> the camera worked fine.
> >> It was tested with the latest upstream firmware [1] and 6.19-rc3 kernel.
> >>
> >> [1] https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/amdgpu/isp_4_1_1.bin
> >>
> >> Thank you for your work :)
> >>
> >> --
> >> BR,
> >> Kate
> > Leave a "Tested-by" tag for my previous testing assertion.
> >
> > Tested-by: Kate Hsuan <hpa@redhat.com>
> >
> >
> >
>
> --
> Regards,
> Bin
>
The ISP driver works perfectly with a clear and sharp video. I tested
it again, and I found a suspend/resume issue.
The ISP can't suspend when the system is set to suspend. The privacy
LED is on when the system is suspended. Therefore, the user will see a
luminous privacy LED when the system is set to suspend.
Today, I made a work to move all the power control to use the runtime
PM, including suspend/resume. This work may be humble and may break
the finite state machine but it works. The major changes of it
include:
1. Support suspend/resume.
2. The power is managed by the runtime PM so the s_power and the related
callback function were dropped.
3. The enable_isp GPIO pin is controlled by the runtime PM.
4. pm_runtime_get_noresume() is used to get the runtime PM at probe()
since the device doesn't have to be set to power on when initialising.
This work stops the video stream on suspend and starts the stream on
resume so the privacy LED is turned on and turned off with the changes
of suspend and resume.
Could you please consider this patch and idea?
Thank you :)
--
BR,
Kate
[-- Attachment #2: 0001-media-platform-amd-isp4-support-suspend-resume-using.patch --]
[-- Type: text/x-patch, Size: 6562 bytes --]
From 62ff53ded5c54f5aaff3f46754e9720bb0e0cad6 Mon Sep 17 00:00:00 2001
From: Kate Hsuan <hpa@redhat.com>
Date: Tue, 13 Jan 2026 21:20:42 +0800
Subject: [PATCH] media: platform: amd: isp4: support suspend/resume using
runtime PM
This work enables the suspend/resume feature of the ISP4. The major
changes includes:
1. Support suspend/resume.
2. The power is managed by the runtime PM so the s_power and the related
callback function were dropped.
3. The enable_isp GPIO pin is controlled by the runtime PM.
4. pm_runtime_get_noresume() is used to get the runtime PM at probe()
since the device doesn't have to be set to power on when initialising.
Signed-off-by: Kate Hsuan <hpa@redhat.com>
---
drivers/media/platform/amd/isp4/isp4.c | 55 ++++++++++++++++++-
drivers/media/platform/amd/isp4/isp4_subdev.c | 48 +++++++++-------
2 files changed, 82 insertions(+), 21 deletions(-)
diff --git a/drivers/media/platform/amd/isp4/isp4.c b/drivers/media/platform/amd/isp4/isp4.c
index f35bc8f1a259..87a07b59356f 100644
--- a/drivers/media/platform/amd/isp4/isp4.c
+++ b/drivers/media/platform/amd/isp4/isp4.c
@@ -160,10 +160,13 @@ static int isp4_capture_probe(struct platform_device *pdev)
goto err_clean_media;
}
- pm_runtime_set_suspended(dev);
- pm_runtime_enable(dev);
spin_lock_init(&isp_subdev->irq_lock);
ret = isp4sd_init(&isp_dev->isp_subdev, &isp_dev->v4l2_dev, irq);
+
+ pm_runtime_set_active(dev);
+ pm_runtime_get_noresume(dev);
+ pm_runtime_enable(dev);
+
if (ret) {
dev_err_probe(dev, ret, "fail init isp4 sub dev\n");
goto err_pm_disable;
@@ -188,6 +191,10 @@ static int isp4_capture_probe(struct platform_device *pdev)
platform_set_drvdata(pdev, isp_dev);
isp_debugfs_create(isp_dev);
+ pm_runtime_set_autosuspend_delay(dev, 1000);
+ pm_runtime_use_autosuspend(dev);
+ pm_runtime_put_autosuspend(dev);
+
return 0;
err_isp4_deinit:
@@ -201,6 +208,42 @@ static int isp4_capture_probe(struct platform_device *pdev)
return ret;
}
+static int isp4_suspend(struct device *dev)
+{
+ struct isp4_device *isp_dev = dev_get_drvdata(dev);
+ struct isp4_subdev *isp_subdev = &isp_dev->isp_subdev;
+ int ret = 0;
+ dev_dbg(dev, "ISP4 power off\n");
+
+ v4l2_subdev_disable_streams (&isp_subdev->sdev, isp_subdev->isp_vdev.vdev_pad.index, BIT(0));
+ ret = gpiod_set_value(isp_subdev->enable_gpio, 0);
+ if (ret) {
+ dev_err(dev, "fail to set enable_isp gpio\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static int isp4_resume(struct device *dev)
+{
+ struct isp4_device *isp_dev = dev_get_drvdata(dev);
+ struct isp4_subdev *isp_subdev = &isp_dev->isp_subdev;
+ int ret;
+ dev_dbg(dev, "ISP4 power on \n");
+
+ ret = gpiod_set_value(isp_subdev->enable_gpio, 1);
+ if (ret) {
+ dev_err(dev, "fail to set enable_isp gpio\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static DEFINE_RUNTIME_DEV_PM_OPS(isp4_pm_ops, isp4_suspend,
+ isp4_resume, NULL);
+
static void isp4_capture_remove(struct platform_device *pdev)
{
struct isp4_device *isp_dev = platform_get_drvdata(pdev);
@@ -213,6 +256,13 @@ static void isp4_capture_remove(struct platform_device *pdev)
v4l2_device_unregister(&isp_dev->v4l2_dev);
media_device_cleanup(&isp_dev->mdev);
pm_runtime_disable(dev);
+
+ /*
+ * Ensure the power is off before removing the device.
+ */
+ if (!pm_runtime_status_suspended(dev))
+ isp4_suspend(dev);
+ pm_runtime_set_suspended(dev);
}
static struct platform_driver isp4_capture_drv = {
@@ -220,6 +270,7 @@ static struct platform_driver isp4_capture_drv = {
.remove = isp4_capture_remove,
.driver = {
.name = ISP4_DRV_NAME,
+ .pm = pm_sleep_ptr(&isp4_pm_ops),
}
};
diff --git a/drivers/media/platform/amd/isp4/isp4_subdev.c b/drivers/media/platform/amd/isp4/isp4_subdev.c
index 2612ca283fc0..d53f2ab66783 100644
--- a/drivers/media/platform/amd/isp4/isp4_subdev.c
+++ b/drivers/media/platform/amd/isp4/isp4_subdev.c
@@ -730,6 +730,12 @@ static int isp4sd_pwron_and_init(struct isp4_subdev *isp_subdev)
enable_irq(isp_subdev->irq[i]);
isp_subdev->irq_enabled = true;
+ /*
+ * Hardware requires at least a 20ms delay between disabling and enabling the module,
+ * so a sleep is added to ensure ISP stability during quick reopen scenarios.
+ */
+ msleep(20);
+
return 0;
err_deinit:
isp4sd_pwroff_and_deinit(isp_subdev);
@@ -748,6 +754,11 @@ static int isp4sd_stop_stream(struct isp4_subdev *isp_subdev,
guard(mutex)(&isp_subdev->ops_mutex);
dev_dbg(dev, "status %i\n", output_info->start_status);
+ if (output_info->start_status == ISP4SD_START_STATUS_OFF) {
+ dev_dbg(dev, "stream already stopped, do nothing\n");
+ return 0;
+ }
+
if (output_info->start_status == ISP4SD_START_STATUS_STARTED) {
struct isp4fw_cmd_enable_out_ch cmd_ch_disable;
@@ -781,6 +792,14 @@ static int isp4sd_stop_stream(struct isp4_subdev *isp_subdev,
isp4sd_uninit_stream(isp_subdev, state, pad);
+ isp4sd_pwroff_and_deinit(isp_subdev);
+
+ ret = pm_runtime_put(isp_subdev->sdev.dev);
+ if (ret) {
+ dev_err(dev, "fail on pm_runtime_put\n");
+ return ret;
+ }
+
/*
* Return success to ensure the stop process proceeds,
* and disregard any errors since they are not fatal.
@@ -799,9 +818,16 @@ static int isp4sd_start_stream(struct isp4_subdev *isp_subdev,
guard(mutex)(&isp_subdev->ops_mutex);
- if (ispif->status != ISP4IF_STATUS_FW_RUNNING) {
- dev_err(dev, "fail, bad fsm %d", ispif->status);
- return -EINVAL;
+ ret = pm_runtime_resume_and_get(isp_subdev->sdev.dev);
+ if (ret) {
+ dev_err(dev, "fail to get runtime pm\n");
+ return ret;
+ }
+
+ ret = isp4sd_pwron_and_init(isp_subdev);
+ if (ret) {
+ dev_err(dev, "fail to power on isp_subdev ret %d\n", ret);
+ return ret;
}
switch (output_info->start_status) {
@@ -894,21 +920,6 @@ static int isp4sd_ioc_send_img_buf(struct v4l2_subdev *sd,
return ret;
}
-static int isp4sd_set_power(struct v4l2_subdev *sd, int on)
-{
- struct isp4_subdev *isp_subdev = to_isp4_subdev(sd);
-
- guard(mutex)(&isp_subdev->ops_mutex);
- if (on)
- return isp4sd_pwron_and_init(isp_subdev);
- else
- return isp4sd_pwroff_and_deinit(isp_subdev);
-}
-
-static const struct v4l2_subdev_core_ops isp4sd_core_ops = {
- .s_power = isp4sd_set_power,
-};
-
static const struct v4l2_subdev_video_ops isp4sd_video_ops = {
.s_stream = v4l2_subdev_s_stream_helper,
};
@@ -979,7 +990,6 @@ static const struct v4l2_subdev_pad_ops isp4sd_pad_ops = {
};
static const struct v4l2_subdev_ops isp4sd_subdev_ops = {
- .core = &isp4sd_core_ops,
.video = &isp4sd_video_ops,
.pad = &isp4sd_pad_ops,
};
--
2.52.0
^ permalink raw reply related [flat|nested] 46+ messages in thread* Re: [PATCH v7 0/7] Add AMD ISP4 driver
2026-01-13 14:11 ` Kate Hsuan
@ 2026-01-13 17:14 ` Nirujogi, Pratap
2026-01-14 8:59 ` Kate Hsuan
0 siblings, 1 reply; 46+ messages in thread
From: Nirujogi, Pratap @ 2026-01-13 17:14 UTC (permalink / raw)
To: Kate Hsuan, Du, Bin
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
sultan, pratap.nirujogi, benjamin.chan, king.li,
gjorgji.rosikopulos, Phil.Jawich, Dominic.Antony,
mario.limonciello, richard.gong, anson.tsao
Hi Kate,
On 1/13/2026 9:11 AM, Kate Hsuan wrote:
> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
>
>
> Hi Bin,
>
> On Tue, Jan 6, 2026 at 4:35 PM Du, Bin <bin.du@amd.com> wrote:
>> Thank you, Kate, for the verification, it gives us more confidence.
>>
>> On 1/6/2026 1:49 PM, Kate Hsuan wrote:
>>> Hi
>>>
>>> On Wed, Dec 31, 2025 at 5:03 PM Kate Hsuan <hpa@redhat.com> wrote:
>>>> On Tue, Dec 16, 2025 at 5:14 PM Bin Du <Bin.Du@amd.com> wrote:
>>>>> Hello,
>>>>>
>>>>> AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC (Ryzen AI Max 300 Series)
>>>>> (https://ubuntu.com/certified/202411-36043)
>>>>> This patch series introduces the initial driver support for the AMD ISP4.
>>>>>
>>>>> Patch summary:
>>>>> - Powers up/off and initializes ISP HW
>>>>> - Configures and kicks off ISP FW
>>>>> - Interacts with APP using standard V4l2 interface by video node
>>>>> - Controls ISP HW and interacts with ISP FW to do image processing
>>>>> - Supports enum/set output image format and resolution
>>>>> - Supports queueing buffer from app and dequeuing ISP filled buffer to App
>>>>> - It is verified on qv4l2, cheese and qcam
>>>>> - It is verified together with following patches
>>>>> platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/20250514215623.522746-1-pratap.nirujogi@amd.com/)
>>>>> pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
>>>>> drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)
>>>>> drm/amd/amdgpu: Declare isp firmware binary file (https://gitlab.freedesktop.org/agd5f/linux/-/commit/35345917bc9f7c86152b270d9d93c220230b667f)
>>>>>
>>>>> AMD ISP4 Key features:
>>>>> - Processes bayer raw data from the connected sensor and output them to different YUV formats
>>>>> - Downscale input image to different output image resolution
>>>>> - Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc.
>>>>>
>>>>> ----------
>>>>>
>>>>> Changes v6 -> v7:
>>>>>
>>>>> - Added missed blank line after some if statements.
>>>>> - Changed the pm_runtime_disable() order in isp4_capture_remove() to align with initialization.
>>>>> - Reset buf_sent_cnt and start_stream_cmd_sent on stream stop.
>>>>> - Removed duplicate buf_sent_cnt and start_stream_cmd_sent reset in isp4sd_pwron_and_init().
>>>>> - Combined isp4sd_reset_stream_info() and isp4sd_reset_camera_info() into isp4sd_uninit_stream() to eliminate redundant stream info reset.
>>>>> - Removed always-false status check in isp4sd_uninit_stream().
>>>>> - Minor style improvements.
>>>>>
>>>>>
>>>>> Changes v5 -> v6:
>>>>>
>>>>> - Lowered FW mempool buffer size from 200M to 100M (actual usage).
>>>>> - Added an irq_enabled member to the ISP subdev for proper IRQ disable handling in both normal and error cases.
>>>>> - Removed unnecessary .owner assignment from isp4_capture_drv definition
>>>>> - Updated IRQ handling to enable and disable interrupts via the ISP register for improved performance.
>>>>> - Revised ring buffer management in isp4if_f2h_resp(), addressing safety checks to ensure the read pointer is validated before memcpy operations, reducing the risk of out-of-bounds access. The ring buffer logic was also streamlined significantly.
>>>>> - Modified ring buffer handling in isp4if_is_cmdq_rb_full(), correcting an off-by-one error in safety checks that previously allowed rd_ptr to equal wr_ptr when the buffer was full.
>>>>> - Refactored ring buffer handling in isp4if_insert_isp_fw_cmd(), simplifying overall logic.
>>>>> - Resolved a regression from v4 to v5 where isp4if_dequeue_buffer() did not protect list_del with bufq_lock.
>>>>> - Addressed a subtle use-after-free issue that could occur if a timeout on a synchronous command coincided with completion.
>>>>> - Added missing pm_runtime_disable() calls to isp4_capture_remove() and to the error path in isp4_capture_probe().
>>>>> - Removed stray semicolons following closing curly braces.
>>>>> - Improved and clarified macro definitions in isp4_interface.h.
>>>>> - Eliminated unnecessary (u8 *) casts.
>>>>> - Added missing memset for firmware command structures in isp4sd_stop_stream().
>>>>> - Excluded streams 2 and 3 from ISP4IF_FW_RESP_RB_IRQ_EN_MASK, preventing their activation in the interrupt enable register.
>>>>> - Enhanced error handling to clean up kthreads in the event of startup failure.
>>>>> - Corrected a race condition during kthread creation where waitqueue head initialization could be delayed, as it was performed by the kthread itself.
>>>>> - Removed status checks in isp4sd_pwroff_and_deinit() that were always false.
>>>>> - Ensured isp4sd_init_stream() is only invoked once per stream start and reordered corresponding status checks in isp4sd_start_stream().
>>>>> - Improved error handling in isp4sd_start_stream() to propagate errors from failed functions.
>>>>> - Relocated debugging messages in isp4sd_stop_stream() to execute under lock protection due to access to output_info->start_status.
>>>>> - Eliminated redundant GET_REG_BASE() macros.
>>>>> - Removed isp4sd_is_stream_running() function.
>>>>> - Corrected error message in isp4sd_init_stream() caused by copy/paste.
>>>>> - Refined struct isp4_interface to remove firmware ring buffer configurations.
>>>>> - Removed obsolete isp4sd_is_stream_running function.
>>>>> - Removed pdev member from struct isp4_device, as it is unnecessary.
>>>>> - Fixed typo in 'isp_mmip' parameter name within isp4if_init().
>>>>> - Removed gap in struct isp4_subdev definition.
>>>>> - Performed extensive dead code removal and minor style improvements throughout the codebase.
>>>>>
>>>>>
>>>>> Changes v4 -> v5:
>>>>>
>>>>> - Transitioned VIDEOBUF2_V4L2 from 'depends' to 'select' within Kconfig.
>>>>> - Standardized object file naming conventions in the Makefile and sorted entries alphabetically.
>>>>> - Removed the unused macro definition to_isp4_device.
>>>>> - Eliminated unused members mem_domain and mem_align from struct isp4if_gpu_mem_info.
>>>>> - Deleted unused fields mc_addr and gpu_pkg from struct isp4if_cmd_element.
>>>>> - Removed obsolete pltf_data, i2c_nb, and notifier elements from struct isp4_device.
>>>>> - Updated platform_get_irq failure handling to return its actual result rather than -ENODEV.
>>>>> - Refined inclusion of header files for clarity and efficiency.
>>>>> - Appended comments following #endif statements in header files.
>>>>> - Improved implementation of isp4if_gpu_mem_free and isp4if_dealloc_fw_gpumem.
>>>>> - Removed isp4if_append_cmd_2_cmdq and revised isp4if_send_fw_cmd accordingly.
>>>>> - Enhanced isp4if_clear_cmdq and isp4if_clear_bufq by eliminating unnecessary list_del operations.
>>>>> - Adopted completion mechanism instead of wait queue and condition for command completion notifications.
>>>>> - Employed memset to ensure proper zeroing of padding bits in structures shared between ISP driver and firmware.
>>>>> - Streamlined IRQs, reducing total from four to two, retaining only essential ones.
>>>>> - Optimized IRQ handler logic using a while loop for greater efficiency.
>>>>> - Introduced dynamic IRQ enable/disable functionality based on camera status (open/close).
>>>>> - Applied distinct identifiers to differentiate multiple threads and IRQs.
>>>>> - Removed unnecessary initialization of local variables.
>>>>> - Refined camera start/stop workflow to mitigate potential synchronization concerns.
>>>>> - Replaced all remaining mutex with guard mutex.
>>>>> - Enhanced command and buffer queue performance by substituting mutexes with spinlocks.
>>>>> - Removed redundant isp4sd_init_meta_buf function and its references.
>>>>> - Limited firmware logging activities to the stream1 thread.
>>>>> - Relocated v4l2_device_unregister_subdev() and media_entity_cleanup() calls from isp4_capture_remove to isp4sd_deinit.
>>>>> - Resolved media device registration sequence issues.
>>>>> - Modified stream processing thread behavior to await IRQ without a timeout.
>>>>> - Addressed cleanup procedures in video device initialization and deinitialization routines.
>>>>> - Corrected typos and made other cosmetic improvements.
>>>>>
>>>>>
>>>>> Changes v3 -> v4:
>>>>>
>>>>> - Replace one mutex with guard mutex.
>>>>> - Remove unnecessary bus_info initialization of v4l2_capability.
>>>>> - Drop V4L2_CAP_IO_MC from capabilities of v4l2_capability.
>>>>> - Modify document with better SOC description.
>>>>> - Fix Test x86 failure in Media CI test https://linux-media.pages.freedesktop.org/-/users/patchwork/-/jobs/83470456/artifacts/report.htm
>>>>> - Modify some commit messages by describing changes in imperative mood.
>>>>> - Add media-ctl output in cover letter.
>>>>> - Create separated dedicated amdgpu patch to add declaration MODULE_FIRMWARE("amdgpu/isp_4_1_1.bin");
>>>>> - Fix typo errors and other cosmetic issues.
>>>>> - Add DRM_AMD_ISP dependency in Kconfig.
>>>>>
>>>>>
>>>>> Changes v2 -> v3:
>>>>>
>>>>> - All the dependent patches in other modules (drm/amd/amdgpu, platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
>>>>> - Removed usage of amdgpu structs in ISP driver. Added helper functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
>>>>> - Moved sensor and MIPI phy control entirely into ISP FW instead of the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
>>>>> - Removed phy configuration and sensor binding as x86 (sensor driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
>>>>> - New FW with built-in sensor driver is submitted on upstream linux-firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
>>>>> - Please note the new FW submitted is not directly compatible with OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
>>>>> - If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
>>>>> - Included critical fixes from Sultan Alsawaf branch (https://github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
>>>>> media: amd: isp4: Add missing refcount tracking to mmap memop
>>>>> media: amd: isp4: Don't put or unmap the dmabuf when detaching
>>>>> media: amd: isp4: Don't increment refcount when dmabuf export fails
>>>>> media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
>>>>> media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
>>>>> media: amd: isp4: Fix implicit dmabuf lifetime tracking
>>>>> media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
>>>>> media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
>>>>> media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
>>>>> media: amd: isp4: Remove unused userptr memops
>>>>> media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
>>>>> media: amd: isp4: Release queued buffers on error in start_streaming
>>>>> - Addressed all code related upstream comments
>>>>> - Fix typo errors and other cosmetic issues.
>>>>>
>>>>>
>>>>> Changes v1 -> v2:
>>>>>
>>>>> - Fix media CI test errors and valid warnings
>>>>> - Reduce patch number in the series from 9 to 8 by merging MAINTAINERS adding patch to the first patch
>>>>> - In patch 5
>>>>> - do modification to use remote endpoint instead of local endpoint
>>>>> - use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint
>>>>>
>>>>> ----------
>>>>>
>>>>> It passes v4l2 compliance test, the test reports for:
>>>>>
>>>>> (a) amd_isp_capture device /dev/video0
>>>>>
>>>>> Compliance test for amd_isp_capture device /dev/video0:
>>>>> -------------------------------------------------------
>>>>>
>>>>> atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
>>>>> v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
>>>>> v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17
>>>>>
>>>>> Compliance test for amd_isp_capture device /dev/video0:
>>>>>
>>>>> Driver Info:
>>>>> Driver name : amd_isp_capture
>>>>> Card type : amd_isp_capture
>>>>> Bus info : platform:amd_isp_capture
>>>>> Driver version : 6.14.0
>>>>> Capabilities : 0xa4200001
>>>>> Video Capture
>>>>> I/O MC
>>>>> Streaming
>>>>> Extended Pix Format
>>>>> Device Capabilities
>>>>> Device Caps : 0x24200001
>>>>> Video Capture
>>>>> I/O MC
>>>>> Streaming
>>>>> Extended Pix Format
>>>>> Media Driver Info:
>>>>> Driver name : amd_isp_capture
>>>>> Model : amd_isp41_mdev
>>>>> Serial :
>>>>> Bus info : platform:amd_isp_capture
>>>>> Media version : 6.14.0
>>>>> Hardware revision: 0x00000000 (0)
>>>>> Driver version : 6.14.0
>>>>> Interface Info:
>>>>> ID : 0x03000005
>>>>> Type : V4L Video
>>>>> Entity Info:
>>>>> ID : 0x00000003 (3)
>>>>> Name : Preview
>>>>> Function : V4L2 I/O
>>>>> Pad 0x01000004 : 0: Sink
>>>>> Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable
>>>>>
>>>>> Required ioctls:
>>>>> test MC information (see 'Media Driver Info' above): OK
>>>>> test VIDIOC_QUERYCAP: OK
>>>>> test invalid ioctls: OK
>>>>>
>>>>> Allow for multiple opens:
>>>>> test second /dev/video0 open: OK
>>>>> test VIDIOC_QUERYCAP: OK
>>>>> test VIDIOC_G/S_PRIORITY: OK
>>>>> test for unlimited opens: OK
>>>>>
>>>>> Debug ioctls:
>>>>> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
>>>>> test VIDIOC_LOG_STATUS: OK (Not Supported)
>>>>>
>>>>> Input ioctls:
>>>>> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
>>>>> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>>>>> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
>>>>> test VIDIOC_ENUMAUDIO: OK (Not Supported)
>>>>> test VIDIOC_G/S/ENUMINPUT: OK
>>>>> test VIDIOC_G/S_AUDIO: OK (Not Supported)
>>>>> Inputs: 1 Audio Inputs: 0 Tuners: 0
>>>>>
>>>>> Output ioctls:
>>>>> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
>>>>> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>>>>> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
>>>>> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
>>>>> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
>>>>> Outputs: 0 Audio Outputs: 0 Modulators: 0
>>>>>
>>>>> Input/Output configuration ioctls:
>>>>> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
>>>>> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
>>>>> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
>>>>> test VIDIOC_G/S_EDID: OK (Not Supported)
>>>>>
>>>>> Control ioctls (Input 0):
>>>>> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
>>>>> test VIDIOC_QUERYCTRL: OK (Not Supported)
>>>>> test VIDIOC_G/S_CTRL: OK (Not Supported)
>>>>> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
>>>>> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
>>>>> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
>>>>> Standard Controls: 0 Private Controls: 0
>>>>>
>>>>> Format ioctls (Input 0):
>>>>> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
>>>>> test VIDIOC_G/S_PARM: OK
>>>>> test VIDIOC_G_FBUF: OK (Not Supported)
>>>>> test VIDIOC_G_FMT: OK
>>>>> test VIDIOC_TRY_FMT: OK
>>>>> test VIDIOC_S_FMT: OK
>>>>> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
>>>>> test Cropping: OK (Not Supported)
>>>>> test Composing: OK (Not Supported)
>>>>> test Scaling: OK (Not Supported)
>>>>>
>>>>> Codec ioctls (Input 0):
>>>>> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
>>>>> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
>>>>> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>>>>>
>>>>> Buffer ioctls (Input 0):
>>>>> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
>>>>> test CREATE_BUFS maximum buffers: OK
>>>>> test VIDIOC_REMOVE_BUFS: OK
>>>>> test VIDIOC_EXPBUF: OK
>>>>> test Requests: OK (Not Supported)
>>>>> test blocking wait: OK
>>>>>
>>>>> Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0
>>>>>
>>>>> The media-ctl output of media device /dev/media0:
>>>>> -------------------------------------------------------
>>>>>
>>>>> atg@atg-HP-PV:~$ media-ctl -p -d /dev/media0
>>>>> Media controller API version 6.17.0
>>>>>
>>>>> Media device information
>>>>> ------------------------
>>>>> driver amd_isp_capture
>>>>> model amd_isp41_mdev
>>>>> serial
>>>>> bus info platform:amd_isp_capture
>>>>> hw revision 0x0
>>>>> driver version 6.17.0
>>>>>
>>>>> Device topology
>>>>> - entity 1: amd isp4 (1 pad, 1 link, 0 routes)
>>>>> type V4L2 subdev subtype Unknown flags 0
>>>>> pad0: Source
>>>>> -> "Preview":0 [ENABLED,IMMUTABLE]
>>>>>
>>>>> - entity 3: Preview (1 pad, 1 link)
>>>>> type Node subtype V4L flags 0
>>>>> device node name /dev/video0
>>>>> pad0: Sink
>>>>> <- "amd isp4":0 [ENABLED,IMMUTABLE]
>>>>>
>>>>> Please review and provide feedback.
>>>>>
>>>>> Many thanks,
>>>>>
>>>>> Bin Du (7):
>>>>> media: platform: amd: Introduce amd isp4 capture driver
>>>>> media: platform: amd: low level support for isp4 firmware
>>>>> media: platform: amd: Add isp4 fw and hw interface
>>>>> media: platform: amd: isp4 subdev and firmware loading handling added
>>>>> media: platform: amd: isp4 video node and buffers handling added
>>>>> media: platform: amd: isp4 debug fs logging and more descriptive
>>>>> errors
>>>>> Documentation: add documentation of AMD isp 4 driver
>>>>>
>>>>> Documentation/admin-guide/media/amdisp4-1.rst | 63 +
>>>>> Documentation/admin-guide/media/amdisp4.dot | 6 +
>>>>> .../admin-guide/media/v4l-drivers.rst | 1 +
>>>>> MAINTAINERS | 25 +
>>>>> drivers/media/platform/Kconfig | 1 +
>>>>> drivers/media/platform/Makefile | 1 +
>>>>> drivers/media/platform/amd/Kconfig | 3 +
>>>>> drivers/media/platform/amd/Makefile | 3 +
>>>>> drivers/media/platform/amd/isp4/Kconfig | 14 +
>>>>> drivers/media/platform/amd/isp4/Makefile | 10 +
>>>>> drivers/media/platform/amd/isp4/isp4.c | 235 ++++
>>>>> drivers/media/platform/amd/isp4/isp4.h | 20 +
>>>>> drivers/media/platform/amd/isp4/isp4_debug.c | 271 ++++
>>>>> drivers/media/platform/amd/isp4/isp4_debug.h | 41 +
>>>>> .../platform/amd/isp4/isp4_fw_cmd_resp.h | 314 +++++
>>>>> drivers/media/platform/amd/isp4/isp4_hw_reg.h | 124 ++
>>>>> .../media/platform/amd/isp4/isp4_interface.c | 789 +++++++++++
>>>>> .../media/platform/amd/isp4/isp4_interface.h | 141 ++
>>>>> drivers/media/platform/amd/isp4/isp4_subdev.c | 1057 +++++++++++++++
>>>>> drivers/media/platform/amd/isp4/isp4_subdev.h | 131 ++
>>>>> drivers/media/platform/amd/isp4/isp4_video.c | 1165 +++++++++++++++++
>>>>> drivers/media/platform/amd/isp4/isp4_video.h | 65 +
>>>>> 22 files changed, 4480 insertions(+)
>>>>> create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst
>>>>> create mode 100644 Documentation/admin-guide/media/amdisp4.dot
>>>>> create mode 100644 drivers/media/platform/amd/Kconfig
>>>>> create mode 100644 drivers/media/platform/amd/Makefile
>>>>> create mode 100644 drivers/media/platform/amd/isp4/Kconfig
>>>>> create mode 100644 drivers/media/platform/amd/isp4/Makefile
>>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4.c
>>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4.h
>>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c
>>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h
>>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
>>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h
>>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c
>>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h
>>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
>>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
>>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c
>>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h
>>>>>
>>>>> --
>>>>> 2.34.1
>>>>>
>>>>>
>>>> v7 worked on my HP ZBook Ultra G1a 14 inch Mobile Workstation PC and
>>>> the camera worked fine.
>>>> It was tested with the latest upstream firmware [1] and 6.19-rc3 kernel.
>>>>
>>>> [1] https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/amdgpu/isp_4_1_1.bin
>>>>
>>>> Thank you for your work :)
>>>>
>>>> --
>>>> BR,
>>>> Kate
>>> Leave a "Tested-by" tag for my previous testing assertion.
>>>
>>> Tested-by: Kate Hsuan <hpa@redhat.com>
>>>
>>>
>>>
>> --
>> Regards,
>> Bin
>>
> The ISP driver works perfectly with a clear and sharp video. I tested
> it again, and I found a suspend/resume issue.
> The ISP can't suspend when the system is set to suspend. The privacy
> LED is on when the system is suspended. Therefore, the user will see a
> luminous privacy LED when the system is set to suspend.
> Today, I made a work to move all the power control to use the runtime
> PM, including suspend/resume. This work may be humble and may break
> the finite state machine but it works. The major changes of it
> include:
> 1. Support suspend/resume.
> 2. The power is managed by the runtime PM so the s_power and the related
> callback function were dropped.
> 3. The enable_isp GPIO pin is controlled by the runtime PM.
> 4. pm_runtime_get_noresume() is used to get the runtime PM at probe()
> since the device doesn't have to be set to power on when initialising.
>
> This work stops the video stream on suspend and starts the stream on
> resume so the privacy LED is turned on and turned off with the changes
> of suspend and resume.
>
> Could you please consider this patch and idea?
>
> Thank you :)
Thanks for reporting this issue and also providing the patch.
We have addressed this issue recently. I suspect the below fix in AMDGPU
available in v6.19-rc5 is missing in your build.
https://github.com/torvalds/linux/commit/7ed51e3a1381422278933d0d3ebda0268b6825de
I have tested locally and this issue is not observed with this change
included. Can you please check and feedback if this solves the problem?
This change takes care of handling isp suspend-resume as part of amdgpu
device suspend-resume instead of genpd, and uses the pm rumtime as you
have suggested.
Thanks,
Pratap
>
> --
> BR,
> Kate
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 0/7] Add AMD ISP4 driver
2026-01-13 17:14 ` Nirujogi, Pratap
@ 2026-01-14 8:59 ` Kate Hsuan
2026-01-15 6:47 ` Du, Bin
0 siblings, 1 reply; 46+ messages in thread
From: Kate Hsuan @ 2026-01-14 8:59 UTC (permalink / raw)
To: Nirujogi, Pratap
Cc: Du, Bin, mchehab, hverkuil, laurent.pinchart+renesas,
bryan.odonoghue, sakari.ailus, prabhakar.mahadev-lad.rj,
linux-media, linux-kernel, sultan, pratap.nirujogi, benjamin.chan,
king.li, gjorgji.rosikopulos, Phil.Jawich, Dominic.Antony,
mario.limonciello, richard.gong, anson.tsao
Hi Pratap,
On Wed, Jan 14, 2026 at 1:14 AM Nirujogi, Pratap <pnirujog@amd.com> wrote:
>
> Hi Kate,
>
> On 1/13/2026 9:11 AM, Kate Hsuan wrote:
> > Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
> >
> >
> > Hi Bin,
> >
> > On Tue, Jan 6, 2026 at 4:35 PM Du, Bin <bin.du@amd.com> wrote:
> >> Thank you, Kate, for the verification, it gives us more confidence.
> >>
> >> On 1/6/2026 1:49 PM, Kate Hsuan wrote:
> >>> Hi
> >>>
> >>> On Wed, Dec 31, 2025 at 5:03 PM Kate Hsuan <hpa@redhat.com> wrote:
> >>>> On Tue, Dec 16, 2025 at 5:14 PM Bin Du <Bin.Du@amd.com> wrote:
> >>>>> Hello,
> >>>>>
> >>>>> AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC (Ryzen AI Max 300 Series)
> >>>>> (https://ubuntu.com/certified/202411-36043)
> >>>>> This patch series introduces the initial driver support for the AMD ISP4.
> >>>>>
> >>>>> Patch summary:
> >>>>> - Powers up/off and initializes ISP HW
> >>>>> - Configures and kicks off ISP FW
> >>>>> - Interacts with APP using standard V4l2 interface by video node
> >>>>> - Controls ISP HW and interacts with ISP FW to do image processing
> >>>>> - Supports enum/set output image format and resolution
> >>>>> - Supports queueing buffer from app and dequeuing ISP filled buffer to App
> >>>>> - It is verified on qv4l2, cheese and qcam
> >>>>> - It is verified together with following patches
> >>>>> platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/20250514215623.522746-1-pratap.nirujogi@amd.com/)
> >>>>> pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
> >>>>> drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)
> >>>>> drm/amd/amdgpu: Declare isp firmware binary file (https://gitlab.freedesktop.org/agd5f/linux/-/commit/35345917bc9f7c86152b270d9d93c220230b667f)
> >>>>>
> >>>>> AMD ISP4 Key features:
> >>>>> - Processes bayer raw data from the connected sensor and output them to different YUV formats
> >>>>> - Downscale input image to different output image resolution
> >>>>> - Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc.
> >>>>>
> >>>>> ----------
> >>>>>
> >>>>> Changes v6 -> v7:
> >>>>>
> >>>>> - Added missed blank line after some if statements.
> >>>>> - Changed the pm_runtime_disable() order in isp4_capture_remove() to align with initialization.
> >>>>> - Reset buf_sent_cnt and start_stream_cmd_sent on stream stop.
> >>>>> - Removed duplicate buf_sent_cnt and start_stream_cmd_sent reset in isp4sd_pwron_and_init().
> >>>>> - Combined isp4sd_reset_stream_info() and isp4sd_reset_camera_info() into isp4sd_uninit_stream() to eliminate redundant stream info reset.
> >>>>> - Removed always-false status check in isp4sd_uninit_stream().
> >>>>> - Minor style improvements.
> >>>>>
> >>>>>
> >>>>> Changes v5 -> v6:
> >>>>>
> >>>>> - Lowered FW mempool buffer size from 200M to 100M (actual usage).
> >>>>> - Added an irq_enabled member to the ISP subdev for proper IRQ disable handling in both normal and error cases.
> >>>>> - Removed unnecessary .owner assignment from isp4_capture_drv definition
> >>>>> - Updated IRQ handling to enable and disable interrupts via the ISP register for improved performance.
> >>>>> - Revised ring buffer management in isp4if_f2h_resp(), addressing safety checks to ensure the read pointer is validated before memcpy operations, reducing the risk of out-of-bounds access. The ring buffer logic was also streamlined significantly.
> >>>>> - Modified ring buffer handling in isp4if_is_cmdq_rb_full(), correcting an off-by-one error in safety checks that previously allowed rd_ptr to equal wr_ptr when the buffer was full.
> >>>>> - Refactored ring buffer handling in isp4if_insert_isp_fw_cmd(), simplifying overall logic.
> >>>>> - Resolved a regression from v4 to v5 where isp4if_dequeue_buffer() did not protect list_del with bufq_lock.
> >>>>> - Addressed a subtle use-after-free issue that could occur if a timeout on a synchronous command coincided with completion.
> >>>>> - Added missing pm_runtime_disable() calls to isp4_capture_remove() and to the error path in isp4_capture_probe().
> >>>>> - Removed stray semicolons following closing curly braces.
> >>>>> - Improved and clarified macro definitions in isp4_interface.h.
> >>>>> - Eliminated unnecessary (u8 *) casts.
> >>>>> - Added missing memset for firmware command structures in isp4sd_stop_stream().
> >>>>> - Excluded streams 2 and 3 from ISP4IF_FW_RESP_RB_IRQ_EN_MASK, preventing their activation in the interrupt enable register.
> >>>>> - Enhanced error handling to clean up kthreads in the event of startup failure.
> >>>>> - Corrected a race condition during kthread creation where waitqueue head initialization could be delayed, as it was performed by the kthread itself.
> >>>>> - Removed status checks in isp4sd_pwroff_and_deinit() that were always false.
> >>>>> - Ensured isp4sd_init_stream() is only invoked once per stream start and reordered corresponding status checks in isp4sd_start_stream().
> >>>>> - Improved error handling in isp4sd_start_stream() to propagate errors from failed functions.
> >>>>> - Relocated debugging messages in isp4sd_stop_stream() to execute under lock protection due to access to output_info->start_status.
> >>>>> - Eliminated redundant GET_REG_BASE() macros.
> >>>>> - Removed isp4sd_is_stream_running() function.
> >>>>> - Corrected error message in isp4sd_init_stream() caused by copy/paste.
> >>>>> - Refined struct isp4_interface to remove firmware ring buffer configurations.
> >>>>> - Removed obsolete isp4sd_is_stream_running function.
> >>>>> - Removed pdev member from struct isp4_device, as it is unnecessary.
> >>>>> - Fixed typo in 'isp_mmip' parameter name within isp4if_init().
> >>>>> - Removed gap in struct isp4_subdev definition.
> >>>>> - Performed extensive dead code removal and minor style improvements throughout the codebase.
> >>>>>
> >>>>>
> >>>>> Changes v4 -> v5:
> >>>>>
> >>>>> - Transitioned VIDEOBUF2_V4L2 from 'depends' to 'select' within Kconfig.
> >>>>> - Standardized object file naming conventions in the Makefile and sorted entries alphabetically.
> >>>>> - Removed the unused macro definition to_isp4_device.
> >>>>> - Eliminated unused members mem_domain and mem_align from struct isp4if_gpu_mem_info.
> >>>>> - Deleted unused fields mc_addr and gpu_pkg from struct isp4if_cmd_element.
> >>>>> - Removed obsolete pltf_data, i2c_nb, and notifier elements from struct isp4_device.
> >>>>> - Updated platform_get_irq failure handling to return its actual result rather than -ENODEV.
> >>>>> - Refined inclusion of header files for clarity and efficiency.
> >>>>> - Appended comments following #endif statements in header files.
> >>>>> - Improved implementation of isp4if_gpu_mem_free and isp4if_dealloc_fw_gpumem.
> >>>>> - Removed isp4if_append_cmd_2_cmdq and revised isp4if_send_fw_cmd accordingly.
> >>>>> - Enhanced isp4if_clear_cmdq and isp4if_clear_bufq by eliminating unnecessary list_del operations.
> >>>>> - Adopted completion mechanism instead of wait queue and condition for command completion notifications.
> >>>>> - Employed memset to ensure proper zeroing of padding bits in structures shared between ISP driver and firmware.
> >>>>> - Streamlined IRQs, reducing total from four to two, retaining only essential ones.
> >>>>> - Optimized IRQ handler logic using a while loop for greater efficiency.
> >>>>> - Introduced dynamic IRQ enable/disable functionality based on camera status (open/close).
> >>>>> - Applied distinct identifiers to differentiate multiple threads and IRQs.
> >>>>> - Removed unnecessary initialization of local variables.
> >>>>> - Refined camera start/stop workflow to mitigate potential synchronization concerns.
> >>>>> - Replaced all remaining mutex with guard mutex.
> >>>>> - Enhanced command and buffer queue performance by substituting mutexes with spinlocks.
> >>>>> - Removed redundant isp4sd_init_meta_buf function and its references.
> >>>>> - Limited firmware logging activities to the stream1 thread.
> >>>>> - Relocated v4l2_device_unregister_subdev() and media_entity_cleanup() calls from isp4_capture_remove to isp4sd_deinit.
> >>>>> - Resolved media device registration sequence issues.
> >>>>> - Modified stream processing thread behavior to await IRQ without a timeout.
> >>>>> - Addressed cleanup procedures in video device initialization and deinitialization routines.
> >>>>> - Corrected typos and made other cosmetic improvements.
> >>>>>
> >>>>>
> >>>>> Changes v3 -> v4:
> >>>>>
> >>>>> - Replace one mutex with guard mutex.
> >>>>> - Remove unnecessary bus_info initialization of v4l2_capability.
> >>>>> - Drop V4L2_CAP_IO_MC from capabilities of v4l2_capability.
> >>>>> - Modify document with better SOC description.
> >>>>> - Fix Test x86 failure in Media CI test https://linux-media.pages.freedesktop.org/-/users/patchwork/-/jobs/83470456/artifacts/report.htm
> >>>>> - Modify some commit messages by describing changes in imperative mood.
> >>>>> - Add media-ctl output in cover letter.
> >>>>> - Create separated dedicated amdgpu patch to add declaration MODULE_FIRMWARE("amdgpu/isp_4_1_1.bin");
> >>>>> - Fix typo errors and other cosmetic issues.
> >>>>> - Add DRM_AMD_ISP dependency in Kconfig.
> >>>>>
> >>>>>
> >>>>> Changes v2 -> v3:
> >>>>>
> >>>>> - All the dependent patches in other modules (drm/amd/amdgpu, platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
> >>>>> - Removed usage of amdgpu structs in ISP driver. Added helper functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
> >>>>> - Moved sensor and MIPI phy control entirely into ISP FW instead of the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
> >>>>> - Removed phy configuration and sensor binding as x86 (sensor driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
> >>>>> - New FW with built-in sensor driver is submitted on upstream linux-firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
> >>>>> - Please note the new FW submitted is not directly compatible with OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
> >>>>> - If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
> >>>>> - Included critical fixes from Sultan Alsawaf branch (https://github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
> >>>>> media: amd: isp4: Add missing refcount tracking to mmap memop
> >>>>> media: amd: isp4: Don't put or unmap the dmabuf when detaching
> >>>>> media: amd: isp4: Don't increment refcount when dmabuf export fails
> >>>>> media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
> >>>>> media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
> >>>>> media: amd: isp4: Fix implicit dmabuf lifetime tracking
> >>>>> media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
> >>>>> media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
> >>>>> media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
> >>>>> media: amd: isp4: Remove unused userptr memops
> >>>>> media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
> >>>>> media: amd: isp4: Release queued buffers on error in start_streaming
> >>>>> - Addressed all code related upstream comments
> >>>>> - Fix typo errors and other cosmetic issues.
> >>>>>
> >>>>>
> >>>>> Changes v1 -> v2:
> >>>>>
> >>>>> - Fix media CI test errors and valid warnings
> >>>>> - Reduce patch number in the series from 9 to 8 by merging MAINTAINERS adding patch to the first patch
> >>>>> - In patch 5
> >>>>> - do modification to use remote endpoint instead of local endpoint
> >>>>> - use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint
> >>>>>
> >>>>> ----------
> >>>>>
> >>>>> It passes v4l2 compliance test, the test reports for:
> >>>>>
> >>>>> (a) amd_isp_capture device /dev/video0
> >>>>>
> >>>>> Compliance test for amd_isp_capture device /dev/video0:
> >>>>> -------------------------------------------------------
> >>>>>
> >>>>> atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
> >>>>> v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
> >>>>> v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17
> >>>>>
> >>>>> Compliance test for amd_isp_capture device /dev/video0:
> >>>>>
> >>>>> Driver Info:
> >>>>> Driver name : amd_isp_capture
> >>>>> Card type : amd_isp_capture
> >>>>> Bus info : platform:amd_isp_capture
> >>>>> Driver version : 6.14.0
> >>>>> Capabilities : 0xa4200001
> >>>>> Video Capture
> >>>>> I/O MC
> >>>>> Streaming
> >>>>> Extended Pix Format
> >>>>> Device Capabilities
> >>>>> Device Caps : 0x24200001
> >>>>> Video Capture
> >>>>> I/O MC
> >>>>> Streaming
> >>>>> Extended Pix Format
> >>>>> Media Driver Info:
> >>>>> Driver name : amd_isp_capture
> >>>>> Model : amd_isp41_mdev
> >>>>> Serial :
> >>>>> Bus info : platform:amd_isp_capture
> >>>>> Media version : 6.14.0
> >>>>> Hardware revision: 0x00000000 (0)
> >>>>> Driver version : 6.14.0
> >>>>> Interface Info:
> >>>>> ID : 0x03000005
> >>>>> Type : V4L Video
> >>>>> Entity Info:
> >>>>> ID : 0x00000003 (3)
> >>>>> Name : Preview
> >>>>> Function : V4L2 I/O
> >>>>> Pad 0x01000004 : 0: Sink
> >>>>> Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable
> >>>>>
> >>>>> Required ioctls:
> >>>>> test MC information (see 'Media Driver Info' above): OK
> >>>>> test VIDIOC_QUERYCAP: OK
> >>>>> test invalid ioctls: OK
> >>>>>
> >>>>> Allow for multiple opens:
> >>>>> test second /dev/video0 open: OK
> >>>>> test VIDIOC_QUERYCAP: OK
> >>>>> test VIDIOC_G/S_PRIORITY: OK
> >>>>> test for unlimited opens: OK
> >>>>>
> >>>>> Debug ioctls:
> >>>>> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> >>>>> test VIDIOC_LOG_STATUS: OK (Not Supported)
> >>>>>
> >>>>> Input ioctls:
> >>>>> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> >>>>> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> >>>>> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> >>>>> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> >>>>> test VIDIOC_G/S/ENUMINPUT: OK
> >>>>> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> >>>>> Inputs: 1 Audio Inputs: 0 Tuners: 0
> >>>>>
> >>>>> Output ioctls:
> >>>>> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> >>>>> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> >>>>> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> >>>>> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> >>>>> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> >>>>> Outputs: 0 Audio Outputs: 0 Modulators: 0
> >>>>>
> >>>>> Input/Output configuration ioctls:
> >>>>> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> >>>>> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> >>>>> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> >>>>> test VIDIOC_G/S_EDID: OK (Not Supported)
> >>>>>
> >>>>> Control ioctls (Input 0):
> >>>>> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> >>>>> test VIDIOC_QUERYCTRL: OK (Not Supported)
> >>>>> test VIDIOC_G/S_CTRL: OK (Not Supported)
> >>>>> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> >>>>> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> >>>>> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> >>>>> Standard Controls: 0 Private Controls: 0
> >>>>>
> >>>>> Format ioctls (Input 0):
> >>>>> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> >>>>> test VIDIOC_G/S_PARM: OK
> >>>>> test VIDIOC_G_FBUF: OK (Not Supported)
> >>>>> test VIDIOC_G_FMT: OK
> >>>>> test VIDIOC_TRY_FMT: OK
> >>>>> test VIDIOC_S_FMT: OK
> >>>>> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> >>>>> test Cropping: OK (Not Supported)
> >>>>> test Composing: OK (Not Supported)
> >>>>> test Scaling: OK (Not Supported)
> >>>>>
> >>>>> Codec ioctls (Input 0):
> >>>>> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> >>>>> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> >>>>> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
> >>>>>
> >>>>> Buffer ioctls (Input 0):
> >>>>> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> >>>>> test CREATE_BUFS maximum buffers: OK
> >>>>> test VIDIOC_REMOVE_BUFS: OK
> >>>>> test VIDIOC_EXPBUF: OK
> >>>>> test Requests: OK (Not Supported)
> >>>>> test blocking wait: OK
> >>>>>
> >>>>> Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0
> >>>>>
> >>>>> The media-ctl output of media device /dev/media0:
> >>>>> -------------------------------------------------------
> >>>>>
> >>>>> atg@atg-HP-PV:~$ media-ctl -p -d /dev/media0
> >>>>> Media controller API version 6.17.0
> >>>>>
> >>>>> Media device information
> >>>>> ------------------------
> >>>>> driver amd_isp_capture
> >>>>> model amd_isp41_mdev
> >>>>> serial
> >>>>> bus info platform:amd_isp_capture
> >>>>> hw revision 0x0
> >>>>> driver version 6.17.0
> >>>>>
> >>>>> Device topology
> >>>>> - entity 1: amd isp4 (1 pad, 1 link, 0 routes)
> >>>>> type V4L2 subdev subtype Unknown flags 0
> >>>>> pad0: Source
> >>>>> -> "Preview":0 [ENABLED,IMMUTABLE]
> >>>>>
> >>>>> - entity 3: Preview (1 pad, 1 link)
> >>>>> type Node subtype V4L flags 0
> >>>>> device node name /dev/video0
> >>>>> pad0: Sink
> >>>>> <- "amd isp4":0 [ENABLED,IMMUTABLE]
> >>>>>
> >>>>> Please review and provide feedback.
> >>>>>
> >>>>> Many thanks,
> >>>>>
> >>>>> Bin Du (7):
> >>>>> media: platform: amd: Introduce amd isp4 capture driver
> >>>>> media: platform: amd: low level support for isp4 firmware
> >>>>> media: platform: amd: Add isp4 fw and hw interface
> >>>>> media: platform: amd: isp4 subdev and firmware loading handling added
> >>>>> media: platform: amd: isp4 video node and buffers handling added
> >>>>> media: platform: amd: isp4 debug fs logging and more descriptive
> >>>>> errors
> >>>>> Documentation: add documentation of AMD isp 4 driver
> >>>>>
> >>>>> Documentation/admin-guide/media/amdisp4-1.rst | 63 +
> >>>>> Documentation/admin-guide/media/amdisp4.dot | 6 +
> >>>>> .../admin-guide/media/v4l-drivers.rst | 1 +
> >>>>> MAINTAINERS | 25 +
> >>>>> drivers/media/platform/Kconfig | 1 +
> >>>>> drivers/media/platform/Makefile | 1 +
> >>>>> drivers/media/platform/amd/Kconfig | 3 +
> >>>>> drivers/media/platform/amd/Makefile | 3 +
> >>>>> drivers/media/platform/amd/isp4/Kconfig | 14 +
> >>>>> drivers/media/platform/amd/isp4/Makefile | 10 +
> >>>>> drivers/media/platform/amd/isp4/isp4.c | 235 ++++
> >>>>> drivers/media/platform/amd/isp4/isp4.h | 20 +
> >>>>> drivers/media/platform/amd/isp4/isp4_debug.c | 271 ++++
> >>>>> drivers/media/platform/amd/isp4/isp4_debug.h | 41 +
> >>>>> .../platform/amd/isp4/isp4_fw_cmd_resp.h | 314 +++++
> >>>>> drivers/media/platform/amd/isp4/isp4_hw_reg.h | 124 ++
> >>>>> .../media/platform/amd/isp4/isp4_interface.c | 789 +++++++++++
> >>>>> .../media/platform/amd/isp4/isp4_interface.h | 141 ++
> >>>>> drivers/media/platform/amd/isp4/isp4_subdev.c | 1057 +++++++++++++++
> >>>>> drivers/media/platform/amd/isp4/isp4_subdev.h | 131 ++
> >>>>> drivers/media/platform/amd/isp4/isp4_video.c | 1165 +++++++++++++++++
> >>>>> drivers/media/platform/amd/isp4/isp4_video.h | 65 +
> >>>>> 22 files changed, 4480 insertions(+)
> >>>>> create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst
> >>>>> create mode 100644 Documentation/admin-guide/media/amdisp4.dot
> >>>>> create mode 100644 drivers/media/platform/amd/Kconfig
> >>>>> create mode 100644 drivers/media/platform/amd/Makefile
> >>>>> create mode 100644 drivers/media/platform/amd/isp4/Kconfig
> >>>>> create mode 100644 drivers/media/platform/amd/isp4/Makefile
> >>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4.c
> >>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4.h
> >>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c
> >>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h
> >>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
> >>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h
> >>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c
> >>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h
> >>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
> >>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
> >>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c
> >>>>> create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h
> >>>>>
> >>>>> --
> >>>>> 2.34.1
> >>>>>
> >>>>>
> >>>> v7 worked on my HP ZBook Ultra G1a 14 inch Mobile Workstation PC and
> >>>> the camera worked fine.
> >>>> It was tested with the latest upstream firmware [1] and 6.19-rc3 kernel.
> >>>>
> >>>> [1] https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/amdgpu/isp_4_1_1.bin
> >>>>
> >>>> Thank you for your work :)
> >>>>
> >>>> --
> >>>> BR,
> >>>> Kate
> >>> Leave a "Tested-by" tag for my previous testing assertion.
> >>>
> >>> Tested-by: Kate Hsuan <hpa@redhat.com>
> >>>
> >>>
> >>>
> >> --
> >> Regards,
> >> Bin
> >>
> > The ISP driver works perfectly with a clear and sharp video. I tested
> > it again, and I found a suspend/resume issue.
> > The ISP can't suspend when the system is set to suspend. The privacy
> > LED is on when the system is suspended. Therefore, the user will see a
> > luminous privacy LED when the system is set to suspend.
> > Today, I made a work to move all the power control to use the runtime
> > PM, including suspend/resume. This work may be humble and may break
> > the finite state machine but it works. The major changes of it
> > include:
> > 1. Support suspend/resume.
> > 2. The power is managed by the runtime PM so the s_power and the related
> > callback function were dropped.
> > 3. The enable_isp GPIO pin is controlled by the runtime PM.
> > 4. pm_runtime_get_noresume() is used to get the runtime PM at probe()
> > since the device doesn't have to be set to power on when initialising.
> >
> > This work stops the video stream on suspend and starts the stream on
> > resume so the privacy LED is turned on and turned off with the changes
> > of suspend and resume.
> >
> > Could you please consider this patch and idea?
> >
> > Thank you :)
>
> Thanks for reporting this issue and also providing the patch.
>
> We have addressed this issue recently. I suspect the below fix in AMDGPU
> available in v6.19-rc5 is missing in your build.
>
> https://github.com/torvalds/linux/commit/7ed51e3a1381422278933d0d3ebda0268b6825de
>
> I have tested locally and this issue is not observed with this change
> included. Can you please check and feedback if this solves the problem?
>
> This change takes care of handling isp suspend-resume as part of amdgpu
> device suspend-resume instead of genpd, and uses the pm rumtime as you
> have suggested.
>
> Thanks,
>
> Pratap
>
>
> >
> > --
> > BR,
> > Kate
>
I tested again with the patch you mentioned. It works but I found some issues.
I also tested it with my work and the test results were shown as follows.
The test step is
1. start the camera app -> 2. set to suspend -> 3. resume the laptop
The results:
1. rebase to 6.19-RC5 with the fix patch (7ed51e3a1)
It works but I found the logs when stopping the video stream
...
[ 527.733851] amd_isp_capture amd_isp_capture: fail to disable stream
...
[ 528.237828] amd_isp_capture amd_isp_capture: fail to stop steam
...
The isp4 tries to stop the stream but it fails to write the data to
the firmware.
I think the ISP stops working before the ISP stops the video stream.
So, it cannot write the data to the firmware.
2. fix patch (7ed51e3a1) + ISP4 runtime PM (my work)
This also works.
The ISP4 complains the error about "static void
isp4if_dealloc_fw_gpumem(struct isp4_interface *ispif)" when suspend.
(This only happens when suspending the system)
[ 223.085419] WARNING: drivers/gpu/drm/amd/amdgpu/amdgpu_object.c:517
at amdgpu_bo_free_kernel+0xe9/0x100 [amdgpu], CPU#30:
kworker/u130:4/794
[ 223.085552] Modules linked in: uinput(E) rfcomm(E) snd_seq_dummy(E)
snd_hrtimer(E) hid_sensor_gyro_3d(E) hid_sensor_trigger(E)
hid_sensor_iio_common(E) industrialio_triggered_buffer(E) kfifo_buf(E)
industrialio(E) hid_sensor_hub(E) sunrpc(E) nf_conntrack_netbios_ns(E)
nf_conntrack_broadcast(E) nft_fib_inet(E) nft_fib_ipv4(E)
<snip>
[ 223.085591] CPU: 30 UID: 0 PID: 794 Comm: kworker/u130:4 Tainted: G
W E 6.19.0-rc5+ #92 PREEMPT(lazy)
[ 223.085592] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE
[ 223.085592] Hardware name: HP HP ZBook Ultra G1a 14 inch Mobile
Workstation PC/8D01, BIOS X89 Ver. 01.01.00 01/16/2025
[ 223.085593] Workqueue: async async_run_entry_fn
[ 223.085594] RIP: 0010:amdgpu_bo_free_kernel+0xe9/0x100 [amdgpu]
[ 223.085725] Code: f4 ff ff 4d 85 e4 74 08 49 c7 04 24 00 00 00 00
48 85 ed 74 08 48 c7 45 00 00 00 00 00 5b 5d 41 5c 41 5d 41 5e c3 cc
cc cc cc <0f> 0b e9 42 ff ff ff 3d 00 fe ff ff 0f 85 a1 34 95 00 eb bd
0f 1f
[ 223.085725] RSP: 0018:ffffcd80413e7ae0 EFLAGS: 00010202
[ 223.085726] RAX: 0000000000000000 RBX: 0000000000000005 RCX: 0000000080800077
[ 223.085726] RDX: ffff8b3719f6eef0 RSI: ffff8b3719f6eee8 RDI: ffff8b3719f6eef8
[ 223.085727] RBP: ffff8b3719f6eee0 R08: 0000000000000000 R09: ffffffffc0558135
[ 223.085727] R10: ffff8b3719f6e7a0 R11: ffffef494467db80 R12: ffff8b370ef86220
[ 223.085727] R13: ffff8b3718003c00 R14: ffff8b372a60ee00 R15: ffff8b370ef86220
[ 223.085728] FS: 0000000000000000(0000) GS:ffff8b5649311000(0000)
knlGS:0000000000000000
[ 223.085728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 223.085729] CR2: 0000563d26ad9000 CR3: 00000001ace2c000 CR4: 0000000000f50ef0
[ 223.085730] PKRU: 55555554
[ 223.085730] Call Trace:
[ 223.085730] <TASK>
[ 223.085731] isp4if_dealloc_fw_gpumem+0xcd/0x100 [amd_capture]
[ 223.085733] isp4if_stop+0x59/0x70 [amd_capture]
[ 223.085735] isp4sd_pwroff_and_deinit.isra.0+0x99/0x160 [amd_capture]
[ 223.085737] isp4sd_stop_stream+0xbe/0x250 [amd_capture]
[ 223.085740] v4l2_subdev_disable_streams+0x1ad/0x390 [videodev]
[ 223.085747] ? dc_dmub_srv_wait_for_idle+0x50/0x150 [amdgpu]
[ 223.085914] isp4_suspend+0x2d/0x80 [amd_capture]
[ 223.085916] genpd_runtime_suspend+0xe7/0x300
[ 223.085917] ? __pfx_isp_suspend_device+0x10/0x10 [amdgpu]
[ 223.086069] pm_runtime_force_suspend+0x71/0x110
[ 223.086070] ? __pfx_genpd_runtime_suspend+0x10/0x10
[ 223.086071] device_for_each_child+0x71/0xb0
[ 223.086073] isp_v4_1_1_hw_suspend+0x22/0x40 [amdgpu]
[ 223.086223] ? amdgpu_dpm_gfx_state_change+0x49/0x60 [amdgpu]
[ 223.086399] amdgpu_ip_block_suspend+0x27/0x50 [amdgpu]
[ 223.086522] amdgpu_device_ip_suspend_phase2+0x13c/0x3d0 [amdgpu]
[ 223.086647] amdgpu_device_suspend+0x161/0x240 [amdgpu]
Both solutions work but need to be integrated.
ISP4 already enables the runtime PM so the power settings can be moved
to the runtime PM and the s_power callback can be dropped. The runtime
PM can be used to power on and power off the ISP4 for the typical
operations, for example, the user turns on the camera and shutdowns
the camera. If the ISP4 media device manages the power status using
the runtime PM, the video stream can be gracefully stopped to ensure
no error events are propagated to the user apps. Moreover, it resolves
the suspend/resume issues and the runtime PM will manage the
suspend/resume status.
--
BR,
Kate
^ permalink raw reply [flat|nested] 46+ messages in thread* Re: [PATCH v7 0/7] Add AMD ISP4 driver
2026-01-14 8:59 ` Kate Hsuan
@ 2026-01-15 6:47 ` Du, Bin
0 siblings, 0 replies; 46+ messages in thread
From: Du, Bin @ 2026-01-15 6:47 UTC (permalink / raw)
To: Kate Hsuan, Nirujogi, Pratap
Cc: mchehab, hverkuil, laurent.pinchart+renesas, bryan.odonoghue,
sakari.ailus, prabhakar.mahadev-lad.rj, linux-media, linux-kernel,
sultan, pratap.nirujogi, benjamin.chan, king.li,
gjorgji.rosikopulos, Phil.Jawich, Dominic.Antony,
mario.limonciello, richard.gong, anson.tsao
On 1/14/2026 4:59 PM, Kate Hsuan wrote:
> Hi Pratap,
>
> On Wed, Jan 14, 2026 at 1:14 AM Nirujogi, Pratap <pnirujog@amd.com> wrote:
>>
>> Hi Kate,
>>
>> On 1/13/2026 9:11 AM, Kate Hsuan wrote:
>>> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
[snip]
>>>> --
>>>> Regards,
>>>> Bin
>>>>
>>> The ISP driver works perfectly with a clear and sharp video. I tested
>>> it again, and I found a suspend/resume issue.
>>> The ISP can't suspend when the system is set to suspend. The privacy
>>> LED is on when the system is suspended. Therefore, the user will see a
>>> luminous privacy LED when the system is set to suspend.
>>> Today, I made a work to move all the power control to use the runtime
>>> PM, including suspend/resume. This work may be humble and may break
>>> the finite state machine but it works. The major changes of it
>>> include:
>>> 1. Support suspend/resume.
>>> 2. The power is managed by the runtime PM so the s_power and the related
>>> callback function were dropped.
>>> 3. The enable_isp GPIO pin is controlled by the runtime PM.
>>> 4. pm_runtime_get_noresume() is used to get the runtime PM at probe()
>>> since the device doesn't have to be set to power on when initialising.
>>>
>>> This work stops the video stream on suspend and starts the stream on
>>> resume so the privacy LED is turned on and turned off with the changes
>>> of suspend and resume.
>>>
>>> Could you please consider this patch and idea?
>>>
>>> Thank you :)
>>
>> Thanks for reporting this issue and also providing the patch.
>>
>> We have addressed this issue recently. I suspect the below fix in AMDGPU
>> available in v6.19-rc5 is missing in your build.
>>
>> https://github.com/torvalds/linux/commit/7ed51e3a1381422278933d0d3ebda0268b6825de
>>
>> I have tested locally and this issue is not observed with this change
>> included. Can you please check and feedback if this solves the problem?
>>
>> This change takes care of handling isp suspend-resume as part of amdgpu
>> device suspend-resume instead of genpd, and uses the pm rumtime as you
>> have suggested.
>>
>> Thanks,
>>
>> Pratap
>>
>>
>>>
>>> --
>>> BR,
>>> Kate
>>
>
> I tested again with the patch you mentioned. It works but I found some issues.
> I also tested it with my work and the test results were shown as follows.
>
> The test step is
> 1. start the camera app -> 2. set to suspend -> 3. resume the laptop
>
> The results:
> 1. rebase to 6.19-RC5 with the fix patch (7ed51e3a1)
> It works but I found the logs when stopping the video stream
> ...
> [ 527.733851] amd_isp_capture amd_isp_capture: fail to disable stream
> ...
> [ 528.237828] amd_isp_capture amd_isp_capture: fail to stop steam
> ...
> The isp4 tries to stop the stream but it fails to write the data to
> the firmware.
> I think the ISP stops working before the ISP stops the video stream.
> So, it cannot write the data to the firmware.
>
> 2. fix patch (7ed51e3a1) + ISP4 runtime PM (my work)
> This also works.
> The ISP4 complains the error about "static void
> isp4if_dealloc_fw_gpumem(struct isp4_interface *ispif)" when suspend.
>
> (This only happens when suspending the system)
> [ 223.085419] WARNING: drivers/gpu/drm/amd/amdgpu/amdgpu_object.c:517
> at amdgpu_bo_free_kernel+0xe9/0x100 [amdgpu], CPU#30:
> kworker/u130:4/794
> [ 223.085552] Modules linked in: uinput(E) rfcomm(E) snd_seq_dummy(E)
> snd_hrtimer(E) hid_sensor_gyro_3d(E) hid_sensor_trigger(E)
> hid_sensor_iio_common(E) industrialio_triggered_buffer(E) kfifo_buf(E)
> industrialio(E) hid_sensor_hub(E) sunrpc(E) nf_conntrack_netbios_ns(E)
> nf_conntrack_broadcast(E) nft_fib_inet(E) nft_fib_ipv4(E)
>
> <snip>
>
> [ 223.085591] CPU: 30 UID: 0 PID: 794 Comm: kworker/u130:4 Tainted: G
> W E 6.19.0-rc5+ #92 PREEMPT(lazy)
> [ 223.085592] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE
> [ 223.085592] Hardware name: HP HP ZBook Ultra G1a 14 inch Mobile
> Workstation PC/8D01, BIOS X89 Ver. 01.01.00 01/16/2025
> [ 223.085593] Workqueue: async async_run_entry_fn
> [ 223.085594] RIP: 0010:amdgpu_bo_free_kernel+0xe9/0x100 [amdgpu]
> [ 223.085725] Code: f4 ff ff 4d 85 e4 74 08 49 c7 04 24 00 00 00 00
> 48 85 ed 74 08 48 c7 45 00 00 00 00 00 5b 5d 41 5c 41 5d 41 5e c3 cc
> cc cc cc <0f> 0b e9 42 ff ff ff 3d 00 fe ff ff 0f 85 a1 34 95 00 eb bd
> 0f 1f
> [ 223.085725] RSP: 0018:ffffcd80413e7ae0 EFLAGS: 00010202
> [ 223.085726] RAX: 0000000000000000 RBX: 0000000000000005 RCX: 0000000080800077
> [ 223.085726] RDX: ffff8b3719f6eef0 RSI: ffff8b3719f6eee8 RDI: ffff8b3719f6eef8
> [ 223.085727] RBP: ffff8b3719f6eee0 R08: 0000000000000000 R09: ffffffffc0558135
> [ 223.085727] R10: ffff8b3719f6e7a0 R11: ffffef494467db80 R12: ffff8b370ef86220
> [ 223.085727] R13: ffff8b3718003c00 R14: ffff8b372a60ee00 R15: ffff8b370ef86220
> [ 223.085728] FS: 0000000000000000(0000) GS:ffff8b5649311000(0000)
> knlGS:0000000000000000
> [ 223.085728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 223.085729] CR2: 0000563d26ad9000 CR3: 00000001ace2c000 CR4: 0000000000f50ef0
> [ 223.085730] PKRU: 55555554
> [ 223.085730] Call Trace:
> [ 223.085730] <TASK>
> [ 223.085731] isp4if_dealloc_fw_gpumem+0xcd/0x100 [amd_capture]
> [ 223.085733] isp4if_stop+0x59/0x70 [amd_capture]
> [ 223.085735] isp4sd_pwroff_and_deinit.isra.0+0x99/0x160 [amd_capture]
> [ 223.085737] isp4sd_stop_stream+0xbe/0x250 [amd_capture]
> [ 223.085740] v4l2_subdev_disable_streams+0x1ad/0x390 [videodev]
> [ 223.085747] ? dc_dmub_srv_wait_for_idle+0x50/0x150 [amdgpu]
> [ 223.085914] isp4_suspend+0x2d/0x80 [amd_capture]
> [ 223.085916] genpd_runtime_suspend+0xe7/0x300
> [ 223.085917] ? __pfx_isp_suspend_device+0x10/0x10 [amdgpu]
> [ 223.086069] pm_runtime_force_suspend+0x71/0x110
> [ 223.086070] ? __pfx_genpd_runtime_suspend+0x10/0x10
> [ 223.086071] device_for_each_child+0x71/0xb0
> [ 223.086073] isp_v4_1_1_hw_suspend+0x22/0x40 [amdgpu]
> [ 223.086223] ? amdgpu_dpm_gfx_state_change+0x49/0x60 [amdgpu]
> [ 223.086399] amdgpu_ip_block_suspend+0x27/0x50 [amdgpu]
> [ 223.086522] amdgpu_device_ip_suspend_phase2+0x13c/0x3d0 [amdgpu]
> [ 223.086647] amdgpu_device_suspend+0x161/0x240 [amdgpu]
>
>
> Both solutions work but need to be integrated.
> ISP4 already enables the runtime PM so the power settings can be moved
> to the runtime PM and the s_power callback can be dropped. The runtime
> PM can be used to power on and power off the ISP4 for the typical
> operations, for example, the user turns on the camera and shutdowns
> the camera. If the ISP4 media device manages the power status using
> the runtime PM, the video stream can be gracefully stopped to ensure
> no error events are propagated to the user apps. Moreover, it resolves
> the suspend/resume issues and the runtime PM will manage the
> suspend/resume status.
>
Thank you, Kate, for verifying and debugging. Yes, s_power is now
obsolete and should be replaced by runtime PM which is also suggested by
Sakari in one of his comments. In our initial version, support for
camera streaming suspend and resume wasn't included, so if you suspend
the laptop while the camera is running, it won't continue to run after
resuming. However, it's easy to fix by simply reopening the camera app.
Therefore, the stream disabling or stopping failure after resume is
expected and harmless. On the other hand, the isp4if_dealloc_fw_gpumem
error was unexpected, as we hadn't encountered it before. Once we've
switched from s_power to runtime PM, we'll test again on 6.19-RC5 as you.
--
Regards,
Bin
^ permalink raw reply [flat|nested] 46+ messages in thread