[PATCH v5 00/16] Add Arm Mali-C55 Image Signal Processor Driver

[PATCH v5 00/16] Add Arm Mali-C55 Image Signal Processor Driver

[Daniel Scally]

Hello all

This patchset introduces a driver for Arm's Mali-C55 Image Signal Processor.
The driver uses the V4L2 / media controller API and implements both of the ISP's
capture pipelines allowing a range of output formats plus downscaling and
cropping. The capture pipelines are named "Full resolution" and "Downscale" and
so abbreviated FR and DS throughout the driver.

The driver exposes 4 V4L2 subdevices:

- mali-c55 isp: input data formatting
- mali-c55 tpg: test pattern generator (modeled as a camera sensor entity)
- mali-c55 resizer fr: downscale / crop and format setting for the FR pipe
- mali-c55 resizer ds: downscale / crop and format setting for the DS pipe

Along with 4 V4L2 Video devices:

- mali-c55 fr: Capture device for the full resolution pipe
- mali-c55 ds: Capture device for the downscale pipe
- mali-c55 3a stats: Capture device for statistics to support 3A algorithms
- mali-c55 3a params: Output device for parameter buffers to configure the ISP

Support is implemented in the parameters video device code for many of the ISP'S
hardware blocks, but not yet all of them. The buffer format is (as far as I am
aware anyway) a novel design that we intend to be extensible so that support for
the C55's remaining hardware blocks can be added later.

Patches 1, 4, 5, 6 and 7 have already had 4 versions on the mailing list...I
decided to post the additional work on the driver as extra patches rather than
merge them all into the existing series as it's already a lot of code to review
and I hoped that that might make it a little easier...if I'm wrong and that's
not liked I can just squash them into a much smaller series.

Thanks
Dan

Daniel Scally (15):
  media: uapi: Add MEDIA_BUS_FMT_RGB202020_1X60 format code
  media: uapi: Add 20-bit bayer formats
  media: v4l2-common: Add RAW16 format info
  dt-bindings: media: Add bindings for ARM mali-c55
  media: mali-c55: Add Mali-C55 ISP driver
  media: Documentation: Add Mali-C55 ISP Documentation
  MAINTAINERS: Add entry for mali-c55 driver
  media: Add MALI_C55_3A_STATS meta format
  media: uapi: Add 3a stats buffer for mali-c55
  media: platform: Add mali-c55 3a stats devnode
  media: platform: Fill stats buffer on ISP_START
  Documentation: mali-c55: Add Statistics documentation
  media: uapi: Add parameters structs to mali-c55-config.h
  media: platform: Add mali-c55 parameters video node
  Documentation: mali-c55: Document the mali-c55 parameter setting

Jacopo Mondi (1):
  media: mali-c55: Add image formats for Mali-C55 parameters buffer

 .../admin-guide/media/mali-c55-graph.dot      |  19 +
 Documentation/admin-guide/media/mali-c55.rst  | 406 ++++++++
 .../admin-guide/media/v4l-drivers.rst         |   1 +
 .../bindings/media/arm,mali-c55.yaml          |  66 ++
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-arm-mali-c55.rst        |  87 ++
 .../media/v4l/subdev-formats.rst              | 268 +++++
 MAINTAINERS                                   |  10 +
 drivers/media/platform/Kconfig                |   1 +
 drivers/media/platform/Makefile               |   1 +
 drivers/media/platform/arm/Kconfig            |   5 +
 drivers/media/platform/arm/Makefile           |   2 +
 drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
 drivers/media/platform/arm/mali-c55/Makefile  |  11 +
 .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
 .../platform/arm/mali-c55/mali-c55-common.h   | 312 ++++++
 .../platform/arm/mali-c55/mali-c55-core.c     | 825 +++++++++++++++
 .../platform/arm/mali-c55/mali-c55-isp.c      | 626 ++++++++++++
 .../platform/arm/mali-c55/mali-c55-params.c   | 615 +++++++++++
 .../arm/mali-c55/mali-c55-registers.h         | 365 +++++++
 .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
 .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
 .../platform/arm/mali-c55/mali-c55-stats.c    | 350 +++++++
 .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
 drivers/media/v4l2-core/v4l2-common.c         |   4 +
 drivers/media/v4l2-core/v4l2-ioctl.c          |   2 +
 include/uapi/linux/media-bus-format.h         |   9 +-
 .../uapi/linux/media/arm/mali-c55-config.h    | 851 ++++++++++++++++
 include/uapi/linux/videodev2.h                |   3 +
 29 files changed, 7370 insertions(+), 2 deletions(-)
 create mode 100644 Documentation/admin-guide/media/mali-c55-graph.dot
 create mode 100644 Documentation/admin-guide/media/mali-c55.rst
 create mode 100644 Documentation/devicetree/bindings/media/arm,mali-c55.yaml
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
 create mode 100644 drivers/media/platform/arm/Kconfig
 create mode 100644 drivers/media/platform/arm/Makefile
 create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
 create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-params.c
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-stats.c
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
 create mode 100644 include/uapi/linux/media/arm/mali-c55-config.h

-- 
2.34.1

[PATCH v5 01/16] media: uapi: Add MEDIA_BUS_FMT_RGB202020_1X60 format code

[Daniel Scally]

The Mali-C55 ISP by ARM requires 20-bits per colour channel input on
the bus. Add a new media bus format code to represent it.

Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- none

Changes in v4:

	- None

Changes in v3:

	- None

Changes in v2:

	- none
 .../media/v4l/subdev-formats.rst              | 168 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   3 +-
 2 files changed, 170 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index d2a6cd2e1eb2..8d164a9a9e15 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -2224,6 +2224,174 @@ The following table list existing packed 48bit wide RGB formats.
 
     \endgroup
 
+The following table list existing packed 60bit wide RGB formats.
+
+.. tabularcolumns:: |p{4.0cm}|p{0.7cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|
+
+.. _v4l2-mbus-pixelcode-rgb-60:
+
+.. raw:: latex
+
+    \begingroup
+    \tiny
+    \setlength{\tabcolsep}{2pt}
+
+.. flat-table:: 60bit RGB formats
+    :header-rows:  3
+    :stub-columns: 0
+    :widths: 36 7 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
+
+    * - Identifier
+      - Code
+      -
+      - :cspan:`31` Data organization
+    * -
+      -
+      - Bit
+      -
+      -
+      -
+      -
+      - 59
+      - 58
+      - 57
+      - 56
+      - 55
+      - 54
+      - 53
+      - 52
+      - 51
+      - 50
+      - 49
+      - 48
+      - 47
+      - 46
+      - 45
+      - 44
+      - 43
+      - 42
+      - 41
+      - 40
+      - 39
+      - 38
+      - 37
+      - 36
+      - 35
+      - 34
+      - 33
+      - 32
+    * -
+      -
+      -
+      - 31
+      - 30
+      - 29
+      - 28
+      - 27
+      - 26
+      - 25
+      - 24
+      - 23
+      - 22
+      - 21
+      - 20
+      - 19
+      - 18
+      - 17
+      - 16
+      - 15
+      - 14
+      - 13
+      - 12
+      - 11
+      - 10
+      - 9
+      - 8
+      - 7
+      - 6
+      - 5
+      - 4
+      - 3
+      - 2
+      - 1
+      - 0
+    * .. _MEDIA-BUS-FMT-RGB202020-1X60:
+
+      - MEDIA_BUS_FMT_RGB202020_1X60
+      - 0x1026
+      -
+      -
+      -
+      -
+      -
+      - r\ :sub:`19`
+      - r\ :sub:`18`
+      - r\ :sub:`17`
+      - r\ :sub:`16`
+      - r\ :sub:`15`
+      - r\ :sub:`14`
+      - r\ :sub:`13`
+      - r\ :sub:`12`
+      - r\ :sub:`11`
+      - r\ :sub:`10`
+      - r\ :sub:`8`
+      - r\ :sub:`8`
+      - r\ :sub:`7`
+      - r\ :sub:`6`
+      - r\ :sub:`5`
+      - r\ :sub:`4`
+      - r\ :sub:`3`
+      - r\ :sub:`2`
+      - r\ :sub:`1`
+      - r\ :sub:`0`
+      - g\ :sub:`19`
+      - g\ :sub:`18`
+      - g\ :sub:`17`
+      - g\ :sub:`16`
+      - g\ :sub:`15`
+      - g\ :sub:`14`
+      - g\ :sub:`13`
+      - g\ :sub:`12`
+    * -
+      -
+      -
+      - g\ :sub:`11`
+      - g\ :sub:`10`
+      - g\ :sub:`9`
+      - g\ :sub:`8`
+      - g\ :sub:`7`
+      - g\ :sub:`6`
+      - g\ :sub:`5`
+      - g\ :sub:`4`
+      - g\ :sub:`3`
+      - g\ :sub:`2`
+      - g\ :sub:`1`
+      - g\ :sub:`0`
+      - b\ :sub:`19`
+      - b\ :sub:`18`
+      - b\ :sub:`17`
+      - b\ :sub:`16`
+      - b\ :sub:`15`
+      - b\ :sub:`14`
+      - b\ :sub:`13`
+      - b\ :sub:`12`
+      - b\ :sub:`11`
+      - b\ :sub:`10`
+      - b\ :sub:`9`
+      - b\ :sub:`8`
+      - b\ :sub:`7`
+      - b\ :sub:`6`
+      - b\ :sub:`5`
+      - b\ :sub:`4`
+      - b\ :sub:`3`
+      - b\ :sub:`2`
+      - b\ :sub:`1`
+      - b\ :sub:`0`
+
+.. raw:: latex
+
+    \endgroup
+
 On LVDS buses, usually each sample is transferred serialized in seven
 time slots per pixel clock, on three (18-bit) or four (24-bit)
 differential data pairs at the same time. The remaining bits are used
diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index d4c1d991014b..49be328d9a3b 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -34,7 +34,7 @@
 
 #define MEDIA_BUS_FMT_FIXED			0x0001
 
-/* RGB - next is	0x1026 */
+/* RGB - next is	0x1027 */
 #define MEDIA_BUS_FMT_RGB444_1X12		0x1016
 #define MEDIA_BUS_FMT_RGB444_2X8_PADHI_BE	0x1001
 #define MEDIA_BUS_FMT_RGB444_2X8_PADHI_LE	0x1002
@@ -72,6 +72,7 @@
 #define MEDIA_BUS_FMT_RGB888_1X36_CPADLO	0x1021
 #define MEDIA_BUS_FMT_RGB121212_1X36		0x1019
 #define MEDIA_BUS_FMT_RGB161616_1X48		0x101a
+#define MEDIA_BUS_FMT_RGB202020_1X60		0x1026
 
 /* YUV (including grey) - next is	0x202f */
 #define MEDIA_BUS_FMT_Y8_1X8			0x2001
-- 
2.34.1

[PATCH v5 02/16] media: uapi: Add 20-bit bayer formats

[Daniel Scally]

The Mali-C55 requires input data be in 20-bit format, MSB aligned.
Add some new media bus format macros to represent that input format.

Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- New patch

 .../media/v4l/subdev-formats.rst              | 100 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   6 +-
 2 files changed, 105 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index 8d164a9a9e15..f986dfc52879 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -3445,6 +3445,106 @@ organization is given as an example for the first pixel only.
       - r\ :sub:`2`
       - r\ :sub:`1`
       - r\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-SBGGR20-1X20:
+
+      - MEDIA_BUS_FMT_SBGGR20_1X20
+      - 0x3021
+      -
+      - r\ :sub:`19`
+      - r\ :sub:`18`
+      - r\ :sub:`17`
+      - r\ :sub:`16`
+      - r\ :sub:`15`
+      - r\ :sub:`14`
+      - r\ :sub:`13`
+      - r\ :sub:`12`
+      - r\ :sub:`11`
+      - r\ :sub:`10`
+      - r\ :sub:`9`
+      - r\ :sub:`8`
+      - r\ :sub:`7`
+      - r\ :sub:`6`
+      - r\ :sub:`5`
+      - r\ :sub:`4`
+      - r\ :sub:`3`
+      - r\ :sub:`2`
+      - r\ :sub:`1`
+      - r\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-SGBRG20-1X20:
+
+      - MEDIA_BUS_FMT_SGBRG20_1X20
+      - 0x3022
+      -
+      - r\ :sub:`19`
+      - r\ :sub:`18`
+      - r\ :sub:`17`
+      - r\ :sub:`16`
+      - r\ :sub:`15`
+      - r\ :sub:`14`
+      - r\ :sub:`13`
+      - r\ :sub:`12`
+      - r\ :sub:`11`
+      - r\ :sub:`10`
+      - r\ :sub:`9`
+      - r\ :sub:`8`
+      - r\ :sub:`7`
+      - r\ :sub:`6`
+      - r\ :sub:`5`
+      - r\ :sub:`4`
+      - r\ :sub:`3`
+      - r\ :sub:`2`
+      - r\ :sub:`1`
+      - r\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-SGRBG20-1X20:
+
+      - MEDIA_BUS_FMT_SGRBG20_1X20
+      - 0x3023
+      -
+      - r\ :sub:`19`
+      - r\ :sub:`18`
+      - r\ :sub:`17`
+      - r\ :sub:`16`
+      - r\ :sub:`15`
+      - r\ :sub:`14`
+      - r\ :sub:`13`
+      - r\ :sub:`12`
+      - r\ :sub:`11`
+      - r\ :sub:`10`
+      - r\ :sub:`9`
+      - r\ :sub:`8`
+      - r\ :sub:`7`
+      - r\ :sub:`6`
+      - r\ :sub:`5`
+      - r\ :sub:`4`
+      - r\ :sub:`3`
+      - r\ :sub:`2`
+      - r\ :sub:`1`
+      - r\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-SRGGB20-1X20:
+
+      - MEDIA_BUS_FMT_SRGGB20_1X20
+      - 0x3024
+      -
+      - r\ :sub:`19`
+      - r\ :sub:`18`
+      - r\ :sub:`17`
+      - r\ :sub:`16`
+      - r\ :sub:`15`
+      - r\ :sub:`14`
+      - r\ :sub:`13`
+      - r\ :sub:`12`
+      - r\ :sub:`11`
+      - r\ :sub:`10`
+      - r\ :sub:`9`
+      - r\ :sub:`8`
+      - r\ :sub:`7`
+      - r\ :sub:`6`
+      - r\ :sub:`5`
+      - r\ :sub:`4`
+      - r\ :sub:`3`
+      - r\ :sub:`2`
+      - r\ :sub:`1`
+      - r\ :sub:`0`
 
 .. raw:: latex
 
diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index 49be328d9a3b..b6acf8c8e383 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -122,7 +122,7 @@
 #define MEDIA_BUS_FMT_YUV16_1X48		0x202a
 #define MEDIA_BUS_FMT_UYYVYY16_0_5X48		0x202b
 
-/* Bayer - next is	0x3021 */
+/* Bayer - next is	0x3025 */
 #define MEDIA_BUS_FMT_SBGGR8_1X8		0x3001
 #define MEDIA_BUS_FMT_SGBRG8_1X8		0x3013
 #define MEDIA_BUS_FMT_SGRBG8_1X8		0x3002
@@ -155,6 +155,10 @@
 #define MEDIA_BUS_FMT_SGBRG16_1X16		0x301e
 #define MEDIA_BUS_FMT_SGRBG16_1X16		0x301f
 #define MEDIA_BUS_FMT_SRGGB16_1X16		0x3020
+#define MEDIA_BUS_FMT_SBGGR20_1X20		0x3021
+#define MEDIA_BUS_FMT_SGBRG20_1X20		0x3022
+#define MEDIA_BUS_FMT_SGRBG20_1X20		0x3023
+#define MEDIA_BUS_FMT_SRGGB20_1X20		0x3024
 
 /* JPEG compressed formats - next is	0x4002 */
 #define MEDIA_BUS_FMT_JPEG_1X8			0x4001
-- 
2.34.1

[PATCH v5 03/16] media: v4l2-common: Add RAW16 format info

[Daniel Scally]

Add entries to v4l2_format_info describing the 16-bit bayer
formats so that they can be used in drivers.

Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- New patch

 drivers/media/v4l2-core/v4l2-common.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-common.c b/drivers/media/v4l2-core/v4l2-common.c
index 4165c815faef..c5d5704af5ee 100644
--- a/drivers/media/v4l2-core/v4l2-common.c
+++ b/drivers/media/v4l2-core/v4l2-common.c
@@ -331,6 +331,10 @@ const struct v4l2_format_info *v4l2_format_info(u32 format)
 		{ .format = V4L2_PIX_FMT_SGBRG12,	.pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
 		{ .format = V4L2_PIX_FMT_SGRBG12,	.pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
 		{ .format = V4L2_PIX_FMT_SRGGB12,	.pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
+		{ .format = V4L2_PIX_FMT_SBGGR16,	.pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
+		{ .format = V4L2_PIX_FMT_SGBRG16,	.pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
+		{ .format = V4L2_PIX_FMT_SGRBG16,	.pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
+		{ .format = V4L2_PIX_FMT_SRGGB16,	.pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
 	};
 	unsigned int i;
 
-- 
2.34.1

[PATCH v5 04/16] dt-bindings: media: Add bindings for ARM mali-c55

[Daniel Scally]

Add the yaml binding for ARM's Mali-C55 Image Signal Processor.

Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- None

Changes in v4:

	- Switched to port instead of ports

Changes in v3:

	- Dropped the video clock as suggested by Laurent. I didn't retain it
	for the purposes of the refcount since this driver will call .s_stream()
	for the sensor driver which will refcount the clock anyway.
	- Clarified that the port is a parallel input port rather (Sakari)

Changes in v2:

	- Added clocks information
	- Fixed the warnings raised by Rob

 .../bindings/media/arm,mali-c55.yaml          | 66 +++++++++++++++++++
 1 file changed, 66 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/arm,mali-c55.yaml

diff --git a/Documentation/devicetree/bindings/media/arm,mali-c55.yaml b/Documentation/devicetree/bindings/media/arm,mali-c55.yaml
new file mode 100644
index 000000000000..9cc2481f2da3
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/arm,mali-c55.yaml
@@ -0,0 +1,66 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/media/arm,mali-c55.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: ARM Mali-C55 Image Signal Processor
+
+maintainers:
+  - Daniel Scally <dan.scally@ideasonboard.com>
+  - Jacopo Mondi <jacopo.mondi@ideasonboard.com>
+
+properties:
+  compatible:
+    const: arm,mali-c55
+
+  reg:
+    maxItems: 1
+
+  interrupts:
+    maxItems: 1
+
+  clocks:
+    items:
+      - description: ISP AXI clock
+      - description: ISP AHB-lite clock
+
+  clock-names:
+    items:
+      - const: aclk
+      - const: hclk
+
+  port:
+    $ref: /schemas/graph.yaml#/properties/port
+    description: Input parallel video bus
+
+    properties:
+      endpoint:
+        $ref: /schemas/graph.yaml#/properties/endpoint
+
+required:
+  - compatible
+  - reg
+  - interrupts
+  - clocks
+  - clock-names
+  - port
+
+additionalProperties: false
+
+examples:
+  - |
+    mali_c55: isp@400000 {
+      compatible = "arm,mali-c55";
+      reg = <0x400000 0x200000>;
+      clocks = <&clk 0>, <&clk 1>;
+      clock-names = "aclk", "hclk";
+      interrupts = <0>;
+
+      port {
+        isp_in: endpoint {
+            remote-endpoint = <&csi2_rx_out>;
+        };
+      };
+    };
+...
-- 
2.34.1

[PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Daniel Scally]

Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
V4L2 and Media Controller compliant and creates subdevices to manage
the ISP itself, its internal test pattern generator as well as the
crop, scaler and output format functionality for each of its two
output devices.

Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- Reworked input formats - previously we allowed representing input data
	  as any 8-16 bit format. Now we only allow input data to be represented
	  by the new 20-bit bayer formats, which is corrected to the equivalent
	  16-bit format in RAW bypass mode.
	- Stopped bypassing blocks that we haven't added supporting parameters
	  for yet.
	- Addressed most of Sakari's comments from the list

Changes not yet made in v5:

	- The output pipelines can still be started and stopped independently of
	  one another - I'd like to discuss that more. 
	- the TPG subdev still uses .s_stream() - I need to rebase onto a tree
	  with working .enable_streams() for a single-source-pad subdevice.

Changes in v4:

	- Reworked mali_c55_update_bits() to internally perform the bit-shift
	- Reworked the resizer to allow cropping during streaming
	- Fixed a bug in NV12 output

Changes in v3:

	- Mostly minor fixes suggested by Sakari
	- Fixed the sequencing of vb2 buffers to be synchronised across the two
	  capture devices.

Changes in v2:

	- Clock handling
	- Fixed the warnings raised by the kernel test robot

 drivers/media/platform/Kconfig                |   1 +
 drivers/media/platform/Makefile               |   1 +
 drivers/media/platform/arm/Kconfig            |   5 +
 drivers/media/platform/arm/Makefile           |   2 +
 drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
 drivers/media/platform/arm/mali-c55/Makefile  |   9 +
 .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
 .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
 .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
 .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
 .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
 .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
 .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
 .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
 14 files changed, 4452 insertions(+)
 create mode 100644 drivers/media/platform/arm/Kconfig
 create mode 100644 drivers/media/platform/arm/Makefile
 create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
 create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c

diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
index 2d79bfc68c15..c929169766aa 100644
--- a/drivers/media/platform/Kconfig
+++ b/drivers/media/platform/Kconfig
@@ -65,6 +65,7 @@ config VIDEO_MUX
 source "drivers/media/platform/allegro-dvt/Kconfig"
 source "drivers/media/platform/amlogic/Kconfig"
 source "drivers/media/platform/amphion/Kconfig"
+source "drivers/media/platform/arm/Kconfig"
 source "drivers/media/platform/aspeed/Kconfig"
 source "drivers/media/platform/atmel/Kconfig"
 source "drivers/media/platform/broadcom/Kconfig"
diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
index da17301f7439..9a647abd5218 100644
--- a/drivers/media/platform/Makefile
+++ b/drivers/media/platform/Makefile
@@ -8,6 +8,7 @@
 obj-y += allegro-dvt/
 obj-y += amlogic/
 obj-y += amphion/
+obj-y += arm/
 obj-y += aspeed/
 obj-y += atmel/
 obj-y += broadcom/
diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
new file mode 100644
index 000000000000..4f0764c329c7
--- /dev/null
+++ b/drivers/media/platform/arm/Kconfig
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+comment "ARM media platform drivers"
+
+source "drivers/media/platform/arm/mali-c55/Kconfig"
diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
new file mode 100644
index 000000000000..8cc4918725ef
--- /dev/null
+++ b/drivers/media/platform/arm/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-y += mali-c55/
diff --git a/drivers/media/platform/arm/mali-c55/Kconfig b/drivers/media/platform/arm/mali-c55/Kconfig
new file mode 100644
index 000000000000..602085e28b01
--- /dev/null
+++ b/drivers/media/platform/arm/mali-c55/Kconfig
@@ -0,0 +1,18 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config VIDEO_MALI_C55
+	tristate "ARM Mali-C55 Image Signal Processor driver"
+	depends on V4L_PLATFORM_DRIVERS
+	depends on VIDEO_DEV && OF
+	depends on ARCH_VEXPRESS || COMPILE_TEST
+	select MEDIA_CONTROLLER
+	select VIDEO_V4L2_SUBDEV_API
+	select VIDEOBUF2_DMA_CONTIG
+	select VIDEOBUF2_VMALLOC
+	select V4L2_FWNODE
+	select GENERIC_PHY_MIPI_DPHY
+	default n
+	help
+	  Enable this to support Arm's Mali-C55 Image Signal Processor.
+
+	  To compile this driver as a module, choose M here: the module
+	  will be called mali-c55.
diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
new file mode 100644
index 000000000000..77dcb2fbf0f4
--- /dev/null
+++ b/drivers/media/platform/arm/mali-c55/Makefile
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0
+
+mali-c55-y := mali-c55-capture.o \
+	      mali-c55-core.o \
+	      mali-c55-isp.o \
+	      mali-c55-tpg.o \
+	      mali-c55-resizer.o
+
+obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
new file mode 100644
index 000000000000..1d539ac9c498
--- /dev/null
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
@@ -0,0 +1,951 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ARM Mali-C55 ISP Driver - Video capture devices
+ *
+ * Copyright (C) 2024 Ideas on Board Oy
+ */
+
+#include <linux/cleanup.h>
+#include <linux/minmax.h>
+#include <linux/pm_runtime.h>
+#include <linux/string.h>
+#include <linux/videodev2.h>
+
+#include <media/v4l2-dev.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-core.h>
+#include <media/videobuf2-dma-contig.h>
+
+#include "mali-c55-common.h"
+#include "mali-c55-registers.h"
+
+static const struct mali_c55_fmt mali_c55_fmts[] = {
+	/*
+	 * This table is missing some entries which need further work or
+	 * investigation:
+	 *
+	 * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
+	 * Base mode 5 is "Generic Data"
+	 * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
+	 * Base mode 9 seems to have no V4L2 equivalent
+	 * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
+	 * equivalent
+	 */
+	{
+		.fourcc = V4L2_PIX_FMT_ARGB2101010,
+		.mbus_codes = {
+			MEDIA_BUS_FMT_RGB121212_1X36,
+			MEDIA_BUS_FMT_RGB202020_1X60,
+		},
+		.is_raw = false,
+		.registers = {
+			.base_mode = MALI_C55_OUTPUT_A2R10G10B10,
+			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
+		}
+	},
+	{
+		.fourcc = V4L2_PIX_FMT_RGB565,
+		.mbus_codes = {
+			MEDIA_BUS_FMT_RGB121212_1X36,
+			MEDIA_BUS_FMT_RGB202020_1X60,
+		},
+		.is_raw = false,
+		.registers = {
+			.base_mode = MALI_C55_OUTPUT_RGB565,
+			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
+		}
+	},
+	{
+		.fourcc = V4L2_PIX_FMT_BGR24,
+		.mbus_codes = {
+			MEDIA_BUS_FMT_RGB121212_1X36,
+			MEDIA_BUS_FMT_RGB202020_1X60,
+		},
+		.is_raw = false,
+		.registers = {
+			.base_mode = MALI_C55_OUTPUT_RGB24,
+			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
+		}
+	},
+	{
+		.fourcc = V4L2_PIX_FMT_YUYV,
+		.mbus_codes = {
+			MEDIA_BUS_FMT_YUV10_1X30,
+		},
+		.is_raw = false,
+		.registers = {
+			.base_mode = MALI_C55_OUTPUT_YUY2,
+			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
+		}
+	},
+	{
+		.fourcc = V4L2_PIX_FMT_UYVY,
+		.mbus_codes = {
+			MEDIA_BUS_FMT_YUV10_1X30,
+		},
+		.is_raw = false,
+		.registers = {
+			.base_mode = MALI_C55_OUTPUT_UYVY,
+			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
+		}
+	},
+	{
+		.fourcc = V4L2_PIX_FMT_Y210,
+		.mbus_codes = {
+			MEDIA_BUS_FMT_YUV10_1X30,
+		},
+		.is_raw = false,
+		.registers = {
+			.base_mode = MALI_C55_OUTPUT_Y210,
+			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
+		}
+	},
+	/*
+	 * This is something of a hack, the ISP thinks it's running NV12M but
+	 * by setting uv_plane = 0 we simply discard that planes and only output
+	 * the Y-plane.
+	 */
+	{
+		.fourcc = V4L2_PIX_FMT_GREY,
+		.mbus_codes = {
+			MEDIA_BUS_FMT_YUV10_1X30,
+		},
+		.is_raw = false,
+		.registers = {
+			.base_mode = MALI_C55_OUTPUT_NV12_21,
+			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
+		}
+	},
+	{
+		.fourcc = V4L2_PIX_FMT_NV12M,
+		.mbus_codes = {
+			MEDIA_BUS_FMT_YUV10_1X30,
+		},
+		.is_raw = false,
+		.registers = {
+			.base_mode = MALI_C55_OUTPUT_NV12_21,
+			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
+		}
+	},
+	{
+		.fourcc = V4L2_PIX_FMT_NV21M,
+		.mbus_codes = {
+			MEDIA_BUS_FMT_YUV10_1X30,
+		},
+		.is_raw = false,
+		.registers = {
+			.base_mode = MALI_C55_OUTPUT_NV12_21,
+			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
+		}
+	},
+	/*
+	 * RAW uncompressed formats are all packed in 16 bpp.
+	 * TODO: Expand this list to encompass all possible RAW formats.
+	 */
+	{
+		.fourcc = V4L2_PIX_FMT_SRGGB16,
+		.mbus_codes = {
+			MEDIA_BUS_FMT_SRGGB16_1X16,
+		},
+		.is_raw = true,
+		.registers = {
+			.base_mode = MALI_C55_OUTPUT_RAW16,
+			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
+		}
+	},
+	{
+		.fourcc = V4L2_PIX_FMT_SBGGR16,
+		.mbus_codes = {
+			MEDIA_BUS_FMT_SBGGR16_1X16,
+		},
+		.is_raw = true,
+		.registers = {
+			.base_mode = MALI_C55_OUTPUT_RAW16,
+			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
+		}
+	},
+	{
+		.fourcc = V4L2_PIX_FMT_SGBRG16,
+		.mbus_codes = {
+			MEDIA_BUS_FMT_SGBRG16_1X16,
+		},
+		.is_raw = true,
+		.registers = {
+			.base_mode = MALI_C55_OUTPUT_RAW16,
+			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
+		}
+	},
+	{
+		.fourcc = V4L2_PIX_FMT_SGRBG16,
+		.mbus_codes = {
+			MEDIA_BUS_FMT_SGRBG16_1X16,
+		},
+		.is_raw = true,
+		.registers = {
+			.base_mode = MALI_C55_OUTPUT_RAW16,
+			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
+		}
+	},
+};
+
+static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
+					       u32 code)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
+		if (fmt->mbus_codes[i] == code)
+			return true;
+	}
+
+	return false;
+}
+
+bool mali_c55_format_is_raw(unsigned int mbus_code)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
+		if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
+			return mali_c55_fmts[i].is_raw;
+	}
+
+	return false;
+}
+
+static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
+		if (mali_c55_fmts[i].fourcc == pixelformat)
+			return &mali_c55_fmts[i];
+	}
+
+	/*
+	 * If we find no matching pixelformat, we'll just default to the first
+	 * one for now.
+	 */
+
+	return &mali_c55_fmts[0];
+}
+
+static const char * const capture_device_names[] = {
+	"mali-c55 fr",
+	"mali-c55 ds",
+	"mali-c55 3a stats",
+	"mali-c55 params",
+};
+
+static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
+{
+	if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
+		return capture_device_names[0];
+
+	if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
+		return capture_device_names[1];
+
+	return "params/stat not supported yet";
+}
+
+static int mali_c55_link_validate(struct media_link *link)
+{
+	struct video_device *vdev =
+		media_entity_to_video_device(link->sink->entity);
+	struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
+	struct v4l2_subdev *sd =
+		media_entity_to_v4l2_subdev(link->source->entity);
+	const struct v4l2_pix_format_mplane *pix_mp;
+	const struct mali_c55_fmt *cap_fmt;
+	struct v4l2_subdev_format sd_fmt = {
+		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
+		.pad = link->source->index,
+	};
+	int ret;
+
+	ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
+	if (ret)
+		return ret;
+
+	pix_mp = &cap_dev->mode.pix_mp;
+	cap_fmt = cap_dev->mode.capture_fmt;
+
+	if (sd_fmt.format.width != pix_mp->width ||
+	    sd_fmt.format.height != pix_mp->height) {
+		dev_dbg(cap_dev->mali_c55->dev,
+			"link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
+			link->source->entity->name, link->source->index,
+			link->sink->entity->name, link->sink->index,
+			sd_fmt.format.width, sd_fmt.format.height,
+			pix_mp->width, pix_mp->height);
+		return -EPIPE;
+	}
+
+	if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
+		dev_dbg(cap_dev->mali_c55->dev,
+			"link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format %p4cc\n",
+			link->source->entity->name, link->source->index,
+			link->sink->entity->name, link->sink->index,
+			sd_fmt.format.code, &pix_mp->pixelformat);
+		return -EPIPE;
+	}
+
+	return 0;
+}
+
+static const struct media_entity_operations mali_c55_media_ops = {
+	.link_validate = mali_c55_link_validate,
+};
+
+static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
+				    unsigned int *num_planes, unsigned int sizes[],
+				    struct device *alloc_devs[])
+{
+	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
+	unsigned int i;
+
+	if (*num_planes) {
+		if (*num_planes != cap_dev->mode.pix_mp.num_planes)
+			return -EINVAL;
+
+		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
+			if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
+				return -EINVAL;
+	} else {
+		*num_planes = cap_dev->mode.pix_mp.num_planes;
+		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
+			sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
+	}
+
+	return 0;
+}
+
+static void mali_c55_buf_queue(struct vb2_buffer *vb)
+{
+	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
+	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
+	struct mali_c55_buffer *buf = container_of(vbuf,
+						   struct mali_c55_buffer, vb);
+	unsigned int i;
+
+	buf->plane_done[MALI_C55_PLANE_Y] = false;
+
+	/*
+	 * If we're in a single-plane format we flag the other plane as done
+	 * already so it's dequeued appropriately later
+	 */
+	buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
+
+	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
+		unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
+
+		vb2_set_plane_payload(vb, i, size);
+	}
+
+	spin_lock(&cap_dev->buffers.lock);
+	list_add_tail(&buf->queue, &cap_dev->buffers.queue);
+	spin_unlock(&cap_dev->buffers.lock);
+}
+
+static int mali_c55_buf_init(struct vb2_buffer *vb)
+{
+	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
+	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
+	struct mali_c55_buffer *buf = container_of(vbuf,
+						   struct mali_c55_buffer, vb);
+	unsigned int i;
+
+	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
+		buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
+
+	return 0;
+}
+
+void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
+{
+	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
+
+	guard(spinlock)(&cap_dev->buffers.lock);
+
+	cap_dev->buffers.curr = cap_dev->buffers.next;
+	cap_dev->buffers.next = NULL;
+
+	if (!list_empty(&cap_dev->buffers.queue)) {
+		struct v4l2_pix_format_mplane *pix_mp;
+		const struct v4l2_format_info *info;
+		u32 *addrs;
+
+		pix_mp = &cap_dev->mode.pix_mp;
+		info = v4l2_format_info(pix_mp->pixelformat);
+
+		mali_c55_update_bits(mali_c55,
+				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
+				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
+		if (cap_dev->mode.capture_fmt->registers.uv_plane)
+			mali_c55_update_bits(mali_c55,
+				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
+				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
+
+		cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
+							 struct mali_c55_buffer,
+							 queue);
+		list_del(&cap_dev->buffers.next->queue);
+
+		addrs = cap_dev->buffers.next->addrs;
+		mali_c55_write(mali_c55,
+			MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
+			addrs[MALI_C55_PLANE_Y]);
+		mali_c55_write(mali_c55,
+			MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
+			addrs[MALI_C55_PLANE_UV]);
+		mali_c55_write(mali_c55,
+			MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
+			pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
+		mali_c55_write(mali_c55,
+			MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
+			pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
+			/ info->hdiv);
+	} else {
+		/*
+		 * If we underflow then we can tell the ISP that we don't want
+		 * to write out the next frame.
+		 */
+		mali_c55_update_bits(mali_c55,
+				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
+				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
+		mali_c55_update_bits(mali_c55,
+				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
+				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
+	}
+}
+
+static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
+				   unsigned int framecount)
+{
+	curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
+	curr_buf->vb.field = V4L2_FIELD_NONE;
+	curr_buf->vb.sequence = framecount;
+	vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
+}
+
+/**
+ * mali_c55_set_plane_done - mark the plane as written and process the buffer if
+ *			     both planes are finished.
+ * @cap_dev:  pointer to the fr or ds pipe output
+ * @plane:    the plane to mark as completed
+ *
+ * The Mali C55 ISP has muliplanar outputs for some formats that come with two
+ * separate "buffer write completed" interrupts - we need to flag each plane's
+ * completion and check whether both planes are done - if so, complete the buf
+ * in vb2.
+ */
+void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
+			     enum mali_c55_planes plane)
+{
+	struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
+	struct mali_c55_buffer *curr_buf;
+
+	guard(spinlock)(&cap_dev->buffers.lock);
+	curr_buf = cap_dev->buffers.curr;
+
+	/*
+	 * This _should_ never happen. If no buffer was available from vb2 then
+	 * we tell the ISP not to bother writing the next frame, which means the
+	 * interrupts that call this function should never trigger. If it does
+	 * happen then one of our assumptions is horribly wrong - complain
+	 * loudly and do nothing.
+	 */
+	if (!curr_buf) {
+		dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
+			mali_c55_cap_dev_to_name(cap_dev), __func__);
+		return;
+	}
+
+	/* If the other plane is also done... */
+	if (curr_buf->plane_done[~plane & 1]) {
+		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
+		cap_dev->buffers.curr = NULL;
+		isp->frame_sequence++;
+	} else {
+		curr_buf->plane_done[plane] = true;
+	}
+}
+
+static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
+{
+	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
+
+	mali_c55_update_bits(mali_c55,
+			     MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
+			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
+	mali_c55_update_bits(mali_c55,
+			     MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
+			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
+}
+
+static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
+{
+	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
+
+	/*
+	 * The Mali ISP can hold up to 5 buffer addresses and simply cycle
+	 * through them, but it's not clear to me that the vb2 queue _guarantees_
+	 * it will queue buffers to the driver in a fixed order, and ensuring
+	 * we call vb2_buffer_done() for the right buffer seems to me to add
+	 * pointless complexity given in multi-context mode we'd need to
+	 * re-write those registers every frame anyway...so we tell the ISP to
+	 * use a single register and update it for each frame.
+	 */
+	mali_c55_update_bits(mali_c55,
+			MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
+			MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
+	mali_c55_update_bits(mali_c55,
+			MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
+			MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
+
+	/*
+	 * We only queue a buffer in the streamon path if this is the first of
+	 * the capture devices to start streaming. If the ISP is already running
+	 * then we rely on the ISP_START interrupt to queue the first buffer for
+	 * this capture device.
+	 */
+	if (mali_c55->pipe.start_count == 1)
+		mali_c55_set_next_buffer(cap_dev);
+}
+
+static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
+					    enum vb2_buffer_state state)
+{
+	struct mali_c55_buffer *buf, *tmp;
+
+	guard(spinlock)(&cap_dev->buffers.lock);
+
+	if (cap_dev->buffers.curr) {
+		vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
+				state);
+		cap_dev->buffers.curr = NULL;
+	}
+
+	if (cap_dev->buffers.next) {
+		vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
+				state);
+		cap_dev->buffers.next = NULL;
+	}
+
+	list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
+		list_del(&buf->queue);
+		vb2_buffer_done(&buf->vb.vb2_buf, state);
+	}
+}
+
+static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
+{
+	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
+	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
+	struct mali_c55_resizer *rzr = cap_dev->rzr;
+	struct mali_c55_isp *isp = &mali_c55->isp;
+	int ret;
+
+	guard(mutex)(&isp->lock);
+
+	ret = pm_runtime_resume_and_get(mali_c55->dev);
+	if (ret)
+		return ret;
+
+	ret = video_device_pipeline_start(&cap_dev->vdev,
+					  &cap_dev->mali_c55->pipe);
+	if (ret) {
+		dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
+			mali_c55_cap_dev_to_name(cap_dev));
+		goto err_pm_put;
+	}
+
+	mali_c55_cap_dev_stream_enable(cap_dev);
+	mali_c55_rzr_start_stream(rzr);
+
+	/*
+	 * We only start the ISP if we're the only capture device that's
+	 * streaming. Otherwise, it'll already be active.
+	 */
+	if (mali_c55->pipe.start_count == 1) {
+		ret = mali_c55_isp_start_stream(isp);
+		if (ret)
+			goto err_disable_cap_dev;
+	}
+
+	return 0;
+
+err_disable_cap_dev:
+	mali_c55_cap_dev_stream_disable(cap_dev);
+	video_device_pipeline_stop(&cap_dev->vdev);
+err_pm_put:
+	pm_runtime_put(mali_c55->dev);
+	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
+
+	return ret;
+}
+
+static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
+{
+	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
+	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
+	struct mali_c55_resizer *rzr = cap_dev->rzr;
+	struct mali_c55_isp *isp = &mali_c55->isp;
+
+	guard(mutex)(&isp->lock);
+
+	/*
+	 * If one of the other capture nodes is streaming, we shouldn't
+	 * disable the ISP here.
+	 */
+	if (mali_c55->pipe.start_count == 1)
+		mali_c55_isp_stop_stream(&mali_c55->isp);
+
+	mali_c55_rzr_stop_stream(rzr);
+	mali_c55_cap_dev_stream_disable(cap_dev);
+	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
+	video_device_pipeline_stop(&cap_dev->vdev);
+	pm_runtime_put(mali_c55->dev);
+}
+
+static const struct vb2_ops mali_c55_vb2_ops = {
+	.queue_setup		= &mali_c55_vb2_queue_setup,
+	.buf_queue		= &mali_c55_buf_queue,
+	.buf_init		= &mali_c55_buf_init,
+	.wait_prepare		= vb2_ops_wait_prepare,
+	.wait_finish		= vb2_ops_wait_finish,
+	.start_streaming	= &mali_c55_vb2_start_streaming,
+	.stop_streaming		= &mali_c55_vb2_stop_streaming,
+};
+
+static const struct v4l2_file_operations mali_c55_v4l2_fops = {
+	.owner = THIS_MODULE,
+	.unlocked_ioctl = video_ioctl2,
+	.open = v4l2_fh_open,
+	.release = vb2_fop_release,
+	.poll = vb2_fop_poll,
+	.mmap = vb2_fop_mmap,
+};
+
+static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
+{
+	const struct mali_c55_fmt *capture_format;
+	const struct v4l2_format_info *info;
+	struct v4l2_plane_pix_format *plane;
+	unsigned int i;
+
+	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
+	pix_mp->pixelformat = capture_format->fourcc;
+
+	pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
+			      MALI_C55_MAX_WIDTH);
+	pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
+			       MALI_C55_MAX_HEIGHT);
+
+	pix_mp->field = V4L2_FIELD_NONE;
+	pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
+	pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
+	pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
+
+	info = v4l2_format_info(pix_mp->pixelformat);
+	pix_mp->num_planes = info->mem_planes;
+	memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
+
+	pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;
+	pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
+				       * pix_mp->height;
+
+	for (i = 1; i < info->comp_planes; i++) {
+		plane = &pix_mp->plane_fmt[i];
+
+		plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
+						   info->hdiv);
+		plane->sizeimage = DIV_ROUND_UP(
+					plane->bytesperline * pix_mp->height,
+					info->vdiv);
+	}
+
+	if (info->mem_planes == 1) {
+		for (i = 1; i < info->comp_planes; i++) {
+			plane = &pix_mp->plane_fmt[i];
+			pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
+		}
+	}
+}
+
+static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
+					   struct v4l2_format *f)
+{
+	mali_c55_try_fmt(&f->fmt.pix_mp);
+
+	return 0;
+}
+
+static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
+				struct v4l2_pix_format_mplane *pix_mp)
+{
+	const struct mali_c55_fmt *capture_format;
+	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
+	const struct v4l2_format_info *info;
+
+	mali_c55_try_fmt(pix_mp);
+	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
+	info = v4l2_format_info(pix_mp->pixelformat);
+	if (WARN_ON(!info))
+		return;
+
+	mali_c55_write(mali_c55,
+		       MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
+		       capture_format->registers.base_mode);
+	mali_c55_write(mali_c55,
+		       MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
+		       MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
+		       MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
+
+	if (info->mem_planes > 1) {
+		mali_c55_write(mali_c55,
+			       MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
+			       capture_format->registers.base_mode);
+		mali_c55_update_bits(mali_c55,
+				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
+				MALI_C55_WRITER_SUBMODE_MASK,
+				capture_format->registers.uv_plane);
+
+		mali_c55_write(mali_c55,
+			MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
+			MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
+			MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
+	}
+
+	if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
+		/*
+		 * TODO: Figure out the colour matrix coefficients and calculate
+		 * and write them here.
+		 */
+
+		mali_c55_write(mali_c55,
+			       MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
+			       MALI_C55_CS_CONV_MATRIX_MASK);
+
+		if (info->hdiv > 1)
+			mali_c55_update_bits(mali_c55,
+				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
+				MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
+		if (info->vdiv > 1)
+			mali_c55_update_bits(mali_c55,
+				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
+				MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
+		if (info->hdiv > 1 || info->vdiv > 1)
+			mali_c55_update_bits(mali_c55,
+				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
+				MALI_C55_CS_CONV_FILTER_MASK, 0x01);
+	}
+
+	cap_dev->mode.pix_mp = *pix_mp;
+	cap_dev->mode.capture_fmt = capture_format;
+}
+
+static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
+					 struct v4l2_format *f)
+{
+	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
+
+	if (vb2_is_busy(&cap_dev->queue))
+		return -EBUSY;
+
+	mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
+
+	return 0;
+}
+
+static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
+					 struct v4l2_format *f)
+{
+	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
+
+	f->fmt.pix_mp = cap_dev->mode.pix_mp;
+
+	return 0;
+}
+
+static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
+					    struct v4l2_fmtdesc *f)
+{
+	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
+	unsigned int j = 0;
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
+		if (f->mbus_code &&
+		    !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
+						       f->mbus_code))
+			continue;
+
+		/* Downscale pipe can't output RAW formats */
+		if (mali_c55_fmts[i].is_raw &&
+		    cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
+			continue;
+
+		if (j++ == f->index) {
+			f->pixelformat = mali_c55_fmts[i].fourcc;
+			return 0;
+		}
+	}
+
+	return -EINVAL;
+}
+
+static int mali_c55_querycap(struct file *file, void *fh,
+			     struct v4l2_capability *cap)
+{
+	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
+	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
+
+	return 0;
+}
+
+static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
+	.vidioc_reqbufs = vb2_ioctl_reqbufs,
+	.vidioc_querybuf = vb2_ioctl_querybuf,
+	.vidioc_create_bufs = vb2_ioctl_create_bufs,
+	.vidioc_qbuf = vb2_ioctl_qbuf,
+	.vidioc_expbuf = vb2_ioctl_expbuf,
+	.vidioc_dqbuf = vb2_ioctl_dqbuf,
+	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+	.vidioc_streamon = vb2_ioctl_streamon,
+	.vidioc_streamoff = vb2_ioctl_streamoff,
+	.vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
+	.vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
+	.vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
+	.vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
+	.vidioc_querycap = mali_c55_querycap,
+	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
+	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
+};
+
+int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
+{
+	struct v4l2_pix_format_mplane pix_mp;
+	struct mali_c55_cap_dev *cap_dev;
+	struct video_device *vdev;
+	struct vb2_queue *vb2q;
+	unsigned int i;
+	int ret;
+
+	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
+		cap_dev = &mali_c55->cap_devs[i];
+		vdev = &cap_dev->vdev;
+		vb2q = &cap_dev->queue;
+
+		/*
+		 * The downscale output pipe is an optional block within the ISP
+		 * so we need to check whether it's actually been fitted or not.
+		 */
+
+		if (i == MALI_C55_CAP_DEV_DS &&
+		    !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
+			continue;
+
+		cap_dev->mali_c55 = mali_c55;
+		mutex_init(&cap_dev->lock);
+		INIT_LIST_HEAD(&cap_dev->buffers.queue);
+
+		switch (i) {
+		case MALI_C55_CAP_DEV_FR:
+			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
+			cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
+			break;
+		case MALI_C55_CAP_DEV_DS:
+			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
+			cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
+			break;
+		default:
+			mutex_destroy(&cap_dev->lock);
+			ret = -EINVAL;
+			goto err_destroy_mutex;
+		}
+
+		cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
+		ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
+		if (ret) {
+			mutex_destroy(&cap_dev->lock);
+			goto err_destroy_mutex;
+		}
+
+		vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+		vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
+		vb2q->drv_priv = cap_dev;
+		vb2q->mem_ops = &vb2_dma_contig_memops;
+		vb2q->ops = &mali_c55_vb2_ops;
+		vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
+		vb2q->min_queued_buffers = 1;
+		vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+		vb2q->lock = &cap_dev->lock;
+		vb2q->dev = mali_c55->dev;
+
+		ret = vb2_queue_init(vb2q);
+		if (ret) {
+			dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
+				mali_c55_cap_dev_to_name(cap_dev));
+			goto err_cleanup_media_entity;
+		}
+
+		strscpy(cap_dev->vdev.name, capture_device_names[i],
+			sizeof(cap_dev->vdev.name));
+		vdev->release = video_device_release_empty;
+		vdev->fops = &mali_c55_v4l2_fops;
+		vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
+		vdev->lock = &cap_dev->lock;
+		vdev->v4l2_dev = &mali_c55->v4l2_dev;
+		vdev->queue = &cap_dev->queue;
+		vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+				    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
+		vdev->entity.ops = &mali_c55_media_ops;
+		video_set_drvdata(vdev, cap_dev);
+
+		memset(&pix_mp, 0, sizeof(pix_mp));
+		pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
+		pix_mp.width = MALI_C55_DEFAULT_WIDTH;
+		pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
+		mali_c55_set_format(cap_dev, &pix_mp);
+
+		ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
+		if (ret) {
+			dev_err(mali_c55->dev,
+				"%s failed to register video device\n",
+				mali_c55_cap_dev_to_name(cap_dev));
+			goto err_release_vb2q;
+		}
+	}
+
+	return 0;
+
+err_release_vb2q:
+	vb2_queue_release(vb2q);
+err_cleanup_media_entity:
+	media_entity_cleanup(&cap_dev->vdev.entity);
+err_destroy_mutex:
+	mutex_destroy(&cap_dev->lock);
+	mali_c55_unregister_capture_devs(mali_c55);
+
+	return ret;
+}
+
+void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
+{
+	struct mali_c55_cap_dev *cap_dev;
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
+		cap_dev = &mali_c55->cap_devs[i];
+
+		if (!video_is_registered(&cap_dev->vdev))
+			continue;
+
+		vb2_video_unregister_device(&cap_dev->vdev);
+		media_entity_cleanup(&cap_dev->vdev.entity);
+		mutex_destroy(&cap_dev->lock);
+	}
+}
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
new file mode 100644
index 000000000000..2d0c4d152beb
--- /dev/null
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
@@ -0,0 +1,266 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * ARM Mali-C55 ISP Driver - Common definitions
+ *
+ * Copyright (C) 2024 Ideas on Board Oy
+ */
+
+#ifndef _MALI_C55_COMMON_H
+#define _MALI_C55_COMMON_H
+
+#include <linux/clk.h>
+#include <linux/io.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/scatterlist.h>
+#include <linux/videodev2.h>
+
+#include <media/media-device.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-dev.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-core.h>
+#include <media/videobuf2-v4l2.h>
+
+#define MALI_C55_DRIVER_NAME		"mali-c55"
+
+/* min and max values for the image sizes */
+#define MALI_C55_MIN_WIDTH		640U
+#define MALI_C55_MIN_HEIGHT		480U
+#define MALI_C55_MAX_WIDTH		8192U
+#define MALI_C55_MAX_HEIGHT		8192U
+#define MALI_C55_DEFAULT_WIDTH		1920U
+#define MALI_C55_DEFAULT_HEIGHT		1080U
+
+#define MALI_C55_DEFAULT_MEDIA_BUS_FMT	MEDIA_BUS_FMT_RGB121212_1X36
+
+struct mali_c55;
+struct mali_c55_cap_dev;
+struct platform_device;
+
+static const char * const mali_c55_clk_names[] = {
+	"aclk",
+	"hclk",
+};
+
+enum mali_c55_interrupts {
+	MALI_C55_IRQ_ISP_START,
+	MALI_C55_IRQ_ISP_DONE,
+	MALI_C55_IRQ_MCM_ERROR,
+	MALI_C55_IRQ_BROKEN_FRAME_ERROR,
+	MALI_C55_IRQ_MET_AF_DONE,
+	MALI_C55_IRQ_MET_AEXP_DONE,
+	MALI_C55_IRQ_MET_AWB_DONE,
+	MALI_C55_IRQ_AEXP_1024_DONE,
+	MALI_C55_IRQ_IRIDIX_MET_DONE,
+	MALI_C55_IRQ_LUT_INIT_DONE,
+	MALI_C55_IRQ_FR_Y_DONE,
+	MALI_C55_IRQ_FR_UV_DONE,
+	MALI_C55_IRQ_DS_Y_DONE,
+	MALI_C55_IRQ_DS_UV_DONE,
+	MALI_C55_IRQ_LINEARIZATION_DONE,
+	MALI_C55_IRQ_RAW_FRONTEND_DONE,
+	MALI_C55_IRQ_NOISE_REDUCTION_DONE,
+	MALI_C55_IRQ_IRIDIX_DONE,
+	MALI_C55_IRQ_BAYER2RGB_DONE,
+	MALI_C55_IRQ_WATCHDOG_TIMER,
+	MALI_C55_IRQ_FRAME_COLLISION,
+	MALI_C55_IRQ_UNUSED,
+	MALI_C55_IRQ_DMA_ERROR,
+	MALI_C55_IRQ_INPUT_STOPPED,
+	MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
+	MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
+	MALI_C55_NUM_IRQ_BITS
+};
+
+enum mali_c55_isp_pads {
+	MALI_C55_ISP_PAD_SINK_VIDEO,
+	MALI_C55_ISP_PAD_SOURCE,
+	MALI_C55_ISP_PAD_SOURCE_BYPASS,
+	MALI_C55_ISP_NUM_PADS,
+};
+
+struct mali_c55_tpg {
+	struct mali_c55 *mali_c55;
+	struct v4l2_subdev sd;
+	struct media_pad pad;
+	struct mutex lock;
+	struct mali_c55_tpg_ctrls {
+		struct v4l2_ctrl_handler handler;
+		struct v4l2_ctrl *test_pattern;
+		struct v4l2_ctrl *hblank;
+		struct v4l2_ctrl *vblank;
+	} ctrls;
+};
+
+struct mali_c55_isp {
+	struct mali_c55 *mali_c55;
+	struct v4l2_subdev sd;
+	struct media_pad pads[MALI_C55_ISP_NUM_PADS];
+	struct media_pad *remote_src;
+	struct v4l2_async_notifier notifier;
+	struct mutex lock;
+	unsigned int frame_sequence;
+};
+
+enum mali_c55_resizer_ids {
+	MALI_C55_RZR_FR,
+	MALI_C55_RZR_DS,
+	MALI_C55_NUM_RZRS,
+};
+
+enum mali_c55_rzr_pads {
+	MALI_C55_RZR_SINK_PAD,
+	MALI_C55_RZR_SOURCE_PAD,
+	MALI_C55_RZR_SINK_BYPASS_PAD,
+	MALI_C55_RZR_NUM_PADS
+};
+
+struct mali_c55_resizer {
+	struct mali_c55 *mali_c55;
+	struct mali_c55_cap_dev *cap_dev;
+	enum mali_c55_resizer_ids id;
+	struct v4l2_subdev sd;
+	struct media_pad pads[MALI_C55_RZR_NUM_PADS];
+	unsigned int num_routes;
+	bool streaming;
+};
+
+enum mali_c55_cap_devs {
+	MALI_C55_CAP_DEV_FR,
+	MALI_C55_CAP_DEV_DS,
+	MALI_C55_NUM_CAP_DEVS
+};
+
+struct mali_c55_fmt {
+	u32 fourcc;
+	unsigned int mbus_codes[2];
+	bool is_raw;
+	struct mali_c55_fmt_registers {
+		unsigned int base_mode;
+		unsigned int uv_plane;
+	} registers;
+};
+
+enum mali_c55_isp_bayer_order {
+	MALI_C55_BAYER_ORDER_RGGB,
+	MALI_C55_BAYER_ORDER_GRBG,
+	MALI_C55_BAYER_ORDER_GBRG,
+	MALI_C55_BAYER_ORDER_BGGR
+};
+
+struct mali_c55_isp_fmt {
+	u32 code;
+	enum v4l2_pixel_encoding encoding;
+	enum mali_c55_isp_bayer_order order;
+};
+
+enum mali_c55_planes {
+	MALI_C55_PLANE_Y,
+	MALI_C55_PLANE_UV,
+	MALI_C55_NUM_PLANES
+};
+
+struct mali_c55_buffer {
+	struct vb2_v4l2_buffer vb;
+	bool plane_done[MALI_C55_NUM_PLANES];
+	struct list_head queue;
+	u32 addrs[MALI_C55_NUM_PLANES];
+};
+
+struct mali_c55_cap_dev {
+	struct mali_c55 *mali_c55;
+	struct mali_c55_resizer *rzr;
+	struct video_device vdev;
+	struct media_pad pad;
+	struct vb2_queue queue;
+	struct mutex lock;
+	unsigned int reg_offset;
+
+	struct mali_c55_mode {
+		const struct mali_c55_fmt *capture_fmt;
+		struct v4l2_pix_format_mplane pix_mp;
+	} mode;
+
+	struct {
+		spinlock_t lock;
+		struct list_head queue;
+		struct mali_c55_buffer *curr;
+		struct mali_c55_buffer *next;
+	} buffers;
+
+	bool streaming;
+};
+
+enum mali_c55_config_spaces {
+	MALI_C55_CONFIG_PING,
+	MALI_C55_CONFIG_PONG,
+	MALI_C55_NUM_CONFIG_SPACES
+};
+
+struct mali_c55_ctx {
+	struct mali_c55 *mali_c55;
+	void *registers;
+	phys_addr_t base;
+	spinlock_t lock;
+	struct list_head list;
+};
+
+struct mali_c55 {
+	struct device *dev;
+	struct resource *res;
+	void __iomem *base;
+	struct dma_chan *channel;
+	struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
+
+	u16 capabilities;
+	struct media_device media_dev;
+	struct v4l2_device v4l2_dev;
+	struct media_pipeline pipe;
+
+	struct mali_c55_tpg tpg;
+	struct mali_c55_isp isp;
+	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
+	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
+
+	struct list_head contexts;
+	enum mali_c55_config_spaces next_config;
+};
+
+void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
+u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
+		  bool force_hardware);
+void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
+			  u32 mask, u32 val);
+int mali_c55_config_write(struct mali_c55_ctx *ctx,
+			  enum mali_c55_config_spaces cfg_space);
+
+int mali_c55_register_isp(struct mali_c55 *mali_c55);
+int mali_c55_register_tpg(struct mali_c55 *mali_c55);
+void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
+void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
+int mali_c55_register_resizers(struct mali_c55 *mali_c55);
+void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
+int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
+void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
+struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
+void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
+			     enum mali_c55_planes plane);
+void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
+void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
+
+bool mali_c55_format_is_raw(unsigned int mbus_code);
+void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
+void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
+int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
+void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
+
+const struct mali_c55_isp_fmt *
+mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
+bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
+#define for_each_mali_isp_fmt(fmt)\
+	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
+
+#endif /* _MALI_C55_COMMON_H */
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
new file mode 100644
index 000000000000..50caf5ee7474
--- /dev/null
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
@@ -0,0 +1,767 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ARM Mali-C55 ISP Driver - Core driver code
+ *
+ * Copyright (C) 2024 Ideas on Board Oy
+ */
+
+#include <linux/bitops.h>
+#include <linux/cleanup.h>
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/dmaengine.h>
+#include <linux/dma-mapping.h>
+#include <linux/interrupt.h>
+#include <linux/iopoll.h>
+#include <linux/ioport.h>
+#include <linux/mod_devicetable.h>
+#include <linux/of.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/scatterlist.h>
+#include <linux/string.h>
+
+#include <media/media-entity.h>
+#include <media/v4l2-device.h>
+#include <media/videobuf2-dma-contig.h>
+
+#include "mali-c55-common.h"
+#include "mali-c55-registers.h"
+
+static const char * const mali_c55_interrupt_names[] = {
+	[MALI_C55_IRQ_ISP_START] = "ISP start",
+	[MALI_C55_IRQ_ISP_DONE] = "ISP done",
+	[MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
+	[MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
+	[MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
+	[MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
+	[MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
+	[MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
+	[MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
+	[MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
+	[MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
+	[MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
+	[MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
+	[MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
+	[MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
+	[MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
+	[MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
+	[MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
+	[MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
+	[MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
+	[MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
+	[MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
+	[MALI_C55_IRQ_DMA_ERROR] = "DMA error",
+	[MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
+	[MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
+	[MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
+};
+
+static unsigned int config_space_addrs[] = {
+	[MALI_C55_CONFIG_PING] = 0x0AB6C,
+	[MALI_C55_CONFIG_PONG] = 0x22B2C,
+};
+
+/* System IO
+ *
+ * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
+ * and 'pong'), with the  expectation that the 'active' space will be left
+ * untouched whilst a frame is being processed and the 'inactive' space
+ * configured ready to be passed during the blanking period before the next
+ * frame processing starts. These spaces should ideally be set via DMA transfer
+ * from a buffer rather than through individual register set operations. There
+ * is also a shared global register space which should be set normally. Of
+ * course, the ISP might be included in a system which lacks a suitable DMA
+ * engine, and the second configuration space might not be fitted at all, which
+ * means we need to support four scenarios:
+ *
+ * 1. Multi config space, with DMA engine.
+ * 2. Multi config space, no DMA engine.
+ * 3. Single config space, with DMA engine.
+ * 4. Single config space, no DMA engine.
+ *
+ * The first case is very easy, but the rest present annoying problems. The best
+ * way to solve them seems to be simply to replicate the concept of DMAing over
+ * the configuration buffer even if there's no DMA engine on the board, for
+ * which we rely on memcpy. To facilitate this any read/write call that is made
+ * to an address within those config spaces should infact be directed to a
+ * buffer that was allocated to hold them rather than the IO memory itself. The
+ * actual copy of that buffer to IO mem will happen on interrupt.
+ */
+
+void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
+{
+	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
+
+	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
+		spin_lock(&ctx->lock);
+		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
+		((u32 *)ctx->registers)[addr] = val;
+		spin_unlock(&ctx->lock);
+
+		return;
+	}
+
+	writel(val, mali_c55->base + addr);
+}
+
+u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
+		  bool force_hardware)
+{
+	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
+	u32 val;
+
+	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
+		spin_lock(&ctx->lock);
+		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
+		val = ((u32 *)ctx->registers)[addr];
+		spin_unlock(&ctx->lock);
+
+		return val;
+	}
+
+	return readl(mali_c55->base + addr);
+}
+
+void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
+			  u32 mask, u32 val)
+{
+	u32 orig, tmp;
+
+	orig = mali_c55_read(mali_c55, addr, false);
+
+	tmp = orig & ~mask;
+	tmp |= (val << (ffs(mask) - 1)) & mask;
+
+	if (tmp != orig)
+		mali_c55_write(mali_c55, addr, tmp);
+}
+
+static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
+			     dma_addr_t dst, enum dma_data_direction dir)
+{
+	struct mali_c55 *mali_c55 = ctx->mali_c55;
+	struct dma_async_tx_descriptor *tx;
+	enum dma_status status;
+	dma_cookie_t cookie;
+
+	tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
+				       MALI_C55_CONFIG_SPACE_SIZE, 0);
+	if (!tx) {
+		dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
+		return -EIO;
+	}
+
+	cookie = dmaengine_submit(tx);
+	if (dma_submit_error(cookie)) {
+		dev_err(mali_c55->dev, "error submitting dma transfer\n");
+		return -EIO;
+	}
+
+	status = dma_sync_wait(mali_c55->channel, cookie);
+	if (status != DMA_COMPLETE) {
+		dev_err(mali_c55->dev, "dma transfer failed\n");
+		return -EIO;
+	}
+
+	return 0;
+}
+
+static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
+			     enum mali_c55_config_spaces cfg_space)
+{
+	struct mali_c55 *mali_c55 = ctx->mali_c55;
+	struct device *dma_dev = mali_c55->channel->device->dev;
+	dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
+	dma_addr_t dst;
+	int ret;
+
+	guard(spinlock)(&ctx->lock);
+
+	dst = dma_map_single(dma_dev, ctx->registers,
+			     MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
+	if (dma_mapping_error(dma_dev, dst)) {
+		dev_err(mali_c55->dev, "failed to map DMA addr\n");
+		return -EIO;
+	}
+
+	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
+	dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
+			 DMA_FROM_DEVICE);
+
+	return ret;
+}
+
+static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
+		       enum mali_c55_config_spaces cfg_space)
+{
+	struct mali_c55 *mali_c55 = ctx->mali_c55;
+	struct device *dma_dev = mali_c55->channel->device->dev;
+	dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
+	dma_addr_t src;
+	int ret;
+
+	guard(spinlock)(&ctx->lock);
+
+	src = dma_map_single(dma_dev, ctx->registers,
+			     MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
+	if (dma_mapping_error(dma_dev, src)) {
+		dev_err(mali_c55->dev, "failed to map DMA addr\n");
+		return -EIO;
+	}
+
+	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
+	dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
+			 DMA_TO_DEVICE);
+
+	return ret;
+}
+
+static int mali_c55_config_read(struct mali_c55_ctx *ctx,
+				enum mali_c55_config_spaces cfg_space)
+{
+	struct mali_c55 *mali_c55 = ctx->mali_c55;
+
+	if (mali_c55->channel) {
+		return mali_c55_dma_read(ctx, cfg_space);
+	} else {
+		memcpy_fromio(ctx->registers,
+			      mali_c55->base + config_space_addrs[cfg_space],
+			      MALI_C55_CONFIG_SPACE_SIZE);
+		return 0;
+	}
+}
+
+int mali_c55_config_write(struct mali_c55_ctx *ctx,
+			  enum mali_c55_config_spaces cfg_space)
+{
+	struct mali_c55 *mali_c55 = ctx->mali_c55;
+
+	if (mali_c55->channel) {
+		return mali_c55_dma_write(ctx, cfg_space);
+	} else {
+		memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
+			    ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
+		return 0;
+	}
+}
+
+struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
+{
+	return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);
+}
+
+static void mali_c55_remove_links(struct mali_c55 *mali_c55)
+{
+	unsigned int i;
+
+	media_entity_remove_links(&mali_c55->tpg.sd.entity);
+	media_entity_remove_links(&mali_c55->isp.sd.entity);
+
+	for (i = 0; i < MALI_C55_NUM_RZRS; i++)
+		media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
+
+	for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
+		media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
+}
+
+static int mali_c55_create_links(struct mali_c55 *mali_c55)
+{
+	int ret;
+
+	/* Test pattern generator to ISP */
+	ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
+				    &mali_c55->isp.sd.entity,
+				    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
+	if (ret) {
+		dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
+		goto err_remove_links;
+	}
+
+	/* Full resolution resizer pipe. */
+	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
+			MALI_C55_ISP_PAD_SOURCE,
+			&mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
+			MALI_C55_RZR_SINK_PAD,
+			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
+	if (ret) {
+		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
+		goto err_remove_links;
+	}
+
+	/* Full resolution bypass. */
+	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
+				    MALI_C55_ISP_PAD_SOURCE_BYPASS,
+				    &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
+				    MALI_C55_RZR_SINK_BYPASS_PAD,
+				    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
+	if (ret) {
+		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
+		goto err_remove_links;
+	}
+
+	/* Resizer pipe to video capture nodes. */
+	ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
+			MALI_C55_RZR_SOURCE_PAD,
+			&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
+			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
+	if (ret) {
+		dev_err(mali_c55->dev,
+			"failed to link FR resizer and video device\n");
+		goto err_remove_links;
+	}
+
+	/* The downscale pipe is an optional hardware block */
+	if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
+		ret = media_create_pad_link(&mali_c55->isp.sd.entity,
+			MALI_C55_ISP_PAD_SOURCE,
+			&mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
+			MALI_C55_RZR_SINK_PAD,
+			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
+		if (ret) {
+			dev_err(mali_c55->dev,
+				"failed to link ISP and DS resizer\n");
+			goto err_remove_links;
+		}
+
+		ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
+			MALI_C55_RZR_SOURCE_PAD,
+			&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
+			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
+		if (ret) {
+			dev_err(mali_c55->dev,
+				"failed to link DS resizer and video device\n");
+			goto err_remove_links;
+		}
+	}
+
+	return 0;
+
+err_remove_links:
+	mali_c55_remove_links(mali_c55);
+	return ret;
+}
+
+static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
+{
+	mali_c55_unregister_tpg(mali_c55);
+	mali_c55_unregister_isp(mali_c55);
+	mali_c55_unregister_resizers(mali_c55);
+	mali_c55_unregister_capture_devs(mali_c55);
+}
+
+static int mali_c55_register_entities(struct mali_c55 *mali_c55)
+{
+	int ret;
+
+	ret = mali_c55_register_tpg(mali_c55);
+	if (ret)
+		return ret;
+
+	ret = mali_c55_register_isp(mali_c55);
+	if (ret)
+		goto err_unregister_entities;
+
+	ret = mali_c55_register_resizers(mali_c55);
+	if (ret)
+		goto err_unregister_entities;
+
+	ret = mali_c55_register_capture_devs(mali_c55);
+	if (ret)
+		goto err_unregister_entities;
+
+	ret = mali_c55_create_links(mali_c55);
+	if (ret)
+		goto err_unregister_entities;
+
+	return 0;
+
+err_unregister_entities:
+	mali_c55_unregister_entities(mali_c55);
+
+	return ret;
+}
+
+static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
+{
+	u32 product, version, revision, capabilities;
+
+	product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
+	version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
+	revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
+
+	dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
+		 product, version, revision);
+
+	capabilities = mali_c55_read(mali_c55,
+				     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
+				     false);
+	mali_c55->capabilities = (capabilities & 0xffff);
+
+	/* TODO: Might as well start some debugfs */
+	dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);
+	return version;
+}
+
+static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
+{
+	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
+	u32 curr_config, next_config;
+
+	curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
+	curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
+		      >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
+	next_config = curr_config ^ 1;
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
+			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
+	mali_c55_config_write(ctx, next_config ?
+			      MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
+}
+
+static irqreturn_t mali_c55_isr(int irq, void *context)
+{
+	struct device *dev = context;
+	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
+	u32 interrupt_status;
+	unsigned int i, j;
+
+	interrupt_status = mali_c55_read(mali_c55,
+					 MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
+					 false);
+	if (!interrupt_status)
+		return IRQ_NONE;
+
+	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
+		       interrupt_status);
+	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
+	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
+
+	for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
+		if (!(interrupt_status & (1 << i)))
+			continue;
+
+		switch (i) {
+		case MALI_C55_IRQ_ISP_START:
+			mali_c55_isp_queue_event_sof(mali_c55);
+
+			for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
+				mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
+
+			mali_c55_swap_next_config(mali_c55);
+
+			break;
+		case MALI_C55_IRQ_ISP_DONE:
+			/*
+			 * TODO: Where the ISP has no Pong config fitted, we'd
+			 * have to do the mali_c55_swap_next_config() call here.
+			 */
+			break;
+		case MALI_C55_IRQ_FR_Y_DONE:
+			mali_c55_set_plane_done(
+				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
+				MALI_C55_PLANE_Y);
+			break;
+		case MALI_C55_IRQ_FR_UV_DONE:
+			mali_c55_set_plane_done(
+				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
+				MALI_C55_PLANE_UV);
+			break;
+		case MALI_C55_IRQ_DS_Y_DONE:
+			mali_c55_set_plane_done(
+				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
+				MALI_C55_PLANE_Y);
+			break;
+		case MALI_C55_IRQ_DS_UV_DONE:
+			mali_c55_set_plane_done(
+				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
+				MALI_C55_PLANE_UV);
+			break;
+		default:
+			/*
+			 * Only the above interrupts are currently unmasked. If
+			 * we receive anything else here then something weird
+			 * has gone on.
+			 */
+			dev_err(dev, "masked interrupt %s triggered\n",
+				mali_c55_interrupt_names[i]);
+		}
+	}
+
+	return IRQ_HANDLED;
+}
+
+static int mali_c55_init_context(struct mali_c55 *mali_c55)
+{
+	struct mali_c55_ctx *ctx;
+	int ret;
+
+	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+	if (!ctx) {
+		dev_err(mali_c55->dev, "failed to allocate new context\n");
+		return -ENOMEM;
+	}
+
+	ctx->base = mali_c55->res->start;
+	ctx->mali_c55 = mali_c55;
+
+	ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
+				 GFP_KERNEL | GFP_DMA);
+	if (!ctx->registers) {
+		ret = -ENOMEM;
+		goto err_free_ctx;
+	}
+
+	/*
+	 * The allocated memory is empty, we need to load the default
+	 * register settings. We just read Ping; it's identical to Pong.
+	 */
+	ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
+	if (ret)
+		goto err_free_registers;
+
+	list_add_tail(&ctx->list, &mali_c55->contexts);
+
+	/*
+	 * Some features of the ISP need to be disabled by default and only
+	 * enabled at the same time as they're configured by a parameters buffer
+	 */
+
+	/* Bypass the sqrt and square compression and expansion modules */
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
+			     MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
+			     MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
+
+	/* Bypass the temper module */
+	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
+		       MALI_C55_REG_BYPASS_2_TEMPER);
+
+	/* Bypass the colour noise reduction  */
+	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
+		       MALI_C55_REG_BYPASS_4_CNR);
+
+	/* Disable the sinter module */
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
+			     MALI_C55_SINTER_ENABLE_MASK, 0x00);
+
+	/* Disable the RGB Gamma module for each output */
+	mali_c55_write(
+		mali_c55,
+		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
+		0x00);
+	mali_c55_write(
+		mali_c55,
+		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
+		0x00);
+
+	/* Disable the colour correction matrix */
+	mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
+
+	return 0;
+
+err_free_registers:
+	kfree(ctx->registers);
+err_free_ctx:
+	kfree(ctx);
+
+	return ret;
+}
+
+static int mali_c55_runtime_resume(struct device *dev)
+{
+	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
+	int ret;
+
+	ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
+				      mali_c55->clks);
+	if (ret)
+		dev_err(mali_c55->dev, "failed to enable clocks\n");
+
+	return ret;
+}
+
+static int mali_c55_runtime_suspend(struct device *dev)
+{
+	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
+
+	clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
+	return 0;
+}
+
+static const struct dev_pm_ops mali_c55_pm_ops = {
+	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
+				pm_runtime_force_resume)
+	SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
+			   NULL)
+};
+
+static int mali_c55_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct mali_c55 *mali_c55;
+	dma_cap_mask_t mask;
+	u32 version;
+	int ret;
+	u32 val;
+
+	mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
+	if (!mali_c55)
+		return dev_err_probe(dev, -ENOMEM,
+				     "failed to allocate memory\n");
+
+	mali_c55->dev = dev;
+	platform_set_drvdata(pdev, mali_c55);
+
+	mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
+								&mali_c55->res);
+	if (IS_ERR(mali_c55->base))
+		return dev_err_probe(dev, PTR_ERR(mali_c55->base),
+				     "failed to map IO memory\n");
+
+	ret = platform_get_irq(pdev, 0);
+	if (ret < 0)
+		return dev_err_probe(dev, ret, "failed to get interrupt num\n");
+
+	ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
+					mali_c55_isr, IRQF_ONESHOT,
+					dev_driver_string(&pdev->dev),
+					&pdev->dev);
+	if (ret)
+		return dev_err_probe(dev, ret, "failed to request irq\n");
+
+	for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
+		mali_c55->clks[i].id = mali_c55_clk_names[i];
+
+	ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
+	if (ret)
+		return dev_err_probe(dev, ret, "failed to acquire clocks\n");
+
+	pm_runtime_enable(&pdev->dev);
+
+	ret = pm_runtime_resume_and_get(&pdev->dev);
+	if (ret)
+		goto err_pm_runtime_disable;
+
+	of_reserved_mem_device_init(dev);
+	version = mali_c55_check_hwcfg(mali_c55);
+	vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
+
+	/* Use "software only" context management. */
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
+			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
+			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
+
+	dma_cap_zero(mask);
+	dma_cap_set(DMA_MEMCPY, mask);
+
+	/*
+	 * No error check, because we will just fallback on memcpy if there is
+	 * no usable DMA channel on the system.
+	 */
+	mali_c55->channel = dma_request_channel(mask, NULL, NULL);
+
+	INIT_LIST_HEAD(&mali_c55->contexts);
+	ret = mali_c55_init_context(mali_c55);
+	if (ret)
+		goto err_release_dma_channel;
+
+	mali_c55->media_dev.dev = dev;
+	strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
+		sizeof(mali_c55->media_dev.model));
+	mali_c55->media_dev.hw_revision = version;
+
+	media_device_init(&mali_c55->media_dev);
+	ret = media_device_register(&mali_c55->media_dev);
+	if (ret)
+		goto err_cleanup_media_device;
+
+	mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
+	ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
+	if (ret) {
+		dev_err(dev, "failed to register V4L2 device\n");
+		goto err_unregister_media_device;
+	};
+
+	ret = mali_c55_register_entities(mali_c55);
+	if (ret) {
+		dev_err(dev, "failed to register entities\n");
+		goto err_unregister_v4l2_device;
+	}
+
+	/* Set safe stop to ensure we're in a non-streaming state */
+	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
+		       MALI_C55_INPUT_SAFE_STOP);
+	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
+			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
+
+	/*
+	 * We're ready to process interrupts. Clear any that are set and then
+	 * unmask them for processing.
+	 */
+	mali_c55_write(mali_c55, 0x30, 0xffffffff);
+	mali_c55_write(mali_c55, 0x34, 0xffffffff);
+	mali_c55_write(mali_c55, 0x40, 0x01);
+	mali_c55_write(mali_c55, 0x40, 0x00);
+	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);
+
+	pm_runtime_put(&pdev->dev);
+
+	return 0;
+
+err_unregister_v4l2_device:
+	v4l2_device_unregister(&mali_c55->v4l2_dev);
+err_unregister_media_device:
+	media_device_unregister(&mali_c55->media_dev);
+err_cleanup_media_device:
+	media_device_cleanup(&mali_c55->media_dev);
+err_release_dma_channel:
+	dma_release_channel(mali_c55->channel);
+err_pm_runtime_disable:
+	pm_runtime_disable(&pdev->dev);
+
+	return ret;
+}
+
+static void mali_c55_remove(struct platform_device *pdev)
+{
+	struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
+	struct mali_c55_ctx *ctx, *tmp;
+
+	list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
+		list_del(&ctx->list);
+		kfree(ctx->registers);
+		kfree(ctx);
+	}
+
+	mali_c55_remove_links(mali_c55);
+	mali_c55_unregister_entities(mali_c55);
+	v4l2_device_put(&mali_c55->v4l2_dev);
+	media_device_unregister(&mali_c55->media_dev);
+	media_device_cleanup(&mali_c55->media_dev);
+	dma_release_channel(mali_c55->channel);
+}
+
+static const struct of_device_id mali_c55_of_match[] = {
+	{ .compatible = "arm,mali-c55", },
+	{},
+};
+MODULE_DEVICE_TABLE(of, mali_c55_of_match);
+
+static struct platform_driver mali_c55_driver = {
+	.driver = {
+		.name = "mali-c55",
+		.of_match_table = of_match_ptr(mali_c55_of_match),
+		.pm = &mali_c55_pm_ops,
+	},
+	.probe = mali_c55_probe,
+	.remove_new = mali_c55_remove,
+};
+
+module_platform_driver(mali_c55_driver);
+MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
+MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
+MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
new file mode 100644
index 000000000000..ea8b7b866e7a
--- /dev/null
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
@@ -0,0 +1,611 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ARM Mali-C55 ISP Driver - Image signal processor
+ *
+ * Copyright (C) 2024 Ideas on Board Oy
+ */
+
+#include <linux/delay.h>
+#include <linux/iopoll.h>
+#include <linux/property.h>
+#include <linux/string.h>
+
+#include <media/media-entity.h>
+#include <media/v4l2-common.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-mc.h>
+#include <media/v4l2-subdev.h>
+
+#include "mali-c55-common.h"
+#include "mali-c55-registers.h"
+
+static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
+	{
+		.code = MEDIA_BUS_FMT_SRGGB20_1X20,
+		.order = MALI_C55_BAYER_ORDER_RGGB,
+		.encoding = V4L2_PIXEL_ENC_BAYER,
+	},
+	{
+		.code = MEDIA_BUS_FMT_SGRBG20_1X20,
+		.order = MALI_C55_BAYER_ORDER_GRBG,
+		.encoding = V4L2_PIXEL_ENC_BAYER,
+	},
+	{
+		.code = MEDIA_BUS_FMT_SGBRG20_1X20,
+		.order = MALI_C55_BAYER_ORDER_GBRG,
+		.encoding = V4L2_PIXEL_ENC_BAYER,
+	},
+	{
+		.code = MEDIA_BUS_FMT_SBGGR20_1X20,
+		.order = MALI_C55_BAYER_ORDER_BGGR,
+		.encoding = V4L2_PIXEL_ENC_BAYER,
+	},
+	{
+		.code = MEDIA_BUS_FMT_RGB202020_1X60,
+		.order = 0, /* Not relevant for this format */
+		.encoding = V4L2_PIXEL_ENC_RGB,
+	}
+	/*
+	 * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
+	 * also support YUV input from a sensor passed-through to the output. At
+	 * present we have no mechanism to test that though so it may have to
+	 * wait a while...
+	 */
+};
+
+const struct mali_c55_isp_fmt *
+mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
+{
+	if (!fmt)
+		fmt = &mali_c55_isp_fmts[0];
+	else
+		fmt++;
+
+	for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
+		return fmt;
+
+	return NULL;
+}
+
+bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
+{
+	const struct mali_c55_isp_fmt *isp_fmt;
+
+	for_each_mali_isp_fmt(isp_fmt) {
+		if (isp_fmt->code == mbus_code)
+			return true;
+	}
+
+	return false;
+}
+
+static const struct mali_c55_isp_fmt *
+mali_c55_isp_get_mbus_config_by_code(u32 code)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
+		if (mali_c55_isp_fmts[i].code == code)
+			return &mali_c55_isp_fmts[i];
+	}
+
+	return NULL;
+}
+
+static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
+{
+	u32 val;
+
+	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);
+	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
+			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
+}
+
+static int mali_c55_isp_start(struct mali_c55 *mali_c55)
+{
+	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
+	const struct mali_c55_isp_fmt *cfg;
+	struct v4l2_mbus_framefmt *format;
+	struct v4l2_subdev_state *state;
+	struct v4l2_rect *crop;
+	struct v4l2_subdev *sd;
+	u32 val;
+	int ret;
+
+	sd = &mali_c55->isp.sd;
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
+			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
+
+	/* Apply input windowing */
+	state = v4l2_subdev_get_locked_active_state(sd);
+	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
+	format = v4l2_subdev_state_get_format(state,
+					      MALI_C55_ISP_PAD_SINK_VIDEO);
+	cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
+
+	mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
+		       MALI_C55_HC_START(crop->left));
+	mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
+		       MALI_C55_HC_SIZE(crop->width));
+	mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
+		       MALI_C55_VC_START(crop->top) |
+		       MALI_C55_VC_SIZE(crop->height));
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
+			     MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
+			     MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
+			     MALI_C55_BAYER_ORDER_MASK, cfg->order);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
+			     MALI_C55_INPUT_WIDTH_MASK,
+			     MALI_C55_INPUT_WIDTH_20BIT);
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
+			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
+			     cfg->encoding == V4L2_PIXEL_ENC_RGB ?
+			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
+
+	ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
+	if (ret) {
+		dev_err(mali_c55->dev, "failed to DMA config\n");
+		return ret;
+	}
+
+	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
+		       MALI_C55_INPUT_SAFE_START);
+	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
+			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
+
+	return 0;
+}
+
+void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)
+{
+	struct mali_c55 *mali_c55 = isp->mali_c55;
+	struct v4l2_subdev *sd;
+
+	if (isp->remote_src) {
+		sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
+		v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
+	}
+	isp->remote_src = NULL;
+
+	mali_c55_isp_stop(mali_c55);
+}
+
+int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
+{
+	struct mali_c55 *mali_c55 = isp->mali_c55;
+	struct media_pad *sink_pad;
+	struct v4l2_subdev *sd;
+	int ret;
+
+	sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
+	isp->remote_src = media_pad_remote_pad_unique(sink_pad);
+	if (IS_ERR(isp->remote_src)) {
+		dev_err(mali_c55->dev, "Failed to get source for ISP\n");
+		return PTR_ERR(isp->remote_src);
+	}
+
+	sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
+
+	isp->frame_sequence = 0;
+	ret = mali_c55_isp_start(mali_c55);
+	if (ret) {
+		dev_err(mali_c55->dev, "Failed to start ISP\n");
+		isp->remote_src = NULL;
+		return ret;
+	}
+
+	/*
+	 * We only support a single input stream, so we can just enable the 1st
+	 * entry in the streams mask.
+	 */
+	ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
+	if (ret) {
+		dev_err(mali_c55->dev, "Failed to start ISP source\n");
+		mali_c55_isp_stop(mali_c55);
+		return ret;
+	}
+
+	return 0;
+}
+
+static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
+				       struct v4l2_subdev_state *state,
+				       struct v4l2_subdev_mbus_code_enum *code)
+{
+	/*
+	 * Only the internal RGB processed format is allowed on the regular
+	 * processing source pad.
+	 */
+	if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
+		if (code->index)
+			return -EINVAL;
+
+		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
+		return 0;
+	}
+
+	/* On the sink and bypass pads all the supported formats are allowed. */
+	if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
+		return -EINVAL;
+
+	code->code = mali_c55_isp_fmts[code->index].code;
+
+	return 0;
+}
+
+static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
+					struct v4l2_subdev_state *state,
+					struct v4l2_subdev_frame_size_enum *fse)
+{
+	const struct mali_c55_isp_fmt *cfg;
+
+	if (fse->index > 0)
+		return -EINVAL;
+
+	/*
+	 * Only the internal RGB processed format is allowed on the regular
+	 * processing source pad.
+	 *
+	 * On the sink and bypass pads all the supported formats are allowed.
+	 */
+	if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
+		if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
+			return -EINVAL;
+	} else {
+		cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
+		if (!cfg)
+			return -EINVAL;
+	}
+
+	fse->min_width = MALI_C55_MIN_WIDTH;
+	fse->min_height = MALI_C55_MIN_HEIGHT;
+	fse->max_width = MALI_C55_MAX_WIDTH;
+	fse->max_height = MALI_C55_MAX_HEIGHT;
+
+	return 0;
+}
+
+static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
+				struct v4l2_subdev_state *state,
+				struct v4l2_subdev_format *format)
+{
+	struct v4l2_mbus_framefmt *fmt = &format->format;
+	struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
+	const struct mali_c55_isp_fmt *cfg;
+	struct v4l2_rect *crop;
+
+	/*
+	 * Disallow set_fmt on the source pads; format is fixed and the sizes
+	 * are the result of applying the sink crop rectangle to the sink
+	 * format.
+	 */
+	if (format->pad)
+		return v4l2_subdev_get_fmt(sd, state, format);
+
+	cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
+	if (!cfg)
+		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
+	fmt->field = V4L2_FIELD_NONE;
+
+	/*
+	 * Clamp sizes in the accepted limits and clamp the crop rectangle in
+	 * the new sizes.
+	 */
+	clamp_t(unsigned int, fmt->width,
+		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
+	clamp_t(unsigned int, fmt->width,
+		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
+
+	sink_fmt = v4l2_subdev_state_get_format(state,
+						MALI_C55_ISP_PAD_SINK_VIDEO);
+	*sink_fmt = *fmt;
+
+	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
+	crop->left = 0;
+	crop->top = 0;
+	crop->width = fmt->width;
+	crop->height = fmt->height;
+
+	/*
+	 * Propagate format to source pads. On the 'regular' output pad use
+	 * the internal RGB processed format, while on the bypass pad simply
+	 * replicate the ISP sink format. The sizes on both pads are the same as
+	 * the ISP sink crop rectangle.
+	 */
+	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
+	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
+	src_fmt->width = crop->width;
+	src_fmt->height = crop->height;
+
+	src_fmt = v4l2_subdev_state_get_format(state,
+					       MALI_C55_ISP_PAD_SOURCE_BYPASS);
+	src_fmt->code = fmt->code;
+	src_fmt->width = crop->width;
+	src_fmt->height = crop->height;
+
+	return 0;
+}
+
+static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
+				      struct v4l2_subdev_state *state,
+				      struct v4l2_subdev_selection *sel)
+{
+	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
+		return -EINVAL;
+
+	sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
+
+	return 0;
+}
+
+static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
+				      struct v4l2_subdev_state *state,
+				      struct v4l2_subdev_selection *sel)
+{
+	struct v4l2_mbus_framefmt *src_fmt;
+	struct v4l2_mbus_framefmt *fmt;
+	struct v4l2_rect *crop;
+
+	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
+		return -EINVAL;
+
+	fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
+
+	clamp_t(unsigned int, sel->r.left, 0, fmt->width);
+	clamp_t(unsigned int, sel->r.top, 0, fmt->height);
+	clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
+		fmt->width - sel->r.left);
+	clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
+		fmt->height - sel->r.top);
+
+	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
+	*crop = sel->r;
+
+	/* Propagate the crop rectangle sizes to the source pad format. */
+	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
+	src_fmt->width = crop->width;
+	src_fmt->height = crop->height;
+
+	return 0;
+}
+
+static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
+	.enum_mbus_code		= mali_c55_isp_enum_mbus_code,
+	.enum_frame_size	= mali_c55_isp_enum_frame_size,
+	.get_fmt		= v4l2_subdev_get_fmt,
+	.set_fmt		= mali_c55_isp_set_fmt,
+	.get_selection		= mali_c55_isp_get_selection,
+	.set_selection		= mali_c55_isp_set_selection,
+	.link_validate		= v4l2_subdev_link_validate_default,
+};
+
+void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
+{
+	struct v4l2_event event = {
+		.type = V4L2_EVENT_FRAME_SYNC,
+	};
+
+	event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
+	v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
+}
+
+static int
+mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
+			     struct v4l2_event_subscription *sub)
+{
+	if (sub->type != V4L2_EVENT_FRAME_SYNC)
+		return -EINVAL;
+
+	/* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
+	if (sub->id != 0)
+		return -EINVAL;
+
+	return v4l2_event_subscribe(fh, sub, 0, NULL);
+}
+
+static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
+	.subscribe_event = mali_c55_isp_subscribe_event,
+	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
+};
+
+static const struct v4l2_subdev_ops mali_c55_isp_ops = {
+	.pad	= &mali_c55_isp_pad_ops,
+	.core	= &mali_c55_isp_core_ops,
+};
+
+static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
+				   struct v4l2_subdev_state *sd_state)
+{
+	struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
+	struct v4l2_rect *in_crop;
+
+	sink_fmt = v4l2_subdev_state_get_format(sd_state,
+						MALI_C55_ISP_PAD_SINK_VIDEO);
+	src_fmt = v4l2_subdev_state_get_format(sd_state,
+					       MALI_C55_ISP_PAD_SOURCE);
+	in_crop = v4l2_subdev_state_get_crop(sd_state,
+					     MALI_C55_ISP_PAD_SINK_VIDEO);
+
+	sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
+	sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
+	sink_fmt->field = V4L2_FIELD_NONE;
+	sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
+
+	*v4l2_subdev_state_get_format(sd_state,
+			      MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
+
+	src_fmt->width = MALI_C55_DEFAULT_WIDTH;
+	src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
+	src_fmt->field = V4L2_FIELD_NONE;
+	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
+
+	in_crop->top = 0;
+	in_crop->left = 0;
+	in_crop->width = MALI_C55_DEFAULT_WIDTH;
+	in_crop->height = MALI_C55_DEFAULT_HEIGHT;
+
+	return 0;
+}
+
+static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
+	.init_state = mali_c55_isp_init_state,
+};
+
+static const struct media_entity_operations mali_c55_isp_media_ops = {
+	.link_validate		= v4l2_subdev_link_validate,
+};
+
+static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
+				       struct v4l2_subdev *subdev,
+				       struct v4l2_async_connection *asc)
+{
+	struct mali_c55_isp *isp = container_of(notifier,
+						struct mali_c55_isp, notifier);
+	struct mali_c55 *mali_c55 = isp->mali_c55;
+	struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
+	int ret;
+
+	/*
+	 * By default we'll flag this link enabled and the TPG disabled, but
+	 * no immutable flag because we need to be able to switch between the
+	 * two.
+	 */
+	ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
+					      MEDIA_LNK_FL_ENABLED);
+	if (ret)
+		dev_err(mali_c55->dev, "failed to create link for %s\n",
+			subdev->name);
+
+	return ret;
+}
+
+static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
+{
+	struct mali_c55_isp *isp = container_of(notifier,
+						struct mali_c55_isp, notifier);
+	struct mali_c55 *mali_c55 = isp->mali_c55;
+
+	return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
+}
+
+static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
+	.bound = mali_c55_isp_notifier_bound,
+	.complete = mali_c55_isp_notifier_complete,
+};
+
+static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
+{
+	struct mali_c55 *mali_c55 = isp->mali_c55;
+	struct v4l2_async_connection *asc;
+	struct fwnode_handle *ep;
+	int ret;
+
+	v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
+
+	/*
+	 * The ISP should have a single endpoint pointing to some flavour of
+	 * CSI-2 receiver...but for now at least we do want everything to work
+	 * normally even with no sensors connected, as we have the TPG. If we
+	 * don't find a sensor just warn and return success.
+	 */
+	ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
+					     0, 0, 0);
+	if (!ep) {
+		dev_warn(mali_c55->dev, "no local endpoint found\n");
+		return 0;
+	}
+
+	asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
+					      struct v4l2_async_connection);
+	if (IS_ERR(asc)) {
+		dev_err(mali_c55->dev, "failed to add remote fwnode\n");
+		ret = PTR_ERR(asc);
+		goto err_put_ep;
+	}
+
+	isp->notifier.ops = &mali_c55_isp_notifier_ops;
+	ret = v4l2_async_nf_register(&isp->notifier);
+	if (ret) {
+		dev_err(mali_c55->dev, "failed to register notifier\n");
+		goto err_cleanup_nf;
+	}
+
+	fwnode_handle_put(ep);
+
+	return 0;
+
+err_cleanup_nf:
+	v4l2_async_nf_cleanup(&isp->notifier);
+err_put_ep:
+	fwnode_handle_put(ep);
+
+	return ret;
+}
+
+int mali_c55_register_isp(struct mali_c55 *mali_c55)
+{
+	struct mali_c55_isp *isp = &mali_c55->isp;
+	struct v4l2_subdev *sd = &isp->sd;
+	int ret;
+
+	isp->mali_c55 = mali_c55;
+
+	v4l2_subdev_init(sd, &mali_c55_isp_ops);
+	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+	sd->entity.ops = &mali_c55_isp_media_ops;
+	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
+	sd->internal_ops = &mali_c55_isp_internal_ops;
+	strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
+
+	isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
+	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
+	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
+
+	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
+				     isp->pads);
+	if (ret)
+		return ret;
+
+	ret = v4l2_subdev_init_finalize(sd);
+	if (ret)
+		goto err_cleanup_media_entity;
+
+	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
+	if (ret)
+		goto err_cleanup_subdev;
+
+	ret = mali_c55_isp_parse_endpoint(isp);
+	if (ret)
+		goto err_cleanup_subdev;
+
+	mutex_init(&isp->lock);
+
+	return 0;
+
+err_cleanup_subdev:
+	v4l2_subdev_cleanup(sd);
+err_cleanup_media_entity:
+	media_entity_cleanup(&sd->entity);
+	isp->mali_c55 = NULL;
+
+	return ret;
+}
+
+void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
+{
+	struct mali_c55_isp *isp = &mali_c55->isp;
+
+	if (!isp->mali_c55)
+		return;
+
+	mutex_destroy(&isp->lock);
+	v4l2_async_nf_unregister(&isp->notifier);
+	v4l2_async_nf_cleanup(&isp->notifier);
+	v4l2_device_unregister_subdev(&isp->sd);
+	v4l2_subdev_cleanup(&isp->sd);
+	media_entity_cleanup(&isp->sd.entity);
+}
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
new file mode 100644
index 000000000000..cb27abde2aa5
--- /dev/null
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
@@ -0,0 +1,258 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * ARM Mali-C55 ISP Driver - Register definitions
+ *
+ * Copyright (C) 2024 Ideas on Board Oy
+ */
+
+#ifndef _MALI_C55_REGISTERS_H
+#define _MALI_C55_REGISTERS_H
+
+#include <linux/bits.h>
+
+/* ISP Common 0x00000 - 0x000ff */
+
+#define MALI_C55_REG_API				0x00000
+#define MALI_C55_REG_PRODUCT				0x00004
+#define MALI_C55_REG_VERSION				0x00008
+#define MALI_C55_REG_REVISION				0x0000c
+#define MALI_C55_REG_PULSE_MODE				0x0003c
+#define MALI_C55_REG_INPUT_MODE_REQUEST			0x0009c
+#define MALI_C55_INPUT_SAFE_STOP			0x00
+#define MALI_C55_INPUT_SAFE_START			0x01
+#define MALI_C55_REG_MODE_STATUS			0x000a0
+#define MALI_C55_REG_INTERRUPT_MASK_VECTOR		0x00030
+#define MALI_C55_INTERRUPT_MASK_ALL			GENMASK(31, 0)
+
+#define MALI_C55_REG_GLOBAL_MONITOR			0x00050
+
+#define MALI_C55_REG_GEN_VIDEO				0x00080
+#define MALI_C55_REG_GEN_VIDEO_ON_MASK			BIT(0)
+#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK		BIT(1)
+#define MALI_C55_REG_GEN_PREFETCH_MASK			GENMASK(31, 16)
+
+#define MALI_C55_REG_MCU_CONFIG				0x00020
+#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK		BIT(0)
+#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK		BIT(1)
+#define MALI_C55_REG_MCU_CONFIG_WRITE_PING		BIT(1)
+#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG		0x00
+#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK		BIT(8)
+#define MALI_C55_REG_PING_PONG_READ			0x00024
+#define MALI_C55_REG_PING_PONG_READ_MASK		BIT(2)
+
+#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR		0x00034
+#define MALI_C55_REG_INTERRUPT_CLEAR			0x00040
+#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR		0x00044
+#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS		0x00068
+#define MALI_C55_GPS_PONG_FITTED			BIT(0)
+#define MALI_C55_GPS_WDR_FITTED				BIT(1)
+#define MALI_C55_GPS_COMPRESSION_FITTED			BIT(2)
+#define MALI_C55_GPS_TEMPER_FITTED			BIT(3)
+#define MALI_C55_GPS_SINTER_LITE_FITTED			BIT(4)
+#define MALI_C55_GPS_SINTER_FITTED			BIT(5)
+#define MALI_C55_GPS_IRIDIX_LTM_FITTED			BIT(6)
+#define MALI_C55_GPS_IRIDIX_GTM_FITTED			BIT(7)
+#define MALI_C55_GPS_CNR_FITTED				BIT(8)
+#define MALI_C55_GPS_FRSCALER_FITTED			BIT(9)
+#define MALI_C55_GPS_DS_PIPE_FITTED			BIT(10)
+
+#define MALI_C55_REG_BLANKING				0x00084
+#define MALI_C55_REG_HBLANK_MASK			GENMASK(15, 0)
+#define MALI_C55_REG_VBLANK_MASK			GENMASK(31, 16)
+
+#define MALI_C55_REG_HC_START				0x00088
+#define MALI_C55_HC_START(h)				(((h) & 0xffff) << 16)
+#define MALI_C55_REG_HC_SIZE				0x0008c
+#define MALI_C55_HC_SIZE(h)				((h) & 0xffff)
+#define MALI_C55_REG_VC_START_SIZE			0x00094
+#define MALI_C55_VC_START(v)				((v) & 0xffff)
+#define MALI_C55_VC_SIZE(v)				(((v) & 0xffff) << 16)
+
+/* Ping/Pong Configuration Space */
+#define MALI_C55_REG_BASE_ADDR				0x18e88
+#define MALI_C55_REG_BYPASS_0				0x18eac
+#define MALI_C55_REG_BYPASS_0_VIDEO_TEST		BIT(0)
+#define MALI_C55_REG_BYPASS_0_INPUT_FMT			BIT(1)
+#define MALI_C55_REG_BYPASS_0_DECOMPANDER		BIT(2)
+#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR		BIT(3)
+#define MALI_C55_REG_BYPASS_0_GAIN_WDR			BIT(4)
+#define MALI_C55_REG_BYPASS_0_FRAME_STITCH		BIT(5)
+#define MALI_C55_REG_BYPASS_1				0x18eb0
+#define MALI_C55_REG_BYPASS_1_DIGI_GAIN			BIT(0)
+#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS		BIT(1)
+#define MALI_C55_REG_BYPASS_1_FE_SQRT			BIT(2)
+#define MALI_C55_REG_BYPASS_1_RAW_FE			BIT(3)
+#define MALI_C55_REG_BYPASS_2				0x18eb8
+#define MALI_C55_REG_BYPASS_2_SINTER			BIT(0)
+#define MALI_C55_REG_BYPASS_2_TEMPER			BIT(1)
+#define MALI_C55_REG_BYPASS_3				0x18ebc
+#define MALI_C55_REG_BYPASS_3_SQUARE_BE			BIT(0)
+#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH	BIT(1)
+#define MALI_C55_REG_BYPASS_3_MESH_SHADING		BIT(3)
+#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE		BIT(4)
+#define MALI_C55_REG_BYPASS_3_IRIDIX			BIT(5)
+#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN		BIT(6)
+#define MALI_C55_REG_BYPASS_4				0x18ec0
+#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB		BIT(1)
+#define MALI_C55_REG_BYPASS_4_PF_CORRECTION		BIT(3)
+#define MALI_C55_REG_BYPASS_4_CCM			BIT(4)
+#define MALI_C55_REG_BYPASS_4_CNR			BIT(5)
+#define MALI_C55_REG_FR_BYPASS				0x18ec4
+#define MALI_C55_REG_DS_BYPASS				0x18ec8
+#define MALI_C55_BYPASS_CROP				BIT(0)
+#define MALI_C55_BYPASS_SCALER				BIT(1)
+#define MALI_C55_BYPASS_GAMMA_RGB			BIT(2)
+#define MALI_C55_BYPASS_SHARPEN				BIT(3)
+#define MALI_C55_BYPASS_CS_CONV				BIT(4)
+#define MALI_C55_REG_ISP_RAW_BYPASS			0x18ecc
+#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK		BIT(0)
+#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK		GENMASK(9, 8)
+#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS		2
+#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS		1
+#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE		BIT(1)
+#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS		BIT(0)
+
+#define MALI_C55_REG_ACTIVE_WIDTH_MASK			0xffff
+#define MALI_C55_REG_ACTIVE_HEIGHT_MASK			0xffff0000
+#define MALI_C55_REG_BAYER_ORDER			0x18e8c
+#define MALI_C55_BAYER_ORDER_MASK			GENMASK(1, 0)
+#define MALI_C55_REG_TPG_CH0				0x18ed8
+#define MALI_C55_TEST_PATTERN_ON_OFF			BIT(0)
+#define MALI_C55_TEST_PATTERN_RGB_MASK			BIT(1)
+#define MALI_C55_REG_TPG_R_BACKGROUND			0x18ee0
+#define MALI_C55_REG_TPG_G_BACKGROUND			0x18ee4
+#define MALI_C55_REG_TPG_B_BACKGROUND			0x18ee8
+#define MALI_C55_TPG_BACKGROUND_MAX			0xfffff
+#define MALI_C55_REG_INPUT_WIDTH			0x18f98
+#define MALI_C55_INPUT_WIDTH_MASK			GENMASK(18, 16)
+#define MALI_C55_INPUT_WIDTH_8BIT			0
+#define MALI_C55_INPUT_WIDTH_10BIT			1
+#define MALI_C55_INPUT_WIDTH_12BIT			2
+#define MALI_C55_INPUT_WIDTH_14BIT			3
+#define MALI_C55_INPUT_WIDTH_16BIT			4
+#define MALI_C55_INPUT_WIDTH_20BIT			5
+#define MALI_C55_REG_SPACE_SIZE				0x4000
+#define MALI_C55_REG_CONFIG_SPACES_OFFSET		0x0ab6c
+#define MALI_C55_CONFIG_SPACE_SIZE			0x1231c
+
+#define MALI_C55_REG_SINTER_CONFIG			0x19348
+#define MALI_C55_SINTER_VIEW_FILTER_MASK		GENMASK(1, 0)
+#define MALI_C55_SINTER_SCALE_MODE_MASK			GENMASK(3, 2)
+#define MALI_C55_SINTER_ENABLE_MASK			BIT(4)
+#define MALI_C55_SINTER_FILTER_SELECT_MASK		BIT(5)
+#define MALI_C55_SINTER_INT_SELECT_MASK			BIT(6)
+#define MALI_C55_SINTER_RM_ENABLE_MASK			BIT(7)
+
+/* Colour Correction Matrix Configuration */
+#define MALI_C55_REG_CCM_ENABLE				0x1b07c
+#define MALI_C55_CCM_ENABLE_MASK			BIT(0)
+#define MALI_C55_REG_CCM_COEF_R_R			0x1b080
+#define MALI_C55_REG_CCM_COEF_R_G			0x1b084
+#define MALI_C55_REG_CCM_COEF_R_B			0x1b088
+#define MALI_C55_REG_CCM_COEF_G_R			0x1b090
+#define MALI_C55_REG_CCM_COEF_G_G			0x1b094
+#define MALI_C55_REG_CCM_COEF_G_B			0x1b098
+#define MALI_C55_REG_CCM_COEF_B_R			0x1b0a0
+#define MALI_C55_REG_CCM_COEF_B_G			0x1b0a4
+#define MALI_C55_REG_CCM_COEF_B_B			0x1b0a8
+#define MALI_C55_CCM_COEF_MASK				GENMASK(12, 0)
+#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R			0x1b0b0
+#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G			0x1b0b4
+#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B			0x1b0b8
+#define MALI_C55_CCM_ANTIFOG_GAIN_MASK			GENMASK(11, 0)
+#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R		0x1b0c0
+#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G		0x1b0c4
+#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B		0x1b0c8
+#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK		GENMASK(11, 0)
+
+/*
+ * The Mali-C55 ISP has up to two output pipes; known as full resolution and
+ * down scaled. The register space for these is laid out identically, but offset
+ * by 372 bytes.
+ */
+#define MALI_C55_CAP_DEV_FR_REG_OFFSET		0x0
+#define MALI_C55_CAP_DEV_DS_REG_OFFSET		0x174
+
+#define MALI_C55_REG_CS_CONV_CONFIG(offset)		(0x1c098 + (offset))
+#define MALI_C55_CS_CONV_MATRIX_MASK			BIT(0)
+#define MALI_C55_CS_CONV_FILTER_MASK			BIT(1)
+#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK		BIT(2)
+#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK		BIT(3)
+#define MALI_C55_REG_Y_WRITER_MODE(offset)		(0x1c0ec + (offset))
+#define MALI_C55_REG_UV_WRITER_MODE(offset)		(0x1c144 + (offset))
+#define MALI_C55_WRITER_MODE_MASK			GENMASK(4, 0)
+#define MALI_C55_WRITER_SUBMODE_MASK			GENMASK(7, 6)
+#define MALI_C55_WRITER_FRAME_WRITE_MASK		BIT(9)
+#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset)		(0x1c0f0 + (offset))
+#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset)		(0x1c148 + (offset))
+#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)		((w) << 0)
+#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)		((h) << 16)
+#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset)	(0x1c0f4 + (offset))
+#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset)	(0x1c108 + (offset))
+#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
+#define MALI_C55_REG_Y_WRITER_BANKS_RESTART		BIT(3)
+#define MALI_C55_REG_Y_WRITER_OFFSET(offset)		(0x1c10c + (offset))
+#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset)	(0x1c14c + (offset))
+#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset)	(0x1c160 + (offset))
+#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
+#define MALI_C55_REG_UV_WRITER_BANKS_RESTART		BIT(3)
+#define MALI_C55_REG_UV_WRITER_OFFSET(offset)		(0x1c164 + (offset))
+
+#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
+#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE		0x18edc
+
+#define MALI_C55_REG_CROP_EN(offset)			(0x1c028 + (offset))
+#define MALI_C55_CROP_ENABLE				BIT(0)
+#define MALI_C55_REG_CROP_X_START(offset)		(0x1c02c + (offset))
+#define MALI_C55_REG_CROP_Y_START(offset)		(0x1c030 + (offset))
+#define MALI_C55_REG_CROP_X_SIZE(offset)		(0x1c034 + (offset))
+#define MALI_C55_REG_CROP_Y_SIZE(offset)		(0x1c038 + (offset))
+#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset)		(0x1c040 + (offset))
+#define MALI_C55_SCALER_TIMEOUT_EN			BIT(4)
+#define MALI_C55_SCALER_TIMEOUT(t)			((t) << 16)
+#define MALI_C55_REG_SCALER_IN_WIDTH(offset)		(0x1c044 + (offset))
+#define MALI_C55_REG_SCALER_IN_HEIGHT(offset)		(0x1c048 + (offset))
+#define MALI_C55_REG_SCALER_OUT_WIDTH(offset)		(0x1c04c + (offset))
+#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset)		(0x1c050 + (offset))
+#define MALI_C55_REG_SCALER_HFILT_TINC(offset)		(0x1c054 + (offset))
+#define MALI_C55_REG_SCALER_HFILT_COEF(offset)		(0x1c058 + (offset))
+#define MALI_C55_REG_SCALER_VFILT_TINC(offset)		(0x1c05c + (offset))
+#define MALI_C55_REG_SCALER_VFILT_COEF(offset)		(0x1c060 + (offset))
+
+#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset)		(0x1c064 + (offset))
+#define MALI_C55_GAMMA_ENABLE_MASK			BIT(0)
+#define MALI_C55_REG_GAMMA_GAINS_1(offset)		(0x1c068 + (offset))
+#define MALI_C55_GAMMA_GAIN_R_MASK			GENMASK(11, 0)
+#define MALI_C55_GAMMA_GAIN_G_MASK			GENMASK(27, 16)
+#define MALI_C55_REG_GAMMA_GAINS_2(offset)		(0x1c06c + (offset))
+#define MALI_C55_GAMMA_GAIN_B_MASK			GENMASK(11, 0)
+#define MALI_C55_REG_GAMMA_OFFSETS_1(offset)		(0x1c070 + (offset))
+#define MALI_C55_GAMMA_OFFSET_R_MASK			GENMASK(11, 0)
+#define MALI_C55_GAMMA_OFFSET_G_MASK			GENMASK(27, 16)
+#define MALI_C55_REG_GAMMA_OFFSETS_2(offset)		(0x1c074 + (offset))
+#define MALI_C55_GAMMA_OFFSET_B_MASK			GENMASK(11, 0)
+
+/* Output DMA Writer */
+
+#define MALI_C55_OUTPUT_DISABLED		0
+#define MALI_C55_OUTPUT_RGB32			1
+#define MALI_C55_OUTPUT_A2R10G10B10		2
+#define MALI_C55_OUTPUT_RGB565			3
+#define MALI_C55_OUTPUT_RGB24			4
+#define MALI_C55_OUTPUT_GEN32			5
+#define MALI_C55_OUTPUT_RAW16			6
+#define MALI_C55_OUTPUT_AYUV			8
+#define MALI_C55_OUTPUT_Y410			9
+#define MALI_C55_OUTPUT_YUY2			10
+#define MALI_C55_OUTPUT_UYVY			11
+#define MALI_C55_OUTPUT_Y210			12
+#define MALI_C55_OUTPUT_NV12_21			13
+#define MALI_C55_OUTPUT_YUV_420_422		17
+#define MALI_C55_OUTPUT_P210_P010		19
+#define MALI_C55_OUTPUT_YUV422			20
+
+#define MALI_C55_OUTPUT_PLANE_ALT0		0
+#define MALI_C55_OUTPUT_PLANE_ALT1		1
+#define MALI_C55_OUTPUT_PLANE_ALT2		2
+
+#endif /* _MALI_C55_REGISTERS_H */
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
new file mode 100644
index 000000000000..8edae87f1e5f
--- /dev/null
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
@@ -0,0 +1,382 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * ARM Mali-C55 ISP Driver - Resizer Coefficients
+ *
+ * Copyright (C) 2024 Ideas on Board Oy
+ */
+
+#ifndef _MALI_C55_RESIZER_COEFS_H
+#define _MALI_C55_RESIZER_COEFS_H
+
+#include "mali-c55-common.h"
+
+#define MALI_C55_RESIZER_COEFS_NUM_BANKS	8
+#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES	64
+
+static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
+	{	/* Bank 0 */
+		0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
+		0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
+		0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
+		0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
+		0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
+		0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
+		0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
+		0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
+		0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
+		0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
+		0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
+		0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
+		0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
+		0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
+		0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
+		0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
+	},
+	{	/* Bank 1 */
+		0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
+		0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
+		0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
+		0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
+		0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
+		0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
+		0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
+		0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
+		0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
+		0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
+		0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
+		0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
+		0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
+		0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
+		0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
+		0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
+	},
+	{	/* Bank 2 */
+		0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
+		0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
+		0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
+		0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
+		0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
+		0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
+		0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
+		0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
+		0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
+		0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
+		0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
+		0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
+		0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
+		0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
+		0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
+		0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
+	},
+	{	/* Bank 3 */
+		0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
+		0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
+		0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
+		0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
+		0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
+		0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
+		0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
+		0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
+		0x20100000, 0x00000010, 0x1f110000, 0x00000010,
+		0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
+		0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
+		0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
+		0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
+		0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
+		0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
+		0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
+	},
+	{	/* Bank 4 */
+		0x17090000, 0x00000917, 0x18090000, 0x00000916,
+		0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
+		0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
+		0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
+		0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
+		0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
+		0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
+		0x190f0300, 0x00000411, 0x18100300, 0x00000411,
+		0x1a100300, 0x00000310, 0x18110400, 0x00000310,
+		0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
+		0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
+		0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
+		0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
+		0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
+		0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
+		0x17160800, 0x0000010a, 0x18160900, 0x00000009,
+	},
+	{	/* Bank 5 */
+		0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
+		0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
+		0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
+		0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
+		0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
+		0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
+		0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
+		0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
+		0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
+		0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
+		0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
+		0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
+		0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
+		0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
+		0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
+		0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
+	},
+	{	/* Bank 6 */
+		0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
+		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
+		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
+		0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
+		0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
+		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
+		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
+		0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
+		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
+		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
+		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
+		0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
+		0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
+		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
+		0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
+		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
+	},
+	{	/* Bank 7 */
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
+	}
+};
+
+static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
+	{	/* Bank 0 */
+		0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
+		0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
+		0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
+		0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
+		0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
+		0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
+		0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
+		0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
+		0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
+		0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
+		0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
+		0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
+		0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
+		0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
+		0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
+		0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
+	},
+	{	/* Bank 1 */
+		0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
+		0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
+		0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
+		0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
+		0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
+		0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
+		0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
+		0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
+		0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
+		0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
+		0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
+		0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
+		0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
+		0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
+		0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
+		0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
+	},
+	{	/* Bank 2 */
+		0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
+		0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
+		0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
+		0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
+		0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
+		0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
+		0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
+		0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
+		0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
+		0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
+		0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
+		0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
+		0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
+		0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
+		0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
+		0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
+	},
+	{	/* Bank 3 */
+		0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
+		0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
+		0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
+		0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
+		0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
+		0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
+		0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
+		0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
+		0x20100000, 0x00000010, 0x1f100000, 0x00000011,
+		0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
+		0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
+		0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
+		0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
+		0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
+		0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
+		0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
+	},
+	{	/* Bank 4 */
+		0x17170900, 0x00000009, 0x18160900, 0x00000009,
+		0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
+		0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
+		0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
+		0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
+		0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
+		0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
+		0x19110400, 0x0000030f, 0x18110400, 0x00000310,
+		0x1a100300, 0x00000310, 0x18100300, 0x00000411,
+		0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
+		0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
+		0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
+		0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
+		0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
+		0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
+		0x170a0100, 0x00000816, 0x18090000, 0x00000916,
+	},
+	{	/* Bank 5 */
+		0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
+		0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
+		0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
+		0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
+		0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
+		0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
+		0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
+		0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
+		0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
+		0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
+		0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
+		0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
+		0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
+		0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
+		0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
+		0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
+	},
+	{	/* Bank 6 */
+		0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
+		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
+		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
+		0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
+		0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
+		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
+		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
+		0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
+		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
+		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
+		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
+		0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
+		0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
+		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
+		0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
+		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
+	},
+	{	/* Bank 7 */
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
+	}
+};
+
+struct mali_c55_resizer_coef_bank {
+	unsigned int bank;
+	unsigned int top;
+	unsigned int bottom;
+};
+
+static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
+	{
+		.bank = 0,
+		.top = 1000,
+		.bottom = 770,
+	},
+	{
+		.bank = 1,
+		.top = 769,
+		.bottom = 600,
+	},
+	{
+		.bank = 2,
+		.top = 599,
+		.bottom = 460,
+	},
+	{
+		.bank = 3,
+		.top = 459,
+		.bottom = 354,
+	},
+	{
+		.bank = 4,
+		.top = 353,
+		.bottom = 273,
+	},
+	{
+		.bank = 5,
+		.top = 272,
+		.bottom = 210,
+	},
+	{
+		.bank = 6,
+		.top = 209,
+		.bottom = 162,
+	},
+	{
+		.bank = 7,
+		.top = 161,
+		.bottom = 125,
+	},
+};
+
+static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,
+						unsigned int crop,
+						unsigned int scale)
+{
+	unsigned int tmp;
+	unsigned int i;
+
+	tmp = (scale * 1000U) / crop;
+
+	for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
+		if (tmp >= mali_c55_coefficient_banks[i].bottom &&
+		    tmp <= mali_c55_coefficient_banks[i].top)
+			return mali_c55_coefficient_banks[i].bank;
+	}
+
+	/*
+	 * We shouldn't ever get here, in theory. As we have no good choices
+	 * simply warn the user and use the first bank of coefficients.
+	 */
+	dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
+	return 0;
+}
+
+#endif /* _MALI_C55_RESIZER_COEFS_H */
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
new file mode 100644
index 000000000000..0a5a2969d3ce
--- /dev/null
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
@@ -0,0 +1,779 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ARM Mali-C55 ISP Driver - Image signal processor
+ *
+ * Copyright (C) 2024 Ideas on Board Oy
+ */
+
+#include <linux/math.h>
+#include <linux/minmax.h>
+
+#include <media/media-entity.h>
+#include <media/v4l2-subdev.h>
+
+#include "mali-c55-common.h"
+#include "mali-c55-registers.h"
+#include "mali-c55-resizer-coefs.h"
+
+/* Scaling factor in Q4.20 format. */
+#define MALI_C55_RZR_SCALER_FACTOR	(1U << 20)
+
+static const u32 rzr_non_bypass_src_fmts[] = {
+	MEDIA_BUS_FMT_RGB121212_1X36,
+	MEDIA_BUS_FMT_YUV10_1X30
+};
+
+static const char * const mali_c55_resizer_names[] = {
+	[MALI_C55_RZR_FR] = "resizer fr",
+	[MALI_C55_RZR_DS] = "resizer ds",
+};
+
+static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
+				     struct v4l2_subdev_state *state)
+{
+	unsigned int reg_offset = rzr->cap_dev->reg_offset;
+	struct mali_c55 *mali_c55 = rzr->mali_c55;
+	struct v4l2_mbus_framefmt *fmt;
+	struct v4l2_rect *crop;
+
+	/* Verify if crop should be enabled. */
+	fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
+	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
+
+	if (fmt->width == crop->width && fmt->height == crop->height)
+		return MALI_C55_BYPASS_CROP;
+
+	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
+		       crop->left);
+	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
+		       crop->top);
+	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
+		       crop->width);
+	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
+		       crop->height);
+
+	mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
+		       MALI_C55_CROP_ENABLE);
+
+	return 0;
+}
+
+static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
+					struct v4l2_subdev_state *state)
+{
+	unsigned int reg_offset = rzr->cap_dev->reg_offset;
+	struct mali_c55 *mali_c55 = rzr->mali_c55;
+	struct v4l2_rect *crop, *scale;
+	unsigned int h_bank, v_bank;
+	u64 h_scale, v_scale;
+
+	/* Verify if scaling should be enabled. */
+	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
+	scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
+
+	if (crop->width == scale->width && crop->height == scale->height)
+		return MALI_C55_BYPASS_SCALER;
+
+	/* Program the V/H scaling factor in Q4.20 format. */
+	h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
+	v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
+
+	do_div(h_scale, scale->width);
+	do_div(v_scale, scale->height);
+
+	mali_c55_write(mali_c55,
+		       MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
+		       crop->width);
+	mali_c55_write(mali_c55,
+		       MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
+		       crop->height);
+
+	mali_c55_write(mali_c55,
+		       MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
+		       scale->width);
+	mali_c55_write(mali_c55,
+		       MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
+		       scale->height);
+
+	mali_c55_write(mali_c55,
+		       MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
+		       h_scale);
+	mali_c55_write(mali_c55,
+		       MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
+		       v_scale);
+
+	h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
+					     scale->width);
+	mali_c55_write(mali_c55,
+		       MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
+		       h_bank);
+
+	v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
+					     scale->height);
+	mali_c55_write(mali_c55,
+		       MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
+		       v_bank);
+
+	return 0;
+}
+
+static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
+				 struct v4l2_subdev_state *state)
+{
+	struct mali_c55 *mali_c55 = rzr->mali_c55;
+	u32 bypass = 0;
+
+	/* Verify if cropping and scaling should be enabled. */
+	bypass |= mali_c55_rzr_program_crop(rzr, state);
+	bypass |= mali_c55_rzr_program_resizer(rzr, state);
+
+	mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
+			     MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
+			     MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
+			     bypass);
+}
+
+/*
+ * Inspect the routing table to know which of the two (mutually exclusive)
+ * routes is enabled and return the sink pad id of the active route.
+ */
+static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
+{
+	struct v4l2_subdev_krouting *routing = &state->routing;
+	struct v4l2_subdev_route *route;
+
+	/* A single route is enabled at a time. */
+	for_each_active_route(routing, route)
+		return route->sink_pad;
+
+	return MALI_C55_RZR_SINK_PAD;
+}
+
+static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
+{
+	u32 corrected_code = 0;
+
+	/*
+	 * The ISP takes input in a 20-bit format, but can only output 16-bit
+	 * RAW bayer data (with the 4 least significant bits from the input
+	 * being lost). Return the 16-bit version of the 20-bit input formats.
+	 */
+	switch (mbus_code) {
+	case MEDIA_BUS_FMT_SBGGR20_1X20:
+		corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
+		break;
+	case MEDIA_BUS_FMT_SGBRG20_1X20:
+		corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
+		break;
+	case MEDIA_BUS_FMT_SGRBG20_1X20:
+		corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
+		break;
+	case MEDIA_BUS_FMT_SRGGB20_1X20:
+		corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
+		break;
+	}
+
+	return corrected_code;
+}
+
+static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
+				      struct v4l2_subdev_state *state,
+				      struct v4l2_subdev_krouting *routing)
+{
+	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
+						    sd);
+	unsigned int active_sink = UINT_MAX;
+	struct v4l2_mbus_framefmt *src_fmt;
+	struct v4l2_rect *crop, *compose;
+	struct v4l2_subdev_route *route;
+	unsigned int active_routes = 0;
+	struct v4l2_mbus_framefmt *fmt;
+	int ret;
+
+	ret = v4l2_subdev_routing_validate(sd, routing, 0);
+	if (ret)
+		return ret;
+
+	/* Only a single route can be enabled at a time. */
+	for_each_active_route(routing, route) {
+		if (++active_routes > 1) {
+			dev_err(rzr->mali_c55->dev,
+				"Only one route can be active");
+			return -EINVAL;
+		}
+
+		active_sink = route->sink_pad;
+	}
+	if (active_sink == UINT_MAX) {
+		dev_err(rzr->mali_c55->dev, "One route has to be active");
+		return -EINVAL;
+	}
+
+	ret = v4l2_subdev_set_routing(sd, state, routing);
+	if (ret) {
+		dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
+		return ret;
+	}
+
+	fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
+	crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
+	compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
+
+	fmt->width = MALI_C55_DEFAULT_WIDTH;
+	fmt->height = MALI_C55_DEFAULT_HEIGHT;
+	fmt->colorspace = V4L2_COLORSPACE_SRGB;
+	fmt->field = V4L2_FIELD_NONE;
+
+	if (active_sink == MALI_C55_RZR_SINK_PAD) {
+		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
+
+		crop->left = crop->top = 0;
+		crop->width = MALI_C55_DEFAULT_WIDTH;
+		crop->height = MALI_C55_DEFAULT_HEIGHT;
+
+		*compose = *crop;
+	} else {
+		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
+	}
+
+	/* Propagate the format to the source pad */
+	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
+					       0);
+	*src_fmt = *fmt;
+
+	/* In the event this is the bypass pad the mbus code needs correcting */
+	if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
+		src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
+
+	return 0;
+}
+
+static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
+				       struct v4l2_subdev_state *state,
+				       struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct v4l2_mbus_framefmt *sink_fmt;
+	const struct mali_c55_isp_fmt *fmt;
+	unsigned int index = 0;
+	u32 sink_pad;
+
+	switch (code->pad) {
+	case MALI_C55_RZR_SINK_PAD:
+		if (code->index)
+			return -EINVAL;
+
+		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
+
+		return 0;
+	case MALI_C55_RZR_SOURCE_PAD:
+		sink_pad = mali_c55_rzr_get_active_sink(state);
+		sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
+
+		/*
+		 * If the active route is from the Bypass sink pad, then the
+		 * source pad is a simple passthrough of the sink format,
+		 * downshifted to 16-bits.
+		 */
+
+		if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
+			if (code->index)
+				return -EINVAL;
+
+			code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
+			if (!code->code)
+				return -EINVAL;
+
+			return 0;
+		}
+
+		/*
+		 * If the active route is from the non-bypass sink then we can
+		 * select either RGB or conversion to YUV.
+		 */
+
+		if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
+			return -EINVAL;
+
+		code->code = rzr_non_bypass_src_fmts[code->index];
+
+		return 0;
+	case MALI_C55_RZR_SINK_BYPASS_PAD:
+		for_each_mali_isp_fmt(fmt) {
+			if (index++ == code->index) {
+				code->code = fmt->code;
+				return 0;
+			}
+		}
+
+		break;
+	}
+
+	return -EINVAL;
+}
+
+static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
+					struct v4l2_subdev_state *state,
+					struct v4l2_subdev_frame_size_enum *fse)
+{
+	if (fse->index)
+		return -EINVAL;
+
+	fse->max_width = MALI_C55_MAX_WIDTH;
+	fse->max_height = MALI_C55_MAX_HEIGHT;
+	fse->min_width = MALI_C55_MIN_WIDTH;
+	fse->min_height = MALI_C55_MIN_HEIGHT;
+
+	return 0;
+}
+
+static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_format *format)
+{
+	struct v4l2_mbus_framefmt *fmt = &format->format;
+	struct v4l2_rect *rect;
+	unsigned int sink_pad;
+
+	/*
+	 * Clamp to min/max and then reset crop and compose rectangles to the
+	 * newly applied size.
+	 */
+	clamp_t(unsigned int, fmt->width,
+		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
+	clamp_t(unsigned int, fmt->height,
+		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
+
+	sink_pad = mali_c55_rzr_get_active_sink(state);
+	if (sink_pad == MALI_C55_RZR_SINK_PAD) {
+		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
+
+		rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
+		rect->left = 0;
+		rect->top = 0;
+		rect->width = fmt->width;
+		rect->height = fmt->height;
+
+		rect = v4l2_subdev_state_get_compose(state,
+						     MALI_C55_RZR_SINK_PAD);
+		rect->left = 0;
+		rect->top = 0;
+		rect->width = fmt->width;
+		rect->height = fmt->height;
+	} else {
+		/*
+		 * Make sure the media bus code is one of the supported
+		 * ISP input media bus codes.
+		 */
+		if (!mali_c55_isp_is_format_supported(fmt->code))
+			fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
+	}
+
+	*v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
+	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
+
+	return 0;
+}
+
+static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
+				       struct v4l2_subdev_state *state,
+				       struct v4l2_subdev_format *format)
+{
+	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
+						    sd);
+	struct v4l2_mbus_framefmt *fmt = &format->format;
+	struct v4l2_mbus_framefmt *sink_fmt;
+	struct v4l2_rect *crop, *compose;
+	unsigned int sink_pad;
+	unsigned int i;
+
+	sink_pad = mali_c55_rzr_get_active_sink(state);
+	sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
+	crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
+	compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
+
+	/* FR Bypass pipe. */
+
+	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
+		/*
+		 * Format on the source pad is the same as the one on the
+		 * sink pad, downshifted to 16-bits.
+		 */
+		fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
+		if (!fmt->code)
+			return -EINVAL;
+
+		/* RAW bypass disables scaling and cropping. */
+		crop->top = compose->top = 0;
+		crop->left = compose->left = 0;
+		fmt->width = crop->width = compose->width = sink_fmt->width;
+		fmt->height = crop->height = compose->height = sink_fmt->height;
+
+		*v4l2_subdev_state_get_format(state,
+					      MALI_C55_RZR_SOURCE_PAD) = *fmt;
+
+		return 0;
+	}
+
+	/* Regular processing pipe. */
+
+	for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
+		if (fmt->code == rzr_non_bypass_src_fmts[i])
+			break;
+	}
+
+	if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
+		dev_dbg(rzr->mali_c55->dev,
+			"Unsupported mbus code 0x%x: using default\n",
+			fmt->code);
+		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
+	}
+
+	/*
+	 * The source pad format size comes directly from the sink pad
+	 * compose rectangle.
+	 */
+	fmt->width = compose->width;
+	fmt->height = compose->height;
+
+	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
+
+	return 0;
+}
+
+static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
+				struct v4l2_subdev_state *state,
+				struct v4l2_subdev_format *format)
+{
+	/*
+	 * On sink pads fmt is either fixed for the 'regular' processing
+	 * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
+	 * pad.
+	 *
+	 * On source pad sizes are the result of crop+compose on the sink
+	 * pad sizes, while the format depends on the active route.
+	 */
+
+	if (format->pad != MALI_C55_RZR_SOURCE_PAD)
+		return mali_c55_rzr_set_sink_fmt(sd, state, format);
+
+	return mali_c55_rzr_set_source_fmt(sd, state, format);
+}
+
+static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
+				      struct v4l2_subdev_state *state,
+				      struct v4l2_subdev_selection *sel)
+{
+	if (sel->pad != MALI_C55_RZR_SINK_PAD)
+		return -EINVAL;
+
+	if (sel->target != V4L2_SEL_TGT_CROP &&
+	    sel->target != V4L2_SEL_TGT_COMPOSE)
+		return -EINVAL;
+
+	sel->r = sel->target == V4L2_SEL_TGT_CROP
+	       ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
+	       : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
+
+	return 0;
+}
+
+static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
+				      struct v4l2_subdev_state *state,
+				      struct v4l2_subdev_selection *sel)
+{
+	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
+						    sd);
+	struct v4l2_mbus_framefmt *source_fmt;
+	struct v4l2_mbus_framefmt *sink_fmt;
+	struct v4l2_rect *crop, *compose;
+
+	if (sel->pad != MALI_C55_RZR_SINK_PAD)
+		return -EINVAL;
+
+	if (sel->target != V4L2_SEL_TGT_CROP &&
+	    sel->target != V4L2_SEL_TGT_COMPOSE)
+		return -EINVAL;
+
+	source_fmt = v4l2_subdev_state_get_format(state,
+						  MALI_C55_RZR_SOURCE_PAD);
+	sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
+	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
+	compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
+
+	/* RAW bypass disables crop/scaling. */
+	if (mali_c55_format_is_raw(source_fmt->code)) {
+		crop->top = compose->top = 0;
+		crop->left = compose->left = 0;
+		crop->width = compose->width = sink_fmt->width;
+		crop->height = compose->height = sink_fmt->height;
+
+		sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
+
+		return 0;
+	}
+
+	/* During streaming, it is allowed to only change the crop rectangle. */
+	if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
+		return -EINVAL;
+
+	 /*
+	  * Update the desired target and then clamp the crop rectangle to the
+	  * sink format sizes and the compose size to the crop sizes.
+	  */
+	if (sel->target == V4L2_SEL_TGT_CROP)
+		*crop = sel->r;
+	else
+		*compose = sel->r;
+
+	clamp_t(unsigned int, crop->left, 0,  sink_fmt->width);
+	clamp_t(unsigned int, crop->top, 0,  sink_fmt->height);
+	clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
+		sink_fmt->width - crop->left);
+	clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
+		sink_fmt->height - crop->top);
+
+	if (rzr->streaming) {
+		/*
+		 * Apply at runtime a crop rectangle on the resizer's sink only
+		 * if it doesn't require re-programming the scaler output sizes
+		 * as it would require changing the output buffer sizes as well.
+		 */
+		if (sel->r.width < compose->width ||
+		    sel->r.height < compose->height)
+			return -EINVAL;
+
+		*crop = sel->r;
+		mali_c55_rzr_program(rzr, state);
+
+		return 0;
+	}
+
+	compose->left = 0;
+	compose->top = 0;
+	clamp_t(unsigned int, compose->left, 0,  sink_fmt->width);
+	clamp_t(unsigned int, compose->top, 0,  sink_fmt->height);
+	clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
+	clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
+
+	sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
+
+	return 0;
+}
+
+static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
+				    struct v4l2_subdev_state *state,
+				    enum v4l2_subdev_format_whence which,
+				    struct v4l2_subdev_krouting *routing)
+{
+	if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
+	    media_entity_is_streaming(&sd->entity))
+		return -EBUSY;
+
+	return __mali_c55_rzr_set_routing(sd, state, routing);
+}
+
+static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
+	.enum_mbus_code		= mali_c55_rzr_enum_mbus_code,
+	.enum_frame_size	= mali_c55_rzr_enum_frame_size,
+	.get_fmt		= v4l2_subdev_get_fmt,
+	.set_fmt		= mali_c55_rzr_set_fmt,
+	.get_selection		= mali_c55_rzr_get_selection,
+	.set_selection		= mali_c55_rzr_set_selection,
+	.set_routing		= mali_c55_rzr_set_routing,
+};
+
+void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
+{
+	struct mali_c55 *mali_c55 = rzr->mali_c55;
+	struct v4l2_subdev *sd = &rzr->sd;
+	struct v4l2_subdev_state *state;
+	unsigned int sink_pad;
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	sink_pad = mali_c55_rzr_get_active_sink(state);
+	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
+		/* Bypass FR pipe processing if the bypass route is active. */
+		mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
+				     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
+				     MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
+		goto unlock_state;
+	}
+
+	/* Disable bypass and use regular processing. */
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
+			     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
+	mali_c55_rzr_program(rzr, state);
+
+unlock_state:
+	rzr->streaming = true;
+	v4l2_subdev_unlock_state(state);
+}
+
+void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
+{
+	struct v4l2_subdev *sd = &rzr->sd;
+	struct v4l2_subdev_state *state;
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+	rzr->streaming = false;
+	v4l2_subdev_unlock_state(state);
+}
+
+static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
+	.pad	= &mali_c55_resizer_pad_ops,
+};
+
+static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
+				   struct v4l2_subdev_state *state)
+{
+	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
+						    sd);
+	struct v4l2_subdev_krouting routing = { };
+	struct v4l2_subdev_route *routes;
+	unsigned int i;
+	int ret;
+
+	routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
+	if (!routes)
+		return -ENOMEM;
+
+	for (i = 0; i < rzr->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routes[i];
+
+		route->sink_pad = i
+				? MALI_C55_RZR_SINK_BYPASS_PAD
+				: MALI_C55_RZR_SINK_PAD;
+		route->source_pad = MALI_C55_RZR_SOURCE_PAD;
+		if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
+			route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
+	}
+
+	routing.num_routes = rzr->num_routes;
+	routing.routes = routes;
+
+	ret = __mali_c55_rzr_set_routing(sd, state, &routing);
+	kfree(routes);
+
+	return ret;
+}
+
+static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
+	.init_state = mali_c55_rzr_init_state,
+};
+
+static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
+						  unsigned int index)
+{
+	const unsigned int scaler_filt_coefmem_addrs[][2] = {
+		[MALI_C55_RZR_FR] = {
+			0x034A8, /* hfilt */
+			0x044A8  /* vfilt */
+		},
+		[MALI_C55_RZR_DS] = {
+			0x014A8, /* hfilt */
+			0x024A8  /* vfilt */
+		},
+	};
+	unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
+	unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
+	unsigned int i, j;
+
+	for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
+		for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
+			mali_c55_write(mali_c55, haddr,
+				mali_c55_scaler_h_filter_coefficients[i][j]);
+			mali_c55_write(mali_c55, vaddr,
+				mali_c55_scaler_v_filter_coefficients[i][j]);
+
+			haddr += sizeof(u32);
+			vaddr += sizeof(u32);
+		}
+	}
+}
+
+int mali_c55_register_resizers(struct mali_c55 *mali_c55)
+{
+	unsigned int i;
+	int ret;
+
+	for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
+		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
+		struct v4l2_subdev *sd = &rzr->sd;
+		unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
+
+		rzr->id = i;
+		rzr->streaming = false;
+
+		if (rzr->id == MALI_C55_RZR_FR)
+			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
+		else
+			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
+
+		mali_c55_resizer_program_coefficients(mali_c55, i);
+
+		v4l2_subdev_init(sd, &mali_c55_resizer_ops);
+		sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
+			     | V4L2_SUBDEV_FL_STREAMS;
+		sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
+		sd->internal_ops = &mali_c55_resizer_internal_ops;
+		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
+			 MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
+
+		rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
+		rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
+
+		/* Only the FR pipe has a bypass pad. */
+		if (rzr->id == MALI_C55_RZR_FR) {
+			rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
+							MEDIA_PAD_FL_SINK;
+			rzr->num_routes = 2;
+		} else {
+			num_pads -= 1;
+			rzr->num_routes = 1;
+		}
+
+		ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
+		if (ret)
+			return ret;
+
+		ret = v4l2_subdev_init_finalize(sd);
+		if (ret)
+			goto err_cleanup;
+
+		ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
+		if (ret)
+			goto err_cleanup;
+
+		rzr->mali_c55 = mali_c55;
+	}
+
+	return 0;
+
+err_cleanup:
+	for (; i >= 0; --i) {
+		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
+		struct v4l2_subdev *sd = &rzr->sd;
+
+		v4l2_subdev_cleanup(sd);
+		media_entity_cleanup(&sd->entity);
+	}
+
+	return ret;
+}
+
+void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
+{
+	unsigned int i;
+
+	for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
+		struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
+
+		if (!resizer->mali_c55)
+			continue;
+
+		v4l2_device_unregister_subdev(&resizer->sd);
+		v4l2_subdev_cleanup(&resizer->sd);
+		media_entity_cleanup(&resizer->sd.entity);
+	}
+}
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
new file mode 100644
index 000000000000..c7e699741c6d
--- /dev/null
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
@@ -0,0 +1,402 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ARM Mali-C55 ISP Driver - Test pattern generator
+ *
+ * Copyright (C) 2024 Ideas on Board Oy
+ */
+
+#include <linux/minmax.h>
+#include <linux/string.h>
+
+#include <media/media-entity.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-subdev.h>
+
+#include "mali-c55-common.h"
+#include "mali-c55-registers.h"
+
+#define MALI_C55_TPG_SRC_PAD		0
+#define MALI_C55_TPG_FIXED_HBLANK	0x20
+#define MALI_C55_TPG_MAX_VBLANK		0xFFFF
+#define MALI_C55_TPG_PIXEL_RATE		100000000
+
+static const char * const mali_c55_tpg_test_pattern_menu[] = {
+	"Flat field",
+	"Horizontal gradient",
+	"Vertical gradient",
+	"Vertical bars",
+	"Arbitrary rectangle",
+	"White frame on black field"
+};
+
+static const u32 mali_c55_tpg_mbus_codes[] = {
+	MEDIA_BUS_FMT_SRGGB20_1X20,
+	MEDIA_BUS_FMT_RGB202020_1X60,
+};
+
+static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
+				       int *def_vblank, int *min_vblank)
+{
+	unsigned int hts;
+	int tgt_fps;
+	int vblank;
+
+	hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
+
+	/*
+	 * The ISP has minimum vertical blanking requirements that must be
+	 * adhered to by the TPG. The minimum is a function of the Iridix blocks
+	 * clocking requirements and the width of the image and horizontal
+	 * blanking, but if we assume the worst case iVariance and sVariance
+	 * values then it boils down to the below.
+	 */
+	*min_vblank = 15 + (120500 / hts);
+
+	/*
+	 * We need to set a sensible default vblank for whatever format height
+	 * we happen to be given from set_fmt(). This function just targets
+	 * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
+	 * If we can't get 5fps we'll take whatever the minimum vblank gives us.
+	 */
+	tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
+
+	if (tgt_fps < 5)
+		vblank = *min_vblank;
+	else
+		vblank = MALI_C55_TPG_PIXEL_RATE / hts
+		       / max(rounddown(tgt_fps, 15), 5);
+
+	*def_vblank = ALIGN_DOWN(vblank, 2) - format->height;
+}
+
+static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct mali_c55_tpg *tpg = container_of(ctrl->handler,
+						struct mali_c55_tpg,
+						ctrls.handler);
+	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
+
+	switch (ctrl->id) {
+	case V4L2_CID_TEST_PATTERN:
+		mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
+			       ctrl->val);
+		break;
+	case V4L2_CID_VBLANK:
+		mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
+				     MALI_C55_REG_VBLANK_MASK, ctrl->val);
+		break;
+	default:
+		dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
+	.s_ctrl = &mali_c55_tpg_s_ctrl,
+};
+
+static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
+				   struct v4l2_subdev *sd)
+{
+	struct v4l2_subdev_state *state;
+	struct v4l2_mbus_framefmt *fmt;
+
+	/*
+	 * hblank needs setting, but is a read-only control and thus won't be
+	 * called during __v4l2_ctrl_handler_setup(). Do it here instead.
+	 */
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
+			     MALI_C55_REG_HBLANK_MASK,
+			     MALI_C55_TPG_FIXED_HBLANK);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
+			     MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
+			     MALI_C55_TEST_PATTERN_RGB_MASK,
+			     fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
+					  0x01 : 0x0);
+
+	v4l2_subdev_unlock_state(state);
+}
+
+static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
+	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
+
+	if (!enable) {
+		mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
+				MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
+		mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
+				MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
+		return 0;
+	}
+
+	/*
+	 * One might reasonably expect the framesize to be set here
+	 * given it's configurable in .set_fmt(), but it's done in the
+	 * ISP subdevice's stream on func instead, as the same register
+	 * is also used to indicate the size of the data coming from the
+	 * sensor.
+	 */
+	mali_c55_tpg_configure(mali_c55, sd);
+	__v4l2_ctrl_handler_setup(sd->ctrl_handler);
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
+			     MALI_C55_TEST_PATTERN_ON_OFF,
+			     MALI_C55_TEST_PATTERN_ON_OFF);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
+			     MALI_C55_REG_GEN_VIDEO_ON_MASK,
+			     MALI_C55_REG_GEN_VIDEO_ON_MASK);
+
+	return 0;
+}
+
+static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
+	.s_stream = &mali_c55_tpg_s_stream,
+};
+
+static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
+				       struct v4l2_subdev_state *state,
+				       struct v4l2_subdev_mbus_code_enum *code)
+{
+	if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
+		return -EINVAL;
+
+	code->code = mali_c55_tpg_mbus_codes[code->index];
+
+	return 0;
+}
+
+static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
+					struct v4l2_subdev_state *state,
+					struct v4l2_subdev_frame_size_enum *fse)
+{
+	if (fse->index > 0 || fse->pad > sd->entity.num_pads)
+		return -EINVAL;
+
+	fse->min_width = MALI_C55_MIN_WIDTH;
+	fse->max_width = MALI_C55_MAX_WIDTH;
+	fse->min_height = MALI_C55_MIN_HEIGHT;
+	fse->max_height = MALI_C55_MAX_HEIGHT;
+
+	return 0;
+}
+
+static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
+				struct v4l2_subdev_state *state,
+				struct v4l2_subdev_format *format)
+{
+	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
+	struct v4l2_mbus_framefmt *fmt = &format->format;
+	int vblank_def, vblank_min;
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
+		if (fmt->code == mali_c55_tpg_mbus_codes[i])
+			break;
+	}
+
+	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
+		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
+
+	/*
+	 * The TPG says that the test frame timing generation logic expects a
+	 * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
+	 * handle anything smaller than 128x128 it seems pointless to allow a
+	 * smaller frame.
+	 */
+	clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
+		MALI_C55_MAX_WIDTH);
+	clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
+		MALI_C55_MAX_HEIGHT);
+
+	*v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;
+
+	if (format->which == V4L2_SUBDEV_FORMAT_TRY)
+		return 0;
+
+	__mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
+	__v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
+				 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
+	__v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);
+
+	return 0;
+}
+
+static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
+	.enum_mbus_code		= mali_c55_tpg_enum_mbus_code,
+	.enum_frame_size	= mali_c55_tpg_enum_frame_size,
+	.get_fmt		= v4l2_subdev_get_fmt,
+	.set_fmt		= mali_c55_tpg_set_fmt,
+};
+
+static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
+	.video	= &mali_c55_tpg_video_ops,
+	.pad	= &mali_c55_tpg_pad_ops,
+};
+
+static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
+				   struct v4l2_subdev_state *sd_state)
+{
+	struct v4l2_mbus_framefmt *fmt =
+		v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
+
+	fmt->width = MALI_C55_DEFAULT_WIDTH;
+	fmt->height = MALI_C55_DEFAULT_HEIGHT;
+	fmt->field = V4L2_FIELD_NONE;
+	fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
+
+	return 0;
+}
+
+static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
+	.init_state = mali_c55_tpg_init_state,
+};
+
+static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
+{
+	struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
+	struct v4l2_subdev *sd = &mali_c55->tpg.sd;
+	struct v4l2_mbus_framefmt *format;
+	struct v4l2_subdev_state *state;
+	int vblank_def, vblank_min;
+	int ret;
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+	format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
+
+	ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);
+	if (ret)
+		goto err_unlock;
+
+	ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
+				&mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
+				ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
+				0, 3, mali_c55_tpg_test_pattern_menu);
+
+	/*
+	 * We fix hblank at the minimum allowed value and control framerate
+	 * solely through the vblank control.
+	 */
+	ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
+				&mali_c55_tpg_ctrl_ops,
+				V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
+				MALI_C55_TPG_FIXED_HBLANK, 1,
+				MALI_C55_TPG_FIXED_HBLANK);
+	if (ctrls->hblank)
+		ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
+
+	__mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);
+	ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
+					  &mali_c55_tpg_ctrl_ops,
+					  V4L2_CID_VBLANK, vblank_min,
+					  MALI_C55_TPG_MAX_VBLANK, 1,
+					  vblank_def);
+
+	if (ctrls->handler.error) {
+		dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
+		ret = ctrls->handler.error;
+		goto err_free_handler;
+	}
+
+	ctrls->handler.lock = &mali_c55->tpg.lock;
+	mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
+
+	v4l2_subdev_unlock_state(state);
+
+	return 0;
+
+err_free_handler:
+	v4l2_ctrl_handler_free(&ctrls->handler);
+err_unlock:
+	v4l2_subdev_unlock_state(state);
+	return ret;
+}
+
+int mali_c55_register_tpg(struct mali_c55 *mali_c55)
+{
+	struct mali_c55_tpg *tpg = &mali_c55->tpg;
+	struct v4l2_subdev *sd = &tpg->sd;
+	struct media_pad *pad = &tpg->pad;
+	int ret;
+
+	mutex_init(&tpg->lock);
+
+	v4l2_subdev_init(sd, &mali_c55_tpg_ops);
+	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+	sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
+	sd->internal_ops = &mali_c55_tpg_internal_ops;
+	strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
+
+	pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;
+	ret = media_entity_pads_init(&sd->entity, 1, pad);
+	if (ret) {
+		dev_err(mali_c55->dev,
+			"Failed to initialize media entity pads\n");
+		goto err_destroy_mutex;
+	}
+
+	ret = v4l2_subdev_init_finalize(sd);
+	if (ret)
+		goto err_cleanup_media_entity;
+
+	ret = mali_c55_tpg_init_controls(mali_c55);
+	if (ret) {
+		dev_err(mali_c55->dev,
+			"Error initialising controls\n");
+		goto err_cleanup_subdev;
+	}
+
+	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
+	if (ret) {
+		dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
+		goto err_free_ctrl_handler;
+	}
+
+	/*
+	 * By default the colour settings lead to a very dim image that is
+	 * nearly indistinguishable from black on some monitor settings. Ramp
+	 * them up a bit so the image is brighter.
+	 */
+	mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
+		       MALI_C55_TPG_BACKGROUND_MAX);
+	mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
+		       MALI_C55_TPG_BACKGROUND_MAX);
+	mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
+		       MALI_C55_TPG_BACKGROUND_MAX);
+
+	tpg->mali_c55 = mali_c55;
+
+	return 0;
+
+err_free_ctrl_handler:
+	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
+err_cleanup_subdev:
+	v4l2_subdev_cleanup(sd);
+err_cleanup_media_entity:
+	media_entity_cleanup(&sd->entity);
+err_destroy_mutex:
+	mutex_destroy(&tpg->lock);
+
+	return ret;
+}
+
+void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
+{
+	struct mali_c55_tpg *tpg = &mali_c55->tpg;
+
+	if (!tpg->mali_c55)
+		return;
+
+	v4l2_device_unregister_subdev(&tpg->sd);
+	v4l2_subdev_cleanup(&tpg->sd);
+	media_entity_cleanup(&tpg->sd.entity);
+	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
+	mutex_destroy(&tpg->lock);
+}
-- 
2.34.1

[PATCH v5 06/16] media: Documentation: Add Mali-C55 ISP Documentation

[Daniel Scally]

Add a documentation page for the mali-c55 driver, which gives a brief
overview of the hardware and explains how to use the driver's capture
devices and the crop/scaler functions.

Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- None

Changes in v4:
	- None

Changes in v3:
	- Documented the synchronised buffer sequence numbers (Sakari)
	- Clarified that the downscale pipe cannot output raw data, the ISP'S
	  resolution limits and choice of media bus format code (Kieran)

Changes in v2:

	- none

 .../admin-guide/media/mali-c55-graph.dot      |  19 +
 Documentation/admin-guide/media/mali-c55.rst  | 333 ++++++++++++++++++
 .../admin-guide/media/v4l-drivers.rst         |   1 +
 3 files changed, 353 insertions(+)
 create mode 100644 Documentation/admin-guide/media/mali-c55-graph.dot
 create mode 100644 Documentation/admin-guide/media/mali-c55.rst

diff --git a/Documentation/admin-guide/media/mali-c55-graph.dot b/Documentation/admin-guide/media/mali-c55-graph.dot
new file mode 100644
index 000000000000..0775ba42bf4c
--- /dev/null
+++ b/Documentation/admin-guide/media/mali-c55-graph.dot
@@ -0,0 +1,19 @@
+digraph board {
+        rankdir=TB
+        n00000001 [label="{{} | mali-c55 tpg\n/dev/v4l-subdev0 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
+        n00000001:port0 -> n00000003:port0 [style=dashed]
+        n00000003 [label="{{<port0> 0} | mali-c55 isp\n/dev/v4l-subdev1 | {<port1> 1 | <port2> 2}}", shape=Mrecord, style=filled, fillcolor=green]
+        n00000003:port1 -> n00000007:port0 [style=bold]
+        n00000003:port2 -> n00000007:port2 [style=bold]
+        n00000003:port1 -> n0000000b:port0 [style=bold]
+        n00000007 [label="{{<port0> 0 | <port2> 2} | mali-c55 resizer fr\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
+        n00000007:port1 -> n0000000e [style=bold]
+        n0000000b [label="{{<port0> 0} | mali-c55 resizer ds\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
+        n0000000b:port1 -> n00000012 [style=bold]
+        n0000000e [label="mali-c55 fr\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
+        n00000012 [label="mali-c55 ds\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
+        n00000022 [label="{{<port0> 0} | csi2-rx\n/dev/v4l-subdev4 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
+        n00000022:port1 -> n00000003:port0
+        n00000027 [label="{{} | imx415 1-001a\n/dev/v4l-subdev5 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
+        n00000027:port0 -> n00000022:port0 [style=bold]
+}
\ No newline at end of file
diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst
new file mode 100644
index 000000000000..cf4176cb1e44
--- /dev/null
+++ b/Documentation/admin-guide/media/mali-c55.rst
@@ -0,0 +1,333 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==========================================
+ARM Mali-C55 Image Signal Processor driver
+==========================================
+
+Introduction
+============
+
+This file documents the driver for ARM's Mali-C55 Image Signal Processor. The
+driver is located under drivers/media/platform/arm/mali-c55.
+
+The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from
+sensors through either a parallel interface or a memory bus before processing it
+and outputting it through an internal DMA engine. Two output pipelines are
+possible (though one may not be fitted, depending on the implementation). These
+are referred to as "Full resolution" and "Downscale", but the naming is historic
+and both pipes are capable of cropping/scaling operations. The full resolution
+pipe is also capable of outputting RAW data, bypassing much of the ISP's
+processing. The downscale pipe cannot output RAW data. An integrated test
+pattern generator can be used to drive the ISP and produce image data in the
+absence of a connected camera sensor. The driver module is named mali_c55, and
+is enabled through the CONFIG_VIDEO_MALI_C55 config option.
+
+The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and
+expects camera sensors connected to the ISP to have V4L2 subdevice interfaces.
+
+Mali-C55 ISP hardware
+=====================
+
+A high level functional view of the Mali-C55 ISP is presented below. The ISP
+takes input from either a live source or through a DMA engine for memory input,
+depending on the SoC integration.::
+
+  +---------+    +----------+                                     +--------+
+  | Sensor  |--->| CSI-2 Rx |                "Full Resolution"    |  DMA   |
+  +---------+    +----------+   |\                 Output    +--->| Writer |
+                       |        | \                          |    +--------+
+                       |        |  \    +----------+  +------+---> Streaming I/O
+  +------------+       +------->|   |   |          |  |
+  |            |                |   |-->| Mali-C55 |--+
+  | DMA Reader |--------------->|   |   |    ISP   |  |
+  |            |                |  /    |          |  |      +---> Streaming I/O
+  +------------+                | /     +----------+  |      |
+                                |/                    +------+
+				                             |    +--------+
+                                                             +--->|  DMA   |
+                                               "Downscaled"       | Writer |
+					          Output          +--------+
+
+Media Controller Topology
+=========================
+
+An example of the ISP's topology (as implemented in a system with an IMX415
+camera sensor and generic CSI-2 receiver) is below:
+
+
+.. kernel-figure:: mali-c55-graph.dot
+    :alt:   mali-c55-graph.dot
+    :align: center
+
+The driver has 4 V4L2 subdevices:
+
+- `mali_c55 isp`: Responsible for configuring input crop and color space
+                  conversion
+- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor.
+- `mali_c55 resizer fr`: The Full-Resolution pipe resizer
+- `mali_c55 resizer ds`: The Downscale pipe resizer
+
+The driver has 2 V4L2 video devices:
+
+- `mali-c55 fr`: The full-resolution pipe's capture device
+- `mali-c55 ds`: The downscale pipe's capture device
+
+Frame sequences are synchronised across to two capture devices, meaning if one
+pipe is started later than the other the sequence numbers returned in its
+buffers will match those of the other pipe rather than starting from zero.
+
+Frame sequences are synchronised across to two capture devices, meaning if one
+pipe is started later than the other the sequence numbers returned in its
+buffers will match those of the other pipe rather than starting from zero.
+
+Idiosyncrasies
+--------------
+
+**mali-c55 isp**
+The `mali-c55 isp` subdevice has a single sink pad to which all sources of data
+should be connected. The active source is selected by enabling the appropriate
+media link and disabling all others. The ISP has two source pads, reflecting the
+different paths through which it can internally route data. Tap points within
+the ISP allow users to divert data to avoid processing by some or all of the
+hardware's processing steps. The diagram below is intended only to highlight how
+the bypassing works and is not a true reflection of those processing steps; for
+a high-level functional block diagram see ARM's developer page for the
+ISP [3]_::
+
+  +--------------------------------------------------------------+
+  |                Possible Internal ISP Data Routes             |
+  |          +------------+  +----------+  +------------+        |
+  +---+      |            |  |          |  |  Colour    |    +---+
+  | 0 |--+-->| Processing |->| Demosaic |->|   Space    |--->| 1 |
+  +---+  |   |            |  |          |  | Conversion |    +---+
+  |      |   +------------+  +----------+  +------------+        |
+  |      |                                                   +---+
+  |      +---------------------------------------------------| 2 |
+  |                                                          +---+
+  |                                                              |
+  +--------------------------------------------------------------+
+
+
+.. flat-table::
+    :header-rows: 1
+
+    * - Pad
+      - Direction
+      - Purpose
+
+    * - 0
+      - sink
+      - Data input, connected to the TPG and camera sensors
+
+    * - 1
+      - source
+      - RGB/YUV data, connected to the FR and DS V4L2 subdevices
+
+    * - 2
+      - source
+      - RAW bayer data, connected to the FR V4L2 subdevices
+
+The ISP is limited to both input and output resolutions between 640x480 and
+8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt()
+operations.
+
+**mali-c55 resizer fr**
+The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different
+insertion points in the hardware (either RAW or demosaiced data):
+
+.. flat-table::
+    :header-rows: 1
+
+    * - Pad
+      - Direction
+      - Purpose
+
+    * - 0
+      - sink
+      - Data input connected to the ISP's demosaiced stream.
+
+    * - 1
+      - source
+      - Data output connected to the capture video device
+
+    * - 2
+      - sink
+      - Data input connected to the ISP's raw data stream
+
+The data source in use is selected through the routing API; two routes each of a
+single stream are available:
+
+.. flat-table::
+    :header-rows: 1
+
+    * - Sink Pad
+      - Source Pad
+      - Purpose
+
+    * - 0
+      - 1
+      - Demosaiced data route
+
+    * - 2
+      - 1
+      - Raw data route
+
+
+If the demosaiced route is active then the FR pipe is only capable of output
+in RGB/YUV formats. If the raw route is active then the output reflects the
+input (which may be either Bayer or RGB/YUV data).
+
+Using the driver to capture video
+=================================
+
+Using the media controller APIs we can configure the input source and ISP to
+capture images in a variety of formats. In the examples below, configuring the
+media graph is done with the v4l-utils [1]_ package's media-ctl utility.
+Capturing the images is done with yavta [2]_.
+
+Configuring the input source
+----------------------------
+
+The first step is to set the input source that we wish by enabling the correct
+media link. Using the example topology above, we can select the TPG as follows:
+
+.. code-block:: none
+
+    media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]"
+    media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]"
+
+Capturing bayer data from the source and processing to RGB/YUV
+--------------------------------------------------------------
+To capture 1920x1080 bayer data from the source and push it through the ISP's
+full processing pipeline, we configure the data formats appropriately on the
+source, ISP and resizer subdevices and set the FR resizer's routing to select
+processed data. The media bus format on the resizer's source pad will be either
+RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or
+YUV. The ISP's debayering block outputs RGB data natively, setting the source
+pad format to YUV10_1X30 enables the colour space conversion block.
+
+In this example we target RGB565 output, so select RGB121212_1X36 as the resizer
+source pad's format:
+
+.. code-block:: none
+
+    # Set formats on the TPG and ISP
+    media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB16_1X16/1920x1080]"
+    media-ctl -V "'mali-c55 isp':0[fmt:SRGGB16_1X16/1920x1080]"
+    media-ctl -V "'mali-c55 isp':1[fmt:SRGGB16_1X16/1920x1080]"
+
+    # Set routing on the FR resizer
+    media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]"
+
+    # Set format on the resizer, must be done AFTER the routing.
+    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]"
+
+The downscale output can also be used to stream data at the same time. In this
+case since only processed data can be captured through the downscale output no
+routing need be set:
+
+.. code-block:: none
+
+    # Set format on the resizer
+    media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]"
+
+Following which images can be captured from both the FR and DS output's video
+devices (simultaneously, if desired):
+
+.. code-block:: none
+
+    yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
+    yavta -f RGB565 -s 1920x1080 -c10 /dev/video1
+
+Cropping the image
+~~~~~~~~~~~~~~~~~~
+
+Both the full resolution and downscale pipes can crop to a minimum resolution of
+640x480. To crop the image simply configure the resizer's sink pad's crop and
+compose rectangles and set the format on the video device:
+
+.. code-block:: none
+
+    media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]"
+    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
+    yavta -f RGB565 -s 640x480 -c10 /dev/video0
+
+Downscaling the image
+~~~~~~~~~~~~~~~~~~~~~
+
+Both the full resolution and downscale pipes can downscale the image by up to 8x
+provided the minimum 640x480 resolution is adhered to. For the best image result
+the scaling ratio for each dimension should be the same. To configure scaling we
+use the compose rectangle on the resizer's sink pad:
+
+.. code-block:: none
+
+    media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]"
+    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
+    yavta -f RGB565 -s 640x480 -c10 /dev/video0
+
+Capturing images in YUV formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If we need to output YUV data rather than RGB the color space conversion block
+needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the
+resizer's source pad. We can then configure a capture format like NV12 (here in
+its multi-planar variant)
+
+.. code-block:: none
+
+    media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]"
+    yavta -f NV12M -s 1920x1080 -c10 /dev/video0
+
+Capturing RGB data from the source and processing it with the resizers
+----------------------------------------------------------------------
+
+The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this
+case although none of the image quality blocks would be used it can still
+crop/scale the data in the usual way.
+
+To achieve this, the ISP's sink pad's format is set to
+MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to
+work with the ISP. Converting the camera sensor's output to that format is the
+responsibility of external hardware.
+
+In this example we ask the test pattern generator to give us RGB data instead of
+bayer.
+
+.. code-block:: none
+
+    media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]"
+    media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]"
+
+Cropping or scaling the data can be done in exactly the same way as outlined
+earlier.
+
+Capturing raw data from the source and outputting it unmodified
+-----------------------------------------------------------------
+
+The ISP can additionally capture raw data from the source and output it on the
+full resolution pipe only, completely unmodified. In this case the downscale
+pipe can still process the data normally and be used at the same time.
+
+To configure raw bypass the FR resizer's subdevice's routing table needs to be
+configured, followed by formats in the appropriate places:
+
+.. code-block:: none
+
+    # We need to configure the routing table for the resizer to use the bypass
+    # path along with set formats on the resizer's bypass sink pad. Doing this
+    # necessitates a single media-ctl command, as multiple calls to the program
+    # reset the routing table.
+    media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"\
+    -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080],"\
+       "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080],"\
+       "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]"
+
+    # Set format on the video device and stream
+    yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
+
+References
+==========
+.. [1] https://git.linuxtv.org/v4l-utils.git/
+.. [2] https://git.ideasonboard.org/yavta.git
+.. [3] https://developer.arm.com/Processors/Mali-C55
diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst
index 4120eded9a13..1d9485860d93 100644
--- a/Documentation/admin-guide/media/v4l-drivers.rst
+++ b/Documentation/admin-guide/media/v4l-drivers.rst
@@ -18,6 +18,7 @@ Video4Linux (V4L) driver-specific documentation
 	ipu3
 	ipu6-isys
 	ivtv
+	mali-c55
 	mgb4
 	omap3isp
 	omap4_camera
-- 
2.34.1

[PATCH v5 07/16] MAINTAINERS: Add entry for mali-c55 driver

[Daniel Scally]

Add a MAINTAINERS entry for the mali-c55 driver and its associated
documentation.

Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- None

Changes in v4:
	- None

Changes in v3:

	- none

Changes in v2:

	- none

 MAINTAINERS | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index ef6be9d95143..c2883b30e0ca 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1688,6 +1688,16 @@ F:	Documentation/gpu/panfrost.rst
 F:	drivers/gpu/drm/panfrost/
 F:	include/uapi/drm/panfrost_drm.h
 
+ARM MALI-C55 ISP DRIVER
+M:	Daniel Scally <dan.scally@ideasonboard.com>
+M:	Jacopo Mondi <jacopo.mondi@ideasonboard.com>
+L:	linux-media@vger.kernel.org
+S:	Maintained
+T:	git git://linuxtv.org/media_tree.git
+F:	Documentation/admin-guide/media/mali-c55.rst
+F:	Documentation/devicetree/bindings/media/arm,mali-c55.yaml
+F:	drivers/media/platform/arm/mali-c55/
+
 ARM MALI-DP DRM DRIVER
 M:	Liviu Dudau <liviu.dudau@arm.com>
 S:	Supported
-- 
2.34.1

[PATCH v5 08/16] media: Add MALI_C55_3A_STATS meta format

[Daniel Scally]

Add a new meta format for the Mali-C55 ISP's 3A Statistics along
with a new descriptor entry.

Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- New patch

 drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
 include/uapi/linux/videodev2.h       | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 4c76d17b4629..0d00b0476762 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1456,6 +1456,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_META_FMT_VIVID:       descr = "Vivid Metadata"; break;
 	case V4L2_META_FMT_RK_ISP1_PARAMS:	descr = "Rockchip ISP1 3A Parameters"; break;
 	case V4L2_META_FMT_RK_ISP1_STAT_3A:	descr = "Rockchip ISP1 3A Statistics"; break;
+	case V4L2_META_FMT_MALI_C55_3A_STATS:	descr = "ARM Mali-C55 ISP 3A Statistics"; break;
 	case V4L2_PIX_FMT_NV12_8L128:	descr = "NV12 (8x128 Linear)"; break;
 	case V4L2_PIX_FMT_NV12M_8L128:	descr = "NV12M (8x128 Linear)"; break;
 	case V4L2_PIX_FMT_NV12_10BE_8L128:	descr = "10-bit NV12 (8x128 Linear, BE)"; break;
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index fe6b67e83751..021424fdaf68 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -841,6 +841,8 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_MALI_C55_3A_STATS	v4l2_fourcc('C', '5', '5', 'S') /* ARM Mali-C55 3A Statistics */
+
 #ifdef __KERNEL__
 /*
  * Line-based metadata formats. Remember to update v4l_fill_fmtdesc() when
-- 
2.34.1

[PATCH v5 09/16] media: uapi: Add 3a stats buffer for mali-c55

[Daniel Scally]

Add a header that describes the format of the 3A statistics buffers
for the mali-c55 ISP.

Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- New patch

 .../uapi/linux/media/arm/mali-c55-config.h    | 182 ++++++++++++++++++
 1 file changed, 182 insertions(+)
 create mode 100644 include/uapi/linux/media/arm/mali-c55-config.h

diff --git a/include/uapi/linux/media/arm/mali-c55-config.h b/include/uapi/linux/media/arm/mali-c55-config.h
new file mode 100644
index 000000000000..8fb89af6c874
--- /dev/null
+++ b/include/uapi/linux/media/arm/mali-c55-config.h
@@ -0,0 +1,182 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * ARM Mali-C55 ISP Driver - Userspace API
+ *
+ * Copyright (C) 2023 Ideas on Board Oy
+ */
+
+#ifndef __UAPI_MALI_C55_CONFIG_H
+#define __UAPI_MALI_C55_CONFIG_H
+
+#include <linux/types.h>
+
+/*
+ * Frames are split into zones of almost equal width and height - a zone is a
+ * rectangular tile of a frame. The metering blocks within the ISP collect
+ * aggregated statistics per zone. A maximum of 15x15 zones can be configured,
+ * and so the statistics buffer within the hardware is sized to accommodate
+ * that.
+ *
+ * The utilised number of zones is runtime configurable.
+ */
+#define MALI_C55_MAX_ZONES	225
+
+/**
+ * struct mali_c55_ae_1024bin_hist - Auto Exposure 1024-bin histogram statistics
+ *
+ * @bins:	1024 element array of 16-bit pixel counts.
+ *
+ * The 1024-bin histogram module collects image-global but zone-weighted
+ * intensity distributions of pixels in fixed-width bins. The modules can be
+ * configured into different "plane modes" which affect the contents of the
+ * collected statistics. In plane mode 0, pixel intensities are taken regardless
+ * of colour plane into a single 1024-bin histogram with a bin width of 4. In
+ * plane mode 1, four 256-bin histograms with a bin width of 16 are collected -
+ * one for each CFA colour plane. In plane modes 4, 5, 6 and 7 two 512-bin
+ * histograms with a bin width of 8 are collected - in each mode one of the
+ * colour planes is collected into the first histogram and all the others are
+ * combined into the second. The histograms are stored consecutively in the bins
+ * array.
+ *
+ * The 16-bit pixel counts are stored as a 4-bit exponent in the most
+ * significant bits followed by a 12-bit mantissa. Conversion to a usable
+ * format can be done according to the following pseudo-code::
+ *
+ *	if (e == 0) {
+ *		bin = m * 2;
+ *	} else {
+ *		bin = (m + 4096) x 2^e
+ *	}
+ *
+ * where
+ *	e is the exponent value in range 0..15
+ *	m is the mantissa value in range 0..4095
+ *
+ * The pixels used in calculating the statistics can be masked using three
+ * methods:
+ *
+ * 1. Pixels can be skipped in X and Y directions independently.
+ * 2. Minimum/Maximum intensities can be configured
+ * 3. Zones can be differentially weighted, including 0 weighted to mask them
+ *
+ * The data for this histogram can be collected from different tap points in the
+ * ISP depending on configuration - after the white balance or digital gain
+ * blocks, or immediately after the input crossbar.
+ */
+struct mali_c55_ae_1024bin_hist {
+	__u16 bins[1024];
+} __attribute__((packed));
+
+/**
+ * struct mali_c55_ae_5bin_hist - Auto Exposure 5-bin histogram statistics
+ *
+ * @hist0:	16-bit normalised pixel count for the 0th intensity bin
+ * @hist1:	16-bit normalised pixel count for the 1st intensity bin
+ * @hist3:	16-bit normalised pixel count for the 3rd intensity bin
+ * @hist4:	16-bit normalised pixel count for the 4th intensity bin
+ *
+ * The ISP generates a 5-bin histogram of normalised pixel counts within bins of
+ * pixel intensity for each of 225 possible zones within a frame. The centre bin
+ * of the histogram for each zone is not available from the hardware and must be
+ * calculated by subtracting the values of hist0, hist1, hist3 and hist4 from
+ * 0xffff as in the following equation:
+ *
+ *	hist2 = 0xffff - (hist0 + hist1 + hist3 + hist4)
+ */
+struct mali_c55_ae_5bin_hist {
+	__u16 hist0;
+	__u16 hist1;
+	__u16 hist3;
+	__u16 hist4;
+} __attribute__((packed));
+
+/**
+ * struct mali_c55_awb_average_ratios - Auto White Balance colour ratios
+ *
+ * @avg_rg_gr:	Average R/G or G/R ratio in Q4.8 format.
+ * @avg_bg_br:	Average B/G or B/R ratio in Q4.8 format.
+ * @num_pixels:	The number of pixels used in the AWB calculation
+ *
+ * The ISP calculates and collects average colour ratios for each zone in an
+ * image and stores them in Q4.8 format (the lowest 8 bits are fractional, with
+ * bits [11:8] representing the integer). The exact ratios collected (either
+ * R/G, B/G or G/R, B/R) are configurable through the parameters buffer. The
+ * value of the 4 high bits is undefined.
+ */
+struct mali_c55_awb_average_ratios {
+	__u16 avg_rg_gr;
+	__u16 avg_bg_br;
+	__u32 num_pixels;
+} __attribute__((packed));
+
+/**
+ * struct mali_c55_af_statistics - Auto Focus edge and intensity statistics
+ *
+ * @intensity_stats:	Packed mantissa and exponent value for pixel intensity
+ * @edge_stats:		Packed mantissa and exponent values for edge intensity
+ *
+ * The ISP collects the squared sum of pixel intensities for each zone within a
+ * configurable Region of Interest on the frame. Additionally, the same data are
+ * collected after being passed through a bandpass filter which removes high and
+ * low frequency components - these are referred to as the edge statistics.
+ *
+ * The intensity and edge statistics for a zone can be used to calculate the
+ * contrast information for a zone
+ *
+ *	C = E2 / I2
+ *
+ * Where I2 is the intensity statistic for a zone and E2 is the edge statistic
+ * for that zone. Optimum focus is reached when C is at its maximum.
+ *
+ * The intensity and edge statistics are stored packed into a non-standard 16
+ * bit floating point format, where the 7 most significant bits represent the
+ * exponent and the 9 least significant bits the mantissa. This format can be
+ * unpacked with the following pseudocode::
+ *
+ *	if (e == 0) {
+ *		x = m;
+ *	} else {
+ *		x = 2^e-1 x (m + 2^9)
+ *	}
+ *
+ * where
+ *	e is the exponent value in range 0..128
+ *	m is the mantissa value in range 0..511
+ */
+struct mali_c55_af_statistics {
+	__u16 intensity_stats;
+	__u16 edge_stats;
+} __attribute__((packed));
+
+/**
+ * struct mali_c55_stats_buffer - 3A statistics for the mali-c55 ISP
+ *
+ * @ae_1024bin_hist:		1024-bin frame-global pixel intensity histogram
+ * @iridix_1024bin_hist:	Post-Iridix block 1024-bin histogram
+ * @ae_5bin_hists:		5-bin pixel intensity histograms for AEC
+ * @reserved1:			Undefined buffer space
+ * @awb_ratios:			Color balance ratios for Auto White Balance
+ * @reserved2:			Undefined buffer space
+ * @af_statistics:		Pixel intensity statistics for Auto Focus
+ * @reserved3:			Undefined buffer space
+ *
+ * This struct describes the metering statistics space in the Mali-C55 ISP's
+ * hardware in its entirety. The space between each defined area is marked as
+ * "unknown" and may not be 0, but should not be used. The @ae_5bin_hists,
+ * @awb_ratios and @af_statistics members are arrays of statistics per-zone.
+ * The zones are arranged in the array in raster order starting from the top
+ * left corner of the image.
+ */
+
+struct mali_c55_stats_buffer {
+	struct mali_c55_ae_1024bin_hist ae_1024bin_hist;
+	struct mali_c55_ae_1024bin_hist iridix_1024bin_hist;
+	struct mali_c55_ae_5bin_hist ae_5bin_hists[MALI_C55_MAX_ZONES];
+	__u32 reserved1[14];
+	struct mali_c55_awb_average_ratios awb_ratios[MALI_C55_MAX_ZONES];
+	__u32 reserved2[14];
+	struct mali_c55_af_statistics af_statistics[MALI_C55_MAX_ZONES];
+	__u32 reserved3[15];
+} __attribute__((packed));
+
+#endif /* __UAPI_MALI_C55_CONFIG_H */
-- 
2.34.1

[PATCH v5 10/16] media: platform: Add mali-c55 3a stats devnode

[Daniel Scally]

Add a new code file to govern the 3a statistics capture node.

Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- New patch

 drivers/media/platform/arm/mali-c55/Makefile  |   3 +-
 .../platform/arm/mali-c55/mali-c55-common.h   |  28 ++
 .../platform/arm/mali-c55/mali-c55-core.c     |  15 +
 .../platform/arm/mali-c55/mali-c55-isp.c      |   1 +
 .../arm/mali-c55/mali-c55-registers.h         |   3 +
 .../platform/arm/mali-c55/mali-c55-stats.c    | 350 ++++++++++++++++++
 6 files changed, 399 insertions(+), 1 deletion(-)
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-stats.c

diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
index 77dcb2fbf0f4..cd5a64bf0c62 100644
--- a/drivers/media/platform/arm/mali-c55/Makefile
+++ b/drivers/media/platform/arm/mali-c55/Makefile
@@ -4,6 +4,7 @@ mali-c55-y := mali-c55-capture.o \
 	      mali-c55-core.o \
 	      mali-c55-isp.o \
 	      mali-c55-tpg.o \
-	      mali-c55-resizer.o
+	      mali-c55-resizer.o \
+	      mali-c55-stats.o
 
 obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
index 2d0c4d152beb..44119e04009b 100644
--- a/drivers/media/platform/arm/mali-c55/mali-c55-common.h
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
@@ -79,6 +79,7 @@ enum mali_c55_isp_pads {
 	MALI_C55_ISP_PAD_SINK_VIDEO,
 	MALI_C55_ISP_PAD_SOURCE,
 	MALI_C55_ISP_PAD_SOURCE_BYPASS,
+	MALI_C55_ISP_PAD_SOURCE_3A,
 	MALI_C55_ISP_NUM_PADS,
 };
 
@@ -194,6 +195,28 @@ struct mali_c55_cap_dev {
 	bool streaming;
 };
 
+struct mali_c55_stats_buf {
+	struct vb2_v4l2_buffer vb;
+	spinlock_t lock;
+	unsigned int segments_remaining;
+	struct list_head queue;
+	bool failed;
+};
+
+struct mali_c55_stats {
+	struct mali_c55 *mali_c55;
+	struct video_device vdev;
+	struct dma_chan *channel;
+	struct vb2_queue queue;
+	struct media_pad pad;
+	struct mutex lock;
+
+	struct {
+		spinlock_t lock;
+		struct list_head queue;
+	} buffers;
+};
+
 enum mali_c55_config_spaces {
 	MALI_C55_CONFIG_PING,
 	MALI_C55_CONFIG_PONG,
@@ -224,6 +247,7 @@ struct mali_c55 {
 	struct mali_c55_isp isp;
 	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
 	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
+	struct mali_c55_stats stats;
 
 	struct list_head contexts;
 	enum mali_c55_config_spaces next_config;
@@ -245,6 +269,8 @@ int mali_c55_register_resizers(struct mali_c55 *mali_c55);
 void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
 int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
 void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
+int mali_c55_register_stats(struct mali_c55 *mali_c55);
+void mali_c55_unregister_stats(struct mali_c55 *mali_c55);
 struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
 void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
 			     enum mali_c55_planes plane);
@@ -262,5 +288,7 @@ mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
 bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
 #define for_each_mali_isp_fmt(fmt)\
 	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
+void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
+				enum mali_c55_config_spaces cfg_space);
 
 #endif /* _MALI_C55_COMMON_H */
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
index 50caf5ee7474..9ea70010876c 100644
--- a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
@@ -337,6 +337,16 @@ static int mali_c55_create_links(struct mali_c55 *mali_c55)
 		}
 	}
 
+	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
+			MALI_C55_ISP_PAD_SOURCE_3A,
+			&mali_c55->stats.vdev.entity, 0,
+			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
+	if (ret) {
+		dev_err(mali_c55->dev,
+			"failed to link ISP and 3a stats node\n");
+		goto err_remove_links;
+	}
+
 	return 0;
 
 err_remove_links:
@@ -350,6 +360,7 @@ static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
 	mali_c55_unregister_isp(mali_c55);
 	mali_c55_unregister_resizers(mali_c55);
 	mali_c55_unregister_capture_devs(mali_c55);
+	mali_c55_unregister_stats(mali_c55);
 }
 
 static int mali_c55_register_entities(struct mali_c55 *mali_c55)
@@ -372,6 +383,10 @@ static int mali_c55_register_entities(struct mali_c55 *mali_c55)
 	if (ret)
 		goto err_unregister_entities;
 
+	ret = mali_c55_register_stats(mali_c55);
+	if (ret)
+		goto err_unregister_entities;
+
 	ret = mali_c55_create_links(mali_c55);
 	if (ret)
 		goto err_unregister_entities;
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
index ea8b7b866e7a..94876fba3353 100644
--- a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
@@ -564,6 +564,7 @@ int mali_c55_register_isp(struct mali_c55 *mali_c55)
 	isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
 	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
 	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
+	isp->pads[MALI_C55_ISP_PAD_SOURCE_3A].flags = MEDIA_PAD_FL_SOURCE;
 
 	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
 				     isp->pads);
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
index cb27abde2aa5..eb3719245ec3 100644
--- a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
@@ -68,6 +68,9 @@
 #define MALI_C55_VC_START(v)				((v) & 0xffff)
 #define MALI_C55_VC_SIZE(v)				(((v) & 0xffff) << 16)
 
+#define MALI_C55_REG_1024BIN_HIST			0x054a8
+#define MALI_C55_1024BIN_HIST_SIZE			4096
+
 /* Ping/Pong Configuration Space */
 #define MALI_C55_REG_BASE_ADDR				0x18e88
 #define MALI_C55_REG_BYPASS_0				0x18eac
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-stats.c b/drivers/media/platform/arm/mali-c55/mali-c55-stats.c
new file mode 100644
index 000000000000..aa40480ed814
--- /dev/null
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-stats.c
@@ -0,0 +1,350 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ARM Mali-C55 ISP Driver - 3A Statistics capture device
+ *
+ * Copyright (C) 2023 Ideas on Board Oy
+ */
+
+#include <linux/dmaengine.h>
+#include <linux/media/arm/mali-c55-config.h>
+#include <linux/spinlock.h>
+
+#include <media/media-entity.h>
+#include <media/v4l2-dev.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-fh.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-core.h>
+#include <media/videobuf2-dma-contig.h>
+
+#include "mali-c55-common.h"
+#include "mali-c55-registers.h"
+
+static unsigned int metering_space_addrs[] = {
+	[MALI_C55_CONFIG_PING] = 0x095AC,
+	[MALI_C55_CONFIG_PONG] = 0x2156C,
+};
+
+static int mali_c55_stats_enum_fmt_meta_cap(struct file *file, void *fh,
+					    struct v4l2_fmtdesc *f)
+{
+	if (f->index || f->type != V4L2_BUF_TYPE_META_CAPTURE)
+		return -EINVAL;
+
+	f->pixelformat = V4L2_META_FMT_MALI_C55_3A_STATS;
+
+	return 0;
+}
+
+static int mali_c55_stats_g_fmt_meta_cap(struct file *file, void *fh,
+					 struct v4l2_format *f)
+{
+	static const struct v4l2_meta_format mfmt = {
+		.dataformat = V4L2_META_FMT_MALI_C55_3A_STATS,
+		.buffersize = sizeof(struct mali_c55_stats_buffer)
+	};
+
+	if (f->type != V4L2_BUF_TYPE_META_CAPTURE)
+		return -EINVAL;
+
+	f->fmt.meta = mfmt;
+
+	return 0;
+}
+
+static int mali_c55_stats_querycap(struct file *file,
+				   void *priv, struct v4l2_capability *cap)
+{
+	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
+	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
+
+	return 0;
+}
+
+static const struct v4l2_ioctl_ops mali_c55_stats_v4l2_ioctl_ops = {
+	.vidioc_reqbufs = vb2_ioctl_reqbufs,
+	.vidioc_querybuf = vb2_ioctl_querybuf,
+	.vidioc_create_bufs = vb2_ioctl_create_bufs,
+	.vidioc_qbuf = vb2_ioctl_qbuf,
+	.vidioc_expbuf = vb2_ioctl_expbuf,
+	.vidioc_dqbuf = vb2_ioctl_dqbuf,
+	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+	.vidioc_streamon = vb2_ioctl_streamon,
+	.vidioc_streamoff = vb2_ioctl_streamoff,
+	.vidioc_enum_fmt_meta_cap = mali_c55_stats_enum_fmt_meta_cap,
+	.vidioc_g_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
+	.vidioc_s_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
+	.vidioc_try_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
+	.vidioc_querycap = mali_c55_stats_querycap,
+	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
+	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
+};
+
+static const struct v4l2_file_operations mali_c55_stats_v4l2_fops = {
+	.owner = THIS_MODULE,
+	.unlocked_ioctl = video_ioctl2,
+	.open = v4l2_fh_open,
+	.release = vb2_fop_release,
+	.poll = vb2_fop_poll,
+	.mmap = vb2_fop_mmap,
+};
+
+static int
+mali_c55_stats_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
+			   unsigned int *num_planes, unsigned int sizes[],
+			   struct device *alloc_devs[])
+{
+	struct mali_c55_stats *stats = vb2_get_drv_priv(q);
+
+	if (*num_planes && *num_planes > 1)
+		return -EINVAL;
+
+	if (sizes[0] && sizes[0] != sizeof(struct mali_c55_stats_buffer))
+		return -EINVAL;
+
+	*num_planes = 1;
+	sizes[0] = sizeof(struct mali_c55_stats_buffer);
+
+	if (stats->channel)
+		alloc_devs[0] = stats->channel->device->dev;
+
+	return 0;
+}
+
+static void mali_c55_stats_buf_queue(struct vb2_buffer *vb)
+{
+	struct mali_c55_stats *stats = vb2_get_drv_priv(vb->vb2_queue);
+	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
+	struct mali_c55_stats_buf *buf = container_of(vbuf,
+						struct mali_c55_stats_buf, vb);
+
+	vb2_set_plane_payload(vb, 0, sizeof(struct mali_c55_stats_buffer));
+	buf->segments_remaining = 2;
+	buf->failed = false;
+
+	spin_lock(&stats->buffers.lock);
+	list_add_tail(&buf->queue, &stats->buffers.queue);
+	spin_unlock(&stats->buffers.lock);
+}
+
+static void mali_c55_stats_stop_streaming(struct vb2_queue *q)
+{
+	struct mali_c55_stats *stats = vb2_get_drv_priv(q);
+	struct mali_c55_stats_buf *buf, *tmp;
+
+	spin_lock(&stats->buffers.lock);
+
+	list_for_each_entry_safe(buf, tmp, &stats->buffers.queue, queue) {
+		list_del(&buf->queue);
+		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
+	}
+
+	spin_unlock(&stats->buffers.lock);
+}
+
+static const struct vb2_ops mali_c55_stats_vb2_ops = {
+	.queue_setup = mali_c55_stats_queue_setup,
+	.buf_queue = mali_c55_stats_buf_queue,
+	.wait_prepare = vb2_ops_wait_prepare,
+	.wait_finish = vb2_ops_wait_finish,
+	.stop_streaming = mali_c55_stats_stop_streaming,
+};
+
+static void
+mali_c55_stats_metering_complete(void *param,
+				 const struct dmaengine_result *result)
+{
+	struct mali_c55_stats_buf *buf = param;
+
+	spin_lock(&buf->lock);
+
+	if (buf->failed)
+		goto out_unlock;
+
+	buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
+
+	if (result->result != DMA_TRANS_NOERROR) {
+		buf->failed = true;
+		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
+		goto out_unlock;
+	}
+
+	if (!--buf->segments_remaining)
+		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
+
+out_unlock:
+	spin_unlock(&buf->lock);
+}
+
+static int mali_c55_stats_dma_xfer(struct mali_c55_stats *stats, dma_addr_t src,
+				   dma_addr_t dst,
+				   struct mali_c55_stats_buf *buf,
+				   size_t length,
+				   void (*callback)(void *, const struct dmaengine_result *result))
+{
+	struct dma_async_tx_descriptor *tx;
+	dma_cookie_t cookie;
+
+	tx = dmaengine_prep_dma_memcpy(stats->channel, dst, src, length, 0);
+	if (!tx) {
+		dev_err(stats->mali_c55->dev, "failed to prep stats DMA\n");
+		return -EIO;
+	}
+
+	tx->callback_result = callback;
+	tx->callback_param = buf;
+
+	cookie = dmaengine_submit(tx);
+	if (dma_submit_error(cookie)) {
+		dev_err(stats->mali_c55->dev, "failed to submit stats DMA\n");
+		return -EIO;
+	}
+
+	dma_async_issue_pending(stats->channel);
+	return 0;
+}
+
+void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
+				enum mali_c55_config_spaces cfg_space)
+{
+	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
+	struct mali_c55_stats *stats = &mali_c55->stats;
+	struct mali_c55_stats_buf *buf = NULL;
+	dma_addr_t src, dst;
+	int ret;
+
+	spin_lock(&stats->buffers.lock);
+	if (!list_empty(&stats->buffers.queue)) {
+		buf = list_first_entry(&stats->buffers.queue,
+				       struct mali_c55_stats_buf, queue);
+		list_del(&buf->queue);
+	}
+	spin_unlock(&stats->buffers.lock);
+
+	if (!buf)
+		return;
+
+	buf->vb.sequence = mali_c55->isp.frame_sequence;
+
+	/*
+	 * There are infact two noncontiguous sections of the ISP's
+	 * memory space that hold statistics for 3a algorithms to use. A
+	 * section in each config space and a global section holding
+	 * histograms which is double buffered and so holds data for the
+	 * last frame. We need to read both.
+	 */
+	src = ctx->base + MALI_C55_REG_1024BIN_HIST;
+	dst = vb2_dma_contig_plane_dma_addr(&buf->vb.vb2_buf, 0);
+
+	ret = mali_c55_stats_dma_xfer(stats, src, dst, buf,
+				      MALI_C55_1024BIN_HIST_SIZE,
+				      mali_c55_stats_metering_complete);
+	if (ret)
+		goto err_fail_buffer;
+
+	src = ctx->base + metering_space_addrs[cfg_space];
+	dst += MALI_C55_1024BIN_HIST_SIZE;
+
+	ret = mali_c55_stats_dma_xfer(
+		stats, src, dst, buf,
+		sizeof(struct mali_c55_stats_buffer) - MALI_C55_1024BIN_HIST_SIZE,
+		mali_c55_stats_metering_complete);
+	if (ret) {
+		dmaengine_terminate_sync(stats->channel);
+		goto err_fail_buffer;
+	}
+
+	return;
+
+err_fail_buffer:
+	vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
+}
+
+void mali_c55_unregister_stats(struct mali_c55 *mali_c55)
+{
+	struct mali_c55_stats *stats = &mali_c55->stats;
+
+	if (!video_is_registered(&stats->vdev))
+		return;
+
+	vb2_video_unregister_device(&stats->vdev);
+	media_entity_cleanup(&stats->vdev.entity);
+	dma_release_channel(stats->channel);
+	mutex_destroy(&stats->lock);
+}
+
+int mali_c55_register_stats(struct mali_c55 *mali_c55)
+{
+	struct mali_c55_stats *stats = &mali_c55->stats;
+	struct video_device *vdev = &stats->vdev;
+	struct vb2_queue *vb2q = &stats->queue;
+	dma_cap_mask_t mask;
+	int ret;
+
+	mutex_init(&stats->lock);
+	INIT_LIST_HEAD(&stats->buffers.queue);
+
+	dma_cap_zero(mask);
+	dma_cap_set(DMA_MEMCPY, mask);
+
+	stats->channel = dma_request_channel(mask, 0, NULL);
+	if (!stats->channel) {
+		ret = -ENODEV;
+		goto err_destroy_mutex;
+	}
+
+	stats->pad.flags = MEDIA_PAD_FL_SINK;
+	ret = media_entity_pads_init(&stats->vdev.entity, 1, &stats->pad);
+	if (ret)
+		goto err_release_dma_channel;
+
+	vb2q->type = V4L2_BUF_TYPE_META_CAPTURE;
+	vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
+	vb2q->drv_priv = stats;
+	vb2q->mem_ops = &vb2_dma_contig_memops;
+	vb2q->ops = &mali_c55_stats_vb2_ops;
+	vb2q->buf_struct_size = sizeof(struct mali_c55_stats_buf);
+	vb2q->min_queued_buffers = 1;
+	vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+	vb2q->lock = &stats->lock;
+	vb2q->dev = mali_c55->dev;
+
+	ret = vb2_queue_init(vb2q);
+	if (ret) {
+		dev_err(mali_c55->dev, "stats vb2 queue init failed\n");
+		goto err_cleanup_entity;
+	}
+
+	strscpy(stats->vdev.name, "mali-c55 3a stats", sizeof(stats->vdev.name));
+	vdev->release = video_device_release_empty;
+	vdev->fops = &mali_c55_stats_v4l2_fops;
+	vdev->ioctl_ops = &mali_c55_stats_v4l2_ioctl_ops;
+	vdev->lock = &stats->lock;
+	vdev->v4l2_dev = &mali_c55->v4l2_dev;
+	vdev->queue = &stats->queue;
+	vdev->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING;
+	vdev->vfl_dir = VFL_DIR_RX;
+	video_set_drvdata(vdev, stats);
+
+	ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
+	if (ret) {
+		dev_err(mali_c55->dev,
+			"failed to register stats video device\n");
+		goto err_release_vb2q;
+	}
+
+	stats->mali_c55 = mali_c55;
+
+	return 0;
+
+err_release_vb2q:
+	vb2_queue_release(vb2q);
+err_cleanup_entity:
+	media_entity_cleanup(&stats->vdev.entity);
+err_release_dma_channel:
+	dma_release_channel(stats->channel);
+err_destroy_mutex:
+	mutex_destroy(&stats->lock);
+
+	return ret;
+}
-- 
2.34.1

[PATCH v5 11/16] media: platform: Fill stats buffer on ISP_START

[Daniel Scally]

On ISP_START, fill the stats buffer by reading out the metering space
in the ISP's memory. This is done for the non-active config just as
the dma transfer of the registers is. To acheive that, move the
checking of the current config outside of mali_c55_swap_next_config()
so we can use it for both functions.

Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- New patch

 .../platform/arm/mali-c55/mali-c55-core.c     | 35 ++++++++++++++-----
 1 file changed, 27 insertions(+), 8 deletions(-)

diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
index 9ea70010876c..2cf8b1169604 100644
--- a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
@@ -420,15 +420,9 @@ static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
 	return version;
 }
 
-static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
+static void mali_c55_swap_next_config(struct mali_c55 *mali_c55, u32 next_config)
 {
 	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
-	u32 curr_config, next_config;
-
-	curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
-	curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
-		      >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
-	next_config = curr_config ^ 1;
 
 	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
 			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
@@ -440,6 +434,7 @@ static irqreturn_t mali_c55_isr(int irq, void *context)
 {
 	struct device *dev = context;
 	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
+	u32 curr_config, next_config;
 	u32 interrupt_status;
 	unsigned int i, j;
 
@@ -465,7 +460,31 @@ static irqreturn_t mali_c55_isr(int irq, void *context)
 			for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
 				mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
 
-			mali_c55_swap_next_config(mali_c55);
+			/*
+			 * When the ISP starts a frame we have some work to do:
+			 *
+			 * 1. Copy over the config for the **next** frame
+			 * 2. Read out the metering stats for the **last** frame
+			 */
+
+			curr_config = mali_c55_read(mali_c55,
+						    MALI_C55_REG_PING_PONG_READ,
+						    false);
+			curr_config &= MALI_C55_REG_PING_PONG_READ_MASK;
+			curr_config >>= ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1;
+			next_config = curr_config ^ 1;
+
+			/*
+			 * The ordering of these two is currently important as
+			 * mali_c55_stats_fill_buffer() is asynchronous whereas
+			 * mali_c55_swap_next_config() is not.
+			 *
+			 * TODO: Should mali_c55_swap_next_config() be async?
+			 */
+			mali_c55_stats_fill_buffer(mali_c55,
+				next_config ? MALI_C55_CONFIG_PING :
+				MALI_C55_CONFIG_PONG);
+			mali_c55_swap_next_config(mali_c55, next_config);
 
 			break;
 		case MALI_C55_IRQ_ISP_DONE:
-- 
2.34.1

[PATCH v5 12/16] Documentation: mali-c55: Add Statistics documentation

[Daniel Scally]

Add documentation explaining the ability to capture statistics from
the mali-c55 driver's new V4L2 device, as well as the various tap
points from which those statistics can be drawn in the ISP's
processing flow. Additionally add a page detailing the new V4L2
meta format for the mali-c55 statistics.

Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- New patch

 Documentation/admin-guide/media/mali-c55.rst  | 60 ++++++++++++++++++-
 .../userspace-api/media/v4l/meta-formats.rst  |  1 +
 .../media/v4l/metafmt-arm-mali-c55.rst        | 28 +++++++++
 3 files changed, 88 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst

diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst
index cf4176cb1e44..b75437f6e96a 100644
--- a/Documentation/admin-guide/media/mali-c55.rst
+++ b/Documentation/admin-guide/media/mali-c55.rst
@@ -67,10 +67,11 @@ The driver has 4 V4L2 subdevices:
 - `mali_c55 resizer fr`: The Full-Resolution pipe resizer
 - `mali_c55 resizer ds`: The Downscale pipe resizer
 
-The driver has 2 V4L2 video devices:
+The driver has 3 V4L2 video devices:
 
 - `mali-c55 fr`: The full-resolution pipe's capture device
 - `mali-c55 ds`: The downscale pipe's capture device
+- `mali-c55 3a stats`: The 3A statistics capture device
 
 Frame sequences are synchronised across to two capture devices, meaning if one
 pipe is started later than the other the sequence numbers returned in its
@@ -326,6 +327,63 @@ configured, followed by formats in the appropriate places:
     # Set format on the video device and stream
     yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
 
+.. _mali-c55-3a-stats:
+
+Capturing ISP Statistics
+========================
+
+The ISP is capable of producing statistics for consumption by image processing
+algorithms running in userspace. These statistics can be captured by queueing
+buffers to the `mali-c55 3a stats` V4L2 Device whilst the ISP is streaming. Only
+the :ref:`V4L2_META_FMT_MALI_C55_3A_STATS <v4l2-meta-fmt-mali-c55-3a-stats>`
+format is supported, so no format-setting need be done:
+
+.. code-block:: none
+
+    # We assume the media graph has been configured to support RGB565 capture
+    # from the mali-c55 fr V4L2 Device, which is at /dev/video0. The statistics
+    # V4L2 device is at /dev/video3
+
+    yavta -f RGB565 -s 1920x1080 -c32 /dev/video0 && \
+    yavta -c10 -F /dev/video3
+
+The layout of the buffer is described by :c:type:`mali_c55_stats_buffer`,
+but broadly statistics are generated to support three image processing
+algorithms; AEXP (Auto-Exposure), AWB (Auto-White Balance) and AF (Auto-Focus).
+These stats can be drawn from various places in the Mali C55 ISP pipeline, known
+as "tap points". This high-level block diagram is intended to explain where in
+the processing flow the statistics can be drawn from::
+
+                  +--> AEXP-2            +----> AEXP-1          +--> AF-0
+                  |                      +----> AF-1            |
+                  |                      |                      |
+      +---------+ |   +--------------+   |   +--------------+   |
+      |  Input  +-+-->+ Digital Gain +---+-->+ Black Level  +---+---+
+      +---------+     +--------------+       +--------------+       |
+  +-----------------------------------------------------------------+
+  |
+  |   +--------------+ +---------+       +----------------+
+  +-->| Sinter Noise +-+  White  +--+--->|  Lens Shading  +--+---------------+
+      |   Reduction  | | Balance |  |    |                |  |               |
+      +--------------+ +---------+  |    +----------------+  |               |
+                                    +---> AEXP-0 (A)         +--> AEXP-0 (B) |
+  +--------------------------------------------------------------------------+
+  |
+  |   +----------------+      +--------------+  +----------------+
+  +-->|  Tone mapping  +-+--->| Demosaicing  +->+ Purple Fringe  +-+-----------+
+      |                | |    +--------------+  |   Correction   | |           |
+      +----------------+ +-> AEXP-IRIDIX        +----------------+ +---> AWB-0 |
+  +----------------------------------------------------------------------------+
+  |                    +-------------+        +-------------+
+  +------------------->|   Colour    +---+--->|    Output   |
+                       | Correction  |   |    |  Pipelines  |
+                       +-------------+   |    +-------------+
+                                         +-->  AWB-1
+
+At present all statistics are drawn from the 0th tap point for each algorithm;
+I.E. AEXP statistics from AEXP-0 (A), AWB statistics from AWB-0 and AF
+statistics from AF-0. In the future this will be configurable.
+
 References
 ==========
 .. [1] https://git.linuxtv.org/v4l-utils.git/
diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index c23aac823d2c..73c05cc35321 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
@@ -15,6 +15,7 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-d4xx
     metafmt-generic
     metafmt-intel-ipu3
+    metafmt-arm-mali-c55
     metafmt-rkisp1
     metafmt-uvc
     metafmt-vivid
diff --git a/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst b/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
new file mode 100644
index 000000000000..219a5dd42d70
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
@@ -0,0 +1,28 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+.. _v4l2-meta-fmt-mali-c55-3a-stats:
+
+****************************************
+V4L2_META_FMT_MALI_C55_3A_STATS ('C55S')
+****************************************
+
+3A Statistics
+=============
+The ISP device collects different statistics over an input bayer frame. Those
+statistics can be obtained by userspace from the
+:ref:`mali-c55 3a stats <mali-c55-3a-stats>` metadata capture video node, using
+the :c:type:`v4l2_meta_format` interface. The buffer contains a single instance
+of the C structure :c:type:`mali_c55_stats_buffer` defined in
+``mali-c55-config.h``, so the structure can be obtained from the buffer by:
+
+.. code-block:: C
+
+	struct mali_c55_stats_buffer *stats =
+		(struct mali_c55_stats_buffer *)buf;
+
+For details of the statistics see :c:type:`mali_c55_stats_buffer`.
+
+Arm Mali-C55 uAPI data types
+============================
+
+.. kernel-doc:: include/uapi/linux/media/arm/mali-c55-config.h
-- 
2.34.1

[PATCH v5 13/16] media: mali-c55: Add image formats for Mali-C55 parameters buffer

[Daniel Scally]

From: Jacopo Mondi <jacopo.mondi@ideasonboard.com>

Add a new V4L2 meta format code for the Mali-C55 parameters

Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- New patch

 drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
 include/uapi/linux/videodev2.h       | 1 +
 2 files changed, 2 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 0d00b0476762..4e3b5e16571c 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1456,6 +1456,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_META_FMT_VIVID:       descr = "Vivid Metadata"; break;
 	case V4L2_META_FMT_RK_ISP1_PARAMS:	descr = "Rockchip ISP1 3A Parameters"; break;
 	case V4L2_META_FMT_RK_ISP1_STAT_3A:	descr = "Rockchip ISP1 3A Statistics"; break;
+	case V4L2_META_FMT_MALI_C55_PARAMS:	descr = "ARM Mali-C55 ISP Parameters"; break;
 	case V4L2_META_FMT_MALI_C55_3A_STATS:	descr = "ARM Mali-C55 ISP 3A Statistics"; break;
 	case V4L2_PIX_FMT_NV12_8L128:	descr = "NV12 (8x128 Linear)"; break;
 	case V4L2_PIX_FMT_NV12M_8L128:	descr = "NV12M (8x128 Linear)"; break;
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 021424fdaf68..e63d5e76d21e 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -841,6 +841,7 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_MALI_C55_PARAMS	v4l2_fourcc('C', '5', '5', 'P') /* ARM Mali-C55 Parameters */
 #define V4L2_META_FMT_MALI_C55_3A_STATS	v4l2_fourcc('C', '5', '5', 'S') /* ARM Mali-C55 3A Statistics */
 
 #ifdef __KERNEL__
-- 
2.34.1

[PATCH v5 14/16] media: uapi: Add parameters structs to mali-c55-config.h

[Daniel Scally]

Add structures describing the ISP parameters to mali-c55-config.h

Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- New patch

 .../uapi/linux/media/arm/mali-c55-config.h    | 669 ++++++++++++++++++
 1 file changed, 669 insertions(+)

diff --git a/include/uapi/linux/media/arm/mali-c55-config.h b/include/uapi/linux/media/arm/mali-c55-config.h
index 8fb89af6c874..fce14bc74f4a 100644
--- a/include/uapi/linux/media/arm/mali-c55-config.h
+++ b/include/uapi/linux/media/arm/mali-c55-config.h
@@ -179,4 +179,673 @@ struct mali_c55_stats_buffer {
 	__u32 reserved3[15];
 } __attribute__((packed));
 
+/**
+ * enum mali_c55_param_buffer_version - Mali-C55 parameters block versioning
+ *
+ * @MALI_C55_PARAM_BUFFER_V0: First version of Mali-C55 parameters block
+ */
+enum mali_c55_param_buffer_version {
+	MALI_C55_PARAM_BUFFER_V0,
+};
+
+/**
+ * enum mali_c55_param_block_type - Enumeration of Mali-C55 parameter blocks
+ *
+ * This enumeration defines the types of Mali-C55 parameters block. Each block
+ * configures a specific processing block of the Mali-C55 ISP. The block
+ * type allows the driver to correctly interpret the parameters block data.
+ *
+ * It is the responsibility of userspace to correctly set the type of each
+ * parameters block.
+ *
+ * @MALI_C55_PARAM_BLOCK_SENSOR_OFFS: Sensor pre-shading black level offset
+ * @MALI_C55_PARAM_BLOCK_AEXP_HIST: Auto-exposure 1024-bin histogram
+ *				    configuration
+ * @MALI_C55_PARAM_BLOCK_AEXP_IHIST: Post-Iridix auto-exposure 1024-bin
+ *				     histogram configuration
+ * @MALI_C55_PARAM_BLOCK_AEXP_HIST_WEIGHTS: Auto-exposure 1024-bin histogram
+ *					    weighting
+ * @MALI_C55_PARAM_BLOCK_AEXP_IHIST_WEIGHTS: Post-Iridix auto-exposure 1024-bin
+ *					     histogram weighting
+ * @MALI_C55_PARAM_BLOCK_DIGITAL_GAIN: Digital gain
+ * @MALI_C55_PARAM_BLOCK_AWB_GAINS: Auto-white balance gains
+ * @MALI_C55_PARAM_BLOCK_AWB_CONFIG: Auto-white balance statistics config
+ * @MALI_C55_PARAM_BLOCK_AWB_GAINS_AEXP: Auto-white balance gains for AEXP-0 tap
+ * @MALI_C55_PARAM_MESH_SHADING_CONFIG : Mesh shading tables configuration
+ * @MALI_C55_PARAM_MESH_SHADING_SELECTION: Mesh shading table selection
+ * @MALI_C55_PARAM_BLOCK_SENTINEL: First non-valid block index
+ */
+enum mali_c55_param_block_type {
+	MALI_C55_PARAM_BLOCK_SENSOR_OFFS,
+	MALI_C55_PARAM_BLOCK_AEXP_HIST,
+	MALI_C55_PARAM_BLOCK_AEXP_IHIST,
+	MALI_C55_PARAM_BLOCK_AEXP_HIST_WEIGHTS,
+	MALI_C55_PARAM_BLOCK_AEXP_IHIST_WEIGHTS,
+	MALI_C55_PARAM_BLOCK_DIGITAL_GAIN,
+	MALI_C55_PARAM_BLOCK_AWB_GAINS,
+	MALI_C55_PARAM_BLOCK_AWB_CONFIG,
+	MALI_C55_PARAM_BLOCK_AWB_GAINS_AEXP,
+	MALI_C55_PARAM_MESH_SHADING_CONFIG,
+	MALI_C55_PARAM_MESH_SHADING_SELECTION,
+	MALI_C55_PARAM_BLOCK_SENTINEL,
+};
+
+/**
+ * struct mali_c55_params_block_header - Mali-C55 parameter block header
+ *
+ * This structure represents the common part of all the ISP configuration
+ * blocks. Each parameters block shall embed an instance of this structure type
+ * as its first member, followed by the block-specific configuration data. The
+ * driver inspects this common header to discern the block type and its size and
+ * properly handle the block content by casting it to the correct block-specific
+ * type.
+ *
+ * The @type field is one of the values enumerated by
+ * :c:type:`mali_c55_param_block_type` and specifies how the data should be
+ * interpreted by the driver. The @size field specifies the size of the
+ * parameters block and is used by the driver for validation purposes. The
+ * @enabled field specifies if the ISP block should be enabled (and configured
+ * according to the provided parameters) or disabled.
+ *
+ * .. code-block:: c
+ *
+ *	struct mali_c55_params_block_header *block = ...;
+ *
+ *	switch (block->type) {
+ *	case MALI_C55_PARAM_BLOCK_SENSOR_OFFS:
+ *		struct mali_c55_params_sensor_off_preshading *sensor_offs =
+ *			(struct mali_c55_params_sensor_off_preshading *)block;
+ *
+ *		if (block->size !=
+ *		    sizeof(struct mali_c55_params_sensor_off_preshading))
+ *			return -EINVAL;
+ *
+ *		handle_sensor_offs(sensor_offs);
+ *		break;
+ *
+ *	case MALI_C55_PARAM_BLOCK_AEXP_HIST:
+ *		struct mali_c55_params_aexp_hist *aexp_hist =
+ *			(struct mali_c55_params_aexp_hist)block;
+ *
+ *		if (block->size != sizeof(mali_c55_params_aexp_hist))
+ *			return -EINVAL;
+ *
+ *		handle_aexp_hist(aesp_hist);
+ *		break;
+ *
+ *	...
+ *
+ *	}
+ *
+ * Userspace is responsible for correctly populating the parameters block header
+ * fields (@type, @enabled and @size) and correctly populate the block-specific
+ * parameters.
+ *
+ * For example:
+ *
+ * .. code-block:: c
+ *
+ *	void populate_sensor_offs(struct mali_c55_params_block_header *block) {
+ *		block->type = MALI_C55_PARAM_BLOCK_SENSOR_OFFS;
+ *		block->enabled = true;
+ *		block->size = sizeof(struct mali_c55_params_sensor_off_preshading);
+ *
+ *		struct mali_c55_params_sensor_off_preshading *sensor_offs =
+ *			(struct mali_c55_params_sensor_off_preshading *)block;
+ *
+ *		sensor_offs->chan00 = offset00;
+ *		sensor_offs->chan01 = offset01;
+ *		sensor_offs->chan10 = offset10;
+ *		sensor_offs->chan11 = offset11;
+ *	}
+ *
+ * @type: The parameters block type (enum mali_c55_param_block_type)
+ * @enabled: Block enabled/disabled flag
+ * @size: Size (in bytes) of the parameters block
+ */
+struct mali_c55_params_block_header {
+	enum mali_c55_param_block_type type;
+	bool enabled;
+	size_t size;
+};
+
+/**
+ * struct mali_c55_params_sensor_off_preshading - offset subtraction for each
+ *						  color channel
+ *
+ * Provides removal of the sensor black level from the sensor data. Separate
+ * offsets are provided for each of the four Bayer component color channels
+ * which are defaulted to R, Gr, Gb, B.
+ *
+ * @header: The Mali-C55 parameters block header
+ * @chan00: Offset for color channel 00 (default: R)
+ * @chan01: Offset for color channel 01 (default: Gr)
+ * @chan10: Offset for color channel 10 (default: Gb)
+ * @chan11: Offset for color channel 11 (default: B)
+ */
+struct mali_c55_params_sensor_off_preshading {
+	struct mali_c55_params_block_header header;
+	__u32 chan00;
+	__u32 chan01;
+	__u32 chan10;
+	__u32 chan11;
+};
+
+/**
+ * enum mali_c55_aexp_hist_tap_points - Tap points for the AEXP histogram
+ * @MALI_C55_AEXP_HIST_TAP_WB: After static white balance
+ * @MALI_C55_AEXP_HIST_TAP_FS: After WDR Frame Stitch
+ * @MALI_C55_AEXP_HIST_TAP_TPG: After the test pattern generator
+ */
+enum mali_c55_aexp_hist_tap_points {
+	MALI_C55_AEXP_HIST_TAP_WB = 0,
+	MALI_C55_AEXP_HIST_TAP_FS,
+	MALI_C55_AEXP_HIST_TAP_TPG,
+};
+
+/**
+ * enum mali_c55_aexp_skip_x - Horizontal pixel skipping
+ * @MALI_C55_AEXP_SKIP_X_EVERY_2ND: Collect every 2nd pixel horizontally
+ * @MALI_C55_AEXP_SKIP_X_EVERY_3RD: Collect every 3rd pixel horizontally
+ * @MALI_C55_AEXP_SKIP_X_EVERY_4TH: Collect every 4th pixel horizontally
+ * @MALI_C55_AEXP_SKIP_X_EVERY_5TH: Collect every 5th pixel horizontally
+ * @MALI_C55_AEXP_SKIP_X_EVERY_8TH: Collect every 8th pixel horizontally
+ * @MALI_C55_AEXP_SKIP_X_EVERY_9TH: Collect every 9th pixel horizontally
+ */
+enum mali_c55_aexp_skip_x {
+	MALI_C55_AEXP_SKIP_X_EVERY_2ND,
+	MALI_C55_AEXP_SKIP_X_EVERY_3RD,
+	MALI_C55_AEXP_SKIP_X_EVERY_4TH,
+	MALI_C55_AEXP_SKIP_X_EVERY_5TH,
+	MALI_C55_AEXP_SKIP_X_EVERY_8TH,
+	MALI_C55_AEXP_SKIP_X_EVERY_9TH
+};
+
+/**
+ * enum mali_c55_aexp_skip_y - Vertical pixel skipping
+ * @MALI_C55_AEXP_SKIP_Y_ALL: Collect every single pixel vertically
+ * @MALI_C55_AEXP_SKIP_Y_EVERY_2ND: Collect every 2nd pixel vertically
+ * @MALI_C55_AEXP_SKIP_Y_EVERY_3RD: Collect every 3rd pixel vertically
+ * @MALI_C55_AEXP_SKIP_Y_EVERY_4TH: Collect every 4th pixel vertically
+ * @MALI_C55_AEXP_SKIP_Y_EVERY_5TH: Collect every 5th pixel vertically
+ * @MALI_C55_AEXP_SKIP_Y_EVERY_8TH: Collect every 8th pixel vertically
+ * @MALI_C55_AEXP_SKIP_Y_EVERY_9TH: Collect every 9th pixel vertically
+ */
+enum mali_c55_aexp_skip_y {
+	MALI_C55_AEXP_SKIP_Y_ALL,
+	MALI_C55_AEXP_SKIP_Y_EVERY_2ND,
+	MALI_C55_AEXP_SKIP_Y_EVERY_3RD,
+	MALI_C55_AEXP_SKIP_Y_EVERY_4TH,
+	MALI_C55_AEXP_SKIP_Y_EVERY_5TH,
+	MALI_C55_AEXP_SKIP_Y_EVERY_8TH,
+	MALI_C55_AEXP_SKIP_Y_EVERY_9TH
+};
+
+/**
+ * enum mali_c55_aexp_row_column_offset - Start from the first or second row or
+ *					  column
+ * @MALI_C55_AEXP_FIRST_ROW_OR_COL:	Start from the first row / column
+ * @MALI_C55_AEXP_SECOND_ROW_OR_COL:	Start from the second row / column
+ */
+enum mali_c55_aexp_row_column_offset {
+	MALI_C55_AEXP_FIRST_ROW_OR_COL = 1,
+	MALI_C55_AEXP_SECOND_ROW_OR_COL = 2,
+};
+
+/**
+ * enum mali_c55_aexp_hist_plane_mode - Mode for the AEXP Histograms
+ * @MALI_C55_AEXP_HIST_COMBINED: All color planes in one 1024-bin histogram
+ * @MALI_C55_AEXP_HIST_SEPARATE: Each color plane in one 256-bin histogram with a bin width of 16
+ * @MALI_C55_AEXP_HIST_FOCUS_00: Top left plane in the first bank, rest in second bank
+ * @MALI_C55_AEXP_HIST_FOCUS_01: Top right plane in the first bank, rest in second bank
+ * @MALI_C55_AEXP_HIST_FOCUS_10: Bottom left plane in the first bank, rest in second bank
+ * @MALI_C55_AEXP_HIST_FOCUS_11: Bottom right plane in the first bank, rest in second bank
+ *
+ * In the "focus" modes statistics are collected into two 512-bin histograms
+ * with a bin width of 8. One colour plane is in the first histogram with the
+ * remainder combined into the second. The four options represent which of the
+ * four positions in a bayer pattern are the focused plane.
+ */
+enum mali_c55_aexp_hist_plane_mode {
+	MALI_C55_AEXP_HIST_COMBINED = 0,
+	MALI_C55_AEXP_HIST_SEPARATE = 1,
+	MALI_C55_AEXP_HIST_FOCUS_00 = 4,
+	MALI_C55_AEXP_HIST_FOCUS_01 = 5,
+	MALI_C55_AEXP_HIST_FOCUS_10 = 6,
+	MALI_C55_AEXP_HIST_FOCUS_11 = 7,
+};
+
+/**
+ * struct mali_c55_params_aexp_hist - configuration for AEXP metering hists
+ *
+ * This struct allows users to configure the 1024-bin AEXP histograms. Broadly
+ * speaking the parameters allow you to mask particular regions of the image and
+ * to select different kinds of histogram.
+ *
+ * @header:		The Mali-C55 parameters block header
+ * @skip_x:		Horizontal decimation. See enum mali_c55_aexp_skip_x
+ * @offset_x:		Column to start from. See enum mali_c55_aexp_row_column_offset
+ * @skip_y:		Vertical decimation. See enum mali_c55_aexp_skip_y
+ * @offset_y:		Row to start from. See enum mali_c55_aexp_row_column_offset
+ * @scale_bottom:	scale of bottom half of range: 0=1x ,1=2x, 2=4x, 4=8x, 4=16x
+ * @scale_top:		scale of top half of range: 0=1x ,1=2x, 2=4x, 4=8x, 4=16x
+ * @plane_mode:		Plane separation mode. See enum mali_c55_aexp_hist_plane_mode
+ * @tap_point:		Tap point for histogram from enum mali_c55_aexp_hist_tap_points.
+ *			This parameter is unused for the post-Iridix Histogram
+ */
+struct mali_c55_params_aexp_hist {
+	struct mali_c55_params_block_header header;
+	__u8 skip_x;
+	__u8 offset_x;
+	__u8 skip_y;
+	__u8 offset_y;
+	__u8 scale_bottom;
+	__u8 scale_top;
+	__u8 plane_mode;
+	__u8 tap_point;
+};
+
+/**
+ * struct mali_c55_params_aexp_weights - Array of weights for AEXP metering
+ *
+ * This struct allows users to configure the weighting for both of the 1024-bin
+ * AEXP histograms. The pixel data collected for each zone is multiplied by the
+ * corresponding weight from this array, which may be zero if the intention is
+ * to mask off the zone entirely.
+ *
+ * @header:		The Mali-C55 parameters block header
+ * @nodes_used_horiz:	Number of active zones horizontally [0..15]
+ * @nodes_used_vert:	Number of active zones vertically [0..15]
+ * @zone_weights:	Zone weighting. Index is row*col where 0,0 is the top
+ * 			left zone continuing in raster order. Each zone can be
+ *			weighted in the range [0..15]. The number of rows and
+ *			columns is defined by @nodes_used_vert and
+ *			@nodes_used_horiz
+ */
+struct mali_c55_params_aexp_weights {
+	struct mali_c55_params_block_header header;
+	__u8 nodes_used_horiz;
+	__u8 nodes_used_vert;
+	__u8 zone_weights[MALI_C55_MAX_ZONES];
+};
+
+/**
+ * struct mali_c55_params_digital_gain - Digital gain value
+ *
+ * This struct carries a digital gain value to set in the ISP
+ *
+ * @header:	The Mali-C55 parameters block header
+ * @gain:	The digital gain value to apply, in Q5.8 format.
+ */
+struct mali_c55_params_digital_gain {
+	struct mali_c55_params_block_header header;
+	__u16 gain;
+};
+
+/**
+ * enum mali_c55_awb_stats_mode - Statistics mode for AWB
+ * @MALI_C55_AWB_MODE_GRBR: Statistics collected as Green/Red and Blue/Red ratios
+ * @MALI_C55_AWB_MODE_RGBG: Statistics collected as Red/Green and Blue/Green ratios
+ */
+enum mali_c55_awb_stats_mode {
+       MALI_C55_AWB_MODE_GRBR = 0,
+       MALI_C55_AWB_MODE_RGBG,
+};
+
+/**
+ * struct mali_c55_params_awb_gains - Gain settings for auto white balance
+ *
+ * This struct allows users to configure the gains for auto-white balance. There
+ * are four gain settings corresponding to each colour channel in the bayer
+ * domain. Although named generically, the association between the gain applied
+ * and the colour channel is done automatically within the ISP depending on the
+ * input format, and so the following mapping always holds true::
+ *
+ *	gain00 = R
+ *	gain01 = Gr
+ *	gain10 = Gb
+ *	gain11 = B
+ *
+ * All of the gains are stored in Q4.8 format.
+ *
+ * @header:	The Mali-C55 parameters block header
+ * @gain00:	Multiplier for colour channel 00
+ * @gain01:	Multiplier for colour channel 01
+ * @gain10:	Multiplier for colour channel 10
+ * @gain11:	Multiplier for colour channel 11
+ */
+struct mali_c55_params_awb_gains {
+	struct mali_c55_params_block_header header;
+	__u16 gain00;
+	__u16 gain01;
+	__u16 gain10;
+	__u16 gain11;
+};
+
+/**
+ * enum mali_c55_params_awb_tap_points - Tap points for the AWB statistics
+ * @MALI_C55_AWB_STATS_TAP_PF: Immediately after the Purple Fringe block
+ * @MALI_C55_AWB_STATS_TAP_CNR: Immediately after the CNR block
+ */
+enum mali_c55_params_awb_tap_points {
+	MALI_C55_AWB_STATS_TAP_PF = 0,
+	MALI_C55_AWB_STATS_TAP_CNR,
+};
+
+/**
+ * struct mali_c55_params_awb_config - Stats settings for auto-white balance
+ *
+ * This struct allows the configuration of the statistics generated for auto
+ * white balance. Pixel intensity limits can be set to exclude overly bright or
+ * dark regions of an image from the statistics entirely. Colour ratio minima
+ * and maxima can be set to discount pixels who's ratios fall outside the
+ * defined boundaries; there are two sets of registers to do this - the
+ * "min/max" ratios which bound a region and the "high/low" ratios which further
+ * trim the upper and lower ratios. For example with the boundaries configured
+ * as follows, only pixels whos colour ratios falls into the region marked "A"
+ * would be counted::
+ *
+ *	                                                          cr_high
+ *	    2.0 |                                                   |
+ *	        |               cb_max --> _________________________v_____
+ *	    1.8 |                         |                         \    |
+ *	        |                         |                          \   |
+ *	    1.6 |                         |                           \  |
+ *	        |                         |                            \ |
+ *	 c  1.4 |               cb_low -->|\              A             \|<--  cb_high
+ *	 b      |                         | \                            |
+ *	    1.2 |                         |  \                           |
+ *	 r      |                         |   \                          |
+ *	 a  1.0 |              cb_min --> |____\_________________________|
+ *	 t      |                         ^    ^                         ^
+ *	 i  0.8 |                         |    |                         |
+ *	 o      |                      cr_min  |                       cr_max
+ *	 s  0.6 |                              |
+ *	        |                             cr_low
+ *	    0.4 |
+ *	        |
+ *	    0.2 |
+ *	        |
+ *	    0.0 |_______________________________________________________________
+ *	        0.0   0.2   0.4   0.6   0.8   1.0   1.2   1.4   1.6   1.8   2.0
+ *	                                   cr ratios
+ *
+ * @header:		The Mali-C55 parameters block header
+ * @tap_point:		The tap point from enum mali_c55_params_awb_tap_points
+ * @stats_mode:		AWB statistics collection mode, see :c:type:`mali_c55_awb_stats_mode`
+ * @white_level:	Upper pixel intensity (I.E. raw pixel values) limit
+ * @black_level:	Lower pixel intensity (I.E. raw pixel values) limit
+ * @cr_max:		Maximum R/G ratio (Q4.8 format)
+ * @cr_min:		Minimum R/G ratio (Q4.8 format)
+ * @cb_max:		Maximum B/G ratio (Q4.8 format)
+ * @cb_min:		Minimum B/G ratio (Q4.8 format)
+ * @nodes_used_horiz:	Number of active zones horizontally [0..15]
+ * @nodes_used_vert:	Number of active zones vertically [0..15]
+ * @cr_high:		R/G ratio trim high (Q4.8 format)
+ * @cr_low:		R/G ratio trim low (Q4.8 format)
+ * @cb_high:		B/G ratio trim high (Q4.8 format)
+ * @cb_low:		B/G ratio trim low (Q4.8 format)
+ */
+struct mali_c55_params_awb_config {
+	struct mali_c55_params_block_header header;
+	__u8 tap_point;
+	__u8 stats_mode;
+	__u16 white_level;
+	__u16 black_level;
+	__u16 cr_max;
+	__u16 cr_min;
+	__u16 cb_max;
+	__u16 cb_min;
+	__u8 nodes_used_horiz;
+	__u8 nodes_used_vert;
+	__u16 cr_high;
+	__u16 cr_low;
+	__u16 cb_high;
+	__u16 cb_low;
+};
+
+#define MALI_C55_NUM_MESH_SHADING_ELEMENTS 3072
+
+/**
+ * struct mali_c55_params_mesh_shading_config - Mesh shading configuration
+ *
+ * The mesh shading correction module allows programming a separate table of
+ * either 16x16 or 32x32 node coefficients for 3 different light sources. The
+ * final correction coefficients applied are computed by blending the
+ * coefficients from two tables together.
+ *
+ * A page of 1024 32-bit integers is associated to each colour channel, with
+ * pages stored consecutively in memory. Each 32-bit integer packs 3 8-bit
+ * correction coefficients for a single node, one for each of the three light
+ * sources. The 8 most significant bits are unused. The following table
+ * describes the layout::
+ *
+ *	+----------- Page (Colour Plane) 0 -------------+
+ *	| @mesh[i]  | Mesh Point | Bits  | Light Source |
+ *	+-----------+------------+-------+--------------+
+ *	|         0 |        0,0 | 16,23 | LS2          |
+ *	|           |            | 08-15 | LS1          |
+ *	|           |            | 00-07 | LS0          |
+ *	+-----------+------------+-------+--------------+
+ *	|         1 |        0,1 | 16,23 | LS2          |
+ *	|           |            | 08-15 | LS1          |
+ *	|           |            | 00-07 | LS0          |
+ *	+-----------+------------+-------+--------------+
+ *	|       ... |        ... | ...   | ...          |
+ *	+-----------+------------+-------+--------------+
+ *	|      1023 |      31,31 | 16,23 | LS2          |
+ *	|           |            | 08-15 | LS1          |
+ *	|           |            | 00-07 | LS0          |
+ *	+----------- Page (Colour Plane) 1 -------------+
+ *	| @mesh[i]  | Mesh Point | Bits  | Light Source |
+ *	+-----------+------------+-------+--------------+
+ *	|      1024 |        0,0 | 16,23 | LS2          |
+ *	|           |            | 08-15 | LS1          |
+ *	|           |            | 00-07 | LS0          |
+ *	+-----------+------------+-------+--------------+
+ *	|      1025 |        0,1 | 16,23 | LS2          |
+ *	|           |            | 08-15 | LS1          |
+ *	|           |            | 00-07 | LS0          |
+ *	+-----------+------------+-------+--------------+
+ *	|       ... |        ... | ...   | ...          |
+ *	+-----------+------------+-------+--------------+
+ *	|      2047 |      31,31 | 16,23 | LS2          |
+ *	|           |            | 08-15 | LS1          |
+ *	|           |            | 00-07 | LS0          |
+ *	+----------- Page (Colour Plane) 2 -------------+
+ *	| @mesh[i]  | Mesh Point | Bits  | Light Source |
+ *	+-----------+------------+-------+--------------+
+ *	|      2048 |        0,0 | 16,23 | LS2          |
+ *	|           |            | 08-15 | LS1          |
+ *	|           |            | 00-07 | LS0          |
+ *	+-----------+------------+-------+--------------+
+ *	|      2049 |        0,1 | 16,23 | LS2          |
+ *	|           |            | 08-15 | LS1          |
+ *	|           |            | 00-07 | LS0          |
+ *	+-----------+------------+-------+--------------+
+ *	|       ... |        ... | ...   | ...          |
+ *	+-----------+------------+-------+--------------+
+ *	|      3071 |      31,31 | 16,23 | LS2          |
+ *	|           |            | 08-15 | LS1          |
+ *	|           |            | 00-07 | LS0          |
+ *	+-----------+------------+-------+--------------+
+ *
+ * The @mesh_scale member determines the precision and minimum and maximum gain.
+ * For example if @mesh_scale is 0 and therefore selects 0 - 2x gain, a value of
+ * 0 in a coefficient means 0.0 gain, a value of 128 means 1.0 gain and 255
+ * means 2.0 gain.
+ *
+ * @header:		The Mali-C55 parameters block header
+ * @mesh_show:		Output the mesh data rather than image data
+ * @mesh_scale:		Set the precision and maximum gain range of mesh shading
+ *				- 0 = 0-2x gain
+ *				- 1 = 0-4x gain
+ *				- 2 = 0-8x gain
+ *				- 3 = 0-16x gain
+ *				- 4 = 1-2x gain
+ *				- 5 = 1-3x gain
+ *				- 6 = 1-5x gain
+ *				- 7 = 1-9x gain
+ * @mesh_page_r:	Mesh page select for red colour plane [0..2]
+ * @mesh_page_g:	Mesh page select for green colour plane [0..2]
+ * @mesh_page_b:	Mesh page select for blue colour plane [0..2]
+ * @mesh_width:		Number of horizontal nodes minus 1 [15,31]
+ * @mesh_height:	Number of vertical nodes minus 1 [15,31]
+ * @mesh:		Mesh shading correction tables
+ */
+struct mali_c55_params_mesh_shading_config {
+	struct mali_c55_params_block_header header;
+	bool mesh_show;
+	__u8 mesh_scale;
+	__u8 mesh_page_r;
+	__u8 mesh_page_g;
+	__u8 mesh_page_b;
+	__u8 mesh_width;
+	__u8 mesh_height;
+	__u32 mesh[MALI_C55_NUM_MESH_SHADING_ELEMENTS];
+};
+
+/** enum mali_c55_params_mesh_alpha_bank - Mesh shading table bank selection
+ * @MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS1 - Select Light Sources 0 and 1
+ * @MALI_C55_MESH_ALPHA_BANK_LS1_AND_LS2 - Select Light Sources 1 and 2
+ * @MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS2 - Select Light Sources 0 and 2
+ */
+enum mali_c55_params_mesh_alpha_bank {
+	MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS1 = 0,
+	MALI_C55_MESH_ALPHA_BANK_LS1_AND_LS2 = 1,
+	MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS2 = 4
+};
+
+/**
+ * struct mali_c55_params_mesh_shading_selection - Mesh table selection
+ *
+ * The module computes the final correction coefficients by blending the ones
+ * from two light source tables, which are selected (independently for each
+ * colour channel) by the @mesh_alpha_bank_r/g/b fields.
+ *
+ * The final blended coefficients for each node are calculated using the
+ * following equation:
+ *
+ *     Final coefficient = (a x LS\ :sub:`b`\ + (256 - a) x LS\ :sub:`a`\) / 256
+ *
+ * Where a is the @mesh_alpha_r/g/b value, and LS\ :sub:`a`\ and LS\ :sub:`b`\
+ * are the node cofficients for the two tables selected by the
+ * @mesh_alpha_bank_r/g/b value.
+ *
+ * The scale of the applied correction may also be controlled by tuning the
+ * @mesh_strength member. This is a modifier to the final coefficients which can
+ * be used to globally reduce the gains applied.
+ *
+ * @header:		The Mali-C55 parameters block header
+ * @mesh_alpha_bank_r:	Red mesh table select (c:type:`enum mali_c55_params_mesh_alpha_bank`)
+ * @mesh_alpha_bank_g:	Green mesh table select (c:type:`enum mali_c55_params_mesh_alpha_bank`)
+ * @mesh_alpha_bank_b:	Blue mesh table select (c:type:`enum mali_c55_params_mesh_alpha_bank`)
+ * @mesh_alpha_r:	Blend coefficient for R [0..255]
+ * @mesh_alpha_g:	Blend coefficient for G [0..255]
+ * @mesh_alpha_b:	Blend coefficient for B [0..255]
+ * @mesh_strength:	Mesh strength in Q4.12 format [0..4096]
+ */
+struct mali_c55_params_mesh_shading_selection {
+	struct mali_c55_params_block_header header;
+	__u8 mesh_alpha_bank_r;
+	__u8 mesh_alpha_bank_g;
+	__u8 mesh_alpha_bank_b;
+	__u8 mesh_alpha_r;
+	__u8 mesh_alpha_g;
+	__u8 mesh_alpha_b;
+	__u16 mesh_strength;
+};
+
+/**
+ * define MALI_C55_PARAMS_MAX_SIZE - Maximum size of all Mali C55 Parameters
+ *
+ * Though the parameters for the Mali-C55 are passed as optional blocks, the
+ * driver still needs to know the absolute maximum size so that it can allocate
+ * a buffer sized appropriately to accomodate userspace attempting to set all
+ * possible parameters in a single frame.
+ */
+#define MALI_C55_PARAMS_MAX_SIZE				\
+	sizeof(struct mali_c55_params_sensor_off_preshading) + 	\
+	sizeof(struct mali_c55_params_aexp_hist) +		\
+	sizeof(struct mali_c55_params_aexp_weights) +		\
+	sizeof(struct mali_c55_params_aexp_hist) +		\
+	sizeof(struct mali_c55_params_aexp_weights) +		\
+	sizeof(struct mali_c55_params_digital_gain) +		\
+	sizeof(struct mali_c55_params_awb_gains) +		\
+	sizeof(struct mali_c55_params_awb_config) +		\
+	sizeof(struct mali_c55_params_awb_gains) +		\
+	sizeof(struct mali_c55_params_mesh_shading_config) +	\
+	sizeof(struct mali_c55_params_mesh_shading_selection)
+
+/**
+ * struct mali_c55_params_buffer - 3A configuration parameters
+ *
+ * This struct contains the configuration parameters of the Mali-C55 ISP
+ * algorithms, serialized by userspace into an opaque data buffer. Each
+ * configuration parameter block is represented by a block-specific structure
+ * which contains a :c:type:`mali_c55_params_block_header` entry as first
+ * member. Userspace populates the @data buffer with configuration parameters
+ * for the blocks that it intends to configure. As a consequence, the data
+ * buffer effective size changes according to the number of ISP blocks that
+ * userspace intends to configure.
+ *
+ * The parameters buffer is versioned by the @version field to allow modifying
+ * and extending its definition. Userspace should populate the @version field to
+ * inform the driver about the version it intends to use. The driver will parse
+ * and handle the @data buffer according to the data layout specific to the
+ * indicated revision and return an error if the desired revision is not
+ * supported.
+ *
+ * For each ISP block that userspace wants to configure, a block-specific
+ * structure is appended to the @data buffer, one after the other without gaps
+ * in between nor overlaps. Userspace shall populate the @total_size field with
+ * the effective size, in bytes, of the @data buffer.
+ *
+ * The expected memory layout of the parameters buffer is::
+ *
+ *	+-------------------- struct mali_c55_params_buffer ------------------+
+ *	| version = MALI_C55_PARAM_BUFFER_V0;                                 |
+ *	| total_size = sizeof(struct mali_c55_params_sensor_off_preshading)   |
+ *	|              sizeof(struct mali_c55_params_aexp_hist);              |
+ *	| +------------------------- data  ---------------------------------+ |
+ *	| | +--------- struct mali_c55_params_sensor_off_preshading ------+ | |
+ *	| | | +-------- struct mali_c55_params_block_header header -----+ | | |
+ *	| | | | type = MALI_C55_PARAM_BLOCK_SENSOR_OFFS;                | | | |
+ *	| | | | enabled = 1;                                            | | | |
+ *	| | | | size =                                                  | | | |
+ *	| | | |    sizeof(struct mali_c55_params_sensor_off_preshading);| | | |
+ *	| | | +---------------------------------------------------------+ | | |
+ *	| | | chan00 = ...;                                               | | |
+ *	| | | chan01 = ...;                                               | | |
+ *	| | | chan10 = ...;                                               | | |
+ *	| | | chan11 = ...;                                               | | |
+ *	| | +------------ struct mali_c55_params_aexp_hist ---------------+ | |
+ *	| | | +-------- struct mali_c55_params_block_header header -----+ | | |
+ *	| | | | type = MALI_C55_PARAM_BLOCK_AEXP_HIST;                  | | | |
+ *	| | | | enabled = 1;                                            | | | |
+ *	| | | | size = sizeof(struct mali_c55_params_aexp_hist);        | | | |
+ *	| | | +---------------------------------------------------------+ | | |
+ *	| | | skip_x = ...;                                               | | |
+ *	| | | offset_x = ...;                                             | | |
+ *	| | | skip_y = ...;                                               | | |
+ *	| | | offset_y = ...;                                             | | |
+ *	| | | scale_bottom = ...;                                         | | |
+ *	| | | scale_top = ...;                                            | | |
+ *	| | | plane_mode = ...;                                           | | |
+ *	| | | tap_point = ...;                                            | | |
+ *	| | +-------------------------------------------------------------+ | |
+ *	| +-----------------------------------------------------------------+ |
+ *	+---------------------------------------------------------------------+
+ *
+ * @version: The Mali-C55 parameters buffer version
+ * @total_size: The Mali-C55 configuration data effective size, excluding this
+ *		header
+ * @data: The Mali-C55 configuration blocks data
+ */
+struct mali_c55_params_buffer {
+	enum mali_c55_param_buffer_version version;
+	size_t total_size;
+	__u8 data[MALI_C55_PARAMS_MAX_SIZE];
+};
+
 #endif /* __UAPI_MALI_C55_CONFIG_H */
-- 
2.34.1

[PATCH v5 15/16] media: platform: Add mali-c55 parameters video node

[Daniel Scally]

Add a new code file to the mali-c55 driver that registers an output
video node for userspace to queue buffers of parameters to. Handlers
are included to program the statistics generation plus the white
balance, black level correction and mesh shading correction blocks.

Update the rest of the driver to register and link the new video node

Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- New patch

 drivers/media/platform/arm/mali-c55/Makefile  |   1 +
 .../platform/arm/mali-c55/mali-c55-common.h   |  18 +
 .../platform/arm/mali-c55/mali-c55-core.c     |  24 +
 .../platform/arm/mali-c55/mali-c55-isp.c      |  16 +-
 .../platform/arm/mali-c55/mali-c55-params.c   | 615 ++++++++++++++++++
 .../arm/mali-c55/mali-c55-registers.h         | 104 +++
 6 files changed, 777 insertions(+), 1 deletion(-)
 create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-params.c

diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
index cd5a64bf0c62..b2443f2d416a 100644
--- a/drivers/media/platform/arm/mali-c55/Makefile
+++ b/drivers/media/platform/arm/mali-c55/Makefile
@@ -5,6 +5,7 @@ mali-c55-y := mali-c55-capture.o \
 	      mali-c55-isp.o \
 	      mali-c55-tpg.o \
 	      mali-c55-resizer.o \
+	      mali-c55-params.o \
 	      mali-c55-stats.o
 
 obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
index 44119e04009b..565d98acfcdd 100644
--- a/drivers/media/platform/arm/mali-c55/mali-c55-common.h
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
@@ -80,6 +80,7 @@ enum mali_c55_isp_pads {
 	MALI_C55_ISP_PAD_SOURCE,
 	MALI_C55_ISP_PAD_SOURCE_BYPASS,
 	MALI_C55_ISP_PAD_SOURCE_3A,
+	MALI_C55_ISP_PAD_SINK_PARAMS,
 	MALI_C55_ISP_NUM_PADS,
 };
 
@@ -217,6 +218,19 @@ struct mali_c55_stats {
 	} buffers;
 };
 
+struct mali_c55_params {
+	struct mali_c55 *mali_c55;
+	struct video_device vdev;
+	struct vb2_queue queue;
+	struct media_pad pad;
+	struct mutex lock;
+
+	struct {
+		spinlock_t lock;
+		struct list_head queue;
+	} buffers;
+};
+
 enum mali_c55_config_spaces {
 	MALI_C55_CONFIG_PING,
 	MALI_C55_CONFIG_PONG,
@@ -247,6 +261,7 @@ struct mali_c55 {
 	struct mali_c55_isp isp;
 	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
 	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
+	struct mali_c55_params params;
 	struct mali_c55_stats stats;
 
 	struct list_head contexts;
@@ -271,6 +286,8 @@ int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
 void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
 int mali_c55_register_stats(struct mali_c55 *mali_c55);
 void mali_c55_unregister_stats(struct mali_c55 *mali_c55);
+int mali_c55_register_params(struct mali_c55 *mali_c55);
+void mali_c55_unregister_params(struct mali_c55 *mali_c55);
 struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
 void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
 			     enum mali_c55_planes plane);
@@ -290,5 +307,6 @@ bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
 	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
 void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
 				enum mali_c55_config_spaces cfg_space);
+void mali_c55_params_write_config(struct mali_c55 *mali_c55);
 
 #endif /* _MALI_C55_COMMON_H */
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
index 2cf8b1169604..6acee3edd03f 100644
--- a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
@@ -347,6 +347,17 @@ static int mali_c55_create_links(struct mali_c55 *mali_c55)
 		goto err_remove_links;
 	}
 
+	ret = media_create_pad_link(&mali_c55->params.vdev.entity, 0,
+				    &mali_c55->isp.sd.entity,
+				    MALI_C55_ISP_PAD_SINK_PARAMS,
+				    MEDIA_LNK_FL_ENABLED |
+				    MEDIA_LNK_FL_IMMUTABLE);
+	if (ret) {
+		dev_err(mali_c55->dev,
+			"failed to link ISP and parameters video node\n");
+		goto err_remove_links;
+	}
+
 	return 0;
 
 err_remove_links:
@@ -360,6 +371,7 @@ static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
 	mali_c55_unregister_isp(mali_c55);
 	mali_c55_unregister_resizers(mali_c55);
 	mali_c55_unregister_capture_devs(mali_c55);
+	mali_c55_unregister_params(mali_c55);
 	mali_c55_unregister_stats(mali_c55);
 }
 
@@ -383,6 +395,10 @@ static int mali_c55_register_entities(struct mali_c55 *mali_c55)
 	if (ret)
 		goto err_unregister_entities;
 
+	ret = mali_c55_register_params(mali_c55);
+	if (ret)
+		goto err_unregister_entities;
+
 	ret = mali_c55_register_stats(mali_c55);
 	if (ret)
 		goto err_unregister_entities;
@@ -474,6 +490,14 @@ static irqreturn_t mali_c55_isr(int irq, void *context)
 			curr_config >>= ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1;
 			next_config = curr_config ^ 1;
 
+			/*
+			 * Write the configuration parameters received from
+			 * userspace into the configuration buffer, which will
+			 * be transferred to the 'next' active config space at
+			 * by mali_c55_swap_next_config().
+			 */
+			mali_c55_params_write_config(mali_c55);
+
 			/*
 			 * The ordering of these two is currently important as
 			 * mali_c55_stats_fill_buffer() is asynchronous whereas
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
index 94876fba3353..8c2b45bfd82d 100644
--- a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
@@ -146,6 +146,7 @@ static int mali_c55_isp_start(struct mali_c55 *mali_c55)
 			     cfg->encoding == V4L2_PIXEL_ENC_RGB ?
 			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
 
+	mali_c55_params_write_config(mali_c55);
 	ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
 	if (ret) {
 		dev_err(mali_c55->dev, "failed to DMA config\n");
@@ -455,8 +456,20 @@ static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
 	.init_state = mali_c55_isp_init_state,
 };
 
+static int mali_c55_subdev_link_validate(struct media_link *link)
+{
+	/*
+	 * Skip validation for the parameters sink pad, as the source is not
+	 * a subdevice.
+	 */
+	if (link->sink->index == MALI_C55_ISP_PAD_SINK_PARAMS)
+		return 0;
+
+	return v4l2_subdev_link_validate(link);
+}
+
 static const struct media_entity_operations mali_c55_isp_media_ops = {
-	.link_validate		= v4l2_subdev_link_validate,
+	.link_validate		= mali_c55_subdev_link_validate,
 };
 
 static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
@@ -565,6 +578,7 @@ int mali_c55_register_isp(struct mali_c55 *mali_c55)
 	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
 	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
 	isp->pads[MALI_C55_ISP_PAD_SOURCE_3A].flags = MEDIA_PAD_FL_SOURCE;
+	isp->pads[MALI_C55_ISP_PAD_SINK_PARAMS].flags = MEDIA_PAD_FL_SINK;
 
 	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
 				     isp->pads);
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-params.c b/drivers/media/platform/arm/mali-c55/mali-c55-params.c
new file mode 100644
index 000000000000..049a7b8e4861
--- /dev/null
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-params.c
@@ -0,0 +1,615 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ARM Mali-C55 ISP Driver - Configuration parameters output device
+ *
+ * Copyright (C) 2024 Ideas on Board Oy
+ */
+#include <linux/media/arm/mali-c55-config.h>
+
+#include <media/media-entity.h>
+#include <media/v4l2-dev.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-fh.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-core.h>
+#include <media/videobuf2-dma-contig.h>
+
+#include "mali-c55-common.h"
+#include "mali-c55-registers.h"
+
+typedef void (*mali_c55_block_handler)(struct mali_c55 *mali_c55,
+				       struct mali_c55_params_block_header *block);
+
+struct mali_c55_block_handler {
+	size_t size;
+	mali_c55_block_handler handler;
+};
+
+static void mali_c55_params_sensor_offs(struct mali_c55 *mali_c55,
+					struct mali_c55_params_block_header *block)
+{
+	struct mali_c55_params_sensor_off_preshading *p =
+		(struct mali_c55_params_sensor_off_preshading *)block;
+	__u32 global_offset;
+
+	if (!block->enabled)
+		return;
+
+	if (!(p->chan00 || p->chan01 || p->chan10 || p->chan11))
+		return;
+
+	mali_c55_write(mali_c55, MALI_C55_REG_SENSOR_OFF_PRE_SHA_00,
+		       p->chan00 & MALI_C55_SENSOR_OFF_PRE_SHA_MASK);
+	mali_c55_write(mali_c55, MALI_C55_REG_SENSOR_OFF_PRE_SHA_01,
+		       p->chan01 & MALI_C55_SENSOR_OFF_PRE_SHA_MASK);
+	mali_c55_write(mali_c55, MALI_C55_REG_SENSOR_OFF_PRE_SHA_10,
+		       p->chan10 & MALI_C55_SENSOR_OFF_PRE_SHA_MASK);
+	mali_c55_write(mali_c55, MALI_C55_REG_SENSOR_OFF_PRE_SHA_11,
+		       p->chan11 & MALI_C55_SENSOR_OFF_PRE_SHA_MASK);
+
+	/*
+	 * The average offset is applied as a global offset for the digital
+	 * gain block
+	 */
+	global_offset = (p->chan00 + p->chan01 + p->chan10 + p->chan11) >> 2;
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_DIGITAL_GAIN_OFFSET,
+			     MALI_C55_DIGITAL_GAIN_OFFSET_MASK, global_offset);
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
+			     MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH, 0x00);
+}
+
+static void mali_c55_params_aexp_hist(struct mali_c55 *mali_c55,
+				struct mali_c55_params_block_header *block)
+{
+	u32 disable_mask = block->type == MALI_C55_PARAM_BLOCK_AEXP_HIST ?
+					  MALI_C55_AEXP_HIST_DISABLE_MASK :
+					  MALI_C55_AEXP_IHIST_DISABLE_MASK;
+	u32 base = block->type == MALI_C55_PARAM_BLOCK_AEXP_HIST ?
+				  MALI_C55_REG_AEXP_HIST_BASE :
+				  MALI_C55_REG_AEXP_IHIST_BASE;
+	struct mali_c55_params_aexp_hist *params =
+		(struct mali_c55_params_aexp_hist *)block;
+
+	if (!block->enabled) {
+		mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
+				     disable_mask, true);
+		return;
+	}
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
+			     disable_mask, false);
+
+	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SKIP_OFFSET,
+			     MALI_C55_AEXP_HIST_SKIP_X_MASK, params->skip_x);
+	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SKIP_OFFSET,
+			     MALI_C55_AEXP_HIST_OFFSET_X_MASK, params->offset_x);
+	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SKIP_OFFSET,
+			     MALI_C55_AEXP_HIST_SKIP_Y_MASK, params->skip_y);
+	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SKIP_OFFSET,
+			     MALI_C55_AEXP_HIST_OFFSET_Y_MASK, params->offset_y);
+
+	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SCALE_OFFSET,
+			     MALI_C55_AEXP_HIST_SCALE_BOTTOM_MASK, params->scale_bottom);
+	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SCALE_OFFSET,
+			     MALI_C55_AEXP_HIST_SCALE_TOP_MASK, params->scale_top);
+
+	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_PLANE_MODE_OFFSET,
+			     MALI_C55_AEXP_HIST_PLANE_MODE_MASK, params->plane_mode);
+
+	if (block->type == MALI_C55_PARAM_BLOCK_AEXP_HIST)
+		mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
+				     MALI_C55_AEXP_HIST_SWITCH_MASK,
+				     params->tap_point);
+}
+
+static void
+mali_c55_params_aexp_hist_weights(struct mali_c55 *mali_c55,
+				  struct mali_c55_params_block_header *block)
+{
+	struct mali_c55_params_aexp_weights *params =
+		(struct mali_c55_params_aexp_weights *)block;
+	u32 base;
+
+	if (!block->enabled)
+		return;
+
+	base = block->type == MALI_C55_PARAM_BLOCK_AEXP_HIST_WEIGHTS ?
+			      MALI_C55_REG_AEXP_HIST_BASE :
+			      MALI_C55_REG_AEXP_IHIST_BASE;
+
+	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_NODES_USED_OFFSET,
+			     MALI_C55_AEXP_HIST_NODES_USED_HORIZ_MASK, params->nodes_used_horiz);
+	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_NODES_USED_OFFSET,
+			     MALI_C55_AEXP_HIST_NODES_USED_VERT_MASK, params->nodes_used_vert);
+
+	/*
+	 * The zone weights array is a 225-element array of u8 values, but that
+	 * is a bit annoying to handle given the ISP expects 32-bit writes. We
+	 * just reinterpret it as a 57-element array of 32-bit values for the
+	 * purposes of this transaction (the 3 bytes of additional space at the
+	 * end of the write is just padding for the array of weights in the ISP
+	 * memory space anyway, so there's no risk of overwriting other
+	 * registers).
+	 */
+	for (unsigned int i = 0; i < 57; i++) {
+		u32 val = ((u32 *)params->zone_weights)[i]
+			    & MALI_C55_AEXP_HIST_ZONE_WEIGHT_MASK;
+		u32 addr = base + MALI_C55_AEXP_HIST_ZONE_WEIGHTS_OFFSET + (4 * i);
+
+		mali_c55_write(mali_c55, addr, val);
+	}
+}
+
+static void mali_c55_params_digital_gain(struct mali_c55 *mali_c55,
+					 struct mali_c55_params_block_header *block)
+{
+	struct mali_c55_params_digital_gain *dgain =
+		(struct mali_c55_params_digital_gain *)block;
+
+	/*
+	 * If the block is flagged as disabled we write a gain of 1.0, which in
+	 * Q5.8 format is 256.
+	 */
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_DIGITAL_GAIN,
+			     MALI_C55_DIGITAL_GAIN_MASK,
+			     block->enabled ? dgain->gain : 256);
+}
+
+static void mali_c55_params_awb_gains(struct mali_c55 *mali_c55,
+				      struct mali_c55_params_block_header *block)
+{
+	struct mali_c55_params_awb_gains *gains =
+		(struct mali_c55_params_awb_gains *)block;
+
+	/*
+	 * There are two places AWB gains can be set in the ISP; one affects the
+	 * image output data and the other affects the statistics for the
+	 * AEXP-0 tap point.
+	 */
+	u32 addr1 = block->type = MALI_C55_PARAM_BLOCK_AWB_GAINS ?
+				  MALI_C55_REG_AWB_GAINS1 :
+				  MALI_C55_REG_AWB_GAINS1_AEXP;
+	u32 addr2 = block->type = MALI_C55_PARAM_BLOCK_AWB_GAINS ?
+				  MALI_C55_REG_AWB_GAINS2 :
+				  MALI_C55_REG_AWB_GAINS2_AEXP;
+
+	mali_c55_update_bits(mali_c55, addr1, MALI_C55_AWB_GAIN00_MASK,
+			     gains->gain00);
+	mali_c55_update_bits(mali_c55, addr1, MALI_C55_AWB_GAIN01_MASK,
+			     gains->gain01);
+	mali_c55_update_bits(mali_c55, addr2, MALI_C55_AWB_GAIN10_MASK,
+			     gains->gain10);
+	mali_c55_update_bits(mali_c55, addr2, MALI_C55_AWB_GAIN11_MASK,
+			     gains->gain11);
+}
+
+static void mali_c55_params_awb_config(struct mali_c55 *mali_c55,
+				      struct mali_c55_params_block_header *block)
+{
+	struct mali_c55_params_awb_config *params =
+		(struct mali_c55_params_awb_config *)block;
+
+	if (!block->enabled) {
+		mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
+				     MALI_C55_AWB_DISABLE_MASK, true);
+		return;
+	}
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
+			     MALI_C55_AWB_DISABLE_MASK, false);
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_STATS_MODE,
+			     MALI_C55_AWB_STATS_MODE_MASK, params->stats_mode);
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_WHITE_LEVEL,
+			     MALI_C55_AWB_WHITE_LEVEL_MASK, params->white_level);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_BLACK_LEVEL,
+			     MALI_C55_AWB_BLACK_LEVEL_MASK, params->black_level);
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CR_MAX,
+			     MALI_C55_AWB_CR_MAX_MASK, params->cr_max);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CR_MIN,
+			     MALI_C55_AWB_CR_MIN_MASK, params->cr_min);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CB_MAX,
+			     MALI_C55_AWB_CB_MAX_MASK, params->cb_max);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CB_MIN,
+			     MALI_C55_AWB_CB_MIN_MASK, params->cb_min);
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_NODES_USED,
+			     MALI_C55_AWB_NODES_USED_HORIZ_MASK,
+			     params->nodes_used_horiz);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_NODES_USED,
+			     MALI_C55_AWB_NODES_USED_VERT_MASK,
+			     params->nodes_used_vert);
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CR_HIGH,
+			     MALI_C55_AWB_CR_HIGH_MASK, params->cr_high);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CR_LOW,
+			     MALI_C55_AWB_CR_LOW_MASK, params->cr_low);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CB_HIGH,
+			     MALI_C55_AWB_CB_HIGH_MASK, params->cb_high);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CB_LOW,
+			     MALI_C55_AWB_CB_LOW_MASK, params->cb_low);
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
+			     MALI_C55_AWB_SWITCH_MASK, params->tap_point);
+}
+
+static void mali_c55_params_lsc_config(struct mali_c55 *mali_c55,
+				       struct mali_c55_params_block_header *block)
+{
+	struct mali_c55_params_mesh_shading_config *params =
+		(struct mali_c55_params_mesh_shading_config *)block;
+	unsigned int i;
+	u32 addr;
+
+	if (!block->enabled) {
+		mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
+				     MALI_C55_MESH_SHADING_ENABLE_MASK, false);
+		return;
+	}
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
+			     MALI_C55_MESH_SHADING_ENABLE_MASK, true);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
+			     MALI_C55_MESH_SHADING_MESH_SHOW, params->mesh_show);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
+			     MALI_C55_MESH_SHADING_SCALE_MASK,
+			     params->mesh_scale);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
+			     MALI_C55_MESH_SHADING_PAGE_R_MASK,
+			     params->mesh_page_r);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
+			     MALI_C55_MESH_SHADING_PAGE_G_MASK,
+			     params->mesh_page_g);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
+			     MALI_C55_MESH_SHADING_PAGE_B_MASK,
+			     params->mesh_page_b);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
+			     MALI_C55_MESH_SHADING_MESH_WIDTH_MASK,
+			     params->mesh_width);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
+			     MALI_C55_MESH_SHADING_MESH_HEIGHT_MASK,
+			     params->mesh_height);
+
+	for (i = 0; i < MALI_C55_NUM_MESH_SHADING_ELEMENTS; i++) {
+		addr = MALI_C55_REG_MESH_SHADING_TABLES + (i * 4);
+		mali_c55_write(mali_c55, addr, params->mesh[i]);
+	}
+}
+
+static void mali_c55_params_lsc_selection(struct mali_c55 *mali_c55,
+					  struct mali_c55_params_block_header *block)
+{
+	struct mali_c55_params_mesh_shading_selection *params =
+		(struct mali_c55_params_mesh_shading_selection *)block;
+
+	if (!block->enabled)
+		return;
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA_BANK,
+			     MALI_C55_MESH_SHADING_ALPHA_BANK_R_MASK,
+			     params->mesh_alpha_bank_r);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA_BANK,
+			     MALI_C55_MESH_SHADING_ALPHA_BANK_G_MASK,
+			     params->mesh_alpha_bank_g);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA_BANK,
+			     MALI_C55_MESH_SHADING_ALPHA_BANK_B_MASK,
+			     params->mesh_alpha_bank_b);
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA,
+			     MALI_C55_MESH_SHADING_ALPHA_R_MASK,
+			     params->mesh_alpha_r);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA,
+			     MALI_C55_MESH_SHADING_ALPHA_G_MASK,
+			     params->mesh_alpha_g);
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA,
+			     MALI_C55_MESH_SHADING_ALPHA_B_MASK,
+			     params->mesh_alpha_b);
+
+	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_MESH_STRENGTH,
+			     MALI_c55_MESH_STRENGTH_MASK,
+			     params->mesh_strength);
+}
+
+static const struct mali_c55_block_handler mali_c55_block_handlers[] = {
+	[MALI_C55_PARAM_BLOCK_SENSOR_OFFS] = {
+		.size = sizeof(struct mali_c55_params_sensor_off_preshading),
+		.handler = &mali_c55_params_sensor_offs,
+	},
+	[MALI_C55_PARAM_BLOCK_AEXP_HIST] = {
+		.size = sizeof(struct mali_c55_params_aexp_hist),
+		.handler = &mali_c55_params_aexp_hist,
+	},
+	[MALI_C55_PARAM_BLOCK_AEXP_IHIST] = {
+		.size = sizeof(struct mali_c55_params_aexp_hist),
+		.handler = &mali_c55_params_aexp_hist,
+	},
+	[MALI_C55_PARAM_BLOCK_AEXP_HIST_WEIGHTS] = {
+		.size = sizeof(struct mali_c55_params_aexp_weights),
+		.handler = &mali_c55_params_aexp_hist_weights,
+	},
+	[MALI_C55_PARAM_BLOCK_AEXP_IHIST_WEIGHTS] = {
+		.size = sizeof(struct mali_c55_params_aexp_weights),
+		.handler = &mali_c55_params_aexp_hist_weights,
+	},
+	[MALI_C55_PARAM_BLOCK_DIGITAL_GAIN] = {
+		.size = sizeof(struct mali_c55_params_digital_gain),
+		.handler = &mali_c55_params_digital_gain,
+	},
+	[MALI_C55_PARAM_BLOCK_AWB_GAINS] = {
+		.size = sizeof(struct mali_c55_params_awb_gains),
+		.handler = &mali_c55_params_awb_gains,
+	},
+	[MALI_C55_PARAM_BLOCK_AWB_CONFIG] = {
+		.size = sizeof(struct mali_c55_params_awb_config),
+		.handler = &mali_c55_params_awb_config,
+	},
+	[MALI_C55_PARAM_BLOCK_AWB_GAINS_AEXP] = {
+		.size = sizeof(struct mali_c55_params_awb_gains),
+		.handler = &mali_c55_params_awb_gains,
+	},
+	[MALI_C55_PARAM_MESH_SHADING_CONFIG] = {
+		.size = sizeof(struct mali_c55_params_mesh_shading_config),
+		.handler = &mali_c55_params_lsc_config,
+	},
+	[MALI_C55_PARAM_MESH_SHADING_SELECTION] = {
+		.size = sizeof(struct mali_c55_params_mesh_shading_selection),
+		.handler = &mali_c55_params_lsc_selection,
+	},
+};
+
+static int mali_c55_params_enum_fmt_meta_out(struct file *file, void *fh,
+					    struct v4l2_fmtdesc *f)
+{
+	if (f->index || f->type != V4L2_BUF_TYPE_META_OUTPUT)
+		return -EINVAL;
+
+	f->pixelformat = V4L2_META_FMT_MALI_C55_PARAMS;
+
+	return 0;
+}
+
+static int mali_c55_params_g_fmt_meta_out(struct file *file, void *fh,
+					 struct v4l2_format *f)
+{
+	static const struct v4l2_meta_format mfmt = {
+		.dataformat = V4L2_META_FMT_MALI_C55_PARAMS,
+		.buffersize = sizeof(struct mali_c55_params_buffer),
+	};
+
+	if (f->type != V4L2_BUF_TYPE_META_OUTPUT)
+		return -EINVAL;
+
+	f->fmt.meta = mfmt;
+
+	return 0;
+}
+
+static int mali_c55_params_querycap(struct file *file,
+				   void *priv, struct v4l2_capability *cap)
+{
+	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
+	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
+
+	return 0;
+}
+
+static const struct v4l2_ioctl_ops mali_c55_params_v4l2_ioctl_ops = {
+	.vidioc_reqbufs = vb2_ioctl_reqbufs,
+	.vidioc_querybuf = vb2_ioctl_querybuf,
+	.vidioc_create_bufs = vb2_ioctl_create_bufs,
+	.vidioc_qbuf = vb2_ioctl_qbuf,
+	.vidioc_expbuf = vb2_ioctl_expbuf,
+	.vidioc_dqbuf = vb2_ioctl_dqbuf,
+	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+	.vidioc_streamon = vb2_ioctl_streamon,
+	.vidioc_streamoff = vb2_ioctl_streamoff,
+	.vidioc_enum_fmt_meta_out = mali_c55_params_enum_fmt_meta_out,
+	.vidioc_g_fmt_meta_out = mali_c55_params_g_fmt_meta_out,
+	.vidioc_s_fmt_meta_out = mali_c55_params_g_fmt_meta_out,
+	.vidioc_try_fmt_meta_out = mali_c55_params_g_fmt_meta_out,
+	.vidioc_querycap = mali_c55_params_querycap,
+	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
+	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
+};
+
+static const struct v4l2_file_operations mali_c55_params_v4l2_fops = {
+	.owner = THIS_MODULE,
+	.unlocked_ioctl = video_ioctl2,
+	.open = v4l2_fh_open,
+	.release = vb2_fop_release,
+	.poll = vb2_fop_poll,
+	.mmap = vb2_fop_mmap,
+};
+
+static int
+mali_c55_params_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
+			   unsigned int *num_planes, unsigned int sizes[],
+			   struct device *alloc_devs[])
+{
+	if (*num_planes && *num_planes > 1)
+		return -EINVAL;
+
+	if (sizes[0] && sizes[0] != sizeof(struct mali_c55_params_buffer))
+		return -EINVAL;
+
+	*num_planes = 1;
+	sizes[0] = sizeof(struct mali_c55_params_buffer);
+
+	return 0;
+}
+
+static void mali_c55_params_buf_queue(struct vb2_buffer *vb)
+{
+	struct mali_c55_params *params = vb2_get_drv_priv(vb->vb2_queue);
+	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
+	struct mali_c55_buffer *buf = container_of(vbuf,
+						   struct mali_c55_buffer, vb);
+
+	vb2_set_plane_payload(vb, 0, sizeof(struct mali_c55_params_buffer));
+
+	spin_lock(&params->buffers.lock);
+	list_add_tail(&buf->queue, &params->buffers.queue);
+	spin_unlock(&params->buffers.lock);
+}
+
+static void mali_c55_params_stop_streaming(struct vb2_queue *q)
+{
+	struct mali_c55_params *params = vb2_get_drv_priv(q);
+	struct mali_c55_buffer *buf, *tmp;
+
+	spin_lock(&params->buffers.lock);
+
+	list_for_each_entry_safe(buf, tmp, &params->buffers.queue, queue) {
+		list_del(&buf->queue);
+		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
+	}
+
+	spin_unlock(&params->buffers.lock);
+}
+
+static const struct vb2_ops mali_c55_params_vb2_ops = {
+	.queue_setup = mali_c55_params_queue_setup,
+	.buf_queue = mali_c55_params_buf_queue,
+	.wait_prepare = vb2_ops_wait_prepare,
+	.wait_finish = vb2_ops_wait_finish,
+	.stop_streaming = mali_c55_params_stop_streaming,
+};
+
+void mali_c55_params_write_config(struct mali_c55 *mali_c55)
+{
+	struct mali_c55_params *params = &mali_c55->params;
+	enum vb2_buffer_state state = VB2_BUF_STATE_DONE;
+	struct mali_c55_params_buffer *config;
+	struct mali_c55_buffer *buf;
+	size_t block_offset = 0;
+
+	spin_lock(&params->buffers.lock);
+
+	buf = list_first_entry_or_null(&params->buffers.queue,
+				       struct mali_c55_buffer, queue);
+	if (buf)
+		list_del(&buf->queue);
+	spin_unlock(&params->buffers.lock);
+
+	if (!buf)
+		return;
+
+	buf->vb.sequence = mali_c55->isp.frame_sequence;
+	config = vb2_plane_vaddr(&buf->vb.vb2_buf, 0);
+
+	if (config->total_size > MALI_C55_PARAMS_MAX_SIZE) {
+		dev_dbg(mali_c55->dev, "Invalid parameters buffer size %lu\n",
+			config->total_size);
+		state = VB2_BUF_STATE_ERROR;
+		goto err_buffer_done;
+	}
+
+	/* Walk the list of parameter blocks and process them. */
+	while (block_offset < config->total_size) {
+		const struct mali_c55_block_handler *block_handler;
+		struct mali_c55_params_block_header *block;
+
+		block = (struct mali_c55_params_block_header *)
+			 &config->data[block_offset];
+
+		if (block->type >= MALI_C55_PARAM_BLOCK_SENTINEL) {
+			dev_dbg(mali_c55->dev, "Invalid parameters block type\n");
+			state = VB2_BUF_STATE_ERROR;
+			goto err_buffer_done;
+		}
+
+		block_handler = &mali_c55_block_handlers[block->type];
+		if (block->size != block_handler->size) {
+			dev_dbg(mali_c55->dev, "Invalid parameters block size\n");
+			state = VB2_BUF_STATE_ERROR;
+			goto err_buffer_done;
+		}
+
+		block_handler->handler(mali_c55, block);
+
+		block_offset += block->size;
+	}
+
+err_buffer_done:
+	vb2_buffer_done(&buf->vb.vb2_buf, state);
+}
+
+void mali_c55_unregister_params(struct mali_c55 *mali_c55)
+{
+	struct mali_c55_params *params = &mali_c55->params;
+
+	if (!video_is_registered(&params->vdev))
+		return;
+
+	vb2_video_unregister_device(&params->vdev);
+	media_entity_cleanup(&params->vdev.entity);
+	mutex_destroy(&params->lock);
+}
+
+int mali_c55_register_params(struct mali_c55 *mali_c55)
+{
+	struct mali_c55_params *params = &mali_c55->params;
+	struct video_device *vdev = &params->vdev;
+	struct vb2_queue *vb2q = &params->queue;
+	int ret;
+
+	mutex_init(&params->lock);
+	INIT_LIST_HEAD(&params->buffers.queue);
+
+	params->pad.flags = MEDIA_PAD_FL_SOURCE;
+	ret = media_entity_pads_init(&params->vdev.entity, 1, &params->pad);
+	if (ret)
+		goto err_destroy_mutex;
+
+	vb2q->type = V4L2_BUF_TYPE_META_OUTPUT;
+	vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
+	vb2q->drv_priv = params;
+	vb2q->mem_ops = &vb2_dma_contig_memops;
+	vb2q->ops = &mali_c55_params_vb2_ops;
+	vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
+	vb2q->min_queued_buffers = 1;
+	vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+	vb2q->lock = &params->lock;
+	vb2q->dev = mali_c55->dev;
+
+	ret = vb2_queue_init(vb2q);
+	if (ret) {
+		dev_err(mali_c55->dev, "params vb2 queue init failed\n");
+		goto err_cleanup_entity;
+	}
+
+	strscpy(params->vdev.name, "mali-c55 3a params",
+		sizeof(params->vdev.name));
+	vdev->release = video_device_release_empty;
+	vdev->fops = &mali_c55_params_v4l2_fops;
+	vdev->ioctl_ops = &mali_c55_params_v4l2_ioctl_ops;
+	vdev->lock = &params->lock;
+	vdev->v4l2_dev = &mali_c55->v4l2_dev;
+	vdev->queue = &params->queue;
+	vdev->device_caps = V4L2_CAP_META_OUTPUT | V4L2_CAP_STREAMING;
+	vdev->vfl_dir = VFL_DIR_TX;
+	video_set_drvdata(vdev, params);
+
+	ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
+	if (ret) {
+		dev_err(mali_c55->dev,
+			"failed to register params video device\n");
+		goto err_release_vb2q;
+	}
+
+	params->mali_c55 = mali_c55;
+
+	return 0;
+
+err_release_vb2q:
+	vb2_queue_release(vb2q);
+err_cleanup_entity:
+	media_entity_cleanup(&params->vdev.entity);
+err_destroy_mutex:
+	mutex_destroy(&params->lock);
+
+	return ret;
+}
diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
index eb3719245ec3..8e6a801077ed 100644
--- a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
+++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
@@ -119,6 +119,19 @@
 #define MALI_C55_REG_ACTIVE_HEIGHT_MASK			0xffff0000
 #define MALI_C55_REG_BAYER_ORDER			0x18e8c
 #define MALI_C55_BAYER_ORDER_MASK			GENMASK(1, 0)
+
+#define MALI_C55_REG_METERING_CONFIG			0x18ed0
+#define MALI_C55_5BIN_HIST_DISABLE_MASK			BIT(0)
+#define MALI_C55_5BIN_HIST_SWITCH_MASK			GENMASK(2, 1)
+#define MALI_C55_AF_DISABLE_MASK			BIT(4)
+#define MALI_C55_AF_SWITCH_MASK				BIT(5)
+#define MALI_C55_AWB_DISABLE_MASK			BIT(8)
+#define MALI_C55_AWB_SWITCH_MASK			BIT(9)
+#define MALI_C55_AEXP_HIST_DISABLE_MASK			BIT(12)
+#define MALI_C55_AEXP_HIST_SWITCH_MASK			GENMASK(14, 13)
+#define MALI_C55_AEXP_IHIST_DISABLE_MASK		BIT(16)
+#define MALI_C55_AEXP_SRC_MASK				BIT(24)
+
 #define MALI_C55_REG_TPG_CH0				0x18ed8
 #define MALI_C55_TEST_PATTERN_ON_OFF			BIT(0)
 #define MALI_C55_TEST_PATTERN_RGB_MASK			BIT(1)
@@ -138,6 +151,11 @@
 #define MALI_C55_REG_CONFIG_SPACES_OFFSET		0x0ab6c
 #define MALI_C55_CONFIG_SPACE_SIZE			0x1231c
 
+#define MALI_C55_REG_DIGITAL_GAIN			0x1926c
+#define MALI_C55_DIGITAL_GAIN_MASK			GENMASK(12, 0)
+#define MALI_C55_REG_DIGITAL_GAIN_OFFSET		0x19270
+#define MALI_C55_DIGITAL_GAIN_OFFSET_MASK		GENMASK(19, 0)
+
 #define MALI_C55_REG_SINTER_CONFIG			0x19348
 #define MALI_C55_SINTER_VIEW_FILTER_MASK		GENMASK(1, 0)
 #define MALI_C55_SINTER_SCALE_MODE_MASK			GENMASK(3, 2)
@@ -146,6 +164,46 @@
 #define MALI_C55_SINTER_INT_SELECT_MASK			BIT(6)
 #define MALI_C55_SINTER_RM_ENABLE_MASK			BIT(7)
 
+/* Black Level Correction Configuration */
+#define MALI_C55_REG_SENSOR_OFF_PRE_SHA_00		0x1abcc
+#define MALI_C55_REG_SENSOR_OFF_PRE_SHA_01		0x1abd0
+#define MALI_C55_REG_SENSOR_OFF_PRE_SHA_10		0x1abd4
+#define MALI_C55_REG_SENSOR_OFF_PRE_SHA_11		0x1abd8
+#define MALI_C55_SENSOR_OFF_PRE_SHA_MASK		0xfffff
+
+/* Lens Mesh Shading Configuration */
+#define MALI_C55_REG_MESH_SHADING_TABLES		0x13074
+#define MALI_C55_REG_MESH_SHADING_CONFIG		0x1abfc
+#define MALI_C55_MESH_SHADING_ENABLE_MASK		BIT(0)
+#define MALI_C55_MESH_SHADING_MESH_SHOW			BIT(1)
+#define MALI_C55_MESH_SHADING_SCALE_MASK		GENMASK(4, 2)
+#define MALI_C55_MESH_SHADING_PAGE_R_MASK		GENMASK(9, 8)
+#define MALI_C55_MESH_SHADING_PAGE_G_MASK		GENMASK(11, 10)
+#define MALI_C55_MESH_SHADING_PAGE_B_MASK		GENMASK(13, 12)
+#define MALI_C55_MESH_SHADING_MESH_WIDTH_MASK		GENMASK(21, 16)
+#define MALI_C55_MESH_SHADING_MESH_HEIGHT_MASK		GENMASK(29, 24)
+
+#define MALI_C55_REG_MESH_SHADING_ALPHA_BANK		0x1ac04
+#define MALI_C55_MESH_SHADING_ALPHA_BANK_R_MASK		GENMASK(2, 0)
+#define MALI_C55_MESH_SHADING_ALPHA_BANK_G_MASK		GENMASK(5, 3)
+#define MALI_C55_MESH_SHADING_ALPHA_BANK_B_MASK		GENMASK(8, 6)
+#define MALI_C55_REG_MESH_SHADING_ALPHA			0x1ac08
+#define MALI_C55_MESH_SHADING_ALPHA_R_MASK		GENMASK(7, 0)
+#define MALI_C55_MESH_SHADING_ALPHA_G_MASK		GENMASK(15, 8)
+#define MALI_C55_MESH_SHADING_ALPHA_B_MASK		GENMASK(23, 16)
+#define MALI_C55_REG_MESH_SHADING_MESH_STRENGTH		0x1ac0c
+#define MALI_c55_MESH_STRENGTH_MASK			GENMASK(15, 0)
+
+/* AWB Gains Configuration */
+#define MALI_C55_REG_AWB_GAINS1				0x1ac10
+#define MALI_C55_AWB_GAIN00_MASK			GENMASK(11, 0)
+#define MALI_C55_AWB_GAIN01_MASK			GENMASK(27, 16)
+#define MALI_C55_REG_AWB_GAINS2				0x1ac14
+#define MALI_C55_AWB_GAIN10_MASK			GENMASK(11, 0)
+#define MALI_C55_AWB_GAIN11_MASK			GENMASK(27, 16)
+#define MALI_C55_REG_AWB_GAINS1_AEXP			0x1ac18
+#define MALI_C55_REG_AWB_GAINS2_AEXP			0x1ac1c
+
 /* Colour Correction Matrix Configuration */
 #define MALI_C55_REG_CCM_ENABLE				0x1b07c
 #define MALI_C55_CCM_ENABLE_MASK			BIT(0)
@@ -168,6 +226,52 @@
 #define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B		0x1b0c8
 #define MALI_C55_CCM_ANTIFOG_OFFSET_MASK		GENMASK(11, 0)
 
+/* AWB Statistics Configuration */
+#define MALI_C55_REG_AWB_STATS_MODE			0x1b29c
+#define MALI_C55_AWB_STATS_MODE_MASK			BIT(0)
+#define MALI_C55_REG_AWB_WHITE_LEVEL			0x1b2a0
+#define MALI_C55_AWB_WHITE_LEVEL_MASK			GENMASK(9, 0)
+#define MALI_C55_REG_AWB_BLACK_LEVEL			0x1b2a4
+#define MALI_C55_AWB_BLACK_LEVEL_MASK			GENMASK(9, 0)
+#define MALI_C55_REG_AWB_CR_MAX				0x1b2a8
+#define MALI_C55_AWB_CR_MAX_MASK			GENMASK(11, 0)
+#define MALI_C55_REG_AWB_CR_MIN				0x1b2ac
+#define MALI_C55_AWB_CR_MIN_MASK			GENMASK(11, 0)
+#define MALI_C55_REG_AWB_CB_MAX				0x1b2b0
+#define MALI_C55_AWB_CB_MAX_MASK			GENMASK(11, 0)
+#define MALI_C55_REG_AWB_CB_MIN				0x1b2b4
+#define MALI_C55_AWB_CB_MIN_MASK			GENMASK(11, 0)
+#define MALI_C55_REG_AWB_NODES_USED			0x1b2c4
+#define MALI_C55_AWB_NODES_USED_HORIZ_MASK		GENMASK(7, 0)
+#define MALI_C55_AWB_NODES_USED_VERT_MASK		GENMASK(15, 8)
+#define MALI_C55_REG_AWB_CR_HIGH			0x1b2c8
+#define MALI_C55_AWB_CR_HIGH_MASK			GENMASK(11, 0)
+#define MALI_C55_REG_AWB_CR_LOW				0x1b2cc
+#define MALI_C55_AWB_CR_LOW_MASK			GENMASK(11, 0)
+#define MALI_C55_REG_AWB_CB_HIGH			0x1b2d0
+#define MALI_C55_AWB_CB_HIGH_MASK			GENMASK(11, 0)
+#define MALI_C55_REG_AWB_CB_LOW				0x1b2d4
+#define MALI_C55_AWB_CB_LOW_MASK			GENMASK(11, 0)
+
+/* AEXP Metering Histogram Configuration */
+#define MALI_C55_REG_AEXP_HIST_BASE			0x1b730
+#define MALI_C55_REG_AEXP_IHIST_BASE			0x1bbac
+#define MALI_C55_AEXP_HIST_SKIP_OFFSET			0
+#define MALI_C55_AEXP_HIST_SKIP_X_MASK			GENMASK(2, 0)
+#define MALI_C55_AEXP_HIST_OFFSET_X_MASK		BIT(3)
+#define MALI_C55_AEXP_HIST_SKIP_Y_MASK			GENMASK(6, 4)
+#define MALI_C55_AEXP_HIST_OFFSET_Y_MASK		BIT(7)
+#define MALI_C55_AEXP_HIST_SCALE_OFFSET			4
+#define MALI_C55_AEXP_HIST_SCALE_BOTTOM_MASK		GENMASK(3, 0)
+#define MALI_C55_AEXP_HIST_SCALE_TOP_MASK		GENMASK(7, 4)
+#define MALI_C55_AEXP_HIST_PLANE_MODE_OFFSET		16
+#define MALI_C55_AEXP_HIST_PLANE_MODE_MASK		GENMASK(2, 0)
+#define MALI_C55_AEXP_HIST_NODES_USED_OFFSET		52
+#define MALI_C55_AEXP_HIST_NODES_USED_HORIZ_MASK	GENMASK(7, 0)
+#define MALI_C55_AEXP_HIST_NODES_USED_VERT_MASK		GENMASK(15, 8)
+#define MALI_C55_AEXP_HIST_ZONE_WEIGHTS_OFFSET		56
+#define MALI_C55_AEXP_HIST_ZONE_WEIGHT_MASK		0x0f0f0f0f
+
 /*
  * The Mali-C55 ISP has up to two output pipes; known as full resolution and
  * down scaled. The register space for these is laid out identically, but offset
-- 
2.34.1

[PATCH v5 16/16] Documentation: mali-c55: Document the mali-c55 parameter setting

[Daniel Scally]

Document the mali-c55 parameter setting by expanding the relevant
pages in both admin-guide/ and userspace-api/.

Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
---
Changes in v5:

	- New patch

 Documentation/admin-guide/media/mali-c55.rst  | 19 +++++-
 .../media/v4l/metafmt-arm-mali-c55.rst        | 65 ++++++++++++++++++-
 2 files changed, 79 insertions(+), 5 deletions(-)

diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst
index b75437f6e96a..c5e6ac826c99 100644
--- a/Documentation/admin-guide/media/mali-c55.rst
+++ b/Documentation/admin-guide/media/mali-c55.rst
@@ -380,9 +380,24 @@ the processing flow the statistics can be drawn from::
                        +-------------+   |    +-------------+
                                          +-->  AWB-1
 
-At present all statistics are drawn from the 0th tap point for each algorithm;
+By default all statistics are drawn from the 0th tap point for each algorithm;
 I.E. AEXP statistics from AEXP-0 (A), AWB statistics from AWB-0 and AF
-statistics from AF-0. In the future this will be configurable.
+statistics from AF-0. This is configurable for AEXP and AWB statsistics through
+programming the ISP's parameters.
+
+.. _mali-c55-3a-params:
+
+Programming ISP Parameters
+==========================
+
+The ISP can be programmed with various parameters from userspace to apply to the
+hardware before and during video stream. This allows userspace to dynamically
+change values such as black level, white balance and lens shading gains and so
+on.
+
+The buffer format and how to populate it are described by the
+:ref:`V4L2_META_FMT_MALI_C55_PARAMS <v4l2-meta-fmt-mali-c55-params>` format,
+which should be set as the data format for the `mali-c55 3a params` video node.
 
 References
 ==========
diff --git a/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst b/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
index 219a5dd42d70..c359d2c83858 100644
--- a/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
+++ b/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
@@ -1,10 +1,11 @@
 .. SPDX-License-Identifier: GPL-2.0
 
+.. _v4l2-meta-fmt-mali-c55-params:
 .. _v4l2-meta-fmt-mali-c55-3a-stats:
 
-****************************************
-V4L2_META_FMT_MALI_C55_3A_STATS ('C55S')
-****************************************
+********************************************************************************
+V4L2_META_FMT_MALI_C55_3A_STATS ('C55S'), V4L2_META_FMT_MALI_C55_PARAMS ('C55P')
+********************************************************************************
 
 3A Statistics
 =============
@@ -22,6 +23,64 @@ of the C structure :c:type:`mali_c55_stats_buffer` defined in
 
 For details of the statistics see :c:type:`mali_c55_stats_buffer`.
 
+Configuration Parameters
+========================
+
+The configuration parameters are passed to the
+:ref:`mali-c55 3a params <mali-c55-3a-params>` metadata output video node, using
+the :c:type:`v4l2_meta_format` interface. Rather than a single struct containing
+sub-structs for each configurable area of the ISP, parameters for the Mali-C55
+are defined as distinct structs or "blocks" which may be added to the data
+member of struct mali_c55_params_buffer. Userspace is responsible for populating
+the data member with the blocks that need to be configured by the driver, but
+need not populate it with **all** the blocks, or indeed with any at all if there
+are no configuration changes to make. Populated blocks **must** be consecutive
+in the buffer. To assist both userspace and the driver in identifying the
+blocks each block-specific struct should embed
+struct mali_c55_params_block_header as its first member and userspace must
+populate the type member with a value from enum mali_c55_param_block_type. Once
+the blocks have been populated into the data buffer, the combined size of all
+populated blocks should be set in the total_size member of
+struct mali_c55_params_buffer. For example:
+
+.. code-block:: c
+
+	struct mali_c55_params_buffer *params =
+		(struct mali_c55_params_buffer *)buffer;
+
+	params->version = MALI_C55_PARAM_BUFFER_V0;
+
+	void *data = (void *)params->data;
+
+	struct mali_c55_params_awb_gains *gains =
+		(struct mali_c55_params_awb_gains *)data;
+
+	gains->header.type = MALI_C55_PARAM_BLOCK_AWB_GAINS;
+	gains->header.enabled = true;
+	gains->header.size = sizeof(struct mali_c55_params_awb_gains);
+
+	gains->gain00 = 256;
+	gains->gain00 = 256;
+	gains->gain00 = 256;
+	gains->gain00 = 256;
+
+	data += sizeof(struct mali_c55_params_awb_gains)
+
+	struct mali_c55_params_sensor_off_preshading *blc =
+		(struct mali_c55_params_sensor_off_preshading *)data;
+
+	blc->header.type = MALI_C55_PARAM_BLOCK_SENSOR_OFFS;
+	blc->header.enabled = true;
+	blc->header.size = sizeof(struct mali_c55_params_sensor_off_preshading);
+
+	blc->chan00 = 51200;
+	blc->chan01 = 51200;
+	blc->chan10 = 51200;
+	blc->chan11 = 51200;
+
+	params->total_size = sizeof(struct mali_c55_params_awb_gains) +
+			     sizeof(struct mali_c55_params_sensor_off_preshading);
+
 Arm Mali-C55 uAPI data types
 ============================
 
-- 
2.34.1

Re: [PATCH v5 01/16] media: uapi: Add MEDIA_BUS_FMT_RGB202020_1X60 format code

[Laurent Pinchart]

Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:43PM +0100, Daniel Scally wrote:
> The Mali-C55 ISP by ARM requires 20-bits per colour channel input on
> the bus. Add a new media bus format code to represent it.
> 
> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
> Changes in v5:
> 
> 	- none
> 
> Changes in v4:
> 
> 	- None
> 
> Changes in v3:
> 
> 	- None
> 
> Changes in v2:
> 
> 	- none
>  .../media/v4l/subdev-formats.rst              | 168 ++++++++++++++++++
>  include/uapi/linux/media-bus-format.h         |   3 +-
>  2 files changed, 170 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index d2a6cd2e1eb2..8d164a9a9e15 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -2224,6 +2224,174 @@ The following table list existing packed 48bit wide RGB formats.
>  
>      \endgroup
>  
> +The following table list existing packed 60bit wide RGB formats.
> +
> +.. tabularcolumns:: |p{4.0cm}|p{0.7cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|p{0.22cm}|

Maybe one day we should try to make those tables more compact.

> +
> +.. _v4l2-mbus-pixelcode-rgb-60:
> +
> +.. raw:: latex
> +
> +    \begingroup
> +    \tiny
> +    \setlength{\tabcolsep}{2pt}
> +
> +.. flat-table:: 60bit RGB formats
> +    :header-rows:  3
> +    :stub-columns: 0
> +    :widths: 36 7 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
> +
> +    * - Identifier
> +      - Code
> +      -
> +      - :cspan:`31` Data organization
> +    * -
> +      -
> +      - Bit
> +      -
> +      -
> +      -
> +      -
> +      - 59
> +      - 58
> +      - 57
> +      - 56
> +      - 55
> +      - 54
> +      - 53
> +      - 52
> +      - 51
> +      - 50
> +      - 49
> +      - 48
> +      - 47
> +      - 46
> +      - 45
> +      - 44
> +      - 43
> +      - 42
> +      - 41
> +      - 40
> +      - 39
> +      - 38
> +      - 37
> +      - 36
> +      - 35
> +      - 34
> +      - 33
> +      - 32
> +    * -
> +      -
> +      -
> +      - 31
> +      - 30
> +      - 29
> +      - 28
> +      - 27
> +      - 26
> +      - 25
> +      - 24
> +      - 23
> +      - 22
> +      - 21
> +      - 20
> +      - 19
> +      - 18
> +      - 17
> +      - 16
> +      - 15
> +      - 14
> +      - 13
> +      - 12
> +      - 11
> +      - 10
> +      - 9
> +      - 8
> +      - 7
> +      - 6
> +      - 5
> +      - 4
> +      - 3
> +      - 2
> +      - 1
> +      - 0
> +    * .. _MEDIA-BUS-FMT-RGB202020-1X60:
> +
> +      - MEDIA_BUS_FMT_RGB202020_1X60
> +      - 0x1026
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - r\ :sub:`19`
> +      - r\ :sub:`18`
> +      - r\ :sub:`17`
> +      - r\ :sub:`16`
> +      - r\ :sub:`15`
> +      - r\ :sub:`14`
> +      - r\ :sub:`13`
> +      - r\ :sub:`12`
> +      - r\ :sub:`11`
> +      - r\ :sub:`10`
> +      - r\ :sub:`8`

I was thinking that reviews of this kind of patches were tedious and not
very useful, and then I found this mistake :-D With that fixed,

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> +      - r\ :sub:`8`
> +      - r\ :sub:`7`
> +      - r\ :sub:`6`
> +      - r\ :sub:`5`
> +      - r\ :sub:`4`
> +      - r\ :sub:`3`
> +      - r\ :sub:`2`
> +      - r\ :sub:`1`
> +      - r\ :sub:`0`
> +      - g\ :sub:`19`
> +      - g\ :sub:`18`
> +      - g\ :sub:`17`
> +      - g\ :sub:`16`
> +      - g\ :sub:`15`
> +      - g\ :sub:`14`
> +      - g\ :sub:`13`
> +      - g\ :sub:`12`
> +    * -
> +      -
> +      -
> +      - g\ :sub:`11`
> +      - g\ :sub:`10`
> +      - g\ :sub:`9`
> +      - g\ :sub:`8`
> +      - g\ :sub:`7`
> +      - g\ :sub:`6`
> +      - g\ :sub:`5`
> +      - g\ :sub:`4`
> +      - g\ :sub:`3`
> +      - g\ :sub:`2`
> +      - g\ :sub:`1`
> +      - g\ :sub:`0`
> +      - b\ :sub:`19`
> +      - b\ :sub:`18`
> +      - b\ :sub:`17`
> +      - b\ :sub:`16`
> +      - b\ :sub:`15`
> +      - b\ :sub:`14`
> +      - b\ :sub:`13`
> +      - b\ :sub:`12`
> +      - b\ :sub:`11`
> +      - b\ :sub:`10`
> +      - b\ :sub:`9`
> +      - b\ :sub:`8`
> +      - b\ :sub:`7`
> +      - b\ :sub:`6`
> +      - b\ :sub:`5`
> +      - b\ :sub:`4`
> +      - b\ :sub:`3`
> +      - b\ :sub:`2`
> +      - b\ :sub:`1`
> +      - b\ :sub:`0`
> +
> +.. raw:: latex
> +
> +    \endgroup
> +
>  On LVDS buses, usually each sample is transferred serialized in seven
>  time slots per pixel clock, on three (18-bit) or four (24-bit)
>  differential data pairs at the same time. The remaining bits are used
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index d4c1d991014b..49be328d9a3b 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -34,7 +34,7 @@
>  
>  #define MEDIA_BUS_FMT_FIXED			0x0001
>  
> -/* RGB - next is	0x1026 */
> +/* RGB - next is	0x1027 */
>  #define MEDIA_BUS_FMT_RGB444_1X12		0x1016
>  #define MEDIA_BUS_FMT_RGB444_2X8_PADHI_BE	0x1001
>  #define MEDIA_BUS_FMT_RGB444_2X8_PADHI_LE	0x1002
> @@ -72,6 +72,7 @@
>  #define MEDIA_BUS_FMT_RGB888_1X36_CPADLO	0x1021
>  #define MEDIA_BUS_FMT_RGB121212_1X36		0x1019
>  #define MEDIA_BUS_FMT_RGB161616_1X48		0x101a
> +#define MEDIA_BUS_FMT_RGB202020_1X60		0x1026
>  
>  /* YUV (including grey) - next is	0x202f */
>  #define MEDIA_BUS_FMT_Y8_1X8			0x2001

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 02/16] media: uapi: Add 20-bit bayer formats

[Laurent Pinchart]

Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:44PM +0100, Daniel Scally wrote:
> The Mali-C55 requires input data be in 20-bit format, MSB aligned.
> Add some new media bus format macros to represent that input format.
> 
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
> Changes in v5:
> 
> 	- New patch
> 
>  .../media/v4l/subdev-formats.rst              | 100 ++++++++++++++++++
>  include/uapi/linux/media-bus-format.h         |   6 +-
>  2 files changed, 105 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index 8d164a9a9e15..f986dfc52879 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -3445,6 +3445,106 @@ organization is given as an example for the first pixel only.
>        - r\ :sub:`2`
>        - r\ :sub:`1`
>        - r\ :sub:`0`
> +    * .. _MEDIA-BUS-FMT-SBGGR20-1X20:
> +
> +      - MEDIA_BUS_FMT_SBGGR20_1X20
> +      - 0x3021
> +      -
> +      - r\ :sub:`19`
> +      - r\ :sub:`18`
> +      - r\ :sub:`17`
> +      - r\ :sub:`16`
> +      - r\ :sub:`15`
> +      - r\ :sub:`14`
> +      - r\ :sub:`13`
> +      - r\ :sub:`12`
> +      - r\ :sub:`11`
> +      - r\ :sub:`10`
> +      - r\ :sub:`9`
> +      - r\ :sub:`8`
> +      - r\ :sub:`7`
> +      - r\ :sub:`6`
> +      - r\ :sub:`5`
> +      - r\ :sub:`4`
> +      - r\ :sub:`3`
> +      - r\ :sub:`2`
> +      - r\ :sub:`1`
> +      - r\ :sub:`0`

The table currently has 16 columns for data bits. You need to expand
that to 20 columns.

> +    * .. _MEDIA-BUS-FMT-SGBRG20-1X20:
> +
> +      - MEDIA_BUS_FMT_SGBRG20_1X20
> +      - 0x3022
> +      -
> +      - r\ :sub:`19`
> +      - r\ :sub:`18`
> +      - r\ :sub:`17`
> +      - r\ :sub:`16`
> +      - r\ :sub:`15`
> +      - r\ :sub:`14`
> +      - r\ :sub:`13`
> +      - r\ :sub:`12`
> +      - r\ :sub:`11`
> +      - r\ :sub:`10`
> +      - r\ :sub:`9`
> +      - r\ :sub:`8`
> +      - r\ :sub:`7`
> +      - r\ :sub:`6`
> +      - r\ :sub:`5`
> +      - r\ :sub:`4`
> +      - r\ :sub:`3`
> +      - r\ :sub:`2`
> +      - r\ :sub:`1`
> +      - r\ :sub:`0`
> +    * .. _MEDIA-BUS-FMT-SGRBG20-1X20:
> +
> +      - MEDIA_BUS_FMT_SGRBG20_1X20
> +      - 0x3023
> +      -
> +      - r\ :sub:`19`
> +      - r\ :sub:`18`
> +      - r\ :sub:`17`
> +      - r\ :sub:`16`
> +      - r\ :sub:`15`
> +      - r\ :sub:`14`
> +      - r\ :sub:`13`
> +      - r\ :sub:`12`
> +      - r\ :sub:`11`
> +      - r\ :sub:`10`
> +      - r\ :sub:`9`
> +      - r\ :sub:`8`
> +      - r\ :sub:`7`
> +      - r\ :sub:`6`
> +      - r\ :sub:`5`
> +      - r\ :sub:`4`
> +      - r\ :sub:`3`
> +      - r\ :sub:`2`
> +      - r\ :sub:`1`
> +      - r\ :sub:`0`
> +    * .. _MEDIA-BUS-FMT-SRGGB20-1X20:
> +
> +      - MEDIA_BUS_FMT_SRGGB20_1X20
> +      - 0x3024
> +      -
> +      - r\ :sub:`19`
> +      - r\ :sub:`18`
> +      - r\ :sub:`17`
> +      - r\ :sub:`16`
> +      - r\ :sub:`15`
> +      - r\ :sub:`14`
> +      - r\ :sub:`13`
> +      - r\ :sub:`12`
> +      - r\ :sub:`11`
> +      - r\ :sub:`10`
> +      - r\ :sub:`9`
> +      - r\ :sub:`8`
> +      - r\ :sub:`7`
> +      - r\ :sub:`6`
> +      - r\ :sub:`5`
> +      - r\ :sub:`4`
> +      - r\ :sub:`3`
> +      - r\ :sub:`2`
> +      - r\ :sub:`1`
> +      - r\ :sub:`0`
>  
>  .. raw:: latex
>  
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index 49be328d9a3b..b6acf8c8e383 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -122,7 +122,7 @@
>  #define MEDIA_BUS_FMT_YUV16_1X48		0x202a
>  #define MEDIA_BUS_FMT_UYYVYY16_0_5X48		0x202b
>  
> -/* Bayer - next is	0x3021 */
> +/* Bayer - next is	0x3025 */
>  #define MEDIA_BUS_FMT_SBGGR8_1X8		0x3001
>  #define MEDIA_BUS_FMT_SGBRG8_1X8		0x3013
>  #define MEDIA_BUS_FMT_SGRBG8_1X8		0x3002
> @@ -155,6 +155,10 @@
>  #define MEDIA_BUS_FMT_SGBRG16_1X16		0x301e
>  #define MEDIA_BUS_FMT_SGRBG16_1X16		0x301f
>  #define MEDIA_BUS_FMT_SRGGB16_1X16		0x3020
> +#define MEDIA_BUS_FMT_SBGGR20_1X20		0x3021
> +#define MEDIA_BUS_FMT_SGBRG20_1X20		0x3022
> +#define MEDIA_BUS_FMT_SGRBG20_1X20		0x3023
> +#define MEDIA_BUS_FMT_SRGGB20_1X20		0x3024
>  
>  /* JPEG compressed formats - next is	0x4002 */
>  #define MEDIA_BUS_FMT_JPEG_1X8			0x4001

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 03/16] media: v4l2-common: Add RAW16 format info

[Laurent Pinchart]

Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:45PM +0100, Daniel Scally wrote:
> Add entries to v4l2_format_info describing the 16-bit bayer
> formats so that they can be used in drivers.
> 
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
> Changes in v5:
> 
> 	- New patch
> 
>  drivers/media/v4l2-core/v4l2-common.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-common.c b/drivers/media/v4l2-core/v4l2-common.c
> index 4165c815faef..c5d5704af5ee 100644
> --- a/drivers/media/v4l2-core/v4l2-common.c
> +++ b/drivers/media/v4l2-core/v4l2-common.c
> @@ -331,6 +331,10 @@ const struct v4l2_format_info *v4l2_format_info(u32 format)
>  		{ .format = V4L2_PIX_FMT_SGBRG12,	.pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
>  		{ .format = V4L2_PIX_FMT_SGRBG12,	.pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
>  		{ .format = V4L2_PIX_FMT_SRGGB12,	.pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },

If you have energy, could you add a patch to also cover the 14-bit
formats ?

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> +		{ .format = V4L2_PIX_FMT_SBGGR16,	.pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
> +		{ .format = V4L2_PIX_FMT_SGBRG16,	.pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
> +		{ .format = V4L2_PIX_FMT_SGRBG16,	.pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
> +		{ .format = V4L2_PIX_FMT_SRGGB16,	.pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
>  	};
>  	unsigned int i;
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 04/16] dt-bindings: media: Add bindings for ARM mali-c55

[Laurent Pinchart]

Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:46PM +0100, Daniel Scally wrote:
> Add the yaml binding for ARM's Mali-C55 Image Signal Processor.
> 
> Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
> Changes in v5:
> 
> 	- None
> 
> Changes in v4:
> 
> 	- Switched to port instead of ports
> 
> Changes in v3:
> 
> 	- Dropped the video clock as suggested by Laurent. I didn't retain it
> 	for the purposes of the refcount since this driver will call .s_stream()
> 	for the sensor driver which will refcount the clock anyway.
> 	- Clarified that the port is a parallel input port rather (Sakari)
> 
> Changes in v2:
> 
> 	- Added clocks information
> 	- Fixed the warnings raised by Rob
> 
>  .../bindings/media/arm,mali-c55.yaml          | 66 +++++++++++++++++++
>  1 file changed, 66 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/arm,mali-c55.yaml
> 
> diff --git a/Documentation/devicetree/bindings/media/arm,mali-c55.yaml b/Documentation/devicetree/bindings/media/arm,mali-c55.yaml
> new file mode 100644
> index 000000000000..9cc2481f2da3
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/arm,mali-c55.yaml
> @@ -0,0 +1,66 @@
> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/media/arm,mali-c55.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: ARM Mali-C55 Image Signal Processor
> +
> +maintainers:
> +  - Daniel Scally <dan.scally@ideasonboard.com>
> +  - Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> +
> +properties:
> +  compatible:
> +    const: arm,mali-c55
> +
> +  reg:
> +    maxItems: 1
> +
> +  interrupts:
> +    maxItems: 1
> +
> +  clocks:
> +    items:
> +      - description: ISP AXI clock
> +      - description: ISP AHB-lite clock
> +
> +  clock-names:
> +    items:
> +      - const: aclk
> +      - const: hclk
> +
> +  port:
> +    $ref: /schemas/graph.yaml#/properties/port
> +    description: Input parallel video bus
> +
> +    properties:
> +      endpoint:
> +        $ref: /schemas/graph.yaml#/properties/endpoint
> +
> +required:
> +  - compatible
> +  - reg
> +  - interrupts
> +  - clocks
> +  - clock-names
> +  - port
> +
> +additionalProperties: false
> +
> +examples:
> +  - |
> +    mali_c55: isp@400000 {
> +      compatible = "arm,mali-c55";
> +      reg = <0x400000 0x200000>;
> +      clocks = <&clk 0>, <&clk 1>;
> +      clock-names = "aclk", "hclk";
> +      interrupts = <0>;
> +
> +      port {
> +        isp_in: endpoint {
> +            remote-endpoint = <&csi2_rx_out>;
> +        };
> +      };
> +    };
> +...

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 07/16] MAINTAINERS: Add entry for mali-c55 driver

[Laurent Pinchart]

Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:49PM +0100, Daniel Scally wrote:
> Add a MAINTAINERS entry for the mali-c55 driver and its associated
> documentation.
> 
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
> Changes in v5:
> 
> 	- None
> 
> Changes in v4:
> 	- None
> 
> Changes in v3:
> 
> 	- none
> 
> Changes in v2:
> 
> 	- none
> 
>  MAINTAINERS | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index ef6be9d95143..c2883b30e0ca 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1688,6 +1688,16 @@ F:	Documentation/gpu/panfrost.rst
>  F:	drivers/gpu/drm/panfrost/
>  F:	include/uapi/drm/panfrost_drm.h
>  
> +ARM MALI-C55 ISP DRIVER
> +M:	Daniel Scally <dan.scally@ideasonboard.com>
> +M:	Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> +L:	linux-media@vger.kernel.org
> +S:	Maintained
> +T:	git git://linuxtv.org/media_tree.git
> +F:	Documentation/admin-guide/media/mali-c55.rst

You're missing mali-c55-graph.dot.

> +F:	Documentation/devicetree/bindings/media/arm,mali-c55.yaml
> +F:	drivers/media/platform/arm/mali-c55/

I'm recording it here to avoid forgetting later, you will need to update
MAINTAINERS in subsequent patches to add

Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
include/uapi/linux/media/arm/mali-c55-config.h

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> +
>  ARM MALI-DP DRM DRIVER
>  M:	Liviu Dudau <liviu.dudau@arm.com>
>  S:	Supported

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 06/16] media: Documentation: Add Mali-C55 ISP Documentation

[Laurent Pinchart]

Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:48PM +0100, Daniel Scally wrote:
> Add a documentation page for the mali-c55 driver, which gives a brief
> overview of the hardware and explains how to use the driver's capture
> devices and the crop/scaler functions.
> 
> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
> Changes in v5:
> 
> 	- None
> 
> Changes in v4:
> 	- None
> 
> Changes in v3:
> 	- Documented the synchronised buffer sequence numbers (Sakari)
> 	- Clarified that the downscale pipe cannot output raw data, the ISP'S
> 	  resolution limits and choice of media bus format code (Kieran)
> 
> Changes in v2:
> 
> 	- none
> 
>  .../admin-guide/media/mali-c55-graph.dot      |  19 +
>  Documentation/admin-guide/media/mali-c55.rst  | 333 ++++++++++++++++++
>  .../admin-guide/media/v4l-drivers.rst         |   1 +
>  3 files changed, 353 insertions(+)
>  create mode 100644 Documentation/admin-guide/media/mali-c55-graph.dot
>  create mode 100644 Documentation/admin-guide/media/mali-c55.rst
> 
> diff --git a/Documentation/admin-guide/media/mali-c55-graph.dot b/Documentation/admin-guide/media/mali-c55-graph.dot
> new file mode 100644
> index 000000000000..0775ba42bf4c
> --- /dev/null
> +++ b/Documentation/admin-guide/media/mali-c55-graph.dot
> @@ -0,0 +1,19 @@
> +digraph board {
> +        rankdir=TB
> +        n00000001 [label="{{} | mali-c55 tpg\n/dev/v4l-subdev0 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
> +        n00000001:port0 -> n00000003:port0 [style=dashed]
> +        n00000003 [label="{{<port0> 0} | mali-c55 isp\n/dev/v4l-subdev1 | {<port1> 1 | <port2> 2}}", shape=Mrecord, style=filled, fillcolor=green]
> +        n00000003:port1 -> n00000007:port0 [style=bold]
> +        n00000003:port2 -> n00000007:port2 [style=bold]
> +        n00000003:port1 -> n0000000b:port0 [style=bold]
> +        n00000007 [label="{{<port0> 0 | <port2> 2} | mali-c55 resizer fr\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
> +        n00000007:port1 -> n0000000e [style=bold]
> +        n0000000b [label="{{<port0> 0} | mali-c55 resizer ds\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
> +        n0000000b:port1 -> n00000012 [style=bold]
> +        n0000000e [label="mali-c55 fr\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
> +        n00000012 [label="mali-c55 ds\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
> +        n00000022 [label="{{<port0> 0} | csi2-rx\n/dev/v4l-subdev4 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
> +        n00000022:port1 -> n00000003:port0
> +        n00000027 [label="{{} | imx415 1-001a\n/dev/v4l-subdev5 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
> +        n00000027:port0 -> n00000022:port0 [style=bold]
> +}
> \ No newline at end of file
> diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst
> new file mode 100644
> index 000000000000..cf4176cb1e44
> --- /dev/null
> +++ b/Documentation/admin-guide/media/mali-c55.rst
> @@ -0,0 +1,333 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==========================================
> +ARM Mali-C55 Image Signal Processor driver
> +==========================================
> +
> +Introduction
> +============
> +
> +This file documents the driver for ARM's Mali-C55 Image Signal Processor. The
> +driver is located under drivers/media/platform/arm/mali-c55.
> +
> +The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from
> +sensors through either a parallel interface or a memory bus before processing it
> +and outputting it through an internal DMA engine. Two output pipelines are
> +possible (though one may not be fitted, depending on the implementation). These
> +are referred to as "Full resolution" and "Downscale", but the naming is historic
> +and both pipes are capable of cropping/scaling operations. The full resolution
> +pipe is also capable of outputting RAW data, bypassing much of the ISP's
> +processing. The downscale pipe cannot output RAW data. An integrated test
> +pattern generator can be used to drive the ISP and produce image data in the
> +absence of a connected camera sensor. The driver module is named mali_c55, and
> +is enabled through the CONFIG_VIDEO_MALI_C55 config option.
> +
> +The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and
> +expects camera sensors connected to the ISP to have V4L2 subdevice interfaces.
> +
> +Mali-C55 ISP hardware
> +=====================
> +
> +A high level functional view of the Mali-C55 ISP is presented below. The ISP
> +takes input from either a live source or through a DMA engine for memory input,
> +depending on the SoC integration.::
> +
> +  +---------+    +----------+                                     +--------+
> +  | Sensor  |--->| CSI-2 Rx |                "Full Resolution"    |  DMA   |
> +  +---------+    +----------+   |\                 Output    +--->| Writer |
> +                       |        | \                          |    +--------+
> +                       |        |  \    +----------+  +------+---> Streaming I/O
> +  +------------+       +------->|   |   |          |  |
> +  |            |                |   |-->| Mali-C55 |--+
> +  | DMA Reader |--------------->|   |   |    ISP   |  |
> +  |            |                |  /    |          |  |      +---> Streaming I/O
> +  +------------+                | /     +----------+  |      |
> +                                |/                    +------+
> +				                             |    +--------+
> +                                                             +--->|  DMA   |
> +                                               "Downscaled"       | Writer |
> +					          Output          +--------+

You have a mix of tabs and spaces here.

> +
> +Media Controller Topology
> +=========================
> +
> +An example of the ISP's topology (as implemented in a system with an IMX415
> +camera sensor and generic CSI-2 receiver) is below:
> +
> +
> +.. kernel-figure:: mali-c55-graph.dot
> +    :alt:   mali-c55-graph.dot
> +    :align: center
> +
> +The driver has 4 V4L2 subdevices:
> +
> +- `mali_c55 isp`: Responsible for configuring input crop and color space
> +                  conversion
> +- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor.
> +- `mali_c55 resizer fr`: The Full-Resolution pipe resizer
> +- `mali_c55 resizer ds`: The Downscale pipe resizer
> +
> +The driver has 2 V4L2 video devices:
> +
> +- `mali-c55 fr`: The full-resolution pipe's capture device
> +- `mali-c55 ds`: The downscale pipe's capture device
> +
> +Frame sequences are synchronised across to two capture devices, meaning if one
> +pipe is started later than the other the sequence numbers returned in its
> +buffers will match those of the other pipe rather than starting from zero.
> +
> +Frame sequences are synchronised across to two capture devices, meaning if one
> +pipe is started later than the other the sequence numbers returned in its
> +buffers will match those of the other pipe rather than starting from zero.

I think you can explain this once only.

> +
> +Idiosyncrasies
> +--------------
> +
> +**mali-c55 isp**
> +The `mali-c55 isp` subdevice has a single sink pad to which all sources of data
> +should be connected. The active source is selected by enabling the appropriate
> +media link and disabling all others.

Modelling this with links prevents switching between sources at runtime.
It also makes it possible to misconfigure the pipeline by disconnecting
the two sources. This would be caught at pipeline validation time, but
it still adds complexity.

I was considering using the subdev routing API instead, which would
allow runtime reconfiguration, and prevent invalid configuration in the
first place. The downside is that we would need a mux subdev in front of
the ISP. In terms of additional complexity, that's clearly not great.

Given that switching between the sensor and TPG at runtime is likely not
an important use case, and that the harware may not even support it at
all, we can probably keep the existing graph and driver implementation.

> The ISP has two source pads, reflecting the
> +different paths through which it can internally route data. Tap points within
> +the ISP allow users to divert data to avoid processing by some or all of the
> +hardware's processing steps. The diagram below is intended only to highlight how
> +the bypassing works and is not a true reflection of those processing steps; for
> +a high-level functional block diagram see ARM's developer page for the
> +ISP [3]_::
> +
> +  +--------------------------------------------------------------+
> +  |                Possible Internal ISP Data Routes             |
> +  |          +------------+  +----------+  +------------+        |
> +  +---+      |            |  |          |  |  Colour    |    +---+
> +  | 0 |--+-->| Processing |->| Demosaic |->|   Space    |--->| 1 |
> +  +---+  |   |            |  |          |  | Conversion |    +---+
> +  |      |   +------------+  +----------+  +------------+        |
> +  |      |                                                   +---+
> +  |      +---------------------------------------------------| 2 |
> +  |                                                          +---+
> +  |                                                              |
> +  +--------------------------------------------------------------+
> +
> +
> +.. flat-table::
> +    :header-rows: 1
> +
> +    * - Pad
> +      - Direction
> +      - Purpose
> +
> +    * - 0
> +      - sink
> +      - Data input, connected to the TPG and camera sensors
> +
> +    * - 1
> +      - source
> +      - RGB/YUV data, connected to the FR and DS V4L2 subdevices
> +
> +    * - 2
> +      - source
> +      - RAW bayer data, connected to the FR V4L2 subdevices
> +
> +The ISP is limited to both input and output resolutions between 640x480 and
> +8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt()
> +operations.
> +
> +**mali-c55 resizer fr**
> +The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different
> +insertion points in the hardware (either RAW or demosaiced data):
> +
> +.. flat-table::
> +    :header-rows: 1
> +
> +    * - Pad
> +      - Direction
> +      - Purpose
> +
> +    * - 0
> +      - sink
> +      - Data input connected to the ISP's demosaiced stream.
> +
> +    * - 1
> +      - source
> +      - Data output connected to the capture video device
> +
> +    * - 2
> +      - sink
> +      - Data input connected to the ISP's raw data stream
> +
> +The data source in use is selected through the routing API; two routes each of a
> +single stream are available:
> +
> +.. flat-table::
> +    :header-rows: 1
> +
> +    * - Sink Pad
> +      - Source Pad
> +      - Purpose
> +
> +    * - 0
> +      - 1
> +      - Demosaiced data route
> +
> +    * - 2
> +      - 1
> +      - Raw data route
> +
> +
> +If the demosaiced route is active then the FR pipe is only capable of output
> +in RGB/YUV formats. If the raw route is active then the output reflects the
> +input (which may be either Bayer or RGB/YUV data).
> +
> +Using the driver to capture video
> +=================================
> +
> +Using the media controller APIs we can configure the input source and ISP to
> +capture images in a variety of formats. In the examples below, configuring the
> +media graph is done with the v4l-utils [1]_ package's media-ctl utility.
> +Capturing the images is done with yavta [2]_.
> +
> +Configuring the input source
> +----------------------------
> +
> +The first step is to set the input source that we wish by enabling the correct
> +media link. Using the example topology above, we can select the TPG as follows:
> +
> +.. code-block:: none
> +
> +    media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]"
> +    media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]"
> +
> +Capturing bayer data from the source and processing to RGB/YUV
> +--------------------------------------------------------------

Missing blank line.

> +To capture 1920x1080 bayer data from the source and push it through the ISP's
> +full processing pipeline, we configure the data formats appropriately on the
> +source, ISP and resizer subdevices and set the FR resizer's routing to select
> +processed data. The media bus format on the resizer's source pad will be either
> +RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or
> +YUV. The ISP's debayering block outputs RGB data natively, setting the source
> +pad format to YUV10_1X30 enables the colour space conversion block.
> +
> +In this example we target RGB565 output, so select RGB121212_1X36 as the resizer
> +source pad's format:
> +
> +.. code-block:: none
> +
> +    # Set formats on the TPG and ISP
> +    media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB16_1X16/1920x1080]"
> +    media-ctl -V "'mali-c55 isp':0[fmt:SRGGB16_1X16/1920x1080]"
> +    media-ctl -V "'mali-c55 isp':1[fmt:SRGGB16_1X16/1920x1080]"
> +
> +    # Set routing on the FR resizer
> +    media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]"
> +
> +    # Set format on the resizer, must be done AFTER the routing.
> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]"
> +
> +The downscale output can also be used to stream data at the same time. In this
> +case since only processed data can be captured through the downscale output no
> +routing need be set:
> +
> +.. code-block:: none
> +
> +    # Set format on the resizer
> +    media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]"
> +
> +Following which images can be captured from both the FR and DS output's video
> +devices (simultaneously, if desired):
> +
> +.. code-block:: none
> +
> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video1
> +
> +Cropping the image
> +~~~~~~~~~~~~~~~~~~
> +
> +Both the full resolution and downscale pipes can crop to a minimum resolution of
> +640x480. To crop the image simply configure the resizer's sink pad's crop and
> +compose rectangles and set the format on the video device:
> +
> +.. code-block:: none
> +
> +    media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]"
> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
> +    yavta -f RGB565 -s 640x480 -c10 /dev/video0
> +
> +Downscaling the image
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +Both the full resolution and downscale pipes can downscale the image by up to 8x
> +provided the minimum 640x480 resolution is adhered to. For the best image result

Maybe "minimum 640x480 output resolution".

> +the scaling ratio for each dimension should be the same. To configure scaling we

s/dimension/direction/

> +use the compose rectangle on the resizer's sink pad:
> +
> +.. code-block:: none
> +
> +    media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]"
> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
> +    yavta -f RGB565 -s 640x480 -c10 /dev/video0
> +
> +Capturing images in YUV formats
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +If we need to output YUV data rather than RGB the color space conversion block
> +needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the
> +resizer's source pad. We can then configure a capture format like NV12 (here in
> +its multi-planar variant)
> +
> +.. code-block:: none
> +
> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]"
> +    yavta -f NV12M -s 1920x1080 -c10 /dev/video0
> +
> +Capturing RGB data from the source and processing it with the resizers
> +----------------------------------------------------------------------
> +
> +The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this
> +case although none of the image quality blocks would be used it can still
> +crop/scale the data in the usual way.
> +
> +To achieve this, the ISP's sink pad's format is set to
> +MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to
> +work with the ISP. Converting the camera sensor's output to that format is the
> +responsibility of external hardware.
> +
> +In this example we ask the test pattern generator to give us RGB data instead of
> +bayer.
> +
> +.. code-block:: none
> +
> +    media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]"
> +    media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]"
> +
> +Cropping or scaling the data can be done in exactly the same way as outlined
> +earlier.

Do we use the ISP's output on pad 1 or pad 2 in this case ? The text
seems to imply that the ISP is bypassed, but the example doesn't mention
any routing change. You may want to clarify this.

> +
> +Capturing raw data from the source and outputting it unmodified
> +-----------------------------------------------------------------
> +
> +The ISP can additionally capture raw data from the source and output it on the
> +full resolution pipe only, completely unmodified. In this case the downscale
> +pipe can still process the data normally and be used at the same time.
> +
> +To configure raw bypass the FR resizer's subdevice's routing table needs to be
> +configured, followed by formats in the appropriate places:
> +
> +.. code-block:: none
> +
> +    # We need to configure the routing table for the resizer to use the bypass
> +    # path along with set formats on the resizer's bypass sink pad. Doing this
> +    # necessitates a single media-ctl command, as multiple calls to the program
> +    # reset the routing table.

Really ? Does -V reset the routing table ? That surprises me.

> +    media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"\
> +    -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080],"\
> +       "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080],"\
> +       "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]"
> +
> +    # Set format on the video device and stream
> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
> +
> +References
> +==========
> +.. [1] https://git.linuxtv.org/v4l-utils.git/
> +.. [2] https://git.ideasonboard.org/yavta.git
> +.. [3] https://developer.arm.com/Processors/Mali-C55
> diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst
> index 4120eded9a13..1d9485860d93 100644
> --- a/Documentation/admin-guide/media/v4l-drivers.rst
> +++ b/Documentation/admin-guide/media/v4l-drivers.rst
> @@ -18,6 +18,7 @@ Video4Linux (V4L) driver-specific documentation
>  	ipu3
>  	ipu6-isys
>  	ivtv
> +	mali-c55
>  	mgb4
>  	omap3isp
>  	omap4_camera

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 06/16] media: Documentation: Add Mali-C55 ISP Documentation

[Dan Scally]

Hi Laurent

On 29/05/2024 21:22, Laurent Pinchart wrote:
> Hi Dan,
>
> Thank you for the patch.
>
> On Wed, May 29, 2024 at 04:28:48PM +0100, Daniel Scally wrote:
>> Add a documentation page for the mali-c55 driver, which gives a brief
>> overview of the hardware and explains how to use the driver's capture
>> devices and the crop/scaler functions.
>>
>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>> ---
>> Changes in v5:
>>
>> 	- None
>>
>> Changes in v4:
>> 	- None
>>
>> Changes in v3:
>> 	- Documented the synchronised buffer sequence numbers (Sakari)
>> 	- Clarified that the downscale pipe cannot output raw data, the ISP'S
>> 	  resolution limits and choice of media bus format code (Kieran)
>>
>> Changes in v2:
>>
>> 	- none
>>
>>   .../admin-guide/media/mali-c55-graph.dot      |  19 +
>>   Documentation/admin-guide/media/mali-c55.rst  | 333 ++++++++++++++++++
>>   .../admin-guide/media/v4l-drivers.rst         |   1 +
>>   3 files changed, 353 insertions(+)
>>   create mode 100644 Documentation/admin-guide/media/mali-c55-graph.dot
>>   create mode 100644 Documentation/admin-guide/media/mali-c55.rst
>>
>> diff --git a/Documentation/admin-guide/media/mali-c55-graph.dot b/Documentation/admin-guide/media/mali-c55-graph.dot
>> new file mode 100644
>> index 000000000000..0775ba42bf4c
>> --- /dev/null
>> +++ b/Documentation/admin-guide/media/mali-c55-graph.dot
>> @@ -0,0 +1,19 @@
>> +digraph board {
>> +        rankdir=TB
>> +        n00000001 [label="{{} | mali-c55 tpg\n/dev/v4l-subdev0 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
>> +        n00000001:port0 -> n00000003:port0 [style=dashed]
>> +        n00000003 [label="{{<port0> 0} | mali-c55 isp\n/dev/v4l-subdev1 | {<port1> 1 | <port2> 2}}", shape=Mrecord, style=filled, fillcolor=green]
>> +        n00000003:port1 -> n00000007:port0 [style=bold]
>> +        n00000003:port2 -> n00000007:port2 [style=bold]
>> +        n00000003:port1 -> n0000000b:port0 [style=bold]
>> +        n00000007 [label="{{<port0> 0 | <port2> 2} | mali-c55 resizer fr\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
>> +        n00000007:port1 -> n0000000e [style=bold]
>> +        n0000000b [label="{{<port0> 0} | mali-c55 resizer ds\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
>> +        n0000000b:port1 -> n00000012 [style=bold]
>> +        n0000000e [label="mali-c55 fr\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
>> +        n00000012 [label="mali-c55 ds\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
>> +        n00000022 [label="{{<port0> 0} | csi2-rx\n/dev/v4l-subdev4 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
>> +        n00000022:port1 -> n00000003:port0
>> +        n00000027 [label="{{} | imx415 1-001a\n/dev/v4l-subdev5 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
>> +        n00000027:port0 -> n00000022:port0 [style=bold]
>> +}
>> \ No newline at end of file
>> diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst
>> new file mode 100644
>> index 000000000000..cf4176cb1e44
>> --- /dev/null
>> +++ b/Documentation/admin-guide/media/mali-c55.rst
>> @@ -0,0 +1,333 @@
>> +.. SPDX-License-Identifier: GPL-2.0
>> +
>> +==========================================
>> +ARM Mali-C55 Image Signal Processor driver
>> +==========================================
>> +
>> +Introduction
>> +============
>> +
>> +This file documents the driver for ARM's Mali-C55 Image Signal Processor. The
>> +driver is located under drivers/media/platform/arm/mali-c55.
>> +
>> +The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from
>> +sensors through either a parallel interface or a memory bus before processing it
>> +and outputting it through an internal DMA engine. Two output pipelines are
>> +possible (though one may not be fitted, depending on the implementation). These
>> +are referred to as "Full resolution" and "Downscale", but the naming is historic
>> +and both pipes are capable of cropping/scaling operations. The full resolution
>> +pipe is also capable of outputting RAW data, bypassing much of the ISP's
>> +processing. The downscale pipe cannot output RAW data. An integrated test
>> +pattern generator can be used to drive the ISP and produce image data in the
>> +absence of a connected camera sensor. The driver module is named mali_c55, and
>> +is enabled through the CONFIG_VIDEO_MALI_C55 config option.
>> +
>> +The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and
>> +expects camera sensors connected to the ISP to have V4L2 subdevice interfaces.
>> +
>> +Mali-C55 ISP hardware
>> +=====================
>> +
>> +A high level functional view of the Mali-C55 ISP is presented below. The ISP
>> +takes input from either a live source or through a DMA engine for memory input,
>> +depending on the SoC integration.::
>> +
>> +  +---------+    +----------+                                     +--------+
>> +  | Sensor  |--->| CSI-2 Rx |                "Full Resolution"    |  DMA   |
>> +  +---------+    +----------+   |\                 Output    +--->| Writer |
>> +                       |        | \                          |    +--------+
>> +                       |        |  \    +----------+  +------+---> Streaming I/O
>> +  +------------+       +------->|   |   |          |  |
>> +  |            |                |   |-->| Mali-C55 |--+
>> +  | DMA Reader |--------------->|   |   |    ISP   |  |
>> +  |            |                |  /    |          |  |      +---> Streaming I/O
>> +  +------------+                | /     +----------+  |      |
>> +                                |/                    +------+
>> +				                             |    +--------+
>> +                                                             +--->|  DMA   |
>> +                                               "Downscaled"       | Writer |
>> +					          Output          +--------+
> You have a mix of tabs and spaces here.
>
>> +
>> +Media Controller Topology
>> +=========================
>> +
>> +An example of the ISP's topology (as implemented in a system with an IMX415
>> +camera sensor and generic CSI-2 receiver) is below:
>> +
>> +
>> +.. kernel-figure:: mali-c55-graph.dot
>> +    :alt:   mali-c55-graph.dot
>> +    :align: center
>> +
>> +The driver has 4 V4L2 subdevices:
>> +
>> +- `mali_c55 isp`: Responsible for configuring input crop and color space
>> +                  conversion
>> +- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor.
>> +- `mali_c55 resizer fr`: The Full-Resolution pipe resizer
>> +- `mali_c55 resizer ds`: The Downscale pipe resizer
>> +
>> +The driver has 2 V4L2 video devices:
>> +
>> +- `mali-c55 fr`: The full-resolution pipe's capture device
>> +- `mali-c55 ds`: The downscale pipe's capture device
>> +
>> +Frame sequences are synchronised across to two capture devices, meaning if one
>> +pipe is started later than the other the sequence numbers returned in its
>> +buffers will match those of the other pipe rather than starting from zero.
>> +
>> +Frame sequences are synchronised across to two capture devices, meaning if one
>> +pipe is started later than the other the sequence numbers returned in its
>> +buffers will match those of the other pipe rather than starting from zero.
> I think you can explain this once only.


Oops...

>
>> +
>> +Idiosyncrasies
>> +--------------
>> +
>> +**mali-c55 isp**
>> +The `mali-c55 isp` subdevice has a single sink pad to which all sources of data
>> +should be connected. The active source is selected by enabling the appropriate
>> +media link and disabling all others.
> Modelling this with links prevents switching between sources at runtime.
> It also makes it possible to misconfigure the pipeline by disconnecting
> the two sources. This would be caught at pipeline validation time, but
> it still adds complexity.
>
> I was considering using the subdev routing API instead, which would
> allow runtime reconfiguration, and prevent invalid configuration in the
> first place. The downside is that we would need a mux subdev in front of
> the ISP. In terms of additional complexity, that's clearly not great.
>
> Given that switching between the sensor and TPG at runtime is likely not
> an important use case, and that the harware may not even support it at
> all, we can probably keep the existing graph and driver implementation.


I suppose in the long term we need to think about how this should be modeled in a multi-context 
system...when we have a media graph with 8 cameras connected (somehow) to the ISP's single sink pad 
how should we select the right input device for a context? Whatever the answer is there, probably we 
should do it here...if we can't do it at runtime with links then probably it has to be a mux...or 
some novel thing.

>
>> The ISP has two source pads, reflecting the
>> +different paths through which it can internally route data. Tap points within
>> +the ISP allow users to divert data to avoid processing by some or all of the
>> +hardware's processing steps. The diagram below is intended only to highlight how
>> +the bypassing works and is not a true reflection of those processing steps; for
>> +a high-level functional block diagram see ARM's developer page for the
>> +ISP [3]_::
>> +
>> +  +--------------------------------------------------------------+
>> +  |                Possible Internal ISP Data Routes             |
>> +  |          +------------+  +----------+  +------------+        |
>> +  +---+      |            |  |          |  |  Colour    |    +---+
>> +  | 0 |--+-->| Processing |->| Demosaic |->|   Space    |--->| 1 |
>> +  +---+  |   |            |  |          |  | Conversion |    +---+
>> +  |      |   +------------+  +----------+  +------------+        |
>> +  |      |                                                   +---+
>> +  |      +---------------------------------------------------| 2 |
>> +  |                                                          +---+
>> +  |                                                              |
>> +  +--------------------------------------------------------------+
>> +
>> +
>> +.. flat-table::
>> +    :header-rows: 1
>> +
>> +    * - Pad
>> +      - Direction
>> +      - Purpose
>> +
>> +    * - 0
>> +      - sink
>> +      - Data input, connected to the TPG and camera sensors
>> +
>> +    * - 1
>> +      - source
>> +      - RGB/YUV data, connected to the FR and DS V4L2 subdevices
>> +
>> +    * - 2
>> +      - source
>> +      - RAW bayer data, connected to the FR V4L2 subdevices
>> +
>> +The ISP is limited to both input and output resolutions between 640x480 and
>> +8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt()
>> +operations.
>> +
>> +**mali-c55 resizer fr**
>> +The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different
>> +insertion points in the hardware (either RAW or demosaiced data):
>> +
>> +.. flat-table::
>> +    :header-rows: 1
>> +
>> +    * - Pad
>> +      - Direction
>> +      - Purpose
>> +
>> +    * - 0
>> +      - sink
>> +      - Data input connected to the ISP's demosaiced stream.
>> +
>> +    * - 1
>> +      - source
>> +      - Data output connected to the capture video device
>> +
>> +    * - 2
>> +      - sink
>> +      - Data input connected to the ISP's raw data stream
>> +
>> +The data source in use is selected through the routing API; two routes each of a
>> +single stream are available:
>> +
>> +.. flat-table::
>> +    :header-rows: 1
>> +
>> +    * - Sink Pad
>> +      - Source Pad
>> +      - Purpose
>> +
>> +    * - 0
>> +      - 1
>> +      - Demosaiced data route
>> +
>> +    * - 2
>> +      - 1
>> +      - Raw data route
>> +
>> +
>> +If the demosaiced route is active then the FR pipe is only capable of output
>> +in RGB/YUV formats. If the raw route is active then the output reflects the
>> +input (which may be either Bayer or RGB/YUV data).
>> +
>> +Using the driver to capture video
>> +=================================
>> +
>> +Using the media controller APIs we can configure the input source and ISP to
>> +capture images in a variety of formats. In the examples below, configuring the
>> +media graph is done with the v4l-utils [1]_ package's media-ctl utility.
>> +Capturing the images is done with yavta [2]_.
>> +
>> +Configuring the input source
>> +----------------------------
>> +
>> +The first step is to set the input source that we wish by enabling the correct
>> +media link. Using the example topology above, we can select the TPG as follows:
>> +
>> +.. code-block:: none
>> +
>> +    media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]"
>> +    media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]"
>> +
>> +Capturing bayer data from the source and processing to RGB/YUV
>> +--------------------------------------------------------------
> Missing blank line.
>
>> +To capture 1920x1080 bayer data from the source and push it through the ISP's
>> +full processing pipeline, we configure the data formats appropriately on the
>> +source, ISP and resizer subdevices and set the FR resizer's routing to select
>> +processed data. The media bus format on the resizer's source pad will be either
>> +RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or
>> +YUV. The ISP's debayering block outputs RGB data natively, setting the source
>> +pad format to YUV10_1X30 enables the colour space conversion block.
>> +
>> +In this example we target RGB565 output, so select RGB121212_1X36 as the resizer
>> +source pad's format:
>> +
>> +.. code-block:: none
>> +
>> +    # Set formats on the TPG and ISP
>> +    media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB16_1X16/1920x1080]"
>> +    media-ctl -V "'mali-c55 isp':0[fmt:SRGGB16_1X16/1920x1080]"
>> +    media-ctl -V "'mali-c55 isp':1[fmt:SRGGB16_1X16/1920x1080]"
>> +
>> +    # Set routing on the FR resizer
>> +    media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]"
>> +
>> +    # Set format on the resizer, must be done AFTER the routing.
>> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]"
>> +
>> +The downscale output can also be used to stream data at the same time. In this
>> +case since only processed data can be captured through the downscale output no
>> +routing need be set:
>> +
>> +.. code-block:: none
>> +
>> +    # Set format on the resizer
>> +    media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]"
>> +
>> +Following which images can be captured from both the FR and DS output's video
>> +devices (simultaneously, if desired):
>> +
>> +.. code-block:: none
>> +
>> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
>> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video1
>> +
>> +Cropping the image
>> +~~~~~~~~~~~~~~~~~~
>> +
>> +Both the full resolution and downscale pipes can crop to a minimum resolution of
>> +640x480. To crop the image simply configure the resizer's sink pad's crop and
>> +compose rectangles and set the format on the video device:
>> +
>> +.. code-block:: none
>> +
>> +    media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]"
>> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
>> +    yavta -f RGB565 -s 640x480 -c10 /dev/video0
>> +
>> +Downscaling the image
>> +~~~~~~~~~~~~~~~~~~~~~
>> +
>> +Both the full resolution and downscale pipes can downscale the image by up to 8x
>> +provided the minimum 640x480 resolution is adhered to. For the best image result
> Maybe "minimum 640x480 output resolution".
>
>> +the scaling ratio for each dimension should be the same. To configure scaling we
> s/dimension/direction/
>
>> +use the compose rectangle on the resizer's sink pad:
>> +
>> +.. code-block:: none
>> +
>> +    media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]"
>> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
>> +    yavta -f RGB565 -s 640x480 -c10 /dev/video0
>> +
>> +Capturing images in YUV formats
>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +If we need to output YUV data rather than RGB the color space conversion block
>> +needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the
>> +resizer's source pad. We can then configure a capture format like NV12 (here in
>> +its multi-planar variant)
>> +
>> +.. code-block:: none
>> +
>> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]"
>> +    yavta -f NV12M -s 1920x1080 -c10 /dev/video0
>> +
>> +Capturing RGB data from the source and processing it with the resizers
>> +----------------------------------------------------------------------
>> +
>> +The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this
>> +case although none of the image quality blocks would be used it can still
>> +crop/scale the data in the usual way.
>> +
>> +To achieve this, the ISP's sink pad's format is set to
>> +MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to
>> +work with the ISP. Converting the camera sensor's output to that format is the
>> +responsibility of external hardware.
>> +
>> +In this example we ask the test pattern generator to give us RGB data instead of
>> +bayer.
>> +
>> +.. code-block:: none
>> +
>> +    media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]"
>> +    media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]"
>> +
>> +Cropping or scaling the data can be done in exactly the same way as outlined
>> +earlier.
> Do we use the ISP's output on pad 1 or pad 2 in this case ? The text
> seems to imply that the ISP is bypassed, but the example doesn't mention
> any routing change. You may want to clarify this.


Pad 1 for bypassed RGB data. I'll make it more clear.

>
>> +
>> +Capturing raw data from the source and outputting it unmodified
>> +-----------------------------------------------------------------
>> +
>> +The ISP can additionally capture raw data from the source and output it on the
>> +full resolution pipe only, completely unmodified. In this case the downscale
>> +pipe can still process the data normally and be used at the same time.
>> +
>> +To configure raw bypass the FR resizer's subdevice's routing table needs to be
>> +configured, followed by formats in the appropriate places:
>> +
>> +.. code-block:: none
>> +
>> +    # We need to configure the routing table for the resizer to use the bypass
>> +    # path along with set formats on the resizer's bypass sink pad. Doing this
>> +    # necessitates a single media-ctl command, as multiple calls to the program
>> +    # reset the routing table.
> Really ?


Yeah

> Does -V reset the routing table ? That surprises me.


It's not -V, it's the fact of opening a subdev which calls .init_state(), which resets the routing 
table...in a single process the fds are held open throughout so all is well, but if you run the 
program twice they're opened each time and the routing is reset.

>
>> +    media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"\
>> +    -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080],"\
>> +       "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080],"\
>> +       "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]"
>> +
>> +    # Set format on the video device and stream
>> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
>> +
>> +References
>> +==========
>> +.. [1] https://git.linuxtv.org/v4l-utils.git/
>> +.. [2] https://git.ideasonboard.org/yavta.git
>> +.. [3] https://developer.arm.com/Processors/Mali-C55
>> diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst
>> index 4120eded9a13..1d9485860d93 100644
>> --- a/Documentation/admin-guide/media/v4l-drivers.rst
>> +++ b/Documentation/admin-guide/media/v4l-drivers.rst
>> @@ -18,6 +18,7 @@ Video4Linux (V4L) driver-specific documentation
>>   	ipu3
>>   	ipu6-isys
>>   	ivtv
>> +	mali-c55
>>   	mgb4
>>   	omap3isp
>>   	omap4_camera

Re: [PATCH v5 06/16] media: Documentation: Add Mali-C55 ISP Documentation

[Laurent Pinchart]

On Wed, May 29, 2024 at 09:35:08PM +0100, Daniel Scally wrote:
> On 29/05/2024 21:22, Laurent Pinchart wrote:
> > On Wed, May 29, 2024 at 04:28:48PM +0100, Daniel Scally wrote:
> >> Add a documentation page for the mali-c55 driver, which gives a brief
> >> overview of the hardware and explains how to use the driver's capture
> >> devices and the crop/scaler functions.
> >>
> >> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
> >> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> >> ---
> >> Changes in v5:
> >>
> >> 	- None
> >>
> >> Changes in v4:
> >> 	- None
> >>
> >> Changes in v3:
> >> 	- Documented the synchronised buffer sequence numbers (Sakari)
> >> 	- Clarified that the downscale pipe cannot output raw data, the ISP'S
> >> 	  resolution limits and choice of media bus format code (Kieran)
> >>
> >> Changes in v2:
> >>
> >> 	- none
> >>
> >>   .../admin-guide/media/mali-c55-graph.dot      |  19 +
> >>   Documentation/admin-guide/media/mali-c55.rst  | 333 ++++++++++++++++++
> >>   .../admin-guide/media/v4l-drivers.rst         |   1 +
> >>   3 files changed, 353 insertions(+)
> >>   create mode 100644 Documentation/admin-guide/media/mali-c55-graph.dot
> >>   create mode 100644 Documentation/admin-guide/media/mali-c55.rst
> >>
> >> diff --git a/Documentation/admin-guide/media/mali-c55-graph.dot b/Documentation/admin-guide/media/mali-c55-graph.dot
> >> new file mode 100644
> >> index 000000000000..0775ba42bf4c
> >> --- /dev/null
> >> +++ b/Documentation/admin-guide/media/mali-c55-graph.dot
> >> @@ -0,0 +1,19 @@
> >> +digraph board {
> >> +        rankdir=TB
> >> +        n00000001 [label="{{} | mali-c55 tpg\n/dev/v4l-subdev0 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
> >> +        n00000001:port0 -> n00000003:port0 [style=dashed]
> >> +        n00000003 [label="{{<port0> 0} | mali-c55 isp\n/dev/v4l-subdev1 | {<port1> 1 | <port2> 2}}", shape=Mrecord, style=filled, fillcolor=green]
> >> +        n00000003:port1 -> n00000007:port0 [style=bold]
> >> +        n00000003:port2 -> n00000007:port2 [style=bold]
> >> +        n00000003:port1 -> n0000000b:port0 [style=bold]
> >> +        n00000007 [label="{{<port0> 0 | <port2> 2} | mali-c55 resizer fr\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
> >> +        n00000007:port1 -> n0000000e [style=bold]
> >> +        n0000000b [label="{{<port0> 0} | mali-c55 resizer ds\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
> >> +        n0000000b:port1 -> n00000012 [style=bold]
> >> +        n0000000e [label="mali-c55 fr\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
> >> +        n00000012 [label="mali-c55 ds\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
> >> +        n00000022 [label="{{<port0> 0} | csi2-rx\n/dev/v4l-subdev4 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
> >> +        n00000022:port1 -> n00000003:port0
> >> +        n00000027 [label="{{} | imx415 1-001a\n/dev/v4l-subdev5 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
> >> +        n00000027:port0 -> n00000022:port0 [style=bold]
> >> +}
> >> \ No newline at end of file
> >> diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst
> >> new file mode 100644
> >> index 000000000000..cf4176cb1e44
> >> --- /dev/null
> >> +++ b/Documentation/admin-guide/media/mali-c55.rst
> >> @@ -0,0 +1,333 @@
> >> +.. SPDX-License-Identifier: GPL-2.0
> >> +
> >> +==========================================
> >> +ARM Mali-C55 Image Signal Processor driver
> >> +==========================================
> >> +
> >> +Introduction
> >> +============
> >> +
> >> +This file documents the driver for ARM's Mali-C55 Image Signal Processor. The
> >> +driver is located under drivers/media/platform/arm/mali-c55.
> >> +
> >> +The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from
> >> +sensors through either a parallel interface or a memory bus before processing it
> >> +and outputting it through an internal DMA engine. Two output pipelines are
> >> +possible (though one may not be fitted, depending on the implementation). These
> >> +are referred to as "Full resolution" and "Downscale", but the naming is historic
> >> +and both pipes are capable of cropping/scaling operations. The full resolution
> >> +pipe is also capable of outputting RAW data, bypassing much of the ISP's
> >> +processing. The downscale pipe cannot output RAW data. An integrated test
> >> +pattern generator can be used to drive the ISP and produce image data in the
> >> +absence of a connected camera sensor. The driver module is named mali_c55, and
> >> +is enabled through the CONFIG_VIDEO_MALI_C55 config option.
> >> +
> >> +The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and
> >> +expects camera sensors connected to the ISP to have V4L2 subdevice interfaces.
> >> +
> >> +Mali-C55 ISP hardware
> >> +=====================
> >> +
> >> +A high level functional view of the Mali-C55 ISP is presented below. The ISP
> >> +takes input from either a live source or through a DMA engine for memory input,
> >> +depending on the SoC integration.::
> >> +
> >> +  +---------+    +----------+                                     +--------+
> >> +  | Sensor  |--->| CSI-2 Rx |                "Full Resolution"    |  DMA   |
> >> +  +---------+    +----------+   |\                 Output    +--->| Writer |
> >> +                       |        | \                          |    +--------+
> >> +                       |        |  \    +----------+  +------+---> Streaming I/O
> >> +  +------------+       +------->|   |   |          |  |
> >> +  |            |                |   |-->| Mali-C55 |--+
> >> +  | DMA Reader |--------------->|   |   |    ISP   |  |
> >> +  |            |                |  /    |          |  |      +---> Streaming I/O
> >> +  +------------+                | /     +----------+  |      |
> >> +                                |/                    +------+
> >> +				                             |    +--------+
> >> +                                                             +--->|  DMA   |
> >> +                                               "Downscaled"       | Writer |
> >> +					          Output          +--------+
> >
> > You have a mix of tabs and spaces here.
> >
> >> +
> >> +Media Controller Topology
> >> +=========================
> >> +
> >> +An example of the ISP's topology (as implemented in a system with an IMX415
> >> +camera sensor and generic CSI-2 receiver) is below:
> >> +
> >> +
> >> +.. kernel-figure:: mali-c55-graph.dot
> >> +    :alt:   mali-c55-graph.dot
> >> +    :align: center
> >> +
> >> +The driver has 4 V4L2 subdevices:
> >> +
> >> +- `mali_c55 isp`: Responsible for configuring input crop and color space
> >> +                  conversion
> >> +- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor.
> >> +- `mali_c55 resizer fr`: The Full-Resolution pipe resizer
> >> +- `mali_c55 resizer ds`: The Downscale pipe resizer
> >> +
> >> +The driver has 2 V4L2 video devices:
> >> +
> >> +- `mali-c55 fr`: The full-resolution pipe's capture device
> >> +- `mali-c55 ds`: The downscale pipe's capture device
> >> +
> >> +Frame sequences are synchronised across to two capture devices, meaning if one
> >> +pipe is started later than the other the sequence numbers returned in its
> >> +buffers will match those of the other pipe rather than starting from zero.
> >> +
> >> +Frame sequences are synchronised across to two capture devices, meaning if one
> >> +pipe is started later than the other the sequence numbers returned in its
> >> +buffers will match those of the other pipe rather than starting from zero.
> >
> > I think you can explain this once only.
> 
> Oops...
> 
> >> +
> >> +Idiosyncrasies
> >> +--------------
> >> +
> >> +**mali-c55 isp**
> >> +The `mali-c55 isp` subdevice has a single sink pad to which all sources of data
> >> +should be connected. The active source is selected by enabling the appropriate
> >> +media link and disabling all others.
> >
> > Modelling this with links prevents switching between sources at runtime.
> > It also makes it possible to misconfigure the pipeline by disconnecting
> > the two sources. This would be caught at pipeline validation time, but
> > it still adds complexity.
> >
> > I was considering using the subdev routing API instead, which would
> > allow runtime reconfiguration, and prevent invalid configuration in the
> > first place. The downside is that we would need a mux subdev in front of
> > the ISP. In terms of additional complexity, that's clearly not great.
> >
> > Given that switching between the sensor and TPG at runtime is likely not
> > an important use case, and that the harware may not even support it at
> > all, we can probably keep the existing graph and driver implementation.
> 
> I suppose in the long term we need to think about how this should be
> modeled in a multi-context system...when we have a media graph with 8
> cameras connected (somehow) to the ISP's single sink pad how should we
> select the right input device for a context? Whatever the answer is
> there, probably we should do it here...if we can't do it at runtime
> with links then probably it has to be a mux...or some novel thing.

Good point. Although... In M2M operation, will we need the TPG at all ?
Userspace could easily supply test patterns in the input buffers. I
don't think we can time-multiplex the ISP in a mode where some of the
inputs would come from memory, and some from a live source (be it the
TPG or a sensor).

Sticking to the technical discussion, I envision that the contexts will
be handled by opening the media device multiple times. Each file handle
will be associated with one context. The link state would then be stored
in the context. The links will still not be configurable at runtime
within a context, but could differ between contexts.

> >> The ISP has two source pads, reflecting the
> >> +different paths through which it can internally route data. Tap points within
> >> +the ISP allow users to divert data to avoid processing by some or all of the
> >> +hardware's processing steps. The diagram below is intended only to highlight how
> >> +the bypassing works and is not a true reflection of those processing steps; for
> >> +a high-level functional block diagram see ARM's developer page for the
> >> +ISP [3]_::
> >> +
> >> +  +--------------------------------------------------------------+
> >> +  |                Possible Internal ISP Data Routes             |
> >> +  |          +------------+  +----------+  +------------+        |
> >> +  +---+      |            |  |          |  |  Colour    |    +---+
> >> +  | 0 |--+-->| Processing |->| Demosaic |->|   Space    |--->| 1 |
> >> +  +---+  |   |            |  |          |  | Conversion |    +---+
> >> +  |      |   +------------+  +----------+  +------------+        |
> >> +  |      |                                                   +---+
> >> +  |      +---------------------------------------------------| 2 |
> >> +  |                                                          +---+
> >> +  |                                                              |
> >> +  +--------------------------------------------------------------+
> >> +
> >> +
> >> +.. flat-table::
> >> +    :header-rows: 1
> >> +
> >> +    * - Pad
> >> +      - Direction
> >> +      - Purpose
> >> +
> >> +    * - 0
> >> +      - sink
> >> +      - Data input, connected to the TPG and camera sensors
> >> +
> >> +    * - 1
> >> +      - source
> >> +      - RGB/YUV data, connected to the FR and DS V4L2 subdevices
> >> +
> >> +    * - 2
> >> +      - source
> >> +      - RAW bayer data, connected to the FR V4L2 subdevices
> >> +
> >> +The ISP is limited to both input and output resolutions between 640x480 and
> >> +8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt()
> >> +operations.
> >> +
> >> +**mali-c55 resizer fr**
> >> +The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different
> >> +insertion points in the hardware (either RAW or demosaiced data):
> >> +
> >> +.. flat-table::
> >> +    :header-rows: 1
> >> +
> >> +    * - Pad
> >> +      - Direction
> >> +      - Purpose
> >> +
> >> +    * - 0
> >> +      - sink
> >> +      - Data input connected to the ISP's demosaiced stream.
> >> +
> >> +    * - 1
> >> +      - source
> >> +      - Data output connected to the capture video device
> >> +
> >> +    * - 2
> >> +      - sink
> >> +      - Data input connected to the ISP's raw data stream
> >> +
> >> +The data source in use is selected through the routing API; two routes each of a
> >> +single stream are available:
> >> +
> >> +.. flat-table::
> >> +    :header-rows: 1
> >> +
> >> +    * - Sink Pad
> >> +      - Source Pad
> >> +      - Purpose
> >> +
> >> +    * - 0
> >> +      - 1
> >> +      - Demosaiced data route
> >> +
> >> +    * - 2
> >> +      - 1
> >> +      - Raw data route
> >> +
> >> +
> >> +If the demosaiced route is active then the FR pipe is only capable of output
> >> +in RGB/YUV formats. If the raw route is active then the output reflects the
> >> +input (which may be either Bayer or RGB/YUV data).
> >> +
> >> +Using the driver to capture video
> >> +=================================
> >> +
> >> +Using the media controller APIs we can configure the input source and ISP to
> >> +capture images in a variety of formats. In the examples below, configuring the
> >> +media graph is done with the v4l-utils [1]_ package's media-ctl utility.
> >> +Capturing the images is done with yavta [2]_.
> >> +
> >> +Configuring the input source
> >> +----------------------------
> >> +
> >> +The first step is to set the input source that we wish by enabling the correct
> >> +media link. Using the example topology above, we can select the TPG as follows:
> >> +
> >> +.. code-block:: none
> >> +
> >> +    media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]"
> >> +    media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]"
> >> +
> >> +Capturing bayer data from the source and processing to RGB/YUV
> >> +--------------------------------------------------------------
> >
> > Missing blank line.
> >
> >> +To capture 1920x1080 bayer data from the source and push it through the ISP's
> >> +full processing pipeline, we configure the data formats appropriately on the
> >> +source, ISP and resizer subdevices and set the FR resizer's routing to select
> >> +processed data. The media bus format on the resizer's source pad will be either
> >> +RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or
> >> +YUV. The ISP's debayering block outputs RGB data natively, setting the source
> >> +pad format to YUV10_1X30 enables the colour space conversion block.
> >> +
> >> +In this example we target RGB565 output, so select RGB121212_1X36 as the resizer
> >> +source pad's format:
> >> +
> >> +.. code-block:: none
> >> +
> >> +    # Set formats on the TPG and ISP
> >> +    media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB16_1X16/1920x1080]"
> >> +    media-ctl -V "'mali-c55 isp':0[fmt:SRGGB16_1X16/1920x1080]"
> >> +    media-ctl -V "'mali-c55 isp':1[fmt:SRGGB16_1X16/1920x1080]"
> >> +
> >> +    # Set routing on the FR resizer
> >> +    media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]"
> >> +
> >> +    # Set format on the resizer, must be done AFTER the routing.
> >> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]"
> >> +
> >> +The downscale output can also be used to stream data at the same time. In this
> >> +case since only processed data can be captured through the downscale output no
> >> +routing need be set:
> >> +
> >> +.. code-block:: none
> >> +
> >> +    # Set format on the resizer
> >> +    media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]"
> >> +
> >> +Following which images can be captured from both the FR and DS output's video
> >> +devices (simultaneously, if desired):
> >> +
> >> +.. code-block:: none
> >> +
> >> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
> >> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video1
> >> +
> >> +Cropping the image
> >> +~~~~~~~~~~~~~~~~~~
> >> +
> >> +Both the full resolution and downscale pipes can crop to a minimum resolution of
> >> +640x480. To crop the image simply configure the resizer's sink pad's crop and
> >> +compose rectangles and set the format on the video device:
> >> +
> >> +.. code-block:: none
> >> +
> >> +    media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]"
> >> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
> >> +    yavta -f RGB565 -s 640x480 -c10 /dev/video0
> >> +
> >> +Downscaling the image
> >> +~~~~~~~~~~~~~~~~~~~~~
> >> +
> >> +Both the full resolution and downscale pipes can downscale the image by up to 8x
> >> +provided the minimum 640x480 resolution is adhered to. For the best image result
> >
> > Maybe "minimum 640x480 output resolution".
> >
> >> +the scaling ratio for each dimension should be the same. To configure scaling we
> >
> > s/dimension/direction/
> >
> >> +use the compose rectangle on the resizer's sink pad:
> >> +
> >> +.. code-block:: none
> >> +
> >> +    media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]"
> >> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
> >> +    yavta -f RGB565 -s 640x480 -c10 /dev/video0
> >> +
> >> +Capturing images in YUV formats
> >> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >> +
> >> +If we need to output YUV data rather than RGB the color space conversion block
> >> +needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the
> >> +resizer's source pad. We can then configure a capture format like NV12 (here in
> >> +its multi-planar variant)
> >> +
> >> +.. code-block:: none
> >> +
> >> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]"
> >> +    yavta -f NV12M -s 1920x1080 -c10 /dev/video0
> >> +
> >> +Capturing RGB data from the source and processing it with the resizers
> >> +----------------------------------------------------------------------
> >> +
> >> +The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this
> >> +case although none of the image quality blocks would be used it can still
> >> +crop/scale the data in the usual way.
> >> +
> >> +To achieve this, the ISP's sink pad's format is set to
> >> +MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to
> >> +work with the ISP. Converting the camera sensor's output to that format is the
> >> +responsibility of external hardware.
> >> +
> >> +In this example we ask the test pattern generator to give us RGB data instead of
> >> +bayer.
> >> +
> >> +.. code-block:: none
> >> +
> >> +    media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]"
> >> +    media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]"
> >> +
> >> +Cropping or scaling the data can be done in exactly the same way as outlined
> >> +earlier.
> >
> > Do we use the ISP's output on pad 1 or pad 2 in this case ? The text
> > seems to imply that the ISP is bypassed, but the example doesn't mention
> > any routing change. You may want to clarify this.
> 
> Pad 1 for bypassed RGB data. I'll make it more clear.

Explaining that this doesn't follow the bypass path, but goes through
the processing path and disables the processing blocks that are not
relevant, would help.

> >> +
> >> +Capturing raw data from the source and outputting it unmodified
> >> +-----------------------------------------------------------------
> >> +
> >> +The ISP can additionally capture raw data from the source and output it on the
> >> +full resolution pipe only, completely unmodified. In this case the downscale
> >> +pipe can still process the data normally and be used at the same time.
> >> +
> >> +To configure raw bypass the FR resizer's subdevice's routing table needs to be
> >> +configured, followed by formats in the appropriate places:
> >> +
> >> +.. code-block:: none
> >> +
> >> +    # We need to configure the routing table for the resizer to use the bypass
> >> +    # path along with set formats on the resizer's bypass sink pad. Doing this
> >> +    # necessitates a single media-ctl command, as multiple calls to the program
> >> +    # reset the routing table.
> >
> > Really ?
> 
> Yeah
> 
> > Does -V reset the routing table ? That surprises me.
> 
> It's not -V, it's the fact of opening a subdev which calls
> .init_state(), which resets the routing table...

Only for the TRY state, not the ACTIVE state. If the TRY state leaks to
the ACTIVE state, you may have a driver bugs :-)

> in a single process the fds are held open throughout so all is well, but if you run the 
> program twice they're opened each time and the routing is reset.
> 
> >> +    media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"\
> >> +    -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080],"\
> >> +       "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080],"\
> >> +       "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]"
> >> +
> >> +    # Set format on the video device and stream
> >> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
> >> +
> >> +References
> >> +==========
> >> +.. [1] https://git.linuxtv.org/v4l-utils.git/
> >> +.. [2] https://git.ideasonboard.org/yavta.git
> >> +.. [3] https://developer.arm.com/Processors/Mali-C55
> >> diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst
> >> index 4120eded9a13..1d9485860d93 100644
> >> --- a/Documentation/admin-guide/media/v4l-drivers.rst
> >> +++ b/Documentation/admin-guide/media/v4l-drivers.rst
> >> @@ -18,6 +18,7 @@ Video4Linux (V4L) driver-specific documentation
> >>   	ipu3
> >>   	ipu6-isys
> >>   	ivtv
> >> +	mali-c55
> >>   	mgb4
> >>   	omap3isp
> >>   	omap4_camera

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 06/16] media: Documentation: Add Mali-C55 ISP Documentation

[Dan Scally]

On 29/05/2024 21:51, Laurent Pinchart wrote:
> On Wed, May 29, 2024 at 09:35:08PM +0100, Daniel Scally wrote:
>> On 29/05/2024 21:22, Laurent Pinchart wrote:
>>> On Wed, May 29, 2024 at 04:28:48PM +0100, Daniel Scally wrote:
>>>> Add a documentation page for the mali-c55 driver, which gives a brief
>>>> overview of the hardware and explains how to use the driver's capture
>>>> devices and the crop/scaler functions.
>>>>
>>>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
>>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>>>> ---
>>>> Changes in v5:
>>>>
>>>> 	- None
>>>>
>>>> Changes in v4:
>>>> 	- None
>>>>
>>>> Changes in v3:
>>>> 	- Documented the synchronised buffer sequence numbers (Sakari)
>>>> 	- Clarified that the downscale pipe cannot output raw data, the ISP'S
>>>> 	  resolution limits and choice of media bus format code (Kieran)
>>>>
>>>> Changes in v2:
>>>>
>>>> 	- none
>>>>
>>>>    .../admin-guide/media/mali-c55-graph.dot      |  19 +
>>>>    Documentation/admin-guide/media/mali-c55.rst  | 333 ++++++++++++++++++
>>>>    .../admin-guide/media/v4l-drivers.rst         |   1 +
>>>>    3 files changed, 353 insertions(+)
>>>>    create mode 100644 Documentation/admin-guide/media/mali-c55-graph.dot
>>>>    create mode 100644 Documentation/admin-guide/media/mali-c55.rst
>>>>
>>>> diff --git a/Documentation/admin-guide/media/mali-c55-graph.dot b/Documentation/admin-guide/media/mali-c55-graph.dot
>>>> new file mode 100644
>>>> index 000000000000..0775ba42bf4c
>>>> --- /dev/null
>>>> +++ b/Documentation/admin-guide/media/mali-c55-graph.dot
>>>> @@ -0,0 +1,19 @@
>>>> +digraph board {
>>>> +        rankdir=TB
>>>> +        n00000001 [label="{{} | mali-c55 tpg\n/dev/v4l-subdev0 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
>>>> +        n00000001:port0 -> n00000003:port0 [style=dashed]
>>>> +        n00000003 [label="{{<port0> 0} | mali-c55 isp\n/dev/v4l-subdev1 | {<port1> 1 | <port2> 2}}", shape=Mrecord, style=filled, fillcolor=green]
>>>> +        n00000003:port1 -> n00000007:port0 [style=bold]
>>>> +        n00000003:port2 -> n00000007:port2 [style=bold]
>>>> +        n00000003:port1 -> n0000000b:port0 [style=bold]
>>>> +        n00000007 [label="{{<port0> 0 | <port2> 2} | mali-c55 resizer fr\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
>>>> +        n00000007:port1 -> n0000000e [style=bold]
>>>> +        n0000000b [label="{{<port0> 0} | mali-c55 resizer ds\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
>>>> +        n0000000b:port1 -> n00000012 [style=bold]
>>>> +        n0000000e [label="mali-c55 fr\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
>>>> +        n00000012 [label="mali-c55 ds\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
>>>> +        n00000022 [label="{{<port0> 0} | csi2-rx\n/dev/v4l-subdev4 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
>>>> +        n00000022:port1 -> n00000003:port0
>>>> +        n00000027 [label="{{} | imx415 1-001a\n/dev/v4l-subdev5 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
>>>> +        n00000027:port0 -> n00000022:port0 [style=bold]
>>>> +}
>>>> \ No newline at end of file
>>>> diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst
>>>> new file mode 100644
>>>> index 000000000000..cf4176cb1e44
>>>> --- /dev/null
>>>> +++ b/Documentation/admin-guide/media/mali-c55.rst
>>>> @@ -0,0 +1,333 @@
>>>> +.. SPDX-License-Identifier: GPL-2.0
>>>> +
>>>> +==========================================
>>>> +ARM Mali-C55 Image Signal Processor driver
>>>> +==========================================
>>>> +
>>>> +Introduction
>>>> +============
>>>> +
>>>> +This file documents the driver for ARM's Mali-C55 Image Signal Processor. The
>>>> +driver is located under drivers/media/platform/arm/mali-c55.
>>>> +
>>>> +The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from
>>>> +sensors through either a parallel interface or a memory bus before processing it
>>>> +and outputting it through an internal DMA engine. Two output pipelines are
>>>> +possible (though one may not be fitted, depending on the implementation). These
>>>> +are referred to as "Full resolution" and "Downscale", but the naming is historic
>>>> +and both pipes are capable of cropping/scaling operations. The full resolution
>>>> +pipe is also capable of outputting RAW data, bypassing much of the ISP's
>>>> +processing. The downscale pipe cannot output RAW data. An integrated test
>>>> +pattern generator can be used to drive the ISP and produce image data in the
>>>> +absence of a connected camera sensor. The driver module is named mali_c55, and
>>>> +is enabled through the CONFIG_VIDEO_MALI_C55 config option.
>>>> +
>>>> +The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and
>>>> +expects camera sensors connected to the ISP to have V4L2 subdevice interfaces.
>>>> +
>>>> +Mali-C55 ISP hardware
>>>> +=====================
>>>> +
>>>> +A high level functional view of the Mali-C55 ISP is presented below. The ISP
>>>> +takes input from either a live source or through a DMA engine for memory input,
>>>> +depending on the SoC integration.::
>>>> +
>>>> +  +---------+    +----------+                                     +--------+
>>>> +  | Sensor  |--->| CSI-2 Rx |                "Full Resolution"    |  DMA   |
>>>> +  +---------+    +----------+   |\                 Output    +--->| Writer |
>>>> +                       |        | \                          |    +--------+
>>>> +                       |        |  \    +----------+  +------+---> Streaming I/O
>>>> +  +------------+       +------->|   |   |          |  |
>>>> +  |            |                |   |-->| Mali-C55 |--+
>>>> +  | DMA Reader |--------------->|   |   |    ISP   |  |
>>>> +  |            |                |  /    |          |  |      +---> Streaming I/O
>>>> +  +------------+                | /     +----------+  |      |
>>>> +                                |/                    +------+
>>>> +				                             |    +--------+
>>>> +                                                             +--->|  DMA   |
>>>> +                                               "Downscaled"       | Writer |
>>>> +					          Output          +--------+
>>> You have a mix of tabs and spaces here.
>>>
>>>> +
>>>> +Media Controller Topology
>>>> +=========================
>>>> +
>>>> +An example of the ISP's topology (as implemented in a system with an IMX415
>>>> +camera sensor and generic CSI-2 receiver) is below:
>>>> +
>>>> +
>>>> +.. kernel-figure:: mali-c55-graph.dot
>>>> +    :alt:   mali-c55-graph.dot
>>>> +    :align: center
>>>> +
>>>> +The driver has 4 V4L2 subdevices:
>>>> +
>>>> +- `mali_c55 isp`: Responsible for configuring input crop and color space
>>>> +                  conversion
>>>> +- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor.
>>>> +- `mali_c55 resizer fr`: The Full-Resolution pipe resizer
>>>> +- `mali_c55 resizer ds`: The Downscale pipe resizer
>>>> +
>>>> +The driver has 2 V4L2 video devices:
>>>> +
>>>> +- `mali-c55 fr`: The full-resolution pipe's capture device
>>>> +- `mali-c55 ds`: The downscale pipe's capture device
>>>> +
>>>> +Frame sequences are synchronised across to two capture devices, meaning if one
>>>> +pipe is started later than the other the sequence numbers returned in its
>>>> +buffers will match those of the other pipe rather than starting from zero.
>>>> +
>>>> +Frame sequences are synchronised across to two capture devices, meaning if one
>>>> +pipe is started later than the other the sequence numbers returned in its
>>>> +buffers will match those of the other pipe rather than starting from zero.
>>> I think you can explain this once only.
>> Oops...
>>
>>>> +
>>>> +Idiosyncrasies
>>>> +--------------
>>>> +
>>>> +**mali-c55 isp**
>>>> +The `mali-c55 isp` subdevice has a single sink pad to which all sources of data
>>>> +should be connected. The active source is selected by enabling the appropriate
>>>> +media link and disabling all others.
>>> Modelling this with links prevents switching between sources at runtime.
>>> It also makes it possible to misconfigure the pipeline by disconnecting
>>> the two sources. This would be caught at pipeline validation time, but
>>> it still adds complexity.
>>>
>>> I was considering using the subdev routing API instead, which would
>>> allow runtime reconfiguration, and prevent invalid configuration in the
>>> first place. The downside is that we would need a mux subdev in front of
>>> the ISP. In terms of additional complexity, that's clearly not great.
>>>
>>> Given that switching between the sensor and TPG at runtime is likely not
>>> an important use case, and that the harware may not even support it at
>>> all, we can probably keep the existing graph and driver implementation.
>> I suppose in the long term we need to think about how this should be
>> modeled in a multi-context system...when we have a media graph with 8
>> cameras connected (somehow) to the ISP's single sink pad how should we
>> select the right input device for a context? Whatever the answer is
>> there, probably we should do it here...if we can't do it at runtime
>> with links then probably it has to be a mux...or some novel thing.
> Good point. Although... In M2M operation, will we need the TPG at all ?
> Userspace could easily supply test patterns in the input buffers.
True, we wouldn't need it.
> I don't think we can time-multiplex the ISP in a mode where some of the
> inputs would come from memory, and some from a live source (be it the
> TPG or a sensor).

Yeah I'm not sure how we could get the timing of that right.

> Sticking to the technical discussion, I envision that the contexts will
> be handled by opening the media device multiple times. Each file handle
> will be associated with one context. The link state would then be stored
> in the context. The links will still not be configurable at runtime
> within a context, but could differ between contexts.


OK I see. In that case keeping the link here gets my vote

>
>>>> The ISP has two source pads, reflecting the
>>>> +different paths through which it can internally route data. Tap points within
>>>> +the ISP allow users to divert data to avoid processing by some or all of the
>>>> +hardware's processing steps. The diagram below is intended only to highlight how
>>>> +the bypassing works and is not a true reflection of those processing steps; for
>>>> +a high-level functional block diagram see ARM's developer page for the
>>>> +ISP [3]_::
>>>> +
>>>> +  +--------------------------------------------------------------+
>>>> +  |                Possible Internal ISP Data Routes             |
>>>> +  |          +------------+  +----------+  +------------+        |
>>>> +  +---+      |            |  |          |  |  Colour    |    +---+
>>>> +  | 0 |--+-->| Processing |->| Demosaic |->|   Space    |--->| 1 |
>>>> +  +---+  |   |            |  |          |  | Conversion |    +---+
>>>> +  |      |   +------------+  +----------+  +------------+        |
>>>> +  |      |                                                   +---+
>>>> +  |      +---------------------------------------------------| 2 |
>>>> +  |                                                          +---+
>>>> +  |                                                              |
>>>> +  +--------------------------------------------------------------+
>>>> +
>>>> +
>>>> +.. flat-table::
>>>> +    :header-rows: 1
>>>> +
>>>> +    * - Pad
>>>> +      - Direction
>>>> +      - Purpose
>>>> +
>>>> +    * - 0
>>>> +      - sink
>>>> +      - Data input, connected to the TPG and camera sensors
>>>> +
>>>> +    * - 1
>>>> +      - source
>>>> +      - RGB/YUV data, connected to the FR and DS V4L2 subdevices
>>>> +
>>>> +    * - 2
>>>> +      - source
>>>> +      - RAW bayer data, connected to the FR V4L2 subdevices
>>>> +
>>>> +The ISP is limited to both input and output resolutions between 640x480 and
>>>> +8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt()
>>>> +operations.
>>>> +
>>>> +**mali-c55 resizer fr**
>>>> +The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different
>>>> +insertion points in the hardware (either RAW or demosaiced data):
>>>> +
>>>> +.. flat-table::
>>>> +    :header-rows: 1
>>>> +
>>>> +    * - Pad
>>>> +      - Direction
>>>> +      - Purpose
>>>> +
>>>> +    * - 0
>>>> +      - sink
>>>> +      - Data input connected to the ISP's demosaiced stream.
>>>> +
>>>> +    * - 1
>>>> +      - source
>>>> +      - Data output connected to the capture video device
>>>> +
>>>> +    * - 2
>>>> +      - sink
>>>> +      - Data input connected to the ISP's raw data stream
>>>> +
>>>> +The data source in use is selected through the routing API; two routes each of a
>>>> +single stream are available:
>>>> +
>>>> +.. flat-table::
>>>> +    :header-rows: 1
>>>> +
>>>> +    * - Sink Pad
>>>> +      - Source Pad
>>>> +      - Purpose
>>>> +
>>>> +    * - 0
>>>> +      - 1
>>>> +      - Demosaiced data route
>>>> +
>>>> +    * - 2
>>>> +      - 1
>>>> +      - Raw data route
>>>> +
>>>> +
>>>> +If the demosaiced route is active then the FR pipe is only capable of output
>>>> +in RGB/YUV formats. If the raw route is active then the output reflects the
>>>> +input (which may be either Bayer or RGB/YUV data).
>>>> +
>>>> +Using the driver to capture video
>>>> +=================================
>>>> +
>>>> +Using the media controller APIs we can configure the input source and ISP to
>>>> +capture images in a variety of formats. In the examples below, configuring the
>>>> +media graph is done with the v4l-utils [1]_ package's media-ctl utility.
>>>> +Capturing the images is done with yavta [2]_.
>>>> +
>>>> +Configuring the input source
>>>> +----------------------------
>>>> +
>>>> +The first step is to set the input source that we wish by enabling the correct
>>>> +media link. Using the example topology above, we can select the TPG as follows:
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> +    media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]"
>>>> +    media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]"
>>>> +
>>>> +Capturing bayer data from the source and processing to RGB/YUV
>>>> +--------------------------------------------------------------
>>> Missing blank line.
>>>
>>>> +To capture 1920x1080 bayer data from the source and push it through the ISP's
>>>> +full processing pipeline, we configure the data formats appropriately on the
>>>> +source, ISP and resizer subdevices and set the FR resizer's routing to select
>>>> +processed data. The media bus format on the resizer's source pad will be either
>>>> +RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or
>>>> +YUV. The ISP's debayering block outputs RGB data natively, setting the source
>>>> +pad format to YUV10_1X30 enables the colour space conversion block.
>>>> +
>>>> +In this example we target RGB565 output, so select RGB121212_1X36 as the resizer
>>>> +source pad's format:
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> +    # Set formats on the TPG and ISP
>>>> +    media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB16_1X16/1920x1080]"
>>>> +    media-ctl -V "'mali-c55 isp':0[fmt:SRGGB16_1X16/1920x1080]"
>>>> +    media-ctl -V "'mali-c55 isp':1[fmt:SRGGB16_1X16/1920x1080]"
>>>> +
>>>> +    # Set routing on the FR resizer
>>>> +    media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]"
>>>> +
>>>> +    # Set format on the resizer, must be done AFTER the routing.
>>>> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]"
>>>> +
>>>> +The downscale output can also be used to stream data at the same time. In this
>>>> +case since only processed data can be captured through the downscale output no
>>>> +routing need be set:
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> +    # Set format on the resizer
>>>> +    media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]"
>>>> +
>>>> +Following which images can be captured from both the FR and DS output's video
>>>> +devices (simultaneously, if desired):
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
>>>> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video1
>>>> +
>>>> +Cropping the image
>>>> +~~~~~~~~~~~~~~~~~~
>>>> +
>>>> +Both the full resolution and downscale pipes can crop to a minimum resolution of
>>>> +640x480. To crop the image simply configure the resizer's sink pad's crop and
>>>> +compose rectangles and set the format on the video device:
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> +    media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]"
>>>> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
>>>> +    yavta -f RGB565 -s 640x480 -c10 /dev/video0
>>>> +
>>>> +Downscaling the image
>>>> +~~~~~~~~~~~~~~~~~~~~~
>>>> +
>>>> +Both the full resolution and downscale pipes can downscale the image by up to 8x
>>>> +provided the minimum 640x480 resolution is adhered to. For the best image result
>>> Maybe "minimum 640x480 output resolution".
>>>
>>>> +the scaling ratio for each dimension should be the same. To configure scaling we
>>> s/dimension/direction/
>>>
>>>> +use the compose rectangle on the resizer's sink pad:
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> +    media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]"
>>>> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
>>>> +    yavta -f RGB565 -s 640x480 -c10 /dev/video0
>>>> +
>>>> +Capturing images in YUV formats
>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> +
>>>> +If we need to output YUV data rather than RGB the color space conversion block
>>>> +needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the
>>>> +resizer's source pad. We can then configure a capture format like NV12 (here in
>>>> +its multi-planar variant)
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]"
>>>> +    yavta -f NV12M -s 1920x1080 -c10 /dev/video0
>>>> +
>>>> +Capturing RGB data from the source and processing it with the resizers
>>>> +----------------------------------------------------------------------
>>>> +
>>>> +The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this
>>>> +case although none of the image quality blocks would be used it can still
>>>> +crop/scale the data in the usual way.
>>>> +
>>>> +To achieve this, the ISP's sink pad's format is set to
>>>> +MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to
>>>> +work with the ISP. Converting the camera sensor's output to that format is the
>>>> +responsibility of external hardware.
>>>> +
>>>> +In this example we ask the test pattern generator to give us RGB data instead of
>>>> +bayer.
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> +    media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]"
>>>> +    media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]"
>>>> +
>>>> +Cropping or scaling the data can be done in exactly the same way as outlined
>>>> +earlier.
>>> Do we use the ISP's output on pad 1 or pad 2 in this case ? The text
>>> seems to imply that the ISP is bypassed, but the example doesn't mention
>>> any routing change. You may want to clarify this.
>> Pad 1 for bypassed RGB data. I'll make it more clear.
> Explaining that this doesn't follow the bypass path, but goes through
> the processing path and disables the processing blocks that are not
> relevant, would help.


Well; kinda. It takes a _different_ bypass path, but in this case the decision as to whether or not 
that path should be taken is just based on the ISP sink pad's format.

>
>>>> +
>>>> +Capturing raw data from the source and outputting it unmodified
>>>> +-----------------------------------------------------------------
>>>> +
>>>> +The ISP can additionally capture raw data from the source and output it on the
>>>> +full resolution pipe only, completely unmodified. In this case the downscale
>>>> +pipe can still process the data normally and be used at the same time.
>>>> +
>>>> +To configure raw bypass the FR resizer's subdevice's routing table needs to be
>>>> +configured, followed by formats in the appropriate places:
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> +    # We need to configure the routing table for the resizer to use the bypass
>>>> +    # path along with set formats on the resizer's bypass sink pad. Doing this
>>>> +    # necessitates a single media-ctl command, as multiple calls to the program
>>>> +    # reset the routing table.
>>> Really ?
>> Yeah
>>
>>> Does -V reset the routing table ? That surprises me.
>> It's not -V, it's the fact of opening a subdev which calls
>> .init_state(), which resets the routing table...
> Only for the TRY state, not the ACTIVE state. If the TRY state leaks to
> the ACTIVE state, you may have a driver bugs :-)

Hmm, of course that's right...maybe I'm misremembering what the problem was, or maybe there's no 
problem at all. I will check and see.
>
>> in a single process the fds are held open throughout so all is well, but if you run the
>> program twice they're opened each time and the routing is reset.
>>
>>>> +    media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"\
>>>> +    -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080],"\
>>>> +       "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080],"\
>>>> +       "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]"
>>>> +
>>>> +    # Set format on the video device and stream
>>>> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
>>>> +
>>>> +References
>>>> +==========
>>>> +.. [1] https://git.linuxtv.org/v4l-utils.git/
>>>> +.. [2] https://git.ideasonboard.org/yavta.git
>>>> +.. [3] https://developer.arm.com/Processors/Mali-C55
>>>> diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst
>>>> index 4120eded9a13..1d9485860d93 100644
>>>> --- a/Documentation/admin-guide/media/v4l-drivers.rst
>>>> +++ b/Documentation/admin-guide/media/v4l-drivers.rst
>>>> @@ -18,6 +18,7 @@ Video4Linux (V4L) driver-specific documentation
>>>>    	ipu3
>>>>    	ipu6-isys
>>>>    	ivtv
>>>> +	mali-c55
>>>>    	mgb4
>>>>    	omap3isp
>>>>    	omap4_camera

Re: [PATCH v5 06/16] media: Documentation: Add Mali-C55 ISP Documentation

[Dan Scally]

Hi Laurent

On 29/05/2024 22:11, Dan Scally wrote:
>
> On 29/05/2024 21:51, Laurent Pinchart wrote:
>> On Wed, May 29, 2024 at 09:35:08PM +0100, Daniel Scally wrote:
>>> On 29/05/2024 21:22, Laurent Pinchart wrote:
>>>> On Wed, May 29, 2024 at 04:28:48PM +0100, Daniel Scally wrote:
>>>>> Add a documentation page for the mali-c55 driver, which gives a brief
>>>>> overview of the hardware and explains how to use the driver's capture
>>>>> devices and the crop/scaler functions.
>>>>>
>>>>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
>>>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>>>>> ---
>>>>> Changes in v5:
>>>>>
>>>>>     - None
>>>>>
>>>>> Changes in v4:
>>>>>     - None
>>>>>
>>>>> Changes in v3:
>>>>>     - Documented the synchronised buffer sequence numbers (Sakari)
>>>>>     - Clarified that the downscale pipe cannot output raw data, the ISP'S
>>>>>       resolution limits and choice of media bus format code (Kieran)
>>>>>
>>>>> Changes in v2:
>>>>>
>>>>>     - none
>>>>>
>>>>>    .../admin-guide/media/mali-c55-graph.dot      |  19 +
>>>>>    Documentation/admin-guide/media/mali-c55.rst  | 333 ++++++++++++++++++
>>>>>    .../admin-guide/media/v4l-drivers.rst         |   1 +
>>>>>    3 files changed, 353 insertions(+)
>>>>>    create mode 100644 Documentation/admin-guide/media/mali-c55-graph.dot
>>>>>    create mode 100644 Documentation/admin-guide/media/mali-c55.rst
>>>>>
>>>>> diff --git a/Documentation/admin-guide/media/mali-c55-graph.dot 
>>>>> b/Documentation/admin-guide/media/mali-c55-graph.dot
>>>>> new file mode 100644
>>>>> index 000000000000..0775ba42bf4c
>>>>> --- /dev/null
>>>>> +++ b/Documentation/admin-guide/media/mali-c55-graph.dot
>>>>> @@ -0,0 +1,19 @@
>>>>> +digraph board {
>>>>> +        rankdir=TB
>>>>> +        n00000001 [label="{{} | mali-c55 tpg\n/dev/v4l-subdev0 | {<port0> 0}}", 
>>>>> shape=Mrecord, style=filled, fillcolor=green]
>>>>> +        n00000001:port0 -> n00000003:port0 [style=dashed]
>>>>> +        n00000003 [label="{{<port0> 0} | mali-c55 isp\n/dev/v4l-subdev1 | {<port1> 1 | 
>>>>> <port2> 2}}", shape=Mrecord, style=filled, fillcolor=green]
>>>>> +        n00000003:port1 -> n00000007:port0 [style=bold]
>>>>> +        n00000003:port2 -> n00000007:port2 [style=bold]
>>>>> +        n00000003:port1 -> n0000000b:port0 [style=bold]
>>>>> +        n00000007 [label="{{<port0> 0 | <port2> 2} | mali-c55 resizer fr\n/dev/v4l-subdev2 | 
>>>>> {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
>>>>> +        n00000007:port1 -> n0000000e [style=bold]
>>>>> +        n0000000b [label="{{<port0> 0} | mali-c55 resizer ds\n/dev/v4l-subdev3 | {<port1> 
>>>>> 1}}", shape=Mrecord, style=filled, fillcolor=green]
>>>>> +        n0000000b:port1 -> n00000012 [style=bold]
>>>>> +        n0000000e [label="mali-c55 fr\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
>>>>> +        n00000012 [label="mali-c55 ds\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
>>>>> +        n00000022 [label="{{<port0> 0} | csi2-rx\n/dev/v4l-subdev4 | {<port1> 1}}", 
>>>>> shape=Mrecord, style=filled, fillcolor=green]
>>>>> +        n00000022:port1 -> n00000003:port0
>>>>> +        n00000027 [label="{{} | imx415 1-001a\n/dev/v4l-subdev5 | {<port0> 0}}", 
>>>>> shape=Mrecord, style=filled, fillcolor=green]
>>>>> +        n00000027:port0 -> n00000022:port0 [style=bold]
>>>>> +}
>>>>> \ No newline at end of file
>>>>> diff --git a/Documentation/admin-guide/media/mali-c55.rst 
>>>>> b/Documentation/admin-guide/media/mali-c55.rst
>>>>> new file mode 100644
>>>>> index 000000000000..cf4176cb1e44
>>>>> --- /dev/null
>>>>> +++ b/Documentation/admin-guide/media/mali-c55.rst
>>>>> @@ -0,0 +1,333 @@
>>>>> +.. SPDX-License-Identifier: GPL-2.0
>>>>> +
>>>>> +==========================================
>>>>> +ARM Mali-C55 Image Signal Processor driver
>>>>> +==========================================
>>>>> +
>>>>> +Introduction
>>>>> +============
>>>>> +
>>>>> +This file documents the driver for ARM's Mali-C55 Image Signal Processor. The
>>>>> +driver is located under drivers/media/platform/arm/mali-c55.
>>>>> +
>>>>> +The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from
>>>>> +sensors through either a parallel interface or a memory bus before processing it
>>>>> +and outputting it through an internal DMA engine. Two output pipelines are
>>>>> +possible (though one may not be fitted, depending on the implementation). These
>>>>> +are referred to as "Full resolution" and "Downscale", but the naming is historic
>>>>> +and both pipes are capable of cropping/scaling operations. The full resolution
>>>>> +pipe is also capable of outputting RAW data, bypassing much of the ISP's
>>>>> +processing. The downscale pipe cannot output RAW data. An integrated test
>>>>> +pattern generator can be used to drive the ISP and produce image data in the
>>>>> +absence of a connected camera sensor. The driver module is named mali_c55, and
>>>>> +is enabled through the CONFIG_VIDEO_MALI_C55 config option.
>>>>> +
>>>>> +The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and
>>>>> +expects camera sensors connected to the ISP to have V4L2 subdevice interfaces.
>>>>> +
>>>>> +Mali-C55 ISP hardware
>>>>> +=====================
>>>>> +
>>>>> +A high level functional view of the Mali-C55 ISP is presented below. The ISP
>>>>> +takes input from either a live source or through a DMA engine for memory input,
>>>>> +depending on the SoC integration.::
>>>>> +
>>>>> +  +---------+ +----------+ +--------+
>>>>> +  | Sensor  |--->| CSI-2 Rx |                "Full Resolution"    |  DMA   |
>>>>> +  +---------+    +----------+   |\ Output    +--->| Writer |
>>>>> +                       |        | \                          |    +--------+
>>>>> +                       |        |  \    +----------+ +------+---> Streaming I/O
>>>>> +  +------------+       +------->|   |   |          | |
>>>>> +  |            |                |   |-->| Mali-C55 |--+
>>>>> +  | DMA Reader |--------------->|   |   |    ISP   | |
>>>>> +  |            |                |  /    |          | |      +---> Streaming I/O
>>>>> +  +------------+                | /     +----------+ |      |
>>>>> +                                |/ +------+
>>>>> +                                             | +--------+
>>>>> +                                                             +--->| DMA   |
>>>>> + "Downscaled"       | Writer |
>>>>> +                              Output          +--------+
>>>> You have a mix of tabs and spaces here.
>>>>
>>>>> +
>>>>> +Media Controller Topology
>>>>> +=========================
>>>>> +
>>>>> +An example of the ISP's topology (as implemented in a system with an IMX415
>>>>> +camera sensor and generic CSI-2 receiver) is below:
>>>>> +
>>>>> +
>>>>> +.. kernel-figure:: mali-c55-graph.dot
>>>>> +    :alt:   mali-c55-graph.dot
>>>>> +    :align: center
>>>>> +
>>>>> +The driver has 4 V4L2 subdevices:
>>>>> +
>>>>> +- `mali_c55 isp`: Responsible for configuring input crop and color space
>>>>> +                  conversion
>>>>> +- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor.
>>>>> +- `mali_c55 resizer fr`: The Full-Resolution pipe resizer
>>>>> +- `mali_c55 resizer ds`: The Downscale pipe resizer
>>>>> +
>>>>> +The driver has 2 V4L2 video devices:
>>>>> +
>>>>> +- `mali-c55 fr`: The full-resolution pipe's capture device
>>>>> +- `mali-c55 ds`: The downscale pipe's capture device
>>>>> +
>>>>> +Frame sequences are synchronised across to two capture devices, meaning if one
>>>>> +pipe is started later than the other the sequence numbers returned in its
>>>>> +buffers will match those of the other pipe rather than starting from zero.
>>>>> +
>>>>> +Frame sequences are synchronised across to two capture devices, meaning if one
>>>>> +pipe is started later than the other the sequence numbers returned in its
>>>>> +buffers will match those of the other pipe rather than starting from zero.
>>>> I think you can explain this once only.
>>> Oops...
>>>
>>>>> +
>>>>> +Idiosyncrasies
>>>>> +--------------
>>>>> +
>>>>> +**mali-c55 isp**
>>>>> +The `mali-c55 isp` subdevice has a single sink pad to which all sources of data
>>>>> +should be connected. The active source is selected by enabling the appropriate
>>>>> +media link and disabling all others.
>>>> Modelling this with links prevents switching between sources at runtime.
>>>> It also makes it possible to misconfigure the pipeline by disconnecting
>>>> the two sources. This would be caught at pipeline validation time, but
>>>> it still adds complexity.
>>>>
>>>> I was considering using the subdev routing API instead, which would
>>>> allow runtime reconfiguration, and prevent invalid configuration in the
>>>> first place. The downside is that we would need a mux subdev in front of
>>>> the ISP. In terms of additional complexity, that's clearly not great.
>>>>
>>>> Given that switching between the sensor and TPG at runtime is likely not
>>>> an important use case, and that the harware may not even support it at
>>>> all, we can probably keep the existing graph and driver implementation.
>>> I suppose in the long term we need to think about how this should be
>>> modeled in a multi-context system...when we have a media graph with 8
>>> cameras connected (somehow) to the ISP's single sink pad how should we
>>> select the right input device for a context? Whatever the answer is
>>> there, probably we should do it here...if we can't do it at runtime
>>> with links then probably it has to be a mux...or some novel thing.
>> Good point. Although... In M2M operation, will we need the TPG at all ?
>> Userspace could easily supply test patterns in the input buffers.
> True, we wouldn't need it.
>> I don't think we can time-multiplex the ISP in a mode where some of the
>> inputs would come from memory, and some from a live source (be it the
>> TPG or a sensor).
>
> Yeah I'm not sure how we could get the timing of that right.
>
>> Sticking to the technical discussion, I envision that the contexts will
>> be handled by opening the media device multiple times. Each file handle
>> will be associated with one context. The link state would then be stored
>> in the context. The links will still not be configurable at runtime
>> within a context, but could differ between contexts.
>
>
> OK I see. In that case keeping the link here gets my vote
>
>>
>>>>> The ISP has two source pads, reflecting the
>>>>> +different paths through which it can internally route data. Tap points within
>>>>> +the ISP allow users to divert data to avoid processing by some or all of the
>>>>> +hardware's processing steps. The diagram below is intended only to highlight how
>>>>> +the bypassing works and is not a true reflection of those processing steps; for
>>>>> +a high-level functional block diagram see ARM's developer page for the
>>>>> +ISP [3]_::
>>>>> +
>>>>> + +--------------------------------------------------------------+
>>>>> +  |                Possible Internal ISP Data Routes             |
>>>>> +  |          +------------+  +----------+ +------------+        |
>>>>> +  +---+      |            |  |          |  |  Colour |    +---+
>>>>> +  | 0 |--+-->| Processing |->| Demosaic |->| Space    |--->| 1 |
>>>>> +  +---+  |   |            |  |          |  | Conversion |    +---+
>>>>> +  |      |   +------------+  +----------+ +------------+        |
>>>>> +  | |                                                   +---+
>>>>> +  | +---------------------------------------------------| 2 |
>>>>> + | +---+
>>>>> + | |
>>>>> + +--------------------------------------------------------------+
>>>>> +
>>>>> +
>>>>> +.. flat-table::
>>>>> +    :header-rows: 1
>>>>> +
>>>>> +    * - Pad
>>>>> +      - Direction
>>>>> +      - Purpose
>>>>> +
>>>>> +    * - 0
>>>>> +      - sink
>>>>> +      - Data input, connected to the TPG and camera sensors
>>>>> +
>>>>> +    * - 1
>>>>> +      - source
>>>>> +      - RGB/YUV data, connected to the FR and DS V4L2 subdevices
>>>>> +
>>>>> +    * - 2
>>>>> +      - source
>>>>> +      - RAW bayer data, connected to the FR V4L2 subdevices
>>>>> +
>>>>> +The ISP is limited to both input and output resolutions between 640x480 and
>>>>> +8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt()
>>>>> +operations.
>>>>> +
>>>>> +**mali-c55 resizer fr**
>>>>> +The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different
>>>>> +insertion points in the hardware (either RAW or demosaiced data):
>>>>> +
>>>>> +.. flat-table::
>>>>> +    :header-rows: 1
>>>>> +
>>>>> +    * - Pad
>>>>> +      - Direction
>>>>> +      - Purpose
>>>>> +
>>>>> +    * - 0
>>>>> +      - sink
>>>>> +      - Data input connected to the ISP's demosaiced stream.
>>>>> +
>>>>> +    * - 1
>>>>> +      - source
>>>>> +      - Data output connected to the capture video device
>>>>> +
>>>>> +    * - 2
>>>>> +      - sink
>>>>> +      - Data input connected to the ISP's raw data stream
>>>>> +
>>>>> +The data source in use is selected through the routing API; two routes each of a
>>>>> +single stream are available:
>>>>> +
>>>>> +.. flat-table::
>>>>> +    :header-rows: 1
>>>>> +
>>>>> +    * - Sink Pad
>>>>> +      - Source Pad
>>>>> +      - Purpose
>>>>> +
>>>>> +    * - 0
>>>>> +      - 1
>>>>> +      - Demosaiced data route
>>>>> +
>>>>> +    * - 2
>>>>> +      - 1
>>>>> +      - Raw data route
>>>>> +
>>>>> +
>>>>> +If the demosaiced route is active then the FR pipe is only capable of output
>>>>> +in RGB/YUV formats. If the raw route is active then the output reflects the
>>>>> +input (which may be either Bayer or RGB/YUV data).
>>>>> +
>>>>> +Using the driver to capture video
>>>>> +=================================
>>>>> +
>>>>> +Using the media controller APIs we can configure the input source and ISP to
>>>>> +capture images in a variety of formats. In the examples below, configuring the
>>>>> +media graph is done with the v4l-utils [1]_ package's media-ctl utility.
>>>>> +Capturing the images is done with yavta [2]_.
>>>>> +
>>>>> +Configuring the input source
>>>>> +----------------------------
>>>>> +
>>>>> +The first step is to set the input source that we wish by enabling the correct
>>>>> +media link. Using the example topology above, we can select the TPG as follows:
>>>>> +
>>>>> +.. code-block:: none
>>>>> +
>>>>> +    media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]"
>>>>> +    media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]"
>>>>> +
>>>>> +Capturing bayer data from the source and processing to RGB/YUV
>>>>> +--------------------------------------------------------------
>>>> Missing blank line.
>>>>
>>>>> +To capture 1920x1080 bayer data from the source and push it through the ISP's
>>>>> +full processing pipeline, we configure the data formats appropriately on the
>>>>> +source, ISP and resizer subdevices and set the FR resizer's routing to select
>>>>> +processed data. The media bus format on the resizer's source pad will be either
>>>>> +RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or
>>>>> +YUV. The ISP's debayering block outputs RGB data natively, setting the source
>>>>> +pad format to YUV10_1X30 enables the colour space conversion block.
>>>>> +
>>>>> +In this example we target RGB565 output, so select RGB121212_1X36 as the resizer
>>>>> +source pad's format:
>>>>> +
>>>>> +.. code-block:: none
>>>>> +
>>>>> +    # Set formats on the TPG and ISP
>>>>> +    media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB16_1X16/1920x1080]"
>>>>> +    media-ctl -V "'mali-c55 isp':0[fmt:SRGGB16_1X16/1920x1080]"
>>>>> +    media-ctl -V "'mali-c55 isp':1[fmt:SRGGB16_1X16/1920x1080]"
>>>>> +
>>>>> +    # Set routing on the FR resizer
>>>>> +    media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]"
>>>>> +
>>>>> +    # Set format on the resizer, must be done AFTER the routing.
>>>>> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]"
>>>>> +
>>>>> +The downscale output can also be used to stream data at the same time. In this
>>>>> +case since only processed data can be captured through the downscale output no
>>>>> +routing need be set:
>>>>> +
>>>>> +.. code-block:: none
>>>>> +
>>>>> +    # Set format on the resizer
>>>>> +    media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]"
>>>>> +
>>>>> +Following which images can be captured from both the FR and DS output's video
>>>>> +devices (simultaneously, if desired):
>>>>> +
>>>>> +.. code-block:: none
>>>>> +
>>>>> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
>>>>> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video1
>>>>> +
>>>>> +Cropping the image
>>>>> +~~~~~~~~~~~~~~~~~~
>>>>> +
>>>>> +Both the full resolution and downscale pipes can crop to a minimum resolution of
>>>>> +640x480. To crop the image simply configure the resizer's sink pad's crop and
>>>>> +compose rectangles and set the format on the video device:
>>>>> +
>>>>> +.. code-block:: none
>>>>> +
>>>>> +    media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 
>>>>> compose:(0,0)/640x480]"
>>>>> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
>>>>> +    yavta -f RGB565 -s 640x480 -c10 /dev/video0
>>>>> +
>>>>> +Downscaling the image
>>>>> +~~~~~~~~~~~~~~~~~~~~~
>>>>> +
>>>>> +Both the full resolution and downscale pipes can downscale the image by up to 8x
>>>>> +provided the minimum 640x480 resolution is adhered to. For the best image result
>>>> Maybe "minimum 640x480 output resolution".
>>>>
>>>>> +the scaling ratio for each dimension should be the same. To configure scaling we
>>>> s/dimension/direction/
>>>>
>>>>> +use the compose rectangle on the resizer's sink pad:
>>>>> +
>>>>> +.. code-block:: none
>>>>> +
>>>>> +    media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 
>>>>> compose:(0,0)/640x480]"
>>>>> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
>>>>> +    yavta -f RGB565 -s 640x480 -c10 /dev/video0
>>>>> +
>>>>> +Capturing images in YUV formats
>>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>> +
>>>>> +If we need to output YUV data rather than RGB the color space conversion block
>>>>> +needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the
>>>>> +resizer's source pad. We can then configure a capture format like NV12 (here in
>>>>> +its multi-planar variant)
>>>>> +
>>>>> +.. code-block:: none
>>>>> +
>>>>> +    media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]"
>>>>> +    yavta -f NV12M -s 1920x1080 -c10 /dev/video0
>>>>> +
>>>>> +Capturing RGB data from the source and processing it with the resizers
>>>>> +----------------------------------------------------------------------
>>>>> +
>>>>> +The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this
>>>>> +case although none of the image quality blocks would be used it can still
>>>>> +crop/scale the data in the usual way.
>>>>> +
>>>>> +To achieve this, the ISP's sink pad's format is set to
>>>>> +MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to
>>>>> +work with the ISP. Converting the camera sensor's output to that format is the
>>>>> +responsibility of external hardware.
>>>>> +
>>>>> +In this example we ask the test pattern generator to give us RGB data instead of
>>>>> +bayer.
>>>>> +
>>>>> +.. code-block:: none
>>>>> +
>>>>> +    media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]"
>>>>> +    media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]"
>>>>> +
>>>>> +Cropping or scaling the data can be done in exactly the same way as outlined
>>>>> +earlier.
>>>> Do we use the ISP's output on pad 1 or pad 2 in this case ? The text
>>>> seems to imply that the ISP is bypassed, but the example doesn't mention
>>>> any routing change. You may want to clarify this.
>>> Pad 1 for bypassed RGB data. I'll make it more clear.
>> Explaining that this doesn't follow the bypass path, but goes through
>> the processing path and disables the processing blocks that are not
>> relevant, would help.
>
>
> Well; kinda. It takes a _different_ bypass path, but in this case the decision as to whether or 
> not that path should be taken is just based on the ISP sink pad's format.
>
>>
>>>>> +
>>>>> +Capturing raw data from the source and outputting it unmodified
>>>>> +-----------------------------------------------------------------
>>>>> +
>>>>> +The ISP can additionally capture raw data from the source and output it on the
>>>>> +full resolution pipe only, completely unmodified. In this case the downscale
>>>>> +pipe can still process the data normally and be used at the same time.
>>>>> +
>>>>> +To configure raw bypass the FR resizer's subdevice's routing table needs to be
>>>>> +configured, followed by formats in the appropriate places:
>>>>> +
>>>>> +.. code-block:: none
>>>>> +
>>>>> +    # We need to configure the routing table for the resizer to use the bypass
>>>>> +    # path along with set formats on the resizer's bypass sink pad. Doing this
>>>>> +    # necessitates a single media-ctl command, as multiple calls to the program
>>>>> +    # reset the routing table.
>>>> Really ?
>>> Yeah
>>>
>>>> Does -V reset the routing table ? That surprises me.
>>> It's not -V, it's the fact of opening a subdev which calls
>>> .init_state(), which resets the routing table...
>> Only for the TRY state, not the ACTIVE state. If the TRY state leaks to
>> the ACTIVE state, you may have a driver bugs :-)
>
> Hmm, of course that's right...maybe I'm misremembering what the problem was, or maybe there's no 
> problem at all. I will check and see.


OK - unclear what I was on about, it all works fine as separate calls :D

>>
>>> in a single process the fds are held open throughout so all is well, but if you run the
>>> program twice they're opened each time and the routing is reset.
>>>
>>>>> +    media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"\
>>>>> +    -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080],"\
>>>>> +       "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080],"\
>>>>> +       "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]"
>>>>> +
>>>>> +    # Set format on the video device and stream
>>>>> +    yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
>>>>> +
>>>>> +References
>>>>> +==========
>>>>> +.. [1] https://git.linuxtv.org/v4l-utils.git/
>>>>> +.. [2] https://git.ideasonboard.org/yavta.git
>>>>> +.. [3] https://developer.arm.com/Processors/Mali-C55
>>>>> diff --git a/Documentation/admin-guide/media/v4l-drivers.rst 
>>>>> b/Documentation/admin-guide/media/v4l-drivers.rst
>>>>> index 4120eded9a13..1d9485860d93 100644
>>>>> --- a/Documentation/admin-guide/media/v4l-drivers.rst
>>>>> +++ b/Documentation/admin-guide/media/v4l-drivers.rst
>>>>> @@ -18,6 +18,7 @@ Video4Linux (V4L) driver-specific documentation
>>>>>        ipu3
>>>>>        ipu6-isys
>>>>>        ivtv
>>>>> +    mali-c55
>>>>>        mgb4
>>>>>        omap3isp
>>>>>        omap4_camera

Re: [PATCH v5 00/16] Add Arm Mali-C55 Image Signal Processor Driver

[Laurent Pinchart]

Hi Dan,

Thank you for the patches.

On Wed, May 29, 2024 at 04:28:42PM +0100, Daniel Scally wrote:
> Hello all
> 
> This patchset introduces a driver for Arm's Mali-C55 Image Signal Processor.
> The driver uses the V4L2 / media controller API and implements both of the ISP's
> capture pipelines allowing a range of output formats plus downscaling and
> cropping. The capture pipelines are named "Full resolution" and "Downscale" and
> so abbreviated FR and DS throughout the driver.
> 
> The driver exposes 4 V4L2 subdevices:
> 
> - mali-c55 isp: input data formatting
> - mali-c55 tpg: test pattern generator (modeled as a camera sensor entity)
> - mali-c55 resizer fr: downscale / crop and format setting for the FR pipe
> - mali-c55 resizer ds: downscale / crop and format setting for the DS pipe
> 
> Along with 4 V4L2 Video devices:
> 
> - mali-c55 fr: Capture device for the full resolution pipe
> - mali-c55 ds: Capture device for the downscale pipe
> - mali-c55 3a stats: Capture device for statistics to support 3A algorithms
> - mali-c55 3a params: Output device for parameter buffers to configure the ISP
> 
> Support is implemented in the parameters video device code for many of the ISP'S
> hardware blocks, but not yet all of them. The buffer format is (as far as I am
> aware anyway) a novel design that we intend to be extensible so that support for
> the C55's remaining hardware blocks can be added later.
> 
> Patches 1, 4, 5, 6 and 7 have already had 4 versions on the mailing list...I
> decided to post the additional work on the driver as extra patches rather than
> merge them all into the existing series as it's already a lot of code to review
> and I hoped that that might make it a little easier...if I'm wrong and that's
> not liked I can just squash them into a much smaller series.

Could you please include the v4l2-compliance report in v6 ? Make sure to
use the very latest version of v4l-utils, compiled from the master
branch.

> Daniel Scally (15):
>   media: uapi: Add MEDIA_BUS_FMT_RGB202020_1X60 format code
>   media: uapi: Add 20-bit bayer formats
>   media: v4l2-common: Add RAW16 format info
>   dt-bindings: media: Add bindings for ARM mali-c55
>   media: mali-c55: Add Mali-C55 ISP driver
>   media: Documentation: Add Mali-C55 ISP Documentation
>   MAINTAINERS: Add entry for mali-c55 driver
>   media: Add MALI_C55_3A_STATS meta format
>   media: uapi: Add 3a stats buffer for mali-c55
>   media: platform: Add mali-c55 3a stats devnode
>   media: platform: Fill stats buffer on ISP_START
>   Documentation: mali-c55: Add Statistics documentation
>   media: uapi: Add parameters structs to mali-c55-config.h
>   media: platform: Add mali-c55 parameters video node
>   Documentation: mali-c55: Document the mali-c55 parameter setting
> 
> Jacopo Mondi (1):
>   media: mali-c55: Add image formats for Mali-C55 parameters buffer
> 
>  .../admin-guide/media/mali-c55-graph.dot      |  19 +
>  Documentation/admin-guide/media/mali-c55.rst  | 406 ++++++++
>  .../admin-guide/media/v4l-drivers.rst         |   1 +
>  .../bindings/media/arm,mali-c55.yaml          |  66 ++
>  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
>  .../media/v4l/metafmt-arm-mali-c55.rst        |  87 ++
>  .../media/v4l/subdev-formats.rst              | 268 +++++
>  MAINTAINERS                                   |  10 +
>  drivers/media/platform/Kconfig                |   1 +
>  drivers/media/platform/Makefile               |   1 +
>  drivers/media/platform/arm/Kconfig            |   5 +
>  drivers/media/platform/arm/Makefile           |   2 +
>  drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
>  drivers/media/platform/arm/mali-c55/Makefile  |  11 +
>  .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
>  .../platform/arm/mali-c55/mali-c55-common.h   | 312 ++++++
>  .../platform/arm/mali-c55/mali-c55-core.c     | 825 +++++++++++++++
>  .../platform/arm/mali-c55/mali-c55-isp.c      | 626 ++++++++++++
>  .../platform/arm/mali-c55/mali-c55-params.c   | 615 +++++++++++
>  .../arm/mali-c55/mali-c55-registers.h         | 365 +++++++
>  .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
>  .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
>  .../platform/arm/mali-c55/mali-c55-stats.c    | 350 +++++++
>  .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
>  drivers/media/v4l2-core/v4l2-common.c         |   4 +
>  drivers/media/v4l2-core/v4l2-ioctl.c          |   2 +
>  include/uapi/linux/media-bus-format.h         |   9 +-
>  .../uapi/linux/media/arm/mali-c55-config.h    | 851 ++++++++++++++++
>  include/uapi/linux/videodev2.h                |   3 +
>  29 files changed, 7370 insertions(+), 2 deletions(-)
>  create mode 100644 Documentation/admin-guide/media/mali-c55-graph.dot
>  create mode 100644 Documentation/admin-guide/media/mali-c55.rst
>  create mode 100644 Documentation/devicetree/bindings/media/arm,mali-c55.yaml
>  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
>  create mode 100644 drivers/media/platform/arm/Kconfig
>  create mode 100644 drivers/media/platform/arm/Makefile
>  create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
>  create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-params.c
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-stats.c
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>  create mode 100644 include/uapi/linux/media/arm/mali-c55-config.h

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Laurent Pinchart]

Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
> V4L2 and Media Controller compliant and creates subdevices to manage
> the ISP itself, its internal test pattern generator as well as the
> crop, scaler and output format functionality for each of its two
> output devices.
> 
> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
> Changes in v5:
> 
> 	- Reworked input formats - previously we allowed representing input data
> 	  as any 8-16 bit format. Now we only allow input data to be represented
> 	  by the new 20-bit bayer formats, which is corrected to the equivalent
> 	  16-bit format in RAW bypass mode.
> 	- Stopped bypassing blocks that we haven't added supporting parameters
> 	  for yet.
> 	- Addressed most of Sakari's comments from the list
> 
> Changes not yet made in v5:
> 
> 	- The output pipelines can still be started and stopped independently of
> 	  one another - I'd like to discuss that more. 
> 	- the TPG subdev still uses .s_stream() - I need to rebase onto a tree
> 	  with working .enable_streams() for a single-source-pad subdevice.
> 
> Changes in v4:
> 
> 	- Reworked mali_c55_update_bits() to internally perform the bit-shift

I really don't like that, it makes the code very confusing, even more so
as it differs from regmap_update_bits().

Look at this for instance:

	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);

It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
BIT(0).

Sorry, I know it will be painful, but this change needs to be reverted.

> 	- Reworked the resizer to allow cropping during streaming
> 	- Fixed a bug in NV12 output
> 
> Changes in v3:
> 
> 	- Mostly minor fixes suggested by Sakari
> 	- Fixed the sequencing of vb2 buffers to be synchronised across the two
> 	  capture devices.
> 
> Changes in v2:
> 
> 	- Clock handling
> 	- Fixed the warnings raised by the kernel test robot
> 
>  drivers/media/platform/Kconfig                |   1 +
>  drivers/media/platform/Makefile               |   1 +
>  drivers/media/platform/arm/Kconfig            |   5 +
>  drivers/media/platform/arm/Makefile           |   2 +
>  drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
>  drivers/media/platform/arm/mali-c55/Makefile  |   9 +
>  .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
>  .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
>  .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
>  .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
>  .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
>  .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
>  .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
>  .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
>  14 files changed, 4452 insertions(+)
>  create mode 100644 drivers/media/platform/arm/Kconfig
>  create mode 100644 drivers/media/platform/arm/Makefile
>  create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
>  create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c

I've skipped review of capture.c and resizer.c as I already have plenty
of comments for the other files, and it's getting late. I'll try to
review the rest tomorrow.

> 
> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> index 2d79bfc68c15..c929169766aa 100644
> --- a/drivers/media/platform/Kconfig
> +++ b/drivers/media/platform/Kconfig
> @@ -65,6 +65,7 @@ config VIDEO_MUX
>  source "drivers/media/platform/allegro-dvt/Kconfig"
>  source "drivers/media/platform/amlogic/Kconfig"
>  source "drivers/media/platform/amphion/Kconfig"
> +source "drivers/media/platform/arm/Kconfig"
>  source "drivers/media/platform/aspeed/Kconfig"
>  source "drivers/media/platform/atmel/Kconfig"
>  source "drivers/media/platform/broadcom/Kconfig"
> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> index da17301f7439..9a647abd5218 100644
> --- a/drivers/media/platform/Makefile
> +++ b/drivers/media/platform/Makefile
> @@ -8,6 +8,7 @@
>  obj-y += allegro-dvt/
>  obj-y += amlogic/
>  obj-y += amphion/
> +obj-y += arm/
>  obj-y += aspeed/
>  obj-y += atmel/
>  obj-y += broadcom/
> diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
> new file mode 100644
> index 000000000000..4f0764c329c7
> --- /dev/null
> +++ b/drivers/media/platform/arm/Kconfig
> @@ -0,0 +1,5 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +
> +comment "ARM media platform drivers"
> +
> +source "drivers/media/platform/arm/mali-c55/Kconfig"
> diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
> new file mode 100644
> index 000000000000..8cc4918725ef
> --- /dev/null
> +++ b/drivers/media/platform/arm/Makefile
> @@ -0,0 +1,2 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +obj-y += mali-c55/
> diff --git a/drivers/media/platform/arm/mali-c55/Kconfig b/drivers/media/platform/arm/mali-c55/Kconfig
> new file mode 100644
> index 000000000000..602085e28b01
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-c55/Kconfig
> @@ -0,0 +1,18 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +config VIDEO_MALI_C55
> +	tristate "ARM Mali-C55 Image Signal Processor driver"
> +	depends on V4L_PLATFORM_DRIVERS
> +	depends on VIDEO_DEV && OF
> +	depends on ARCH_VEXPRESS || COMPILE_TEST
> +	select MEDIA_CONTROLLER
> +	select VIDEO_V4L2_SUBDEV_API
> +	select VIDEOBUF2_DMA_CONTIG
> +	select VIDEOBUF2_VMALLOC
> +	select V4L2_FWNODE
> +	select GENERIC_PHY_MIPI_DPHY

Alphabetical order ?

> +	default n

That's the default, you don't have to specify ti.

> +	help
> +	  Enable this to support Arm's Mali-C55 Image Signal Processor.
> +
> +	  To compile this driver as a module, choose M here: the module
> +	  will be called mali-c55.
> diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
> new file mode 100644
> index 000000000000..77dcb2fbf0f4
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-c55/Makefile
> @@ -0,0 +1,9 @@
> +# SPDX-License-Identifier: GPL-2.0
> +
> +mali-c55-y := mali-c55-capture.o \
> +	      mali-c55-core.o \
> +	      mali-c55-isp.o \
> +	      mali-c55-tpg.o \
> +	      mali-c55-resizer.o

Alphabetical order here too.

> +
> +obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> new file mode 100644
> index 000000000000..1d539ac9c498
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> @@ -0,0 +1,951 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ARM Mali-C55 ISP Driver - Video capture devices
> + *
> + * Copyright (C) 2024 Ideas on Board Oy
> + */
> +
> +#include <linux/cleanup.h>
> +#include <linux/minmax.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/string.h>
> +#include <linux/videodev2.h>
> +
> +#include <media/v4l2-dev.h>
> +#include <media/v4l2-event.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-core.h>
> +#include <media/videobuf2-dma-contig.h>
> +
> +#include "mali-c55-common.h"
> +#include "mali-c55-registers.h"
> +
> +static const struct mali_c55_fmt mali_c55_fmts[] = {
> +	/*
> +	 * This table is missing some entries which need further work or
> +	 * investigation:
> +	 *
> +	 * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
> +	 * Base mode 5 is "Generic Data"
> +	 * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
> +	 * Base mode 9 seems to have no V4L2 equivalent
> +	 * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
> +	 * equivalent
> +	 */
> +	{
> +		.fourcc = V4L2_PIX_FMT_ARGB2101010,
> +		.mbus_codes = {
> +			MEDIA_BUS_FMT_RGB121212_1X36,
> +			MEDIA_BUS_FMT_RGB202020_1X60,
> +		},
> +		.is_raw = false,
> +		.registers = {
> +			.base_mode = MALI_C55_OUTPUT_A2R10G10B10,
> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> +		}
> +	},
> +	{
> +		.fourcc = V4L2_PIX_FMT_RGB565,
> +		.mbus_codes = {
> +			MEDIA_BUS_FMT_RGB121212_1X36,
> +			MEDIA_BUS_FMT_RGB202020_1X60,
> +		},
> +		.is_raw = false,
> +		.registers = {
> +			.base_mode = MALI_C55_OUTPUT_RGB565,
> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> +		}
> +	},
> +	{
> +		.fourcc = V4L2_PIX_FMT_BGR24,
> +		.mbus_codes = {
> +			MEDIA_BUS_FMT_RGB121212_1X36,
> +			MEDIA_BUS_FMT_RGB202020_1X60,
> +		},
> +		.is_raw = false,
> +		.registers = {
> +			.base_mode = MALI_C55_OUTPUT_RGB24,
> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> +		}
> +	},
> +	{
> +		.fourcc = V4L2_PIX_FMT_YUYV,
> +		.mbus_codes = {
> +			MEDIA_BUS_FMT_YUV10_1X30,
> +		},
> +		.is_raw = false,
> +		.registers = {
> +			.base_mode = MALI_C55_OUTPUT_YUY2,
> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> +		}
> +	},
> +	{
> +		.fourcc = V4L2_PIX_FMT_UYVY,
> +		.mbus_codes = {
> +			MEDIA_BUS_FMT_YUV10_1X30,
> +		},
> +		.is_raw = false,
> +		.registers = {
> +			.base_mode = MALI_C55_OUTPUT_UYVY,
> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> +		}
> +	},
> +	{
> +		.fourcc = V4L2_PIX_FMT_Y210,
> +		.mbus_codes = {
> +			MEDIA_BUS_FMT_YUV10_1X30,
> +		},
> +		.is_raw = false,
> +		.registers = {
> +			.base_mode = MALI_C55_OUTPUT_Y210,
> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> +		}
> +	},
> +	/*
> +	 * This is something of a hack, the ISP thinks it's running NV12M but
> +	 * by setting uv_plane = 0 we simply discard that planes and only output
> +	 * the Y-plane.
> +	 */
> +	{
> +		.fourcc = V4L2_PIX_FMT_GREY,
> +		.mbus_codes = {
> +			MEDIA_BUS_FMT_YUV10_1X30,
> +		},
> +		.is_raw = false,
> +		.registers = {
> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> +		}
> +	},
> +	{
> +		.fourcc = V4L2_PIX_FMT_NV12M,
> +		.mbus_codes = {
> +			MEDIA_BUS_FMT_YUV10_1X30,
> +		},
> +		.is_raw = false,
> +		.registers = {
> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
> +		}
> +	},
> +	{
> +		.fourcc = V4L2_PIX_FMT_NV21M,
> +		.mbus_codes = {
> +			MEDIA_BUS_FMT_YUV10_1X30,
> +		},
> +		.is_raw = false,
> +		.registers = {
> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
> +		}
> +	},
> +	/*
> +	 * RAW uncompressed formats are all packed in 16 bpp.
> +	 * TODO: Expand this list to encompass all possible RAW formats.
> +	 */
> +	{
> +		.fourcc = V4L2_PIX_FMT_SRGGB16,
> +		.mbus_codes = {
> +			MEDIA_BUS_FMT_SRGGB16_1X16,
> +		},
> +		.is_raw = true,
> +		.registers = {
> +			.base_mode = MALI_C55_OUTPUT_RAW16,
> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> +		}
> +	},
> +	{
> +		.fourcc = V4L2_PIX_FMT_SBGGR16,
> +		.mbus_codes = {
> +			MEDIA_BUS_FMT_SBGGR16_1X16,
> +		},
> +		.is_raw = true,
> +		.registers = {
> +			.base_mode = MALI_C55_OUTPUT_RAW16,
> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> +		}
> +	},
> +	{
> +		.fourcc = V4L2_PIX_FMT_SGBRG16,
> +		.mbus_codes = {
> +			MEDIA_BUS_FMT_SGBRG16_1X16,
> +		},
> +		.is_raw = true,
> +		.registers = {
> +			.base_mode = MALI_C55_OUTPUT_RAW16,
> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> +		}
> +	},
> +	{
> +		.fourcc = V4L2_PIX_FMT_SGRBG16,
> +		.mbus_codes = {
> +			MEDIA_BUS_FMT_SGRBG16_1X16,
> +		},
> +		.is_raw = true,
> +		.registers = {
> +			.base_mode = MALI_C55_OUTPUT_RAW16,
> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> +		}
> +	},
> +};
> +
> +static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
> +					       u32 code)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
> +		if (fmt->mbus_codes[i] == code)
> +			return true;
> +	}
> +
> +	return false;
> +}
> +
> +bool mali_c55_format_is_raw(unsigned int mbus_code)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> +		if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
> +			return mali_c55_fmts[i].is_raw;
> +	}
> +
> +	return false;
> +}
> +
> +static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> +		if (mali_c55_fmts[i].fourcc == pixelformat)
> +			return &mali_c55_fmts[i];
> +	}
> +
> +	/*
> +	 * If we find no matching pixelformat, we'll just default to the first
> +	 * one for now.
> +	 */
> +
> +	return &mali_c55_fmts[0];
> +}
> +
> +static const char * const capture_device_names[] = {
> +	"mali-c55 fr",
> +	"mali-c55 ds",
> +	"mali-c55 3a stats",
> +	"mali-c55 params",
> +};
> +
> +static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
> +{
> +	if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
> +		return capture_device_names[0];
> +
> +	if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
> +		return capture_device_names[1];
> +
> +	return "params/stat not supported yet";
> +}
> +
> +static int mali_c55_link_validate(struct media_link *link)
> +{
> +	struct video_device *vdev =
> +		media_entity_to_video_device(link->sink->entity);
> +	struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
> +	struct v4l2_subdev *sd =
> +		media_entity_to_v4l2_subdev(link->source->entity);
> +	const struct v4l2_pix_format_mplane *pix_mp;
> +	const struct mali_c55_fmt *cap_fmt;
> +	struct v4l2_subdev_format sd_fmt = {
> +		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
> +		.pad = link->source->index,
> +	};
> +	int ret;
> +
> +	ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
> +	if (ret)
> +		return ret;
> +
> +	pix_mp = &cap_dev->mode.pix_mp;
> +	cap_fmt = cap_dev->mode.capture_fmt;
> +
> +	if (sd_fmt.format.width != pix_mp->width ||
> +	    sd_fmt.format.height != pix_mp->height) {
> +		dev_dbg(cap_dev->mali_c55->dev,
> +			"link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
> +			link->source->entity->name, link->source->index,
> +			link->sink->entity->name, link->sink->index,
> +			sd_fmt.format.width, sd_fmt.format.height,
> +			pix_mp->width, pix_mp->height);
> +		return -EPIPE;
> +	}
> +
> +	if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
> +		dev_dbg(cap_dev->mali_c55->dev,
> +			"link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format %p4cc\n",
> +			link->source->entity->name, link->source->index,
> +			link->sink->entity->name, link->sink->index,
> +			sd_fmt.format.code, &pix_mp->pixelformat);
> +		return -EPIPE;
> +	}
> +
> +	return 0;
> +}
> +
> +static const struct media_entity_operations mali_c55_media_ops = {
> +	.link_validate = mali_c55_link_validate,
> +};
> +
> +static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
> +				    unsigned int *num_planes, unsigned int sizes[],
> +				    struct device *alloc_devs[])
> +{
> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> +	unsigned int i;
> +
> +	if (*num_planes) {
> +		if (*num_planes != cap_dev->mode.pix_mp.num_planes)
> +			return -EINVAL;
> +
> +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> +			if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
> +				return -EINVAL;
> +	} else {
> +		*num_planes = cap_dev->mode.pix_mp.num_planes;
> +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> +			sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
> +	}
> +
> +	return 0;
> +}
> +
> +static void mali_c55_buf_queue(struct vb2_buffer *vb)
> +{
> +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> +	struct mali_c55_buffer *buf = container_of(vbuf,
> +						   struct mali_c55_buffer, vb);
> +	unsigned int i;
> +
> +	buf->plane_done[MALI_C55_PLANE_Y] = false;
> +
> +	/*
> +	 * If we're in a single-plane format we flag the other plane as done
> +	 * already so it's dequeued appropriately later
> +	 */
> +	buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
> +
> +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
> +		unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
> +
> +		vb2_set_plane_payload(vb, i, size);
> +	}
> +
> +	spin_lock(&cap_dev->buffers.lock);
> +	list_add_tail(&buf->queue, &cap_dev->buffers.queue);
> +	spin_unlock(&cap_dev->buffers.lock);
> +}
> +
> +static int mali_c55_buf_init(struct vb2_buffer *vb)
> +{
> +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> +	struct mali_c55_buffer *buf = container_of(vbuf,
> +						   struct mali_c55_buffer, vb);
> +	unsigned int i;
> +
> +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> +		buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
> +
> +	return 0;
> +}
> +
> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
> +{
> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> +
> +	guard(spinlock)(&cap_dev->buffers.lock);
> +
> +	cap_dev->buffers.curr = cap_dev->buffers.next;
> +	cap_dev->buffers.next = NULL;
> +
> +	if (!list_empty(&cap_dev->buffers.queue)) {
> +		struct v4l2_pix_format_mplane *pix_mp;
> +		const struct v4l2_format_info *info;
> +		u32 *addrs;
> +
> +		pix_mp = &cap_dev->mode.pix_mp;
> +		info = v4l2_format_info(pix_mp->pixelformat);
> +
> +		mali_c55_update_bits(mali_c55,
> +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
> +		if (cap_dev->mode.capture_fmt->registers.uv_plane)
> +			mali_c55_update_bits(mali_c55,
> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
> +
> +		cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
> +							 struct mali_c55_buffer,
> +							 queue);
> +		list_del(&cap_dev->buffers.next->queue);
> +
> +		addrs = cap_dev->buffers.next->addrs;
> +		mali_c55_write(mali_c55,
> +			MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
> +			addrs[MALI_C55_PLANE_Y]);
> +		mali_c55_write(mali_c55,
> +			MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
> +			addrs[MALI_C55_PLANE_UV]);
> +		mali_c55_write(mali_c55,
> +			MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
> +			pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
> +		mali_c55_write(mali_c55,
> +			MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
> +			pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
> +			/ info->hdiv);
> +	} else {
> +		/*
> +		 * If we underflow then we can tell the ISP that we don't want
> +		 * to write out the next frame.
> +		 */
> +		mali_c55_update_bits(mali_c55,
> +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> +		mali_c55_update_bits(mali_c55,
> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> +	}
> +}
> +
> +static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
> +				   unsigned int framecount)
> +{
> +	curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> +	curr_buf->vb.field = V4L2_FIELD_NONE;
> +	curr_buf->vb.sequence = framecount;
> +	vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
> +}
> +
> +/**
> + * mali_c55_set_plane_done - mark the plane as written and process the buffer if
> + *			     both planes are finished.
> + * @cap_dev:  pointer to the fr or ds pipe output
> + * @plane:    the plane to mark as completed
> + *
> + * The Mali C55 ISP has muliplanar outputs for some formats that come with two
> + * separate "buffer write completed" interrupts - we need to flag each plane's
> + * completion and check whether both planes are done - if so, complete the buf
> + * in vb2.
> + */
> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> +			     enum mali_c55_planes plane)
> +{
> +	struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
> +	struct mali_c55_buffer *curr_buf;
> +
> +	guard(spinlock)(&cap_dev->buffers.lock);
> +	curr_buf = cap_dev->buffers.curr;
> +
> +	/*
> +	 * This _should_ never happen. If no buffer was available from vb2 then
> +	 * we tell the ISP not to bother writing the next frame, which means the
> +	 * interrupts that call this function should never trigger. If it does
> +	 * happen then one of our assumptions is horribly wrong - complain
> +	 * loudly and do nothing.
> +	 */
> +	if (!curr_buf) {
> +		dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
> +			mali_c55_cap_dev_to_name(cap_dev), __func__);
> +		return;
> +	}
> +
> +	/* If the other plane is also done... */
> +	if (curr_buf->plane_done[~plane & 1]) {
> +		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> +		cap_dev->buffers.curr = NULL;
> +		isp->frame_sequence++;
> +	} else {
> +		curr_buf->plane_done[plane] = true;
> +	}
> +}
> +
> +static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
> +{
> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> +
> +	mali_c55_update_bits(mali_c55,
> +			     MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> +	mali_c55_update_bits(mali_c55,
> +			     MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> +}
> +
> +static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
> +{
> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> +
> +	/*
> +	 * The Mali ISP can hold up to 5 buffer addresses and simply cycle
> +	 * through them, but it's not clear to me that the vb2 queue _guarantees_
> +	 * it will queue buffers to the driver in a fixed order, and ensuring
> +	 * we call vb2_buffer_done() for the right buffer seems to me to add
> +	 * pointless complexity given in multi-context mode we'd need to
> +	 * re-write those registers every frame anyway...so we tell the ISP to
> +	 * use a single register and update it for each frame.
> +	 */
> +	mali_c55_update_bits(mali_c55,
> +			MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
> +			MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
> +	mali_c55_update_bits(mali_c55,
> +			MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
> +			MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
> +
> +	/*
> +	 * We only queue a buffer in the streamon path if this is the first of
> +	 * the capture devices to start streaming. If the ISP is already running
> +	 * then we rely on the ISP_START interrupt to queue the first buffer for
> +	 * this capture device.
> +	 */
> +	if (mali_c55->pipe.start_count == 1)
> +		mali_c55_set_next_buffer(cap_dev);
> +}
> +
> +static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
> +					    enum vb2_buffer_state state)
> +{
> +	struct mali_c55_buffer *buf, *tmp;
> +
> +	guard(spinlock)(&cap_dev->buffers.lock);
> +
> +	if (cap_dev->buffers.curr) {
> +		vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
> +				state);
> +		cap_dev->buffers.curr = NULL;
> +	}
> +
> +	if (cap_dev->buffers.next) {
> +		vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
> +				state);
> +		cap_dev->buffers.next = NULL;
> +	}
> +
> +	list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
> +		list_del(&buf->queue);
> +		vb2_buffer_done(&buf->vb.vb2_buf, state);
> +	}
> +}
> +
> +static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
> +{
> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> +	struct mali_c55_resizer *rzr = cap_dev->rzr;
> +	struct mali_c55_isp *isp = &mali_c55->isp;
> +	int ret;
> +
> +	guard(mutex)(&isp->lock);
> +
> +	ret = pm_runtime_resume_and_get(mali_c55->dev);
> +	if (ret)
> +		return ret;
> +
> +	ret = video_device_pipeline_start(&cap_dev->vdev,
> +					  &cap_dev->mali_c55->pipe);
> +	if (ret) {
> +		dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
> +			mali_c55_cap_dev_to_name(cap_dev));
> +		goto err_pm_put;
> +	}
> +
> +	mali_c55_cap_dev_stream_enable(cap_dev);
> +	mali_c55_rzr_start_stream(rzr);
> +
> +	/*
> +	 * We only start the ISP if we're the only capture device that's
> +	 * streaming. Otherwise, it'll already be active.
> +	 */
> +	if (mali_c55->pipe.start_count == 1) {
> +		ret = mali_c55_isp_start_stream(isp);
> +		if (ret)
> +			goto err_disable_cap_dev;
> +	}
> +
> +	return 0;
> +
> +err_disable_cap_dev:
> +	mali_c55_cap_dev_stream_disable(cap_dev);
> +	video_device_pipeline_stop(&cap_dev->vdev);
> +err_pm_put:
> +	pm_runtime_put(mali_c55->dev);
> +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
> +
> +	return ret;
> +}
> +
> +static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
> +{
> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> +	struct mali_c55_resizer *rzr = cap_dev->rzr;
> +	struct mali_c55_isp *isp = &mali_c55->isp;
> +
> +	guard(mutex)(&isp->lock);
> +
> +	/*
> +	 * If one of the other capture nodes is streaming, we shouldn't
> +	 * disable the ISP here.
> +	 */
> +	if (mali_c55->pipe.start_count == 1)
> +		mali_c55_isp_stop_stream(&mali_c55->isp);
> +
> +	mali_c55_rzr_stop_stream(rzr);
> +	mali_c55_cap_dev_stream_disable(cap_dev);
> +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
> +	video_device_pipeline_stop(&cap_dev->vdev);
> +	pm_runtime_put(mali_c55->dev);
> +}
> +
> +static const struct vb2_ops mali_c55_vb2_ops = {
> +	.queue_setup		= &mali_c55_vb2_queue_setup,
> +	.buf_queue		= &mali_c55_buf_queue,
> +	.buf_init		= &mali_c55_buf_init,
> +	.wait_prepare		= vb2_ops_wait_prepare,
> +	.wait_finish		= vb2_ops_wait_finish,
> +	.start_streaming	= &mali_c55_vb2_start_streaming,
> +	.stop_streaming		= &mali_c55_vb2_stop_streaming,
> +};
> +
> +static const struct v4l2_file_operations mali_c55_v4l2_fops = {
> +	.owner = THIS_MODULE,
> +	.unlocked_ioctl = video_ioctl2,
> +	.open = v4l2_fh_open,
> +	.release = vb2_fop_release,
> +	.poll = vb2_fop_poll,
> +	.mmap = vb2_fop_mmap,
> +};
> +
> +static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
> +{
> +	const struct mali_c55_fmt *capture_format;
> +	const struct v4l2_format_info *info;
> +	struct v4l2_plane_pix_format *plane;
> +	unsigned int i;
> +
> +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
> +	pix_mp->pixelformat = capture_format->fourcc;
> +
> +	pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
> +			      MALI_C55_MAX_WIDTH);
> +	pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
> +			       MALI_C55_MAX_HEIGHT);
> +
> +	pix_mp->field = V4L2_FIELD_NONE;
> +	pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
> +	pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> +	pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
> +
> +	info = v4l2_format_info(pix_mp->pixelformat);
> +	pix_mp->num_planes = info->mem_planes;
> +	memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
> +
> +	pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;
> +	pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
> +				       * pix_mp->height;
> +
> +	for (i = 1; i < info->comp_planes; i++) {
> +		plane = &pix_mp->plane_fmt[i];
> +
> +		plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
> +						   info->hdiv);
> +		plane->sizeimage = DIV_ROUND_UP(
> +					plane->bytesperline * pix_mp->height,
> +					info->vdiv);
> +	}
> +
> +	if (info->mem_planes == 1) {
> +		for (i = 1; i < info->comp_planes; i++) {
> +			plane = &pix_mp->plane_fmt[i];
> +			pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
> +		}
> +	}
> +}
> +
> +static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
> +					   struct v4l2_format *f)
> +{
> +	mali_c55_try_fmt(&f->fmt.pix_mp);
> +
> +	return 0;
> +}
> +
> +static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
> +				struct v4l2_pix_format_mplane *pix_mp)
> +{
> +	const struct mali_c55_fmt *capture_format;
> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> +	const struct v4l2_format_info *info;
> +
> +	mali_c55_try_fmt(pix_mp);
> +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
> +	info = v4l2_format_info(pix_mp->pixelformat);
> +	if (WARN_ON(!info))
> +		return;
> +
> +	mali_c55_write(mali_c55,
> +		       MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> +		       capture_format->registers.base_mode);
> +	mali_c55_write(mali_c55,
> +		       MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
> +		       MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
> +		       MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
> +
> +	if (info->mem_planes > 1) {
> +		mali_c55_write(mali_c55,
> +			       MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> +			       capture_format->registers.base_mode);
> +		mali_c55_update_bits(mali_c55,
> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> +				MALI_C55_WRITER_SUBMODE_MASK,
> +				capture_format->registers.uv_plane);
> +
> +		mali_c55_write(mali_c55,
> +			MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
> +			MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
> +			MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
> +	}
> +
> +	if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
> +		/*
> +		 * TODO: Figure out the colour matrix coefficients and calculate
> +		 * and write them here.
> +		 */
> +
> +		mali_c55_write(mali_c55,
> +			       MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> +			       MALI_C55_CS_CONV_MATRIX_MASK);
> +
> +		if (info->hdiv > 1)
> +			mali_c55_update_bits(mali_c55,
> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> +				MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
> +		if (info->vdiv > 1)
> +			mali_c55_update_bits(mali_c55,
> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> +				MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
> +		if (info->hdiv > 1 || info->vdiv > 1)
> +			mali_c55_update_bits(mali_c55,
> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> +				MALI_C55_CS_CONV_FILTER_MASK, 0x01);
> +	}
> +
> +	cap_dev->mode.pix_mp = *pix_mp;
> +	cap_dev->mode.capture_fmt = capture_format;
> +}
> +
> +static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
> +					 struct v4l2_format *f)
> +{
> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> +
> +	if (vb2_is_busy(&cap_dev->queue))
> +		return -EBUSY;
> +
> +	mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
> +
> +	return 0;
> +}
> +
> +static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
> +					 struct v4l2_format *f)
> +{
> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> +
> +	f->fmt.pix_mp = cap_dev->mode.pix_mp;
> +
> +	return 0;
> +}
> +
> +static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
> +					    struct v4l2_fmtdesc *f)
> +{
> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> +	unsigned int j = 0;
> +	unsigned int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> +		if (f->mbus_code &&
> +		    !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
> +						       f->mbus_code))
> +			continue;
> +
> +		/* Downscale pipe can't output RAW formats */
> +		if (mali_c55_fmts[i].is_raw &&
> +		    cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
> +			continue;
> +
> +		if (j++ == f->index) {
> +			f->pixelformat = mali_c55_fmts[i].fourcc;
> +			return 0;
> +		}
> +	}
> +
> +	return -EINVAL;
> +}
> +
> +static int mali_c55_querycap(struct file *file, void *fh,
> +			     struct v4l2_capability *cap)
> +{
> +	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
> +	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
> +	.vidioc_querybuf = vb2_ioctl_querybuf,
> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
> +	.vidioc_qbuf = vb2_ioctl_qbuf,
> +	.vidioc_expbuf = vb2_ioctl_expbuf,
> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> +	.vidioc_streamon = vb2_ioctl_streamon,
> +	.vidioc_streamoff = vb2_ioctl_streamoff,
> +	.vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
> +	.vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
> +	.vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
> +	.vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
> +	.vidioc_querycap = mali_c55_querycap,
> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
> +};
> +
> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
> +{
> +	struct v4l2_pix_format_mplane pix_mp;
> +	struct mali_c55_cap_dev *cap_dev;
> +	struct video_device *vdev;
> +	struct vb2_queue *vb2q;
> +	unsigned int i;
> +	int ret;
> +
> +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
> +		cap_dev = &mali_c55->cap_devs[i];
> +		vdev = &cap_dev->vdev;
> +		vb2q = &cap_dev->queue;
> +
> +		/*
> +		 * The downscale output pipe is an optional block within the ISP
> +		 * so we need to check whether it's actually been fitted or not.
> +		 */
> +
> +		if (i == MALI_C55_CAP_DEV_DS &&
> +		    !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
> +			continue;
> +
> +		cap_dev->mali_c55 = mali_c55;
> +		mutex_init(&cap_dev->lock);
> +		INIT_LIST_HEAD(&cap_dev->buffers.queue);
> +
> +		switch (i) {
> +		case MALI_C55_CAP_DEV_FR:
> +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
> +			cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
> +			break;
> +		case MALI_C55_CAP_DEV_DS:
> +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
> +			cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
> +			break;
> +		default:
> +			mutex_destroy(&cap_dev->lock);
> +			ret = -EINVAL;
> +			goto err_destroy_mutex;
> +		}
> +
> +		cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
> +		ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
> +		if (ret) {
> +			mutex_destroy(&cap_dev->lock);
> +			goto err_destroy_mutex;
> +		}
> +
> +		vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> +		vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
> +		vb2q->drv_priv = cap_dev;
> +		vb2q->mem_ops = &vb2_dma_contig_memops;
> +		vb2q->ops = &mali_c55_vb2_ops;
> +		vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
> +		vb2q->min_queued_buffers = 1;
> +		vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> +		vb2q->lock = &cap_dev->lock;
> +		vb2q->dev = mali_c55->dev;
> +
> +		ret = vb2_queue_init(vb2q);
> +		if (ret) {
> +			dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
> +				mali_c55_cap_dev_to_name(cap_dev));
> +			goto err_cleanup_media_entity;
> +		}
> +
> +		strscpy(cap_dev->vdev.name, capture_device_names[i],
> +			sizeof(cap_dev->vdev.name));
> +		vdev->release = video_device_release_empty;
> +		vdev->fops = &mali_c55_v4l2_fops;
> +		vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
> +		vdev->lock = &cap_dev->lock;
> +		vdev->v4l2_dev = &mali_c55->v4l2_dev;
> +		vdev->queue = &cap_dev->queue;
> +		vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +				    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
> +		vdev->entity.ops = &mali_c55_media_ops;
> +		video_set_drvdata(vdev, cap_dev);
> +
> +		memset(&pix_mp, 0, sizeof(pix_mp));
> +		pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
> +		pix_mp.width = MALI_C55_DEFAULT_WIDTH;
> +		pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
> +		mali_c55_set_format(cap_dev, &pix_mp);
> +
> +		ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
> +		if (ret) {
> +			dev_err(mali_c55->dev,
> +				"%s failed to register video device\n",
> +				mali_c55_cap_dev_to_name(cap_dev));
> +			goto err_release_vb2q;
> +		}
> +	}
> +
> +	return 0;
> +
> +err_release_vb2q:
> +	vb2_queue_release(vb2q);
> +err_cleanup_media_entity:
> +	media_entity_cleanup(&cap_dev->vdev.entity);
> +err_destroy_mutex:
> +	mutex_destroy(&cap_dev->lock);
> +	mali_c55_unregister_capture_devs(mali_c55);
> +
> +	return ret;
> +}
> +
> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
> +{
> +	struct mali_c55_cap_dev *cap_dev;
> +	unsigned int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
> +		cap_dev = &mali_c55->cap_devs[i];
> +
> +		if (!video_is_registered(&cap_dev->vdev))
> +			continue;
> +
> +		vb2_video_unregister_device(&cap_dev->vdev);
> +		media_entity_cleanup(&cap_dev->vdev.entity);
> +		mutex_destroy(&cap_dev->lock);
> +	}
> +}
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> new file mode 100644
> index 000000000000..2d0c4d152beb
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> @@ -0,0 +1,266 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * ARM Mali-C55 ISP Driver - Common definitions
> + *
> + * Copyright (C) 2024 Ideas on Board Oy
> + */
> +
> +#ifndef _MALI_C55_COMMON_H
> +#define _MALI_C55_COMMON_H
> +
> +#include <linux/clk.h>
> +#include <linux/io.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/scatterlist.h>

I don't think this is needed. You're however missing spinlock.h.

> +#include <linux/videodev2.h>
> +
> +#include <media/media-device.h>
> +#include <media/v4l2-async.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-dev.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-core.h>
> +#include <media/videobuf2-v4l2.h>
> +
> +#define MALI_C55_DRIVER_NAME		"mali-c55"
> +
> +/* min and max values for the image sizes */
> +#define MALI_C55_MIN_WIDTH		640U
> +#define MALI_C55_MIN_HEIGHT		480U
> +#define MALI_C55_MAX_WIDTH		8192U
> +#define MALI_C55_MAX_HEIGHT		8192U
> +#define MALI_C55_DEFAULT_WIDTH		1920U
> +#define MALI_C55_DEFAULT_HEIGHT		1080U
> +
> +#define MALI_C55_DEFAULT_MEDIA_BUS_FMT	MEDIA_BUS_FMT_RGB121212_1X36
> +
> +struct mali_c55;
> +struct mali_c55_cap_dev;
> +struct platform_device;

You should also forward-declare

struct device;
struct dma_chan;
struct resource;

> +
> +static const char * const mali_c55_clk_names[] = {
> +	"aclk",
> +	"hclk",
> +};

This will end up duplicating the array in each compilation unit, not
great. Move it to mali-c55-core.c. You use it in this file just for its
size, replace that with a macro that defines the size, or allocate
mali_c55.clks dynamically with devm_kcalloc().

> +
> +enum mali_c55_interrupts {
> +	MALI_C55_IRQ_ISP_START,
> +	MALI_C55_IRQ_ISP_DONE,
> +	MALI_C55_IRQ_MCM_ERROR,
> +	MALI_C55_IRQ_BROKEN_FRAME_ERROR,
> +	MALI_C55_IRQ_MET_AF_DONE,
> +	MALI_C55_IRQ_MET_AEXP_DONE,
> +	MALI_C55_IRQ_MET_AWB_DONE,
> +	MALI_C55_IRQ_AEXP_1024_DONE,
> +	MALI_C55_IRQ_IRIDIX_MET_DONE,
> +	MALI_C55_IRQ_LUT_INIT_DONE,
> +	MALI_C55_IRQ_FR_Y_DONE,
> +	MALI_C55_IRQ_FR_UV_DONE,
> +	MALI_C55_IRQ_DS_Y_DONE,
> +	MALI_C55_IRQ_DS_UV_DONE,
> +	MALI_C55_IRQ_LINEARIZATION_DONE,
> +	MALI_C55_IRQ_RAW_FRONTEND_DONE,
> +	MALI_C55_IRQ_NOISE_REDUCTION_DONE,
> +	MALI_C55_IRQ_IRIDIX_DONE,
> +	MALI_C55_IRQ_BAYER2RGB_DONE,
> +	MALI_C55_IRQ_WATCHDOG_TIMER,
> +	MALI_C55_IRQ_FRAME_COLLISION,
> +	MALI_C55_IRQ_UNUSED,
> +	MALI_C55_IRQ_DMA_ERROR,
> +	MALI_C55_IRQ_INPUT_STOPPED,
> +	MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
> +	MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
> +	MALI_C55_NUM_IRQ_BITS

Those are register bits, I think they belong to mali-c55-registers.h,
and should probably be macros instead of an enum.

> +};
> +
> +enum mali_c55_isp_pads {
> +	MALI_C55_ISP_PAD_SINK_VIDEO,

As there's a single sink pad, maybe MALI_C55_ISP_PAD_SINK ? Ah, you're
probably preparing for ISP parameters support. It's fine.

> +	MALI_C55_ISP_PAD_SOURCE,

Then maybe this should be named MALI_C55_ISP_PAD_SOURCE_VIDEO as I
assume there will be a stats source pad.

> +	MALI_C55_ISP_PAD_SOURCE_BYPASS,
> +	MALI_C55_ISP_NUM_PADS,
> +};
> +
> +struct mali_c55_tpg {
> +	struct mali_c55 *mali_c55;
> +	struct v4l2_subdev sd;
> +	struct media_pad pad;
> +	struct mutex lock;
> +	struct mali_c55_tpg_ctrls {
> +		struct v4l2_ctrl_handler handler;
> +		struct v4l2_ctrl *test_pattern;

Set but never used. You can drop it.

> +		struct v4l2_ctrl *hblank;

Set and used only once, in the same function. You can make it a local
variable.

> +		struct v4l2_ctrl *vblank;
> +	} ctrls;
> +};

I wonder if this file should be split, with mali-c55-capture.h,
mali-c55-core.h, mali-c55-isp.h, ... I think it could increase
readability by clearly separating the different elements. Up to you.

> +
> +struct mali_c55_isp {
> +	struct mali_c55 *mali_c55;
> +	struct v4l2_subdev sd;
> +	struct media_pad pads[MALI_C55_ISP_NUM_PADS];
> +	struct media_pad *remote_src;
> +	struct v4l2_async_notifier notifier;

I'm tempted to move the notifier to mali_c55, as it's related to
components external to the whole ISP, not to the ISP subdev itself.
Could you give it a try, to see if it could be done without any drawback
?

> +	struct mutex lock;

Locks require a comment to explain what they protect. Same below where
applicable (for both mutexes and spinlocks).

> +	unsigned int frame_sequence;
> +};
> +
> +enum mali_c55_resizer_ids {
> +	MALI_C55_RZR_FR,
> +	MALI_C55_RZR_DS,
> +	MALI_C55_NUM_RZRS,

The usual abbreviation for "resizer" in drivers/media/ is "rsz", not
"rzr". I would have said we can leave it as-is as changing it would be a
bit annoying, but I then realized that "rzr" is not just unusual, it's
actually not used at all. Would you mind applying a sed globally ? :-)

> +};
> +
> +enum mali_c55_rzr_pads {

Same enums/structs use abbreviations, some don't. Consistency would
help.

> +	MALI_C55_RZR_SINK_PAD,
> +	MALI_C55_RZR_SOURCE_PAD,
> +	MALI_C55_RZR_SINK_BYPASS_PAD,
> +	MALI_C55_RZR_NUM_PADS
> +};
> +
> +struct mali_c55_resizer {
> +	struct mali_c55 *mali_c55;
> +	struct mali_c55_cap_dev *cap_dev;
> +	enum mali_c55_resizer_ids id;
> +	struct v4l2_subdev sd;
> +	struct media_pad pads[MALI_C55_RZR_NUM_PADS];
> +	unsigned int num_routes;
> +	bool streaming;
> +};
> +
> +enum mali_c55_cap_devs {
> +	MALI_C55_CAP_DEV_FR,
> +	MALI_C55_CAP_DEV_DS,
> +	MALI_C55_NUM_CAP_DEVS
> +};
> +
> +struct mali_c55_fmt {

mali_c55_format_info would be a better name I think, as this stores
format information, not formats.

> +	u32 fourcc;
> +	unsigned int mbus_codes[2];

A comment to explain why we have two media bus codes would be useful.
You can document the whole structure if desired :-)

> +	bool is_raw;
> +	struct mali_c55_fmt_registers {

Make it an anonymous structure, it's never used anywhere else.

> +		unsigned int base_mode;
> +		unsigned int uv_plane;

If those are register field values, use u32 instead of unsigned int.

> +	} registers;

It's funny, we tend to abbreviate different things, I would have used
"regs" here but written "format" in full in the structure name :-)

> +};
> +
> +enum mali_c55_isp_bayer_order {
> +	MALI_C55_BAYER_ORDER_RGGB,
> +	MALI_C55_BAYER_ORDER_GRBG,
> +	MALI_C55_BAYER_ORDER_GBRG,
> +	MALI_C55_BAYER_ORDER_BGGR

These are registers values too, they belong to mali-c55-registers.h.

> +};
> +
> +struct mali_c55_isp_fmt {

mali_c55_isp_format_info

> +	u32 code;
> +	enum v4l2_pixel_encoding encoding;

Here you use v4l2_pixel_encoding, for mali_c55_fmt you use is_raw. Maybe
pick the same option for both structures ?

> +	enum mali_c55_isp_bayer_order order;
> +};
> +
> +enum mali_c55_planes {
> +	MALI_C55_PLANE_Y,
> +	MALI_C55_PLANE_UV,
> +	MALI_C55_NUM_PLANES
> +};
> +
> +struct mali_c55_buffer {
> +	struct vb2_v4l2_buffer vb;
> +	bool plane_done[MALI_C55_NUM_PLANES];

I think tracking the pending state would simplify the logic in
mali_c55_set_plane_done(), which would become

	curr_buf->plane_pending[plane] = false;

	if (!curr_buf->plane_pending[0] && !curr_buf->plane_pending[1]) {
		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
		cap_dev->buffers.curr = NULL;
		isp->frame_sequence++;
	}

Or a counter may be even easier (and would consume less memory).

> +	struct list_head queue;
> +	u32 addrs[MALI_C55_NUM_PLANES];

This stores DMA addresses, use dma_addr_t.

> +};
> +
> +struct mali_c55_cap_dev {
> +	struct mali_c55 *mali_c55;
> +	struct mali_c55_resizer *rzr;
> +	struct video_device vdev;
> +	struct media_pad pad;
> +	struct vb2_queue queue;
> +	struct mutex lock;
> +	unsigned int reg_offset;

Manual handling of the offset everywhere, with parametric macros for the
resizer register addresses, isn't very nice. Introduce resizer-specific
accessors and capture-specific accessors (mali_c55_rsz_write(), ...)
that take a mali_c55_resizer or mali_c55_cap_dev pointer, and handle the
offset there. The register macros should loose their offset parameter.

You could also use a single set of accessors that would become
path/output-specific (mali_c55_path_write() ? mali_c55_output_write()
?), that may make the code easier to read.

You can also replace reg_offset with a void __iomem * base, which would
avoid the computation at runtime.

> +
> +	struct mali_c55_mode {

Make the structure anonymous.

> +		const struct mali_c55_fmt *capture_fmt;
> +		struct v4l2_pix_format_mplane pix_mp;
> +	} mode;

What's a "mode" ? I think I'd name this

	struct {
		const struct mali_c55_fmt *info;
		struct v4l2_pix_format_mplane format;
	} format;

Or you could just drop the structure and have

	const struct mali_c55_fmt *format_info;
	struct v4l2_pix_format_mplane format;

or something similar.

> +
> +	struct {
> +		spinlock_t lock;
> +		struct list_head queue;
> +		struct mali_c55_buffer *curr;
> +		struct mali_c55_buffer *next;
> +	} buffers;
> +
> +	bool streaming;
> +};
> +
> +enum mali_c55_config_spaces {
> +	MALI_C55_CONFIG_PING,
> +	MALI_C55_CONFIG_PONG,
> +	MALI_C55_NUM_CONFIG_SPACES

The last enumerator is not used.

> +};
> +
> +struct mali_c55_ctx {

mali_c55_context ?

> +	struct mali_c55 *mali_c55;
> +	void *registers;

Please document this structure and explain that this field points to a
copy of the register space in system memory, I was about to write you're
missing __iomem :-)

> +	phys_addr_t base;
> +	spinlock_t lock;
> +	struct list_head list;
> +};
> +
> +struct mali_c55 {
> +	struct device *dev;
> +	struct resource *res;

You could possibly drop this field by passing the physical address of
the register space from mali_c55_probe() to mali_c55_init_context() as a
function parameter.

> +	void __iomem *base;
> +	struct dma_chan *channel;
> +	struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
> +
> +	u16 capabilities;
> +	struct media_device media_dev;
> +	struct v4l2_device v4l2_dev;
> +	struct media_pipeline pipe;
> +
> +	struct mali_c55_tpg tpg;
> +	struct mali_c55_isp isp;
> +	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
> +	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
> +
> +	struct list_head contexts;
> +	enum mali_c55_config_spaces next_config;
> +};
> +
> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
> +		  bool force_hardware);
> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
> +			  u32 mask, u32 val);
> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
> +			  enum mali_c55_config_spaces cfg_space);
> +
> +int mali_c55_register_isp(struct mali_c55 *mali_c55);
> +int mali_c55_register_tpg(struct mali_c55 *mali_c55);
> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
> +int mali_c55_register_resizers(struct mali_c55 *mali_c55);
> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> +			     enum mali_c55_planes plane);
> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
> +
> +bool mali_c55_format_is_raw(unsigned int mbus_code);
> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
> +
> +const struct mali_c55_isp_fmt *
> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
> +#define for_each_mali_isp_fmt(fmt)\

#define for_each_mali_isp_fmt(fmt) \

> +	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)

Looks like parentheses were on sale :-)

	for ((fmt) = NULL; (fmt) = mali_c55_isp_fmt_next(fmt); )

This macro is used in two places only, in the mali-c55-isp.c file where
open-coding the loop without using mali_c55_isp_fmt_next() would be more
efficient, and in mali-c55-resizer.c where a function to return format
i'th would be more efficient. I think you can drop the macro and the
mali_c55_isp_fmt_next() function.

> +
> +#endif /* _MALI_C55_COMMON_H */
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> new file mode 100644
> index 000000000000..50caf5ee7474
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> @@ -0,0 +1,767 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ARM Mali-C55 ISP Driver - Core driver code
> + *
> + * Copyright (C) 2024 Ideas on Board Oy
> + */
> +
> +#include <linux/bitops.h>
> +#include <linux/cleanup.h>
> +#include <linux/clk.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/dmaengine.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/interrupt.h>
> +#include <linux/iopoll.h>
> +#include <linux/ioport.h>
> +#include <linux/mod_devicetable.h>
> +#include <linux/of.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/platform_device.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/scatterlist.h>

I don't think this is needed.

Missing slab.h.

> +#include <linux/string.h>
> +
> +#include <media/media-entity.h>
> +#include <media/v4l2-device.h>
> +#include <media/videobuf2-dma-contig.h>
> +
> +#include "mali-c55-common.h"
> +#include "mali-c55-registers.h"
> +
> +static const char * const mali_c55_interrupt_names[] = {
> +	[MALI_C55_IRQ_ISP_START] = "ISP start",
> +	[MALI_C55_IRQ_ISP_DONE] = "ISP done",
> +	[MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
> +	[MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
> +	[MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
> +	[MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
> +	[MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
> +	[MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
> +	[MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
> +	[MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
> +	[MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
> +	[MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
> +	[MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
> +	[MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
> +	[MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
> +	[MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
> +	[MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
> +	[MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
> +	[MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
> +	[MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
> +	[MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
> +	[MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
> +	[MALI_C55_IRQ_DMA_ERROR] = "DMA error",
> +	[MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
> +	[MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
> +	[MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
> +};
> +
> +static unsigned int config_space_addrs[] = {

const

> +	[MALI_C55_CONFIG_PING] = 0x0AB6C,
> +	[MALI_C55_CONFIG_PONG] = 0x22B2C,

Lowercase hex constants.

Don't the values belong to mali-c55-registers.h ?

> +};
> +
> +/* System IO

/*
 * System IO

> + *
> + * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
> + * and 'pong'), with the  expectation that the 'active' space will be left

s/the  /the /

> + * untouched whilst a frame is being processed and the 'inactive' space
> + * configured ready to be passed during the blanking period before the next

s/to be passed/to be switched to/ ?

> + * frame processing starts. These spaces should ideally be set via DMA transfer
> + * from a buffer rather than through individual register set operations. There
> + * is also a shared global register space which should be set normally. Of
> + * course, the ISP might be included in a system which lacks a suitable DMA
> + * engine, and the second configuration space might not be fitted at all, which
> + * means we need to support four scenarios:
> + *
> + * 1. Multi config space, with DMA engine.
> + * 2. Multi config space, no DMA engine.
> + * 3. Single config space, with DMA engine.
> + * 4. Single config space, no DMA engine.
> + *
> + * The first case is very easy, but the rest present annoying problems. The best
> + * way to solve them seems to be simply to replicate the concept of DMAing over
> + * the configuration buffer even if there's no DMA engine on the board, for
> + * which we rely on memcpy. To facilitate this any read/write call that is made
> + * to an address within those config spaces should infact be directed to a
> + * buffer that was allocated to hold them rather than the IO memory itself. The
> + * actual copy of that buffer to IO mem will happen on interrupt.
> + */
> +
> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
> +{
> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> +
> +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
> +		spin_lock(&ctx->lock);
> +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
> +		((u32 *)ctx->registers)[addr] = val;
> +		spin_unlock(&ctx->lock);
> +
> +		return;
> +	}

Ouch. This is likely the second comment you really won't like (after the
comment regarding mali_c55_update_bits() at the very top). I apologize
in advance.

I really don't like this. Directing writes either to hardware registers
or to the shadow registers in the context makes the callers of the
read/write accessors very hard to read. The probe code, for instance,
mixes writes to hardware registers and writes to the context shadow
registers to initialize the value of some of the shadow registers.

I'd like to split the read/write accessors into functions that access
the hardware registers (that's easy) and functions that access the
shadow registers. I think the latter should receive a mali_c55_ctx
pointer instead of a mali_c55 pointer to prepare for multi-context
support.

You can add WARN_ON() guards to the two sets of functions, to ensure
that no register from the "other" space gets passed to the wrong
function by mistake.

> +
> +	writel(val, mali_c55->base + addr);
> +}
> +
> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
> +		  bool force_hardware)

force_hardware is never set to true.

> +{
> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> +	u32 val;
> +
> +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
> +		spin_lock(&ctx->lock);
> +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
> +		val = ((u32 *)ctx->registers)[addr];
> +		spin_unlock(&ctx->lock);
> +
> +		return val;
> +	}
> +
> +	return readl(mali_c55->base + addr);
> +}
> +
> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
> +			  u32 mask, u32 val)
> +{
> +	u32 orig, tmp;
> +
> +	orig = mali_c55_read(mali_c55, addr, false);
> +
> +	tmp = orig & ~mask;
> +	tmp |= (val << (ffs(mask) - 1)) & mask;
> +
> +	if (tmp != orig)
> +		mali_c55_write(mali_c55, addr, tmp);
> +}
> +
> +static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
> +			     dma_addr_t dst, enum dma_data_direction dir)
> +{
> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> +	struct dma_async_tx_descriptor *tx;
> +	enum dma_status status;
> +	dma_cookie_t cookie;
> +
> +	tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
> +				       MALI_C55_CONFIG_SPACE_SIZE, 0);
> +	if (!tx) {
> +		dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
> +		return -EIO;
> +	}
> +
> +	cookie = dmaengine_submit(tx);
> +	if (dma_submit_error(cookie)) {
> +		dev_err(mali_c55->dev, "error submitting dma transfer\n");
> +		return -EIO;
> +	}
> +
> +	status = dma_sync_wait(mali_c55->channel, cookie);

I've just realized this performs a busy-wait :-S See the comment in the
probe function about the threaded IRQ handler. I think we'll need to
rework all this. It could be done on top though.

> +	if (status != DMA_COMPLETE) {
> +		dev_err(mali_c55->dev, "dma transfer failed\n");
> +		return -EIO;
> +	}
> +
> +	return 0;
> +}
> +
> +static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
> +			     enum mali_c55_config_spaces cfg_space)
> +{
> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> +	struct device *dma_dev = mali_c55->channel->device->dev;
> +	dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
> +	dma_addr_t dst;
> +	int ret;
> +
> +	guard(spinlock)(&ctx->lock);
> +
> +	dst = dma_map_single(dma_dev, ctx->registers,
> +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
> +	if (dma_mapping_error(dma_dev, dst)) {
> +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
> +		return -EIO;
> +	}
> +
> +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
> +	dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
> +			 DMA_FROM_DEVICE);
> +
> +	return ret;
> +}
> +
> +static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
> +		       enum mali_c55_config_spaces cfg_space)
> +{
> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> +	struct device *dma_dev = mali_c55->channel->device->dev;
> +	dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
> +	dma_addr_t src;
> +	int ret;
> +
> +	guard(spinlock)(&ctx->lock);

The code below can take a large amount of time, holding a spinlock will
disable interrupts on the local CPU, that's not good :-(

> +
> +	src = dma_map_single(dma_dev, ctx->registers,
> +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
> +	if (dma_mapping_error(dma_dev, src)) {
> +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
> +		return -EIO;
> +	}
> +
> +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
> +	dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
> +			 DMA_TO_DEVICE);
> +
> +	return ret;
> +}
> +
> +static int mali_c55_config_read(struct mali_c55_ctx *ctx,
> +				enum mali_c55_config_spaces cfg_space)
> +{
> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> +
> +	if (mali_c55->channel) {
> +		return mali_c55_dma_read(ctx, cfg_space);

As this function is used at probe time only, to initialize the context,
I think DMA is overkill.

> +	} else {
> +		memcpy_fromio(ctx->registers,
> +			      mali_c55->base + config_space_addrs[cfg_space],
> +			      MALI_C55_CONFIG_SPACE_SIZE);
> +		return 0;
> +	}
> +}
> +
> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
> +			  enum mali_c55_config_spaces cfg_space)
> +{
> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> +
> +	if (mali_c55->channel) {
> +		return mali_c55_dma_write(ctx, cfg_space);
> +	} else {
> +		memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
> +			    ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
> +		return 0;
> +	}

Could you measure the time it typically takes to write the registers
using DMA compared to using memcpy_toio() ?

> +}
> +
> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
> +{
> +	return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);

I think it's too early to tell how multi-context support will look like.
I'm fine keeping mali_c55_get_active_context() as changing that would be
very intrusive (even if I think it will need to be changed), but the
list of contexts is neither the mechanism we'll use, nor something we
need now. Drop the list, embed the context in struct mali_c55, and
return the pointer to that single context from this function.

> +}
> +
> +static void mali_c55_remove_links(struct mali_c55 *mali_c55)
> +{
> +	unsigned int i;
> +
> +	media_entity_remove_links(&mali_c55->tpg.sd.entity);
> +	media_entity_remove_links(&mali_c55->isp.sd.entity);
> +
> +	for (i = 0; i < MALI_C55_NUM_RZRS; i++)
> +		media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
> +
> +	for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
> +		media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
> +}
> +
> +static int mali_c55_create_links(struct mali_c55 *mali_c55)
> +{
> +	int ret;
> +
> +	/* Test pattern generator to ISP */
> +	ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
> +				    &mali_c55->isp.sd.entity,
> +				    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
> +	if (ret) {
> +		dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
> +		goto err_remove_links;
> +	}
> +
> +	/* Full resolution resizer pipe. */
> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> +			MALI_C55_ISP_PAD_SOURCE,
> +			&mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
> +			MALI_C55_RZR_SINK_PAD,
> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> +	if (ret) {
> +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
> +		goto err_remove_links;
> +	}
> +
> +	/* Full resolution bypass. */
> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> +				    MALI_C55_ISP_PAD_SOURCE_BYPASS,
> +				    &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
> +				    MALI_C55_RZR_SINK_BYPASS_PAD,
> +				    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> +	if (ret) {
> +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
> +		goto err_remove_links;
> +	}
> +
> +	/* Resizer pipe to video capture nodes. */
> +	ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
> +			MALI_C55_RZR_SOURCE_PAD,
> +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
> +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> +	if (ret) {
> +		dev_err(mali_c55->dev,
> +			"failed to link FR resizer and video device\n");
> +		goto err_remove_links;
> +	}
> +
> +	/* The downscale pipe is an optional hardware block */
> +	if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
> +		ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> +			MALI_C55_ISP_PAD_SOURCE,
> +			&mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
> +			MALI_C55_RZR_SINK_PAD,
> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> +		if (ret) {
> +			dev_err(mali_c55->dev,
> +				"failed to link ISP and DS resizer\n");
> +			goto err_remove_links;
> +		}
> +
> +		ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
> +			MALI_C55_RZR_SOURCE_PAD,
> +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
> +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> +		if (ret) {
> +			dev_err(mali_c55->dev,
> +				"failed to link DS resizer and video device\n");
> +			goto err_remove_links;
> +		}
> +	}
> +
> +	return 0;
> +
> +err_remove_links:
> +	mali_c55_remove_links(mali_c55);
> +	return ret;
> +}
> +
> +static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
> +{
> +	mali_c55_unregister_tpg(mali_c55);
> +	mali_c55_unregister_isp(mali_c55);
> +	mali_c55_unregister_resizers(mali_c55);
> +	mali_c55_unregister_capture_devs(mali_c55);
> +}
> +
> +static int mali_c55_register_entities(struct mali_c55 *mali_c55)
> +{
> +	int ret;
> +
> +	ret = mali_c55_register_tpg(mali_c55);
> +	if (ret)
> +		return ret;
> +
> +	ret = mali_c55_register_isp(mali_c55);
> +	if (ret)
> +		goto err_unregister_entities;
> +
> +	ret = mali_c55_register_resizers(mali_c55);
> +	if (ret)
> +		goto err_unregister_entities;
> +
> +	ret = mali_c55_register_capture_devs(mali_c55);
> +	if (ret)
> +		goto err_unregister_entities;
> +
> +	ret = mali_c55_create_links(mali_c55);
> +	if (ret)
> +		goto err_unregister_entities;
> +
> +	return 0;
> +
> +err_unregister_entities:
> +	mali_c55_unregister_entities(mali_c55);
> +
> +	return ret;
> +}
> +
> +static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
> +{
> +	u32 product, version, revision, capabilities;
> +
> +	product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
> +	version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
> +	revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
> +
> +	dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
> +		 product, version, revision);
> +
> +	capabilities = mali_c55_read(mali_c55,
> +				     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
> +				     false);
> +	mali_c55->capabilities = (capabilities & 0xffff);
> +
> +	/* TODO: Might as well start some debugfs */

If it's just to expose the version and capabilities, I think that's
overkill. It's not needed for debug purpose (you can get it from the
kernel log already). debugfs isn't meant to be accessible in production,
so an application that would need access to the information wouldn't be
able to use it.

> +	dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);

Combine the two messages into one.

> +	return version;
> +}
> +
> +static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
> +{
> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> +	u32 curr_config, next_config;
> +
> +	curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
> +	curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
> +		      >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
> +	next_config = curr_config ^ 1;
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
> +	mali_c55_config_write(ctx, next_config ?
> +			      MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
> +}
> +
> +static irqreturn_t mali_c55_isr(int irq, void *context)
> +{
> +	struct device *dev = context;
> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> +	u32 interrupt_status;
> +	unsigned int i, j;
> +
> +	interrupt_status = mali_c55_read(mali_c55,
> +					 MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
> +					 false);
> +	if (!interrupt_status)
> +		return IRQ_NONE;
> +
> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
> +		       interrupt_status);
> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
> +
> +	for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
> +		if (!(interrupt_status & (1 << i)))
> +			continue;
> +
> +		switch (i) {
> +		case MALI_C55_IRQ_ISP_START:
> +			mali_c55_isp_queue_event_sof(mali_c55);
> +
> +			for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
> +				mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
> +
> +			mali_c55_swap_next_config(mali_c55);
> +
> +			break;
> +		case MALI_C55_IRQ_ISP_DONE:
> +			/*
> +			 * TODO: Where the ISP has no Pong config fitted, we'd
> +			 * have to do the mali_c55_swap_next_config() call here.
> +			 */
> +			break;
> +		case MALI_C55_IRQ_FR_Y_DONE:
> +			mali_c55_set_plane_done(
> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
> +				MALI_C55_PLANE_Y);
> +			break;
> +		case MALI_C55_IRQ_FR_UV_DONE:
> +			mali_c55_set_plane_done(
> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
> +				MALI_C55_PLANE_UV);
> +			break;
> +		case MALI_C55_IRQ_DS_Y_DONE:
> +			mali_c55_set_plane_done(
> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
> +				MALI_C55_PLANE_Y);
> +			break;
> +		case MALI_C55_IRQ_DS_UV_DONE:
> +			mali_c55_set_plane_done(
> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
> +				MALI_C55_PLANE_UV);
> +			break;
> +		default:
> +			/*
> +			 * Only the above interrupts are currently unmasked. If
> +			 * we receive anything else here then something weird
> +			 * has gone on.
> +			 */
> +			dev_err(dev, "masked interrupt %s triggered\n",
> +				mali_c55_interrupt_names[i]);
> +		}
> +	}
> +
> +	return IRQ_HANDLED;
> +}
> +
> +static int mali_c55_init_context(struct mali_c55 *mali_c55)
> +{
> +	struct mali_c55_ctx *ctx;
> +	int ret;
> +
> +	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
> +	if (!ctx) {
> +		dev_err(mali_c55->dev, "failed to allocate new context\n");

No need for an error message when memory allocation fails.

> +		return -ENOMEM;
> +	}
> +
> +	ctx->base = mali_c55->res->start;
> +	ctx->mali_c55 = mali_c55;
> +
> +	ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
> +				 GFP_KERNEL | GFP_DMA);
> +	if (!ctx->registers) {
> +		ret = -ENOMEM;
> +		goto err_free_ctx;
> +	}
> +
> +	/*
> +	 * The allocated memory is empty, we need to load the default
> +	 * register settings. We just read Ping; it's identical to Pong.
> +	 */
> +	ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
> +	if (ret)
> +		goto err_free_registers;
> +
> +	list_add_tail(&ctx->list, &mali_c55->contexts);
> +
> +	/*
> +	 * Some features of the ISP need to be disabled by default and only
> +	 * enabled at the same time as they're configured by a parameters buffer
> +	 */
> +
> +	/* Bypass the sqrt and square compression and expansion modules */
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
> +			     MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
> +			     MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
> +
> +	/* Bypass the temper module */
> +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
> +		       MALI_C55_REG_BYPASS_2_TEMPER);
> +
> +	/* Bypass the colour noise reduction  */
> +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
> +		       MALI_C55_REG_BYPASS_4_CNR);
> +
> +	/* Disable the sinter module */
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
> +			     MALI_C55_SINTER_ENABLE_MASK, 0x00);
> +
> +	/* Disable the RGB Gamma module for each output */
> +	mali_c55_write(
> +		mali_c55,
> +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
> +		0x00);
> +	mali_c55_write(
> +		mali_c55,
> +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
> +		0x00);
> +
> +	/* Disable the colour correction matrix */
> +	mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
> +
> +	return 0;
> +
> +err_free_registers:
> +	kfree(ctx->registers);
> +err_free_ctx:
> +	kfree(ctx);
> +
> +	return ret;
> +}
> +
> +static int mali_c55_runtime_resume(struct device *dev)
> +{
> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> +	int ret;
> +
> +	ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
> +				      mali_c55->clks);
> +	if (ret)
> +		dev_err(mali_c55->dev, "failed to enable clocks\n");
> +
> +	return ret;
> +}
> +
> +static int mali_c55_runtime_suspend(struct device *dev)
> +{
> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> +
> +	clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
> +	return 0;
> +}
> +
> +static const struct dev_pm_ops mali_c55_pm_ops = {
> +	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> +				pm_runtime_force_resume)
> +	SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
> +			   NULL)
> +};
> +
> +static int mali_c55_probe(struct platform_device *pdev)
> +{
> +	struct device *dev = &pdev->dev;
> +	struct mali_c55 *mali_c55;
> +	dma_cap_mask_t mask;
> +	u32 version;
> +	int ret;
> +	u32 val;
> +
> +	mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
> +	if (!mali_c55)
> +		return dev_err_probe(dev, -ENOMEM,
> +				     "failed to allocate memory\n");

		return -ENOMEM;

There's no need to print messages for memory allocation failures.

> +
> +	mali_c55->dev = dev;
> +	platform_set_drvdata(pdev, mali_c55);
> +
> +	mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
> +								&mali_c55->res);
> +	if (IS_ERR(mali_c55->base))
> +		return dev_err_probe(dev, PTR_ERR(mali_c55->base),
> +				     "failed to map IO memory\n");
> +
> +	ret = platform_get_irq(pdev, 0);
> +	if (ret < 0)
> +		return dev_err_probe(dev, ret, "failed to get interrupt num\n");

s/ num// or s/num/number/

> +
> +	ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
> +					mali_c55_isr, IRQF_ONESHOT,
> +					dev_driver_string(&pdev->dev),
> +					&pdev->dev);

Requested the IRQ should be done much lower, after you have initialized
everything, or an IRQ that would fire early would have really bad
consequences.

A comment to explain why you need a threaded interrupt handler would be
good. I assume it is due only to the need to transfer the registers
using DMA. I wonder if we should then split the interrupt handler in
two, with a non-threaded part for the operations that can run quickly,
and a threaded part for the reprogramming.

It may also be that we could just start the DMA transfer in the
non-threaded handler without waiting synchronously for it to complete.
That would be a bigger change, and would require checking race
conditions carefully. On the other hand, I'm a bit concerned about the
current implementation, have you tested what happens if the DMA transfer
takes too long to complete, and spans frame boundaries ? This part could
be addressed by a patch on top of this one.

> +	if (ret)
> +		return dev_err_probe(dev, ret, "failed to request irq\n");
> +
> +	for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
> +		mali_c55->clks[i].id = mali_c55_clk_names[i];
> +
> +	ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
> +	if (ret)
> +		return dev_err_probe(dev, ret, "failed to acquire clocks\n");
> +
> +	pm_runtime_enable(&pdev->dev);
> +
> +	ret = pm_runtime_resume_and_get(&pdev->dev);
> +	if (ret)
> +		goto err_pm_runtime_disable;
> +
> +	of_reserved_mem_device_init(dev);

I'd move this, the vb2_dma_contig_set_max_seg_size() call and the
dma_cap_* calls before pm_runtime_enable() as they don't need the device
to be powered.

> +	version = mali_c55_check_hwcfg(mali_c55);
> +	vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
> +
> +	/* Use "software only" context management. */
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);

You handle that in mali_c55_isp_start(), does the register have to be
set here too ?

> +
> +	dma_cap_zero(mask);
> +	dma_cap_set(DMA_MEMCPY, mask);
> +
> +	/*
> +	 * No error check, because we will just fallback on memcpy if there is
> +	 * no usable DMA channel on the system.
> +	 */
> +	mali_c55->channel = dma_request_channel(mask, NULL, NULL);
> +
> +	INIT_LIST_HEAD(&mali_c55->contexts);
> +	ret = mali_c55_init_context(mali_c55);
> +	if (ret)
> +		goto err_release_dma_channel;
> +

I'd move all the code from here ...

> +	mali_c55->media_dev.dev = dev;
> +	strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
> +		sizeof(mali_c55->media_dev.model));
> +	mali_c55->media_dev.hw_revision = version;
> +
> +	media_device_init(&mali_c55->media_dev);
> +	ret = media_device_register(&mali_c55->media_dev);
> +	if (ret)
> +		goto err_cleanup_media_device;
> +
> +	mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
> +	ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
> +	if (ret) {
> +		dev_err(dev, "failed to register V4L2 device\n");
> +		goto err_unregister_media_device;
> +	};
> +
> +	ret = mali_c55_register_entities(mali_c55);
> +	if (ret) {
> +		dev_err(dev, "failed to register entities\n");
> +		goto err_unregister_v4l2_device;
> +	}

... to here to a separate function, or maybe fold it all in
mali_c55_register_entities() (which should the be renamed). Same thing
for the cleanup code.

> +
> +	/* Set safe stop to ensure we're in a non-streaming state */
> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> +		       MALI_C55_INPUT_SAFE_STOP);
> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> +
> +	/*
> +	 * We're ready to process interrupts. Clear any that are set and then
> +	 * unmask them for processing.
> +	 */
> +	mali_c55_write(mali_c55, 0x30, 0xffffffff);
> +	mali_c55_write(mali_c55, 0x34, 0xffffffff);
> +	mali_c55_write(mali_c55, 0x40, 0x01);
> +	mali_c55_write(mali_c55, 0x40, 0x00);

Please replace the register addresses with macros.

> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);

The value should use the interrupt bits macros.

> +
> +	pm_runtime_put(&pdev->dev);

Once power gets cut, the registers your programmed above may be lost. I
think you should programe them in the runtime PM resume handler.

> +
> +	return 0;
> +
> +err_unregister_v4l2_device:
> +	v4l2_device_unregister(&mali_c55->v4l2_dev);
> +err_unregister_media_device:
> +	media_device_unregister(&mali_c55->media_dev);
> +err_cleanup_media_device:
> +	media_device_cleanup(&mali_c55->media_dev);
> +err_release_dma_channel:
> +	dma_release_channel(mali_c55->channel);
> +err_pm_runtime_disable:
> +	pm_runtime_disable(&pdev->dev);
> +
> +	return ret;
> +}
> +
> +static void mali_c55_remove(struct platform_device *pdev)
> +{
> +	struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
> +	struct mali_c55_ctx *ctx, *tmp;
> +
> +	list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
> +		list_del(&ctx->list);
> +		kfree(ctx->registers);
> +		kfree(ctx);
> +	}
> +
> +	mali_c55_remove_links(mali_c55);
> +	mali_c55_unregister_entities(mali_c55);
> +	v4l2_device_put(&mali_c55->v4l2_dev);
> +	media_device_unregister(&mali_c55->media_dev);
> +	media_device_cleanup(&mali_c55->media_dev);
> +	dma_release_channel(mali_c55->channel);
> +}
> +
> +static const struct of_device_id mali_c55_of_match[] = {
> +	{ .compatible = "arm,mali-c55", },
> +	{},

	{ /* Sentinel */ },

> +};
> +MODULE_DEVICE_TABLE(of, mali_c55_of_match);
> +
> +static struct platform_driver mali_c55_driver = {
> +	.driver = {
> +		.name = "mali-c55",
> +		.of_match_table = of_match_ptr(mali_c55_of_match),

Drop of_match_ptr().

> +		.pm = &mali_c55_pm_ops,
> +	},
> +	.probe = mali_c55_probe,
> +	.remove_new = mali_c55_remove,
> +};
> +
> +module_platform_driver(mali_c55_driver);

Blank line.

> +MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
> +MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
> +MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
> +MODULE_LICENSE("GPL");
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> new file mode 100644
> index 000000000000..ea8b7b866e7a
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> @@ -0,0 +1,611 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ARM Mali-C55 ISP Driver - Image signal processor
> + *
> + * Copyright (C) 2024 Ideas on Board Oy
> + */
> +
> +#include <linux/delay.h>
> +#include <linux/iopoll.h>
> +#include <linux/property.h>
> +#include <linux/string.h>
> +
> +#include <media/media-entity.h>
> +#include <media/v4l2-common.h>
> +#include <media/v4l2-event.h>
> +#include <media/v4l2-mc.h>
> +#include <media/v4l2-subdev.h>
> +
> +#include "mali-c55-common.h"
> +#include "mali-c55-registers.h"
> +
> +static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
> +	{
> +		.code = MEDIA_BUS_FMT_SRGGB20_1X20,
> +		.order = MALI_C55_BAYER_ORDER_RGGB,
> +		.encoding = V4L2_PIXEL_ENC_BAYER,
> +	},
> +	{
> +		.code = MEDIA_BUS_FMT_SGRBG20_1X20,
> +		.order = MALI_C55_BAYER_ORDER_GRBG,
> +		.encoding = V4L2_PIXEL_ENC_BAYER,
> +	},
> +	{
> +		.code = MEDIA_BUS_FMT_SGBRG20_1X20,
> +		.order = MALI_C55_BAYER_ORDER_GBRG,
> +		.encoding = V4L2_PIXEL_ENC_BAYER,
> +	},
> +	{
> +		.code = MEDIA_BUS_FMT_SBGGR20_1X20,
> +		.order = MALI_C55_BAYER_ORDER_BGGR,
> +		.encoding = V4L2_PIXEL_ENC_BAYER,
> +	},
> +	{
> +		.code = MEDIA_BUS_FMT_RGB202020_1X60,
> +		.order = 0, /* Not relevant for this format */
> +		.encoding = V4L2_PIXEL_ENC_RGB,
> +	}
> +	/*
> +	 * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
> +	 * also support YUV input from a sensor passed-through to the output. At
> +	 * present we have no mechanism to test that though so it may have to
> +	 * wait a while...
> +	 */
> +};
> +
> +const struct mali_c55_isp_fmt *
> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
> +{
> +	if (!fmt)
> +		fmt = &mali_c55_isp_fmts[0];
> +	else
> +		fmt++;
> +
> +	for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
> +		return fmt;

That's peculiar.

	if (!fmt)
		fmt = &mali_c55_isp_fmts[0];
	else if (fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)])
		return ++fmt;
	else
		return NULL;

> +
> +	return NULL;
> +}
> +
> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
> +{
> +	const struct mali_c55_isp_fmt *isp_fmt;
> +
> +	for_each_mali_isp_fmt(isp_fmt) {

I would open-code the loop instead of using the macro, like you do
below. It will be more efficient.

> +		if (isp_fmt->code == mbus_code)
> +			return true;
> +	}
> +
> +	return false;
> +}
> +
> +static const struct mali_c55_isp_fmt *
> +mali_c55_isp_get_mbus_config_by_code(u32 code)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
> +		if (mali_c55_isp_fmts[i].code == code)
> +			return &mali_c55_isp_fmts[i];
> +	}
> +
> +	return NULL;
> +}
> +
> +static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
> +{
> +	u32 val;
> +
> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);

	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
		       MALI_C55_INPUT_SAFE_STOP);

> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> +}
> +
> +static int mali_c55_isp_start(struct mali_c55 *mali_c55)
> +{
> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> +	const struct mali_c55_isp_fmt *cfg;
> +	struct v4l2_mbus_framefmt *format;

const

> +	struct v4l2_subdev_state *state;
> +	struct v4l2_rect *crop;

const

> +	struct v4l2_subdev *sd;
> +	u32 val;
> +	int ret;
> +
> +	sd = &mali_c55->isp.sd;

Assign when declaring the variable.

> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
> +
> +	/* Apply input windowing */
> +	state = v4l2_subdev_get_locked_active_state(sd);

Using .enable_streams() (see below) you'll get this for free.

> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> +	format = v4l2_subdev_state_get_format(state,
> +					      MALI_C55_ISP_PAD_SINK_VIDEO);
> +	cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
> +
> +	mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
> +		       MALI_C55_HC_START(crop->left));
> +	mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
> +		       MALI_C55_HC_SIZE(crop->width));
> +	mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
> +		       MALI_C55_VC_START(crop->top) |
> +		       MALI_C55_VC_SIZE(crop->height));
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
> +			     MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
> +			     MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
> +			     MALI_C55_BAYER_ORDER_MASK, cfg->order);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
> +			     MALI_C55_INPUT_WIDTH_MASK,
> +			     MALI_C55_INPUT_WIDTH_20BIT);
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
> +			     cfg->encoding == V4L2_PIXEL_ENC_RGB ?
> +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
> +
> +	ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
> +	if (ret) {
> +		dev_err(mali_c55->dev, "failed to DMA config\n");
> +		return ret;
> +	}
> +
> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> +		       MALI_C55_INPUT_SAFE_START);
> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);

Should you return an error in case of timeout ?

> +
> +	return 0;
> +}
> +
> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)

Why is this not handled wired to .s_stream() ? Or better,
.enable_streams() and .disable_streams().

> +{
> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> +	struct v4l2_subdev *sd;
> +
> +	if (isp->remote_src) {
> +		sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
> +		v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
> +	}
> +	isp->remote_src = NULL;
> +
> +	mali_c55_isp_stop(mali_c55);
> +}
> +
> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
> +{
> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> +	struct media_pad *sink_pad;
> +	struct v4l2_subdev *sd;
> +	int ret;
> +
> +	sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
> +	isp->remote_src = media_pad_remote_pad_unique(sink_pad);
> +	if (IS_ERR(isp->remote_src)) {

If you set the MUST_CONNECT flag for the MALI_C55_ISP_PAD_SINK_VIDEO pad
I think you can drop this check.

> +		dev_err(mali_c55->dev, "Failed to get source for ISP\n");
> +		return PTR_ERR(isp->remote_src);
> +	}
> +
> +	sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
> +
> +	isp->frame_sequence = 0;
> +	ret = mali_c55_isp_start(mali_c55);
> +	if (ret) {
> +		dev_err(mali_c55->dev, "Failed to start ISP\n");
> +		isp->remote_src = NULL;
> +		return ret;
> +	}
> +
> +	/*
> +	 * We only support a single input stream, so we can just enable the 1st
> +	 * entry in the streams mask.
> +	 */
> +	ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
> +	if (ret) {
> +		dev_err(mali_c55->dev, "Failed to start ISP source\n");
> +		mali_c55_isp_stop(mali_c55);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
> +				       struct v4l2_subdev_state *state,
> +				       struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	/*
> +	 * Only the internal RGB processed format is allowed on the regular
> +	 * processing source pad.
> +	 */
> +	if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
> +		if (code->index)
> +			return -EINVAL;
> +
> +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> +		return 0;
> +	}
> +
> +	/* On the sink and bypass pads all the supported formats are allowed. */
> +	if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
> +		return -EINVAL;
> +
> +	code->code = mali_c55_isp_fmts[code->index].code;
> +
> +	return 0;
> +}
> +
> +static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
> +					struct v4l2_subdev_state *state,
> +					struct v4l2_subdev_frame_size_enum *fse)
> +{
> +	const struct mali_c55_isp_fmt *cfg;
> +
> +	if (fse->index > 0)
> +		return -EINVAL;
> +
> +	/*
> +	 * Only the internal RGB processed format is allowed on the regular
> +	 * processing source pad.
> +	 *
> +	 * On the sink and bypass pads all the supported formats are allowed.
> +	 */
> +	if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
> +		if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
> +			return -EINVAL;
> +	} else {
> +		cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
> +		if (!cfg)
> +			return -EINVAL;
> +	}
> +
> +	fse->min_width = MALI_C55_MIN_WIDTH;
> +	fse->min_height = MALI_C55_MIN_HEIGHT;
> +	fse->max_width = MALI_C55_MAX_WIDTH;
> +	fse->max_height = MALI_C55_MAX_HEIGHT;
> +
> +	return 0;
> +}
> +
> +static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
> +				struct v4l2_subdev_state *state,
> +				struct v4l2_subdev_format *format)
> +{
> +	struct v4l2_mbus_framefmt *fmt = &format->format;
> +	struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
> +	const struct mali_c55_isp_fmt *cfg;
> +	struct v4l2_rect *crop;
> +
> +	/*
> +	 * Disallow set_fmt on the source pads; format is fixed and the sizes
> +	 * are the result of applying the sink crop rectangle to the sink
> +	 * format.
> +	 */
> +	if (format->pad)

	if (format->pad != MALI_C55_ISP_PAD_SINK_VIDEO)

> +		return v4l2_subdev_get_fmt(sd, state, format);
> +
> +	cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
> +	if (!cfg)
> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> +	fmt->field = V4L2_FIELD_NONE;

Do you intentionally allow the colorspace fields to be overwritten to
any value ? rkisp1_isp_set_src_fmt() and rkisp1_isp_set_sink_fmt() may
show you how this could be handled.

> +
> +	/*
> +	 * Clamp sizes in the accepted limits and clamp the crop rectangle in
> +	 * the new sizes.
> +	 */
> +	clamp_t(unsigned int, fmt->width,
> +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> +	clamp_t(unsigned int, fmt->width,
> +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);

clamp_t() returns a value, which you ignore. Those are no-ops. You meant

	fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
			     MALI_C55_MAX_WIDTH);
	fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
			      MALI_C55_MAX_HEIGHT);

Same for every use of clamp_t() through the whole driver.

Also, do you need clamp_t() ? I think all values are unsigned int, you
can use clamp().

Are there any alignment constraints, such a multiples of two for bayer
formats ? Same in all the other locations where applicable.

> +
> +	sink_fmt = v4l2_subdev_state_get_format(state,
> +						MALI_C55_ISP_PAD_SINK_VIDEO);
> +	*sink_fmt = *fmt;
> +
> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> +	crop->left = 0;
> +	crop->top = 0;
> +	crop->width = fmt->width;
> +	crop->height = fmt->height;
> +
> +	/*
> +	 * Propagate format to source pads. On the 'regular' output pad use
> +	 * the internal RGB processed format, while on the bypass pad simply
> +	 * replicate the ISP sink format. The sizes on both pads are the same as
> +	 * the ISP sink crop rectangle.
> +	 */

Colorspace information will need to be propagated too.

> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
> +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> +	src_fmt->width = crop->width;
> +	src_fmt->height = crop->height;
> +
> +	src_fmt = v4l2_subdev_state_get_format(state,
> +					       MALI_C55_ISP_PAD_SOURCE_BYPASS);
> +	src_fmt->code = fmt->code;
> +	src_fmt->width = crop->width;
> +	src_fmt->height = crop->height;
> +
> +	return 0;
> +}
> +
> +static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
> +				      struct v4l2_subdev_state *state,
> +				      struct v4l2_subdev_selection *sel)
> +{
> +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)

	sel->pad != MALI_C55_ISP_PAD_SINK_VIDEO

> +		return -EINVAL;
> +
> +	sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> +
> +	return 0;
> +}
> +
> +static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
> +				      struct v4l2_subdev_state *state,
> +				      struct v4l2_subdev_selection *sel)
> +{
> +	struct v4l2_mbus_framefmt *src_fmt;
> +	struct v4l2_mbus_framefmt *fmt;

const

> +	struct v4l2_rect *crop;
> +
> +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)

Ditto.

> +		return -EINVAL;
> +
> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> +
> +	clamp_t(unsigned int, sel->r.left, 0, fmt->width);
> +	clamp_t(unsigned int, sel->r.top, 0, fmt->height);
> +	clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
> +		fmt->width - sel->r.left);
> +	clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
> +		fmt->height - sel->r.top);
> +
> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> +	*crop = sel->r;
> +
> +	/* Propagate the crop rectangle sizes to the source pad format. */
> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
> +	src_fmt->width = crop->width;
> +	src_fmt->height = crop->height;

Can you confirm that cropping doesn't affect the bypass path ? And maybe
add a comment to mention it.

> +
> +	return 0;
> +}
> +
> +static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
> +	.enum_mbus_code		= mali_c55_isp_enum_mbus_code,
> +	.enum_frame_size	= mali_c55_isp_enum_frame_size,
> +	.get_fmt		= v4l2_subdev_get_fmt,
> +	.set_fmt		= mali_c55_isp_set_fmt,
> +	.get_selection		= mali_c55_isp_get_selection,
> +	.set_selection		= mali_c55_isp_set_selection,
> +	.link_validate		= v4l2_subdev_link_validate_default,
> +};
> +
> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
> +{
> +	struct v4l2_event event = {
> +		.type = V4L2_EVENT_FRAME_SYNC,
> +	};
> +
> +	event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
> +	v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
> +}
> +
> +static int
> +mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
> +			     struct v4l2_event_subscription *sub)
> +{
> +	if (sub->type != V4L2_EVENT_FRAME_SYNC)
> +		return -EINVAL;
> +
> +	/* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
> +	if (sub->id != 0)
> +		return -EINVAL;
> +
> +	return v4l2_event_subscribe(fh, sub, 0, NULL);
> +}
> +
> +static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
> +	.subscribe_event = mali_c55_isp_subscribe_event,
> +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
> +};
> +
> +static const struct v4l2_subdev_ops mali_c55_isp_ops = {
> +	.pad	= &mali_c55_isp_pad_ops,
> +	.core	= &mali_c55_isp_core_ops,
> +};
> +
> +static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
> +				   struct v4l2_subdev_state *sd_state)

You name this variable state in every other subdev operation handler.

> +{
> +	struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> +	struct v4l2_rect *in_crop;
> +
> +	sink_fmt = v4l2_subdev_state_get_format(sd_state,
> +						MALI_C55_ISP_PAD_SINK_VIDEO);
> +	src_fmt = v4l2_subdev_state_get_format(sd_state,
> +					       MALI_C55_ISP_PAD_SOURCE);
> +	in_crop = v4l2_subdev_state_get_crop(sd_state,
> +					     MALI_C55_ISP_PAD_SINK_VIDEO);
> +
> +	sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
> +	sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
> +	sink_fmt->field = V4L2_FIELD_NONE;
> +	sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;

You should initialize the colorspace fields too. Same below.

> +
> +	*v4l2_subdev_state_get_format(sd_state,
> +			      MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
> +
> +	src_fmt->width = MALI_C55_DEFAULT_WIDTH;
> +	src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
> +	src_fmt->field = V4L2_FIELD_NONE;
> +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> +
> +	in_crop->top = 0;
> +	in_crop->left = 0;
> +	in_crop->width = MALI_C55_DEFAULT_WIDTH;
> +	in_crop->height = MALI_C55_DEFAULT_HEIGHT;
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
> +	.init_state = mali_c55_isp_init_state,
> +};
> +
> +static const struct media_entity_operations mali_c55_isp_media_ops = {
> +	.link_validate		= v4l2_subdev_link_validate,

	.link_validate = v4l2_subdev_link_validate,

to match mali_c55_isp_internal_ops.

> +};
> +
> +static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
> +				       struct v4l2_subdev *subdev,
> +				       struct v4l2_async_connection *asc)
> +{
> +	struct mali_c55_isp *isp = container_of(notifier,
> +						struct mali_c55_isp, notifier);
> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> +	struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
> +	int ret;
> +
> +	/*
> +	 * By default we'll flag this link enabled and the TPG disabled, but
> +	 * no immutable flag because we need to be able to switch between the
> +	 * two.
> +	 */
> +	ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
> +					      MEDIA_LNK_FL_ENABLED);
> +	if (ret)
> +		dev_err(mali_c55->dev, "failed to create link for %s\n",
> +			subdev->name);
> +
> +	return ret;
> +}
> +
> +static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
> +{
> +	struct mali_c55_isp *isp = container_of(notifier,
> +						struct mali_c55_isp, notifier);
> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> +
> +	return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
> +}
> +
> +static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
> +	.bound = mali_c55_isp_notifier_bound,
> +	.complete = mali_c55_isp_notifier_complete,
> +};
> +
> +static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
> +{
> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> +	struct v4l2_async_connection *asc;
> +	struct fwnode_handle *ep;
> +	int ret;
> +
> +	v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
> +
> +	/*
> +	 * The ISP should have a single endpoint pointing to some flavour of
> +	 * CSI-2 receiver...but for now at least we do want everything to work
> +	 * normally even with no sensors connected, as we have the TPG. If we
> +	 * don't find a sensor just warn and return success.
> +	 */
> +	ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
> +					     0, 0, 0);
> +	if (!ep) {
> +		dev_warn(mali_c55->dev, "no local endpoint found\n");
> +		return 0;
> +	}
> +
> +	asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
> +					      struct v4l2_async_connection);
> +	if (IS_ERR(asc)) {
> +		dev_err(mali_c55->dev, "failed to add remote fwnode\n");
> +		ret = PTR_ERR(asc);
> +		goto err_put_ep;
> +	}
> +
> +	isp->notifier.ops = &mali_c55_isp_notifier_ops;
> +	ret = v4l2_async_nf_register(&isp->notifier);
> +	if (ret) {
> +		dev_err(mali_c55->dev, "failed to register notifier\n");
> +		goto err_cleanup_nf;
> +	}
> +
> +	fwnode_handle_put(ep);
> +
> +	return 0;
> +
> +err_cleanup_nf:
> +	v4l2_async_nf_cleanup(&isp->notifier);
> +err_put_ep:
> +	fwnode_handle_put(ep);
> +
> +	return ret;
> +}
> +
> +int mali_c55_register_isp(struct mali_c55 *mali_c55)
> +{
> +	struct mali_c55_isp *isp = &mali_c55->isp;
> +	struct v4l2_subdev *sd = &isp->sd;
> +	int ret;
> +
> +	isp->mali_c55 = mali_c55;
> +
> +	v4l2_subdev_init(sd, &mali_c55_isp_ops);
> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> +	sd->entity.ops = &mali_c55_isp_media_ops;
> +	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
> +	sd->internal_ops = &mali_c55_isp_internal_ops;
> +	strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
> +
> +	isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;

The MUST_CONNECT flag would make sense here.

> +	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> +	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
> +
> +	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
> +				     isp->pads);
> +	if (ret)
> +		return ret;
> +
> +	ret = v4l2_subdev_init_finalize(sd);
> +	if (ret)
> +		goto err_cleanup_media_entity;
> +
> +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> +	if (ret)
> +		goto err_cleanup_subdev;
> +
> +	ret = mali_c55_isp_parse_endpoint(isp);
> +	if (ret)
> +		goto err_cleanup_subdev;

As noted elsewhere, I think this belongs to mali-c55-core.c.

> +
> +	mutex_init(&isp->lock);

This lock is used in mali-c55-capture.c only, that seems weird.

> +
> +	return 0;
> +
> +err_cleanup_subdev:
> +	v4l2_subdev_cleanup(sd);
> +err_cleanup_media_entity:
> +	media_entity_cleanup(&sd->entity);
> +	isp->mali_c55 = NULL;
> +
> +	return ret;
> +}
> +
> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
> +{
> +	struct mali_c55_isp *isp = &mali_c55->isp;
> +
> +	if (!isp->mali_c55)
> +		return;
> +
> +	mutex_destroy(&isp->lock);
> +	v4l2_async_nf_unregister(&isp->notifier);
> +	v4l2_async_nf_cleanup(&isp->notifier);
> +	v4l2_device_unregister_subdev(&isp->sd);
> +	v4l2_subdev_cleanup(&isp->sd);
> +	media_entity_cleanup(&isp->sd.entity);
> +}
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> new file mode 100644
> index 000000000000..cb27abde2aa5
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> @@ -0,0 +1,258 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * ARM Mali-C55 ISP Driver - Register definitions
> + *
> + * Copyright (C) 2024 Ideas on Board Oy
> + */
> +
> +#ifndef _MALI_C55_REGISTERS_H
> +#define _MALI_C55_REGISTERS_H
> +
> +#include <linux/bits.h>
> +
> +/* ISP Common 0x00000 - 0x000ff */
> +
> +#define MALI_C55_REG_API				0x00000
> +#define MALI_C55_REG_PRODUCT				0x00004
> +#define MALI_C55_REG_VERSION				0x00008
> +#define MALI_C55_REG_REVISION				0x0000c
> +#define MALI_C55_REG_PULSE_MODE				0x0003c
> +#define MALI_C55_REG_INPUT_MODE_REQUEST			0x0009c
> +#define MALI_C55_INPUT_SAFE_STOP			0x00
> +#define MALI_C55_INPUT_SAFE_START			0x01
> +#define MALI_C55_REG_MODE_STATUS			0x000a0
> +#define MALI_C55_REG_INTERRUPT_MASK_VECTOR		0x00030
> +#define MALI_C55_INTERRUPT_MASK_ALL			GENMASK(31, 0)
> +
> +#define MALI_C55_REG_GLOBAL_MONITOR			0x00050
> +
> +#define MALI_C55_REG_GEN_VIDEO				0x00080
> +#define MALI_C55_REG_GEN_VIDEO_ON_MASK			BIT(0)
> +#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK		BIT(1)
> +#define MALI_C55_REG_GEN_PREFETCH_MASK			GENMASK(31, 16)
> +
> +#define MALI_C55_REG_MCU_CONFIG				0x00020
> +#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK		BIT(0)

#define MALI_C55_REG_MCU_CONFIG_OVERRIDE		BIT(0)

Same in other places where applicable.

> +#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK		BIT(1)
> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PING		BIT(1)
> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG		0x00
> +#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK		BIT(8)
> +#define MALI_C55_REG_PING_PONG_READ			0x00024
> +#define MALI_C55_REG_PING_PONG_READ_MASK		BIT(2)
> +
> +#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR		0x00034
> +#define MALI_C55_REG_INTERRUPT_CLEAR			0x00040
> +#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR		0x00044
> +#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS		0x00068
> +#define MALI_C55_GPS_PONG_FITTED			BIT(0)
> +#define MALI_C55_GPS_WDR_FITTED				BIT(1)
> +#define MALI_C55_GPS_COMPRESSION_FITTED			BIT(2)
> +#define MALI_C55_GPS_TEMPER_FITTED			BIT(3)
> +#define MALI_C55_GPS_SINTER_LITE_FITTED			BIT(4)
> +#define MALI_C55_GPS_SINTER_FITTED			BIT(5)
> +#define MALI_C55_GPS_IRIDIX_LTM_FITTED			BIT(6)
> +#define MALI_C55_GPS_IRIDIX_GTM_FITTED			BIT(7)
> +#define MALI_C55_GPS_CNR_FITTED				BIT(8)
> +#define MALI_C55_GPS_FRSCALER_FITTED			BIT(9)
> +#define MALI_C55_GPS_DS_PIPE_FITTED			BIT(10)
> +
> +#define MALI_C55_REG_BLANKING				0x00084
> +#define MALI_C55_REG_HBLANK_MASK			GENMASK(15, 0)
> +#define MALI_C55_REG_VBLANK_MASK			GENMASK(31, 16)
> +
> +#define MALI_C55_REG_HC_START				0x00088
> +#define MALI_C55_HC_START(h)				(((h) & 0xffff) << 16)
> +#define MALI_C55_REG_HC_SIZE				0x0008c
> +#define MALI_C55_HC_SIZE(h)				((h) & 0xffff)
> +#define MALI_C55_REG_VC_START_SIZE			0x00094
> +#define MALI_C55_VC_START(v)				((v) & 0xffff)
> +#define MALI_C55_VC_SIZE(v)				(((v) & 0xffff) << 16)
> +
> +/* Ping/Pong Configuration Space */
> +#define MALI_C55_REG_BASE_ADDR				0x18e88
> +#define MALI_C55_REG_BYPASS_0				0x18eac
> +#define MALI_C55_REG_BYPASS_0_VIDEO_TEST		BIT(0)
> +#define MALI_C55_REG_BYPASS_0_INPUT_FMT			BIT(1)
> +#define MALI_C55_REG_BYPASS_0_DECOMPANDER		BIT(2)
> +#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR		BIT(3)
> +#define MALI_C55_REG_BYPASS_0_GAIN_WDR			BIT(4)
> +#define MALI_C55_REG_BYPASS_0_FRAME_STITCH		BIT(5)
> +#define MALI_C55_REG_BYPASS_1				0x18eb0
> +#define MALI_C55_REG_BYPASS_1_DIGI_GAIN			BIT(0)
> +#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS		BIT(1)
> +#define MALI_C55_REG_BYPASS_1_FE_SQRT			BIT(2)
> +#define MALI_C55_REG_BYPASS_1_RAW_FE			BIT(3)
> +#define MALI_C55_REG_BYPASS_2				0x18eb8
> +#define MALI_C55_REG_BYPASS_2_SINTER			BIT(0)
> +#define MALI_C55_REG_BYPASS_2_TEMPER			BIT(1)
> +#define MALI_C55_REG_BYPASS_3				0x18ebc
> +#define MALI_C55_REG_BYPASS_3_SQUARE_BE			BIT(0)
> +#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH	BIT(1)
> +#define MALI_C55_REG_BYPASS_3_MESH_SHADING		BIT(3)
> +#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE		BIT(4)
> +#define MALI_C55_REG_BYPASS_3_IRIDIX			BIT(5)
> +#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN		BIT(6)
> +#define MALI_C55_REG_BYPASS_4				0x18ec0
> +#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB		BIT(1)
> +#define MALI_C55_REG_BYPASS_4_PF_CORRECTION		BIT(3)
> +#define MALI_C55_REG_BYPASS_4_CCM			BIT(4)
> +#define MALI_C55_REG_BYPASS_4_CNR			BIT(5)
> +#define MALI_C55_REG_FR_BYPASS				0x18ec4
> +#define MALI_C55_REG_DS_BYPASS				0x18ec8
> +#define MALI_C55_BYPASS_CROP				BIT(0)
> +#define MALI_C55_BYPASS_SCALER				BIT(1)
> +#define MALI_C55_BYPASS_GAMMA_RGB			BIT(2)
> +#define MALI_C55_BYPASS_SHARPEN				BIT(3)
> +#define MALI_C55_BYPASS_CS_CONV				BIT(4)
> +#define MALI_C55_REG_ISP_RAW_BYPASS			0x18ecc
> +#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK		BIT(0)
> +#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK		GENMASK(9, 8)
> +#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS		2
> +#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS		1
> +#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE		BIT(1)
> +#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS		BIT(0)
> +
> +#define MALI_C55_REG_ACTIVE_WIDTH_MASK			0xffff
> +#define MALI_C55_REG_ACTIVE_HEIGHT_MASK			0xffff0000
> +#define MALI_C55_REG_BAYER_ORDER			0x18e8c
> +#define MALI_C55_BAYER_ORDER_MASK			GENMASK(1, 0)
> +#define MALI_C55_REG_TPG_CH0				0x18ed8
> +#define MALI_C55_TEST_PATTERN_ON_OFF			BIT(0)
> +#define MALI_C55_TEST_PATTERN_RGB_MASK			BIT(1)
> +#define MALI_C55_REG_TPG_R_BACKGROUND			0x18ee0
> +#define MALI_C55_REG_TPG_G_BACKGROUND			0x18ee4
> +#define MALI_C55_REG_TPG_B_BACKGROUND			0x18ee8
> +#define MALI_C55_TPG_BACKGROUND_MAX			0xfffff
> +#define MALI_C55_REG_INPUT_WIDTH			0x18f98
> +#define MALI_C55_INPUT_WIDTH_MASK			GENMASK(18, 16)
> +#define MALI_C55_INPUT_WIDTH_8BIT			0
> +#define MALI_C55_INPUT_WIDTH_10BIT			1
> +#define MALI_C55_INPUT_WIDTH_12BIT			2
> +#define MALI_C55_INPUT_WIDTH_14BIT			3
> +#define MALI_C55_INPUT_WIDTH_16BIT			4
> +#define MALI_C55_INPUT_WIDTH_20BIT			5
> +#define MALI_C55_REG_SPACE_SIZE				0x4000
> +#define MALI_C55_REG_CONFIG_SPACES_OFFSET		0x0ab6c
> +#define MALI_C55_CONFIG_SPACE_SIZE			0x1231c
> +
> +#define MALI_C55_REG_SINTER_CONFIG			0x19348
> +#define MALI_C55_SINTER_VIEW_FILTER_MASK		GENMASK(1, 0)
> +#define MALI_C55_SINTER_SCALE_MODE_MASK			GENMASK(3, 2)
> +#define MALI_C55_SINTER_ENABLE_MASK			BIT(4)
> +#define MALI_C55_SINTER_FILTER_SELECT_MASK		BIT(5)
> +#define MALI_C55_SINTER_INT_SELECT_MASK			BIT(6)
> +#define MALI_C55_SINTER_RM_ENABLE_MASK			BIT(7)
> +
> +/* Colour Correction Matrix Configuration */
> +#define MALI_C55_REG_CCM_ENABLE				0x1b07c
> +#define MALI_C55_CCM_ENABLE_MASK			BIT(0)
> +#define MALI_C55_REG_CCM_COEF_R_R			0x1b080
> +#define MALI_C55_REG_CCM_COEF_R_G			0x1b084
> +#define MALI_C55_REG_CCM_COEF_R_B			0x1b088
> +#define MALI_C55_REG_CCM_COEF_G_R			0x1b090
> +#define MALI_C55_REG_CCM_COEF_G_G			0x1b094
> +#define MALI_C55_REG_CCM_COEF_G_B			0x1b098
> +#define MALI_C55_REG_CCM_COEF_B_R			0x1b0a0
> +#define MALI_C55_REG_CCM_COEF_B_G			0x1b0a4
> +#define MALI_C55_REG_CCM_COEF_B_B			0x1b0a8
> +#define MALI_C55_CCM_COEF_MASK				GENMASK(12, 0)
> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R			0x1b0b0
> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G			0x1b0b4
> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B			0x1b0b8
> +#define MALI_C55_CCM_ANTIFOG_GAIN_MASK			GENMASK(11, 0)
> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R		0x1b0c0
> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G		0x1b0c4
> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B		0x1b0c8
> +#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK		GENMASK(11, 0)
> +
> +/*
> + * The Mali-C55 ISP has up to two output pipes; known as full resolution and
> + * down scaled. The register space for these is laid out identically, but offset
> + * by 372 bytes.
> + */
> +#define MALI_C55_CAP_DEV_FR_REG_OFFSET		0x0
> +#define MALI_C55_CAP_DEV_DS_REG_OFFSET		0x174
> +
> +#define MALI_C55_REG_CS_CONV_CONFIG(offset)		(0x1c098 + (offset))
> +#define MALI_C55_CS_CONV_MATRIX_MASK			BIT(0)
> +#define MALI_C55_CS_CONV_FILTER_MASK			BIT(1)
> +#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK		BIT(2)
> +#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK		BIT(3)
> +#define MALI_C55_REG_Y_WRITER_MODE(offset)		(0x1c0ec + (offset))
> +#define MALI_C55_REG_UV_WRITER_MODE(offset)		(0x1c144 + (offset))
> +#define MALI_C55_WRITER_MODE_MASK			GENMASK(4, 0)
> +#define MALI_C55_WRITER_SUBMODE_MASK			GENMASK(7, 6)
> +#define MALI_C55_WRITER_FRAME_WRITE_MASK		BIT(9)
> +#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset)		(0x1c0f0 + (offset))
> +#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset)		(0x1c148 + (offset))
> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)		((w) << 0)
> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)		((h) << 16)
> +#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset)	(0x1c0f4 + (offset))
> +#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset)	(0x1c108 + (offset))
> +#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
> +#define MALI_C55_REG_Y_WRITER_BANKS_RESTART		BIT(3)
> +#define MALI_C55_REG_Y_WRITER_OFFSET(offset)		(0x1c10c + (offset))
> +#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset)	(0x1c14c + (offset))
> +#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset)	(0x1c160 + (offset))
> +#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
> +#define MALI_C55_REG_UV_WRITER_BANKS_RESTART		BIT(3)
> +#define MALI_C55_REG_UV_WRITER_OFFSET(offset)		(0x1c164 + (offset))
> +
> +#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
> +#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE		0x18edc
> +
> +#define MALI_C55_REG_CROP_EN(offset)			(0x1c028 + (offset))
> +#define MALI_C55_CROP_ENABLE				BIT(0)
> +#define MALI_C55_REG_CROP_X_START(offset)		(0x1c02c + (offset))
> +#define MALI_C55_REG_CROP_Y_START(offset)		(0x1c030 + (offset))
> +#define MALI_C55_REG_CROP_X_SIZE(offset)		(0x1c034 + (offset))
> +#define MALI_C55_REG_CROP_Y_SIZE(offset)		(0x1c038 + (offset))
> +#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset)		(0x1c040 + (offset))
> +#define MALI_C55_SCALER_TIMEOUT_EN			BIT(4)
> +#define MALI_C55_SCALER_TIMEOUT(t)			((t) << 16)
> +#define MALI_C55_REG_SCALER_IN_WIDTH(offset)		(0x1c044 + (offset))
> +#define MALI_C55_REG_SCALER_IN_HEIGHT(offset)		(0x1c048 + (offset))
> +#define MALI_C55_REG_SCALER_OUT_WIDTH(offset)		(0x1c04c + (offset))
> +#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset)		(0x1c050 + (offset))
> +#define MALI_C55_REG_SCALER_HFILT_TINC(offset)		(0x1c054 + (offset))
> +#define MALI_C55_REG_SCALER_HFILT_COEF(offset)		(0x1c058 + (offset))
> +#define MALI_C55_REG_SCALER_VFILT_TINC(offset)		(0x1c05c + (offset))
> +#define MALI_C55_REG_SCALER_VFILT_COEF(offset)		(0x1c060 + (offset))
> +
> +#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset)		(0x1c064 + (offset))
> +#define MALI_C55_GAMMA_ENABLE_MASK			BIT(0)
> +#define MALI_C55_REG_GAMMA_GAINS_1(offset)		(0x1c068 + (offset))
> +#define MALI_C55_GAMMA_GAIN_R_MASK			GENMASK(11, 0)
> +#define MALI_C55_GAMMA_GAIN_G_MASK			GENMASK(27, 16)
> +#define MALI_C55_REG_GAMMA_GAINS_2(offset)		(0x1c06c + (offset))
> +#define MALI_C55_GAMMA_GAIN_B_MASK			GENMASK(11, 0)
> +#define MALI_C55_REG_GAMMA_OFFSETS_1(offset)		(0x1c070 + (offset))
> +#define MALI_C55_GAMMA_OFFSET_R_MASK			GENMASK(11, 0)
> +#define MALI_C55_GAMMA_OFFSET_G_MASK			GENMASK(27, 16)
> +#define MALI_C55_REG_GAMMA_OFFSETS_2(offset)		(0x1c074 + (offset))
> +#define MALI_C55_GAMMA_OFFSET_B_MASK			GENMASK(11, 0)
> +
> +/* Output DMA Writer */
> +
> +#define MALI_C55_OUTPUT_DISABLED		0
> +#define MALI_C55_OUTPUT_RGB32			1
> +#define MALI_C55_OUTPUT_A2R10G10B10		2
> +#define MALI_C55_OUTPUT_RGB565			3
> +#define MALI_C55_OUTPUT_RGB24			4
> +#define MALI_C55_OUTPUT_GEN32			5
> +#define MALI_C55_OUTPUT_RAW16			6
> +#define MALI_C55_OUTPUT_AYUV			8
> +#define MALI_C55_OUTPUT_Y410			9
> +#define MALI_C55_OUTPUT_YUY2			10
> +#define MALI_C55_OUTPUT_UYVY			11
> +#define MALI_C55_OUTPUT_Y210			12
> +#define MALI_C55_OUTPUT_NV12_21			13
> +#define MALI_C55_OUTPUT_YUV_420_422		17
> +#define MALI_C55_OUTPUT_P210_P010		19
> +#define MALI_C55_OUTPUT_YUV422			20

I'd define this just below the MALI_C55_REG_UV_WRITER_MODE register
macro.

> +
> +#define MALI_C55_OUTPUT_PLANE_ALT0		0
> +#define MALI_C55_OUTPUT_PLANE_ALT1		1
> +#define MALI_C55_OUTPUT_PLANE_ALT2		2

Same here ?

> +
> +#endif /* _MALI_C55_REGISTERS_H */
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> new file mode 100644
> index 000000000000..8edae87f1e5f
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> @@ -0,0 +1,382 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * ARM Mali-C55 ISP Driver - Resizer Coefficients
> + *
> + * Copyright (C) 2024 Ideas on Board Oy
> + */
> +
> +#ifndef _MALI_C55_RESIZER_COEFS_H
> +#define _MALI_C55_RESIZER_COEFS_H
> +
> +#include "mali-c55-common.h"
> +
> +#define MALI_C55_RESIZER_COEFS_NUM_BANKS	8
> +#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES	64

Do these belongs to mali-c55-registers.h ?

> +
> +static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
> +	{	/* Bank 0 */
> +		0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
> +		0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
> +		0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
> +		0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
> +		0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
> +		0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
> +		0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
> +		0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
> +		0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
> +		0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
> +		0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
> +		0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
> +		0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
> +		0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
> +		0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
> +		0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
> +	},
> +	{	/* Bank 1 */
> +		0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
> +		0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
> +		0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
> +		0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
> +		0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
> +		0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
> +		0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
> +		0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
> +		0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
> +		0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
> +		0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
> +		0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
> +		0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
> +		0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
> +		0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
> +		0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
> +	},
> +	{	/* Bank 2 */
> +		0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
> +		0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
> +		0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
> +		0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
> +		0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
> +		0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
> +		0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
> +		0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
> +		0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
> +		0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
> +		0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
> +		0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
> +		0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
> +		0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
> +		0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
> +		0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
> +	},
> +	{	/* Bank 3 */
> +		0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
> +		0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
> +		0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
> +		0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
> +		0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
> +		0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
> +		0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
> +		0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
> +		0x20100000, 0x00000010, 0x1f110000, 0x00000010,
> +		0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
> +		0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
> +		0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
> +		0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
> +		0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
> +		0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
> +		0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
> +	},
> +	{	/* Bank 4 */
> +		0x17090000, 0x00000917, 0x18090000, 0x00000916,
> +		0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
> +		0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
> +		0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
> +		0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
> +		0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
> +		0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
> +		0x190f0300, 0x00000411, 0x18100300, 0x00000411,
> +		0x1a100300, 0x00000310, 0x18110400, 0x00000310,
> +		0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
> +		0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
> +		0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
> +		0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
> +		0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
> +		0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
> +		0x17160800, 0x0000010a, 0x18160900, 0x00000009,
> +	},
> +	{	/* Bank 5 */
> +		0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
> +		0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
> +		0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
> +		0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
> +		0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
> +		0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
> +		0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
> +		0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
> +		0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
> +		0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
> +		0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
> +		0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
> +		0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
> +		0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
> +		0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
> +		0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
> +	},
> +	{	/* Bank 6 */
> +		0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
> +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
> +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
> +		0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
> +		0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> +		0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
> +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> +		0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
> +		0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
> +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
> +		0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> +	},
> +	{	/* Bank 7 */
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> +	}
> +};
> +
> +static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
> +	{	/* Bank 0 */
> +		0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
> +		0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
> +		0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
> +		0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
> +		0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
> +		0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
> +		0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
> +		0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
> +		0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
> +		0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
> +		0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
> +		0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
> +		0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
> +		0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
> +		0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
> +		0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
> +	},
> +	{	/* Bank 1 */
> +		0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
> +		0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
> +		0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
> +		0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
> +		0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
> +		0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
> +		0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
> +		0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
> +		0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
> +		0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
> +		0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
> +		0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
> +		0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
> +		0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
> +		0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
> +		0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
> +	},
> +	{	/* Bank 2 */
> +		0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
> +		0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
> +		0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
> +		0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
> +		0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
> +		0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
> +		0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
> +		0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
> +		0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
> +		0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
> +		0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
> +		0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
> +		0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
> +		0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
> +		0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
> +		0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
> +	},
> +	{	/* Bank 3 */
> +		0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
> +		0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
> +		0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
> +		0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
> +		0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
> +		0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
> +		0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
> +		0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
> +		0x20100000, 0x00000010, 0x1f100000, 0x00000011,
> +		0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
> +		0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
> +		0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
> +		0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
> +		0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
> +		0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
> +		0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
> +	},
> +	{	/* Bank 4 */
> +		0x17170900, 0x00000009, 0x18160900, 0x00000009,
> +		0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
> +		0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
> +		0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
> +		0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
> +		0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
> +		0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
> +		0x19110400, 0x0000030f, 0x18110400, 0x00000310,
> +		0x1a100300, 0x00000310, 0x18100300, 0x00000411,
> +		0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
> +		0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
> +		0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
> +		0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
> +		0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
> +		0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
> +		0x170a0100, 0x00000816, 0x18090000, 0x00000916,
> +	},
> +	{	/* Bank 5 */
> +		0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
> +		0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
> +		0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
> +		0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
> +		0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
> +		0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
> +		0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
> +		0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
> +		0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
> +		0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
> +		0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
> +		0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
> +		0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
> +		0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
> +		0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
> +		0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
> +	},
> +	{	/* Bank 6 */
> +		0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
> +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
> +		0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
> +		0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> +		0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
> +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> +		0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
> +		0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
> +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
> +		0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
> +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
> +	},
> +	{	/* Bank 7 */
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> +	}
> +};
> +
> +struct mali_c55_resizer_coef_bank {
> +	unsigned int bank;

This is always equal to the index of the entry in the
mali_c55_coefficient_banks array, you can drop it.

> +	unsigned int top;
> +	unsigned int bottom;

The bottom value of bank N is always equal to the top value of bank N+1.
You can simplify this by storing a single value.

> +};
> +
> +static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
> +	{
> +		.bank = 0,
> +		.top = 1000,
> +		.bottom = 770,
> +	},
> +	{
> +		.bank = 1,
> +		.top = 769,
> +		.bottom = 600,
> +	},
> +	{
> +		.bank = 2,
> +		.top = 599,
> +		.bottom = 460,
> +	},
> +	{
> +		.bank = 3,
> +		.top = 459,
> +		.bottom = 354,
> +	},
> +	{
> +		.bank = 4,
> +		.top = 353,
> +		.bottom = 273,
> +	},
> +	{
> +		.bank = 5,
> +		.top = 272,
> +		.bottom = 210,
> +	},
> +	{
> +		.bank = 6,
> +		.top = 209,
> +		.bottom = 162,
> +	},
> +	{
> +		.bank = 7,
> +		.top = 161,
> +		.bottom = 125,
> +	},
> +};
> +

A small comment would be nice, such as

/* Select a bank of resizer coefficients, based on the scaling ratio. */

> +static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,

This function is related to the resizers. Add "rsz" somewhere in the
function name, and pass a resizer pointer.

> +						unsigned int crop,
> +						unsigned int scale)

I think those are the input and output sizes to the scaler. Rename them
to make it clearer.

> +{
> +	unsigned int tmp;

tmp is almost always a bad variable name. Please use a more descriptive
name, size as rsz_ratio.

> +	unsigned int i;
> +
> +	tmp = (scale * 1000U) / crop;
> +
> +	for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
> +		if (tmp >= mali_c55_coefficient_banks[i].bottom &&
> +		    tmp <= mali_c55_coefficient_banks[i].top)
> +			return mali_c55_coefficient_banks[i].bank;
> +	}
> +
> +	/*
> +	 * We shouldn't ever get here, in theory. As we have no good choices
> +	 * simply warn the user and use the first bank of coefficients.
> +	 */
> +	dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
> +	return 0;
> +}

And everything else belongs to mali-c55-resizer.c. Drop this header
file.

> +
> +#endif /* _MALI_C55_RESIZER_COEFS_H */
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> new file mode 100644
> index 000000000000..0a5a2969d3ce
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> @@ -0,0 +1,779 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ARM Mali-C55 ISP Driver - Image signal processor
> + *
> + * Copyright (C) 2024 Ideas on Board Oy
> + */
> +
> +#include <linux/math.h>
> +#include <linux/minmax.h>
> +
> +#include <media/media-entity.h>
> +#include <media/v4l2-subdev.h>
> +
> +#include "mali-c55-common.h"
> +#include "mali-c55-registers.h"
> +#include "mali-c55-resizer-coefs.h"
> +
> +/* Scaling factor in Q4.20 format. */
> +#define MALI_C55_RZR_SCALER_FACTOR	(1U << 20)
> +
> +static const u32 rzr_non_bypass_src_fmts[] = {
> +	MEDIA_BUS_FMT_RGB121212_1X36,
> +	MEDIA_BUS_FMT_YUV10_1X30
> +};
> +
> +static const char * const mali_c55_resizer_names[] = {
> +	[MALI_C55_RZR_FR] = "resizer fr",
> +	[MALI_C55_RZR_DS] = "resizer ds",
> +};
> +
> +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
> +				     struct v4l2_subdev_state *state)
> +{
> +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> +	struct v4l2_mbus_framefmt *fmt;
> +	struct v4l2_rect *crop;
> +
> +	/* Verify if crop should be enabled. */
> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> +
> +	if (fmt->width == crop->width && fmt->height == crop->height)
> +		return MALI_C55_BYPASS_CROP;
> +
> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
> +		       crop->left);
> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
> +		       crop->top);
> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
> +		       crop->width);
> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
> +		       crop->height);
> +
> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
> +		       MALI_C55_CROP_ENABLE);
> +
> +	return 0;
> +}
> +
> +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
> +					struct v4l2_subdev_state *state)
> +{
> +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> +	struct v4l2_rect *crop, *scale;
> +	unsigned int h_bank, v_bank;
> +	u64 h_scale, v_scale;
> +
> +	/* Verify if scaling should be enabled. */
> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> +	scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
> +
> +	if (crop->width == scale->width && crop->height == scale->height)
> +		return MALI_C55_BYPASS_SCALER;
> +
> +	/* Program the V/H scaling factor in Q4.20 format. */
> +	h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
> +	v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
> +
> +	do_div(h_scale, scale->width);
> +	do_div(v_scale, scale->height);
> +
> +	mali_c55_write(mali_c55,
> +		       MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
> +		       crop->width);
> +	mali_c55_write(mali_c55,
> +		       MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
> +		       crop->height);
> +
> +	mali_c55_write(mali_c55,
> +		       MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
> +		       scale->width);
> +	mali_c55_write(mali_c55,
> +		       MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
> +		       scale->height);
> +
> +	mali_c55_write(mali_c55,
> +		       MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
> +		       h_scale);
> +	mali_c55_write(mali_c55,
> +		       MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
> +		       v_scale);
> +
> +	h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
> +					     scale->width);
> +	mali_c55_write(mali_c55,
> +		       MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
> +		       h_bank);
> +
> +	v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
> +					     scale->height);
> +	mali_c55_write(mali_c55,
> +		       MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
> +		       v_bank);
> +
> +	return 0;
> +}
> +
> +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
> +				 struct v4l2_subdev_state *state)
> +{
> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> +	u32 bypass = 0;
> +
> +	/* Verify if cropping and scaling should be enabled. */
> +	bypass |= mali_c55_rzr_program_crop(rzr, state);
> +	bypass |= mali_c55_rzr_program_resizer(rzr, state);
> +
> +	mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
> +			     MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
> +			     MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
> +			     bypass);
> +}
> +
> +/*
> + * Inspect the routing table to know which of the two (mutually exclusive)
> + * routes is enabled and return the sink pad id of the active route.
> + */
> +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
> +{
> +	struct v4l2_subdev_krouting *routing = &state->routing;
> +	struct v4l2_subdev_route *route;
> +
> +	/* A single route is enabled at a time. */
> +	for_each_active_route(routing, route)
> +		return route->sink_pad;
> +
> +	return MALI_C55_RZR_SINK_PAD;
> +}
> +
> +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
> +{
> +	u32 corrected_code = 0;
> +
> +	/*
> +	 * The ISP takes input in a 20-bit format, but can only output 16-bit
> +	 * RAW bayer data (with the 4 least significant bits from the input
> +	 * being lost). Return the 16-bit version of the 20-bit input formats.
> +	 */
> +	switch (mbus_code) {
> +	case MEDIA_BUS_FMT_SBGGR20_1X20:
> +		corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
> +		break;
> +	case MEDIA_BUS_FMT_SGBRG20_1X20:
> +		corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
> +		break;
> +	case MEDIA_BUS_FMT_SGRBG20_1X20:
> +		corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
> +		break;
> +	case MEDIA_BUS_FMT_SRGGB20_1X20:
> +		corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
> +		break;
> +	}
> +
> +	return corrected_code;
> +}
> +
> +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> +				      struct v4l2_subdev_state *state,
> +				      struct v4l2_subdev_krouting *routing)
> +{
> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> +						    sd);
> +	unsigned int active_sink = UINT_MAX;
> +	struct v4l2_mbus_framefmt *src_fmt;
> +	struct v4l2_rect *crop, *compose;
> +	struct v4l2_subdev_route *route;
> +	unsigned int active_routes = 0;
> +	struct v4l2_mbus_framefmt *fmt;
> +	int ret;
> +
> +	ret = v4l2_subdev_routing_validate(sd, routing, 0);
> +	if (ret)
> +		return ret;
> +
> +	/* Only a single route can be enabled at a time. */
> +	for_each_active_route(routing, route) {
> +		if (++active_routes > 1) {
> +			dev_err(rzr->mali_c55->dev,
> +				"Only one route can be active");
> +			return -EINVAL;
> +		}
> +
> +		active_sink = route->sink_pad;
> +	}
> +	if (active_sink == UINT_MAX) {
> +		dev_err(rzr->mali_c55->dev, "One route has to be active");
> +		return -EINVAL;
> +	}
> +
> +	ret = v4l2_subdev_set_routing(sd, state, routing);
> +	if (ret) {
> +		dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
> +		return ret;
> +	}
> +
> +	fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
> +	crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
> +	compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
> +
> +	fmt->width = MALI_C55_DEFAULT_WIDTH;
> +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
> +	fmt->colorspace = V4L2_COLORSPACE_SRGB;
> +	fmt->field = V4L2_FIELD_NONE;
> +
> +	if (active_sink == MALI_C55_RZR_SINK_PAD) {
> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> +
> +		crop->left = crop->top = 0;
> +		crop->width = MALI_C55_DEFAULT_WIDTH;
> +		crop->height = MALI_C55_DEFAULT_HEIGHT;
> +
> +		*compose = *crop;
> +	} else {
> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> +	}
> +
> +	/* Propagate the format to the source pad */
> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
> +					       0);
> +	*src_fmt = *fmt;
> +
> +	/* In the event this is the bypass pad the mbus code needs correcting */
> +	if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
> +		src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
> +
> +	return 0;
> +}
> +
> +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
> +				       struct v4l2_subdev_state *state,
> +				       struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	struct v4l2_mbus_framefmt *sink_fmt;
> +	const struct mali_c55_isp_fmt *fmt;
> +	unsigned int index = 0;
> +	u32 sink_pad;
> +
> +	switch (code->pad) {
> +	case MALI_C55_RZR_SINK_PAD:
> +		if (code->index)
> +			return -EINVAL;
> +
> +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> +
> +		return 0;
> +	case MALI_C55_RZR_SOURCE_PAD:
> +		sink_pad = mali_c55_rzr_get_active_sink(state);
> +		sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> +
> +		/*
> +		 * If the active route is from the Bypass sink pad, then the
> +		 * source pad is a simple passthrough of the sink format,
> +		 * downshifted to 16-bits.
> +		 */
> +
> +		if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> +			if (code->index)
> +				return -EINVAL;
> +
> +			code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> +			if (!code->code)
> +				return -EINVAL;
> +
> +			return 0;
> +		}
> +
> +		/*
> +		 * If the active route is from the non-bypass sink then we can
> +		 * select either RGB or conversion to YUV.
> +		 */
> +
> +		if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
> +			return -EINVAL;
> +
> +		code->code = rzr_non_bypass_src_fmts[code->index];
> +
> +		return 0;
> +	case MALI_C55_RZR_SINK_BYPASS_PAD:
> +		for_each_mali_isp_fmt(fmt) {
> +			if (index++ == code->index) {
> +				code->code = fmt->code;
> +				return 0;
> +			}
> +		}
> +
> +		break;
> +	}
> +
> +	return -EINVAL;
> +}
> +
> +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
> +					struct v4l2_subdev_state *state,
> +					struct v4l2_subdev_frame_size_enum *fse)
> +{
> +	if (fse->index)
> +		return -EINVAL;
> +
> +	fse->max_width = MALI_C55_MAX_WIDTH;
> +	fse->max_height = MALI_C55_MAX_HEIGHT;
> +	fse->min_width = MALI_C55_MIN_WIDTH;
> +	fse->min_height = MALI_C55_MIN_HEIGHT;
> +
> +	return 0;
> +}
> +
> +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
> +				     struct v4l2_subdev_state *state,
> +				     struct v4l2_subdev_format *format)
> +{
> +	struct v4l2_mbus_framefmt *fmt = &format->format;
> +	struct v4l2_rect *rect;
> +	unsigned int sink_pad;
> +
> +	/*
> +	 * Clamp to min/max and then reset crop and compose rectangles to the
> +	 * newly applied size.
> +	 */
> +	clamp_t(unsigned int, fmt->width,
> +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> +	clamp_t(unsigned int, fmt->height,
> +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> +
> +	sink_pad = mali_c55_rzr_get_active_sink(state);
> +	if (sink_pad == MALI_C55_RZR_SINK_PAD) {
> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> +
> +		rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> +		rect->left = 0;
> +		rect->top = 0;
> +		rect->width = fmt->width;
> +		rect->height = fmt->height;
> +
> +		rect = v4l2_subdev_state_get_compose(state,
> +						     MALI_C55_RZR_SINK_PAD);
> +		rect->left = 0;
> +		rect->top = 0;
> +		rect->width = fmt->width;
> +		rect->height = fmt->height;
> +	} else {
> +		/*
> +		 * Make sure the media bus code is one of the supported
> +		 * ISP input media bus codes.
> +		 */
> +		if (!mali_c55_isp_is_format_supported(fmt->code))
> +			fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
> +	}
> +
> +	*v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
> +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
> +
> +	return 0;
> +}
> +
> +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> +				       struct v4l2_subdev_state *state,
> +				       struct v4l2_subdev_format *format)
> +{
> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> +						    sd);
> +	struct v4l2_mbus_framefmt *fmt = &format->format;
> +	struct v4l2_mbus_framefmt *sink_fmt;
> +	struct v4l2_rect *crop, *compose;
> +	unsigned int sink_pad;
> +	unsigned int i;
> +
> +	sink_pad = mali_c55_rzr_get_active_sink(state);
> +	sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> +	crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
> +	compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
> +
> +	/* FR Bypass pipe. */
> +
> +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> +		/*
> +		 * Format on the source pad is the same as the one on the
> +		 * sink pad, downshifted to 16-bits.
> +		 */
> +		fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> +		if (!fmt->code)
> +			return -EINVAL;
> +
> +		/* RAW bypass disables scaling and cropping. */
> +		crop->top = compose->top = 0;
> +		crop->left = compose->left = 0;
> +		fmt->width = crop->width = compose->width = sink_fmt->width;
> +		fmt->height = crop->height = compose->height = sink_fmt->height;
> +
> +		*v4l2_subdev_state_get_format(state,
> +					      MALI_C55_RZR_SOURCE_PAD) = *fmt;
> +
> +		return 0;
> +	}
> +
> +	/* Regular processing pipe. */
> +
> +	for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> +		if (fmt->code == rzr_non_bypass_src_fmts[i])
> +			break;
> +	}
> +
> +	if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
> +		dev_dbg(rzr->mali_c55->dev,
> +			"Unsupported mbus code 0x%x: using default\n",
> +			fmt->code);
> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> +	}
> +
> +	/*
> +	 * The source pad format size comes directly from the sink pad
> +	 * compose rectangle.
> +	 */
> +	fmt->width = compose->width;
> +	fmt->height = compose->height;
> +
> +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
> +
> +	return 0;
> +}
> +
> +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
> +				struct v4l2_subdev_state *state,
> +				struct v4l2_subdev_format *format)
> +{
> +	/*
> +	 * On sink pads fmt is either fixed for the 'regular' processing
> +	 * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
> +	 * pad.
> +	 *
> +	 * On source pad sizes are the result of crop+compose on the sink
> +	 * pad sizes, while the format depends on the active route.
> +	 */
> +
> +	if (format->pad != MALI_C55_RZR_SOURCE_PAD)
> +		return mali_c55_rzr_set_sink_fmt(sd, state, format);
> +
> +	return mali_c55_rzr_set_source_fmt(sd, state, format);
> +}
> +
> +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
> +				      struct v4l2_subdev_state *state,
> +				      struct v4l2_subdev_selection *sel)
> +{
> +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
> +		return -EINVAL;
> +
> +	if (sel->target != V4L2_SEL_TGT_CROP &&
> +	    sel->target != V4L2_SEL_TGT_COMPOSE)
> +		return -EINVAL;
> +
> +	sel->r = sel->target == V4L2_SEL_TGT_CROP
> +	       ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
> +	       : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> +
> +	return 0;
> +}
> +
> +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
> +				      struct v4l2_subdev_state *state,
> +				      struct v4l2_subdev_selection *sel)
> +{
> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> +						    sd);
> +	struct v4l2_mbus_framefmt *source_fmt;
> +	struct v4l2_mbus_framefmt *sink_fmt;
> +	struct v4l2_rect *crop, *compose;
> +
> +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
> +		return -EINVAL;
> +
> +	if (sel->target != V4L2_SEL_TGT_CROP &&
> +	    sel->target != V4L2_SEL_TGT_COMPOSE)
> +		return -EINVAL;
> +
> +	source_fmt = v4l2_subdev_state_get_format(state,
> +						  MALI_C55_RZR_SOURCE_PAD);
> +	sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> +	compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> +
> +	/* RAW bypass disables crop/scaling. */
> +	if (mali_c55_format_is_raw(source_fmt->code)) {
> +		crop->top = compose->top = 0;
> +		crop->left = compose->left = 0;
> +		crop->width = compose->width = sink_fmt->width;
> +		crop->height = compose->height = sink_fmt->height;
> +
> +		sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> +
> +		return 0;
> +	}
> +
> +	/* During streaming, it is allowed to only change the crop rectangle. */
> +	if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
> +		return -EINVAL;
> +
> +	 /*
> +	  * Update the desired target and then clamp the crop rectangle to the
> +	  * sink format sizes and the compose size to the crop sizes.
> +	  */
> +	if (sel->target == V4L2_SEL_TGT_CROP)
> +		*crop = sel->r;
> +	else
> +		*compose = sel->r;
> +
> +	clamp_t(unsigned int, crop->left, 0,  sink_fmt->width);
> +	clamp_t(unsigned int, crop->top, 0,  sink_fmt->height);
> +	clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
> +		sink_fmt->width - crop->left);
> +	clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
> +		sink_fmt->height - crop->top);
> +
> +	if (rzr->streaming) {
> +		/*
> +		 * Apply at runtime a crop rectangle on the resizer's sink only
> +		 * if it doesn't require re-programming the scaler output sizes
> +		 * as it would require changing the output buffer sizes as well.
> +		 */
> +		if (sel->r.width < compose->width ||
> +		    sel->r.height < compose->height)
> +			return -EINVAL;
> +
> +		*crop = sel->r;
> +		mali_c55_rzr_program(rzr, state);
> +
> +		return 0;
> +	}
> +
> +	compose->left = 0;
> +	compose->top = 0;
> +	clamp_t(unsigned int, compose->left, 0,  sink_fmt->width);
> +	clamp_t(unsigned int, compose->top, 0,  sink_fmt->height);
> +	clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
> +	clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
> +
> +	sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> +
> +	return 0;
> +}
> +
> +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> +				    struct v4l2_subdev_state *state,
> +				    enum v4l2_subdev_format_whence which,
> +				    struct v4l2_subdev_krouting *routing)
> +{
> +	if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
> +	    media_entity_is_streaming(&sd->entity))
> +		return -EBUSY;
> +
> +	return __mali_c55_rzr_set_routing(sd, state, routing);
> +}
> +
> +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
> +	.enum_mbus_code		= mali_c55_rzr_enum_mbus_code,
> +	.enum_frame_size	= mali_c55_rzr_enum_frame_size,
> +	.get_fmt		= v4l2_subdev_get_fmt,
> +	.set_fmt		= mali_c55_rzr_set_fmt,
> +	.get_selection		= mali_c55_rzr_get_selection,
> +	.set_selection		= mali_c55_rzr_set_selection,
> +	.set_routing		= mali_c55_rzr_set_routing,
> +};
> +
> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
> +{
> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> +	struct v4l2_subdev *sd = &rzr->sd;
> +	struct v4l2_subdev_state *state;
> +	unsigned int sink_pad;
> +
> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> +
> +	sink_pad = mali_c55_rzr_get_active_sink(state);
> +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> +		/* Bypass FR pipe processing if the bypass route is active. */
> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> +				     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
> +				     MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
> +		goto unlock_state;
> +	}
> +
> +	/* Disable bypass and use regular processing. */
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> +			     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
> +	mali_c55_rzr_program(rzr, state);
> +
> +unlock_state:
> +	rzr->streaming = true;
> +	v4l2_subdev_unlock_state(state);
> +}
> +
> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
> +{
> +	struct v4l2_subdev *sd = &rzr->sd;
> +	struct v4l2_subdev_state *state;
> +
> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> +	rzr->streaming = false;
> +	v4l2_subdev_unlock_state(state);
> +}
> +
> +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
> +	.pad	= &mali_c55_resizer_pad_ops,
> +};
> +
> +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
> +				   struct v4l2_subdev_state *state)
> +{
> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> +						    sd);
> +	struct v4l2_subdev_krouting routing = { };
> +	struct v4l2_subdev_route *routes;
> +	unsigned int i;
> +	int ret;
> +
> +	routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
> +	if (!routes)
> +		return -ENOMEM;
> +
> +	for (i = 0; i < rzr->num_routes; ++i) {
> +		struct v4l2_subdev_route *route = &routes[i];
> +
> +		route->sink_pad = i
> +				? MALI_C55_RZR_SINK_BYPASS_PAD
> +				: MALI_C55_RZR_SINK_PAD;
> +		route->source_pad = MALI_C55_RZR_SOURCE_PAD;
> +		if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
> +			route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
> +	}
> +
> +	routing.num_routes = rzr->num_routes;
> +	routing.routes = routes;
> +
> +	ret = __mali_c55_rzr_set_routing(sd, state, &routing);
> +	kfree(routes);
> +
> +	return ret;
> +}
> +
> +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
> +	.init_state = mali_c55_rzr_init_state,
> +};
> +
> +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
> +						  unsigned int index)
> +{
> +	const unsigned int scaler_filt_coefmem_addrs[][2] = {
> +		[MALI_C55_RZR_FR] = {
> +			0x034A8, /* hfilt */
> +			0x044A8  /* vfilt */

Lowercase hex constants.

> +		},
> +		[MALI_C55_RZR_DS] = {
> +			0x014A8, /* hfilt */
> +			0x024A8  /* vfilt */
> +		},
> +	};
> +	unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
> +	unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
> +	unsigned int i, j;
> +
> +	for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
> +		for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
> +			mali_c55_write(mali_c55, haddr,
> +				mali_c55_scaler_h_filter_coefficients[i][j]);
> +			mali_c55_write(mali_c55, vaddr,
> +				mali_c55_scaler_v_filter_coefficients[i][j]);
> +
> +			haddr += sizeof(u32);
> +			vaddr += sizeof(u32);
> +		}
> +	}
> +}
> +
> +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
> +{
> +	unsigned int i;
> +	int ret;
> +
> +	for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
> +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> +		struct v4l2_subdev *sd = &rzr->sd;
> +		unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
> +
> +		rzr->id = i;
> +		rzr->streaming = false;
> +
> +		if (rzr->id == MALI_C55_RZR_FR)
> +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
> +		else
> +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
> +
> +		mali_c55_resizer_program_coefficients(mali_c55, i);
> +
> +		v4l2_subdev_init(sd, &mali_c55_resizer_ops);
> +		sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
> +			     | V4L2_SUBDEV_FL_STREAMS;
> +		sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
> +		sd->internal_ops = &mali_c55_resizer_internal_ops;
> +		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
> +			 MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
> +
> +		rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
> +		rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
> +
> +		/* Only the FR pipe has a bypass pad. */
> +		if (rzr->id == MALI_C55_RZR_FR) {
> +			rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
> +							MEDIA_PAD_FL_SINK;
> +			rzr->num_routes = 2;
> +		} else {
> +			num_pads -= 1;
> +			rzr->num_routes = 1;
> +		}
> +
> +		ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
> +		if (ret)
> +			return ret;
> +
> +		ret = v4l2_subdev_init_finalize(sd);
> +		if (ret)
> +			goto err_cleanup;
> +
> +		ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> +		if (ret)
> +			goto err_cleanup;
> +
> +		rzr->mali_c55 = mali_c55;
> +	}
> +
> +	return 0;
> +
> +err_cleanup:
> +	for (; i >= 0; --i) {
> +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> +		struct v4l2_subdev *sd = &rzr->sd;
> +
> +		v4l2_subdev_cleanup(sd);
> +		media_entity_cleanup(&sd->entity);
> +	}
> +
> +	return ret;
> +}
> +
> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
> +		struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
> +
> +		if (!resizer->mali_c55)
> +			continue;
> +
> +		v4l2_device_unregister_subdev(&resizer->sd);
> +		v4l2_subdev_cleanup(&resizer->sd);
> +		media_entity_cleanup(&resizer->sd.entity);
> +	}
> +}
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> new file mode 100644
> index 000000000000..c7e699741c6d
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> @@ -0,0 +1,402 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ARM Mali-C55 ISP Driver - Test pattern generator
> + *
> + * Copyright (C) 2024 Ideas on Board Oy
> + */
> +
> +#include <linux/minmax.h>
> +#include <linux/string.h>
> +
> +#include <media/media-entity.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-subdev.h>
> +
> +#include "mali-c55-common.h"
> +#include "mali-c55-registers.h"
> +
> +#define MALI_C55_TPG_SRC_PAD		0
> +#define MALI_C55_TPG_FIXED_HBLANK	0x20
> +#define MALI_C55_TPG_MAX_VBLANK		0xFFFF

Lowercase hex constants.

> +#define MALI_C55_TPG_PIXEL_RATE		100000000

This should be exposed to applications using the V4L2_CID_PIXEL_RATE
control (read-only).

> +
> +static const char * const mali_c55_tpg_test_pattern_menu[] = {
> +	"Flat field",
> +	"Horizontal gradient",
> +	"Vertical gradient",
> +	"Vertical bars",
> +	"Arbitrary rectangle",
> +	"White frame on black field"
> +};
> +
> +static const u32 mali_c55_tpg_mbus_codes[] = {
> +	MEDIA_BUS_FMT_SRGGB20_1X20,
> +	MEDIA_BUS_FMT_RGB202020_1X60,
> +};
> +
> +static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
> +				       int *def_vblank, int *min_vblank)

unsigned int ?

> +{
> +	unsigned int hts;
> +	int tgt_fps;
> +	int vblank;
> +
> +	hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
> +
> +	/*
> +	 * The ISP has minimum vertical blanking requirements that must be
> +	 * adhered to by the TPG. The minimum is a function of the Iridix blocks
> +	 * clocking requirements and the width of the image and horizontal
> +	 * blanking, but if we assume the worst case iVariance and sVariance
> +	 * values then it boils down to the below.
> +	 */
> +	*min_vblank = 15 + (120500 / hts);

I wonder if this should round up.

> +
> +	/*
> +	 * We need to set a sensible default vblank for whatever format height
> +	 * we happen to be given from set_fmt(). This function just targets
> +	 * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
> +	 * If we can't get 5fps we'll take whatever the minimum vblank gives us.
> +	 */
> +	tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
> +
> +	if (tgt_fps < 5)
> +		vblank = *min_vblank;
> +	else
> +		vblank = MALI_C55_TPG_PIXEL_RATE / hts
> +		       / max(rounddown(tgt_fps, 15), 5);
> +
> +	*def_vblank = ALIGN_DOWN(vblank, 2) - format->height;

"vblank = vblank - height" doesn't seem right. The "else" branch stores
a vts in vblank, which doesn't seem right either. Maybe you meant
something like

	if (tgt_fps < 5)
		def_vts = *min_vblank + format->height;
	else
		def_vts = MALI_C55_TPG_PIXEL_RATE / hts
			/ max(rounddown(tgt_fps, 15), 5);

	*def_vblank = ALIGN_DOWN(def_vts, 2) - format->height;

> +}
> +
> +static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
> +{
> +	struct mali_c55_tpg *tpg = container_of(ctrl->handler,
> +						struct mali_c55_tpg,
> +						ctrls.handler);
> +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
> +

Should you return here if the pipeline isn't streaming ?

> +	switch (ctrl->id) {
> +	case V4L2_CID_TEST_PATTERN:
> +		mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
> +			       ctrl->val);
> +		break;
> +	case V4L2_CID_VBLANK:
> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
> +				     MALI_C55_REG_VBLANK_MASK, ctrl->val);
> +		break;
> +	default:
> +		dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
> +		return -EINVAL;

Can this happen ?

> +	}
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
> +	.s_ctrl = &mali_c55_tpg_s_ctrl,
> +};
> +
> +static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
> +				   struct v4l2_subdev *sd)
> +{
> +	struct v4l2_subdev_state *state;
> +	struct v4l2_mbus_framefmt *fmt;
> +
> +	/*
> +	 * hblank needs setting, but is a read-only control and thus won't be
> +	 * called during __v4l2_ctrl_handler_setup(). Do it here instead.
> +	 */
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
> +			     MALI_C55_REG_HBLANK_MASK,
> +			     MALI_C55_TPG_FIXED_HBLANK);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> +			     MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
> +
> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> +			     MALI_C55_TEST_PATTERN_RGB_MASK,
> +			     fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
> +					  0x01 : 0x0);
> +
> +	v4l2_subdev_unlock_state(state);
> +}
> +
> +static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
> +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
> +
> +	if (!enable) {
> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> +				MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> +				MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
> +		return 0;
> +	}
> +
> +	/*
> +	 * One might reasonably expect the framesize to be set here
> +	 * given it's configurable in .set_fmt(), but it's done in the
> +	 * ISP subdevice's stream on func instead, as the same register

s/func/function/

> +	 * is also used to indicate the size of the data coming from the
> +	 * sensor.
> +	 */
> +	mali_c55_tpg_configure(mali_c55, sd);

	mali_c55_tpg_configure(tpg);

> +	__v4l2_ctrl_handler_setup(sd->ctrl_handler);
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> +			     MALI_C55_TEST_PATTERN_ON_OFF,
> +			     MALI_C55_TEST_PATTERN_ON_OFF);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> +			     MALI_C55_REG_GEN_VIDEO_ON_MASK,
> +			     MALI_C55_REG_GEN_VIDEO_ON_MASK);
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
> +	.s_stream = &mali_c55_tpg_s_stream,

Can we use .enable_streams() and .disable_streams() ?

> +};
> +
> +static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
> +				       struct v4l2_subdev_state *state,
> +				       struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> +		return -EINVAL;
> +
> +	code->code = mali_c55_tpg_mbus_codes[code->index];
> +
> +	return 0;
> +}
> +
> +static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
> +					struct v4l2_subdev_state *state,
> +					struct v4l2_subdev_frame_size_enum *fse)
> +{

You sohuld verify here that fse->code is a supported value and return
-EINVAL otherwise.

> +	if (fse->index > 0 || fse->pad > sd->entity.num_pads)

Drop the pad check, it's done in the subdev core already.

> +		return -EINVAL;
> +
> +	fse->min_width = MALI_C55_MIN_WIDTH;
> +	fse->max_width = MALI_C55_MAX_WIDTH;
> +	fse->min_height = MALI_C55_MIN_HEIGHT;
> +	fse->max_height = MALI_C55_MAX_HEIGHT;
> +
> +	return 0;
> +}
> +
> +static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
> +				struct v4l2_subdev_state *state,
> +				struct v4l2_subdev_format *format)
> +{
> +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
> +	struct v4l2_mbus_framefmt *fmt = &format->format;
> +	int vblank_def, vblank_min;
> +	unsigned int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> +		if (fmt->code == mali_c55_tpg_mbus_codes[i])
> +			break;
> +	}
> +
> +	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> +
> +	/*
> +	 * The TPG says that the test frame timing generation logic expects a
> +	 * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
> +	 * handle anything smaller than 128x128 it seems pointless to allow a
> +	 * smaller frame.
> +	 */
> +	clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> +		MALI_C55_MAX_WIDTH);
> +	clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> +		MALI_C55_MAX_HEIGHT);
> +
> +	*v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;

You're allowing userspace to set fmt->field, as well as all the
colorspace parameters, to random values. I would instead do something
like

	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
		if (format->format.code == mali_c55_tpg_mbus_codes[i])
			break;  
	}

	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
		format->format.code = MEDIA_BUS_FMT_SRGGB20_1X20;

	format->format.width = clamp(format->format.width,
				     MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
	format->format.height = clamp(format->format.height,
				      MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);

	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
	fmt->code = format->format.code;
	fmt->width = format->format.width;
	fmt->height = format->format.height;

	format->format = *fmt;

Alternatively (which I think I like better),

	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);

	fmt->code = format->format.code;

	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
		if (fmt->code == mali_c55_tpg_mbus_codes[i])
			break;  
	}

	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;

	fmt->width = clamp(format->format.width,
			   MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
	fmt->height = clamp(format->format.height,
			    MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);

	format->format = *fmt;

> +
> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY)
> +		return 0;
> +
> +	__mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
> +	__v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
> +				 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
> +	__v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);

Move those three calls to a separate function, it will be reused below.
I'd name is mali_c55_tpg_update_vblank(). You can fold
__mali_c55_tpg_calc_vblank() in it.

> +
> +	return 0;
> +}
> +
> +static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
> +	.enum_mbus_code		= mali_c55_tpg_enum_mbus_code,
> +	.enum_frame_size	= mali_c55_tpg_enum_frame_size,
> +	.get_fmt		= v4l2_subdev_get_fmt,
> +	.set_fmt		= mali_c55_tpg_set_fmt,
> +};
> +
> +static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
> +	.video	= &mali_c55_tpg_video_ops,
> +	.pad	= &mali_c55_tpg_pad_ops,
> +};
> +
> +static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
> +				   struct v4l2_subdev_state *sd_state)

You name this variable state in every other subdev operation handler.

> +{
> +	struct v4l2_mbus_framefmt *fmt =
> +		v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
> +
> +	fmt->width = MALI_C55_DEFAULT_WIDTH;
> +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
> +	fmt->field = V4L2_FIELD_NONE;
> +	fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;

Initialize the colorspace fields too.

> +
> +	return 0;
> +}
> +
> +static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
> +	.init_state = mali_c55_tpg_init_state,
> +};
> +
> +static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
> +{
> +	struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
> +	struct v4l2_subdev *sd = &mali_c55->tpg.sd;
> +	struct v4l2_mbus_framefmt *format;
> +	struct v4l2_subdev_state *state;
> +	int vblank_def, vblank_min;
> +	int ret;
> +
> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> +	format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> +
> +	ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);

You have 3 controls.

> +	if (ret)
> +		goto err_unlock;
> +
> +	ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
> +				&mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
> +				ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
> +				0, 3, mali_c55_tpg_test_pattern_menu);
> +
> +	/*
> +	 * We fix hblank at the minimum allowed value and control framerate
> +	 * solely through the vblank control.
> +	 */
> +	ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
> +				&mali_c55_tpg_ctrl_ops,
> +				V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
> +				MALI_C55_TPG_FIXED_HBLANK, 1,
> +				MALI_C55_TPG_FIXED_HBLANK);
> +	if (ctrls->hblank)
> +		ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
> +
> +	__mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);

Drop this and initialize the control with default values. You can then
update the value by calling mali_c55_tpg_update_vblank() in
mali_c55_register_tpg().

The reason is to share the same mutex between the control handler and
the subdev active state without having to add a separate mutex in the
mali_c55_tpg structure. The simplest way to do so is to initialize the
controls first, set sd->state_lock to point to the control handler lock,
and call v4l2_subdev_init_finalize() as the last step. As a consequence,
you can't access the active state when initializing controls.

You can alternatively keep the lock in mali_c55_tpg and set
sd->state_lock to point to it, but I think that's more complex.

> +	ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
> +					  &mali_c55_tpg_ctrl_ops,
> +					  V4L2_CID_VBLANK, vblank_min,
> +					  MALI_C55_TPG_MAX_VBLANK, 1,
> +					  vblank_def);
> +
> +	if (ctrls->handler.error) {
> +		dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
> +		ret = ctrls->handler.error;
> +		goto err_free_handler;
> +	}
> +
> +	ctrls->handler.lock = &mali_c55->tpg.lock;

Drop this and drop the mutex. The control handler will use its internal
mutex.

> +	mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
> +
> +	v4l2_subdev_unlock_state(state);
> +
> +	return 0;
> +
> +err_free_handler:
> +	v4l2_ctrl_handler_free(&ctrls->handler);
> +err_unlock:
> +	v4l2_subdev_unlock_state(state);
> +	return ret;
> +}
> +
> +int mali_c55_register_tpg(struct mali_c55 *mali_c55)
> +{
> +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
> +	struct v4l2_subdev *sd = &tpg->sd;
> +	struct media_pad *pad = &tpg->pad;
> +	int ret;
> +
> +	mutex_init(&tpg->lock);
> +
> +	v4l2_subdev_init(sd, &mali_c55_tpg_ops);
> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;

Should we introduce a TPG function ?

> +	sd->internal_ops = &mali_c55_tpg_internal_ops;
> +	strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
> +
> +	pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;

I don't think MEDIA_PAD_FL_MUST_CONNECT is right.

> +	ret = media_entity_pads_init(&sd->entity, 1, pad);
> +	if (ret) {
> +		dev_err(mali_c55->dev,
> +			"Failed to initialize media entity pads\n");
> +		goto err_destroy_mutex;
> +	}
> +

	sd->state_lock = sd->ctrl_handler->lock;

to use the same lock for the controls and the active state. You need to
move this line and the v4l2_subdev_init_finalize() call after
mali_c55_tpg_init_controls() to get the control handler lock initialized
first.

> +	ret = v4l2_subdev_init_finalize(sd);
> +	if (ret)
> +		goto err_cleanup_media_entity;
> +
> +	ret = mali_c55_tpg_init_controls(mali_c55);
> +	if (ret) {
> +		dev_err(mali_c55->dev,
> +			"Error initialising controls\n");
> +		goto err_cleanup_subdev;
> +	}
> +
> +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> +	if (ret) {
> +		dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
> +		goto err_free_ctrl_handler;
> +	}
> +
> +	/*
> +	 * By default the colour settings lead to a very dim image that is
> +	 * nearly indistinguishable from black on some monitor settings. Ramp
> +	 * them up a bit so the image is brighter.
> +	 */
> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
> +		       MALI_C55_TPG_BACKGROUND_MAX);
> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
> +		       MALI_C55_TPG_BACKGROUND_MAX);
> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
> +		       MALI_C55_TPG_BACKGROUND_MAX);
> +
> +	tpg->mali_c55 = mali_c55;
> +
> +	return 0;
> +
> +err_free_ctrl_handler:
> +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> +err_cleanup_subdev:
> +	v4l2_subdev_cleanup(sd);
> +err_cleanup_media_entity:
> +	media_entity_cleanup(&sd->entity);
> +err_destroy_mutex:
> +	mutex_destroy(&tpg->lock);
> +
> +	return ret;
> +}
> +
> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
> +{
> +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
> +
> +	if (!tpg->mali_c55)
> +		return;
> +
> +	v4l2_device_unregister_subdev(&tpg->sd);
> +	v4l2_subdev_cleanup(&tpg->sd);
> +	media_entity_cleanup(&tpg->sd.entity);
> +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);

Free the control handler just after v4l2_device_unregister_subdev() to
match the order in mali_c55_register_tpg().

> +	mutex_destroy(&tpg->lock);
> +}

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 14/16] media: uapi: Add parameters structs to mali-c55-config.h

[kernel test robot]

Hi Daniel,

kernel test robot noticed the following build errors:

[auto build test ERROR on media-tree/master]
[cannot apply to linuxtv-media-stage/master sailus-media-tree/master linus/master sailus-media-tree/streams v6.10-rc1 next-20240529]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Daniel-Scally/media-uapi-Add-MEDIA_BUS_FMT_RGB202020_1X60-format-code/20240529-233239
base:   git://linuxtv.org/media_tree.git master
patch link:    https://lore.kernel.org/r/20240529152858.183799-15-dan.scally%40ideasonboard.com
patch subject: [PATCH v5 14/16] media: uapi: Add parameters structs to mali-c55-config.h
config: i386-buildonly-randconfig-004-20240530 (https://download.01.org/0day-ci/archive/20240530/202405301513.etiNs24g-lkp@intel.com/config)
compiler: clang version 18.1.5 (https://github.com/llvm/llvm-project 617a15a9eac96088ae5e9134248d8236e34b91b1)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240530/202405301513.etiNs24g-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202405301513.etiNs24g-lkp@intel.com/

All errors (new ones prefixed by >>):

   In file included from <built-in>:1:
>> ./usr/include/linux/media/arm/mali-c55-config.h:308:2: error: unknown type name 'bool'
     308 |         bool enabled;
         |         ^
>> ./usr/include/linux/media/arm/mali-c55-config.h:309:2: error: unknown type name 'size_t'
     309 |         size_t size;
         |         ^
   ./usr/include/linux/media/arm/mali-c55-config.h:698:2: error: unknown type name 'bool'
     698 |         bool mesh_show;
         |         ^
   ./usr/include/linux/media/arm/mali-c55-config.h:847:2: error: unknown type name 'size_t'
     847 |         size_t total_size;
         |         ^
   4 errors generated.

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH v5 15/16] media: platform: Add mali-c55 parameters video node

[kernel test robot]

Hi Daniel,

kernel test robot noticed the following build warnings:

[auto build test WARNING on media-tree/master]
[cannot apply to linuxtv-media-stage/master sailus-media-tree/master linus/master sailus-media-tree/streams v6.10-rc1 next-20240529]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Daniel-Scally/media-uapi-Add-MEDIA_BUS_FMT_RGB202020_1X60-format-code/20240529-233239
base:   git://linuxtv.org/media_tree.git master
patch link:    https://lore.kernel.org/r/20240529152858.183799-16-dan.scally%40ideasonboard.com
patch subject: [PATCH v5 15/16] media: platform: Add mali-c55 parameters video node
config: hexagon-allyesconfig (https://download.01.org/0day-ci/archive/20240530/202405301558.no1nWGU1-lkp@intel.com/config)
compiler: clang version 19.0.0git (https://github.com/llvm/llvm-project bafda89a0944d947fc4b3b5663185e07a397ac30)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240530/202405301558.no1nWGU1-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202405301558.no1nWGU1-lkp@intel.com/

All warnings (new ones prefixed by >>):

   In file included from drivers/media/platform/arm/mali-c55/mali-c55-params.c:14:
   In file included from include/media/videobuf2-core.h:18:
   In file included from include/linux/dma-buf.h:16:
   In file included from include/linux/iosys-map.h:10:
   In file included from include/linux/io.h:13:
   In file included from arch/hexagon/include/asm/io.h:328:
   include/asm-generic/io.h:547:31: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     547 |         val = __raw_readb(PCI_IOBASE + addr);
         |                           ~~~~~~~~~~ ^
   include/asm-generic/io.h:560:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     560 |         val = __le16_to_cpu((__le16 __force)__raw_readw(PCI_IOBASE + addr));
         |                                                         ~~~~~~~~~~ ^
   include/uapi/linux/byteorder/little_endian.h:37:51: note: expanded from macro '__le16_to_cpu'
      37 | #define __le16_to_cpu(x) ((__force __u16)(__le16)(x))
         |                                                   ^
   In file included from drivers/media/platform/arm/mali-c55/mali-c55-params.c:14:
   In file included from include/media/videobuf2-core.h:18:
   In file included from include/linux/dma-buf.h:16:
   In file included from include/linux/iosys-map.h:10:
   In file included from include/linux/io.h:13:
   In file included from arch/hexagon/include/asm/io.h:328:
   include/asm-generic/io.h:573:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     573 |         val = __le32_to_cpu((__le32 __force)__raw_readl(PCI_IOBASE + addr));
         |                                                         ~~~~~~~~~~ ^
   include/uapi/linux/byteorder/little_endian.h:35:51: note: expanded from macro '__le32_to_cpu'
      35 | #define __le32_to_cpu(x) ((__force __u32)(__le32)(x))
         |                                                   ^
   In file included from drivers/media/platform/arm/mali-c55/mali-c55-params.c:14:
   In file included from include/media/videobuf2-core.h:18:
   In file included from include/linux/dma-buf.h:16:
   In file included from include/linux/iosys-map.h:10:
   In file included from include/linux/io.h:13:
   In file included from arch/hexagon/include/asm/io.h:328:
   include/asm-generic/io.h:584:33: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     584 |         __raw_writeb(value, PCI_IOBASE + addr);
         |                             ~~~~~~~~~~ ^
   include/asm-generic/io.h:594:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     594 |         __raw_writew((u16 __force)cpu_to_le16(value), PCI_IOBASE + addr);
         |                                                       ~~~~~~~~~~ ^
   include/asm-generic/io.h:604:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     604 |         __raw_writel((u32 __force)cpu_to_le32(value), PCI_IOBASE + addr);
         |                                                       ~~~~~~~~~~ ^
   In file included from drivers/media/platform/arm/mali-c55/mali-c55-params.c:14:
   In file included from include/media/videobuf2-core.h:18:
   In file included from include/linux/dma-buf.h:19:
   In file included from include/linux/scatterlist.h:8:
   In file included from include/linux/mm.h:2208:
   include/linux/vmstat.h:522:36: warning: arithmetic between different enumeration types ('enum node_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
     522 |         return node_stat_name(NR_LRU_BASE + lru) + 3; // skip "nr_"
         |                               ~~~~~~~~~~~ ^ ~~~
>> drivers/media/platform/arm/mali-c55/mali-c55-params.c:505:4: warning: format specifies type 'unsigned long' but the argument has type 'size_t' (aka 'unsigned int') [-Wformat]
     504 |                 dev_dbg(mali_c55->dev, "Invalid parameters buffer size %lu\n",
         |                                                                        ~~~
         |                                                                        %zu
     505 |                         config->total_size);
         |                         ^~~~~~~~~~~~~~~~~~
   include/linux/dev_printk.h:155:39: note: expanded from macro 'dev_dbg'
     155 |         dynamic_dev_dbg(dev, dev_fmt(fmt), ##__VA_ARGS__)
         |                                      ~~~     ^~~~~~~~~~~
   include/linux/dynamic_debug.h:274:19: note: expanded from macro 'dynamic_dev_dbg'
     274 |                            dev, fmt, ##__VA_ARGS__)
         |                                 ~~~    ^~~~~~~~~~~
   include/linux/dynamic_debug.h:250:59: note: expanded from macro '_dynamic_func_call'
     250 |         _dynamic_func_call_cls(_DPRINTK_CLASS_DFLT, fmt, func, ##__VA_ARGS__)
         |                                                                  ^~~~~~~~~~~
   include/linux/dynamic_debug.h:248:65: note: expanded from macro '_dynamic_func_call_cls'
     248 |         __dynamic_func_call_cls(__UNIQUE_ID(ddebug), cls, fmt, func, ##__VA_ARGS__)
         |                                                                        ^~~~~~~~~~~
   include/linux/dynamic_debug.h:224:15: note: expanded from macro '__dynamic_func_call_cls'
     224 |                 func(&id, ##__VA_ARGS__);                       \
         |                             ^~~~~~~~~~~
   8 warnings generated.


vim +505 drivers/media/platform/arm/mali-c55/mali-c55-params.c

   480	
   481	void mali_c55_params_write_config(struct mali_c55 *mali_c55)
   482	{
   483		struct mali_c55_params *params = &mali_c55->params;
   484		enum vb2_buffer_state state = VB2_BUF_STATE_DONE;
   485		struct mali_c55_params_buffer *config;
   486		struct mali_c55_buffer *buf;
   487		size_t block_offset = 0;
   488	
   489		spin_lock(&params->buffers.lock);
   490	
   491		buf = list_first_entry_or_null(&params->buffers.queue,
   492					       struct mali_c55_buffer, queue);
   493		if (buf)
   494			list_del(&buf->queue);
   495		spin_unlock(&params->buffers.lock);
   496	
   497		if (!buf)
   498			return;
   499	
   500		buf->vb.sequence = mali_c55->isp.frame_sequence;
   501		config = vb2_plane_vaddr(&buf->vb.vb2_buf, 0);
   502	
   503		if (config->total_size > MALI_C55_PARAMS_MAX_SIZE) {
   504			dev_dbg(mali_c55->dev, "Invalid parameters buffer size %lu\n",
 > 505				config->total_size);
   506			state = VB2_BUF_STATE_ERROR;
   507			goto err_buffer_done;
   508		}
   509	
   510		/* Walk the list of parameter blocks and process them. */
   511		while (block_offset < config->total_size) {
   512			const struct mali_c55_block_handler *block_handler;
   513			struct mali_c55_params_block_header *block;
   514	
   515			block = (struct mali_c55_params_block_header *)
   516				 &config->data[block_offset];
   517	
   518			if (block->type >= MALI_C55_PARAM_BLOCK_SENTINEL) {
   519				dev_dbg(mali_c55->dev, "Invalid parameters block type\n");
   520				state = VB2_BUF_STATE_ERROR;
   521				goto err_buffer_done;
   522			}
   523	
   524			block_handler = &mali_c55_block_handlers[block->type];
   525			if (block->size != block_handler->size) {
   526				dev_dbg(mali_c55->dev, "Invalid parameters block size\n");
   527				state = VB2_BUF_STATE_ERROR;
   528				goto err_buffer_done;
   529			}
   530	
   531			block_handler->handler(mali_c55, block);
   532	
   533			block_offset += block->size;
   534		}
   535	
   536	err_buffer_done:
   537		vb2_buffer_done(&buf->vb.vb2_buf, state);
   538	}
   539	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH v5 15/16] media: platform: Add mali-c55 parameters video node

[kernel test robot]

Hi Daniel,

kernel test robot noticed the following build warnings:

[auto build test WARNING on media-tree/master]
[cannot apply to linuxtv-media-stage/master sailus-media-tree/master linus/master sailus-media-tree/streams v6.10-rc1 next-20240529]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Daniel-Scally/media-uapi-Add-MEDIA_BUS_FMT_RGB202020_1X60-format-code/20240529-233239
base:   git://linuxtv.org/media_tree.git master
patch link:    https://lore.kernel.org/r/20240529152858.183799-16-dan.scally%40ideasonboard.com
patch subject: [PATCH v5 15/16] media: platform: Add mali-c55 parameters video node
config: nios2-allyesconfig (https://download.01.org/0day-ci/archive/20240530/202405302048.Twi0r71r-lkp@intel.com/config)
compiler: nios2-linux-gcc (GCC) 13.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240530/202405302048.Twi0r71r-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202405302048.Twi0r71r-lkp@intel.com/

All warnings (new ones prefixed by >>):

   In file included from include/linux/printk.h:566,
                    from include/asm-generic/bug.h:22,
                    from ./arch/nios2/include/generated/asm/bug.h:1,
                    from include/linux/bug.h:5,
                    from include/media/media-entity.h:15,
                    from drivers/media/platform/arm/mali-c55/mali-c55-params.c:9:
   drivers/media/platform/arm/mali-c55/mali-c55-params.c: In function 'mali_c55_params_write_config':
>> drivers/media/platform/arm/mali-c55/mali-c55-params.c:504:40: warning: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'size_t' {aka 'unsigned int'} [-Wformat=]
     504 |                 dev_dbg(mali_c55->dev, "Invalid parameters buffer size %lu\n",
         |                                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   include/linux/dynamic_debug.h:224:29: note: in definition of macro '__dynamic_func_call_cls'
     224 |                 func(&id, ##__VA_ARGS__);                       \
         |                             ^~~~~~~~~~~
   include/linux/dynamic_debug.h:250:9: note: in expansion of macro '_dynamic_func_call_cls'
     250 |         _dynamic_func_call_cls(_DPRINTK_CLASS_DFLT, fmt, func, ##__VA_ARGS__)
         |         ^~~~~~~~~~~~~~~~~~~~~~
   include/linux/dynamic_debug.h:273:9: note: in expansion of macro '_dynamic_func_call'
     273 |         _dynamic_func_call(fmt, __dynamic_dev_dbg,              \
         |         ^~~~~~~~~~~~~~~~~~
   include/linux/dev_printk.h:155:9: note: in expansion of macro 'dynamic_dev_dbg'
     155 |         dynamic_dev_dbg(dev, dev_fmt(fmt), ##__VA_ARGS__)
         |         ^~~~~~~~~~~~~~~
   include/linux/dev_printk.h:155:30: note: in expansion of macro 'dev_fmt'
     155 |         dynamic_dev_dbg(dev, dev_fmt(fmt), ##__VA_ARGS__)
         |                              ^~~~~~~
   drivers/media/platform/arm/mali-c55/mali-c55-params.c:504:17: note: in expansion of macro 'dev_dbg'
     504 |                 dev_dbg(mali_c55->dev, "Invalid parameters buffer size %lu\n",
         |                 ^~~~~~~
   drivers/media/platform/arm/mali-c55/mali-c55-params.c:504:74: note: format string is defined here
     504 |                 dev_dbg(mali_c55->dev, "Invalid parameters buffer size %lu\n",
         |                                                                        ~~^
         |                                                                          |
         |                                                                          long unsigned int
         |                                                                        %u


vim +504 drivers/media/platform/arm/mali-c55/mali-c55-params.c

   480	
   481	void mali_c55_params_write_config(struct mali_c55 *mali_c55)
   482	{
   483		struct mali_c55_params *params = &mali_c55->params;
   484		enum vb2_buffer_state state = VB2_BUF_STATE_DONE;
   485		struct mali_c55_params_buffer *config;
   486		struct mali_c55_buffer *buf;
   487		size_t block_offset = 0;
   488	
   489		spin_lock(&params->buffers.lock);
   490	
   491		buf = list_first_entry_or_null(&params->buffers.queue,
   492					       struct mali_c55_buffer, queue);
   493		if (buf)
   494			list_del(&buf->queue);
   495		spin_unlock(&params->buffers.lock);
   496	
   497		if (!buf)
   498			return;
   499	
   500		buf->vb.sequence = mali_c55->isp.frame_sequence;
   501		config = vb2_plane_vaddr(&buf->vb.vb2_buf, 0);
   502	
   503		if (config->total_size > MALI_C55_PARAMS_MAX_SIZE) {
 > 504			dev_dbg(mali_c55->dev, "Invalid parameters buffer size %lu\n",
   505				config->total_size);
   506			state = VB2_BUF_STATE_ERROR;
   507			goto err_buffer_done;
   508		}
   509	
   510		/* Walk the list of parameter blocks and process them. */
   511		while (block_offset < config->total_size) {
   512			const struct mali_c55_block_handler *block_handler;
   513			struct mali_c55_params_block_header *block;
   514	
   515			block = (struct mali_c55_params_block_header *)
   516				 &config->data[block_offset];
   517	
   518			if (block->type >= MALI_C55_PARAM_BLOCK_SENTINEL) {
   519				dev_dbg(mali_c55->dev, "Invalid parameters block type\n");
   520				state = VB2_BUF_STATE_ERROR;
   521				goto err_buffer_done;
   522			}
   523	
   524			block_handler = &mali_c55_block_handlers[block->type];
   525			if (block->size != block_handler->size) {
   526				dev_dbg(mali_c55->dev, "Invalid parameters block size\n");
   527				state = VB2_BUF_STATE_ERROR;
   528				goto err_buffer_done;
   529			}
   530	
   531			block_handler->handler(mali_c55, block);
   532	
   533			block_offset += block->size;
   534		}
   535	
   536	err_buffer_done:
   537		vb2_buffer_done(&buf->vb.vb2_buf, state);
   538	}
   539	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Laurent Pinchart]

And now the second part of the review, addressing mali-c55-capture.c and
mali-c55-resizer.c. I've reviewed the code from the bottom up, so some
messages may be repeated in an order that seems weird. Sorry about that.

On Thu, May 30, 2024 at 03:15:10AM +0300, Laurent Pinchart wrote:
> On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
> > Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
> > V4L2 and Media Controller compliant and creates subdevices to manage
> > the ISP itself, its internal test pattern generator as well as the
> > crop, scaler and output format functionality for each of its two
> > output devices.
> > 
> > Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
> > Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> > ---
> > Changes in v5:
> > 
> > 	- Reworked input formats - previously we allowed representing input data
> > 	  as any 8-16 bit format. Now we only allow input data to be represented
> > 	  by the new 20-bit bayer formats, which is corrected to the equivalent
> > 	  16-bit format in RAW bypass mode.
> > 	- Stopped bypassing blocks that we haven't added supporting parameters
> > 	  for yet.
> > 	- Addressed most of Sakari's comments from the list
> > 
> > Changes not yet made in v5:
> > 
> > 	- The output pipelines can still be started and stopped independently of
> > 	  one another - I'd like to discuss that more. 
> > 	- the TPG subdev still uses .s_stream() - I need to rebase onto a tree
> > 	  with working .enable_streams() for a single-source-pad subdevice.
> > 
> > Changes in v4:
> > 
> > 	- Reworked mali_c55_update_bits() to internally perform the bit-shift
> 
> I really don't like that, it makes the code very confusing, even more so
> as it differs from regmap_update_bits().
> 
> Look at this for instance:
> 
> 	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> 
> It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
> BIT(0).
> 
> Sorry, I know it will be painful, but this change needs to be reverted.
> 
> > 	- Reworked the resizer to allow cropping during streaming
> > 	- Fixed a bug in NV12 output
> > 
> > Changes in v3:
> > 
> > 	- Mostly minor fixes suggested by Sakari
> > 	- Fixed the sequencing of vb2 buffers to be synchronised across the two
> > 	  capture devices.
> > 
> > Changes in v2:
> > 
> > 	- Clock handling
> > 	- Fixed the warnings raised by the kernel test robot
> > 
> >  drivers/media/platform/Kconfig                |   1 +
> >  drivers/media/platform/Makefile               |   1 +
> >  drivers/media/platform/arm/Kconfig            |   5 +
> >  drivers/media/platform/arm/Makefile           |   2 +
> >  drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
> >  drivers/media/platform/arm/mali-c55/Makefile  |   9 +
> >  .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
> >  .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
> >  .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
> >  .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
> >  .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
> >  .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
> >  .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
> >  .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
> >  14 files changed, 4452 insertions(+)
> >  create mode 100644 drivers/media/platform/arm/Kconfig
> >  create mode 100644 drivers/media/platform/arm/Makefile
> >  create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
> >  create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
> >  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> 
> I've skipped review of capture.c and resizer.c as I already have plenty
> of comments for the other files, and it's getting late. I'll try to
> review the rest tomorrow.
> 
> > 
> > diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> > index 2d79bfc68c15..c929169766aa 100644
> > --- a/drivers/media/platform/Kconfig
> > +++ b/drivers/media/platform/Kconfig
> > @@ -65,6 +65,7 @@ config VIDEO_MUX
> >  source "drivers/media/platform/allegro-dvt/Kconfig"
> >  source "drivers/media/platform/amlogic/Kconfig"
> >  source "drivers/media/platform/amphion/Kconfig"
> > +source "drivers/media/platform/arm/Kconfig"
> >  source "drivers/media/platform/aspeed/Kconfig"
> >  source "drivers/media/platform/atmel/Kconfig"
> >  source "drivers/media/platform/broadcom/Kconfig"
> > diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> > index da17301f7439..9a647abd5218 100644
> > --- a/drivers/media/platform/Makefile
> > +++ b/drivers/media/platform/Makefile
> > @@ -8,6 +8,7 @@
> >  obj-y += allegro-dvt/
> >  obj-y += amlogic/
> >  obj-y += amphion/
> > +obj-y += arm/
> >  obj-y += aspeed/
> >  obj-y += atmel/
> >  obj-y += broadcom/
> > diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
> > new file mode 100644
> > index 000000000000..4f0764c329c7
> > --- /dev/null
> > +++ b/drivers/media/platform/arm/Kconfig
> > @@ -0,0 +1,5 @@
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +
> > +comment "ARM media platform drivers"
> > +
> > +source "drivers/media/platform/arm/mali-c55/Kconfig"
> > diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
> > new file mode 100644
> > index 000000000000..8cc4918725ef
> > --- /dev/null
> > +++ b/drivers/media/platform/arm/Makefile
> > @@ -0,0 +1,2 @@
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +obj-y += mali-c55/
> > diff --git a/drivers/media/platform/arm/mali-c55/Kconfig b/drivers/media/platform/arm/mali-c55/Kconfig
> > new file mode 100644
> > index 000000000000..602085e28b01
> > --- /dev/null
> > +++ b/drivers/media/platform/arm/mali-c55/Kconfig
> > @@ -0,0 +1,18 @@
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +config VIDEO_MALI_C55
> > +	tristate "ARM Mali-C55 Image Signal Processor driver"
> > +	depends on V4L_PLATFORM_DRIVERS
> > +	depends on VIDEO_DEV && OF
> > +	depends on ARCH_VEXPRESS || COMPILE_TEST
> > +	select MEDIA_CONTROLLER
> > +	select VIDEO_V4L2_SUBDEV_API
> > +	select VIDEOBUF2_DMA_CONTIG
> > +	select VIDEOBUF2_VMALLOC
> > +	select V4L2_FWNODE
> > +	select GENERIC_PHY_MIPI_DPHY
> 
> Alphabetical order ?
> 
> > +	default n
> 
> That's the default, you don't have to specify ti.
> 
> > +	help
> > +	  Enable this to support Arm's Mali-C55 Image Signal Processor.
> > +
> > +	  To compile this driver as a module, choose M here: the module
> > +	  will be called mali-c55.
> > diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
> > new file mode 100644
> > index 000000000000..77dcb2fbf0f4
> > --- /dev/null
> > +++ b/drivers/media/platform/arm/mali-c55/Makefile
> > @@ -0,0 +1,9 @@
> > +# SPDX-License-Identifier: GPL-2.0
> > +
> > +mali-c55-y := mali-c55-capture.o \
> > +	      mali-c55-core.o \
> > +	      mali-c55-isp.o \
> > +	      mali-c55-tpg.o \
> > +	      mali-c55-resizer.o
> 
> Alphabetical order here too.
> 
> > +
> > +obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
> > diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> > new file mode 100644
> > index 000000000000..1d539ac9c498
> > --- /dev/null
> > +++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> > @@ -0,0 +1,951 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * ARM Mali-C55 ISP Driver - Video capture devices
> > + *
> > + * Copyright (C) 2024 Ideas on Board Oy
> > + */
> > +
> > +#include <linux/cleanup.h>
> > +#include <linux/minmax.h>
> > +#include <linux/pm_runtime.h>
> > +#include <linux/string.h>
> > +#include <linux/videodev2.h>
> > +
> > +#include <media/v4l2-dev.h>
> > +#include <media/v4l2-event.h>
> > +#include <media/v4l2-ioctl.h>
> > +#include <media/v4l2-subdev.h>
> > +#include <media/videobuf2-core.h>
> > +#include <media/videobuf2-dma-contig.h>
> > +
> > +#include "mali-c55-common.h"
> > +#include "mali-c55-registers.h"
> > +
> > +static const struct mali_c55_fmt mali_c55_fmts[] = {
> > +	/*
> > +	 * This table is missing some entries which need further work or
> > +	 * investigation:
> > +	 *
> > +	 * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
> > +	 * Base mode 5 is "Generic Data"
> > +	 * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
> > +	 * Base mode 9 seems to have no V4L2 equivalent
> > +	 * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
> > +	 * equivalent
> > +	 */
> > +	{
> > +		.fourcc = V4L2_PIX_FMT_ARGB2101010,
> > +		.mbus_codes = {
> > +			MEDIA_BUS_FMT_RGB121212_1X36,
> > +			MEDIA_BUS_FMT_RGB202020_1X60,
> > +		},
> > +		.is_raw = false,
> > +		.registers = {
> > +			.base_mode = MALI_C55_OUTPUT_A2R10G10B10,
> > +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> > +		}
> > +	},
> > +	{
> > +		.fourcc = V4L2_PIX_FMT_RGB565,
> > +		.mbus_codes = {
> > +			MEDIA_BUS_FMT_RGB121212_1X36,
> > +			MEDIA_BUS_FMT_RGB202020_1X60,
> > +		},
> > +		.is_raw = false,
> > +		.registers = {
> > +			.base_mode = MALI_C55_OUTPUT_RGB565,
> > +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> > +		}
> > +	},
> > +	{
> > +		.fourcc = V4L2_PIX_FMT_BGR24,
> > +		.mbus_codes = {
> > +			MEDIA_BUS_FMT_RGB121212_1X36,
> > +			MEDIA_BUS_FMT_RGB202020_1X60,
> > +		},
> > +		.is_raw = false,
> > +		.registers = {
> > +			.base_mode = MALI_C55_OUTPUT_RGB24,
> > +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> > +		}
> > +	},
> > +	{
> > +		.fourcc = V4L2_PIX_FMT_YUYV,
> > +		.mbus_codes = {
> > +			MEDIA_BUS_FMT_YUV10_1X30,
> > +		},
> > +		.is_raw = false,
> > +		.registers = {
> > +			.base_mode = MALI_C55_OUTPUT_YUY2,
> > +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> > +		}
> > +	},
> > +	{
> > +		.fourcc = V4L2_PIX_FMT_UYVY,
> > +		.mbus_codes = {
> > +			MEDIA_BUS_FMT_YUV10_1X30,
> > +		},
> > +		.is_raw = false,
> > +		.registers = {
> > +			.base_mode = MALI_C55_OUTPUT_UYVY,
> > +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> > +		}
> > +	},
> > +	{
> > +		.fourcc = V4L2_PIX_FMT_Y210,
> > +		.mbus_codes = {
> > +			MEDIA_BUS_FMT_YUV10_1X30,
> > +		},
> > +		.is_raw = false,
> > +		.registers = {
> > +			.base_mode = MALI_C55_OUTPUT_Y210,
> > +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> > +		}
> > +	},
> > +	/*
> > +	 * This is something of a hack, the ISP thinks it's running NV12M but
> > +	 * by setting uv_plane = 0 we simply discard that planes and only output
> > +	 * the Y-plane.
> > +	 */
> > +	{
> > +		.fourcc = V4L2_PIX_FMT_GREY,
> > +		.mbus_codes = {
> > +			MEDIA_BUS_FMT_YUV10_1X30,
> > +		},
> > +		.is_raw = false,
> > +		.registers = {
> > +			.base_mode = MALI_C55_OUTPUT_NV12_21,
> > +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> > +		}
> > +	},
> > +	{
> > +		.fourcc = V4L2_PIX_FMT_NV12M,
> > +		.mbus_codes = {
> > +			MEDIA_BUS_FMT_YUV10_1X30,
> > +		},
> > +		.is_raw = false,
> > +		.registers = {
> > +			.base_mode = MALI_C55_OUTPUT_NV12_21,
> > +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
> > +		}
> > +	},
> > +	{
> > +		.fourcc = V4L2_PIX_FMT_NV21M,
> > +		.mbus_codes = {
> > +			MEDIA_BUS_FMT_YUV10_1X30,
> > +		},
> > +		.is_raw = false,
> > +		.registers = {
> > +			.base_mode = MALI_C55_OUTPUT_NV12_21,
> > +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
> > +		}
> > +	},
> > +	/*
> > +	 * RAW uncompressed formats are all packed in 16 bpp.
> > +	 * TODO: Expand this list to encompass all possible RAW formats.
> > +	 */
> > +	{
> > +		.fourcc = V4L2_PIX_FMT_SRGGB16,
> > +		.mbus_codes = {
> > +			MEDIA_BUS_FMT_SRGGB16_1X16,
> > +		},
> > +		.is_raw = true,
> > +		.registers = {
> > +			.base_mode = MALI_C55_OUTPUT_RAW16,
> > +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> > +		}
> > +	},
> > +	{
> > +		.fourcc = V4L2_PIX_FMT_SBGGR16,
> > +		.mbus_codes = {
> > +			MEDIA_BUS_FMT_SBGGR16_1X16,
> > +		},
> > +		.is_raw = true,
> > +		.registers = {
> > +			.base_mode = MALI_C55_OUTPUT_RAW16,
> > +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> > +		}
> > +	},
> > +	{
> > +		.fourcc = V4L2_PIX_FMT_SGBRG16,
> > +		.mbus_codes = {
> > +			MEDIA_BUS_FMT_SGBRG16_1X16,
> > +		},
> > +		.is_raw = true,
> > +		.registers = {
> > +			.base_mode = MALI_C55_OUTPUT_RAW16,
> > +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> > +		}
> > +	},
> > +	{
> > +		.fourcc = V4L2_PIX_FMT_SGRBG16,
> > +		.mbus_codes = {
> > +			MEDIA_BUS_FMT_SGRBG16_1X16,
> > +		},
> > +		.is_raw = true,
> > +		.registers = {
> > +			.base_mode = MALI_C55_OUTPUT_RAW16,
> > +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> > +		}
> > +	},
> > +};
> > +
> > +static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
> > +					       u32 code)
> > +{
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
> > +		if (fmt->mbus_codes[i] == code)
> > +			return true;
> > +	}
> > +
> > +	return false;
> > +}
> > +
> > +bool mali_c55_format_is_raw(unsigned int mbus_code)
> > +{
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> > +		if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
> > +			return mali_c55_fmts[i].is_raw;
> > +	}
> > +
> > +	return false;
> > +}
> > +
> > +static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
> > +{
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> > +		if (mali_c55_fmts[i].fourcc == pixelformat)
> > +			return &mali_c55_fmts[i];
> > +	}
> > +
> > +	/*
> > +	 * If we find no matching pixelformat, we'll just default to the first
> > +	 * one for now.
> > +	 */
> > +
> > +	return &mali_c55_fmts[0];
> > +}
> > +
> > +static const char * const capture_device_names[] = {
> > +	"mali-c55 fr",
> > +	"mali-c55 ds",
> > +	"mali-c55 3a stats",
> > +	"mali-c55 params",

The last two entries are not used AFAICT, neither here, nor in
subsequent patches.

> > +};
> > +
> > +static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
> > +{
> > +	if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
> > +		return capture_device_names[0];
> > +
> > +	if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
> > +		return capture_device_names[1];
> > +
> > +	return "params/stat not supported yet";
> > +}

Use cap_dev->vdev.name instead of mali_c55_cap_dev_to_name(cap_dev) and
drop this function.

> > +
> > +static int mali_c55_link_validate(struct media_link *link)
> > +{
> > +	struct video_device *vdev =
> > +		media_entity_to_video_device(link->sink->entity);
> > +	struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
> > +	struct v4l2_subdev *sd =
> > +		media_entity_to_v4l2_subdev(link->source->entity);
> > +	const struct v4l2_pix_format_mplane *pix_mp;
> > +	const struct mali_c55_fmt *cap_fmt;
> > +	struct v4l2_subdev_format sd_fmt = {
> > +		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
> > +		.pad = link->source->index,
> > +	};
> > +	int ret;
> > +
> > +	ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
> > +	if (ret)
> > +		return ret;
> > +
> > +	pix_mp = &cap_dev->mode.pix_mp;
> > +	cap_fmt = cap_dev->mode.capture_fmt;
> > +
> > +	if (sd_fmt.format.width != pix_mp->width ||
> > +	    sd_fmt.format.height != pix_mp->height) {
> > +		dev_dbg(cap_dev->mali_c55->dev,
> > +			"link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
> > +			link->source->entity->name, link->source->index,
> > +			link->sink->entity->name, link->sink->index,
> > +			sd_fmt.format.width, sd_fmt.format.height,
> > +			pix_mp->width, pix_mp->height);
> > +		return -EPIPE;
> > +	}
> > +
> > +	if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
> > +		dev_dbg(cap_dev->mali_c55->dev,
> > +			"link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format %p4cc\n",
> > +			link->source->entity->name, link->source->index,
> > +			link->sink->entity->name, link->sink->index,
> > +			sd_fmt.format.code, &pix_mp->pixelformat);
> > +		return -EPIPE;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static const struct media_entity_operations mali_c55_media_ops = {
> > +	.link_validate = mali_c55_link_validate,
> > +};
> > +
> > +static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
> > +				    unsigned int *num_planes, unsigned int sizes[],
> > +				    struct device *alloc_devs[])
> > +{
> > +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> > +	unsigned int i;
> > +
> > +	if (*num_planes) {
> > +		if (*num_planes != cap_dev->mode.pix_mp.num_planes)
> > +			return -EINVAL;
> > +
> > +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> > +			if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
> > +				return -EINVAL;
> > +	} else {
> > +		*num_planes = cap_dev->mode.pix_mp.num_planes;
> > +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> > +			sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static void mali_c55_buf_queue(struct vb2_buffer *vb)
> > +{
> > +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
> > +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> > +	struct mali_c55_buffer *buf = container_of(vbuf,
> > +						   struct mali_c55_buffer, vb);
> > +	unsigned int i;
> > +
> > +	buf->plane_done[MALI_C55_PLANE_Y] = false;
> > +
> > +	/*
> > +	 * If we're in a single-plane format we flag the other plane as done
> > +	 * already so it's dequeued appropriately later
> > +	 */
> > +	buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
> > +
> > +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
> > +		unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
> > +
> > +		vb2_set_plane_payload(vb, i, size);
> > +	}
> > +
> > +	spin_lock(&cap_dev->buffers.lock);
> > +	list_add_tail(&buf->queue, &cap_dev->buffers.queue);
> > +	spin_unlock(&cap_dev->buffers.lock);
> > +}
> > +
> > +static int mali_c55_buf_init(struct vb2_buffer *vb)
> > +{
> > +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
> > +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> > +	struct mali_c55_buffer *buf = container_of(vbuf,
> > +						   struct mali_c55_buffer, vb);
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> > +		buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
> > +
> > +	return 0;
> > +}
> > +
> > +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
> > +{
> > +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> > +
> > +	guard(spinlock)(&cap_dev->buffers.lock);
> > +
> > +	cap_dev->buffers.curr = cap_dev->buffers.next;
> > +	cap_dev->buffers.next = NULL;
> > +
> > +	if (!list_empty(&cap_dev->buffers.queue)) {
> > +		struct v4l2_pix_format_mplane *pix_mp;
> > +		const struct v4l2_format_info *info;
> > +		u32 *addrs;
> > +
> > +		pix_mp = &cap_dev->mode.pix_mp;
> > +		info = v4l2_format_info(pix_mp->pixelformat);
> > +
> > +		mali_c55_update_bits(mali_c55,
> > +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> > +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
> > +		if (cap_dev->mode.capture_fmt->registers.uv_plane)
> > +			mali_c55_update_bits(mali_c55,
> > +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> > +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
> > +
> > +		cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
> > +							 struct mali_c55_buffer,
> > +							 queue);
> > +		list_del(&cap_dev->buffers.next->queue);
> > +
> > +		addrs = cap_dev->buffers.next->addrs;
> > +		mali_c55_write(mali_c55,
> > +			MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
> > +			addrs[MALI_C55_PLANE_Y]);
> > +		mali_c55_write(mali_c55,
> > +			MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
> > +			addrs[MALI_C55_PLANE_UV]);
> > +		mali_c55_write(mali_c55,
> > +			MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
> > +			pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
> > +		mali_c55_write(mali_c55,
> > +			MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
> > +			pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
> > +			/ info->hdiv);
> > +	} else {
> > +		/*
> > +		 * If we underflow then we can tell the ISP that we don't want
> > +		 * to write out the next frame.
> > +		 */
> > +		mali_c55_update_bits(mali_c55,
> > +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> > +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> > +		mali_c55_update_bits(mali_c55,
> > +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> > +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> > +	}
> > +}
> > +
> > +static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
> > +				   unsigned int framecount)
> > +{
> > +	curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> > +	curr_buf->vb.field = V4L2_FIELD_NONE;

The could be set already when the buffer is queued.

> > +	curr_buf->vb.sequence = framecount;
> > +	vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
> > +}
> > +
> > +/**
> > + * mali_c55_set_plane_done - mark the plane as written and process the buffer if
> > + *			     both planes are finished.
> > + * @cap_dev:  pointer to the fr or ds pipe output
> > + * @plane:    the plane to mark as completed
> > + *
> > + * The Mali C55 ISP has muliplanar outputs for some formats that come with two
> > + * separate "buffer write completed" interrupts - we need to flag each plane's
> > + * completion and check whether both planes are done - if so, complete the buf
> > + * in vb2.
> > + */
> > +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> > +			     enum mali_c55_planes plane)
> > +{
> > +	struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
> > +	struct mali_c55_buffer *curr_buf;
> > +
> > +	guard(spinlock)(&cap_dev->buffers.lock);
> > +	curr_buf = cap_dev->buffers.curr;
> > +
> > +	/*
> > +	 * This _should_ never happen. If no buffer was available from vb2 then
> > +	 * we tell the ISP not to bother writing the next frame, which means the
> > +	 * interrupts that call this function should never trigger. If it does
> > +	 * happen then one of our assumptions is horribly wrong - complain
> > +	 * loudly and do nothing.
> > +	 */
> > +	if (!curr_buf) {
> > +		dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
> > +			mali_c55_cap_dev_to_name(cap_dev), __func__);
> > +		return;
> > +	}
> > +
> > +	/* If the other plane is also done... */
> > +	if (curr_buf->plane_done[~plane & 1]) {
> > +		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> > +		cap_dev->buffers.curr = NULL;
> > +		isp->frame_sequence++;
> > +	} else {
> > +		curr_buf->plane_done[plane] = true;
> > +	}
> > +}
> > +
> > +static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
> > +{
> > +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> > +
> > +	mali_c55_update_bits(mali_c55,
> > +			     MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> > +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> > +	mali_c55_update_bits(mali_c55,
> > +			     MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> > +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> > +}
> > +
> > +static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
> > +{
> > +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> > +
> > +	/*
> > +	 * The Mali ISP can hold up to 5 buffer addresses and simply cycle
> > +	 * through them, but it's not clear to me that the vb2 queue _guarantees_
> > +	 * it will queue buffers to the driver in a fixed order, and ensuring
> > +	 * we call vb2_buffer_done() for the right buffer seems to me to add
> > +	 * pointless complexity given in multi-context mode we'd need to
> > +	 * re-write those registers every frame anyway...so we tell the ISP to
> > +	 * use a single register and update it for each frame.
> > +	 */

A single register sounds prone to error conditions. Is it at least
shadowed in the hardware, or do you have to make sure you reprogram it
during the vertical blanking only ?

I'll mostly skip buffer handling in this review, I need to first
understand how the hardware operates to make an informed opinion.

> > +	mali_c55_update_bits(mali_c55,
> > +			MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
> > +			MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
> > +	mali_c55_update_bits(mali_c55,
> > +			MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
> > +			MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
> > +
> > +	/*
> > +	 * We only queue a buffer in the streamon path if this is the first of
> > +	 * the capture devices to start streaming. If the ISP is already running
> > +	 * then we rely on the ISP_START interrupt to queue the first buffer for
> > +	 * this capture device.
> > +	 */
> > +	if (mali_c55->pipe.start_count == 1)
> > +		mali_c55_set_next_buffer(cap_dev);

I think we'll have to revisit buffer handling to make sure it's 100%
race-free.

> > +}
> > +
> > +static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
> > +					    enum vb2_buffer_state state)
> > +{
> > +	struct mali_c55_buffer *buf, *tmp;
> > +
> > +	guard(spinlock)(&cap_dev->buffers.lock);
> > +
> > +	if (cap_dev->buffers.curr) {
> > +		vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
> > +				state);
> > +		cap_dev->buffers.curr = NULL;
> > +	}
> > +
> > +	if (cap_dev->buffers.next) {
> > +		vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
> > +				state);
> > +		cap_dev->buffers.next = NULL;
> > +	}
> > +
> > +	list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
> > +		list_del(&buf->queue);
> > +		vb2_buffer_done(&buf->vb.vb2_buf, state);
> > +	}
> > +}
> > +
> > +static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
> > +{
> > +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> > +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> > +	struct mali_c55_resizer *rzr = cap_dev->rzr;
> > +	struct mali_c55_isp *isp = &mali_c55->isp;
> > +	int ret;
> > +
> > +	guard(mutex)(&isp->lock);

What's the reason for using the isp lock here and in
mali_c55_vb2_stop_streaming() ? If you need a lock that covers all video
nodes in order to synchronize start/stop, you may want to use the
graph_mutex of the media device instead.

> > +
> > +	ret = pm_runtime_resume_and_get(mali_c55->dev);
> > +	if (ret)
> > +		return ret;
> > +
> > +	ret = video_device_pipeline_start(&cap_dev->vdev,
> > +					  &cap_dev->mali_c55->pipe);
> > +	if (ret) {
> > +		dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
> > +			mali_c55_cap_dev_to_name(cap_dev));

Drop the message or make it dev_dbg() as it can be triggered by
userspace.

> > +		goto err_pm_put;
> > +	}
> > +
> > +	mali_c55_cap_dev_stream_enable(cap_dev);
> > +	mali_c55_rzr_start_stream(rzr);
> > +
> > +	/*
> > +	 * We only start the ISP if we're the only capture device that's
> > +	 * streaming. Otherwise, it'll already be active.
> > +	 */

I still think we should use link setup to indicate which video devices
userspace plans to use, and then only start when they're all started.
That includes stats and parameters buffers. We can continue this
discussion in the context of the previous version of the patch series,
or here, up to you.

> > +	if (mali_c55->pipe.start_count == 1) {
> > +		ret = mali_c55_isp_start_stream(isp);
> > +		if (ret)
> > +			goto err_disable_cap_dev;
> > +	}
> > +
> > +	return 0;
> > +
> > +err_disable_cap_dev:
> > +	mali_c55_cap_dev_stream_disable(cap_dev);
> > +	video_device_pipeline_stop(&cap_dev->vdev);
> > +err_pm_put:
> > +	pm_runtime_put(mali_c55->dev);
> > +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
> > +
> > +	return ret;
> > +}
> > +
> > +static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
> > +{
> > +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> > +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> > +	struct mali_c55_resizer *rzr = cap_dev->rzr;
> > +	struct mali_c55_isp *isp = &mali_c55->isp;
> > +
> > +	guard(mutex)(&isp->lock);
> > +
> > +	/*
> > +	 * If one of the other capture nodes is streaming, we shouldn't
> > +	 * disable the ISP here.
> > +	 */
> > +	if (mali_c55->pipe.start_count == 1)
> > +		mali_c55_isp_stop_stream(&mali_c55->isp);
> > +
> > +	mali_c55_rzr_stop_stream(rzr);
> > +	mali_c55_cap_dev_stream_disable(cap_dev);
> > +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
> > +	video_device_pipeline_stop(&cap_dev->vdev);
> > +	pm_runtime_put(mali_c55->dev);

I think runtime PM autosuspend would be very useful, as it will ensure
that stop-reconfigure-start cycles get handled as efficiently as
possible without powering the device down. It could be done on top as a
separate patch.

> > +}
> > +
> > +static const struct vb2_ops mali_c55_vb2_ops = {
> > +	.queue_setup		= &mali_c55_vb2_queue_setup,
> > +	.buf_queue		= &mali_c55_buf_queue,
> > +	.buf_init		= &mali_c55_buf_init,
> > +	.wait_prepare		= vb2_ops_wait_prepare,
> > +	.wait_finish		= vb2_ops_wait_finish,
> > +	.start_streaming	= &mali_c55_vb2_start_streaming,
> > +	.stop_streaming		= &mali_c55_vb2_stop_streaming,
> > +};
> > +
> > +static const struct v4l2_file_operations mali_c55_v4l2_fops = {
> > +	.owner = THIS_MODULE,
> > +	.unlocked_ioctl = video_ioctl2,
> > +	.open = v4l2_fh_open,
> > +	.release = vb2_fop_release,
> > +	.poll = vb2_fop_poll,
> > +	.mmap = vb2_fop_mmap,
> > +};
> > +
> > +static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
> > +{
> > +	const struct mali_c55_fmt *capture_format;
> > +	const struct v4l2_format_info *info;
> > +	struct v4l2_plane_pix_format *plane;
> > +	unsigned int i;
> > +
> > +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
> > +	pix_mp->pixelformat = capture_format->fourcc;
> > +
> > +	pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
> > +			      MALI_C55_MAX_WIDTH);
> > +	pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
> > +			       MALI_C55_MAX_HEIGHT);

Ah, these clamps are right :-)

> > +
> > +	pix_mp->field = V4L2_FIELD_NONE;
> > +	pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
> > +	pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> > +	pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
> > +
> > +	info = v4l2_format_info(pix_mp->pixelformat);

This function may return NULL. That shouldn't be the case as long as it
supports all formats that the C55 driver supports, so I suppose it's
safe.

> > +	pix_mp->num_planes = info->mem_planes;
> > +	memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
> > +
> > +	pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;

Does the hardware support configurable line strides ? If so we should
support it.

> > +	pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
> > +				       * pix_mp->height;

	pix_mp->plane_fmt[0].sizeimage = pix_mp->plane_fmt[0].bytesperline
				       * pix_mp->height;

> > +
> > +	for (i = 1; i < info->comp_planes; i++) {
> > +		plane = &pix_mp->plane_fmt[i];
> > +
> > +		plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
> > +						   info->hdiv);
> > +		plane->sizeimage = DIV_ROUND_UP(
> > +					plane->bytesperline * pix_mp->height,
> > +					info->vdiv);
> > +	}
> > +
> > +	if (info->mem_planes == 1) {
> > +		for (i = 1; i < info->comp_planes; i++) {
> > +			plane = &pix_mp->plane_fmt[i];
> > +			pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
> > +		}
> > +	}

I'm wondering, could v4l2_fill_pixfmt_mp() help ? It doesn't support
configurable strides though :-S Maybe the helper could be improved, if
it's close enough to what you need ?

> > +}
> > +
> > +static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
> > +					   struct v4l2_format *f)
> > +{
> > +	mali_c55_try_fmt(&f->fmt.pix_mp);
> > +
> > +	return 0;
> > +}
> > +
> > +static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
> > +				struct v4l2_pix_format_mplane *pix_mp)
> > +{
> > +	const struct mali_c55_fmt *capture_format;
> > +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> > +	const struct v4l2_format_info *info;
> > +
> > +	mali_c55_try_fmt(pix_mp);
> > +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
> > +	info = v4l2_format_info(pix_mp->pixelformat);
> > +	if (WARN_ON(!info))
> > +		return;
> > +
> > +	mali_c55_write(mali_c55,
> > +		       MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> > +		       capture_format->registers.base_mode);
> > +	mali_c55_write(mali_c55,
> > +		       MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
> > +		       MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
> > +		       MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));

Could the register writes be moved to stream start time ?

> > +
> > +	if (info->mem_planes > 1) {
> > +		mali_c55_write(mali_c55,
> > +			       MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> > +			       capture_format->registers.base_mode);
> > +		mali_c55_update_bits(mali_c55,
> > +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> > +				MALI_C55_WRITER_SUBMODE_MASK,
> > +				capture_format->registers.uv_plane);
> > +
> > +		mali_c55_write(mali_c55,
> > +			MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
> > +			MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
> > +			MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
> > +	}
> > +
> > +	if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
> > +		/*
> > +		 * TODO: Figure out the colour matrix coefficients and calculate
> > +		 * and write them here.
> > +		 */

Ideally they should also be exposed directly to userspace as ISP
parameters. I would probably go as far as saying that they should come
directly from userspace, and not derived from the colorspace fields.

> > +
> > +		mali_c55_write(mali_c55,
> > +			       MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> > +			       MALI_C55_CS_CONV_MATRIX_MASK);
> > +
> > +		if (info->hdiv > 1)
> > +			mali_c55_update_bits(mali_c55,
> > +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> > +				MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
> > +		if (info->vdiv > 1)
> > +			mali_c55_update_bits(mali_c55,
> > +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> > +				MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
> > +		if (info->hdiv > 1 || info->vdiv > 1)
> > +			mali_c55_update_bits(mali_c55,
> > +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> > +				MALI_C55_CS_CONV_FILTER_MASK, 0x01);
> > +	}
> > +
> > +	cap_dev->mode.pix_mp = *pix_mp;
> > +	cap_dev->mode.capture_fmt = capture_format;
> > +}
> > +
> > +static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
> > +					 struct v4l2_format *f)
> > +{
> > +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> > +
> > +	if (vb2_is_busy(&cap_dev->queue))
> > +		return -EBUSY;
> > +
> > +	mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
> > +					 struct v4l2_format *f)
> > +{
> > +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> > +
> > +	f->fmt.pix_mp = cap_dev->mode.pix_mp;
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
> > +					    struct v4l2_fmtdesc *f)
> > +{
> > +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> > +	unsigned int j = 0;
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> > +		if (f->mbus_code &&
> > +		    !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
> > +						       f->mbus_code))

Small indentation mistake.

> > +			continue;
> > +
> > +		/* Downscale pipe can't output RAW formats */
> > +		if (mali_c55_fmts[i].is_raw &&
> > +		    cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
> > +			continue;
> > +
> > +		if (j++ == f->index) {
> > +			f->pixelformat = mali_c55_fmts[i].fourcc;
> > +			return 0;
> > +		}
> > +	}
> > +
> > +	return -EINVAL;
> > +}
> > +
> > +static int mali_c55_querycap(struct file *file, void *fh,
> > +			     struct v4l2_capability *cap)
> > +{
> > +	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
> > +	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
> > +
> > +	return 0;
> > +}
> > +
> > +static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
> > +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
> > +	.vidioc_querybuf = vb2_ioctl_querybuf,
> > +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
> > +	.vidioc_qbuf = vb2_ioctl_qbuf,
> > +	.vidioc_expbuf = vb2_ioctl_expbuf,
> > +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
> > +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> > +	.vidioc_streamon = vb2_ioctl_streamon,
> > +	.vidioc_streamoff = vb2_ioctl_streamoff,
> > +	.vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
> > +	.vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
> > +	.vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
> > +	.vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
> > +	.vidioc_querycap = mali_c55_querycap,
> > +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
> > +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
> > +};
> > +
> > +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
> > +{
> > +	struct v4l2_pix_format_mplane pix_mp;
> > +	struct mali_c55_cap_dev *cap_dev;
> > +	struct video_device *vdev;
> > +	struct vb2_queue *vb2q;
> > +	unsigned int i;
> > +	int ret;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {

Moving the inner content to a separate mali_c55_register_capture_dev()
function would increase readability I think, and remove usage of gotos.
I would probably do the same for unregistration too, for consistency.

> > +		cap_dev = &mali_c55->cap_devs[i];
> > +		vdev = &cap_dev->vdev;
> > +		vb2q = &cap_dev->queue;
> > +
> > +		/*
> > +		 * The downscale output pipe is an optional block within the ISP
> > +		 * so we need to check whether it's actually been fitted or not.
> > +		 */
> > +
> > +		if (i == MALI_C55_CAP_DEV_DS &&
> > +		    !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
> > +			continue;

Given that there's only two capture devices, and one is optional, when
moving the inner code to a separate function you could unroll the loop.
Up to you.

> > +
> > +		cap_dev->mali_c55 = mali_c55;
> > +		mutex_init(&cap_dev->lock);
> > +		INIT_LIST_HEAD(&cap_dev->buffers.queue);
> > +
> > +		switch (i) {
> > +		case MALI_C55_CAP_DEV_FR:
> > +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
> > +			cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
> > +			break;
> > +		case MALI_C55_CAP_DEV_DS:
> > +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
> > +			cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
> > +			break;
> > +		default:

That can't happen.

> > +			mutex_destroy(&cap_dev->lock);
> > +			ret = -EINVAL;
> > +			goto err_destroy_mutex;
> > +		}
> > +
> > +		cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
> > +		ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
> > +		if (ret) {
> > +			mutex_destroy(&cap_dev->lock);
> > +			goto err_destroy_mutex;
> > +		}
> > +
> > +		vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> > +		vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
> > +		vb2q->drv_priv = cap_dev;
> > +		vb2q->mem_ops = &vb2_dma_contig_memops;
> > +		vb2q->ops = &mali_c55_vb2_ops;
> > +		vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
> > +		vb2q->min_queued_buffers = 1;
> > +		vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> > +		vb2q->lock = &cap_dev->lock;
> > +		vb2q->dev = mali_c55->dev;
> > +
> > +		ret = vb2_queue_init(vb2q);
> > +		if (ret) {
> > +			dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
> > +				mali_c55_cap_dev_to_name(cap_dev));
> > +			goto err_cleanup_media_entity;
> > +		}
> > +
> > +		strscpy(cap_dev->vdev.name, capture_device_names[i],
> > +			sizeof(cap_dev->vdev.name));
> > +		vdev->release = video_device_release_empty;
> > +		vdev->fops = &mali_c55_v4l2_fops;
> > +		vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
> > +		vdev->lock = &cap_dev->lock;
> > +		vdev->v4l2_dev = &mali_c55->v4l2_dev;
> > +		vdev->queue = &cap_dev->queue;
> > +		vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> > +				    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
> > +		vdev->entity.ops = &mali_c55_media_ops;
> > +		video_set_drvdata(vdev, cap_dev);
> > +
> > +		memset(&pix_mp, 0, sizeof(pix_mp));
> > +		pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
> > +		pix_mp.width = MALI_C55_DEFAULT_WIDTH;
> > +		pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
> > +		mali_c55_set_format(cap_dev, &pix_mp);
> > +
> > +		ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
> > +		if (ret) {
> > +			dev_err(mali_c55->dev,
> > +				"%s failed to register video device\n",
> > +				mali_c55_cap_dev_to_name(cap_dev));
> > +			goto err_release_vb2q;
> > +		}
> > +	}
> > +
> > +	return 0;
> > +
> > +err_release_vb2q:
> > +	vb2_queue_release(vb2q);
> > +err_cleanup_media_entity:
> > +	media_entity_cleanup(&cap_dev->vdev.entity);
> > +err_destroy_mutex:
> > +	mutex_destroy(&cap_dev->lock);
> > +	mali_c55_unregister_capture_devs(mali_c55);
> > +
> > +	return ret;
> > +}
> > +
> > +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
> > +{
> > +	struct mali_c55_cap_dev *cap_dev;
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
> > +		cap_dev = &mali_c55->cap_devs[i];
> > +
> > +		if (!video_is_registered(&cap_dev->vdev))
> > +			continue;
> > +
> > +		vb2_video_unregister_device(&cap_dev->vdev);
> > +		media_entity_cleanup(&cap_dev->vdev.entity);
> > +		mutex_destroy(&cap_dev->lock);
> > +	}
> > +}
> > diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> > new file mode 100644
> > index 000000000000..2d0c4d152beb
> > --- /dev/null
> > +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> > @@ -0,0 +1,266 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * ARM Mali-C55 ISP Driver - Common definitions
> > + *
> > + * Copyright (C) 2024 Ideas on Board Oy
> > + */
> > +
> > +#ifndef _MALI_C55_COMMON_H
> > +#define _MALI_C55_COMMON_H
> > +
> > +#include <linux/clk.h>
> > +#include <linux/io.h>
> > +#include <linux/list.h>
> > +#include <linux/mutex.h>
> > +#include <linux/scatterlist.h>
> 
> I don't think this is needed. You're however missing spinlock.h.
> 
> > +#include <linux/videodev2.h>
> > +
> > +#include <media/media-device.h>
> > +#include <media/v4l2-async.h>
> > +#include <media/v4l2-ctrls.h>
> > +#include <media/v4l2-dev.h>
> > +#include <media/v4l2-device.h>
> > +#include <media/v4l2-subdev.h>
> > +#include <media/videobuf2-core.h>
> > +#include <media/videobuf2-v4l2.h>
> > +
> > +#define MALI_C55_DRIVER_NAME		"mali-c55"
> > +
> > +/* min and max values for the image sizes */
> > +#define MALI_C55_MIN_WIDTH		640U
> > +#define MALI_C55_MIN_HEIGHT		480U
> > +#define MALI_C55_MAX_WIDTH		8192U
> > +#define MALI_C55_MAX_HEIGHT		8192U
> > +#define MALI_C55_DEFAULT_WIDTH		1920U
> > +#define MALI_C55_DEFAULT_HEIGHT		1080U
> > +
> > +#define MALI_C55_DEFAULT_MEDIA_BUS_FMT	MEDIA_BUS_FMT_RGB121212_1X36
> > +
> > +struct mali_c55;
> > +struct mali_c55_cap_dev;
> > +struct platform_device;
> 
> You should also forward-declare
> 
> struct device;
> struct dma_chan;
> struct resource;
> 
> > +
> > +static const char * const mali_c55_clk_names[] = {
> > +	"aclk",
> > +	"hclk",
> > +};
> 
> This will end up duplicating the array in each compilation unit, not
> great. Move it to mali-c55-core.c. You use it in this file just for its
> size, replace that with a macro that defines the size, or allocate
> mali_c55.clks dynamically with devm_kcalloc().
> 
> > +
> > +enum mali_c55_interrupts {
> > +	MALI_C55_IRQ_ISP_START,
> > +	MALI_C55_IRQ_ISP_DONE,
> > +	MALI_C55_IRQ_MCM_ERROR,
> > +	MALI_C55_IRQ_BROKEN_FRAME_ERROR,
> > +	MALI_C55_IRQ_MET_AF_DONE,
> > +	MALI_C55_IRQ_MET_AEXP_DONE,
> > +	MALI_C55_IRQ_MET_AWB_DONE,
> > +	MALI_C55_IRQ_AEXP_1024_DONE,
> > +	MALI_C55_IRQ_IRIDIX_MET_DONE,
> > +	MALI_C55_IRQ_LUT_INIT_DONE,
> > +	MALI_C55_IRQ_FR_Y_DONE,
> > +	MALI_C55_IRQ_FR_UV_DONE,
> > +	MALI_C55_IRQ_DS_Y_DONE,
> > +	MALI_C55_IRQ_DS_UV_DONE,
> > +	MALI_C55_IRQ_LINEARIZATION_DONE,
> > +	MALI_C55_IRQ_RAW_FRONTEND_DONE,
> > +	MALI_C55_IRQ_NOISE_REDUCTION_DONE,
> > +	MALI_C55_IRQ_IRIDIX_DONE,
> > +	MALI_C55_IRQ_BAYER2RGB_DONE,
> > +	MALI_C55_IRQ_WATCHDOG_TIMER,
> > +	MALI_C55_IRQ_FRAME_COLLISION,
> > +	MALI_C55_IRQ_UNUSED,
> > +	MALI_C55_IRQ_DMA_ERROR,
> > +	MALI_C55_IRQ_INPUT_STOPPED,
> > +	MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
> > +	MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
> > +	MALI_C55_NUM_IRQ_BITS
> 
> Those are register bits, I think they belong to mali-c55-registers.h,
> and should probably be macros instead of an enum.
> 
> > +};
> > +
> > +enum mali_c55_isp_pads {
> > +	MALI_C55_ISP_PAD_SINK_VIDEO,
> 
> As there's a single sink pad, maybe MALI_C55_ISP_PAD_SINK ? Ah, you're
> probably preparing for ISP parameters support. It's fine.
> 
> > +	MALI_C55_ISP_PAD_SOURCE,
> 
> Then maybe this should be named MALI_C55_ISP_PAD_SOURCE_VIDEO as I
> assume there will be a stats source pad.
> 
> > +	MALI_C55_ISP_PAD_SOURCE_BYPASS,
> > +	MALI_C55_ISP_NUM_PADS,
> > +};
> > +
> > +struct mali_c55_tpg {
> > +	struct mali_c55 *mali_c55;
> > +	struct v4l2_subdev sd;
> > +	struct media_pad pad;
> > +	struct mutex lock;
> > +	struct mali_c55_tpg_ctrls {
> > +		struct v4l2_ctrl_handler handler;
> > +		struct v4l2_ctrl *test_pattern;
> 
> Set but never used. You can drop it.
> 
> > +		struct v4l2_ctrl *hblank;
> 
> Set and used only once, in the same function. You can make it a local
> variable.
> 
> > +		struct v4l2_ctrl *vblank;
> > +	} ctrls;
> > +};
> 
> I wonder if this file should be split, with mali-c55-capture.h,
> mali-c55-core.h, mali-c55-isp.h, ... I think it could increase
> readability by clearly separating the different elements. Up to you.
> 
> > +
> > +struct mali_c55_isp {
> > +	struct mali_c55 *mali_c55;
> > +	struct v4l2_subdev sd;
> > +	struct media_pad pads[MALI_C55_ISP_NUM_PADS];
> > +	struct media_pad *remote_src;
> > +	struct v4l2_async_notifier notifier;
> 
> I'm tempted to move the notifier to mali_c55, as it's related to
> components external to the whole ISP, not to the ISP subdev itself.
> Could you give it a try, to see if it could be done without any drawback
> ?
> 
> > +	struct mutex lock;
> 
> Locks require a comment to explain what they protect. Same below where
> applicable (for both mutexes and spinlocks).
> 
> > +	unsigned int frame_sequence;
> > +};
> > +
> > +enum mali_c55_resizer_ids {
> > +	MALI_C55_RZR_FR,
> > +	MALI_C55_RZR_DS,
> > +	MALI_C55_NUM_RZRS,
> 
> The usual abbreviation for "resizer" in drivers/media/ is "rsz", not
> "rzr". I would have said we can leave it as-is as changing it would be a
> bit annoying, but I then realized that "rzr" is not just unusual, it's
> actually not used at all. Would you mind applying a sed globally ? :-)
> 
> > +};
> > +
> > +enum mali_c55_rzr_pads {
> 
> Same enums/structs use abbreviations, some don't. Consistency would
> help.
> 
> > +	MALI_C55_RZR_SINK_PAD,
> > +	MALI_C55_RZR_SOURCE_PAD,
> > +	MALI_C55_RZR_SINK_BYPASS_PAD,
> > +	MALI_C55_RZR_NUM_PADS
> > +};
> > +
> > +struct mali_c55_resizer {
> > +	struct mali_c55 *mali_c55;
> > +	struct mali_c55_cap_dev *cap_dev;
> > +	enum mali_c55_resizer_ids id;
> > +	struct v4l2_subdev sd;
> > +	struct media_pad pads[MALI_C55_RZR_NUM_PADS];
> > +	unsigned int num_routes;
> > +	bool streaming;
> > +};
> > +
> > +enum mali_c55_cap_devs {
> > +	MALI_C55_CAP_DEV_FR,
> > +	MALI_C55_CAP_DEV_DS,
> > +	MALI_C55_NUM_CAP_DEVS
> > +};
> > +
> > +struct mali_c55_fmt {
> 
> mali_c55_format_info would be a better name I think, as this stores
> format information, not formats.
> 
> > +	u32 fourcc;
> > +	unsigned int mbus_codes[2];
> 
> A comment to explain why we have two media bus codes would be useful.
> You can document the whole structure if desired :-)
> 
> > +	bool is_raw;
> > +	struct mali_c55_fmt_registers {
> 
> Make it an anonymous structure, it's never used anywhere else.
> 
> > +		unsigned int base_mode;
> > +		unsigned int uv_plane;
> 
> If those are register field values, use u32 instead of unsigned int.
> 
> > +	} registers;
> 
> It's funny, we tend to abbreviate different things, I would have used
> "regs" here but written "format" in full in the structure name :-)
> 
> > +};
> > +
> > +enum mali_c55_isp_bayer_order {
> > +	MALI_C55_BAYER_ORDER_RGGB,
> > +	MALI_C55_BAYER_ORDER_GRBG,
> > +	MALI_C55_BAYER_ORDER_GBRG,
> > +	MALI_C55_BAYER_ORDER_BGGR
> 
> These are registers values too, they belong to mali-c55-registers.h.
> 
> > +};
> > +
> > +struct mali_c55_isp_fmt {
> 
> mali_c55_isp_format_info
> 
> > +	u32 code;
> > +	enum v4l2_pixel_encoding encoding;
> 
> Here you use v4l2_pixel_encoding, for mali_c55_fmt you use is_raw. Maybe
> pick the same option for both structures ?
> 
> > +	enum mali_c55_isp_bayer_order order;
> > +};
> > +
> > +enum mali_c55_planes {
> > +	MALI_C55_PLANE_Y,
> > +	MALI_C55_PLANE_UV,
> > +	MALI_C55_NUM_PLANES
> > +};
> > +
> > +struct mali_c55_buffer {
> > +	struct vb2_v4l2_buffer vb;
> > +	bool plane_done[MALI_C55_NUM_PLANES];
> 
> I think tracking the pending state would simplify the logic in
> mali_c55_set_plane_done(), which would become
> 
> 	curr_buf->plane_pending[plane] = false;
> 
> 	if (!curr_buf->plane_pending[0] && !curr_buf->plane_pending[1]) {
> 		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> 		cap_dev->buffers.curr = NULL;
> 		isp->frame_sequence++;
> 	}
> 
> Or a counter may be even easier (and would consume less memory).
> 
> > +	struct list_head queue;
> > +	u32 addrs[MALI_C55_NUM_PLANES];
> 
> This stores DMA addresses, use dma_addr_t.
> 
> > +};
> > +
> > +struct mali_c55_cap_dev {
> > +	struct mali_c55 *mali_c55;
> > +	struct mali_c55_resizer *rzr;
> > +	struct video_device vdev;
> > +	struct media_pad pad;
> > +	struct vb2_queue queue;
> > +	struct mutex lock;
> > +	unsigned int reg_offset;
> 
> Manual handling of the offset everywhere, with parametric macros for the
> resizer register addresses, isn't very nice. Introduce resizer-specific
> accessors and capture-specific accessors (mali_c55_rsz_write(), ...)
> that take a mali_c55_resizer or mali_c55_cap_dev pointer, and handle the
> offset there. The register macros should loose their offset parameter.
> 
> You could also use a single set of accessors that would become
> path/output-specific (mali_c55_path_write() ? mali_c55_output_write()
> ?), that may make the code easier to read.
> 
> You can also replace reg_offset with a void __iomem * base, which would
> avoid the computation at runtime.
> 
> > +
> > +	struct mali_c55_mode {
> 
> Make the structure anonymous.
> 
> > +		const struct mali_c55_fmt *capture_fmt;
> > +		struct v4l2_pix_format_mplane pix_mp;
> > +	} mode;
> 
> What's a "mode" ? I think I'd name this
> 
> 	struct {
> 		const struct mali_c55_fmt *info;
> 		struct v4l2_pix_format_mplane format;
> 	} format;
> 
> Or you could just drop the structure and have
> 
> 	const struct mali_c55_fmt *format_info;
> 	struct v4l2_pix_format_mplane format;
> 
> or something similar.
> 
> > +
> > +	struct {
> > +		spinlock_t lock;
> > +		struct list_head queue;
> > +		struct mali_c55_buffer *curr;
> > +		struct mali_c55_buffer *next;
> > +	} buffers;
> > +
> > +	bool streaming;
> > +};
> > +
> > +enum mali_c55_config_spaces {
> > +	MALI_C55_CONFIG_PING,
> > +	MALI_C55_CONFIG_PONG,
> > +	MALI_C55_NUM_CONFIG_SPACES
> 
> The last enumerator is not used.
> 
> > +};
> > +
> > +struct mali_c55_ctx {
> 
> mali_c55_context ?
> 
> > +	struct mali_c55 *mali_c55;
> > +	void *registers;
> 
> Please document this structure and explain that this field points to a
> copy of the register space in system memory, I was about to write you're
> missing __iomem :-)
> 
> > +	phys_addr_t base;
> > +	spinlock_t lock;
> > +	struct list_head list;
> > +};
> > +
> > +struct mali_c55 {
> > +	struct device *dev;
> > +	struct resource *res;
> 
> You could possibly drop this field by passing the physical address of
> the register space from mali_c55_probe() to mali_c55_init_context() as a
> function parameter.
> 
> > +	void __iomem *base;
> > +	struct dma_chan *channel;
> > +	struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
> > +
> > +	u16 capabilities;
> > +	struct media_device media_dev;
> > +	struct v4l2_device v4l2_dev;
> > +	struct media_pipeline pipe;
> > +
> > +	struct mali_c55_tpg tpg;
> > +	struct mali_c55_isp isp;
> > +	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
> > +	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
> > +
> > +	struct list_head contexts;
> > +	enum mali_c55_config_spaces next_config;
> > +};
> > +
> > +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
> > +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
> > +		  bool force_hardware);
> > +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
> > +			  u32 mask, u32 val);
> > +int mali_c55_config_write(struct mali_c55_ctx *ctx,
> > +			  enum mali_c55_config_spaces cfg_space);
> > +
> > +int mali_c55_register_isp(struct mali_c55 *mali_c55);
> > +int mali_c55_register_tpg(struct mali_c55 *mali_c55);
> > +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
> > +void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
> > +int mali_c55_register_resizers(struct mali_c55 *mali_c55);
> > +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
> > +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
> > +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
> > +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
> > +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> > +			     enum mali_c55_planes plane);
> > +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
> > +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
> > +
> > +bool mali_c55_format_is_raw(unsigned int mbus_code);
> > +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
> > +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
> > +int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
> > +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
> > +
> > +const struct mali_c55_isp_fmt *
> > +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
> > +bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
> > +#define for_each_mali_isp_fmt(fmt)\
> 
> #define for_each_mali_isp_fmt(fmt) \
> 
> > +	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
> 
> Looks like parentheses were on sale :-)
> 
> 	for ((fmt) = NULL; (fmt) = mali_c55_isp_fmt_next(fmt); )
> 
> This macro is used in two places only, in the mali-c55-isp.c file where
> open-coding the loop without using mali_c55_isp_fmt_next() would be more
> efficient, and in mali-c55-resizer.c where a function to return format
> i'th would be more efficient. I think you can drop the macro and the
> mali_c55_isp_fmt_next() function.
> 
> > +
> > +#endif /* _MALI_C55_COMMON_H */
> > diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> > new file mode 100644
> > index 000000000000..50caf5ee7474
> > --- /dev/null
> > +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> > @@ -0,0 +1,767 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * ARM Mali-C55 ISP Driver - Core driver code
> > + *
> > + * Copyright (C) 2024 Ideas on Board Oy
> > + */
> > +
> > +#include <linux/bitops.h>
> > +#include <linux/cleanup.h>
> > +#include <linux/clk.h>
> > +#include <linux/delay.h>
> > +#include <linux/device.h>
> > +#include <linux/dmaengine.h>
> > +#include <linux/dma-mapping.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/iopoll.h>
> > +#include <linux/ioport.h>
> > +#include <linux/mod_devicetable.h>
> > +#include <linux/of.h>
> > +#include <linux/of_reserved_mem.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/pm_runtime.h>
> > +#include <linux/scatterlist.h>
> 
> I don't think this is needed.
> 
> Missing slab.h.
> 
> > +#include <linux/string.h>
> > +
> > +#include <media/media-entity.h>
> > +#include <media/v4l2-device.h>
> > +#include <media/videobuf2-dma-contig.h>
> > +
> > +#include "mali-c55-common.h"
> > +#include "mali-c55-registers.h"
> > +
> > +static const char * const mali_c55_interrupt_names[] = {
> > +	[MALI_C55_IRQ_ISP_START] = "ISP start",
> > +	[MALI_C55_IRQ_ISP_DONE] = "ISP done",
> > +	[MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
> > +	[MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
> > +	[MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
> > +	[MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
> > +	[MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
> > +	[MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
> > +	[MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
> > +	[MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
> > +	[MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
> > +	[MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
> > +	[MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
> > +	[MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
> > +	[MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
> > +	[MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
> > +	[MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
> > +	[MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
> > +	[MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
> > +	[MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
> > +	[MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
> > +	[MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
> > +	[MALI_C55_IRQ_DMA_ERROR] = "DMA error",
> > +	[MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
> > +	[MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
> > +	[MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
> > +};
> > +
> > +static unsigned int config_space_addrs[] = {
> 
> const
> 
> > +	[MALI_C55_CONFIG_PING] = 0x0AB6C,
> > +	[MALI_C55_CONFIG_PONG] = 0x22B2C,
> 
> Lowercase hex constants.
> 
> Don't the values belong to mali-c55-registers.h ?
> 
> > +};
> > +
> > +/* System IO
> 
> /*
>  * System IO
> 
> > + *
> > + * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
> > + * and 'pong'), with the  expectation that the 'active' space will be left
> 
> s/the  /the /
> 
> > + * untouched whilst a frame is being processed and the 'inactive' space
> > + * configured ready to be passed during the blanking period before the next
> 
> s/to be passed/to be switched to/ ?
> 
> > + * frame processing starts. These spaces should ideally be set via DMA transfer
> > + * from a buffer rather than through individual register set operations. There
> > + * is also a shared global register space which should be set normally. Of
> > + * course, the ISP might be included in a system which lacks a suitable DMA
> > + * engine, and the second configuration space might not be fitted at all, which
> > + * means we need to support four scenarios:
> > + *
> > + * 1. Multi config space, with DMA engine.
> > + * 2. Multi config space, no DMA engine.
> > + * 3. Single config space, with DMA engine.
> > + * 4. Single config space, no DMA engine.
> > + *
> > + * The first case is very easy, but the rest present annoying problems. The best
> > + * way to solve them seems to be simply to replicate the concept of DMAing over
> > + * the configuration buffer even if there's no DMA engine on the board, for
> > + * which we rely on memcpy. To facilitate this any read/write call that is made
> > + * to an address within those config spaces should infact be directed to a
> > + * buffer that was allocated to hold them rather than the IO memory itself. The
> > + * actual copy of that buffer to IO mem will happen on interrupt.
> > + */
> > +
> > +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
> > +{
> > +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> > +
> > +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
> > +		spin_lock(&ctx->lock);
> > +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
> > +		((u32 *)ctx->registers)[addr] = val;
> > +		spin_unlock(&ctx->lock);
> > +
> > +		return;
> > +	}
> 
> Ouch. This is likely the second comment you really won't like (after the
> comment regarding mali_c55_update_bits() at the very top). I apologize
> in advance.
> 
> I really don't like this. Directing writes either to hardware registers
> or to the shadow registers in the context makes the callers of the
> read/write accessors very hard to read. The probe code, for instance,
> mixes writes to hardware registers and writes to the context shadow
> registers to initialize the value of some of the shadow registers.
> 
> I'd like to split the read/write accessors into functions that access
> the hardware registers (that's easy) and functions that access the
> shadow registers. I think the latter should receive a mali_c55_ctx
> pointer instead of a mali_c55 pointer to prepare for multi-context
> support.
> 
> You can add WARN_ON() guards to the two sets of functions, to ensure
> that no register from the "other" space gets passed to the wrong
> function by mistake.
> 
> > +
> > +	writel(val, mali_c55->base + addr);
> > +}
> > +
> > +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
> > +		  bool force_hardware)
> 
> force_hardware is never set to true.
> 
> > +{
> > +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> > +	u32 val;
> > +
> > +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
> > +		spin_lock(&ctx->lock);
> > +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
> > +		val = ((u32 *)ctx->registers)[addr];
> > +		spin_unlock(&ctx->lock);
> > +
> > +		return val;
> > +	}
> > +
> > +	return readl(mali_c55->base + addr);
> > +}
> > +
> > +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
> > +			  u32 mask, u32 val)
> > +{
> > +	u32 orig, tmp;
> > +
> > +	orig = mali_c55_read(mali_c55, addr, false);
> > +
> > +	tmp = orig & ~mask;
> > +	tmp |= (val << (ffs(mask) - 1)) & mask;
> > +
> > +	if (tmp != orig)
> > +		mali_c55_write(mali_c55, addr, tmp);
> > +}
> > +
> > +static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
> > +			     dma_addr_t dst, enum dma_data_direction dir)
> > +{
> > +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> > +	struct dma_async_tx_descriptor *tx;
> > +	enum dma_status status;
> > +	dma_cookie_t cookie;
> > +
> > +	tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
> > +				       MALI_C55_CONFIG_SPACE_SIZE, 0);
> > +	if (!tx) {
> > +		dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
> > +		return -EIO;
> > +	}
> > +
> > +	cookie = dmaengine_submit(tx);
> > +	if (dma_submit_error(cookie)) {
> > +		dev_err(mali_c55->dev, "error submitting dma transfer\n");
> > +		return -EIO;
> > +	}
> > +
> > +	status = dma_sync_wait(mali_c55->channel, cookie);
> 
> I've just realized this performs a busy-wait :-S See the comment in the
> probe function about the threaded IRQ handler. I think we'll need to
> rework all this. It could be done on top though.
> 
> > +	if (status != DMA_COMPLETE) {
> > +		dev_err(mali_c55->dev, "dma transfer failed\n");
> > +		return -EIO;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
> > +			     enum mali_c55_config_spaces cfg_space)
> > +{
> > +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> > +	struct device *dma_dev = mali_c55->channel->device->dev;
> > +	dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
> > +	dma_addr_t dst;
> > +	int ret;
> > +
> > +	guard(spinlock)(&ctx->lock);
> > +
> > +	dst = dma_map_single(dma_dev, ctx->registers,
> > +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
> > +	if (dma_mapping_error(dma_dev, dst)) {
> > +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
> > +		return -EIO;
> > +	}
> > +
> > +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
> > +	dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
> > +			 DMA_FROM_DEVICE);
> > +
> > +	return ret;
> > +}
> > +
> > +static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
> > +		       enum mali_c55_config_spaces cfg_space)
> > +{
> > +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> > +	struct device *dma_dev = mali_c55->channel->device->dev;
> > +	dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
> > +	dma_addr_t src;
> > +	int ret;
> > +
> > +	guard(spinlock)(&ctx->lock);
> 
> The code below can take a large amount of time, holding a spinlock will
> disable interrupts on the local CPU, that's not good :-(
> 
> > +
> > +	src = dma_map_single(dma_dev, ctx->registers,
> > +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
> > +	if (dma_mapping_error(dma_dev, src)) {
> > +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
> > +		return -EIO;
> > +	}
> > +
> > +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
> > +	dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
> > +			 DMA_TO_DEVICE);
> > +
> > +	return ret;
> > +}
> > +
> > +static int mali_c55_config_read(struct mali_c55_ctx *ctx,
> > +				enum mali_c55_config_spaces cfg_space)
> > +{
> > +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> > +
> > +	if (mali_c55->channel) {
> > +		return mali_c55_dma_read(ctx, cfg_space);
> 
> As this function is used at probe time only, to initialize the context,
> I think DMA is overkill.
> 
> > +	} else {
> > +		memcpy_fromio(ctx->registers,
> > +			      mali_c55->base + config_space_addrs[cfg_space],
> > +			      MALI_C55_CONFIG_SPACE_SIZE);
> > +		return 0;
> > +	}
> > +}
> > +
> > +int mali_c55_config_write(struct mali_c55_ctx *ctx,
> > +			  enum mali_c55_config_spaces cfg_space)
> > +{
> > +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> > +
> > +	if (mali_c55->channel) {
> > +		return mali_c55_dma_write(ctx, cfg_space);
> > +	} else {
> > +		memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
> > +			    ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
> > +		return 0;
> > +	}
> 
> Could you measure the time it typically takes to write the registers
> using DMA compared to using memcpy_toio() ?
> 
> > +}
> > +
> > +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
> > +{
> > +	return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);
> 
> I think it's too early to tell how multi-context support will look like.
> I'm fine keeping mali_c55_get_active_context() as changing that would be
> very intrusive (even if I think it will need to be changed), but the
> list of contexts is neither the mechanism we'll use, nor something we
> need now. Drop the list, embed the context in struct mali_c55, and
> return the pointer to that single context from this function.
> 
> > +}
> > +
> > +static void mali_c55_remove_links(struct mali_c55 *mali_c55)
> > +{
> > +	unsigned int i;
> > +
> > +	media_entity_remove_links(&mali_c55->tpg.sd.entity);
> > +	media_entity_remove_links(&mali_c55->isp.sd.entity);
> > +
> > +	for (i = 0; i < MALI_C55_NUM_RZRS; i++)
> > +		media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
> > +
> > +	for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
> > +		media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
> > +}
> > +
> > +static int mali_c55_create_links(struct mali_c55 *mali_c55)
> > +{
> > +	int ret;
> > +
> > +	/* Test pattern generator to ISP */
> > +	ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
> > +				    &mali_c55->isp.sd.entity,
> > +				    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
> > +	if (ret) {
> > +		dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
> > +		goto err_remove_links;
> > +	}
> > +
> > +	/* Full resolution resizer pipe. */
> > +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> > +			MALI_C55_ISP_PAD_SOURCE,
> > +			&mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
> > +			MALI_C55_RZR_SINK_PAD,
> > +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> > +	if (ret) {
> > +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
> > +		goto err_remove_links;
> > +	}
> > +
> > +	/* Full resolution bypass. */
> > +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> > +				    MALI_C55_ISP_PAD_SOURCE_BYPASS,
> > +				    &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
> > +				    MALI_C55_RZR_SINK_BYPASS_PAD,
> > +				    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> > +	if (ret) {
> > +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
> > +		goto err_remove_links;
> > +	}
> > +
> > +	/* Resizer pipe to video capture nodes. */
> > +	ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
> > +			MALI_C55_RZR_SOURCE_PAD,
> > +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
> > +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> > +	if (ret) {
> > +		dev_err(mali_c55->dev,
> > +			"failed to link FR resizer and video device\n");
> > +		goto err_remove_links;
> > +	}
> > +
> > +	/* The downscale pipe is an optional hardware block */
> > +	if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
> > +		ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> > +			MALI_C55_ISP_PAD_SOURCE,
> > +			&mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
> > +			MALI_C55_RZR_SINK_PAD,
> > +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> > +		if (ret) {
> > +			dev_err(mali_c55->dev,
> > +				"failed to link ISP and DS resizer\n");
> > +			goto err_remove_links;
> > +		}
> > +
> > +		ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
> > +			MALI_C55_RZR_SOURCE_PAD,
> > +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
> > +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> > +		if (ret) {
> > +			dev_err(mali_c55->dev,
> > +				"failed to link DS resizer and video device\n");
> > +			goto err_remove_links;
> > +		}
> > +	}
> > +
> > +	return 0;
> > +
> > +err_remove_links:
> > +	mali_c55_remove_links(mali_c55);
> > +	return ret;
> > +}
> > +
> > +static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
> > +{
> > +	mali_c55_unregister_tpg(mali_c55);
> > +	mali_c55_unregister_isp(mali_c55);
> > +	mali_c55_unregister_resizers(mali_c55);
> > +	mali_c55_unregister_capture_devs(mali_c55);
> > +}
> > +
> > +static int mali_c55_register_entities(struct mali_c55 *mali_c55)
> > +{
> > +	int ret;
> > +
> > +	ret = mali_c55_register_tpg(mali_c55);
> > +	if (ret)
> > +		return ret;
> > +
> > +	ret = mali_c55_register_isp(mali_c55);
> > +	if (ret)
> > +		goto err_unregister_entities;
> > +
> > +	ret = mali_c55_register_resizers(mali_c55);
> > +	if (ret)
> > +		goto err_unregister_entities;
> > +
> > +	ret = mali_c55_register_capture_devs(mali_c55);
> > +	if (ret)
> > +		goto err_unregister_entities;
> > +
> > +	ret = mali_c55_create_links(mali_c55);
> > +	if (ret)
> > +		goto err_unregister_entities;
> > +
> > +	return 0;
> > +
> > +err_unregister_entities:
> > +	mali_c55_unregister_entities(mali_c55);
> > +
> > +	return ret;
> > +}
> > +
> > +static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
> > +{
> > +	u32 product, version, revision, capabilities;
> > +
> > +	product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
> > +	version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
> > +	revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
> > +
> > +	dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
> > +		 product, version, revision);
> > +
> > +	capabilities = mali_c55_read(mali_c55,
> > +				     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
> > +				     false);
> > +	mali_c55->capabilities = (capabilities & 0xffff);
> > +
> > +	/* TODO: Might as well start some debugfs */
> 
> If it's just to expose the version and capabilities, I think that's
> overkill. It's not needed for debug purpose (you can get it from the
> kernel log already). debugfs isn't meant to be accessible in production,
> so an application that would need access to the information wouldn't be
> able to use it.
> 
> > +	dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);
> 
> Combine the two messages into one.
> 
> > +	return version;
> > +}
> > +
> > +static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
> > +{
> > +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> > +	u32 curr_config, next_config;
> > +
> > +	curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
> > +	curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
> > +		      >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
> > +	next_config = curr_config ^ 1;
> > +
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> > +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
> > +	mali_c55_config_write(ctx, next_config ?
> > +			      MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
> > +}
> > +
> > +static irqreturn_t mali_c55_isr(int irq, void *context)
> > +{
> > +	struct device *dev = context;
> > +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> > +	u32 interrupt_status;
> > +	unsigned int i, j;
> > +
> > +	interrupt_status = mali_c55_read(mali_c55,
> > +					 MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
> > +					 false);
> > +	if (!interrupt_status)
> > +		return IRQ_NONE;
> > +
> > +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
> > +		       interrupt_status);
> > +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
> > +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
> > +
> > +	for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
> > +		if (!(interrupt_status & (1 << i)))
> > +			continue;
> > +
> > +		switch (i) {
> > +		case MALI_C55_IRQ_ISP_START:
> > +			mali_c55_isp_queue_event_sof(mali_c55);
> > +
> > +			for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
> > +				mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
> > +
> > +			mali_c55_swap_next_config(mali_c55);
> > +
> > +			break;
> > +		case MALI_C55_IRQ_ISP_DONE:
> > +			/*
> > +			 * TODO: Where the ISP has no Pong config fitted, we'd
> > +			 * have to do the mali_c55_swap_next_config() call here.
> > +			 */
> > +			break;
> > +		case MALI_C55_IRQ_FR_Y_DONE:
> > +			mali_c55_set_plane_done(
> > +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
> > +				MALI_C55_PLANE_Y);
> > +			break;
> > +		case MALI_C55_IRQ_FR_UV_DONE:
> > +			mali_c55_set_plane_done(
> > +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
> > +				MALI_C55_PLANE_UV);
> > +			break;
> > +		case MALI_C55_IRQ_DS_Y_DONE:
> > +			mali_c55_set_plane_done(
> > +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
> > +				MALI_C55_PLANE_Y);
> > +			break;
> > +		case MALI_C55_IRQ_DS_UV_DONE:
> > +			mali_c55_set_plane_done(
> > +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
> > +				MALI_C55_PLANE_UV);
> > +			break;
> > +		default:
> > +			/*
> > +			 * Only the above interrupts are currently unmasked. If
> > +			 * we receive anything else here then something weird
> > +			 * has gone on.
> > +			 */
> > +			dev_err(dev, "masked interrupt %s triggered\n",
> > +				mali_c55_interrupt_names[i]);
> > +		}
> > +	}
> > +
> > +	return IRQ_HANDLED;
> > +}
> > +
> > +static int mali_c55_init_context(struct mali_c55 *mali_c55)
> > +{
> > +	struct mali_c55_ctx *ctx;
> > +	int ret;
> > +
> > +	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
> > +	if (!ctx) {
> > +		dev_err(mali_c55->dev, "failed to allocate new context\n");
> 
> No need for an error message when memory allocation fails.
> 
> > +		return -ENOMEM;
> > +	}
> > +
> > +	ctx->base = mali_c55->res->start;
> > +	ctx->mali_c55 = mali_c55;
> > +
> > +	ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
> > +				 GFP_KERNEL | GFP_DMA);
> > +	if (!ctx->registers) {
> > +		ret = -ENOMEM;
> > +		goto err_free_ctx;
> > +	}
> > +
> > +	/*
> > +	 * The allocated memory is empty, we need to load the default
> > +	 * register settings. We just read Ping; it's identical to Pong.
> > +	 */
> > +	ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
> > +	if (ret)
> > +		goto err_free_registers;
> > +
> > +	list_add_tail(&ctx->list, &mali_c55->contexts);
> > +
> > +	/*
> > +	 * Some features of the ISP need to be disabled by default and only
> > +	 * enabled at the same time as they're configured by a parameters buffer
> > +	 */
> > +
> > +	/* Bypass the sqrt and square compression and expansion modules */
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
> > +			     MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
> > +			     MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
> > +
> > +	/* Bypass the temper module */
> > +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
> > +		       MALI_C55_REG_BYPASS_2_TEMPER);
> > +
> > +	/* Bypass the colour noise reduction  */
> > +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
> > +		       MALI_C55_REG_BYPASS_4_CNR);
> > +
> > +	/* Disable the sinter module */
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
> > +			     MALI_C55_SINTER_ENABLE_MASK, 0x00);
> > +
> > +	/* Disable the RGB Gamma module for each output */
> > +	mali_c55_write(
> > +		mali_c55,
> > +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
> > +		0x00);
> > +	mali_c55_write(
> > +		mali_c55,
> > +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
> > +		0x00);
> > +
> > +	/* Disable the colour correction matrix */
> > +	mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
> > +
> > +	return 0;
> > +
> > +err_free_registers:
> > +	kfree(ctx->registers);
> > +err_free_ctx:
> > +	kfree(ctx);
> > +
> > +	return ret;
> > +}
> > +
> > +static int mali_c55_runtime_resume(struct device *dev)
> > +{
> > +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> > +	int ret;
> > +
> > +	ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
> > +				      mali_c55->clks);
> > +	if (ret)
> > +		dev_err(mali_c55->dev, "failed to enable clocks\n");
> > +
> > +	return ret;
> > +}
> > +
> > +static int mali_c55_runtime_suspend(struct device *dev)
> > +{
> > +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> > +
> > +	clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
> > +	return 0;
> > +}
> > +
> > +static const struct dev_pm_ops mali_c55_pm_ops = {
> > +	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> > +				pm_runtime_force_resume)
> > +	SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
> > +			   NULL)
> > +};
> > +
> > +static int mali_c55_probe(struct platform_device *pdev)
> > +{
> > +	struct device *dev = &pdev->dev;
> > +	struct mali_c55 *mali_c55;
> > +	dma_cap_mask_t mask;
> > +	u32 version;
> > +	int ret;
> > +	u32 val;
> > +
> > +	mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
> > +	if (!mali_c55)
> > +		return dev_err_probe(dev, -ENOMEM,
> > +				     "failed to allocate memory\n");
> 
> 		return -ENOMEM;
> 
> There's no need to print messages for memory allocation failures.
> 
> > +
> > +	mali_c55->dev = dev;
> > +	platform_set_drvdata(pdev, mali_c55);
> > +
> > +	mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
> > +								&mali_c55->res);
> > +	if (IS_ERR(mali_c55->base))
> > +		return dev_err_probe(dev, PTR_ERR(mali_c55->base),
> > +				     "failed to map IO memory\n");
> > +
> > +	ret = platform_get_irq(pdev, 0);
> > +	if (ret < 0)
> > +		return dev_err_probe(dev, ret, "failed to get interrupt num\n");
> 
> s/ num// or s/num/number/
> 
> > +
> > +	ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
> > +					mali_c55_isr, IRQF_ONESHOT,
> > +					dev_driver_string(&pdev->dev),
> > +					&pdev->dev);
> 
> Requested the IRQ should be done much lower, after you have initialized
> everything, or an IRQ that would fire early would have really bad
> consequences.
> 
> A comment to explain why you need a threaded interrupt handler would be
> good. I assume it is due only to the need to transfer the registers
> using DMA. I wonder if we should then split the interrupt handler in
> two, with a non-threaded part for the operations that can run quickly,
> and a threaded part for the reprogramming.
> 
> It may also be that we could just start the DMA transfer in the
> non-threaded handler without waiting synchronously for it to complete.
> That would be a bigger change, and would require checking race
> conditions carefully. On the other hand, I'm a bit concerned about the
> current implementation, have you tested what happens if the DMA transfer
> takes too long to complete, and spans frame boundaries ? This part could
> be addressed by a patch on top of this one.
> 
> > +	if (ret)
> > +		return dev_err_probe(dev, ret, "failed to request irq\n");
> > +
> > +	for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
> > +		mali_c55->clks[i].id = mali_c55_clk_names[i];
> > +
> > +	ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
> > +	if (ret)
> > +		return dev_err_probe(dev, ret, "failed to acquire clocks\n");
> > +
> > +	pm_runtime_enable(&pdev->dev);
> > +
> > +	ret = pm_runtime_resume_and_get(&pdev->dev);
> > +	if (ret)
> > +		goto err_pm_runtime_disable;
> > +
> > +	of_reserved_mem_device_init(dev);
> 
> I'd move this, the vb2_dma_contig_set_max_seg_size() call and the
> dma_cap_* calls before pm_runtime_enable() as they don't need the device
> to be powered.
> 
> > +	version = mali_c55_check_hwcfg(mali_c55);
> > +	vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
> > +
> > +	/* Use "software only" context management. */
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> > +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> > +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> 
> You handle that in mali_c55_isp_start(), does the register have to be
> set here too ?
> 
> > +
> > +	dma_cap_zero(mask);
> > +	dma_cap_set(DMA_MEMCPY, mask);
> > +
> > +	/*
> > +	 * No error check, because we will just fallback on memcpy if there is
> > +	 * no usable DMA channel on the system.
> > +	 */
> > +	mali_c55->channel = dma_request_channel(mask, NULL, NULL);
> > +
> > +	INIT_LIST_HEAD(&mali_c55->contexts);
> > +	ret = mali_c55_init_context(mali_c55);
> > +	if (ret)
> > +		goto err_release_dma_channel;
> > +
> 
> I'd move all the code from here ...
> 
> > +	mali_c55->media_dev.dev = dev;
> > +	strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
> > +		sizeof(mali_c55->media_dev.model));
> > +	mali_c55->media_dev.hw_revision = version;
> > +
> > +	media_device_init(&mali_c55->media_dev);
> > +	ret = media_device_register(&mali_c55->media_dev);
> > +	if (ret)
> > +		goto err_cleanup_media_device;
> > +
> > +	mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
> > +	ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
> > +	if (ret) {
> > +		dev_err(dev, "failed to register V4L2 device\n");
> > +		goto err_unregister_media_device;
> > +	};
> > +
> > +	ret = mali_c55_register_entities(mali_c55);
> > +	if (ret) {
> > +		dev_err(dev, "failed to register entities\n");
> > +		goto err_unregister_v4l2_device;
> > +	}
> 
> ... to here to a separate function, or maybe fold it all in
> mali_c55_register_entities() (which should the be renamed). Same thing
> for the cleanup code.
> 
> > +
> > +	/* Set safe stop to ensure we're in a non-streaming state */
> > +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> > +		       MALI_C55_INPUT_SAFE_STOP);
> > +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> > +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> > +
> > +	/*
> > +	 * We're ready to process interrupts. Clear any that are set and then
> > +	 * unmask them for processing.
> > +	 */
> > +	mali_c55_write(mali_c55, 0x30, 0xffffffff);
> > +	mali_c55_write(mali_c55, 0x34, 0xffffffff);
> > +	mali_c55_write(mali_c55, 0x40, 0x01);
> > +	mali_c55_write(mali_c55, 0x40, 0x00);
> 
> Please replace the register addresses with macros.
> 
> > +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);
> 
> The value should use the interrupt bits macros.
> 
> > +
> > +	pm_runtime_put(&pdev->dev);
> 
> Once power gets cut, the registers your programmed above may be lost. I
> think you should programe them in the runtime PM resume handler.
> 
> > +
> > +	return 0;
> > +
> > +err_unregister_v4l2_device:
> > +	v4l2_device_unregister(&mali_c55->v4l2_dev);
> > +err_unregister_media_device:
> > +	media_device_unregister(&mali_c55->media_dev);
> > +err_cleanup_media_device:
> > +	media_device_cleanup(&mali_c55->media_dev);
> > +err_release_dma_channel:
> > +	dma_release_channel(mali_c55->channel);
> > +err_pm_runtime_disable:
> > +	pm_runtime_disable(&pdev->dev);
> > +
> > +	return ret;
> > +}
> > +
> > +static void mali_c55_remove(struct platform_device *pdev)
> > +{
> > +	struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
> > +	struct mali_c55_ctx *ctx, *tmp;
> > +
> > +	list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
> > +		list_del(&ctx->list);
> > +		kfree(ctx->registers);
> > +		kfree(ctx);
> > +	}
> > +
> > +	mali_c55_remove_links(mali_c55);
> > +	mali_c55_unregister_entities(mali_c55);
> > +	v4l2_device_put(&mali_c55->v4l2_dev);
> > +	media_device_unregister(&mali_c55->media_dev);
> > +	media_device_cleanup(&mali_c55->media_dev);
> > +	dma_release_channel(mali_c55->channel);
> > +}
> > +
> > +static const struct of_device_id mali_c55_of_match[] = {
> > +	{ .compatible = "arm,mali-c55", },
> > +	{},
> 
> 	{ /* Sentinel */ },
> 
> > +};
> > +MODULE_DEVICE_TABLE(of, mali_c55_of_match);
> > +
> > +static struct platform_driver mali_c55_driver = {
> > +	.driver = {
> > +		.name = "mali-c55",
> > +		.of_match_table = of_match_ptr(mali_c55_of_match),
> 
> Drop of_match_ptr().
> 
> > +		.pm = &mali_c55_pm_ops,
> > +	},
> > +	.probe = mali_c55_probe,
> > +	.remove_new = mali_c55_remove,
> > +};
> > +
> > +module_platform_driver(mali_c55_driver);
> 
> Blank line.
> 
> > +MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
> > +MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
> > +MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
> > +MODULE_LICENSE("GPL");
> > diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> > new file mode 100644
> > index 000000000000..ea8b7b866e7a
> > --- /dev/null
> > +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> > @@ -0,0 +1,611 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * ARM Mali-C55 ISP Driver - Image signal processor
> > + *
> > + * Copyright (C) 2024 Ideas on Board Oy
> > + */
> > +
> > +#include <linux/delay.h>
> > +#include <linux/iopoll.h>
> > +#include <linux/property.h>
> > +#include <linux/string.h>
> > +
> > +#include <media/media-entity.h>
> > +#include <media/v4l2-common.h>
> > +#include <media/v4l2-event.h>
> > +#include <media/v4l2-mc.h>
> > +#include <media/v4l2-subdev.h>
> > +
> > +#include "mali-c55-common.h"
> > +#include "mali-c55-registers.h"
> > +
> > +static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
> > +	{
> > +		.code = MEDIA_BUS_FMT_SRGGB20_1X20,
> > +		.order = MALI_C55_BAYER_ORDER_RGGB,
> > +		.encoding = V4L2_PIXEL_ENC_BAYER,
> > +	},
> > +	{
> > +		.code = MEDIA_BUS_FMT_SGRBG20_1X20,
> > +		.order = MALI_C55_BAYER_ORDER_GRBG,
> > +		.encoding = V4L2_PIXEL_ENC_BAYER,
> > +	},
> > +	{
> > +		.code = MEDIA_BUS_FMT_SGBRG20_1X20,
> > +		.order = MALI_C55_BAYER_ORDER_GBRG,
> > +		.encoding = V4L2_PIXEL_ENC_BAYER,
> > +	},
> > +	{
> > +		.code = MEDIA_BUS_FMT_SBGGR20_1X20,
> > +		.order = MALI_C55_BAYER_ORDER_BGGR,
> > +		.encoding = V4L2_PIXEL_ENC_BAYER,
> > +	},
> > +	{
> > +		.code = MEDIA_BUS_FMT_RGB202020_1X60,
> > +		.order = 0, /* Not relevant for this format */
> > +		.encoding = V4L2_PIXEL_ENC_RGB,
> > +	}
> > +	/*
> > +	 * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
> > +	 * also support YUV input from a sensor passed-through to the output. At
> > +	 * present we have no mechanism to test that though so it may have to
> > +	 * wait a while...
> > +	 */
> > +};
> > +
> > +const struct mali_c55_isp_fmt *
> > +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
> > +{
> > +	if (!fmt)
> > +		fmt = &mali_c55_isp_fmts[0];
> > +	else
> > +		fmt++;
> > +
> > +	for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
> > +		return fmt;
> 
> That's peculiar.
> 
> 	if (!fmt)
> 		fmt = &mali_c55_isp_fmts[0];
> 	else if (fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)])
> 		return ++fmt;
> 	else
> 		return NULL;
> 
> > +
> > +	return NULL;
> > +}
> > +
> > +bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
> > +{
> > +	const struct mali_c55_isp_fmt *isp_fmt;
> > +
> > +	for_each_mali_isp_fmt(isp_fmt) {
> 
> I would open-code the loop instead of using the macro, like you do
> below. It will be more efficient.
> 
> > +		if (isp_fmt->code == mbus_code)
> > +			return true;
> > +	}
> > +
> > +	return false;
> > +}
> > +
> > +static const struct mali_c55_isp_fmt *
> > +mali_c55_isp_get_mbus_config_by_code(u32 code)
> > +{
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
> > +		if (mali_c55_isp_fmts[i].code == code)
> > +			return &mali_c55_isp_fmts[i];
> > +	}
> > +
> > +	return NULL;
> > +}
> > +
> > +static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
> > +{
> > +	u32 val;
> > +
> > +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);
> 
> 	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> 		       MALI_C55_INPUT_SAFE_STOP);
> 
> > +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> > +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> > +}
> > +
> > +static int mali_c55_isp_start(struct mali_c55 *mali_c55)
> > +{
> > +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> > +	const struct mali_c55_isp_fmt *cfg;
> > +	struct v4l2_mbus_framefmt *format;
> 
> const
> 
> > +	struct v4l2_subdev_state *state;
> > +	struct v4l2_rect *crop;
> 
> const
> 
> > +	struct v4l2_subdev *sd;
> > +	u32 val;
> > +	int ret;
> > +
> > +	sd = &mali_c55->isp.sd;
> 
> Assign when declaring the variable.
> 
> > +
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> > +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
> > +
> > +	/* Apply input windowing */
> > +	state = v4l2_subdev_get_locked_active_state(sd);
> 
> Using .enable_streams() (see below) you'll get this for free.
> 
> > +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> > +	format = v4l2_subdev_state_get_format(state,
> > +					      MALI_C55_ISP_PAD_SINK_VIDEO);
> > +	cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
> > +
> > +	mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
> > +		       MALI_C55_HC_START(crop->left));
> > +	mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
> > +		       MALI_C55_HC_SIZE(crop->width));
> > +	mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
> > +		       MALI_C55_VC_START(crop->top) |
> > +		       MALI_C55_VC_SIZE(crop->height));
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
> > +			     MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
> > +			     MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
> > +			     MALI_C55_BAYER_ORDER_MASK, cfg->order);
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
> > +			     MALI_C55_INPUT_WIDTH_MASK,
> > +			     MALI_C55_INPUT_WIDTH_20BIT);
> > +
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> > +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
> > +			     cfg->encoding == V4L2_PIXEL_ENC_RGB ?
> > +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
> > +
> > +	ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
> > +	if (ret) {
> > +		dev_err(mali_c55->dev, "failed to DMA config\n");
> > +		return ret;
> > +	}
> > +
> > +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> > +		       MALI_C55_INPUT_SAFE_START);
> > +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> > +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> 
> Should you return an error in case of timeout ?
> 
> > +
> > +	return 0;
> > +}
> > +
> > +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)
> 
> Why is this not handled wired to .s_stream() ? Or better,
> .enable_streams() and .disable_streams().
> 
> > +{
> > +	struct mali_c55 *mali_c55 = isp->mali_c55;
> > +	struct v4l2_subdev *sd;
> > +
> > +	if (isp->remote_src) {
> > +		sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
> > +		v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
> > +	}
> > +	isp->remote_src = NULL;
> > +
> > +	mali_c55_isp_stop(mali_c55);
> > +}
> > +
> > +int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
> > +{
> > +	struct mali_c55 *mali_c55 = isp->mali_c55;
> > +	struct media_pad *sink_pad;
> > +	struct v4l2_subdev *sd;
> > +	int ret;
> > +
> > +	sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
> > +	isp->remote_src = media_pad_remote_pad_unique(sink_pad);
> > +	if (IS_ERR(isp->remote_src)) {
> 
> If you set the MUST_CONNECT flag for the MALI_C55_ISP_PAD_SINK_VIDEO pad
> I think you can drop this check.
> 
> > +		dev_err(mali_c55->dev, "Failed to get source for ISP\n");
> > +		return PTR_ERR(isp->remote_src);
> > +	}
> > +
> > +	sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
> > +
> > +	isp->frame_sequence = 0;
> > +	ret = mali_c55_isp_start(mali_c55);
> > +	if (ret) {
> > +		dev_err(mali_c55->dev, "Failed to start ISP\n");
> > +		isp->remote_src = NULL;
> > +		return ret;
> > +	}
> > +
> > +	/*
> > +	 * We only support a single input stream, so we can just enable the 1st
> > +	 * entry in the streams mask.
> > +	 */
> > +	ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
> > +	if (ret) {
> > +		dev_err(mali_c55->dev, "Failed to start ISP source\n");
> > +		mali_c55_isp_stop(mali_c55);
> > +		return ret;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
> > +				       struct v4l2_subdev_state *state,
> > +				       struct v4l2_subdev_mbus_code_enum *code)
> > +{
> > +	/*
> > +	 * Only the internal RGB processed format is allowed on the regular
> > +	 * processing source pad.
> > +	 */
> > +	if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
> > +		if (code->index)
> > +			return -EINVAL;
> > +
> > +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > +		return 0;
> > +	}
> > +
> > +	/* On the sink and bypass pads all the supported formats are allowed. */
> > +	if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
> > +		return -EINVAL;
> > +
> > +	code->code = mali_c55_isp_fmts[code->index].code;
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
> > +					struct v4l2_subdev_state *state,
> > +					struct v4l2_subdev_frame_size_enum *fse)
> > +{
> > +	const struct mali_c55_isp_fmt *cfg;
> > +
> > +	if (fse->index > 0)
> > +		return -EINVAL;
> > +
> > +	/*
> > +	 * Only the internal RGB processed format is allowed on the regular
> > +	 * processing source pad.
> > +	 *
> > +	 * On the sink and bypass pads all the supported formats are allowed.
> > +	 */
> > +	if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
> > +		if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
> > +			return -EINVAL;
> > +	} else {
> > +		cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
> > +		if (!cfg)
> > +			return -EINVAL;
> > +	}
> > +
> > +	fse->min_width = MALI_C55_MIN_WIDTH;
> > +	fse->min_height = MALI_C55_MIN_HEIGHT;
> > +	fse->max_width = MALI_C55_MAX_WIDTH;
> > +	fse->max_height = MALI_C55_MAX_HEIGHT;
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
> > +				struct v4l2_subdev_state *state,
> > +				struct v4l2_subdev_format *format)
> > +{
> > +	struct v4l2_mbus_framefmt *fmt = &format->format;
> > +	struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
> > +	const struct mali_c55_isp_fmt *cfg;
> > +	struct v4l2_rect *crop;
> > +
> > +	/*
> > +	 * Disallow set_fmt on the source pads; format is fixed and the sizes
> > +	 * are the result of applying the sink crop rectangle to the sink
> > +	 * format.
> > +	 */
> > +	if (format->pad)
> 
> 	if (format->pad != MALI_C55_ISP_PAD_SINK_VIDEO)
> 
> > +		return v4l2_subdev_get_fmt(sd, state, format);
> > +
> > +	cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
> > +	if (!cfg)
> > +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> > +	fmt->field = V4L2_FIELD_NONE;
> 
> Do you intentionally allow the colorspace fields to be overwritten to
> any value ? rkisp1_isp_set_src_fmt() and rkisp1_isp_set_sink_fmt() may
> show you how this could be handled.
> 
> > +
> > +	/*
> > +	 * Clamp sizes in the accepted limits and clamp the crop rectangle in
> > +	 * the new sizes.
> > +	 */
> > +	clamp_t(unsigned int, fmt->width,
> > +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> > +	clamp_t(unsigned int, fmt->width,
> > +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> 
> clamp_t() returns a value, which you ignore. Those are no-ops. You meant
> 
> 	fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> 			     MALI_C55_MAX_WIDTH);
> 	fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> 			      MALI_C55_MAX_HEIGHT);
> 
> Same for every use of clamp_t() through the whole driver.
> 
> Also, do you need clamp_t() ? I think all values are unsigned int, you
> can use clamp().
> 
> Are there any alignment constraints, such a multiples of two for bayer
> formats ? Same in all the other locations where applicable.
> 
> > +
> > +	sink_fmt = v4l2_subdev_state_get_format(state,
> > +						MALI_C55_ISP_PAD_SINK_VIDEO);
> > +	*sink_fmt = *fmt;
> > +
> > +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> > +	crop->left = 0;
> > +	crop->top = 0;
> > +	crop->width = fmt->width;
> > +	crop->height = fmt->height;
> > +
> > +	/*
> > +	 * Propagate format to source pads. On the 'regular' output pad use
> > +	 * the internal RGB processed format, while on the bypass pad simply
> > +	 * replicate the ISP sink format. The sizes on both pads are the same as
> > +	 * the ISP sink crop rectangle.
> > +	 */
> 
> Colorspace information will need to be propagated too.
> 
> > +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
> > +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > +	src_fmt->width = crop->width;
> > +	src_fmt->height = crop->height;
> > +
> > +	src_fmt = v4l2_subdev_state_get_format(state,
> > +					       MALI_C55_ISP_PAD_SOURCE_BYPASS);
> > +	src_fmt->code = fmt->code;
> > +	src_fmt->width = crop->width;
> > +	src_fmt->height = crop->height;
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
> > +				      struct v4l2_subdev_state *state,
> > +				      struct v4l2_subdev_selection *sel)
> > +{
> > +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> 
> 	sel->pad != MALI_C55_ISP_PAD_SINK_VIDEO
> 
> > +		return -EINVAL;
> > +
> > +	sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
> > +				      struct v4l2_subdev_state *state,
> > +				      struct v4l2_subdev_selection *sel)
> > +{
> > +	struct v4l2_mbus_framefmt *src_fmt;
> > +	struct v4l2_mbus_framefmt *fmt;
> 
> const
> 
> > +	struct v4l2_rect *crop;
> > +
> > +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> 
> Ditto.
> 
> > +		return -EINVAL;
> > +
> > +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> > +
> > +	clamp_t(unsigned int, sel->r.left, 0, fmt->width);
> > +	clamp_t(unsigned int, sel->r.top, 0, fmt->height);
> > +	clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
> > +		fmt->width - sel->r.left);
> > +	clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
> > +		fmt->height - sel->r.top);
> > +
> > +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> > +	*crop = sel->r;
> > +
> > +	/* Propagate the crop rectangle sizes to the source pad format. */
> > +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
> > +	src_fmt->width = crop->width;
> > +	src_fmt->height = crop->height;
> 
> Can you confirm that cropping doesn't affect the bypass path ? And maybe
> add a comment to mention it.
> 
> > +
> > +	return 0;
> > +}
> > +
> > +static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
> > +	.enum_mbus_code		= mali_c55_isp_enum_mbus_code,
> > +	.enum_frame_size	= mali_c55_isp_enum_frame_size,
> > +	.get_fmt		= v4l2_subdev_get_fmt,
> > +	.set_fmt		= mali_c55_isp_set_fmt,
> > +	.get_selection		= mali_c55_isp_get_selection,
> > +	.set_selection		= mali_c55_isp_set_selection,
> > +	.link_validate		= v4l2_subdev_link_validate_default,
> > +};
> > +
> > +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
> > +{
> > +	struct v4l2_event event = {
> > +		.type = V4L2_EVENT_FRAME_SYNC,
> > +	};
> > +
> > +	event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
> > +	v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
> > +}
> > +
> > +static int
> > +mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
> > +			     struct v4l2_event_subscription *sub)
> > +{
> > +	if (sub->type != V4L2_EVENT_FRAME_SYNC)
> > +		return -EINVAL;
> > +
> > +	/* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
> > +	if (sub->id != 0)
> > +		return -EINVAL;
> > +
> > +	return v4l2_event_subscribe(fh, sub, 0, NULL);
> > +}
> > +
> > +static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
> > +	.subscribe_event = mali_c55_isp_subscribe_event,
> > +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
> > +};
> > +
> > +static const struct v4l2_subdev_ops mali_c55_isp_ops = {
> > +	.pad	= &mali_c55_isp_pad_ops,
> > +	.core	= &mali_c55_isp_core_ops,
> > +};
> > +
> > +static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
> > +				   struct v4l2_subdev_state *sd_state)
> 
> You name this variable state in every other subdev operation handler.
> 
> > +{
> > +	struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> > +	struct v4l2_rect *in_crop;
> > +
> > +	sink_fmt = v4l2_subdev_state_get_format(sd_state,
> > +						MALI_C55_ISP_PAD_SINK_VIDEO);
> > +	src_fmt = v4l2_subdev_state_get_format(sd_state,
> > +					       MALI_C55_ISP_PAD_SOURCE);
> > +	in_crop = v4l2_subdev_state_get_crop(sd_state,
> > +					     MALI_C55_ISP_PAD_SINK_VIDEO);
> > +
> > +	sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
> > +	sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
> > +	sink_fmt->field = V4L2_FIELD_NONE;
> > +	sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> 
> You should initialize the colorspace fields too. Same below.
> 
> > +
> > +	*v4l2_subdev_state_get_format(sd_state,
> > +			      MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
> > +
> > +	src_fmt->width = MALI_C55_DEFAULT_WIDTH;
> > +	src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
> > +	src_fmt->field = V4L2_FIELD_NONE;
> > +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > +
> > +	in_crop->top = 0;
> > +	in_crop->left = 0;
> > +	in_crop->width = MALI_C55_DEFAULT_WIDTH;
> > +	in_crop->height = MALI_C55_DEFAULT_HEIGHT;
> > +
> > +	return 0;
> > +}
> > +
> > +static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
> > +	.init_state = mali_c55_isp_init_state,
> > +};
> > +
> > +static const struct media_entity_operations mali_c55_isp_media_ops = {
> > +	.link_validate		= v4l2_subdev_link_validate,
> 
> 	.link_validate = v4l2_subdev_link_validate,
> 
> to match mali_c55_isp_internal_ops.
> 
> > +};
> > +
> > +static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
> > +				       struct v4l2_subdev *subdev,
> > +				       struct v4l2_async_connection *asc)
> > +{
> > +	struct mali_c55_isp *isp = container_of(notifier,
> > +						struct mali_c55_isp, notifier);
> > +	struct mali_c55 *mali_c55 = isp->mali_c55;
> > +	struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
> > +	int ret;
> > +
> > +	/*
> > +	 * By default we'll flag this link enabled and the TPG disabled, but
> > +	 * no immutable flag because we need to be able to switch between the
> > +	 * two.
> > +	 */
> > +	ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
> > +					      MEDIA_LNK_FL_ENABLED);
> > +	if (ret)
> > +		dev_err(mali_c55->dev, "failed to create link for %s\n",
> > +			subdev->name);
> > +
> > +	return ret;
> > +}
> > +
> > +static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
> > +{
> > +	struct mali_c55_isp *isp = container_of(notifier,
> > +						struct mali_c55_isp, notifier);
> > +	struct mali_c55 *mali_c55 = isp->mali_c55;
> > +
> > +	return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
> > +}
> > +
> > +static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
> > +	.bound = mali_c55_isp_notifier_bound,
> > +	.complete = mali_c55_isp_notifier_complete,
> > +};
> > +
> > +static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
> > +{
> > +	struct mali_c55 *mali_c55 = isp->mali_c55;
> > +	struct v4l2_async_connection *asc;
> > +	struct fwnode_handle *ep;
> > +	int ret;
> > +
> > +	v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
> > +
> > +	/*
> > +	 * The ISP should have a single endpoint pointing to some flavour of
> > +	 * CSI-2 receiver...but for now at least we do want everything to work
> > +	 * normally even with no sensors connected, as we have the TPG. If we
> > +	 * don't find a sensor just warn and return success.
> > +	 */
> > +	ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
> > +					     0, 0, 0);
> > +	if (!ep) {
> > +		dev_warn(mali_c55->dev, "no local endpoint found\n");
> > +		return 0;
> > +	}
> > +
> > +	asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
> > +					      struct v4l2_async_connection);
> > +	if (IS_ERR(asc)) {
> > +		dev_err(mali_c55->dev, "failed to add remote fwnode\n");
> > +		ret = PTR_ERR(asc);
> > +		goto err_put_ep;
> > +	}
> > +
> > +	isp->notifier.ops = &mali_c55_isp_notifier_ops;
> > +	ret = v4l2_async_nf_register(&isp->notifier);
> > +	if (ret) {
> > +		dev_err(mali_c55->dev, "failed to register notifier\n");
> > +		goto err_cleanup_nf;
> > +	}
> > +
> > +	fwnode_handle_put(ep);
> > +
> > +	return 0;
> > +
> > +err_cleanup_nf:
> > +	v4l2_async_nf_cleanup(&isp->notifier);
> > +err_put_ep:
> > +	fwnode_handle_put(ep);
> > +
> > +	return ret;
> > +}
> > +
> > +int mali_c55_register_isp(struct mali_c55 *mali_c55)
> > +{
> > +	struct mali_c55_isp *isp = &mali_c55->isp;
> > +	struct v4l2_subdev *sd = &isp->sd;
> > +	int ret;
> > +
> > +	isp->mali_c55 = mali_c55;
> > +
> > +	v4l2_subdev_init(sd, &mali_c55_isp_ops);
> > +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> > +	sd->entity.ops = &mali_c55_isp_media_ops;
> > +	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
> > +	sd->internal_ops = &mali_c55_isp_internal_ops;
> > +	strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
> > +
> > +	isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
> 
> The MUST_CONNECT flag would make sense here.
> 
> > +	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> > +	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
> > +
> > +	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
> > +				     isp->pads);
> > +	if (ret)
> > +		return ret;
> > +
> > +	ret = v4l2_subdev_init_finalize(sd);
> > +	if (ret)
> > +		goto err_cleanup_media_entity;
> > +
> > +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> > +	if (ret)
> > +		goto err_cleanup_subdev;
> > +
> > +	ret = mali_c55_isp_parse_endpoint(isp);
> > +	if (ret)
> > +		goto err_cleanup_subdev;
> 
> As noted elsewhere, I think this belongs to mali-c55-core.c.
> 
> > +
> > +	mutex_init(&isp->lock);
> 
> This lock is used in mali-c55-capture.c only, that seems weird.
> 
> > +
> > +	return 0;
> > +
> > +err_cleanup_subdev:
> > +	v4l2_subdev_cleanup(sd);
> > +err_cleanup_media_entity:
> > +	media_entity_cleanup(&sd->entity);
> > +	isp->mali_c55 = NULL;
> > +
> > +	return ret;
> > +}
> > +
> > +void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
> > +{
> > +	struct mali_c55_isp *isp = &mali_c55->isp;
> > +
> > +	if (!isp->mali_c55)
> > +		return;
> > +
> > +	mutex_destroy(&isp->lock);
> > +	v4l2_async_nf_unregister(&isp->notifier);
> > +	v4l2_async_nf_cleanup(&isp->notifier);
> > +	v4l2_device_unregister_subdev(&isp->sd);
> > +	v4l2_subdev_cleanup(&isp->sd);
> > +	media_entity_cleanup(&isp->sd.entity);
> > +}
> > diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> > new file mode 100644
> > index 000000000000..cb27abde2aa5
> > --- /dev/null
> > +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> > @@ -0,0 +1,258 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * ARM Mali-C55 ISP Driver - Register definitions
> > + *
> > + * Copyright (C) 2024 Ideas on Board Oy
> > + */
> > +
> > +#ifndef _MALI_C55_REGISTERS_H
> > +#define _MALI_C55_REGISTERS_H
> > +
> > +#include <linux/bits.h>
> > +
> > +/* ISP Common 0x00000 - 0x000ff */
> > +
> > +#define MALI_C55_REG_API				0x00000
> > +#define MALI_C55_REG_PRODUCT				0x00004
> > +#define MALI_C55_REG_VERSION				0x00008
> > +#define MALI_C55_REG_REVISION				0x0000c
> > +#define MALI_C55_REG_PULSE_MODE				0x0003c
> > +#define MALI_C55_REG_INPUT_MODE_REQUEST			0x0009c
> > +#define MALI_C55_INPUT_SAFE_STOP			0x00
> > +#define MALI_C55_INPUT_SAFE_START			0x01
> > +#define MALI_C55_REG_MODE_STATUS			0x000a0
> > +#define MALI_C55_REG_INTERRUPT_MASK_VECTOR		0x00030
> > +#define MALI_C55_INTERRUPT_MASK_ALL			GENMASK(31, 0)
> > +
> > +#define MALI_C55_REG_GLOBAL_MONITOR			0x00050
> > +
> > +#define MALI_C55_REG_GEN_VIDEO				0x00080
> > +#define MALI_C55_REG_GEN_VIDEO_ON_MASK			BIT(0)
> > +#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK		BIT(1)
> > +#define MALI_C55_REG_GEN_PREFETCH_MASK			GENMASK(31, 16)
> > +
> > +#define MALI_C55_REG_MCU_CONFIG				0x00020
> > +#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK		BIT(0)
> 
> #define MALI_C55_REG_MCU_CONFIG_OVERRIDE		BIT(0)
> 
> Same in other places where applicable.
> 
> > +#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK		BIT(1)
> > +#define MALI_C55_REG_MCU_CONFIG_WRITE_PING		BIT(1)
> > +#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG		0x00
> > +#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK		BIT(8)
> > +#define MALI_C55_REG_PING_PONG_READ			0x00024
> > +#define MALI_C55_REG_PING_PONG_READ_MASK		BIT(2)
> > +
> > +#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR		0x00034
> > +#define MALI_C55_REG_INTERRUPT_CLEAR			0x00040
> > +#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR		0x00044
> > +#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS		0x00068
> > +#define MALI_C55_GPS_PONG_FITTED			BIT(0)
> > +#define MALI_C55_GPS_WDR_FITTED				BIT(1)
> > +#define MALI_C55_GPS_COMPRESSION_FITTED			BIT(2)
> > +#define MALI_C55_GPS_TEMPER_FITTED			BIT(3)
> > +#define MALI_C55_GPS_SINTER_LITE_FITTED			BIT(4)
> > +#define MALI_C55_GPS_SINTER_FITTED			BIT(5)
> > +#define MALI_C55_GPS_IRIDIX_LTM_FITTED			BIT(6)
> > +#define MALI_C55_GPS_IRIDIX_GTM_FITTED			BIT(7)
> > +#define MALI_C55_GPS_CNR_FITTED				BIT(8)
> > +#define MALI_C55_GPS_FRSCALER_FITTED			BIT(9)
> > +#define MALI_C55_GPS_DS_PIPE_FITTED			BIT(10)
> > +
> > +#define MALI_C55_REG_BLANKING				0x00084
> > +#define MALI_C55_REG_HBLANK_MASK			GENMASK(15, 0)
> > +#define MALI_C55_REG_VBLANK_MASK			GENMASK(31, 16)
> > +
> > +#define MALI_C55_REG_HC_START				0x00088
> > +#define MALI_C55_HC_START(h)				(((h) & 0xffff) << 16)
> > +#define MALI_C55_REG_HC_SIZE				0x0008c
> > +#define MALI_C55_HC_SIZE(h)				((h) & 0xffff)
> > +#define MALI_C55_REG_VC_START_SIZE			0x00094
> > +#define MALI_C55_VC_START(v)				((v) & 0xffff)
> > +#define MALI_C55_VC_SIZE(v)				(((v) & 0xffff) << 16)
> > +
> > +/* Ping/Pong Configuration Space */
> > +#define MALI_C55_REG_BASE_ADDR				0x18e88
> > +#define MALI_C55_REG_BYPASS_0				0x18eac
> > +#define MALI_C55_REG_BYPASS_0_VIDEO_TEST		BIT(0)
> > +#define MALI_C55_REG_BYPASS_0_INPUT_FMT			BIT(1)
> > +#define MALI_C55_REG_BYPASS_0_DECOMPANDER		BIT(2)
> > +#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR		BIT(3)
> > +#define MALI_C55_REG_BYPASS_0_GAIN_WDR			BIT(4)
> > +#define MALI_C55_REG_BYPASS_0_FRAME_STITCH		BIT(5)
> > +#define MALI_C55_REG_BYPASS_1				0x18eb0
> > +#define MALI_C55_REG_BYPASS_1_DIGI_GAIN			BIT(0)
> > +#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS		BIT(1)
> > +#define MALI_C55_REG_BYPASS_1_FE_SQRT			BIT(2)
> > +#define MALI_C55_REG_BYPASS_1_RAW_FE			BIT(3)
> > +#define MALI_C55_REG_BYPASS_2				0x18eb8
> > +#define MALI_C55_REG_BYPASS_2_SINTER			BIT(0)
> > +#define MALI_C55_REG_BYPASS_2_TEMPER			BIT(1)
> > +#define MALI_C55_REG_BYPASS_3				0x18ebc
> > +#define MALI_C55_REG_BYPASS_3_SQUARE_BE			BIT(0)
> > +#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH	BIT(1)
> > +#define MALI_C55_REG_BYPASS_3_MESH_SHADING		BIT(3)
> > +#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE		BIT(4)
> > +#define MALI_C55_REG_BYPASS_3_IRIDIX			BIT(5)
> > +#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN		BIT(6)
> > +#define MALI_C55_REG_BYPASS_4				0x18ec0
> > +#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB		BIT(1)
> > +#define MALI_C55_REG_BYPASS_4_PF_CORRECTION		BIT(3)
> > +#define MALI_C55_REG_BYPASS_4_CCM			BIT(4)
> > +#define MALI_C55_REG_BYPASS_4_CNR			BIT(5)
> > +#define MALI_C55_REG_FR_BYPASS				0x18ec4
> > +#define MALI_C55_REG_DS_BYPASS				0x18ec8
> > +#define MALI_C55_BYPASS_CROP				BIT(0)
> > +#define MALI_C55_BYPASS_SCALER				BIT(1)
> > +#define MALI_C55_BYPASS_GAMMA_RGB			BIT(2)
> > +#define MALI_C55_BYPASS_SHARPEN				BIT(3)
> > +#define MALI_C55_BYPASS_CS_CONV				BIT(4)
> > +#define MALI_C55_REG_ISP_RAW_BYPASS			0x18ecc
> > +#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK		BIT(0)
> > +#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK		GENMASK(9, 8)
> > +#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS		2
> > +#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS		1
> > +#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE		BIT(1)
> > +#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS		BIT(0)
> > +
> > +#define MALI_C55_REG_ACTIVE_WIDTH_MASK			0xffff
> > +#define MALI_C55_REG_ACTIVE_HEIGHT_MASK			0xffff0000
> > +#define MALI_C55_REG_BAYER_ORDER			0x18e8c
> > +#define MALI_C55_BAYER_ORDER_MASK			GENMASK(1, 0)
> > +#define MALI_C55_REG_TPG_CH0				0x18ed8
> > +#define MALI_C55_TEST_PATTERN_ON_OFF			BIT(0)
> > +#define MALI_C55_TEST_PATTERN_RGB_MASK			BIT(1)
> > +#define MALI_C55_REG_TPG_R_BACKGROUND			0x18ee0
> > +#define MALI_C55_REG_TPG_G_BACKGROUND			0x18ee4
> > +#define MALI_C55_REG_TPG_B_BACKGROUND			0x18ee8
> > +#define MALI_C55_TPG_BACKGROUND_MAX			0xfffff
> > +#define MALI_C55_REG_INPUT_WIDTH			0x18f98
> > +#define MALI_C55_INPUT_WIDTH_MASK			GENMASK(18, 16)
> > +#define MALI_C55_INPUT_WIDTH_8BIT			0
> > +#define MALI_C55_INPUT_WIDTH_10BIT			1
> > +#define MALI_C55_INPUT_WIDTH_12BIT			2
> > +#define MALI_C55_INPUT_WIDTH_14BIT			3
> > +#define MALI_C55_INPUT_WIDTH_16BIT			4
> > +#define MALI_C55_INPUT_WIDTH_20BIT			5
> > +#define MALI_C55_REG_SPACE_SIZE				0x4000
> > +#define MALI_C55_REG_CONFIG_SPACES_OFFSET		0x0ab6c
> > +#define MALI_C55_CONFIG_SPACE_SIZE			0x1231c
> > +
> > +#define MALI_C55_REG_SINTER_CONFIG			0x19348
> > +#define MALI_C55_SINTER_VIEW_FILTER_MASK		GENMASK(1, 0)
> > +#define MALI_C55_SINTER_SCALE_MODE_MASK			GENMASK(3, 2)
> > +#define MALI_C55_SINTER_ENABLE_MASK			BIT(4)
> > +#define MALI_C55_SINTER_FILTER_SELECT_MASK		BIT(5)
> > +#define MALI_C55_SINTER_INT_SELECT_MASK			BIT(6)
> > +#define MALI_C55_SINTER_RM_ENABLE_MASK			BIT(7)
> > +
> > +/* Colour Correction Matrix Configuration */
> > +#define MALI_C55_REG_CCM_ENABLE				0x1b07c
> > +#define MALI_C55_CCM_ENABLE_MASK			BIT(0)
> > +#define MALI_C55_REG_CCM_COEF_R_R			0x1b080
> > +#define MALI_C55_REG_CCM_COEF_R_G			0x1b084
> > +#define MALI_C55_REG_CCM_COEF_R_B			0x1b088
> > +#define MALI_C55_REG_CCM_COEF_G_R			0x1b090
> > +#define MALI_C55_REG_CCM_COEF_G_G			0x1b094
> > +#define MALI_C55_REG_CCM_COEF_G_B			0x1b098
> > +#define MALI_C55_REG_CCM_COEF_B_R			0x1b0a0
> > +#define MALI_C55_REG_CCM_COEF_B_G			0x1b0a4
> > +#define MALI_C55_REG_CCM_COEF_B_B			0x1b0a8
> > +#define MALI_C55_CCM_COEF_MASK				GENMASK(12, 0)
> > +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R			0x1b0b0
> > +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G			0x1b0b4
> > +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B			0x1b0b8
> > +#define MALI_C55_CCM_ANTIFOG_GAIN_MASK			GENMASK(11, 0)
> > +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R		0x1b0c0
> > +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G		0x1b0c4
> > +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B		0x1b0c8
> > +#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK		GENMASK(11, 0)
> > +
> > +/*
> > + * The Mali-C55 ISP has up to two output pipes; known as full resolution and
> > + * down scaled. The register space for these is laid out identically, but offset
> > + * by 372 bytes.
> > + */
> > +#define MALI_C55_CAP_DEV_FR_REG_OFFSET		0x0
> > +#define MALI_C55_CAP_DEV_DS_REG_OFFSET		0x174
> > +
> > +#define MALI_C55_REG_CS_CONV_CONFIG(offset)		(0x1c098 + (offset))
> > +#define MALI_C55_CS_CONV_MATRIX_MASK			BIT(0)
> > +#define MALI_C55_CS_CONV_FILTER_MASK			BIT(1)
> > +#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK		BIT(2)
> > +#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK		BIT(3)
> > +#define MALI_C55_REG_Y_WRITER_MODE(offset)		(0x1c0ec + (offset))
> > +#define MALI_C55_REG_UV_WRITER_MODE(offset)		(0x1c144 + (offset))
> > +#define MALI_C55_WRITER_MODE_MASK			GENMASK(4, 0)
> > +#define MALI_C55_WRITER_SUBMODE_MASK			GENMASK(7, 6)
> > +#define MALI_C55_WRITER_FRAME_WRITE_MASK		BIT(9)
> > +#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset)		(0x1c0f0 + (offset))
> > +#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset)		(0x1c148 + (offset))
> > +#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)		((w) << 0)
> > +#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)		((h) << 16)
> > +#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset)	(0x1c0f4 + (offset))
> > +#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset)	(0x1c108 + (offset))
> > +#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
> > +#define MALI_C55_REG_Y_WRITER_BANKS_RESTART		BIT(3)
> > +#define MALI_C55_REG_Y_WRITER_OFFSET(offset)		(0x1c10c + (offset))
> > +#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset)	(0x1c14c + (offset))
> > +#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset)	(0x1c160 + (offset))
> > +#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
> > +#define MALI_C55_REG_UV_WRITER_BANKS_RESTART		BIT(3)
> > +#define MALI_C55_REG_UV_WRITER_OFFSET(offset)		(0x1c164 + (offset))
> > +
> > +#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
> > +#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE		0x18edc
> > +
> > +#define MALI_C55_REG_CROP_EN(offset)			(0x1c028 + (offset))
> > +#define MALI_C55_CROP_ENABLE				BIT(0)
> > +#define MALI_C55_REG_CROP_X_START(offset)		(0x1c02c + (offset))
> > +#define MALI_C55_REG_CROP_Y_START(offset)		(0x1c030 + (offset))
> > +#define MALI_C55_REG_CROP_X_SIZE(offset)		(0x1c034 + (offset))
> > +#define MALI_C55_REG_CROP_Y_SIZE(offset)		(0x1c038 + (offset))
> > +#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset)		(0x1c040 + (offset))
> > +#define MALI_C55_SCALER_TIMEOUT_EN			BIT(4)
> > +#define MALI_C55_SCALER_TIMEOUT(t)			((t) << 16)
> > +#define MALI_C55_REG_SCALER_IN_WIDTH(offset)		(0x1c044 + (offset))
> > +#define MALI_C55_REG_SCALER_IN_HEIGHT(offset)		(0x1c048 + (offset))
> > +#define MALI_C55_REG_SCALER_OUT_WIDTH(offset)		(0x1c04c + (offset))
> > +#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset)		(0x1c050 + (offset))
> > +#define MALI_C55_REG_SCALER_HFILT_TINC(offset)		(0x1c054 + (offset))
> > +#define MALI_C55_REG_SCALER_HFILT_COEF(offset)		(0x1c058 + (offset))
> > +#define MALI_C55_REG_SCALER_VFILT_TINC(offset)		(0x1c05c + (offset))
> > +#define MALI_C55_REG_SCALER_VFILT_COEF(offset)		(0x1c060 + (offset))
> > +
> > +#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset)		(0x1c064 + (offset))
> > +#define MALI_C55_GAMMA_ENABLE_MASK			BIT(0)
> > +#define MALI_C55_REG_GAMMA_GAINS_1(offset)		(0x1c068 + (offset))
> > +#define MALI_C55_GAMMA_GAIN_R_MASK			GENMASK(11, 0)
> > +#define MALI_C55_GAMMA_GAIN_G_MASK			GENMASK(27, 16)
> > +#define MALI_C55_REG_GAMMA_GAINS_2(offset)		(0x1c06c + (offset))
> > +#define MALI_C55_GAMMA_GAIN_B_MASK			GENMASK(11, 0)
> > +#define MALI_C55_REG_GAMMA_OFFSETS_1(offset)		(0x1c070 + (offset))
> > +#define MALI_C55_GAMMA_OFFSET_R_MASK			GENMASK(11, 0)
> > +#define MALI_C55_GAMMA_OFFSET_G_MASK			GENMASK(27, 16)
> > +#define MALI_C55_REG_GAMMA_OFFSETS_2(offset)		(0x1c074 + (offset))
> > +#define MALI_C55_GAMMA_OFFSET_B_MASK			GENMASK(11, 0)
> > +
> > +/* Output DMA Writer */
> > +
> > +#define MALI_C55_OUTPUT_DISABLED		0
> > +#define MALI_C55_OUTPUT_RGB32			1
> > +#define MALI_C55_OUTPUT_A2R10G10B10		2
> > +#define MALI_C55_OUTPUT_RGB565			3
> > +#define MALI_C55_OUTPUT_RGB24			4
> > +#define MALI_C55_OUTPUT_GEN32			5
> > +#define MALI_C55_OUTPUT_RAW16			6
> > +#define MALI_C55_OUTPUT_AYUV			8
> > +#define MALI_C55_OUTPUT_Y410			9
> > +#define MALI_C55_OUTPUT_YUY2			10
> > +#define MALI_C55_OUTPUT_UYVY			11
> > +#define MALI_C55_OUTPUT_Y210			12
> > +#define MALI_C55_OUTPUT_NV12_21			13
> > +#define MALI_C55_OUTPUT_YUV_420_422		17
> > +#define MALI_C55_OUTPUT_P210_P010		19
> > +#define MALI_C55_OUTPUT_YUV422			20
> 
> I'd define this just below the MALI_C55_REG_UV_WRITER_MODE register
> macro.
> 
> > +
> > +#define MALI_C55_OUTPUT_PLANE_ALT0		0
> > +#define MALI_C55_OUTPUT_PLANE_ALT1		1
> > +#define MALI_C55_OUTPUT_PLANE_ALT2		2
> 
> Same here ?
> 
> > +
> > +#endif /* _MALI_C55_REGISTERS_H */
> > diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> > new file mode 100644
> > index 000000000000..8edae87f1e5f
> > --- /dev/null
> > +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> > @@ -0,0 +1,382 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * ARM Mali-C55 ISP Driver - Resizer Coefficients
> > + *
> > + * Copyright (C) 2024 Ideas on Board Oy
> > + */
> > +
> > +#ifndef _MALI_C55_RESIZER_COEFS_H
> > +#define _MALI_C55_RESIZER_COEFS_H
> > +
> > +#include "mali-c55-common.h"
> > +
> > +#define MALI_C55_RESIZER_COEFS_NUM_BANKS	8
> > +#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES	64
> 
> Do these belongs to mali-c55-registers.h ?
> 
> > +
> > +static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
> > +	{	/* Bank 0 */
> > +		0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
> > +		0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
> > +		0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
> > +		0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
> > +		0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
> > +		0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
> > +		0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
> > +		0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
> > +		0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
> > +		0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
> > +		0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
> > +		0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
> > +		0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
> > +		0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
> > +		0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
> > +		0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
> > +	},
> > +	{	/* Bank 1 */
> > +		0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
> > +		0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
> > +		0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
> > +		0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
> > +		0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
> > +		0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
> > +		0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
> > +		0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
> > +		0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
> > +		0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
> > +		0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
> > +		0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
> > +		0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
> > +		0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
> > +		0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
> > +		0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
> > +	},
> > +	{	/* Bank 2 */
> > +		0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
> > +		0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
> > +		0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
> > +		0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
> > +		0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
> > +		0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
> > +		0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
> > +		0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
> > +		0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
> > +		0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
> > +		0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
> > +		0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
> > +		0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
> > +		0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
> > +		0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
> > +		0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
> > +	},
> > +	{	/* Bank 3 */
> > +		0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
> > +		0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
> > +		0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
> > +		0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
> > +		0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
> > +		0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
> > +		0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
> > +		0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
> > +		0x20100000, 0x00000010, 0x1f110000, 0x00000010,
> > +		0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
> > +		0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
> > +		0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
> > +		0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
> > +		0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
> > +		0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
> > +		0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
> > +	},
> > +	{	/* Bank 4 */
> > +		0x17090000, 0x00000917, 0x18090000, 0x00000916,
> > +		0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
> > +		0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
> > +		0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
> > +		0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
> > +		0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
> > +		0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
> > +		0x190f0300, 0x00000411, 0x18100300, 0x00000411,
> > +		0x1a100300, 0x00000310, 0x18110400, 0x00000310,
> > +		0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
> > +		0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
> > +		0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
> > +		0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
> > +		0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
> > +		0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
> > +		0x17160800, 0x0000010a, 0x18160900, 0x00000009,
> > +	},
> > +	{	/* Bank 5 */
> > +		0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
> > +		0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
> > +		0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
> > +		0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
> > +		0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
> > +		0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
> > +		0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
> > +		0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
> > +		0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
> > +		0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
> > +		0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
> > +		0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
> > +		0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
> > +		0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
> > +		0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
> > +		0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
> > +	},
> > +	{	/* Bank 6 */
> > +		0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
> > +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
> > +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
> > +		0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
> > +		0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
> > +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> > +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> > +		0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
> > +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
> > +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> > +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> > +		0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
> > +		0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
> > +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
> > +		0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> > +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> > +	},
> > +	{	/* Bank 7 */
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> > +	}
> > +};
> > +
> > +static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
> > +	{	/* Bank 0 */
> > +		0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
> > +		0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
> > +		0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
> > +		0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
> > +		0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
> > +		0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
> > +		0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
> > +		0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
> > +		0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
> > +		0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
> > +		0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
> > +		0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
> > +		0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
> > +		0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
> > +		0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
> > +		0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
> > +	},
> > +	{	/* Bank 1 */
> > +		0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
> > +		0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
> > +		0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
> > +		0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
> > +		0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
> > +		0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
> > +		0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
> > +		0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
> > +		0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
> > +		0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
> > +		0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
> > +		0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
> > +		0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
> > +		0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
> > +		0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
> > +		0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
> > +	},
> > +	{	/* Bank 2 */
> > +		0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
> > +		0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
> > +		0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
> > +		0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
> > +		0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
> > +		0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
> > +		0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
> > +		0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
> > +		0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
> > +		0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
> > +		0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
> > +		0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
> > +		0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
> > +		0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
> > +		0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
> > +		0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
> > +	},
> > +	{	/* Bank 3 */
> > +		0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
> > +		0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
> > +		0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
> > +		0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
> > +		0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
> > +		0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
> > +		0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
> > +		0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
> > +		0x20100000, 0x00000010, 0x1f100000, 0x00000011,
> > +		0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
> > +		0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
> > +		0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
> > +		0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
> > +		0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
> > +		0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
> > +		0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
> > +	},
> > +	{	/* Bank 4 */
> > +		0x17170900, 0x00000009, 0x18160900, 0x00000009,
> > +		0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
> > +		0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
> > +		0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
> > +		0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
> > +		0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
> > +		0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
> > +		0x19110400, 0x0000030f, 0x18110400, 0x00000310,
> > +		0x1a100300, 0x00000310, 0x18100300, 0x00000411,
> > +		0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
> > +		0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
> > +		0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
> > +		0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
> > +		0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
> > +		0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
> > +		0x170a0100, 0x00000816, 0x18090000, 0x00000916,
> > +	},
> > +	{	/* Bank 5 */
> > +		0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
> > +		0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
> > +		0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
> > +		0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
> > +		0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
> > +		0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
> > +		0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
> > +		0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
> > +		0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
> > +		0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
> > +		0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
> > +		0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
> > +		0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
> > +		0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
> > +		0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
> > +		0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
> > +	},
> > +	{	/* Bank 6 */
> > +		0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
> > +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> > +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
> > +		0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
> > +		0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
> > +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> > +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> > +		0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
> > +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
> > +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> > +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> > +		0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
> > +		0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
> > +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
> > +		0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
> > +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
> > +	},
> > +	{	/* Bank 7 */
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> > +	}
> > +};
> > +
> > +struct mali_c55_resizer_coef_bank {
> > +	unsigned int bank;
> 
> This is always equal to the index of the entry in the
> mali_c55_coefficient_banks array, you can drop it.
> 
> > +	unsigned int top;
> > +	unsigned int bottom;
> 
> The bottom value of bank N is always equal to the top value of bank N+1.
> You can simplify this by storing a single value.
> 
> > +};
> > +
> > +static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
> > +	{
> > +		.bank = 0,
> > +		.top = 1000,
> > +		.bottom = 770,
> > +	},
> > +	{
> > +		.bank = 1,
> > +		.top = 769,
> > +		.bottom = 600,
> > +	},
> > +	{
> > +		.bank = 2,
> > +		.top = 599,
> > +		.bottom = 460,
> > +	},
> > +	{
> > +		.bank = 3,
> > +		.top = 459,
> > +		.bottom = 354,
> > +	},
> > +	{
> > +		.bank = 4,
> > +		.top = 353,
> > +		.bottom = 273,
> > +	},
> > +	{
> > +		.bank = 5,
> > +		.top = 272,
> > +		.bottom = 210,
> > +	},
> > +	{
> > +		.bank = 6,
> > +		.top = 209,
> > +		.bottom = 162,
> > +	},
> > +	{
> > +		.bank = 7,
> > +		.top = 161,
> > +		.bottom = 125,
> > +	},
> > +};
> > +
> 
> A small comment would be nice, such as
> 
> /* Select a bank of resizer coefficients, based on the scaling ratio. */
> 
> > +static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,
> 
> This function is related to the resizers. Add "rsz" somewhere in the
> function name, and pass a resizer pointer.
> 
> > +						unsigned int crop,
> > +						unsigned int scale)
> 
> I think those are the input and output sizes to the scaler. Rename them
> to make it clearer.
> 
> > +{
> > +	unsigned int tmp;
> 
> tmp is almost always a bad variable name. Please use a more descriptive
> name, size as rsz_ratio.
> 
> > +	unsigned int i;
> > +
> > +	tmp = (scale * 1000U) / crop;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
> > +		if (tmp >= mali_c55_coefficient_banks[i].bottom &&
> > +		    tmp <= mali_c55_coefficient_banks[i].top)
> > +			return mali_c55_coefficient_banks[i].bank;
> > +	}
> > +
> > +	/*
> > +	 * We shouldn't ever get here, in theory. As we have no good choices
> > +	 * simply warn the user and use the first bank of coefficients.
> > +	 */
> > +	dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
> > +	return 0;
> > +}
> 
> And everything else belongs to mali-c55-resizer.c. Drop this header
> file.
> 
> > +
> > +#endif /* _MALI_C55_RESIZER_COEFS_H */
> > diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> > new file mode 100644
> > index 000000000000..0a5a2969d3ce
> > --- /dev/null
> > +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> > @@ -0,0 +1,779 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * ARM Mali-C55 ISP Driver - Image signal processor
> > + *
> > + * Copyright (C) 2024 Ideas on Board Oy
> > + */
> > +
> > +#include <linux/math.h>
> > +#include <linux/minmax.h>
> > +
> > +#include <media/media-entity.h>
> > +#include <media/v4l2-subdev.h>
> > +
> > +#include "mali-c55-common.h"
> > +#include "mali-c55-registers.h"
> > +#include "mali-c55-resizer-coefs.h"
> > +
> > +/* Scaling factor in Q4.20 format. */
> > +#define MALI_C55_RZR_SCALER_FACTOR	(1U << 20)
> > +
> > +static const u32 rzr_non_bypass_src_fmts[] = {
> > +	MEDIA_BUS_FMT_RGB121212_1X36,
> > +	MEDIA_BUS_FMT_YUV10_1X30
> > +};
> > +
> > +static const char * const mali_c55_resizer_names[] = {
> > +	[MALI_C55_RZR_FR] = "resizer fr",
> > +	[MALI_C55_RZR_DS] = "resizer ds",
> > +};
> > +
> > +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
> > +				     struct v4l2_subdev_state *state)
> > +{
> > +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
> > +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> > +	struct v4l2_mbus_framefmt *fmt;
> > +	struct v4l2_rect *crop;

const

> > +
> > +	/* Verify if crop should be enabled. */
> > +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
> > +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> > +
> > +	if (fmt->width == crop->width && fmt->height == crop->height)
> > +		return MALI_C55_BYPASS_CROP;
> > +
> > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
> > +		       crop->left);
> > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
> > +		       crop->top);
> > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
> > +		       crop->width);
> > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
> > +		       crop->height);
> > +
> > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
> > +		       MALI_C55_CROP_ENABLE);
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
> > +					struct v4l2_subdev_state *state)
> > +{
> > +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
> > +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> > +	struct v4l2_rect *crop, *scale;

const

Once "[PATCH v4 0/3] media: v4l2-subdev: Support const-awareness in
state accessors" gets merged, the state argument to this function can be
made const too. Same for other functions, as applicable.

> > +	unsigned int h_bank, v_bank;
> > +	u64 h_scale, v_scale;
> > +
> > +	/* Verify if scaling should be enabled. */
> > +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> > +	scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
> > +
> > +	if (crop->width == scale->width && crop->height == scale->height)
> > +		return MALI_C55_BYPASS_SCALER;
> > +
> > +	/* Program the V/H scaling factor in Q4.20 format. */
> > +	h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
> > +	v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
> > +
> > +	do_div(h_scale, scale->width);
> > +	do_div(v_scale, scale->height);
> > +
> > +	mali_c55_write(mali_c55,
> > +		       MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
> > +		       crop->width);
> > +	mali_c55_write(mali_c55,
> > +		       MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
> > +		       crop->height);
> > +
> > +	mali_c55_write(mali_c55,
> > +		       MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
> > +		       scale->width);
> > +	mali_c55_write(mali_c55,
> > +		       MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
> > +		       scale->height);
> > +
> > +	mali_c55_write(mali_c55,
> > +		       MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
> > +		       h_scale);
> > +	mali_c55_write(mali_c55,
> > +		       MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
> > +		       v_scale);
> > +
> > +	h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
> > +					     scale->width);
> > +	mali_c55_write(mali_c55,
> > +		       MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
> > +		       h_bank);
> > +
> > +	v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
> > +					     scale->height);
> > +	mali_c55_write(mali_c55,
> > +		       MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
> > +		       v_bank);
> > +
> > +	return 0;
> > +}
> > +
> > +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
> > +				 struct v4l2_subdev_state *state)
> > +{
> > +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> > +	u32 bypass = 0;
> > +
> > +	/* Verify if cropping and scaling should be enabled. */
> > +	bypass |= mali_c55_rzr_program_crop(rzr, state);
> > +	bypass |= mali_c55_rzr_program_resizer(rzr, state);
> > +
> > +	mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
> > +			     MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
> > +			     MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
> > +			     bypass);
> > +}
> > +
> > +/*
> > + * Inspect the routing table to know which of the two (mutually exclusive)
> > + * routes is enabled and return the sink pad id of the active route.
> > + */
> > +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
> > +{
> > +	struct v4l2_subdev_krouting *routing = &state->routing;
> > +	struct v4l2_subdev_route *route;
> > +
> > +	/* A single route is enabled at a time. */
> > +	for_each_active_route(routing, route)
> > +		return route->sink_pad;
> > +
> > +	return MALI_C55_RZR_SINK_PAD;
> > +}
> > +
> > +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
> > +{
> > +	u32 corrected_code = 0;
> > +
> > +	/*
> > +	 * The ISP takes input in a 20-bit format, but can only output 16-bit
> > +	 * RAW bayer data (with the 4 least significant bits from the input
> > +	 * being lost). Return the 16-bit version of the 20-bit input formats.
> > +	 */
> > +	switch (mbus_code) {
> > +	case MEDIA_BUS_FMT_SBGGR20_1X20:
> > +		corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
> > +		break;
> > +	case MEDIA_BUS_FMT_SGBRG20_1X20:
> > +		corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
> > +		break;
> > +	case MEDIA_BUS_FMT_SGRBG20_1X20:
> > +		corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
> > +		break;
> > +	case MEDIA_BUS_FMT_SRGGB20_1X20:
> > +		corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
> > +		break;

Would it make sense to add the shifted code to mali_c55_isp_fmt ?

> > +	}
> > +
> > +	return corrected_code;
> > +}
> > +
> > +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> > +				      struct v4l2_subdev_state *state,
> > +				      struct v4l2_subdev_krouting *routing)

I think the last argument can be const.

> > +{
> > +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> > +						    sd);

A to_mali_c55_resizer() static inline function would be useful. Same for
other components, where applicable.

> > +	unsigned int active_sink = UINT_MAX;
> > +	struct v4l2_mbus_framefmt *src_fmt;
> > +	struct v4l2_rect *crop, *compose;
> > +	struct v4l2_subdev_route *route;
> > +	unsigned int active_routes = 0;
> > +	struct v4l2_mbus_framefmt *fmt;
> > +	int ret;
> > +
> > +	ret = v4l2_subdev_routing_validate(sd, routing, 0);
> > +	if (ret)
> > +		return ret;
> > +
> > +	/* Only a single route can be enabled at a time. */
> > +	for_each_active_route(routing, route) {
> > +		if (++active_routes > 1) {
> > +			dev_err(rzr->mali_c55->dev,
> > +				"Only one route can be active");

No kernel log message with a level higher than dev_dbg() from
user-controlled paths please, here and where applicable. This is to
avoid giving applications an easy way to flood the kernel log.

> > +			return -EINVAL;
> > +		}
> > +
> > +		active_sink = route->sink_pad;
> > +	}
> > +	if (active_sink == UINT_MAX) {
> > +		dev_err(rzr->mali_c55->dev, "One route has to be active");
> > +		return -EINVAL;
> > +	}

The recommended handling of invalid routing is to adjust the routing
table, not to return errors.

> > +
> > +	ret = v4l2_subdev_set_routing(sd, state, routing);
> > +	if (ret) {
> > +		dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
> > +		return ret;
> > +	}
> > +
> > +	fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
> > +	crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
> > +	compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
> > +
> > +	fmt->width = MALI_C55_DEFAULT_WIDTH;
> > +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
> > +	fmt->colorspace = V4L2_COLORSPACE_SRGB;

There are other colorspace-related fields.

> > +	fmt->field = V4L2_FIELD_NONE;

I wonder if we should really update the sink pad format, or just
propagate it. If we update it, I think it should be set to defaults on
both sink pads, not just the active sink pad.

> > +
> > +	if (active_sink == MALI_C55_RZR_SINK_PAD) {
> > +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > +
> > +		crop->left = crop->top = 0;

		crop->left = 0;
		crop->top = 0;

> > +		crop->width = MALI_C55_DEFAULT_WIDTH;
> > +		crop->height = MALI_C55_DEFAULT_HEIGHT;
> > +
> > +		*compose = *crop;
> > +	} else {
> > +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> > +	}
> > +
> > +	/* Propagate the format to the source pad */
> > +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
> > +					       0);
> > +	*src_fmt = *fmt;
> > +
> > +	/* In the event this is the bypass pad the mbus code needs correcting */
> > +	if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
> > +		src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
> > +				       struct v4l2_subdev_state *state,
> > +				       struct v4l2_subdev_mbus_code_enum *code)
> > +{
> > +	struct v4l2_mbus_framefmt *sink_fmt;
> > +	const struct mali_c55_isp_fmt *fmt;
> > +	unsigned int index = 0;
> > +	u32 sink_pad;
> > +
> > +	switch (code->pad) {
> > +	case MALI_C55_RZR_SINK_PAD:
> > +		if (code->index)
> > +			return -EINVAL;
> > +
> > +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > +
> > +		return 0;
> > +	case MALI_C55_RZR_SOURCE_PAD:
> > +		sink_pad = mali_c55_rzr_get_active_sink(state);
> > +		sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> > +
> > +		/*
> > +		 * If the active route is from the Bypass sink pad, then the
> > +		 * source pad is a simple passthrough of the sink format,
> > +		 * downshifted to 16-bits.
> > +		 */
> > +
> > +		if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> > +			if (code->index)
> > +				return -EINVAL;
> > +
> > +			code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> > +			if (!code->code)
> > +				return -EINVAL;
> > +
> > +			return 0;
> > +		}
> > +
> > +		/*
> > +		 * If the active route is from the non-bypass sink then we can
> > +		 * select either RGB or conversion to YUV.
> > +		 */
> > +
> > +		if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
> > +			return -EINVAL;
> > +
> > +		code->code = rzr_non_bypass_src_fmts[code->index];
> > +
> > +		return 0;
> > +	case MALI_C55_RZR_SINK_BYPASS_PAD:
> > +		for_each_mali_isp_fmt(fmt) {
> > +			if (index++ == code->index) {
> > +				code->code = fmt->code;
> > +				return 0;
> > +			}
> > +		}
> > +
> > +		break;
> > +	}
> > +
> > +	return -EINVAL;
> > +}
> > +
> > +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
> > +					struct v4l2_subdev_state *state,
> > +					struct v4l2_subdev_frame_size_enum *fse)
> > +{
> > +	if (fse->index)
> > +		return -EINVAL;
> > +
> > +	fse->max_width = MALI_C55_MAX_WIDTH;
> > +	fse->max_height = MALI_C55_MAX_HEIGHT;
> > +	fse->min_width = MALI_C55_MIN_WIDTH;
> > +	fse->min_height = MALI_C55_MIN_HEIGHT;
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
> > +				     struct v4l2_subdev_state *state,
> > +				     struct v4l2_subdev_format *format)
> > +{
> > +	struct v4l2_mbus_framefmt *fmt = &format->format;
> > +	struct v4l2_rect *rect;
> > +	unsigned int sink_pad;
> > +
> > +	/*
> > +	 * Clamp to min/max and then reset crop and compose rectangles to the
> > +	 * newly applied size.
> > +	 */
> > +	clamp_t(unsigned int, fmt->width,
> > +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> > +	clamp_t(unsigned int, fmt->height,
> > +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);

Please check comments for other components related to the colorspace
fields, to decide how to handle them here.

> > +
> > +	sink_pad = mali_c55_rzr_get_active_sink(state);
> > +	if (sink_pad == MALI_C55_RZR_SINK_PAD) {

The selection here should depend on format->pad, not the active sink
pad.

> > +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > +
> > +		rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> > +		rect->left = 0;
> > +		rect->top = 0;
> > +		rect->width = fmt->width;
> > +		rect->height = fmt->height;
> > +
> > +		rect = v4l2_subdev_state_get_compose(state,
> > +						     MALI_C55_RZR_SINK_PAD);
> > +		rect->left = 0;
> > +		rect->top = 0;
> > +		rect->width = fmt->width;
> > +		rect->height = fmt->height;
> > +	} else {
> > +		/*
> > +		 * Make sure the media bus code is one of the supported
> > +		 * ISP input media bus codes.
> > +		 */
> > +		if (!mali_c55_isp_is_format_supported(fmt->code))
> > +			fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
> > +	}
> > +
> > +	*v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
> > +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;

Propagation to the source pad, however, should depend on the active
route. If format->pad is routed to the source pad, you should propagate,
otherwise, you shouldn't.

> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> > +				       struct v4l2_subdev_state *state,
> > +				       struct v4l2_subdev_format *format)
> > +{
> > +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> > +						    sd);
> > +	struct v4l2_mbus_framefmt *fmt = &format->format;
> > +	struct v4l2_mbus_framefmt *sink_fmt;
> > +	struct v4l2_rect *crop, *compose;
> > +	unsigned int sink_pad;
> > +	unsigned int i;
> > +
> > +	sink_pad = mali_c55_rzr_get_active_sink(state);
> > +	sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> > +	crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
> > +	compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
> > +
> > +	/* FR Bypass pipe. */
> > +
> > +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> > +		/*
> > +		 * Format on the source pad is the same as the one on the
> > +		 * sink pad, downshifted to 16-bits.
> > +		 */
> > +		fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> > +		if (!fmt->code)
> > +			return -EINVAL;
> > +
> > +		/* RAW bypass disables scaling and cropping. */
> > +		crop->top = compose->top = 0;
> > +		crop->left = compose->left = 0;
> > +		fmt->width = crop->width = compose->width = sink_fmt->width;
> > +		fmt->height = crop->height = compose->height = sink_fmt->height;

I don't think this is right. This function sets the format on the source
pad. Subdevs should propagate formats from the sink to the source, not
the other way around.

The only parameter that can be modified on the source pad (as far as I
understand) is the media bus code. In the bypass path, I understand it's
fixed, while in the other path, you can select between RGB and YUV. I
think the following code is what you need to implement this function.

static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
				       struct v4l2_subdev_state *state,
				       struct v4l2_subdev_format *format)
{
	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
						    sd);
	struct v4l2_mbus_framefmt *fmt;

	fmt = v4l2_subdev_state_get_format(state, format->pad);

	/* In the non-bypass path the output format can be selected. */
	if (mali_c55_rzr_get_active_sink(state) == MALI_C55_RZR_SINK_PAD) {
		unsigned int i;

		fmt->code = format->format.code;

		for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
			if (fmt->code == rzr_non_bypass_src_fmts[i])
				break;
		}

		if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts))
			fmt->code = rzr_non_bypass_src_fmts[0];
	}

	format->format = *fmt;

	return 0;
}

> > +
> > +		*v4l2_subdev_state_get_format(state,
> > +					      MALI_C55_RZR_SOURCE_PAD) = *fmt;
> > +
> > +		return 0;
> > +	}
> > +
> > +	/* Regular processing pipe. */
> > +
> > +	for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> > +		if (fmt->code == rzr_non_bypass_src_fmts[i])
> > +			break;
> > +	}
> > +
> > +	if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
> > +		dev_dbg(rzr->mali_c55->dev,
> > +			"Unsupported mbus code 0x%x: using default\n",
> > +			fmt->code);

I think you can drop this message.

> > +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > +	}
> > +
> > +	/*
> > +	 * The source pad format size comes directly from the sink pad
> > +	 * compose rectangle.
> > +	 */
> > +	fmt->width = compose->width;
> > +	fmt->height = compose->height;
> > +
> > +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
> > +				struct v4l2_subdev_state *state,
> > +				struct v4l2_subdev_format *format)
> > +{
> > +	/*
> > +	 * On sink pads fmt is either fixed for the 'regular' processing
> > +	 * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
> > +	 * pad.
> > +	 *
> > +	 * On source pad sizes are the result of crop+compose on the sink
> > +	 * pad sizes, while the format depends on the active route.
> > +	 */
> > +
> > +	if (format->pad != MALI_C55_RZR_SOURCE_PAD)
> > +		return mali_c55_rzr_set_sink_fmt(sd, state, format);
> > +
> > +	return mali_c55_rzr_set_source_fmt(sd, state, format);

Nitpicking,

	if (format->pad == MALI_C55_RZR_SOURCE_PAD)
		return mali_c55_rzr_set_source_fmt(sd, state, format);

	return mali_c55_rzr_set_sink_fmt(sd, state, format);

to match SOURCE_PAD and source_fmt.

> > +}
> > +
> > +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
> > +				      struct v4l2_subdev_state *state,
> > +				      struct v4l2_subdev_selection *sel)
> > +{
> > +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
> > +		return -EINVAL;
> > +
> > +	if (sel->target != V4L2_SEL_TGT_CROP &&
> > +	    sel->target != V4L2_SEL_TGT_COMPOSE)
> > +		return -EINVAL;
> > +
> > +	sel->r = sel->target == V4L2_SEL_TGT_CROP
> > +	       ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
> > +	       : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
> > +				      struct v4l2_subdev_state *state,
> > +				      struct v4l2_subdev_selection *sel)
> > +{
> > +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> > +						    sd);
> > +	struct v4l2_mbus_framefmt *source_fmt;
> > +	struct v4l2_mbus_framefmt *sink_fmt;
> > +	struct v4l2_rect *crop, *compose;
> > +
> > +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
> > +		return -EINVAL;
> > +
> > +	if (sel->target != V4L2_SEL_TGT_CROP &&
> > +	    sel->target != V4L2_SEL_TGT_COMPOSE)
> > +		return -EINVAL;
> > +
> > +	source_fmt = v4l2_subdev_state_get_format(state,
> > +						  MALI_C55_RZR_SOURCE_PAD);
> > +	sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
> > +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> > +	compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> > +
> > +	/* RAW bypass disables crop/scaling. */
> > +	if (mali_c55_format_is_raw(source_fmt->code)) {
> > +		crop->top = compose->top = 0;
> > +		crop->left = compose->left = 0;
> > +		crop->width = compose->width = sink_fmt->width;
> > +		crop->height = compose->height = sink_fmt->height;
> > +
> > +		sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> > +
> > +		return 0;
> > +	}
> > +
> > +	/* During streaming, it is allowed to only change the crop rectangle. */
> > +	if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
> > +		return -EINVAL;
> > +
> > +	 /*
> > +	  * Update the desired target and then clamp the crop rectangle to the
> > +	  * sink format sizes and the compose size to the crop sizes.
> > +	  */
> > +	if (sel->target == V4L2_SEL_TGT_CROP)
> > +		*crop = sel->r;
> > +	else
> > +		*compose = sel->r;
> > +
> > +	clamp_t(unsigned int, crop->left, 0,  sink_fmt->width);
> > +	clamp_t(unsigned int, crop->top, 0,  sink_fmt->height);
> > +	clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
> > +		sink_fmt->width - crop->left);
> > +	clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
> > +		sink_fmt->height - crop->top);
> > +
> > +	if (rzr->streaming) {
> > +		/*
> > +		 * Apply at runtime a crop rectangle on the resizer's sink only
> > +		 * if it doesn't require re-programming the scaler output sizes
> > +		 * as it would require changing the output buffer sizes as well.
> > +		 */
> > +		if (sel->r.width < compose->width ||
> > +		    sel->r.height < compose->height)
> > +			return -EINVAL;
> > +
> > +		*crop = sel->r;
> > +		mali_c55_rzr_program(rzr, state);
> > +
> > +		return 0;
> > +	}
> > +
> > +	compose->left = 0;
> > +	compose->top = 0;
> > +	clamp_t(unsigned int, compose->left, 0,  sink_fmt->width);
> > +	clamp_t(unsigned int, compose->top, 0,  sink_fmt->height);
> > +	clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
> > +	clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
> > +
> > +	sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> > +				    struct v4l2_subdev_state *state,
> > +				    enum v4l2_subdev_format_whence which,
> > +				    struct v4l2_subdev_krouting *routing)
> > +{
> > +	if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
> > +	    media_entity_is_streaming(&sd->entity))
> > +		return -EBUSY;
> > +
> > +	return __mali_c55_rzr_set_routing(sd, state, routing);
> > +}
> > +
> > +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
> > +	.enum_mbus_code		= mali_c55_rzr_enum_mbus_code,
> > +	.enum_frame_size	= mali_c55_rzr_enum_frame_size,
> > +	.get_fmt		= v4l2_subdev_get_fmt,
> > +	.set_fmt		= mali_c55_rzr_set_fmt,
> > +	.get_selection		= mali_c55_rzr_get_selection,
> > +	.set_selection		= mali_c55_rzr_set_selection,
> > +	.set_routing		= mali_c55_rzr_set_routing,
> > +};
> > +
> > +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)

Could this be handled through the .enable_streams() and
.disable_streams() operations ? They ensure that the stream state stored
internal is correct. That may not matter much today, but I think it will
become increasingly important in the future for the V4L2 core.

> > +{
> > +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> > +	struct v4l2_subdev *sd = &rzr->sd;
> > +	struct v4l2_subdev_state *state;
> > +	unsigned int sink_pad;
> > +
> > +	state = v4l2_subdev_lock_and_get_active_state(sd);
> > +
> > +	sink_pad = mali_c55_rzr_get_active_sink(state);
> > +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> > +		/* Bypass FR pipe processing if the bypass route is active. */
> > +		mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> > +				     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
> > +				     MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
> > +		goto unlock_state;
> > +	}
> > +
> > +	/* Disable bypass and use regular processing. */
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> > +			     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
> > +	mali_c55_rzr_program(rzr, state);
> > +
> > +unlock_state:
> > +	rzr->streaming = true;

And hopefully you'll be able to replace this with
v4l2_subdev_is_streaming(), introduced in "[PATCH v6 00/11] media:
subdev: Improve stream enable/disable machinery" (Sakari has sent a pull
request for v6.11 yesterday).

> > +	v4l2_subdev_unlock_state(state);
> > +}
> > +
> > +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
> > +{
> > +	struct v4l2_subdev *sd = &rzr->sd;
> > +	struct v4l2_subdev_state *state;
> > +
> > +	state = v4l2_subdev_lock_and_get_active_state(sd);
> > +	rzr->streaming = false;
> > +	v4l2_subdev_unlock_state(state);
> > +}
> > +
> > +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
> > +	.pad	= &mali_c55_resizer_pad_ops,
> > +};
> > +
> > +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
> > +				   struct v4l2_subdev_state *state)
> > +{
> > +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> > +						    sd);
> > +	struct v4l2_subdev_krouting routing = { };
> > +	struct v4l2_subdev_route *routes;
> > +	unsigned int i;
> > +	int ret;
> > +
> > +	routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
> > +	if (!routes)
> > +		return -ENOMEM;
> > +
> > +	for (i = 0; i < rzr->num_routes; ++i) {
> > +		struct v4l2_subdev_route *route = &routes[i];
> > +
> > +		route->sink_pad = i
> > +				? MALI_C55_RZR_SINK_BYPASS_PAD
> > +				: MALI_C55_RZR_SINK_PAD;
> > +		route->source_pad = MALI_C55_RZR_SOURCE_PAD;
> > +		if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
> > +			route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
> > +	}
> > +
> > +	routing.num_routes = rzr->num_routes;
> > +	routing.routes = routes;
> > +
> > +	ret = __mali_c55_rzr_set_routing(sd, state, &routing);
> > +	kfree(routes);
> > +
> > +	return ret;

I think this could be simplified.

	struct v4l2_subdev_route routes[2] = {
		{
			.sink_pad = MALI_C55_RZR_SINK_PAD,
			.source_pad = MALI_C55_RZR_SOURCE_PAD,
			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
		}, {
			.sink_pad = MALI_C55_RZR_SINK_BYPASS_PAD,
			.source_pad = MALI_C55_RZR_SOURCE_PAD,
		},
	};
	struct v4l2_subdev_krouting routing = {
		.num_routes = rzr->num_routes,
		.routes = routes,
	};

	return __mali_c55_rzr_set_routing(sd, state, &routing);

> > +}
> > +
> > +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
> > +	.init_state = mali_c55_rzr_init_state,
> > +};
> > +
> > +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
> > +						  unsigned int index)
> > +{
> > +	const unsigned int scaler_filt_coefmem_addrs[][2] = {
> > +		[MALI_C55_RZR_FR] = {
> > +			0x034A8, /* hfilt */
> > +			0x044A8  /* vfilt */
> 
> Lowercase hex constants.

And addresses belong to the mali-c55-registers.h file.

> > +		},
> > +		[MALI_C55_RZR_DS] = {
> > +			0x014A8, /* hfilt */
> > +			0x024A8  /* vfilt */
> > +		},
> > +	};
> > +	unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
> > +	unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
> > +	unsigned int i, j;
> > +
> > +	for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
> > +		for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
> > +			mali_c55_write(mali_c55, haddr,
> > +				mali_c55_scaler_h_filter_coefficients[i][j]);
> > +			mali_c55_write(mali_c55, vaddr,
> > +				mali_c55_scaler_v_filter_coefficients[i][j]);
> > +
> > +			haddr += sizeof(u32);
> > +			vaddr += sizeof(u32);
> > +		}
> > +	}

How about memcpy_toio() ? I suppose this function isn't
performance sensitive, so maybe usage of mali_c55_write() is better from
a consistency point of view.

> > +}
> > +
> > +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
> > +{
> > +	unsigned int i;
> > +	int ret;
> > +
> > +	for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {

Moving the inner content to a separate mali_c55_register_resizer()
function would increase readability I think, and remove usage of gotos.
I would probably do the same for unregistration too, for consistency.

> > +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> > +		struct v4l2_subdev *sd = &rzr->sd;
> > +		unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
> > +
> > +		rzr->id = i;
> > +		rzr->streaming = false;
> > +
> > +		if (rzr->id == MALI_C55_RZR_FR)
> > +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
> > +		else
> > +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
> > +
> > +		mali_c55_resizer_program_coefficients(mali_c55, i);

Should this be done at stream start, given that power may be cut off
between streaming sessions ?

> > +
> > +		v4l2_subdev_init(sd, &mali_c55_resizer_ops);
> > +		sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
> > +			     | V4L2_SUBDEV_FL_STREAMS;
> > +		sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
> > +		sd->internal_ops = &mali_c55_resizer_internal_ops;
> > +		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",

		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s resizer %s",

and drop the "resizer " prefix from mali_c55_resizer_names. You can also
make mali_c55_resizer_names a local static const variable.

> > +			 MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
> > +
> > +		rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
> > +		rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
> > +
> > +		/* Only the FR pipe has a bypass pad. */
> > +		if (rzr->id == MALI_C55_RZR_FR) {
> > +			rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
> > +							MEDIA_PAD_FL_SINK;
> > +			rzr->num_routes = 2;
> > +		} else {
> > +			num_pads -= 1;
> > +			rzr->num_routes = 1;
> > +		}
> > +
> > +		ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
> > +		if (ret)
> > +			return ret;
> > +
> > +		ret = v4l2_subdev_init_finalize(sd);
> > +		if (ret)
> > +			goto err_cleanup;
> > +
> > +		ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> > +		if (ret)
> > +			goto err_cleanup;
> > +
> > +		rzr->mali_c55 = mali_c55;
> > +	}
> > +
> > +	return 0;
> > +
> > +err_cleanup:
> > +	for (; i >= 0; --i) {
> > +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> > +		struct v4l2_subdev *sd = &rzr->sd;
> > +
> > +		v4l2_subdev_cleanup(sd);
> > +		media_entity_cleanup(&sd->entity);
> > +	}
> > +
> > +	return ret;
> > +}
> > +
> > +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
> > +{
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
> > +		struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
> > +
> > +		if (!resizer->mali_c55)
> > +			continue;
> > +
> > +		v4l2_device_unregister_subdev(&resizer->sd);
> > +		v4l2_subdev_cleanup(&resizer->sd);
> > +		media_entity_cleanup(&resizer->sd.entity);
> > +	}
> > +}
> > diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> > new file mode 100644
> > index 000000000000..c7e699741c6d
> > --- /dev/null
> > +++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> > @@ -0,0 +1,402 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * ARM Mali-C55 ISP Driver - Test pattern generator
> > + *
> > + * Copyright (C) 2024 Ideas on Board Oy
> > + */
> > +
> > +#include <linux/minmax.h>
> > +#include <linux/string.h>
> > +
> > +#include <media/media-entity.h>
> > +#include <media/v4l2-ctrls.h>
> > +#include <media/v4l2-subdev.h>
> > +
> > +#include "mali-c55-common.h"
> > +#include "mali-c55-registers.h"
> > +
> > +#define MALI_C55_TPG_SRC_PAD		0
> > +#define MALI_C55_TPG_FIXED_HBLANK	0x20
> > +#define MALI_C55_TPG_MAX_VBLANK		0xFFFF
> 
> Lowercase hex constants.
> 
> > +#define MALI_C55_TPG_PIXEL_RATE		100000000
> 
> This should be exposed to applications using the V4L2_CID_PIXEL_RATE
> control (read-only).
> 
> > +
> > +static const char * const mali_c55_tpg_test_pattern_menu[] = {
> > +	"Flat field",
> > +	"Horizontal gradient",
> > +	"Vertical gradient",
> > +	"Vertical bars",
> > +	"Arbitrary rectangle",
> > +	"White frame on black field"
> > +};
> > +
> > +static const u32 mali_c55_tpg_mbus_codes[] = {
> > +	MEDIA_BUS_FMT_SRGGB20_1X20,
> > +	MEDIA_BUS_FMT_RGB202020_1X60,
> > +};
> > +
> > +static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
> > +				       int *def_vblank, int *min_vblank)
> 
> unsigned int ?
> 
> > +{
> > +	unsigned int hts;
> > +	int tgt_fps;
> > +	int vblank;
> > +
> > +	hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
> > +
> > +	/*
> > +	 * The ISP has minimum vertical blanking requirements that must be
> > +	 * adhered to by the TPG. The minimum is a function of the Iridix blocks
> > +	 * clocking requirements and the width of the image and horizontal
> > +	 * blanking, but if we assume the worst case iVariance and sVariance
> > +	 * values then it boils down to the below.
> > +	 */
> > +	*min_vblank = 15 + (120500 / hts);
> 
> I wonder if this should round up.
> 
> > +
> > +	/*
> > +	 * We need to set a sensible default vblank for whatever format height
> > +	 * we happen to be given from set_fmt(). This function just targets
> > +	 * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
> > +	 * If we can't get 5fps we'll take whatever the minimum vblank gives us.
> > +	 */
> > +	tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
> > +
> > +	if (tgt_fps < 5)
> > +		vblank = *min_vblank;
> > +	else
> > +		vblank = MALI_C55_TPG_PIXEL_RATE / hts
> > +		       / max(rounddown(tgt_fps, 15), 5);
> > +
> > +	*def_vblank = ALIGN_DOWN(vblank, 2) - format->height;
> 
> "vblank = vblank - height" doesn't seem right. The "else" branch stores
> a vts in vblank, which doesn't seem right either. Maybe you meant
> something like
> 
> 	if (tgt_fps < 5)
> 		def_vts = *min_vblank + format->height;
> 	else
> 		def_vts = MALI_C55_TPG_PIXEL_RATE / hts
> 			/ max(rounddown(tgt_fps, 15), 5);
> 
> 	*def_vblank = ALIGN_DOWN(def_vts, 2) - format->height;
> 
> > +}
> > +
> > +static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
> > +{
> > +	struct mali_c55_tpg *tpg = container_of(ctrl->handler,
> > +						struct mali_c55_tpg,
> > +						ctrls.handler);
> > +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
> > +
> 
> Should you return here if the pipeline isn't streaming ?
> 
> > +	switch (ctrl->id) {
> > +	case V4L2_CID_TEST_PATTERN:
> > +		mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
> > +			       ctrl->val);
> > +		break;
> > +	case V4L2_CID_VBLANK:
> > +		mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
> > +				     MALI_C55_REG_VBLANK_MASK, ctrl->val);
> > +		break;
> > +	default:
> > +		dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
> > +		return -EINVAL;
> 
> Can this happen ?
> 
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
> > +	.s_ctrl = &mali_c55_tpg_s_ctrl,
> > +};
> > +
> > +static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
> > +				   struct v4l2_subdev *sd)
> > +{
> > +	struct v4l2_subdev_state *state;
> > +	struct v4l2_mbus_framefmt *fmt;
> > +
> > +	/*
> > +	 * hblank needs setting, but is a read-only control and thus won't be
> > +	 * called during __v4l2_ctrl_handler_setup(). Do it here instead.
> > +	 */
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
> > +			     MALI_C55_REG_HBLANK_MASK,
> > +			     MALI_C55_TPG_FIXED_HBLANK);
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> > +			     MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
> > +
> > +	state = v4l2_subdev_lock_and_get_active_state(sd);
> > +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> > +
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> > +			     MALI_C55_TEST_PATTERN_RGB_MASK,
> > +			     fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
> > +					  0x01 : 0x0);
> > +
> > +	v4l2_subdev_unlock_state(state);
> > +}
> > +
> > +static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
> > +{
> > +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
> > +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
> > +
> > +	if (!enable) {
> > +		mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> > +				MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
> > +		mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> > +				MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
> > +		return 0;
> > +	}
> > +
> > +	/*
> > +	 * One might reasonably expect the framesize to be set here
> > +	 * given it's configurable in .set_fmt(), but it's done in the
> > +	 * ISP subdevice's stream on func instead, as the same register
> 
> s/func/function/
> 
> > +	 * is also used to indicate the size of the data coming from the
> > +	 * sensor.
> > +	 */
> > +	mali_c55_tpg_configure(mali_c55, sd);
> 
> 	mali_c55_tpg_configure(tpg);
> 
> > +	__v4l2_ctrl_handler_setup(sd->ctrl_handler);
> > +
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> > +			     MALI_C55_TEST_PATTERN_ON_OFF,
> > +			     MALI_C55_TEST_PATTERN_ON_OFF);
> > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> > +			     MALI_C55_REG_GEN_VIDEO_ON_MASK,
> > +			     MALI_C55_REG_GEN_VIDEO_ON_MASK);
> > +
> > +	return 0;
> > +}
> > +
> > +static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
> > +	.s_stream = &mali_c55_tpg_s_stream,
> 
> Can we use .enable_streams() and .disable_streams() ?
> 
> > +};
> > +
> > +static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
> > +				       struct v4l2_subdev_state *state,
> > +				       struct v4l2_subdev_mbus_code_enum *code)
> > +{
> > +	if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> > +		return -EINVAL;
> > +
> > +	code->code = mali_c55_tpg_mbus_codes[code->index];
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
> > +					struct v4l2_subdev_state *state,
> > +					struct v4l2_subdev_frame_size_enum *fse)
> > +{
> 
> You sohuld verify here that fse->code is a supported value and return
> -EINVAL otherwise.
> 
> > +	if (fse->index > 0 || fse->pad > sd->entity.num_pads)
> 
> Drop the pad check, it's done in the subdev core already.
> 
> > +		return -EINVAL;
> > +
> > +	fse->min_width = MALI_C55_MIN_WIDTH;
> > +	fse->max_width = MALI_C55_MAX_WIDTH;
> > +	fse->min_height = MALI_C55_MIN_HEIGHT;
> > +	fse->max_height = MALI_C55_MAX_HEIGHT;
> > +
> > +	return 0;
> > +}
> > +
> > +static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
> > +				struct v4l2_subdev_state *state,
> > +				struct v4l2_subdev_format *format)
> > +{
> > +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
> > +	struct v4l2_mbus_framefmt *fmt = &format->format;
> > +	int vblank_def, vblank_min;
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> > +		if (fmt->code == mali_c55_tpg_mbus_codes[i])
> > +			break;
> > +	}
> > +
> > +	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> > +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> > +
> > +	/*
> > +	 * The TPG says that the test frame timing generation logic expects a
> > +	 * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
> > +	 * handle anything smaller than 128x128 it seems pointless to allow a
> > +	 * smaller frame.
> > +	 */
> > +	clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> > +		MALI_C55_MAX_WIDTH);
> > +	clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> > +		MALI_C55_MAX_HEIGHT);
> > +
> > +	*v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;
> 
> You're allowing userspace to set fmt->field, as well as all the
> colorspace parameters, to random values. I would instead do something
> like
> 
> 	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> 		if (format->format.code == mali_c55_tpg_mbus_codes[i])
> 			break;  
> 	}
> 
> 	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> 		format->format.code = MEDIA_BUS_FMT_SRGGB20_1X20;
> 
> 	format->format.width = clamp(format->format.width,
> 				     MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> 	format->format.height = clamp(format->format.height,
> 				      MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> 
> 	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> 	fmt->code = format->format.code;
> 	fmt->width = format->format.width;
> 	fmt->height = format->format.height;
> 
> 	format->format = *fmt;
> 
> Alternatively (which I think I like better),
> 
> 	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> 
> 	fmt->code = format->format.code;
> 
> 	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> 		if (fmt->code == mali_c55_tpg_mbus_codes[i])
> 			break;  
> 	}
> 
> 	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> 		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> 
> 	fmt->width = clamp(format->format.width,
> 			   MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> 	fmt->height = clamp(format->format.height,
> 			    MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> 
> 	format->format = *fmt;
> 
> > +
> > +	if (format->which == V4L2_SUBDEV_FORMAT_TRY)
> > +		return 0;
> > +
> > +	__mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
> > +	__v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
> > +				 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
> > +	__v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);
> 
> Move those three calls to a separate function, it will be reused below.
> I'd name is mali_c55_tpg_update_vblank(). You can fold
> __mali_c55_tpg_calc_vblank() in it.
> 
> > +
> > +	return 0;
> > +}
> > +
> > +static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
> > +	.enum_mbus_code		= mali_c55_tpg_enum_mbus_code,
> > +	.enum_frame_size	= mali_c55_tpg_enum_frame_size,
> > +	.get_fmt		= v4l2_subdev_get_fmt,
> > +	.set_fmt		= mali_c55_tpg_set_fmt,
> > +};
> > +
> > +static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
> > +	.video	= &mali_c55_tpg_video_ops,
> > +	.pad	= &mali_c55_tpg_pad_ops,
> > +};
> > +
> > +static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
> > +				   struct v4l2_subdev_state *sd_state)
> 
> You name this variable state in every other subdev operation handler.
> 
> > +{
> > +	struct v4l2_mbus_framefmt *fmt =
> > +		v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
> > +
> > +	fmt->width = MALI_C55_DEFAULT_WIDTH;
> > +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
> > +	fmt->field = V4L2_FIELD_NONE;
> > +	fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> 
> Initialize the colorspace fields too.
> 
> > +
> > +	return 0;
> > +}
> > +
> > +static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
> > +	.init_state = mali_c55_tpg_init_state,
> > +};
> > +
> > +static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
> > +{
> > +	struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
> > +	struct v4l2_subdev *sd = &mali_c55->tpg.sd;
> > +	struct v4l2_mbus_framefmt *format;
> > +	struct v4l2_subdev_state *state;
> > +	int vblank_def, vblank_min;
> > +	int ret;
> > +
> > +	state = v4l2_subdev_lock_and_get_active_state(sd);
> > +	format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> > +
> > +	ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);
> 
> You have 3 controls.
> 
> > +	if (ret)
> > +		goto err_unlock;
> > +
> > +	ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
> > +				&mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
> > +				ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
> > +				0, 3, mali_c55_tpg_test_pattern_menu);
> > +
> > +	/*
> > +	 * We fix hblank at the minimum allowed value and control framerate
> > +	 * solely through the vblank control.
> > +	 */
> > +	ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
> > +				&mali_c55_tpg_ctrl_ops,
> > +				V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
> > +				MALI_C55_TPG_FIXED_HBLANK, 1,
> > +				MALI_C55_TPG_FIXED_HBLANK);
> > +	if (ctrls->hblank)
> > +		ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
> > +
> > +	__mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);
> 
> Drop this and initialize the control with default values. You can then
> update the value by calling mali_c55_tpg_update_vblank() in
> mali_c55_register_tpg().
> 
> The reason is to share the same mutex between the control handler and
> the subdev active state without having to add a separate mutex in the
> mali_c55_tpg structure. The simplest way to do so is to initialize the
> controls first, set sd->state_lock to point to the control handler lock,
> and call v4l2_subdev_init_finalize() as the last step. As a consequence,
> you can't access the active state when initializing controls.
> 
> You can alternatively keep the lock in mali_c55_tpg and set
> sd->state_lock to point to it, but I think that's more complex.
> 
> > +	ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
> > +					  &mali_c55_tpg_ctrl_ops,
> > +					  V4L2_CID_VBLANK, vblank_min,
> > +					  MALI_C55_TPG_MAX_VBLANK, 1,
> > +					  vblank_def);
> > +
> > +	if (ctrls->handler.error) {
> > +		dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
> > +		ret = ctrls->handler.error;
> > +		goto err_free_handler;
> > +	}
> > +
> > +	ctrls->handler.lock = &mali_c55->tpg.lock;
> 
> Drop this and drop the mutex. The control handler will use its internal
> mutex.
> 
> > +	mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
> > +
> > +	v4l2_subdev_unlock_state(state);
> > +
> > +	return 0;
> > +
> > +err_free_handler:
> > +	v4l2_ctrl_handler_free(&ctrls->handler);
> > +err_unlock:
> > +	v4l2_subdev_unlock_state(state);
> > +	return ret;
> > +}
> > +
> > +int mali_c55_register_tpg(struct mali_c55 *mali_c55)
> > +{
> > +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
> > +	struct v4l2_subdev *sd = &tpg->sd;
> > +	struct media_pad *pad = &tpg->pad;
> > +	int ret;
> > +
> > +	mutex_init(&tpg->lock);
> > +
> > +	v4l2_subdev_init(sd, &mali_c55_tpg_ops);
> > +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> > +	sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
> 
> Should we introduce a TPG function ?
> 
> > +	sd->internal_ops = &mali_c55_tpg_internal_ops;
> > +	strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
> > +
> > +	pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;
> 
> I don't think MEDIA_PAD_FL_MUST_CONNECT is right.
> 
> > +	ret = media_entity_pads_init(&sd->entity, 1, pad);
> > +	if (ret) {
> > +		dev_err(mali_c55->dev,
> > +			"Failed to initialize media entity pads\n");
> > +		goto err_destroy_mutex;
> > +	}
> > +
> 
> 	sd->state_lock = sd->ctrl_handler->lock;
> 
> to use the same lock for the controls and the active state. You need to
> move this line and the v4l2_subdev_init_finalize() call after
> mali_c55_tpg_init_controls() to get the control handler lock initialized
> first.
> 
> > +	ret = v4l2_subdev_init_finalize(sd);
> > +	if (ret)
> > +		goto err_cleanup_media_entity;
> > +
> > +	ret = mali_c55_tpg_init_controls(mali_c55);
> > +	if (ret) {
> > +		dev_err(mali_c55->dev,
> > +			"Error initialising controls\n");
> > +		goto err_cleanup_subdev;
> > +	}
> > +
> > +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> > +	if (ret) {
> > +		dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
> > +		goto err_free_ctrl_handler;
> > +	}
> > +
> > +	/*
> > +	 * By default the colour settings lead to a very dim image that is
> > +	 * nearly indistinguishable from black on some monitor settings. Ramp
> > +	 * them up a bit so the image is brighter.
> > +	 */
> > +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
> > +		       MALI_C55_TPG_BACKGROUND_MAX);
> > +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
> > +		       MALI_C55_TPG_BACKGROUND_MAX);
> > +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
> > +		       MALI_C55_TPG_BACKGROUND_MAX);
> > +
> > +	tpg->mali_c55 = mali_c55;
> > +
> > +	return 0;
> > +
> > +err_free_ctrl_handler:
> > +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> > +err_cleanup_subdev:
> > +	v4l2_subdev_cleanup(sd);
> > +err_cleanup_media_entity:
> > +	media_entity_cleanup(&sd->entity);
> > +err_destroy_mutex:
> > +	mutex_destroy(&tpg->lock);
> > +
> > +	return ret;
> > +}
> > +
> > +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
> > +{
> > +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
> > +
> > +	if (!tpg->mali_c55)
> > +		return;
> > +
> > +	v4l2_device_unregister_subdev(&tpg->sd);
> > +	v4l2_subdev_cleanup(&tpg->sd);
> > +	media_entity_cleanup(&tpg->sd.entity);
> > +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> 
> Free the control handler just after v4l2_device_unregister_subdev() to
> match the order in mali_c55_register_tpg().
> 
> > +	mutex_destroy(&tpg->lock);
> > +}

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 08/16] media: Add MALI_C55_3A_STATS meta format

[Laurent Pinchart]

Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:50PM +0100, Daniel Scally wrote:
> Add a new meta format for the Mali-C55 ISP's 3A Statistics along
> with a new descriptor entry.
> 
> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
> Changes in v5:
> 
> 	- New patch
> 
>  drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
>  include/uapi/linux/videodev2.h       | 2 ++
>  2 files changed, 3 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 4c76d17b4629..0d00b0476762 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1456,6 +1456,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>  	case V4L2_META_FMT_VIVID:       descr = "Vivid Metadata"; break;
>  	case V4L2_META_FMT_RK_ISP1_PARAMS:	descr = "Rockchip ISP1 3A Parameters"; break;
>  	case V4L2_META_FMT_RK_ISP1_STAT_3A:	descr = "Rockchip ISP1 3A Statistics"; break;
> +	case V4L2_META_FMT_MALI_C55_3A_STATS:	descr = "ARM Mali-C55 ISP 3A Statistics"; break;
>  	case V4L2_PIX_FMT_NV12_8L128:	descr = "NV12 (8x128 Linear)"; break;
>  	case V4L2_PIX_FMT_NV12M_8L128:	descr = "NV12M (8x128 Linear)"; break;
>  	case V4L2_PIX_FMT_NV12_10BE_8L128:	descr = "10-bit NV12 (8x128 Linear, BE)"; break;
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index fe6b67e83751..021424fdaf68 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -841,6 +841,8 @@ struct v4l2_pix_format {
>  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
>  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
>  
> +#define V4L2_META_FMT_MALI_C55_3A_STATS	v4l2_fourcc('C', '5', '5', 'S') /* ARM Mali-C55 3A Statistics */
> +
>  #ifdef __KERNEL__
>  /*
>   * Line-based metadata formats. Remember to update v4l_fill_fmtdesc() when

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 09/16] media: uapi: Add 3a stats buffer for mali-c55

[Laurent Pinchart]

Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:51PM +0100, Daniel Scally wrote:
> Add a header that describes the format of the 3A statistics buffers
> for the mali-c55 ISP.
> 
> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
> Changes in v5:
> 
> 	- New patch
> 
>  .../uapi/linux/media/arm/mali-c55-config.h    | 182 ++++++++++++++++++
>  1 file changed, 182 insertions(+)
>  create mode 100644 include/uapi/linux/media/arm/mali-c55-config.h
> 
> diff --git a/include/uapi/linux/media/arm/mali-c55-config.h b/include/uapi/linux/media/arm/mali-c55-config.h
> new file mode 100644
> index 000000000000..8fb89af6c874
> --- /dev/null
> +++ b/include/uapi/linux/media/arm/mali-c55-config.h
> @@ -0,0 +1,182 @@
> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> +/*
> + * ARM Mali-C55 ISP Driver - Userspace API
> + *
> + * Copyright (C) 2023 Ideas on Board Oy
> + */
> +
> +#ifndef __UAPI_MALI_C55_CONFIG_H
> +#define __UAPI_MALI_C55_CONFIG_H
> +
> +#include <linux/types.h>
> +
> +/*
> + * Frames are split into zones of almost equal width and height - a zone is a
> + * rectangular tile of a frame. The metering blocks within the ISP collect
> + * aggregated statistics per zone. A maximum of 15x15 zones can be configured,
> + * and so the statistics buffer within the hardware is sized to accommodate
> + * that.
> + *
> + * The utilised number of zones is runtime configurable.
> + */
> +#define MALI_C55_MAX_ZONES	225

Maybe

#define MALI_C55_MAX_ZONES	(15 * 15)

to match the comment above ? Or do you think the result of the
multiplication is more important to see at a quick glance ?

> +
> +/**
> + * struct mali_c55_ae_1024bin_hist - Auto Exposure 1024-bin histogram statistics
> + *
> + * @bins:	1024 element array of 16-bit pixel counts.
> + *
> + * The 1024-bin histogram module collects image-global but zone-weighted
> + * intensity distributions of pixels in fixed-width bins. The modules can be
> + * configured into different "plane modes" which affect the contents of the
> + * collected statistics. In plane mode 0, pixel intensities are taken regardless
> + * of colour plane into a single 1024-bin histogram with a bin width of 4. In
> + * plane mode 1, four 256-bin histograms with a bin width of 16 are collected -
> + * one for each CFA colour plane. In plane modes 4, 5, 6 and 7 two 512-bin
> + * histograms with a bin width of 8 are collected - in each mode one of the
> + * colour planes is collected into the first histogram and all the others are
> + * combined into the second. The histograms are stored consecutively in the bins
> + * array.
> + *
> + * The 16-bit pixel counts are stored as a 4-bit exponent in the most
> + * significant bits followed by a 12-bit mantissa. Conversion to a usable
> + * format can be done according to the following pseudo-code::
> + *
> + *	if (e == 0) {
> + *		bin = m * 2;
> + *	} else {
> + *		bin = (m + 4096) x 2^e

x or *, up to you, but pick one :-) Same below.

> + *	}
> + *
> + * where
> + *	e is the exponent value in range 0..15
> + *	m is the mantissa value in range 0..4095
> + *
> + * The pixels used in calculating the statistics can be masked using three
> + * methods:
> + *
> + * 1. Pixels can be skipped in X and Y directions independently.
> + * 2. Minimum/Maximum intensities can be configured
> + * 3. Zones can be differentially weighted, including 0 weighted to mask them
> + *
> + * The data for this histogram can be collected from different tap points in the
> + * ISP depending on configuration - after the white balance or digital gain
> + * blocks, or immediately after the input crossbar.
> + */
> +struct mali_c55_ae_1024bin_hist {
> +	__u16 bins[1024];
> +} __attribute__((packed));
> +
> +/**
> + * struct mali_c55_ae_5bin_hist - Auto Exposure 5-bin histogram statistics
> + *
> + * @hist0:	16-bit normalised pixel count for the 0th intensity bin
> + * @hist1:	16-bit normalised pixel count for the 1st intensity bin
> + * @hist3:	16-bit normalised pixel count for the 3rd intensity bin
> + * @hist4:	16-bit normalised pixel count for the 4th intensity bin
> + *
> + * The ISP generates a 5-bin histogram of normalised pixel counts within bins of
> + * pixel intensity for each of 225 possible zones within a frame. The centre bin
> + * of the histogram for each zone is not available from the hardware and must be
> + * calculated by subtracting the values of hist0, hist1, hist3 and hist4 from
> + * 0xffff as in the following equation:
> + *
> + *	hist2 = 0xffff - (hist0 + hist1 + hist3 + hist4)
> + */
> +struct mali_c55_ae_5bin_hist {
> +	__u16 hist0;
> +	__u16 hist1;
> +	__u16 hist3;
> +	__u16 hist4;
> +} __attribute__((packed));
> +
> +/**
> + * struct mali_c55_awb_average_ratios - Auto White Balance colour ratios
> + *
> + * @avg_rg_gr:	Average R/G or G/R ratio in Q4.8 format.
> + * @avg_bg_br:	Average B/G or B/R ratio in Q4.8 format.
> + * @num_pixels:	The number of pixels used in the AWB calculation
> + *
> + * The ISP calculates and collects average colour ratios for each zone in an
> + * image and stores them in Q4.8 format (the lowest 8 bits are fractional, with
> + * bits [11:8] representing the integer). The exact ratios collected (either
> + * R/G, B/G or G/R, B/R) are configurable through the parameters buffer. The
> + * value of the 4 high bits is undefined.
> + */
> +struct mali_c55_awb_average_ratios {
> +	__u16 avg_rg_gr;
> +	__u16 avg_bg_br;
> +	__u32 num_pixels;
> +} __attribute__((packed));
> +
> +/**
> + * struct mali_c55_af_statistics - Auto Focus edge and intensity statistics
> + *
> + * @intensity_stats:	Packed mantissa and exponent value for pixel intensity
> + * @edge_stats:		Packed mantissa and exponent values for edge intensity
> + *
> + * The ISP collects the squared sum of pixel intensities for each zone within a
> + * configurable Region of Interest on the frame. Additionally, the same data are
> + * collected after being passed through a bandpass filter which removes high and
> + * low frequency components - these are referred to as the edge statistics.
> + *
> + * The intensity and edge statistics for a zone can be used to calculate the
> + * contrast information for a zone
> + *
> + *	C = E2 / I2
> + *
> + * Where I2 is the intensity statistic for a zone and E2 is the edge statistic
> + * for that zone. Optimum focus is reached when C is at its maximum.
> + *
> + * The intensity and edge statistics are stored packed into a non-standard 16
> + * bit floating point format, where the 7 most significant bits represent the
> + * exponent and the 9 least significant bits the mantissa. This format can be
> + * unpacked with the following pseudocode::
> + *
> + *	if (e == 0) {
> + *		x = m;
> + *	} else {
> + *		x = 2^e-1 x (m + 2^9)
> + *	}
> + *
> + * where
> + *	e is the exponent value in range 0..128

If the exponent is stored on 7 bits, isn't this 0..127 ?

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> + *	m is the mantissa value in range 0..511
> + */
> +struct mali_c55_af_statistics {
> +	__u16 intensity_stats;
> +	__u16 edge_stats;
> +} __attribute__((packed));
> +
> +/**
> + * struct mali_c55_stats_buffer - 3A statistics for the mali-c55 ISP
> + *
> + * @ae_1024bin_hist:		1024-bin frame-global pixel intensity histogram
> + * @iridix_1024bin_hist:	Post-Iridix block 1024-bin histogram
> + * @ae_5bin_hists:		5-bin pixel intensity histograms for AEC
> + * @reserved1:			Undefined buffer space
> + * @awb_ratios:			Color balance ratios for Auto White Balance
> + * @reserved2:			Undefined buffer space
> + * @af_statistics:		Pixel intensity statistics for Auto Focus
> + * @reserved3:			Undefined buffer space
> + *
> + * This struct describes the metering statistics space in the Mali-C55 ISP's
> + * hardware in its entirety. The space between each defined area is marked as
> + * "unknown" and may not be 0, but should not be used. The @ae_5bin_hists,
> + * @awb_ratios and @af_statistics members are arrays of statistics per-zone.
> + * The zones are arranged in the array in raster order starting from the top
> + * left corner of the image.
> + */
> +
> +struct mali_c55_stats_buffer {
> +	struct mali_c55_ae_1024bin_hist ae_1024bin_hist;
> +	struct mali_c55_ae_1024bin_hist iridix_1024bin_hist;
> +	struct mali_c55_ae_5bin_hist ae_5bin_hists[MALI_C55_MAX_ZONES];
> +	__u32 reserved1[14];
> +	struct mali_c55_awb_average_ratios awb_ratios[MALI_C55_MAX_ZONES];
> +	__u32 reserved2[14];
> +	struct mali_c55_af_statistics af_statistics[MALI_C55_MAX_ZONES];
> +	__u32 reserved3[15];
> +} __attribute__((packed));
> +
> +#endif /* __UAPI_MALI_C55_CONFIG_H */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 12/16] Documentation: mali-c55: Add Statistics documentation

[Laurent Pinchart]

Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:54PM +0100, Daniel Scally wrote:
> Add documentation explaining the ability to capture statistics from
> the mali-c55 driver's new V4L2 device, as well as the various tap
> points from which those statistics can be drawn in the ISP's
> processing flow. Additionally add a page detailing the new V4L2
> meta format for the mali-c55 statistics.
> 
> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
> Changes in v5:
> 
> 	- New patch
> 
>  Documentation/admin-guide/media/mali-c55.rst  | 60 ++++++++++++++++++-
>  .../userspace-api/media/v4l/meta-formats.rst  |  1 +
>  .../media/v4l/metafmt-arm-mali-c55.rst        | 28 +++++++++
>  3 files changed, 88 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
> 
> diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst
> index cf4176cb1e44..b75437f6e96a 100644
> --- a/Documentation/admin-guide/media/mali-c55.rst
> +++ b/Documentation/admin-guide/media/mali-c55.rst
> @@ -67,10 +67,11 @@ The driver has 4 V4L2 subdevices:
>  - `mali_c55 resizer fr`: The Full-Resolution pipe resizer
>  - `mali_c55 resizer ds`: The Downscale pipe resizer
>  
> -The driver has 2 V4L2 video devices:
> +The driver has 3 V4L2 video devices:
>  
>  - `mali-c55 fr`: The full-resolution pipe's capture device
>  - `mali-c55 ds`: The downscale pipe's capture device
> +- `mali-c55 3a stats`: The 3A statistics capture device
>  
>  Frame sequences are synchronised across to two capture devices, meaning if one
>  pipe is started later than the other the sequence numbers returned in its
> @@ -326,6 +327,63 @@ configured, followed by formats in the appropriate places:
>      # Set format on the video device and stream
>      yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
>  
> +.. _mali-c55-3a-stats:
> +
> +Capturing ISP Statistics
> +========================
> +
> +The ISP is capable of producing statistics for consumption by image processing
> +algorithms running in userspace. These statistics can be captured by queueing
> +buffers to the `mali-c55 3a stats` V4L2 Device whilst the ISP is streaming. Only
> +the :ref:`V4L2_META_FMT_MALI_C55_3A_STATS <v4l2-meta-fmt-mali-c55-3a-stats>`
> +format is supported, so no format-setting need be done:
> +
> +.. code-block:: none
> +
> +    # We assume the media graph has been configured to support RGB565 capture
> +    # from the mali-c55 fr V4L2 Device, which is at /dev/video0. The statistics
> +    # V4L2 device is at /dev/video3
> +
> +    yavta -f RGB565 -s 1920x1080 -c32 /dev/video0 && \
> +    yavta -c10 -F /dev/video3
> +
> +The layout of the buffer is described by :c:type:`mali_c55_stats_buffer`,
> +but broadly statistics are generated to support three image processing
> +algorithms; AEXP (Auto-Exposure), AWB (Auto-White Balance) and AF (Auto-Focus).
> +These stats can be drawn from various places in the Mali C55 ISP pipeline, known
> +as "tap points". This high-level block diagram is intended to explain where in
> +the processing flow the statistics can be drawn from::
> +
> +                  +--> AEXP-2            +----> AEXP-1          +--> AF-0
> +                  |                      +----> AF-1            |
> +                  |                      |                      |
> +      +---------+ |   +--------------+   |   +--------------+   |
> +      |  Input  +-+-->+ Digital Gain +---+-->+ Black Level  +---+---+
> +      +---------+     +--------------+       +--------------+       |
> +  +-----------------------------------------------------------------+
> +  |
> +  |   +--------------+ +---------+       +----------------+
> +  +-->| Sinter Noise +-+  White  +--+--->|  Lens Shading  +--+---------------+
> +      |   Reduction  | | Balance |  |    |                |  |               |
> +      +--------------+ +---------+  |    +----------------+  |               |
> +                                    +---> AEXP-0 (A)         +--> AEXP-0 (B) |
> +  +--------------------------------------------------------------------------+
> +  |
> +  |   +----------------+      +--------------+  +----------------+
> +  +-->|  Tone mapping  +-+--->| Demosaicing  +->+ Purple Fringe  +-+-----------+
> +      |                | |    +--------------+  |   Correction   | |           |
> +      +----------------+ +-> AEXP-IRIDIX        +----------------+ +---> AWB-0 |
> +  +----------------------------------------------------------------------------+
> +  |                    +-------------+        +-------------+
> +  +------------------->|   Colour    +---+--->|    Output   |
> +                       | Correction  |   |    |  Pipelines  |
> +                       +-------------+   |    +-------------+
> +                                         +-->  AWB-1
> +
> +At present all statistics are drawn from the 0th tap point for each algorithm;
> +I.E. AEXP statistics from AEXP-0 (A), AWB statistics from AWB-0 and AF
> +statistics from AF-0. In the future this will be configurable.
> +
>  References
>  ==========
>  .. [1] https://git.linuxtv.org/v4l-utils.git/
> diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> index c23aac823d2c..73c05cc35321 100644
> --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> @@ -15,6 +15,7 @@ These formats are used for the :ref:`metadata` interface only.
>      metafmt-d4xx
>      metafmt-generic
>      metafmt-intel-ipu3
> +    metafmt-arm-mali-c55

Alphabetical order please.

>      metafmt-rkisp1
>      metafmt-uvc
>      metafmt-vivid
> diff --git a/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst b/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
> new file mode 100644
> index 000000000000..219a5dd42d70
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
> @@ -0,0 +1,28 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +.. _v4l2-meta-fmt-mali-c55-3a-stats:
> +
> +****************************************
> +V4L2_META_FMT_MALI_C55_3A_STATS ('C55S')
> +****************************************
> +
> +3A Statistics
> +=============

Missing blank line.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> +The ISP device collects different statistics over an input bayer frame. Those
> +statistics can be obtained by userspace from the
> +:ref:`mali-c55 3a stats <mali-c55-3a-stats>` metadata capture video node, using
> +the :c:type:`v4l2_meta_format` interface. The buffer contains a single instance
> +of the C structure :c:type:`mali_c55_stats_buffer` defined in
> +``mali-c55-config.h``, so the structure can be obtained from the buffer by:
> +
> +.. code-block:: C
> +
> +	struct mali_c55_stats_buffer *stats =
> +		(struct mali_c55_stats_buffer *)buf;
> +
> +For details of the statistics see :c:type:`mali_c55_stats_buffer`.
> +
> +Arm Mali-C55 uAPI data types
> +============================
> +
> +.. kernel-doc:: include/uapi/linux/media/arm/mali-c55-config.h

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 13/16] media: mali-c55: Add image formats for Mali-C55 parameters buffer

[Laurent Pinchart]

Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:55PM +0100, Daniel Scally wrote:
> From: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> 
> Add a new V4L2 meta format code for the Mali-C55 parameters

s/$/./

> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
> Changes in v5:
> 
> 	- New patch
> 
>  drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
>  include/uapi/linux/videodev2.h       | 1 +
>  2 files changed, 2 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 0d00b0476762..4e3b5e16571c 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1456,6 +1456,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>  	case V4L2_META_FMT_VIVID:       descr = "Vivid Metadata"; break;
>  	case V4L2_META_FMT_RK_ISP1_PARAMS:	descr = "Rockchip ISP1 3A Parameters"; break;
>  	case V4L2_META_FMT_RK_ISP1_STAT_3A:	descr = "Rockchip ISP1 3A Statistics"; break;
> +	case V4L2_META_FMT_MALI_C55_PARAMS:	descr = "ARM Mali-C55 ISP Parameters"; break;
>  	case V4L2_META_FMT_MALI_C55_3A_STATS:	descr = "ARM Mali-C55 ISP 3A Statistics"; break;
>  	case V4L2_PIX_FMT_NV12_8L128:	descr = "NV12 (8x128 Linear)"; break;
>  	case V4L2_PIX_FMT_NV12M_8L128:	descr = "NV12M (8x128 Linear)"; break;
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 021424fdaf68..e63d5e76d21e 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -841,6 +841,7 @@ struct v4l2_pix_format {
>  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
>  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
>  
> +#define V4L2_META_FMT_MALI_C55_PARAMS	v4l2_fourcc('C', '5', '5', 'P') /* ARM Mali-C55 Parameters */
>  #define V4L2_META_FMT_MALI_C55_3A_STATS	v4l2_fourcc('C', '5', '5', 'S') /* ARM Mali-C55 3A Statistics */
>  
>  #ifdef __KERNEL__

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 16/16] Documentation: mali-c55: Document the mali-c55 parameter setting

[Laurent Pinchart]

Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:58PM +0100, Daniel Scally wrote:
> Document the mali-c55 parameter setting by expanding the relevant
> pages in both admin-guide/ and userspace-api/.
> 
> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
> Changes in v5:
> 
> 	- New patch
> 
>  Documentation/admin-guide/media/mali-c55.rst  | 19 +++++-
>  .../media/v4l/metafmt-arm-mali-c55.rst        | 65 ++++++++++++++++++-
>  2 files changed, 79 insertions(+), 5 deletions(-)
> 
> diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst
> index b75437f6e96a..c5e6ac826c99 100644
> --- a/Documentation/admin-guide/media/mali-c55.rst
> +++ b/Documentation/admin-guide/media/mali-c55.rst
> @@ -380,9 +380,24 @@ the processing flow the statistics can be drawn from::
>                         +-------------+   |    +-------------+
>                                           +-->  AWB-1
>  
> -At present all statistics are drawn from the 0th tap point for each algorithm;
> +By default all statistics are drawn from the 0th tap point for each algorithm;
>  I.E. AEXP statistics from AEXP-0 (A), AWB statistics from AWB-0 and AF
> -statistics from AF-0. In the future this will be configurable.
> +statistics from AF-0. This is configurable for AEXP and AWB statsistics through
> +programming the ISP's parameters.
> +
> +.. _mali-c55-3a-params:
> +
> +Programming ISP Parameters
> +==========================
> +
> +The ISP can be programmed with various parameters from userspace to apply to the
> +hardware before and during video stream. This allows userspace to dynamically
> +change values such as black level, white balance and lens shading gains and so
> +on.
> +
> +The buffer format and how to populate it are described by the
> +:ref:`V4L2_META_FMT_MALI_C55_PARAMS <v4l2-meta-fmt-mali-c55-params>` format,
> +which should be set as the data format for the `mali-c55 3a params` video node.
>  
>  References
>  ==========
> diff --git a/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst b/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
> index 219a5dd42d70..c359d2c83858 100644
> --- a/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
> +++ b/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
> @@ -1,10 +1,11 @@
>  .. SPDX-License-Identifier: GPL-2.0
>  
> +.. _v4l2-meta-fmt-mali-c55-params:
>  .. _v4l2-meta-fmt-mali-c55-3a-stats:
>  
> -****************************************
> -V4L2_META_FMT_MALI_C55_3A_STATS ('C55S')
> -****************************************
> +********************************************************************************
> +V4L2_META_FMT_MALI_C55_3A_STATS ('C55S'), V4L2_META_FMT_MALI_C55_PARAMS ('C55P')
> +********************************************************************************
>  
>  3A Statistics
>  =============
> @@ -22,6 +23,64 @@ of the C structure :c:type:`mali_c55_stats_buffer` defined in
>  
>  For details of the statistics see :c:type:`mali_c55_stats_buffer`.
>  
> +Configuration Parameters
> +========================
> +
> +The configuration parameters are passed to the
> +:ref:`mali-c55 3a params <mali-c55-3a-params>` metadata output video node, using
> +the :c:type:`v4l2_meta_format` interface. Rather than a single struct containing
> +sub-structs for each configurable area of the ISP, parameters for the Mali-C55
> +are defined as distinct structs or "blocks" which may be added to the data
> +member of struct mali_c55_params_buffer. Userspace is responsible for populating
> +the data member with the blocks that need to be configured by the driver, but
> +need not populate it with **all** the blocks, or indeed with any at all if there
> +are no configuration changes to make. Populated blocks **must** be consecutive
> +in the buffer. To assist both userspace and the driver in identifying the
> +blocks each block-specific struct should embed

s/should embed/embeds/

> +struct mali_c55_params_block_header as its first member and userspace must
> +populate the type member with a value from enum mali_c55_param_block_type. Once

It would be nice to add references to the struct and enum.

> +the blocks have been populated into the data buffer, the combined size of all
> +populated blocks should be set in the total_size member of

s/should be/is/

or "must" or "shall".

> +struct mali_c55_params_buffer. For example:
> +
> +.. code-block:: c
> +
> +	struct mali_c55_params_buffer *params =
> +		(struct mali_c55_params_buffer *)buffer;
> +
> +	params->version = MALI_C55_PARAM_BUFFER_V0;

I think applications will likely handle the size as follows:

	params->total_size = 0;

> +
> +	void *data = (void *)params->data;
> +
> +	struct mali_c55_params_awb_gains *gains =
> +		(struct mali_c55_params_awb_gains *)data;
> +
> +	gains->header.type = MALI_C55_PARAM_BLOCK_AWB_GAINS;
> +	gains->header.enabled = true;
> +	gains->header.size = sizeof(struct mali_c55_params_awb_gains);
> +
> +	gains->gain00 = 256;
> +	gains->gain00 = 256;
> +	gains->gain00 = 256;
> +	gains->gain00 = 256;
> +
> +	data += sizeof(struct mali_c55_params_awb_gains)

	params->total_size += sizeof(struct mali_c55_params_awb_gains);

> +
> +	struct mali_c55_params_sensor_off_preshading *blc =
> +		(struct mali_c55_params_sensor_off_preshading *)data;
> +
> +	blc->header.type = MALI_C55_PARAM_BLOCK_SENSOR_OFFS;
> +	blc->header.enabled = true;
> +	blc->header.size = sizeof(struct mali_c55_params_sensor_off_preshading);
> +
> +	blc->chan00 = 51200;
> +	blc->chan01 = 51200;
> +	blc->chan10 = 51200;
> +	blc->chan11 = 51200;
> +
> +	params->total_size = sizeof(struct mali_c55_params_awb_gains) +
> +			     sizeof(struct mali_c55_params_sensor_off_preshading);

	params->total_size += sizeof(struct mali_c55_params_sensor_off_preshading);

I could be wrong though. Up to you.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> +
>  Arm Mali-C55 uAPI data types
>  ============================
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 14/16] media: uapi: Add parameters structs to mali-c55-config.h

[Laurent Pinchart]

Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:56PM +0100, Daniel Scally wrote:
> Add structures describing the ISP parameters to mali-c55-config.h
> 
> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
> Changes in v5:
> 
> 	- New patch
> 
>  .../uapi/linux/media/arm/mali-c55-config.h    | 669 ++++++++++++++++++
>  1 file changed, 669 insertions(+)
> 
> diff --git a/include/uapi/linux/media/arm/mali-c55-config.h b/include/uapi/linux/media/arm/mali-c55-config.h
> index 8fb89af6c874..fce14bc74f4a 100644
> --- a/include/uapi/linux/media/arm/mali-c55-config.h
> +++ b/include/uapi/linux/media/arm/mali-c55-config.h
> @@ -179,4 +179,673 @@ struct mali_c55_stats_buffer {
>  	__u32 reserved3[15];
>  } __attribute__((packed));
>  
> +/**
> + * enum mali_c55_param_buffer_version - Mali-C55 parameters block versioning
> + *
> + * @MALI_C55_PARAM_BUFFER_V0: First version of Mali-C55 parameters block

So you like versions to be 0-based ? :-)

> + */
> +enum mali_c55_param_buffer_version {
> +	MALI_C55_PARAM_BUFFER_V0,
> +};
> +
> +/**
> + * enum mali_c55_param_block_type - Enumeration of Mali-C55 parameter blocks
> + *
> + * This enumeration defines the types of Mali-C55 parameters block. Each block
> + * configures a specific processing block of the Mali-C55 ISP. The block
> + * type allows the driver to correctly interpret the parameters block data.
> + *
> + * It is the responsibility of userspace to correctly set the type of each
> + * parameters block.
> + *
> + * @MALI_C55_PARAM_BLOCK_SENSOR_OFFS: Sensor pre-shading black level offset
> + * @MALI_C55_PARAM_BLOCK_AEXP_HIST: Auto-exposure 1024-bin histogram
> + *				    configuration
> + * @MALI_C55_PARAM_BLOCK_AEXP_IHIST: Post-Iridix auto-exposure 1024-bin
> + *				     histogram configuration
> + * @MALI_C55_PARAM_BLOCK_AEXP_HIST_WEIGHTS: Auto-exposure 1024-bin histogram
> + *					    weighting
> + * @MALI_C55_PARAM_BLOCK_AEXP_IHIST_WEIGHTS: Post-Iridix auto-exposure 1024-bin
> + *					     histogram weighting

Do you expect applications to need to set the histogram configuration
and weights separately ?

> + * @MALI_C55_PARAM_BLOCK_DIGITAL_GAIN: Digital gain
> + * @MALI_C55_PARAM_BLOCK_AWB_GAINS: Auto-white balance gains

I was thinking that those two could be combined, but if the former is
used by AEC and the latter by AWB, maybe keeping them separate would
make like easier for userspace.

> + * @MALI_C55_PARAM_BLOCK_AWB_CONFIG: Auto-white balance statistics config
> + * @MALI_C55_PARAM_BLOCK_AWB_GAINS_AEXP: Auto-white balance gains for AEXP-0 tap
> + * @MALI_C55_PARAM_MESH_SHADING_CONFIG : Mesh shading tables configuration
> + * @MALI_C55_PARAM_MESH_SHADING_SELECTION: Mesh shading table selection
> + * @MALI_C55_PARAM_BLOCK_SENTINEL: First non-valid block index

You should indicate somewhere the correspondance between the block type
and the block data structure. It could be done in the definition of each
data structure for instance.

> + */
> +enum mali_c55_param_block_type {
> +	MALI_C55_PARAM_BLOCK_SENSOR_OFFS,
> +	MALI_C55_PARAM_BLOCK_AEXP_HIST,
> +	MALI_C55_PARAM_BLOCK_AEXP_IHIST,
> +	MALI_C55_PARAM_BLOCK_AEXP_HIST_WEIGHTS,
> +	MALI_C55_PARAM_BLOCK_AEXP_IHIST_WEIGHTS,
> +	MALI_C55_PARAM_BLOCK_DIGITAL_GAIN,
> +	MALI_C55_PARAM_BLOCK_AWB_GAINS,
> +	MALI_C55_PARAM_BLOCK_AWB_CONFIG,
> +	MALI_C55_PARAM_BLOCK_AWB_GAINS_AEXP,
> +	MALI_C55_PARAM_MESH_SHADING_CONFIG,
> +	MALI_C55_PARAM_MESH_SHADING_SELECTION,
> +	MALI_C55_PARAM_BLOCK_SENTINEL,
> +};
> +
> +/**
> + * struct mali_c55_params_block_header - Mali-C55 parameter block header
> + *
> + * This structure represents the common part of all the ISP configuration
> + * blocks. Each parameters block shall embed an instance of this structure type

s/shall embed/embeds/

> + * as its first member, followed by the block-specific configuration data. The
> + * driver inspects this common header to discern the block type and its size and
> + * properly handle the block content by casting it to the correct block-specific
> + * type.
> + *
> + * The @type field is one of the values enumerated by
> + * :c:type:`mali_c55_param_block_type` and specifies how the data should be
> + * interpreted by the driver. The @size field specifies the size of the
> + * parameters block and is used by the driver for validation purposes. The
> + * @enabled field specifies if the ISP block should be enabled (and configured
> + * according to the provided parameters) or disabled.
> + *
> + * .. code-block:: c
> + *
> + *	struct mali_c55_params_block_header *block = ...;
> + *
> + *	switch (block->type) {
> + *	case MALI_C55_PARAM_BLOCK_SENSOR_OFFS:
> + *		struct mali_c55_params_sensor_off_preshading *sensor_offs =
> + *			(struct mali_c55_params_sensor_off_preshading *)block;
> + *
> + *		if (block->size !=
> + *		    sizeof(struct mali_c55_params_sensor_off_preshading))
> + *			return -EINVAL;
> + *
> + *		handle_sensor_offs(sensor_offs);
> + *		break;
> + *
> + *	case MALI_C55_PARAM_BLOCK_AEXP_HIST:
> + *		struct mali_c55_params_aexp_hist *aexp_hist =
> + *			(struct mali_c55_params_aexp_hist)block;
> + *
> + *		if (block->size != sizeof(mali_c55_params_aexp_hist))
> + *			return -EINVAL;
> + *
> + *		handle_aexp_hist(aesp_hist);
> + *		break;
> + *
> + *	...
> + *
> + *	}

I would probably skip this. The kernel-side of the implementation can be
found in the driver. I would document the UAPI here with examples
focussing on the userspace side only.

> + *
> + * Userspace is responsible for correctly populating the parameters block header
> + * fields (@type, @enabled and @size) and correctly populate the block-specific
> + * parameters.
> + *
> + * For example:
> + *
> + * .. code-block:: c
> + *
> + *	void populate_sensor_offs(struct mali_c55_params_block_header *block) {
> + *		block->type = MALI_C55_PARAM_BLOCK_SENSOR_OFFS;
> + *		block->enabled = true;
> + *		block->size = sizeof(struct mali_c55_params_sensor_off_preshading);
> + *
> + *		struct mali_c55_params_sensor_off_preshading *sensor_offs =
> + *			(struct mali_c55_params_sensor_off_preshading *)block;
> + *
> + *		sensor_offs->chan00 = offset00;
> + *		sensor_offs->chan01 = offset01;
> + *		sensor_offs->chan10 = offset10;
> + *		sensor_offs->chan11 = offset11;
> + *	}
> + *
> + * @type: The parameters block type (enum mali_c55_param_block_type)
> + * @enabled: Block enabled/disabled flag

Does this flag apply to all blocks ? If not, it would be good to
indicate which blocks it applies to.

Also, I think we discussed previously that, when a block is disabled,
its configuration parameters are not applicable anymore, and could then
be skipped. Is that something useful to do ?

> + * @size: Size (in bytes) of the parameters block
> + */
> +struct mali_c55_params_block_header {
> +	enum mali_c55_param_block_type type;
> +	bool enabled;
> +	size_t size;

enums, bool and size_t shouldn't be used in a UAPI, as they size may
vary between 32- and 64-bit architectures, making 32-bit userspace on a
64-bit kernel require compat handling in the driver. Use types with
well-defined sizes. Same below.

> +};
> +
> +/**
> + * struct mali_c55_params_sensor_off_preshading - offset subtraction for each
> + *						  color channel
> + *
> + * Provides removal of the sensor black level from the sensor data. Separate
> + * offsets are provided for each of the four Bayer component color channels

That's a weird spelling of offsets.

> + * which are defaulted to R, Gr, Gb, B.

I think it would be useful to indicate, for each structure, what block
it corresponds to in the ascii diagram in patch 12/16. On a side note,
at some point turning that diagram into SVG may make sense.

> + *
> + * @header: The Mali-C55 parameters block header
> + * @chan00: Offset for color channel 00 (default: R)
> + * @chan01: Offset for color channel 01 (default: Gr)
> + * @chan10: Offset for color channel 10 (default: Gb)
> + * @chan11: Offset for color channel 11 (default: B)
> + */
> +struct mali_c55_params_sensor_off_preshading {
> +	struct mali_c55_params_block_header header;
> +	__u32 chan00;
> +	__u32 chan01;
> +	__u32 chan10;
> +	__u32 chan11;
> +};
> +
> +/**
> + * enum mali_c55_aexp_hist_tap_points - Tap points for the AEXP histogram
> + * @MALI_C55_AEXP_HIST_TAP_WB: After static white balance
> + * @MALI_C55_AEXP_HIST_TAP_FS: After WDR Frame Stitch
> + * @MALI_C55_AEXP_HIST_TAP_TPG: After the test pattern generator
> + */
> +enum mali_c55_aexp_hist_tap_points {
> +	MALI_C55_AEXP_HIST_TAP_WB = 0,
> +	MALI_C55_AEXP_HIST_TAP_FS,
> +	MALI_C55_AEXP_HIST_TAP_TPG,
> +};
> +
> +/**
> + * enum mali_c55_aexp_skip_x - Horizontal pixel skipping
> + * @MALI_C55_AEXP_SKIP_X_EVERY_2ND: Collect every 2nd pixel horizontally
> + * @MALI_C55_AEXP_SKIP_X_EVERY_3RD: Collect every 3rd pixel horizontally
> + * @MALI_C55_AEXP_SKIP_X_EVERY_4TH: Collect every 4th pixel horizontally
> + * @MALI_C55_AEXP_SKIP_X_EVERY_5TH: Collect every 5th pixel horizontally
> + * @MALI_C55_AEXP_SKIP_X_EVERY_8TH: Collect every 8th pixel horizontally
> + * @MALI_C55_AEXP_SKIP_X_EVERY_9TH: Collect every 9th pixel horizontally
> + */
> +enum mali_c55_aexp_skip_x {
> +	MALI_C55_AEXP_SKIP_X_EVERY_2ND,
> +	MALI_C55_AEXP_SKIP_X_EVERY_3RD,
> +	MALI_C55_AEXP_SKIP_X_EVERY_4TH,
> +	MALI_C55_AEXP_SKIP_X_EVERY_5TH,
> +	MALI_C55_AEXP_SKIP_X_EVERY_8TH,
> +	MALI_C55_AEXP_SKIP_X_EVERY_9TH
> +};

Does this mean that that the histogram can't operate on every pixels,
but will always skip at least every other pixel ?

> +
> +/**
> + * enum mali_c55_aexp_skip_y - Vertical pixel skipping
> + * @MALI_C55_AEXP_SKIP_Y_ALL: Collect every single pixel vertically
> + * @MALI_C55_AEXP_SKIP_Y_EVERY_2ND: Collect every 2nd pixel vertically
> + * @MALI_C55_AEXP_SKIP_Y_EVERY_3RD: Collect every 3rd pixel vertically
> + * @MALI_C55_AEXP_SKIP_Y_EVERY_4TH: Collect every 4th pixel vertically
> + * @MALI_C55_AEXP_SKIP_Y_EVERY_5TH: Collect every 5th pixel vertically
> + * @MALI_C55_AEXP_SKIP_Y_EVERY_8TH: Collect every 8th pixel vertically
> + * @MALI_C55_AEXP_SKIP_Y_EVERY_9TH: Collect every 9th pixel vertically
> + */
> +enum mali_c55_aexp_skip_y {
> +	MALI_C55_AEXP_SKIP_Y_ALL,
> +	MALI_C55_AEXP_SKIP_Y_EVERY_2ND,
> +	MALI_C55_AEXP_SKIP_Y_EVERY_3RD,
> +	MALI_C55_AEXP_SKIP_Y_EVERY_4TH,
> +	MALI_C55_AEXP_SKIP_Y_EVERY_5TH,
> +	MALI_C55_AEXP_SKIP_Y_EVERY_8TH,
> +	MALI_C55_AEXP_SKIP_Y_EVERY_9TH
> +};
> +
> +/**
> + * enum mali_c55_aexp_row_column_offset - Start from the first or second row or
> + *					  column
> + * @MALI_C55_AEXP_FIRST_ROW_OR_COL:	Start from the first row / column
> + * @MALI_C55_AEXP_SECOND_ROW_OR_COL:	Start from the second row / column
> + */
> +enum mali_c55_aexp_row_column_offset {
> +	MALI_C55_AEXP_FIRST_ROW_OR_COL = 1,
> +	MALI_C55_AEXP_SECOND_ROW_OR_COL = 2,
> +};
> +
> +/**
> + * enum mali_c55_aexp_hist_plane_mode - Mode for the AEXP Histograms

As histograms are computed on bayer data, I'd talk about "component"
instead of "plane" here, unless the word plane matches documentation.

> + * @MALI_C55_AEXP_HIST_COMBINED: All color planes in one 1024-bin histogram
> + * @MALI_C55_AEXP_HIST_SEPARATE: Each color plane in one 256-bin histogram with a bin width of 16
> + * @MALI_C55_AEXP_HIST_FOCUS_00: Top left plane in the first bank, rest in second bank
> + * @MALI_C55_AEXP_HIST_FOCUS_01: Top right plane in the first bank, rest in second bank
> + * @MALI_C55_AEXP_HIST_FOCUS_10: Bottom left plane in the first bank, rest in second bank
> + * @MALI_C55_AEXP_HIST_FOCUS_11: Bottom right plane in the first bank, rest in second bank
> + *
> + * In the "focus" modes statistics are collected into two 512-bin histograms
> + * with a bin width of 8. One colour plane is in the first histogram with the
> + * remainder combined into the second. The four options represent which of the
> + * four positions in a bayer pattern are the focused plane.

How does that work with x/y skipping ? Is skipping then applied to 2x2
blocks, or still to pixels ?

> + */
> +enum mali_c55_aexp_hist_plane_mode {
> +	MALI_C55_AEXP_HIST_COMBINED = 0,
> +	MALI_C55_AEXP_HIST_SEPARATE = 1,
> +	MALI_C55_AEXP_HIST_FOCUS_00 = 4,
> +	MALI_C55_AEXP_HIST_FOCUS_01 = 5,
> +	MALI_C55_AEXP_HIST_FOCUS_10 = 6,
> +	MALI_C55_AEXP_HIST_FOCUS_11 = 7,
> +};
> +
> +/**
> + * struct mali_c55_params_aexp_hist - configuration for AEXP metering hists
> + *
> + * This struct allows users to configure the 1024-bin AEXP histograms. Broadly
> + * speaking the parameters allow you to mask particular regions of the image and
> + * to select different kinds of histogram.
> + *
> + * @header:		The Mali-C55 parameters block header
> + * @skip_x:		Horizontal decimation. See enum mali_c55_aexp_skip_x
> + * @offset_x:		Column to start from. See enum mali_c55_aexp_row_column_offset
> + * @skip_y:		Vertical decimation. See enum mali_c55_aexp_skip_y
> + * @offset_y:		Row to start from. See enum mali_c55_aexp_row_column_offset

Do the offsets need to be multiples of 2 ?

> + * @scale_bottom:	scale of bottom half of range: 0=1x ,1=2x, 2=4x, 4=8x, 4=16x

Could you elaborate on this ?

> + * @scale_top:		scale of top half of range: 0=1x ,1=2x, 2=4x, 4=8x, 4=16x
> + * @plane_mode:		Plane separation mode. See enum mali_c55_aexp_hist_plane_mode
> + * @tap_point:		Tap point for histogram from enum mali_c55_aexp_hist_tap_points.
> + *			This parameter is unused for the post-Iridix Histogram
> + */
> +struct mali_c55_params_aexp_hist {
> +	struct mali_c55_params_block_header header;
> +	__u8 skip_x;
> +	__u8 offset_x;
> +	__u8 skip_y;
> +	__u8 offset_y;
> +	__u8 scale_bottom;
> +	__u8 scale_top;
> +	__u8 plane_mode;
> +	__u8 tap_point;
> +};
> +
> +/**
> + * struct mali_c55_params_aexp_weights - Array of weights for AEXP metering
> + *
> + * This struct allows users to configure the weighting for both of the 1024-bin
> + * AEXP histograms. The pixel data collected for each zone is multiplied by the
> + * corresponding weight from this array, which may be zero if the intention is
> + * to mask off the zone entirely.
> + *
> + * @header:		The Mali-C55 parameters block header
> + * @nodes_used_horiz:	Number of active zones horizontally [0..15]
> + * @nodes_used_vert:	Number of active zones vertically [0..15]

Is the image automatically split in the number of zones with a uniform
distribution ?

What happens if the number of zones is 0 ?

> + * @zone_weights:	Zone weighting. Index is row*col where 0,0 is the top
> + * 			left zone continuing in raster order. Each zone can be
> + *			weighted in the range [0..15]. The number of rows and
> + *			columns is defined by @nodes_used_vert and
> + *			@nodes_used_horiz
> + */
> +struct mali_c55_params_aexp_weights {
> +	struct mali_c55_params_block_header header;
> +	__u8 nodes_used_horiz;
> +	__u8 nodes_used_vert;
> +	__u8 zone_weights[MALI_C55_MAX_ZONES];
> +};
> +
> +/**
> + * struct mali_c55_params_digital_gain - Digital gain value
> + *
> + * This struct carries a digital gain value to set in the ISP
> + *
> + * @header:	The Mali-C55 parameters block header
> + * @gain:	The digital gain value to apply, in Q5.8 format.
> + */
> +struct mali_c55_params_digital_gain {
> +	struct mali_c55_params_block_header header;
> +	__u16 gain;
> +};
> +
> +/**
> + * enum mali_c55_awb_stats_mode - Statistics mode for AWB
> + * @MALI_C55_AWB_MODE_GRBR: Statistics collected as Green/Red and Blue/Red ratios
> + * @MALI_C55_AWB_MODE_RGBG: Statistics collected as Red/Green and Blue/Green ratios
> + */
> +enum mali_c55_awb_stats_mode {
> +       MALI_C55_AWB_MODE_GRBR = 0,
> +       MALI_C55_AWB_MODE_RGBG,
> +};
> +
> +/**
> + * struct mali_c55_params_awb_gains - Gain settings for auto white balance
> + *
> + * This struct allows users to configure the gains for auto-white balance. There
> + * are four gain settings corresponding to each colour channel in the bayer
> + * domain. Although named generically, the association between the gain applied
> + * and the colour channel is done automatically within the ISP depending on the
> + * input format, and so the following mapping always holds true::
> + *
> + *	gain00 = R
> + *	gain01 = Gr
> + *	gain10 = Gb
> + *	gain11 = B

How about naming them accordingly then ? :-)

> + *
> + * All of the gains are stored in Q4.8 format.
> + *
> + * @header:	The Mali-C55 parameters block header
> + * @gain00:	Multiplier for colour channel 00
> + * @gain01:	Multiplier for colour channel 01
> + * @gain10:	Multiplier for colour channel 10
> + * @gain11:	Multiplier for colour channel 11
> + */
> +struct mali_c55_params_awb_gains {
> +	struct mali_c55_params_block_header header;
> +	__u16 gain00;
> +	__u16 gain01;
> +	__u16 gain10;
> +	__u16 gain11;
> +};
> +
> +/**
> + * enum mali_c55_params_awb_tap_points - Tap points for the AWB statistics
> + * @MALI_C55_AWB_STATS_TAP_PF: Immediately after the Purple Fringe block
> + * @MALI_C55_AWB_STATS_TAP_CNR: Immediately after the CNR block
> + */
> +enum mali_c55_params_awb_tap_points {
> +	MALI_C55_AWB_STATS_TAP_PF = 0,
> +	MALI_C55_AWB_STATS_TAP_CNR,
> +};
> +
> +/**
> + * struct mali_c55_params_awb_config - Stats settings for auto-white balance
> + *
> + * This struct allows the configuration of the statistics generated for auto
> + * white balance. Pixel intensity limits can be set to exclude overly bright or
> + * dark regions of an image from the statistics entirely. Colour ratio minima
> + * and maxima can be set to discount pixels who's ratios fall outside the
> + * defined boundaries; there are two sets of registers to do this - the
> + * "min/max" ratios which bound a region and the "high/low" ratios which further
> + * trim the upper and lower ratios. For example with the boundaries configured
> + * as follows, only pixels whos colour ratios falls into the region marked "A"
> + * would be counted::
> + *
> + *	                                                          cr_high
> + *	    2.0 |                                                   |
> + *	        |               cb_max --> _________________________v_____
> + *	    1.8 |                         |                         \    |
> + *	        |                         |                          \   |
> + *	    1.6 |                         |                           \  |
> + *	        |                         |                            \ |
> + *	 c  1.4 |               cb_low -->|\              A             \|<--  cb_high
> + *	 b      |                         | \                            |
> + *	    1.2 |                         |  \                           |
> + *	 r      |                         |   \                          |
> + *	 a  1.0 |              cb_min --> |____\_________________________|
> + *	 t      |                         ^    ^                         ^
> + *	 i  0.8 |                         |    |                         |
> + *	 o      |                      cr_min  |                       cr_max
> + *	 s  0.6 |                              |
> + *	        |                             cr_low
> + *	    0.4 |
> + *	        |
> + *	    0.2 |
> + *	        |
> + *	    0.0 |_______________________________________________________________
> + *	        0.0   0.2   0.4   0.6   0.8   1.0   1.2   1.4   1.6   1.8   2.0
> + *	                                   cr ratios
> + *
> + * @header:		The Mali-C55 parameters block header
> + * @tap_point:		The tap point from enum mali_c55_params_awb_tap_points
> + * @stats_mode:		AWB statistics collection mode, see :c:type:`mali_c55_awb_stats_mode`
> + * @white_level:	Upper pixel intensity (I.E. raw pixel values) limit
> + * @black_level:	Lower pixel intensity (I.E. raw pixel values) limit
> + * @cr_max:		Maximum R/G ratio (Q4.8 format)
> + * @cr_min:		Minimum R/G ratio (Q4.8 format)
> + * @cb_max:		Maximum B/G ratio (Q4.8 format)
> + * @cb_min:		Minimum B/G ratio (Q4.8 format)
> + * @nodes_used_horiz:	Number of active zones horizontally [0..15]
> + * @nodes_used_vert:	Number of active zones vertically [0..15]
> + * @cr_high:		R/G ratio trim high (Q4.8 format)
> + * @cr_low:		R/G ratio trim low (Q4.8 format)
> + * @cb_high:		B/G ratio trim high (Q4.8 format)
> + * @cb_low:		B/G ratio trim low (Q4.8 format)
> + */
> +struct mali_c55_params_awb_config {
> +	struct mali_c55_params_block_header header;
> +	__u8 tap_point;
> +	__u8 stats_mode;
> +	__u16 white_level;
> +	__u16 black_level;
> +	__u16 cr_max;
> +	__u16 cr_min;
> +	__u16 cb_max;
> +	__u16 cb_min;
> +	__u8 nodes_used_horiz;
> +	__u8 nodes_used_vert;
> +	__u16 cr_high;
> +	__u16 cr_low;
> +	__u16 cb_high;
> +	__u16 cb_low;
> +};
> +
> +#define MALI_C55_NUM_MESH_SHADING_ELEMENTS 3072
> +
> +/**
> + * struct mali_c55_params_mesh_shading_config - Mesh shading configuration
> + *
> + * The mesh shading correction module allows programming a separate table of
> + * either 16x16 or 32x32 node coefficients for 3 different light sources. The
> + * final correction coefficients applied are computed by blending the
> + * coefficients from two tables together.
> + *
> + * A page of 1024 32-bit integers is associated to each colour channel, with
> + * pages stored consecutively in memory. Each 32-bit integer packs 3 8-bit
> + * correction coefficients for a single node, one for each of the three light
> + * sources. The 8 most significant bits are unused. The following table
> + * describes the layout::
> + *
> + *	+----------- Page (Colour Plane) 0 -------------+
> + *	| @mesh[i]  | Mesh Point | Bits  | Light Source |
> + *	+-----------+------------+-------+--------------+
> + *	|         0 |        0,0 | 16,23 | LS2          |
> + *	|           |            | 08-15 | LS1          |
> + *	|           |            | 00-07 | LS0          |
> + *	+-----------+------------+-------+--------------+
> + *	|         1 |        0,1 | 16,23 | LS2          |
> + *	|           |            | 08-15 | LS1          |
> + *	|           |            | 00-07 | LS0          |
> + *	+-----------+------------+-------+--------------+
> + *	|       ... |        ... | ...   | ...          |
> + *	+-----------+------------+-------+--------------+
> + *	|      1023 |      31,31 | 16,23 | LS2          |
> + *	|           |            | 08-15 | LS1          |
> + *	|           |            | 00-07 | LS0          |
> + *	+----------- Page (Colour Plane) 1 -------------+
> + *	| @mesh[i]  | Mesh Point | Bits  | Light Source |
> + *	+-----------+------------+-------+--------------+
> + *	|      1024 |        0,0 | 16,23 | LS2          |
> + *	|           |            | 08-15 | LS1          |
> + *	|           |            | 00-07 | LS0          |
> + *	+-----------+------------+-------+--------------+
> + *	|      1025 |        0,1 | 16,23 | LS2          |
> + *	|           |            | 08-15 | LS1          |
> + *	|           |            | 00-07 | LS0          |
> + *	+-----------+------------+-------+--------------+
> + *	|       ... |        ... | ...   | ...          |
> + *	+-----------+------------+-------+--------------+
> + *	|      2047 |      31,31 | 16,23 | LS2          |
> + *	|           |            | 08-15 | LS1          |
> + *	|           |            | 00-07 | LS0          |
> + *	+----------- Page (Colour Plane) 2 -------------+
> + *	| @mesh[i]  | Mesh Point | Bits  | Light Source |
> + *	+-----------+------------+-------+--------------+
> + *	|      2048 |        0,0 | 16,23 | LS2          |
> + *	|           |            | 08-15 | LS1          |
> + *	|           |            | 00-07 | LS0          |
> + *	+-----------+------------+-------+--------------+
> + *	|      2049 |        0,1 | 16,23 | LS2          |
> + *	|           |            | 08-15 | LS1          |
> + *	|           |            | 00-07 | LS0          |
> + *	+-----------+------------+-------+--------------+
> + *	|       ... |        ... | ...   | ...          |
> + *	+-----------+------------+-------+--------------+
> + *	|      3071 |      31,31 | 16,23 | LS2          |
> + *	|           |            | 08-15 | LS1          |
> + *	|           |            | 00-07 | LS0          |
> + *	+-----------+------------+-------+--------------+
> + *
> + * The @mesh_scale member determines the precision and minimum and maximum gain.
> + * For example if @mesh_scale is 0 and therefore selects 0 - 2x gain, a value of
> + * 0 in a coefficient means 0.0 gain, a value of 128 means 1.0 gain and 255
> + * means 2.0 gain.
> + *
> + * @header:		The Mali-C55 parameters block header
> + * @mesh_show:		Output the mesh data rather than image data
> + * @mesh_scale:		Set the precision and maximum gain range of mesh shading
> + *				- 0 = 0-2x gain
> + *				- 1 = 0-4x gain
> + *				- 2 = 0-8x gain
> + *				- 3 = 0-16x gain
> + *				- 4 = 1-2x gain
> + *				- 5 = 1-3x gain
> + *				- 6 = 1-5x gain
> + *				- 7 = 1-9x gain
> + * @mesh_page_r:	Mesh page select for red colour plane [0..2]
> + * @mesh_page_g:	Mesh page select for green colour plane [0..2]
> + * @mesh_page_b:	Mesh page select for blue colour plane [0..2]
> + * @mesh_width:		Number of horizontal nodes minus 1 [15,31]
> + * @mesh_height:	Number of vertical nodes minus 1 [15,31]
> + * @mesh:		Mesh shading correction tables
> + */
> +struct mali_c55_params_mesh_shading_config {
> +	struct mali_c55_params_block_header header;
> +	bool mesh_show;
> +	__u8 mesh_scale;
> +	__u8 mesh_page_r;
> +	__u8 mesh_page_g;
> +	__u8 mesh_page_b;
> +	__u8 mesh_width;
> +	__u8 mesh_height;
> +	__u32 mesh[MALI_C55_NUM_MESH_SHADING_ELEMENTS];
> +};
> +
> +/** enum mali_c55_params_mesh_alpha_bank - Mesh shading table bank selection
> + * @MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS1 - Select Light Sources 0 and 1
> + * @MALI_C55_MESH_ALPHA_BANK_LS1_AND_LS2 - Select Light Sources 1 and 2
> + * @MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS2 - Select Light Sources 0 and 2
> + */
> +enum mali_c55_params_mesh_alpha_bank {
> +	MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS1 = 0,
> +	MALI_C55_MESH_ALPHA_BANK_LS1_AND_LS2 = 1,
> +	MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS2 = 4
> +};
> +
> +/**
> + * struct mali_c55_params_mesh_shading_selection - Mesh table selection
> + *
> + * The module computes the final correction coefficients by blending the ones
> + * from two light source tables, which are selected (independently for each
> + * colour channel) by the @mesh_alpha_bank_r/g/b fields.
> + *
> + * The final blended coefficients for each node are calculated using the
> + * following equation:
> + *
> + *     Final coefficient = (a x LS\ :sub:`b`\ + (256 - a) x LS\ :sub:`a`\) / 256
> + *
> + * Where a is the @mesh_alpha_r/g/b value, and LS\ :sub:`a`\ and LS\ :sub:`b`\
> + * are the node cofficients for the two tables selected by the
> + * @mesh_alpha_bank_r/g/b value.
> + *
> + * The scale of the applied correction may also be controlled by tuning the
> + * @mesh_strength member. This is a modifier to the final coefficients which can
> + * be used to globally reduce the gains applied.
> + *
> + * @header:		The Mali-C55 parameters block header
> + * @mesh_alpha_bank_r:	Red mesh table select (c:type:`enum mali_c55_params_mesh_alpha_bank`)
> + * @mesh_alpha_bank_g:	Green mesh table select (c:type:`enum mali_c55_params_mesh_alpha_bank`)
> + * @mesh_alpha_bank_b:	Blue mesh table select (c:type:`enum mali_c55_params_mesh_alpha_bank`)
> + * @mesh_alpha_r:	Blend coefficient for R [0..255]
> + * @mesh_alpha_g:	Blend coefficient for G [0..255]
> + * @mesh_alpha_b:	Blend coefficient for B [0..255]
> + * @mesh_strength:	Mesh strength in Q4.12 format [0..4096]
> + */
> +struct mali_c55_params_mesh_shading_selection {
> +	struct mali_c55_params_block_header header;
> +	__u8 mesh_alpha_bank_r;
> +	__u8 mesh_alpha_bank_g;
> +	__u8 mesh_alpha_bank_b;
> +	__u8 mesh_alpha_r;
> +	__u8 mesh_alpha_g;
> +	__u8 mesh_alpha_b;
> +	__u16 mesh_strength;
> +};
> +
> +/**
> + * define MALI_C55_PARAMS_MAX_SIZE - Maximum size of all Mali C55 Parameters
> + *
> + * Though the parameters for the Mali-C55 are passed as optional blocks, the
> + * driver still needs to know the absolute maximum size so that it can allocate
> + * a buffer sized appropriately to accomodate userspace attempting to set all
> + * possible parameters in a single frame.
> + */
> +#define MALI_C55_PARAMS_MAX_SIZE				\
> +	sizeof(struct mali_c55_params_sensor_off_preshading) + 	\
> +	sizeof(struct mali_c55_params_aexp_hist) +		\
> +	sizeof(struct mali_c55_params_aexp_weights) +		\
> +	sizeof(struct mali_c55_params_aexp_hist) +		\
> +	sizeof(struct mali_c55_params_aexp_weights) +		\
> +	sizeof(struct mali_c55_params_digital_gain) +		\
> +	sizeof(struct mali_c55_params_awb_gains) +		\
> +	sizeof(struct mali_c55_params_awb_config) +		\
> +	sizeof(struct mali_c55_params_awb_gains) +		\

Some data structures are duplicated, please explain why somewhere, or it
may appear to be a bug.

> +	sizeof(struct mali_c55_params_mesh_shading_config) +	\
> +	sizeof(struct mali_c55_params_mesh_shading_selection)

I think this will work fine now, but as soon as you add a block that
contains a 64-bit field, you'll have issues. Many CPUs expect 64-bit
accesses to be aligned. If some structures have a size that are not
multiples of 64-bit, you'll end up having unaligned blocks.

I would recommend aligning the size of each data structure to 64 bits. A
simple way to do so would be to align the size of the header to 64 bits.
Please test the result to make sure I'm right though :-)

> +
> +/**
> + * struct mali_c55_params_buffer - 3A configuration parameters
> + *
> + * This struct contains the configuration parameters of the Mali-C55 ISP
> + * algorithms, serialized by userspace into an opaque data buffer. Each

It's not opaque, you've documented it fully :-)

s/opaque //

> + * configuration parameter block is represented by a block-specific structure
> + * which contains a :c:type:`mali_c55_params_block_header` entry as first
> + * member. Userspace populates the @data buffer with configuration parameters
> + * for the blocks that it intends to configure. As a consequence, the data
> + * buffer effective size changes according to the number of ISP blocks that
> + * userspace intends to configure.
> + *
> + * The parameters buffer is versioned by the @version field to allow modifying
> + * and extending its definition. Userspace should populate the @version field to

s/should/shall/

> + * inform the driver about the version it intends to use. The driver will parse
> + * and handle the @data buffer according to the data layout specific to the
> + * indicated revision and return an error if the desired revision is not

s/revision/version/g

> + * supported.
> + *
> + * For each ISP block that userspace wants to configure, a block-specific
> + * structure is appended to the @data buffer, one after the other without gaps
> + * in between nor overlaps. Userspace shall populate the @total_size field with
> + * the effective size, in bytes, of the @data buffer.
> + *
> + * The expected memory layout of the parameters buffer is::
> + *
> + *	+-------------------- struct mali_c55_params_buffer ------------------+
> + *	| version = MALI_C55_PARAM_BUFFER_V0;                                 |
> + *	| total_size = sizeof(struct mali_c55_params_sensor_off_preshading)   |
> + *	|              sizeof(struct mali_c55_params_aexp_hist);              |
> + *	| +------------------------- data  ---------------------------------+ |
> + *	| | +--------- struct mali_c55_params_sensor_off_preshading ------+ | |
> + *	| | | +-------- struct mali_c55_params_block_header header -----+ | | |
> + *	| | | | type = MALI_C55_PARAM_BLOCK_SENSOR_OFFS;                | | | |
> + *	| | | | enabled = 1;                                            | | | |
> + *	| | | | size =                                                  | | | |
> + *	| | | |    sizeof(struct mali_c55_params_sensor_off_preshading);| | | |
> + *	| | | +---------------------------------------------------------+ | | |
> + *	| | | chan00 = ...;                                               | | |
> + *	| | | chan01 = ...;                                               | | |
> + *	| | | chan10 = ...;                                               | | |
> + *	| | | chan11 = ...;                                               | | |
> + *	| | +------------ struct mali_c55_params_aexp_hist ---------------+ | |
> + *	| | | +-------- struct mali_c55_params_block_header header -----+ | | |
> + *	| | | | type = MALI_C55_PARAM_BLOCK_AEXP_HIST;                  | | | |
> + *	| | | | enabled = 1;                                            | | | |
> + *	| | | | size = sizeof(struct mali_c55_params_aexp_hist);        | | | |
> + *	| | | +---------------------------------------------------------+ | | |
> + *	| | | skip_x = ...;                                               | | |
> + *	| | | offset_x = ...;                                             | | |
> + *	| | | skip_y = ...;                                               | | |
> + *	| | | offset_y = ...;                                             | | |
> + *	| | | scale_bottom = ...;                                         | | |
> + *	| | | scale_top = ...;                                            | | |
> + *	| | | plane_mode = ...;                                           | | |
> + *	| | | tap_point = ...;                                            | | |
> + *	| | +-------------------------------------------------------------+ | |
> + *	| +-----------------------------------------------------------------+ |
> + *	+---------------------------------------------------------------------+
> + *
> + * @version: The Mali-C55 parameters buffer version
> + * @total_size: The Mali-C55 configuration data effective size, excluding this
> + *		header
> + * @data: The Mali-C55 configuration blocks data
> + */
> +struct mali_c55_params_buffer {
> +	enum mali_c55_param_buffer_version version;
> +	size_t total_size;
> +	__u8 data[MALI_C55_PARAMS_MAX_SIZE];
> +};
> +
>  #endif /* __UAPI_MALI_C55_CONFIG_H */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 14/16] media: uapi: Add parameters structs to mali-c55-config.h

[Dan Scally]

Morning Laurent - thanks for the review

On 31/05/2024 01:09, Laurent Pinchart wrote:
> Hi Dan,
>
> Thank you for the patch.
>
> On Wed, May 29, 2024 at 04:28:56PM +0100, Daniel Scally wrote:
>> Add structures describing the ISP parameters to mali-c55-config.h
>>
>> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>> ---
>> Changes in v5:
>>
>> 	- New patch
>>
>>   .../uapi/linux/media/arm/mali-c55-config.h    | 669 ++++++++++++++++++
>>   1 file changed, 669 insertions(+)
>>
>> diff --git a/include/uapi/linux/media/arm/mali-c55-config.h b/include/uapi/linux/media/arm/mali-c55-config.h
>> index 8fb89af6c874..fce14bc74f4a 100644
>> --- a/include/uapi/linux/media/arm/mali-c55-config.h
>> +++ b/include/uapi/linux/media/arm/mali-c55-config.h
>> @@ -179,4 +179,673 @@ struct mali_c55_stats_buffer {
>>   	__u32 reserved3[15];
>>   } __attribute__((packed));
>>   
>> +/**
>> + * enum mali_c55_param_buffer_version - Mali-C55 parameters block versioning
>> + *
>> + * @MALI_C55_PARAM_BUFFER_V0: First version of Mali-C55 parameters block
> So you like versions to be 0-based ? :-)
Hah - sure. I'll switch it to V1.
>
>> + */
>> +enum mali_c55_param_buffer_version {
>> +	MALI_C55_PARAM_BUFFER_V0,
>> +};
>> +
>> +/**
>> + * enum mali_c55_param_block_type - Enumeration of Mali-C55 parameter blocks
>> + *
>> + * This enumeration defines the types of Mali-C55 parameters block. Each block
>> + * configures a specific processing block of the Mali-C55 ISP. The block
>> + * type allows the driver to correctly interpret the parameters block data.
>> + *
>> + * It is the responsibility of userspace to correctly set the type of each
>> + * parameters block.
>> + *
>> + * @MALI_C55_PARAM_BLOCK_SENSOR_OFFS: Sensor pre-shading black level offset
>> + * @MALI_C55_PARAM_BLOCK_AEXP_HIST: Auto-exposure 1024-bin histogram
>> + *				    configuration
>> + * @MALI_C55_PARAM_BLOCK_AEXP_IHIST: Post-Iridix auto-exposure 1024-bin
>> + *				     histogram configuration
>> + * @MALI_C55_PARAM_BLOCK_AEXP_HIST_WEIGHTS: Auto-exposure 1024-bin histogram
>> + *					    weighting
>> + * @MALI_C55_PARAM_BLOCK_AEXP_IHIST_WEIGHTS: Post-Iridix auto-exposure 1024-bin
>> + *					     histogram weighting
> Do you expect applications to need to set the histogram configuration
> and weights separately ?


**Need** to no, but it's convenient. I don't particularly expect the histogram configuration to be 
changed past the first frame, but changing the metering mode for example would require a new weights 
array and separating them avoids re-writing the config unnecessarily.

>
>> + * @MALI_C55_PARAM_BLOCK_DIGITAL_GAIN: Digital gain
>> + * @MALI_C55_PARAM_BLOCK_AWB_GAINS: Auto-white balance gains
> I was thinking that those two could be combined, but if the former is
> used by AEC and the latter by AWB, maybe keeping them separate would
> make like easier for userspace.


They're used by the separate algorithms yes.

>
>> + * @MALI_C55_PARAM_BLOCK_AWB_CONFIG: Auto-white balance statistics config
>> + * @MALI_C55_PARAM_BLOCK_AWB_GAINS_AEXP: Auto-white balance gains for AEXP-0 tap
>> + * @MALI_C55_PARAM_MESH_SHADING_CONFIG : Mesh shading tables configuration
>> + * @MALI_C55_PARAM_MESH_SHADING_SELECTION: Mesh shading table selection
>> + * @MALI_C55_PARAM_BLOCK_SENTINEL: First non-valid block index
> You should indicate somewhere the correspondance between the block type
> and the block data structure. It could be done in the definition of each
> data structure for instance.


Hmm...just in the documentary comment? Or something more strict?

>
>> + */
>> +enum mali_c55_param_block_type {
>> +	MALI_C55_PARAM_BLOCK_SENSOR_OFFS,
>> +	MALI_C55_PARAM_BLOCK_AEXP_HIST,
>> +	MALI_C55_PARAM_BLOCK_AEXP_IHIST,
>> +	MALI_C55_PARAM_BLOCK_AEXP_HIST_WEIGHTS,
>> +	MALI_C55_PARAM_BLOCK_AEXP_IHIST_WEIGHTS,
>> +	MALI_C55_PARAM_BLOCK_DIGITAL_GAIN,
>> +	MALI_C55_PARAM_BLOCK_AWB_GAINS,
>> +	MALI_C55_PARAM_BLOCK_AWB_CONFIG,
>> +	MALI_C55_PARAM_BLOCK_AWB_GAINS_AEXP,
>> +	MALI_C55_PARAM_MESH_SHADING_CONFIG,
>> +	MALI_C55_PARAM_MESH_SHADING_SELECTION,
>> +	MALI_C55_PARAM_BLOCK_SENTINEL,
>> +};
>> +
>> +/**
>> + * struct mali_c55_params_block_header - Mali-C55 parameter block header
>> + *
>> + * This structure represents the common part of all the ISP configuration
>> + * blocks. Each parameters block shall embed an instance of this structure type
> s/shall embed/embeds/
>
>> + * as its first member, followed by the block-specific configuration data. The
>> + * driver inspects this common header to discern the block type and its size and
>> + * properly handle the block content by casting it to the correct block-specific
>> + * type.
>> + *
>> + * The @type field is one of the values enumerated by
>> + * :c:type:`mali_c55_param_block_type` and specifies how the data should be
>> + * interpreted by the driver. The @size field specifies the size of the
>> + * parameters block and is used by the driver for validation purposes. The
>> + * @enabled field specifies if the ISP block should be enabled (and configured
>> + * according to the provided parameters) or disabled.
>> + *
>> + * .. code-block:: c
>> + *
>> + *	struct mali_c55_params_block_header *block = ...;
>> + *
>> + *	switch (block->type) {
>> + *	case MALI_C55_PARAM_BLOCK_SENSOR_OFFS:
>> + *		struct mali_c55_params_sensor_off_preshading *sensor_offs =
>> + *			(struct mali_c55_params_sensor_off_preshading *)block;
>> + *
>> + *		if (block->size !=
>> + *		    sizeof(struct mali_c55_params_sensor_off_preshading))
>> + *			return -EINVAL;
>> + *
>> + *		handle_sensor_offs(sensor_offs);
>> + *		break;
>> + *
>> + *	case MALI_C55_PARAM_BLOCK_AEXP_HIST:
>> + *		struct mali_c55_params_aexp_hist *aexp_hist =
>> + *			(struct mali_c55_params_aexp_hist)block;
>> + *
>> + *		if (block->size != sizeof(mali_c55_params_aexp_hist))
>> + *			return -EINVAL;
>> + *
>> + *		handle_aexp_hist(aesp_hist);
>> + *		break;
>> + *
>> + *	...
>> + *
>> + *	}
> I would probably skip this. The kernel-side of the implementation can be
> found in the driver. I would document the UAPI here with examples
> focussing on the userspace side only.


OK.

>
>> + *
>> + * Userspace is responsible for correctly populating the parameters block header
>> + * fields (@type, @enabled and @size) and correctly populate the block-specific
>> + * parameters.
>> + *
>> + * For example:
>> + *
>> + * .. code-block:: c
>> + *
>> + *	void populate_sensor_offs(struct mali_c55_params_block_header *block) {
>> + *		block->type = MALI_C55_PARAM_BLOCK_SENSOR_OFFS;
>> + *		block->enabled = true;
>> + *		block->size = sizeof(struct mali_c55_params_sensor_off_preshading);
>> + *
>> + *		struct mali_c55_params_sensor_off_preshading *sensor_offs =
>> + *			(struct mali_c55_params_sensor_off_preshading *)block;
>> + *
>> + *		sensor_offs->chan00 = offset00;
>> + *		sensor_offs->chan01 = offset01;
>> + *		sensor_offs->chan10 = offset10;
>> + *		sensor_offs->chan11 = offset11;
>> + *	}
>> + *
>> + * @type: The parameters block type (enum mali_c55_param_block_type)
>> + * @enabled: Block enabled/disabled flag
> Does this flag apply to all blocks ? If not, it would be good to
> indicate which blocks it applies to.


It's part of the header, so it's available to all blocks...they can all be disabled somehow though 
the exact mechanism differs.

>
> Also, I think we discussed previously that, when a block is disabled,
> its configuration parameters are not applicable anymore, and could then
> be skipped. Is that something useful to do ?


We don't pass blocks where the values are unchanged (so only a couple go in each frame - white 
balance and digital gain IIRC), but if we want to transition a block from enabled to disabled for 
whatever reason the driver would need to see it

>
>> + * @size: Size (in bytes) of the parameters block
>> + */
>> +struct mali_c55_params_block_header {
>> +	enum mali_c55_param_block_type type;
>> +	bool enabled;
>> +	size_t size;
> enums, bool and size_t shouldn't be used in a UAPI, as they size may
> vary between 32- and 64-bit architectures, making 32-bit userspace on a
> 64-bit kernel require compat handling in the driver. Use types with
> well-defined sizes. Same below.


Ack

>> +};
>> +
>> +/**
>> + * struct mali_c55_params_sensor_off_preshading - offset subtraction for each
>> + *						  color channel
>> + *
>> + * Provides removal of the sensor black level from the sensor data. Separate
>> + * offsets are provided for each of the four Bayer component color channels
> That's a weird spelling of offsets.


Err...you mean "off"? It's from the documentation - I can spell it properly though.

>
>> + * which are defaulted to R, Gr, Gb, B.
> I think it would be useful to indicate, for each structure, what block
> it corresponds to in the ascii diagram in patch 12/16.


Sure, ok.

> On a side note,
> at some point turning that diagram into SVG may make sense.


That diagram is incomplete...it's only supposed to highlight the tap points and the blocks that 
surround them. So we'll need to extend it to accommodate that (which probably will be easier as an SVG)

>
>> + *
>> + * @header: The Mali-C55 parameters block header
>> + * @chan00: Offset for color channel 00 (default: R)
>> + * @chan01: Offset for color channel 01 (default: Gr)
>> + * @chan10: Offset for color channel 10 (default: Gb)
>> + * @chan11: Offset for color channel 11 (default: B)
>> + */
>> +struct mali_c55_params_sensor_off_preshading {
>> +	struct mali_c55_params_block_header header;
>> +	__u32 chan00;
>> +	__u32 chan01;
>> +	__u32 chan10;
>> +	__u32 chan11;
>> +};
>> +
>> +/**
>> + * enum mali_c55_aexp_hist_tap_points - Tap points for the AEXP histogram
>> + * @MALI_C55_AEXP_HIST_TAP_WB: After static white balance
>> + * @MALI_C55_AEXP_HIST_TAP_FS: After WDR Frame Stitch
>> + * @MALI_C55_AEXP_HIST_TAP_TPG: After the test pattern generator
>> + */
>> +enum mali_c55_aexp_hist_tap_points {
>> +	MALI_C55_AEXP_HIST_TAP_WB = 0,
>> +	MALI_C55_AEXP_HIST_TAP_FS,
>> +	MALI_C55_AEXP_HIST_TAP_TPG,
>> +};
>> +
>> +/**
>> + * enum mali_c55_aexp_skip_x - Horizontal pixel skipping
>> + * @MALI_C55_AEXP_SKIP_X_EVERY_2ND: Collect every 2nd pixel horizontally
>> + * @MALI_C55_AEXP_SKIP_X_EVERY_3RD: Collect every 3rd pixel horizontally
>> + * @MALI_C55_AEXP_SKIP_X_EVERY_4TH: Collect every 4th pixel horizontally
>> + * @MALI_C55_AEXP_SKIP_X_EVERY_5TH: Collect every 5th pixel horizontally
>> + * @MALI_C55_AEXP_SKIP_X_EVERY_8TH: Collect every 8th pixel horizontally
>> + * @MALI_C55_AEXP_SKIP_X_EVERY_9TH: Collect every 9th pixel horizontally
>> + */
>> +enum mali_c55_aexp_skip_x {
>> +	MALI_C55_AEXP_SKIP_X_EVERY_2ND,
>> +	MALI_C55_AEXP_SKIP_X_EVERY_3RD,
>> +	MALI_C55_AEXP_SKIP_X_EVERY_4TH,
>> +	MALI_C55_AEXP_SKIP_X_EVERY_5TH,
>> +	MALI_C55_AEXP_SKIP_X_EVERY_8TH,
>> +	MALI_C55_AEXP_SKIP_X_EVERY_9TH
>> +};
> Does this mean that that the histogram can't operate on every pixels,
> but will always skip at least every other pixel ?


Horizontally, yes. So the maximum pixel count would be (width / 2) x height.

>
>> +
>> +/**
>> + * enum mali_c55_aexp_skip_y - Vertical pixel skipping
>> + * @MALI_C55_AEXP_SKIP_Y_ALL: Collect every single pixel vertically
>> + * @MALI_C55_AEXP_SKIP_Y_EVERY_2ND: Collect every 2nd pixel vertically
>> + * @MALI_C55_AEXP_SKIP_Y_EVERY_3RD: Collect every 3rd pixel vertically
>> + * @MALI_C55_AEXP_SKIP_Y_EVERY_4TH: Collect every 4th pixel vertically
>> + * @MALI_C55_AEXP_SKIP_Y_EVERY_5TH: Collect every 5th pixel vertically
>> + * @MALI_C55_AEXP_SKIP_Y_EVERY_8TH: Collect every 8th pixel vertically
>> + * @MALI_C55_AEXP_SKIP_Y_EVERY_9TH: Collect every 9th pixel vertically
>> + */
>> +enum mali_c55_aexp_skip_y {
>> +	MALI_C55_AEXP_SKIP_Y_ALL,
>> +	MALI_C55_AEXP_SKIP_Y_EVERY_2ND,
>> +	MALI_C55_AEXP_SKIP_Y_EVERY_3RD,
>> +	MALI_C55_AEXP_SKIP_Y_EVERY_4TH,
>> +	MALI_C55_AEXP_SKIP_Y_EVERY_5TH,
>> +	MALI_C55_AEXP_SKIP_Y_EVERY_8TH,
>> +	MALI_C55_AEXP_SKIP_Y_EVERY_9TH
>> +};
>> +
>> +/**
>> + * enum mali_c55_aexp_row_column_offset - Start from the first or second row or
>> + *					  column
>> + * @MALI_C55_AEXP_FIRST_ROW_OR_COL:	Start from the first row / column
>> + * @MALI_C55_AEXP_SECOND_ROW_OR_COL:	Start from the second row / column
>> + */
>> +enum mali_c55_aexp_row_column_offset {
>> +	MALI_C55_AEXP_FIRST_ROW_OR_COL = 1,
>> +	MALI_C55_AEXP_SECOND_ROW_OR_COL = 2,
>> +};
>> +
>> +/**
>> + * enum mali_c55_aexp_hist_plane_mode - Mode for the AEXP Histograms
> As histograms are computed on bayer data, I'd talk about "component"
> instead of "plane" here, unless the word plane matches documentation.


Plane is from the documentation yes.

>
>> + * @MALI_C55_AEXP_HIST_COMBINED: All color planes in one 1024-bin histogram
>> + * @MALI_C55_AEXP_HIST_SEPARATE: Each color plane in one 256-bin histogram with a bin width of 16
>> + * @MALI_C55_AEXP_HIST_FOCUS_00: Top left plane in the first bank, rest in second bank
>> + * @MALI_C55_AEXP_HIST_FOCUS_01: Top right plane in the first bank, rest in second bank
>> + * @MALI_C55_AEXP_HIST_FOCUS_10: Bottom left plane in the first bank, rest in second bank
>> + * @MALI_C55_AEXP_HIST_FOCUS_11: Bottom right plane in the first bank, rest in second bank
>> + *
>> + * In the "focus" modes statistics are collected into two 512-bin histograms
>> + * with a bin width of 8. One colour plane is in the first histogram with the
>> + * remainder combined into the second. The four options represent which of the
>> + * four positions in a bayer pattern are the focused plane.
> How does that work with x/y skipping ? Is skipping then applied to 2x2
> blocks, or still to pixels ?


To pixels, so particular configurations of the skipping and the offsets can cause some of the colour 
component histograms to be empty.

>> + */
>> +enum mali_c55_aexp_hist_plane_mode {
>> +	MALI_C55_AEXP_HIST_COMBINED = 0,
>> +	MALI_C55_AEXP_HIST_SEPARATE = 1,
>> +	MALI_C55_AEXP_HIST_FOCUS_00 = 4,
>> +	MALI_C55_AEXP_HIST_FOCUS_01 = 5,
>> +	MALI_C55_AEXP_HIST_FOCUS_10 = 6,
>> +	MALI_C55_AEXP_HIST_FOCUS_11 = 7,
>> +};
>> +
>> +/**
>> + * struct mali_c55_params_aexp_hist - configuration for AEXP metering hists
>> + *
>> + * This struct allows users to configure the 1024-bin AEXP histograms. Broadly
>> + * speaking the parameters allow you to mask particular regions of the image and
>> + * to select different kinds of histogram.
>> + *
>> + * @header:		The Mali-C55 parameters block header
>> + * @skip_x:		Horizontal decimation. See enum mali_c55_aexp_skip_x
>> + * @offset_x:		Column to start from. See enum mali_c55_aexp_row_column_offset
>> + * @skip_y:		Vertical decimation. See enum mali_c55_aexp_skip_y
>> + * @offset_y:		Row to start from. See enum mali_c55_aexp_row_column_offset
> Do the offsets need to be multiples of 2 ?
It's effectively a boolean field to decide whether to skip the first column/row or not.
>
>> + * @scale_bottom:	scale of bottom half of range: 0=1x ,1=2x, 2=4x, 4=8x, 4=16x
> Could you elaborate on this ?


Will do

>
>> + * @scale_top:		scale of top half of range: 0=1x ,1=2x, 2=4x, 4=8x, 4=16x
>> + * @plane_mode:		Plane separation mode. See enum mali_c55_aexp_hist_plane_mode
>> + * @tap_point:		Tap point for histogram from enum mali_c55_aexp_hist_tap_points.
>> + *			This parameter is unused for the post-Iridix Histogram
>> + */
>> +struct mali_c55_params_aexp_hist {
>> +	struct mali_c55_params_block_header header;
>> +	__u8 skip_x;
>> +	__u8 offset_x;
>> +	__u8 skip_y;
>> +	__u8 offset_y;
>> +	__u8 scale_bottom;
>> +	__u8 scale_top;
>> +	__u8 plane_mode;
>> +	__u8 tap_point;
>> +};
>> +
>> +/**
>> + * struct mali_c55_params_aexp_weights - Array of weights for AEXP metering
>> + *
>> + * This struct allows users to configure the weighting for both of the 1024-bin
>> + * AEXP histograms. The pixel data collected for each zone is multiplied by the
>> + * corresponding weight from this array, which may be zero if the intention is
>> + * to mask off the zone entirely.
>> + *
>> + * @header:		The Mali-C55 parameters block header
>> + * @nodes_used_horiz:	Number of active zones horizontally [0..15]
>> + * @nodes_used_vert:	Number of active zones vertically [0..15]
> Is the image automatically split in the number of zones with a uniform
> distribution ?


That's my understanding yes

>
> What happens if the number of zones is 0 ?


I don't know; haven't tried. I'll test it out.

>
>> + * @zone_weights:	Zone weighting. Index is row*col where 0,0 is the top
>> + * 			left zone continuing in raster order. Each zone can be
>> + *			weighted in the range [0..15]. The number of rows and
>> + *			columns is defined by @nodes_used_vert and
>> + *			@nodes_used_horiz
>> + */
>> +struct mali_c55_params_aexp_weights {
>> +	struct mali_c55_params_block_header header;
>> +	__u8 nodes_used_horiz;
>> +	__u8 nodes_used_vert;
>> +	__u8 zone_weights[MALI_C55_MAX_ZONES];
>> +};
>> +
>> +/**
>> + * struct mali_c55_params_digital_gain - Digital gain value
>> + *
>> + * This struct carries a digital gain value to set in the ISP
>> + *
>> + * @header:	The Mali-C55 parameters block header
>> + * @gain:	The digital gain value to apply, in Q5.8 format.
>> + */
>> +struct mali_c55_params_digital_gain {
>> +	struct mali_c55_params_block_header header;
>> +	__u16 gain;
>> +};
>> +
>> +/**
>> + * enum mali_c55_awb_stats_mode - Statistics mode for AWB
>> + * @MALI_C55_AWB_MODE_GRBR: Statistics collected as Green/Red and Blue/Red ratios
>> + * @MALI_C55_AWB_MODE_RGBG: Statistics collected as Red/Green and Blue/Green ratios
>> + */
>> +enum mali_c55_awb_stats_mode {
>> +       MALI_C55_AWB_MODE_GRBR = 0,
>> +       MALI_C55_AWB_MODE_RGBG,
>> +};
>> +
>> +/**
>> + * struct mali_c55_params_awb_gains - Gain settings for auto white balance
>> + *
>> + * This struct allows users to configure the gains for auto-white balance. There
>> + * are four gain settings corresponding to each colour channel in the bayer
>> + * domain. Although named generically, the association between the gain applied
>> + * and the colour channel is done automatically within the ISP depending on the
>> + * input format, and so the following mapping always holds true::
>> + *
>> + *	gain00 = R
>> + *	gain01 = Gr
>> + *	gain10 = Gb
>> + *	gain11 = B
> How about naming them accordingly then ? :-)


The gainNN comes from the register field names; I've used them for everything...does doing that 
consistently have value? Or should I rename them?

>
>> + *
>> + * All of the gains are stored in Q4.8 format.
>> + *
>> + * @header:	The Mali-C55 parameters block header
>> + * @gain00:	Multiplier for colour channel 00
>> + * @gain01:	Multiplier for colour channel 01
>> + * @gain10:	Multiplier for colour channel 10
>> + * @gain11:	Multiplier for colour channel 11
>> + */
>> +struct mali_c55_params_awb_gains {
>> +	struct mali_c55_params_block_header header;
>> +	__u16 gain00;
>> +	__u16 gain01;
>> +	__u16 gain10;
>> +	__u16 gain11;
>> +};
>> +
>> +/**
>> + * enum mali_c55_params_awb_tap_points - Tap points for the AWB statistics
>> + * @MALI_C55_AWB_STATS_TAP_PF: Immediately after the Purple Fringe block
>> + * @MALI_C55_AWB_STATS_TAP_CNR: Immediately after the CNR block
>> + */
>> +enum mali_c55_params_awb_tap_points {
>> +	MALI_C55_AWB_STATS_TAP_PF = 0,
>> +	MALI_C55_AWB_STATS_TAP_CNR,
>> +};
>> +
>> +/**
>> + * struct mali_c55_params_awb_config - Stats settings for auto-white balance
>> + *
>> + * This struct allows the configuration of the statistics generated for auto
>> + * white balance. Pixel intensity limits can be set to exclude overly bright or
>> + * dark regions of an image from the statistics entirely. Colour ratio minima
>> + * and maxima can be set to discount pixels who's ratios fall outside the
>> + * defined boundaries; there are two sets of registers to do this - the
>> + * "min/max" ratios which bound a region and the "high/low" ratios which further
>> + * trim the upper and lower ratios. For example with the boundaries configured
>> + * as follows, only pixels whos colour ratios falls into the region marked "A"
>> + * would be counted::
>> + *
>> + *	                                                          cr_high
>> + *	    2.0 |                                                   |
>> + *	        |               cb_max --> _________________________v_____
>> + *	    1.8 |                         |                         \    |
>> + *	        |                         |                          \   |
>> + *	    1.6 |                         |                           \  |
>> + *	        |                         |                            \ |
>> + *	 c  1.4 |               cb_low -->|\              A             \|<--  cb_high
>> + *	 b      |                         | \                            |
>> + *	    1.2 |                         |  \                           |
>> + *	 r      |                         |   \                          |
>> + *	 a  1.0 |              cb_min --> |____\_________________________|
>> + *	 t      |                         ^    ^                         ^
>> + *	 i  0.8 |                         |    |                         |
>> + *	 o      |                      cr_min  |                       cr_max
>> + *	 s  0.6 |                              |
>> + *	        |                             cr_low
>> + *	    0.4 |
>> + *	        |
>> + *	    0.2 |
>> + *	        |
>> + *	    0.0 |_______________________________________________________________
>> + *	        0.0   0.2   0.4   0.6   0.8   1.0   1.2   1.4   1.6   1.8   2.0
>> + *	                                   cr ratios
>> + *
>> + * @header:		The Mali-C55 parameters block header
>> + * @tap_point:		The tap point from enum mali_c55_params_awb_tap_points
>> + * @stats_mode:		AWB statistics collection mode, see :c:type:`mali_c55_awb_stats_mode`
>> + * @white_level:	Upper pixel intensity (I.E. raw pixel values) limit
>> + * @black_level:	Lower pixel intensity (I.E. raw pixel values) limit
>> + * @cr_max:		Maximum R/G ratio (Q4.8 format)
>> + * @cr_min:		Minimum R/G ratio (Q4.8 format)
>> + * @cb_max:		Maximum B/G ratio (Q4.8 format)
>> + * @cb_min:		Minimum B/G ratio (Q4.8 format)
>> + * @nodes_used_horiz:	Number of active zones horizontally [0..15]
>> + * @nodes_used_vert:	Number of active zones vertically [0..15]
>> + * @cr_high:		R/G ratio trim high (Q4.8 format)
>> + * @cr_low:		R/G ratio trim low (Q4.8 format)
>> + * @cb_high:		B/G ratio trim high (Q4.8 format)
>> + * @cb_low:		B/G ratio trim low (Q4.8 format)
>> + */
>> +struct mali_c55_params_awb_config {
>> +	struct mali_c55_params_block_header header;
>> +	__u8 tap_point;
>> +	__u8 stats_mode;
>> +	__u16 white_level;
>> +	__u16 black_level;
>> +	__u16 cr_max;
>> +	__u16 cr_min;
>> +	__u16 cb_max;
>> +	__u16 cb_min;
>> +	__u8 nodes_used_horiz;
>> +	__u8 nodes_used_vert;
>> +	__u16 cr_high;
>> +	__u16 cr_low;
>> +	__u16 cb_high;
>> +	__u16 cb_low;
>> +};
>> +
>> +#define MALI_C55_NUM_MESH_SHADING_ELEMENTS 3072
>> +
>> +/**
>> + * struct mali_c55_params_mesh_shading_config - Mesh shading configuration
>> + *
>> + * The mesh shading correction module allows programming a separate table of
>> + * either 16x16 or 32x32 node coefficients for 3 different light sources. The
>> + * final correction coefficients applied are computed by blending the
>> + * coefficients from two tables together.
>> + *
>> + * A page of 1024 32-bit integers is associated to each colour channel, with
>> + * pages stored consecutively in memory. Each 32-bit integer packs 3 8-bit
>> + * correction coefficients for a single node, one for each of the three light
>> + * sources. The 8 most significant bits are unused. The following table
>> + * describes the layout::
>> + *
>> + *	+----------- Page (Colour Plane) 0 -------------+
>> + *	| @mesh[i]  | Mesh Point | Bits  | Light Source |
>> + *	+-----------+------------+-------+--------------+
>> + *	|         0 |        0,0 | 16,23 | LS2          |
>> + *	|           |            | 08-15 | LS1          |
>> + *	|           |            | 00-07 | LS0          |
>> + *	+-----------+------------+-------+--------------+
>> + *	|         1 |        0,1 | 16,23 | LS2          |
>> + *	|           |            | 08-15 | LS1          |
>> + *	|           |            | 00-07 | LS0          |
>> + *	+-----------+------------+-------+--------------+
>> + *	|       ... |        ... | ...   | ...          |
>> + *	+-----------+------------+-------+--------------+
>> + *	|      1023 |      31,31 | 16,23 | LS2          |
>> + *	|           |            | 08-15 | LS1          |
>> + *	|           |            | 00-07 | LS0          |
>> + *	+----------- Page (Colour Plane) 1 -------------+
>> + *	| @mesh[i]  | Mesh Point | Bits  | Light Source |
>> + *	+-----------+------------+-------+--------------+
>> + *	|      1024 |        0,0 | 16,23 | LS2          |
>> + *	|           |            | 08-15 | LS1          |
>> + *	|           |            | 00-07 | LS0          |
>> + *	+-----------+------------+-------+--------------+
>> + *	|      1025 |        0,1 | 16,23 | LS2          |
>> + *	|           |            | 08-15 | LS1          |
>> + *	|           |            | 00-07 | LS0          |
>> + *	+-----------+------------+-------+--------------+
>> + *	|       ... |        ... | ...   | ...          |
>> + *	+-----------+------------+-------+--------------+
>> + *	|      2047 |      31,31 | 16,23 | LS2          |
>> + *	|           |            | 08-15 | LS1          |
>> + *	|           |            | 00-07 | LS0          |
>> + *	+----------- Page (Colour Plane) 2 -------------+
>> + *	| @mesh[i]  | Mesh Point | Bits  | Light Source |
>> + *	+-----------+------------+-------+--------------+
>> + *	|      2048 |        0,0 | 16,23 | LS2          |
>> + *	|           |            | 08-15 | LS1          |
>> + *	|           |            | 00-07 | LS0          |
>> + *	+-----------+------------+-------+--------------+
>> + *	|      2049 |        0,1 | 16,23 | LS2          |
>> + *	|           |            | 08-15 | LS1          |
>> + *	|           |            | 00-07 | LS0          |
>> + *	+-----------+------------+-------+--------------+
>> + *	|       ... |        ... | ...   | ...          |
>> + *	+-----------+------------+-------+--------------+
>> + *	|      3071 |      31,31 | 16,23 | LS2          |
>> + *	|           |            | 08-15 | LS1          |
>> + *	|           |            | 00-07 | LS0          |
>> + *	+-----------+------------+-------+--------------+
>> + *
>> + * The @mesh_scale member determines the precision and minimum and maximum gain.
>> + * For example if @mesh_scale is 0 and therefore selects 0 - 2x gain, a value of
>> + * 0 in a coefficient means 0.0 gain, a value of 128 means 1.0 gain and 255
>> + * means 2.0 gain.
>> + *
>> + * @header:		The Mali-C55 parameters block header
>> + * @mesh_show:		Output the mesh data rather than image data
>> + * @mesh_scale:		Set the precision and maximum gain range of mesh shading
>> + *				- 0 = 0-2x gain
>> + *				- 1 = 0-4x gain
>> + *				- 2 = 0-8x gain
>> + *				- 3 = 0-16x gain
>> + *				- 4 = 1-2x gain
>> + *				- 5 = 1-3x gain
>> + *				- 6 = 1-5x gain
>> + *				- 7 = 1-9x gain
>> + * @mesh_page_r:	Mesh page select for red colour plane [0..2]
>> + * @mesh_page_g:	Mesh page select for green colour plane [0..2]
>> + * @mesh_page_b:	Mesh page select for blue colour plane [0..2]
>> + * @mesh_width:		Number of horizontal nodes minus 1 [15,31]
>> + * @mesh_height:	Number of vertical nodes minus 1 [15,31]
>> + * @mesh:		Mesh shading correction tables
>> + */
>> +struct mali_c55_params_mesh_shading_config {
>> +	struct mali_c55_params_block_header header;
>> +	bool mesh_show;
>> +	__u8 mesh_scale;
>> +	__u8 mesh_page_r;
>> +	__u8 mesh_page_g;
>> +	__u8 mesh_page_b;
>> +	__u8 mesh_width;
>> +	__u8 mesh_height;
>> +	__u32 mesh[MALI_C55_NUM_MESH_SHADING_ELEMENTS];
>> +};
>> +
>> +/** enum mali_c55_params_mesh_alpha_bank - Mesh shading table bank selection
>> + * @MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS1 - Select Light Sources 0 and 1
>> + * @MALI_C55_MESH_ALPHA_BANK_LS1_AND_LS2 - Select Light Sources 1 and 2
>> + * @MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS2 - Select Light Sources 0 and 2
>> + */
>> +enum mali_c55_params_mesh_alpha_bank {
>> +	MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS1 = 0,
>> +	MALI_C55_MESH_ALPHA_BANK_LS1_AND_LS2 = 1,
>> +	MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS2 = 4
>> +};
>> +
>> +/**
>> + * struct mali_c55_params_mesh_shading_selection - Mesh table selection
>> + *
>> + * The module computes the final correction coefficients by blending the ones
>> + * from two light source tables, which are selected (independently for each
>> + * colour channel) by the @mesh_alpha_bank_r/g/b fields.
>> + *
>> + * The final blended coefficients for each node are calculated using the
>> + * following equation:
>> + *
>> + *     Final coefficient = (a x LS\ :sub:`b`\ + (256 - a) x LS\ :sub:`a`\) / 256
>> + *
>> + * Where a is the @mesh_alpha_r/g/b value, and LS\ :sub:`a`\ and LS\ :sub:`b`\
>> + * are the node cofficients for the two tables selected by the
>> + * @mesh_alpha_bank_r/g/b value.
>> + *
>> + * The scale of the applied correction may also be controlled by tuning the
>> + * @mesh_strength member. This is a modifier to the final coefficients which can
>> + * be used to globally reduce the gains applied.
>> + *
>> + * @header:		The Mali-C55 parameters block header
>> + * @mesh_alpha_bank_r:	Red mesh table select (c:type:`enum mali_c55_params_mesh_alpha_bank`)
>> + * @mesh_alpha_bank_g:	Green mesh table select (c:type:`enum mali_c55_params_mesh_alpha_bank`)
>> + * @mesh_alpha_bank_b:	Blue mesh table select (c:type:`enum mali_c55_params_mesh_alpha_bank`)
>> + * @mesh_alpha_r:	Blend coefficient for R [0..255]
>> + * @mesh_alpha_g:	Blend coefficient for G [0..255]
>> + * @mesh_alpha_b:	Blend coefficient for B [0..255]
>> + * @mesh_strength:	Mesh strength in Q4.12 format [0..4096]
>> + */
>> +struct mali_c55_params_mesh_shading_selection {
>> +	struct mali_c55_params_block_header header;
>> +	__u8 mesh_alpha_bank_r;
>> +	__u8 mesh_alpha_bank_g;
>> +	__u8 mesh_alpha_bank_b;
>> +	__u8 mesh_alpha_r;
>> +	__u8 mesh_alpha_g;
>> +	__u8 mesh_alpha_b;
>> +	__u16 mesh_strength;
>> +};
>> +
>> +/**
>> + * define MALI_C55_PARAMS_MAX_SIZE - Maximum size of all Mali C55 Parameters
>> + *
>> + * Though the parameters for the Mali-C55 are passed as optional blocks, the
>> + * driver still needs to know the absolute maximum size so that it can allocate
>> + * a buffer sized appropriately to accomodate userspace attempting to set all
>> + * possible parameters in a single frame.
>> + */
>> +#define MALI_C55_PARAMS_MAX_SIZE				\
>> +	sizeof(struct mali_c55_params_sensor_off_preshading) + 	\
>> +	sizeof(struct mali_c55_params_aexp_hist) +		\
>> +	sizeof(struct mali_c55_params_aexp_weights) +		\
>> +	sizeof(struct mali_c55_params_aexp_hist) +		\
>> +	sizeof(struct mali_c55_params_aexp_weights) +		\
>> +	sizeof(struct mali_c55_params_digital_gain) +		\
>> +	sizeof(struct mali_c55_params_awb_gains) +		\
>> +	sizeof(struct mali_c55_params_awb_config) +		\
>> +	sizeof(struct mali_c55_params_awb_gains) +		\
> Some data structures are duplicated, please explain why somewhere, or it
> may appear to be a bug.


I'll add an explanatory comment

>
>> +	sizeof(struct mali_c55_params_mesh_shading_config) +	\
>> +	sizeof(struct mali_c55_params_mesh_shading_selection)
> I think this will work fine now, but as soon as you add a block that
> contains a 64-bit field, you'll have issues. Many CPUs expect 64-bit
> accesses to be aligned. If some structures have a size that are not
> multiples of 64-bit, you'll end up having unaligned blocks.
>
> I would recommend aligning the size of each data structure to 64 bits. A
> simple way to do so would be to align the size of the header to 64 bits.
> Please test the result to make sure I'm right though :-)
>
>> +
>> +/**
>> + * struct mali_c55_params_buffer - 3A configuration parameters
>> + *
>> + * This struct contains the configuration parameters of the Mali-C55 ISP
>> + * algorithms, serialized by userspace into an opaque data buffer. Each
> It's not opaque, you've documented it fully :-)
>
> s/opaque //


Yeah wrong word...I can't think of a better one - I'll just remove it.

>
>> + * configuration parameter block is represented by a block-specific structure
>> + * which contains a :c:type:`mali_c55_params_block_header` entry as first
>> + * member. Userspace populates the @data buffer with configuration parameters
>> + * for the blocks that it intends to configure. As a consequence, the data
>> + * buffer effective size changes according to the number of ISP blocks that
>> + * userspace intends to configure.
>> + *
>> + * The parameters buffer is versioned by the @version field to allow modifying
>> + * and extending its definition. Userspace should populate the @version field to
> s/should/shall/
>
>> + * inform the driver about the version it intends to use. The driver will parse
>> + * and handle the @data buffer according to the data layout specific to the
>> + * indicated revision and return an error if the desired revision is not
> s/revision/version/g
>
>> + * supported.
>> + *
>> + * For each ISP block that userspace wants to configure, a block-specific
>> + * structure is appended to the @data buffer, one after the other without gaps
>> + * in between nor overlaps. Userspace shall populate the @total_size field with
>> + * the effective size, in bytes, of the @data buffer.
>> + *
>> + * The expected memory layout of the parameters buffer is::
>> + *
>> + *	+-------------------- struct mali_c55_params_buffer ------------------+
>> + *	| version = MALI_C55_PARAM_BUFFER_V0;                                 |
>> + *	| total_size = sizeof(struct mali_c55_params_sensor_off_preshading)   |
>> + *	|              sizeof(struct mali_c55_params_aexp_hist);              |
>> + *	| +------------------------- data  ---------------------------------+ |
>> + *	| | +--------- struct mali_c55_params_sensor_off_preshading ------+ | |
>> + *	| | | +-------- struct mali_c55_params_block_header header -----+ | | |
>> + *	| | | | type = MALI_C55_PARAM_BLOCK_SENSOR_OFFS;                | | | |
>> + *	| | | | enabled = 1;                                            | | | |
>> + *	| | | | size =                                                  | | | |
>> + *	| | | |    sizeof(struct mali_c55_params_sensor_off_preshading);| | | |
>> + *	| | | +---------------------------------------------------------+ | | |
>> + *	| | | chan00 = ...;                                               | | |
>> + *	| | | chan01 = ...;                                               | | |
>> + *	| | | chan10 = ...;                                               | | |
>> + *	| | | chan11 = ...;                                               | | |
>> + *	| | +------------ struct mali_c55_params_aexp_hist ---------------+ | |
>> + *	| | | +-------- struct mali_c55_params_block_header header -----+ | | |
>> + *	| | | | type = MALI_C55_PARAM_BLOCK_AEXP_HIST;                  | | | |
>> + *	| | | | enabled = 1;                                            | | | |
>> + *	| | | | size = sizeof(struct mali_c55_params_aexp_hist);        | | | |
>> + *	| | | +---------------------------------------------------------+ | | |
>> + *	| | | skip_x = ...;                                               | | |
>> + *	| | | offset_x = ...;                                             | | |
>> + *	| | | skip_y = ...;                                               | | |
>> + *	| | | offset_y = ...;                                             | | |
>> + *	| | | scale_bottom = ...;                                         | | |
>> + *	| | | scale_top = ...;                                            | | |
>> + *	| | | plane_mode = ...;                                           | | |
>> + *	| | | tap_point = ...;                                            | | |
>> + *	| | +-------------------------------------------------------------+ | |
>> + *	| +-----------------------------------------------------------------+ |
>> + *	+---------------------------------------------------------------------+
>> + *
>> + * @version: The Mali-C55 parameters buffer version
>> + * @total_size: The Mali-C55 configuration data effective size, excluding this
>> + *		header
>> + * @data: The Mali-C55 configuration blocks data
>> + */
>> +struct mali_c55_params_buffer {
>> +	enum mali_c55_param_buffer_version version;
>> +	size_t total_size;
>> +	__u8 data[MALI_C55_PARAMS_MAX_SIZE];
>> +};
>> +
>>   #endif /* __UAPI_MALI_C55_CONFIG_H */

Re: [PATCH v5 14/16] media: uapi: Add parameters structs to mali-c55-config.h

[Laurent Pinchart]

Hi Dan,

On Fri, May 31, 2024 at 08:30:21AM +0100, Daniel Scally wrote:
> On 31/05/2024 01:09, Laurent Pinchart wrote:
> > On Wed, May 29, 2024 at 04:28:56PM +0100, Daniel Scally wrote:
> >> Add structures describing the ISP parameters to mali-c55-config.h
> >>
> >> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
> >> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> >> ---
> >> Changes in v5:
> >>
> >> 	- New patch
> >>
> >>   .../uapi/linux/media/arm/mali-c55-config.h    | 669 ++++++++++++++++++
> >>   1 file changed, 669 insertions(+)
> >>
> >> diff --git a/include/uapi/linux/media/arm/mali-c55-config.h b/include/uapi/linux/media/arm/mali-c55-config.h
> >> index 8fb89af6c874..fce14bc74f4a 100644
> >> --- a/include/uapi/linux/media/arm/mali-c55-config.h
> >> +++ b/include/uapi/linux/media/arm/mali-c55-config.h
> >> @@ -179,4 +179,673 @@ struct mali_c55_stats_buffer {
> >>   	__u32 reserved3[15];
> >>   } __attribute__((packed));
> >>   
> >> +/**
> >> + * enum mali_c55_param_buffer_version - Mali-C55 parameters block versioning
> >> + *
> >> + * @MALI_C55_PARAM_BUFFER_V0: First version of Mali-C55 parameters block
> >
> > So you like versions to be 0-based ? :-)
> 
> Hah - sure. I'll switch it to V1.
>
> >> + */
> >> +enum mali_c55_param_buffer_version {
> >> +	MALI_C55_PARAM_BUFFER_V0,
> >> +};
> >> +
> >> +/**
> >> + * enum mali_c55_param_block_type - Enumeration of Mali-C55 parameter blocks
> >> + *
> >> + * This enumeration defines the types of Mali-C55 parameters block. Each block
> >> + * configures a specific processing block of the Mali-C55 ISP. The block
> >> + * type allows the driver to correctly interpret the parameters block data.
> >> + *
> >> + * It is the responsibility of userspace to correctly set the type of each
> >> + * parameters block.
> >> + *
> >> + * @MALI_C55_PARAM_BLOCK_SENSOR_OFFS: Sensor pre-shading black level offset
> >> + * @MALI_C55_PARAM_BLOCK_AEXP_HIST: Auto-exposure 1024-bin histogram
> >> + *				    configuration
> >> + * @MALI_C55_PARAM_BLOCK_AEXP_IHIST: Post-Iridix auto-exposure 1024-bin
> >> + *				     histogram configuration
> >> + * @MALI_C55_PARAM_BLOCK_AEXP_HIST_WEIGHTS: Auto-exposure 1024-bin histogram
> >> + *					    weighting
> >> + * @MALI_C55_PARAM_BLOCK_AEXP_IHIST_WEIGHTS: Post-Iridix auto-exposure 1024-bin
> >> + *					     histogram weighting
> >
> > Do you expect applications to need to set the histogram configuration
> > and weights separately ?
> 
> **Need** to no, but it's convenient. I don't particularly expect the histogram configuration to be 
> changed past the first frame, but changing the metering mode for example would require a new weights 
> array and separating them avoids re-writing the config unnecessarily.

Fine with me.

> >> + * @MALI_C55_PARAM_BLOCK_DIGITAL_GAIN: Digital gain
> >> + * @MALI_C55_PARAM_BLOCK_AWB_GAINS: Auto-white balance gains
> >
> > I was thinking that those two could be combined, but if the former is
> > used by AEC and the latter by AWB, maybe keeping them separate would
> > make like easier for userspace.
> 
> They're used by the separate algorithms yes.
> 
> >> + * @MALI_C55_PARAM_BLOCK_AWB_CONFIG: Auto-white balance statistics config
> >> + * @MALI_C55_PARAM_BLOCK_AWB_GAINS_AEXP: Auto-white balance gains for AEXP-0 tap
> >> + * @MALI_C55_PARAM_MESH_SHADING_CONFIG : Mesh shading tables configuration
> >> + * @MALI_C55_PARAM_MESH_SHADING_SELECTION: Mesh shading table selection
> >> + * @MALI_C55_PARAM_BLOCK_SENTINEL: First non-valid block index
> >
> > You should indicate somewhere the correspondance between the block type
> > and the block data structure. It could be done in the definition of each
> > data structure for instance.
> 
> Hmm...just in the documentary comment? Or something more strict?

Just in the comments. As the mapping isn't 1:1 (here are structures used
by multiple block types), documenting the mapping explicitly could avoid
confusion and mistakes.

> >> + */
> >> +enum mali_c55_param_block_type {
> >> +	MALI_C55_PARAM_BLOCK_SENSOR_OFFS,
> >> +	MALI_C55_PARAM_BLOCK_AEXP_HIST,
> >> +	MALI_C55_PARAM_BLOCK_AEXP_IHIST,
> >> +	MALI_C55_PARAM_BLOCK_AEXP_HIST_WEIGHTS,
> >> +	MALI_C55_PARAM_BLOCK_AEXP_IHIST_WEIGHTS,
> >> +	MALI_C55_PARAM_BLOCK_DIGITAL_GAIN,
> >> +	MALI_C55_PARAM_BLOCK_AWB_GAINS,
> >> +	MALI_C55_PARAM_BLOCK_AWB_CONFIG,
> >> +	MALI_C55_PARAM_BLOCK_AWB_GAINS_AEXP,
> >> +	MALI_C55_PARAM_MESH_SHADING_CONFIG,
> >> +	MALI_C55_PARAM_MESH_SHADING_SELECTION,
> >> +	MALI_C55_PARAM_BLOCK_SENTINEL,
> >> +};
> >> +
> >> +/**
> >> + * struct mali_c55_params_block_header - Mali-C55 parameter block header
> >> + *
> >> + * This structure represents the common part of all the ISP configuration
> >> + * blocks. Each parameters block shall embed an instance of this structure type
> >
> > s/shall embed/embeds/
> >
> >> + * as its first member, followed by the block-specific configuration data. The
> >> + * driver inspects this common header to discern the block type and its size and
> >> + * properly handle the block content by casting it to the correct block-specific
> >> + * type.
> >> + *
> >> + * The @type field is one of the values enumerated by
> >> + * :c:type:`mali_c55_param_block_type` and specifies how the data should be
> >> + * interpreted by the driver. The @size field specifies the size of the
> >> + * parameters block and is used by the driver for validation purposes. The
> >> + * @enabled field specifies if the ISP block should be enabled (and configured
> >> + * according to the provided parameters) or disabled.
> >> + *
> >> + * .. code-block:: c
> >> + *
> >> + *	struct mali_c55_params_block_header *block = ...;
> >> + *
> >> + *	switch (block->type) {
> >> + *	case MALI_C55_PARAM_BLOCK_SENSOR_OFFS:
> >> + *		struct mali_c55_params_sensor_off_preshading *sensor_offs =
> >> + *			(struct mali_c55_params_sensor_off_preshading *)block;
> >> + *
> >> + *		if (block->size !=
> >> + *		    sizeof(struct mali_c55_params_sensor_off_preshading))
> >> + *			return -EINVAL;
> >> + *
> >> + *		handle_sensor_offs(sensor_offs);
> >> + *		break;
> >> + *
> >> + *	case MALI_C55_PARAM_BLOCK_AEXP_HIST:
> >> + *		struct mali_c55_params_aexp_hist *aexp_hist =
> >> + *			(struct mali_c55_params_aexp_hist)block;
> >> + *
> >> + *		if (block->size != sizeof(mali_c55_params_aexp_hist))
> >> + *			return -EINVAL;
> >> + *
> >> + *		handle_aexp_hist(aesp_hist);
> >> + *		break;
> >> + *
> >> + *	...
> >> + *
> >> + *	}
> >
> > I would probably skip this. The kernel-side of the implementation can be
> > found in the driver. I would document the UAPI here with examples
> > focussing on the userspace side only.
> 
> OK.
> 
> >> + *
> >> + * Userspace is responsible for correctly populating the parameters block header
> >> + * fields (@type, @enabled and @size) and correctly populate the block-specific
> >> + * parameters.
> >> + *
> >> + * For example:
> >> + *
> >> + * .. code-block:: c
> >> + *
> >> + *	void populate_sensor_offs(struct mali_c55_params_block_header *block) {
> >> + *		block->type = MALI_C55_PARAM_BLOCK_SENSOR_OFFS;
> >> + *		block->enabled = true;
> >> + *		block->size = sizeof(struct mali_c55_params_sensor_off_preshading);
> >> + *
> >> + *		struct mali_c55_params_sensor_off_preshading *sensor_offs =
> >> + *			(struct mali_c55_params_sensor_off_preshading *)block;
> >> + *
> >> + *		sensor_offs->chan00 = offset00;
> >> + *		sensor_offs->chan01 = offset01;
> >> + *		sensor_offs->chan10 = offset10;
> >> + *		sensor_offs->chan11 = offset11;
> >> + *	}
> >> + *
> >> + * @type: The parameters block type (enum mali_c55_param_block_type)
> >> + * @enabled: Block enabled/disabled flag
> >
> > Does this flag apply to all blocks ? If not, it would be good to
> > indicate which blocks it applies to.
> 
> It's part of the header, so it's available to all blocks...they can all be disabled somehow though 
> the exact mechanism differs.
> 
> > Also, I think we discussed previously that, when a block is disabled,
> > its configuration parameters are not applicable anymore, and could then
> > be skipped. Is that something useful to do ?
> 
> We don't pass blocks where the values are unchanged (so only a couple go in each frame - white 
> balance and digital gain IIRC), but if we want to transition a block from enabled to disabled for 
> whatever reason the driver would need to see it

When userspace wants to disable a block, it will include the block in
the params buffer with .enabeld set to false. That's not an issue. What
I was wondering is if the rest of the block (after the header) was
needed in that case, or if we could include the header only in that
case.

> >> + * @size: Size (in bytes) of the parameters block
> >> + */
> >> +struct mali_c55_params_block_header {
> >> +	enum mali_c55_param_block_type type;
> >> +	bool enabled;
> >> +	size_t size;
> >
> > enums, bool and size_t shouldn't be used in a UAPI, as they size may
> > vary between 32- and 64-bit architectures, making 32-bit userspace on a
> > 64-bit kernel require compat handling in the driver. Use types with
> > well-defined sizes. Same below.
> 
> Ack
> 
> >> +};
> >> +
> >> +/**
> >> + * struct mali_c55_params_sensor_off_preshading - offset subtraction for each
> >> + *						  color channel
> >> + *
> >> + * Provides removal of the sensor black level from the sensor data. Separate
> >> + * offsets are provided for each of the four Bayer component color channels
> >
> > That's a weird spelling of offsets.
> 
> Err...you mean "off"? It's from the documentation - I can spell it properly though.

I mean "offsets" vs. "offsets". But now that you mention it, the
shortened "off" spelling in the structure name can be a bit confusing.

> >> + * which are defaulted to R, Gr, Gb, B.
> >
> > I think it would be useful to indicate, for each structure, what block
> > it corresponds to in the ascii diagram in patch 12/16.
> 
> Sure, ok.
> 
> > On a side note,
> > at some point turning that diagram into SVG may make sense.
> 
> That diagram is incomplete...it's only supposed to highlight the tap points and the blocks that 
> surround them. So we'll need to extend it to accommodate that (which probably will be easier as an SVG)

Agreed.

> >> + *
> >> + * @header: The Mali-C55 parameters block header
> >> + * @chan00: Offset for color channel 00 (default: R)
> >> + * @chan01: Offset for color channel 01 (default: Gr)
> >> + * @chan10: Offset for color channel 10 (default: Gb)
> >> + * @chan11: Offset for color channel 11 (default: B)
> >> + */
> >> +struct mali_c55_params_sensor_off_preshading {
> >> +	struct mali_c55_params_block_header header;
> >> +	__u32 chan00;
> >> +	__u32 chan01;
> >> +	__u32 chan10;
> >> +	__u32 chan11;
> >> +};
> >> +
> >> +/**
> >> + * enum mali_c55_aexp_hist_tap_points - Tap points for the AEXP histogram
> >> + * @MALI_C55_AEXP_HIST_TAP_WB: After static white balance
> >> + * @MALI_C55_AEXP_HIST_TAP_FS: After WDR Frame Stitch
> >> + * @MALI_C55_AEXP_HIST_TAP_TPG: After the test pattern generator
> >> + */
> >> +enum mali_c55_aexp_hist_tap_points {
> >> +	MALI_C55_AEXP_HIST_TAP_WB = 0,
> >> +	MALI_C55_AEXP_HIST_TAP_FS,
> >> +	MALI_C55_AEXP_HIST_TAP_TPG,
> >> +};
> >> +
> >> +/**
> >> + * enum mali_c55_aexp_skip_x - Horizontal pixel skipping
> >> + * @MALI_C55_AEXP_SKIP_X_EVERY_2ND: Collect every 2nd pixel horizontally
> >> + * @MALI_C55_AEXP_SKIP_X_EVERY_3RD: Collect every 3rd pixel horizontally
> >> + * @MALI_C55_AEXP_SKIP_X_EVERY_4TH: Collect every 4th pixel horizontally
> >> + * @MALI_C55_AEXP_SKIP_X_EVERY_5TH: Collect every 5th pixel horizontally
> >> + * @MALI_C55_AEXP_SKIP_X_EVERY_8TH: Collect every 8th pixel horizontally
> >> + * @MALI_C55_AEXP_SKIP_X_EVERY_9TH: Collect every 9th pixel horizontally
> >> + */
> >> +enum mali_c55_aexp_skip_x {
> >> +	MALI_C55_AEXP_SKIP_X_EVERY_2ND,
> >> +	MALI_C55_AEXP_SKIP_X_EVERY_3RD,
> >> +	MALI_C55_AEXP_SKIP_X_EVERY_4TH,
> >> +	MALI_C55_AEXP_SKIP_X_EVERY_5TH,
> >> +	MALI_C55_AEXP_SKIP_X_EVERY_8TH,
> >> +	MALI_C55_AEXP_SKIP_X_EVERY_9TH
> >> +};
> >
> > Does this mean that that the histogram can't operate on every pixels,
> > but will always skip at least every other pixel ?
> 
> Horizontally, yes. So the maximum pixel count would be (width / 2) x height.
> 
> >> +
> >> +/**
> >> + * enum mali_c55_aexp_skip_y - Vertical pixel skipping
> >> + * @MALI_C55_AEXP_SKIP_Y_ALL: Collect every single pixel vertically
> >> + * @MALI_C55_AEXP_SKIP_Y_EVERY_2ND: Collect every 2nd pixel vertically
> >> + * @MALI_C55_AEXP_SKIP_Y_EVERY_3RD: Collect every 3rd pixel vertically
> >> + * @MALI_C55_AEXP_SKIP_Y_EVERY_4TH: Collect every 4th pixel vertically
> >> + * @MALI_C55_AEXP_SKIP_Y_EVERY_5TH: Collect every 5th pixel vertically
> >> + * @MALI_C55_AEXP_SKIP_Y_EVERY_8TH: Collect every 8th pixel vertically
> >> + * @MALI_C55_AEXP_SKIP_Y_EVERY_9TH: Collect every 9th pixel vertically
> >> + */
> >> +enum mali_c55_aexp_skip_y {
> >> +	MALI_C55_AEXP_SKIP_Y_ALL,
> >> +	MALI_C55_AEXP_SKIP_Y_EVERY_2ND,
> >> +	MALI_C55_AEXP_SKIP_Y_EVERY_3RD,
> >> +	MALI_C55_AEXP_SKIP_Y_EVERY_4TH,
> >> +	MALI_C55_AEXP_SKIP_Y_EVERY_5TH,
> >> +	MALI_C55_AEXP_SKIP_Y_EVERY_8TH,
> >> +	MALI_C55_AEXP_SKIP_Y_EVERY_9TH
> >> +};
> >> +
> >> +/**
> >> + * enum mali_c55_aexp_row_column_offset - Start from the first or second row or
> >> + *					  column
> >> + * @MALI_C55_AEXP_FIRST_ROW_OR_COL:	Start from the first row / column
> >> + * @MALI_C55_AEXP_SECOND_ROW_OR_COL:	Start from the second row / column
> >> + */
> >> +enum mali_c55_aexp_row_column_offset {
> >> +	MALI_C55_AEXP_FIRST_ROW_OR_COL = 1,
> >> +	MALI_C55_AEXP_SECOND_ROW_OR_COL = 2,
> >> +};
> >> +
> >> +/**
> >> + * enum mali_c55_aexp_hist_plane_mode - Mode for the AEXP Histograms
> >
> > As histograms are computed on bayer data, I'd talk about "component"
> > instead of "plane" here, unless the word plane matches documentation.
> 
> Plane is from the documentation yes.
> 
> >> + * @MALI_C55_AEXP_HIST_COMBINED: All color planes in one 1024-bin histogram
> >> + * @MALI_C55_AEXP_HIST_SEPARATE: Each color plane in one 256-bin histogram with a bin width of 16
> >> + * @MALI_C55_AEXP_HIST_FOCUS_00: Top left plane in the first bank, rest in second bank
> >> + * @MALI_C55_AEXP_HIST_FOCUS_01: Top right plane in the first bank, rest in second bank
> >> + * @MALI_C55_AEXP_HIST_FOCUS_10: Bottom left plane in the first bank, rest in second bank
> >> + * @MALI_C55_AEXP_HIST_FOCUS_11: Bottom right plane in the first bank, rest in second bank
> >> + *
> >> + * In the "focus" modes statistics are collected into two 512-bin histograms
> >> + * with a bin width of 8. One colour plane is in the first histogram with the
> >> + * remainder combined into the second. The four options represent which of the
> >> + * four positions in a bayer pattern are the focused plane.
> >
> > How does that work with x/y skipping ? Is skipping then applied to 2x2
> > blocks, or still to pixels ?
> 
> To pixels, so particular configurations of the skipping and the offsets can cause some of the colour 
> component histograms to be empty.

Does it mean that in modes other than MALI_C55_AEXP_HIST_COMBINED the
user should pick a /3, /5 or /9 downsampling ? I think it would be
useful to explain this a bit better.

> >> + */
> >> +enum mali_c55_aexp_hist_plane_mode {
> >> +	MALI_C55_AEXP_HIST_COMBINED = 0,
> >> +	MALI_C55_AEXP_HIST_SEPARATE = 1,
> >> +	MALI_C55_AEXP_HIST_FOCUS_00 = 4,
> >> +	MALI_C55_AEXP_HIST_FOCUS_01 = 5,
> >> +	MALI_C55_AEXP_HIST_FOCUS_10 = 6,
> >> +	MALI_C55_AEXP_HIST_FOCUS_11 = 7,
> >> +};
> >> +
> >> +/**
> >> + * struct mali_c55_params_aexp_hist - configuration for AEXP metering hists
> >> + *
> >> + * This struct allows users to configure the 1024-bin AEXP histograms. Broadly
> >> + * speaking the parameters allow you to mask particular regions of the image and
> >> + * to select different kinds of histogram.
> >> + *
> >> + * @header:		The Mali-C55 parameters block header
> >> + * @skip_x:		Horizontal decimation. See enum mali_c55_aexp_skip_x
> >> + * @offset_x:		Column to start from. See enum mali_c55_aexp_row_column_offset
> >> + * @skip_y:		Vertical decimation. See enum mali_c55_aexp_skip_y
> >> + * @offset_y:		Row to start from. See enum mali_c55_aexp_row_column_offset
> >
> > Do the offsets need to be multiples of 2 ?
>
> It's effectively a boolean field to decide whether to skip the first column/row or not.

That should be mentionedThat should be mentioned..

> >> + * @scale_bottom:	scale of bottom half of range: 0=1x ,1=2x, 2=4x, 4=8x, 4=16x
> >
> > Could you elaborate on this ?
> 
> Will do
> 
> >> + * @scale_top:		scale of top half of range: 0=1x ,1=2x, 2=4x, 4=8x, 4=16x
> >> + * @plane_mode:		Plane separation mode. See enum mali_c55_aexp_hist_plane_mode
> >> + * @tap_point:		Tap point for histogram from enum mali_c55_aexp_hist_tap_points.
> >> + *			This parameter is unused for the post-Iridix Histogram
> >> + */
> >> +struct mali_c55_params_aexp_hist {
> >> +	struct mali_c55_params_block_header header;
> >> +	__u8 skip_x;
> >> +	__u8 offset_x;
> >> +	__u8 skip_y;
> >> +	__u8 offset_y;
> >> +	__u8 scale_bottom;
> >> +	__u8 scale_top;
> >> +	__u8 plane_mode;
> >> +	__u8 tap_point;
> >> +};
> >> +
> >> +/**
> >> + * struct mali_c55_params_aexp_weights - Array of weights for AEXP metering
> >> + *
> >> + * This struct allows users to configure the weighting for both of the 1024-bin
> >> + * AEXP histograms. The pixel data collected for each zone is multiplied by the
> >> + * corresponding weight from this array, which may be zero if the intention is
> >> + * to mask off the zone entirely.
> >> + *
> >> + * @header:		The Mali-C55 parameters block header
> >> + * @nodes_used_horiz:	Number of active zones horizontally [0..15]
> >> + * @nodes_used_vert:	Number of active zones vertically [0..15]
> >
> > Is the image automatically split in the number of zones with a uniform
> > distribution ?
> 
> That's my understanding yes
> 
> > What happens if the number of zones is 0 ?
> 
> I don't know; haven't tried. I'll test it out.

I hope it wont cause the blue smoke to come out of the chip :-)

> >> + * @zone_weights:	Zone weighting. Index is row*col where 0,0 is the top
> >> + * 			left zone continuing in raster order. Each zone can be
> >> + *			weighted in the range [0..15]. The number of rows and
> >> + *			columns is defined by @nodes_used_vert and
> >> + *			@nodes_used_horiz
> >> + */
> >> +struct mali_c55_params_aexp_weights {
> >> +	struct mali_c55_params_block_header header;
> >> +	__u8 nodes_used_horiz;
> >> +	__u8 nodes_used_vert;
> >> +	__u8 zone_weights[MALI_C55_MAX_ZONES];
> >> +};
> >> +
> >> +/**
> >> + * struct mali_c55_params_digital_gain - Digital gain value
> >> + *
> >> + * This struct carries a digital gain value to set in the ISP
> >> + *
> >> + * @header:	The Mali-C55 parameters block header
> >> + * @gain:	The digital gain value to apply, in Q5.8 format.
> >> + */
> >> +struct mali_c55_params_digital_gain {
> >> +	struct mali_c55_params_block_header header;
> >> +	__u16 gain;
> >> +};
> >> +
> >> +/**
> >> + * enum mali_c55_awb_stats_mode - Statistics mode for AWB
> >> + * @MALI_C55_AWB_MODE_GRBR: Statistics collected as Green/Red and Blue/Red ratios
> >> + * @MALI_C55_AWB_MODE_RGBG: Statistics collected as Red/Green and Blue/Green ratios
> >> + */
> >> +enum mali_c55_awb_stats_mode {
> >> +       MALI_C55_AWB_MODE_GRBR = 0,
> >> +       MALI_C55_AWB_MODE_RGBG,
> >> +};
> >> +
> >> +/**
> >> + * struct mali_c55_params_awb_gains - Gain settings for auto white balance
> >> + *
> >> + * This struct allows users to configure the gains for auto-white balance. There
> >> + * are four gain settings corresponding to each colour channel in the bayer
> >> + * domain. Although named generically, the association between the gain applied
> >> + * and the colour channel is done automatically within the ISP depending on the
> >> + * input format, and so the following mapping always holds true::
> >> + *
> >> + *	gain00 = R
> >> + *	gain01 = Gr
> >> + *	gain10 = Gb
> >> + *	gain11 = B
> >
> > How about naming them accordingly then ? :-)
> 
> The gainNN comes from the register field names; I've used them for everything...does doing that 
> consistently have value? Or should I rename them?

I'm OK keeping the current names.

> >> + *
> >> + * All of the gains are stored in Q4.8 format.
> >> + *
> >> + * @header:	The Mali-C55 parameters block header
> >> + * @gain00:	Multiplier for colour channel 00
> >> + * @gain01:	Multiplier for colour channel 01
> >> + * @gain10:	Multiplier for colour channel 10
> >> + * @gain11:	Multiplier for colour channel 11
> >> + */
> >> +struct mali_c55_params_awb_gains {
> >> +	struct mali_c55_params_block_header header;
> >> +	__u16 gain00;
> >> +	__u16 gain01;
> >> +	__u16 gain10;
> >> +	__u16 gain11;
> >> +};
> >> +
> >> +/**
> >> + * enum mali_c55_params_awb_tap_points - Tap points for the AWB statistics
> >> + * @MALI_C55_AWB_STATS_TAP_PF: Immediately after the Purple Fringe block
> >> + * @MALI_C55_AWB_STATS_TAP_CNR: Immediately after the CNR block
> >> + */
> >> +enum mali_c55_params_awb_tap_points {
> >> +	MALI_C55_AWB_STATS_TAP_PF = 0,
> >> +	MALI_C55_AWB_STATS_TAP_CNR,
> >> +};
> >> +
> >> +/**
> >> + * struct mali_c55_params_awb_config - Stats settings for auto-white balance
> >> + *
> >> + * This struct allows the configuration of the statistics generated for auto
> >> + * white balance. Pixel intensity limits can be set to exclude overly bright or
> >> + * dark regions of an image from the statistics entirely. Colour ratio minima
> >> + * and maxima can be set to discount pixels who's ratios fall outside the
> >> + * defined boundaries; there are two sets of registers to do this - the
> >> + * "min/max" ratios which bound a region and the "high/low" ratios which further
> >> + * trim the upper and lower ratios. For example with the boundaries configured
> >> + * as follows, only pixels whos colour ratios falls into the region marked "A"
> >> + * would be counted::
> >> + *
> >> + *	                                                          cr_high
> >> + *	    2.0 |                                                   |
> >> + *	        |               cb_max --> _________________________v_____
> >> + *	    1.8 |                         |                         \    |
> >> + *	        |                         |                          \   |
> >> + *	    1.6 |                         |                           \  |
> >> + *	        |                         |                            \ |
> >> + *	 c  1.4 |               cb_low -->|\              A             \|<--  cb_high
> >> + *	 b      |                         | \                            |
> >> + *	    1.2 |                         |  \                           |
> >> + *	 r      |                         |   \                          |
> >> + *	 a  1.0 |              cb_min --> |____\_________________________|
> >> + *	 t      |                         ^    ^                         ^
> >> + *	 i  0.8 |                         |    |                         |
> >> + *	 o      |                      cr_min  |                       cr_max
> >> + *	 s  0.6 |                              |
> >> + *	        |                             cr_low
> >> + *	    0.4 |
> >> + *	        |
> >> + *	    0.2 |
> >> + *	        |
> >> + *	    0.0 |_______________________________________________________________
> >> + *	        0.0   0.2   0.4   0.6   0.8   1.0   1.2   1.4   1.6   1.8   2.0
> >> + *	                                   cr ratios
> >> + *
> >> + * @header:		The Mali-C55 parameters block header
> >> + * @tap_point:		The tap point from enum mali_c55_params_awb_tap_points
> >> + * @stats_mode:		AWB statistics collection mode, see :c:type:`mali_c55_awb_stats_mode`
> >> + * @white_level:	Upper pixel intensity (I.E. raw pixel values) limit
> >> + * @black_level:	Lower pixel intensity (I.E. raw pixel values) limit
> >> + * @cr_max:		Maximum R/G ratio (Q4.8 format)
> >> + * @cr_min:		Minimum R/G ratio (Q4.8 format)
> >> + * @cb_max:		Maximum B/G ratio (Q4.8 format)
> >> + * @cb_min:		Minimum B/G ratio (Q4.8 format)
> >> + * @nodes_used_horiz:	Number of active zones horizontally [0..15]
> >> + * @nodes_used_vert:	Number of active zones vertically [0..15]
> >> + * @cr_high:		R/G ratio trim high (Q4.8 format)
> >> + * @cr_low:		R/G ratio trim low (Q4.8 format)
> >> + * @cb_high:		B/G ratio trim high (Q4.8 format)
> >> + * @cb_low:		B/G ratio trim low (Q4.8 format)
> >> + */
> >> +struct mali_c55_params_awb_config {
> >> +	struct mali_c55_params_block_header header;
> >> +	__u8 tap_point;
> >> +	__u8 stats_mode;
> >> +	__u16 white_level;
> >> +	__u16 black_level;
> >> +	__u16 cr_max;
> >> +	__u16 cr_min;
> >> +	__u16 cb_max;
> >> +	__u16 cb_min;
> >> +	__u8 nodes_used_horiz;
> >> +	__u8 nodes_used_vert;
> >> +	__u16 cr_high;
> >> +	__u16 cr_low;
> >> +	__u16 cb_high;
> >> +	__u16 cb_low;
> >> +};
> >> +
> >> +#define MALI_C55_NUM_MESH_SHADING_ELEMENTS 3072
> >> +
> >> +/**
> >> + * struct mali_c55_params_mesh_shading_config - Mesh shading configuration
> >> + *
> >> + * The mesh shading correction module allows programming a separate table of
> >> + * either 16x16 or 32x32 node coefficients for 3 different light sources. The
> >> + * final correction coefficients applied are computed by blending the
> >> + * coefficients from two tables together.
> >> + *
> >> + * A page of 1024 32-bit integers is associated to each colour channel, with
> >> + * pages stored consecutively in memory. Each 32-bit integer packs 3 8-bit
> >> + * correction coefficients for a single node, one for each of the three light
> >> + * sources. The 8 most significant bits are unused. The following table
> >> + * describes the layout::
> >> + *
> >> + *	+----------- Page (Colour Plane) 0 -------------+
> >> + *	| @mesh[i]  | Mesh Point | Bits  | Light Source |
> >> + *	+-----------+------------+-------+--------------+
> >> + *	|         0 |        0,0 | 16,23 | LS2          |
> >> + *	|           |            | 08-15 | LS1          |
> >> + *	|           |            | 00-07 | LS0          |
> >> + *	+-----------+------------+-------+--------------+
> >> + *	|         1 |        0,1 | 16,23 | LS2          |
> >> + *	|           |            | 08-15 | LS1          |
> >> + *	|           |            | 00-07 | LS0          |
> >> + *	+-----------+------------+-------+--------------+
> >> + *	|       ... |        ... | ...   | ...          |
> >> + *	+-----------+------------+-------+--------------+
> >> + *	|      1023 |      31,31 | 16,23 | LS2          |
> >> + *	|           |            | 08-15 | LS1          |
> >> + *	|           |            | 00-07 | LS0          |
> >> + *	+----------- Page (Colour Plane) 1 -------------+
> >> + *	| @mesh[i]  | Mesh Point | Bits  | Light Source |
> >> + *	+-----------+------------+-------+--------------+
> >> + *	|      1024 |        0,0 | 16,23 | LS2          |
> >> + *	|           |            | 08-15 | LS1          |
> >> + *	|           |            | 00-07 | LS0          |
> >> + *	+-----------+------------+-------+--------------+
> >> + *	|      1025 |        0,1 | 16,23 | LS2          |
> >> + *	|           |            | 08-15 | LS1          |
> >> + *	|           |            | 00-07 | LS0          |
> >> + *	+-----------+------------+-------+--------------+
> >> + *	|       ... |        ... | ...   | ...          |
> >> + *	+-----------+------------+-------+--------------+
> >> + *	|      2047 |      31,31 | 16,23 | LS2          |
> >> + *	|           |            | 08-15 | LS1          |
> >> + *	|           |            | 00-07 | LS0          |
> >> + *	+----------- Page (Colour Plane) 2 -------------+
> >> + *	| @mesh[i]  | Mesh Point | Bits  | Light Source |
> >> + *	+-----------+------------+-------+--------------+
> >> + *	|      2048 |        0,0 | 16,23 | LS2          |
> >> + *	|           |            | 08-15 | LS1          |
> >> + *	|           |            | 00-07 | LS0          |
> >> + *	+-----------+------------+-------+--------------+
> >> + *	|      2049 |        0,1 | 16,23 | LS2          |
> >> + *	|           |            | 08-15 | LS1          |
> >> + *	|           |            | 00-07 | LS0          |
> >> + *	+-----------+------------+-------+--------------+
> >> + *	|       ... |        ... | ...   | ...          |
> >> + *	+-----------+------------+-------+--------------+
> >> + *	|      3071 |      31,31 | 16,23 | LS2          |
> >> + *	|           |            | 08-15 | LS1          |
> >> + *	|           |            | 00-07 | LS0          |
> >> + *	+-----------+------------+-------+--------------+
> >> + *
> >> + * The @mesh_scale member determines the precision and minimum and maximum gain.
> >> + * For example if @mesh_scale is 0 and therefore selects 0 - 2x gain, a value of
> >> + * 0 in a coefficient means 0.0 gain, a value of 128 means 1.0 gain and 255
> >> + * means 2.0 gain.
> >> + *
> >> + * @header:		The Mali-C55 parameters block header
> >> + * @mesh_show:		Output the mesh data rather than image data
> >> + * @mesh_scale:		Set the precision and maximum gain range of mesh shading
> >> + *				- 0 = 0-2x gain
> >> + *				- 1 = 0-4x gain
> >> + *				- 2 = 0-8x gain
> >> + *				- 3 = 0-16x gain
> >> + *				- 4 = 1-2x gain
> >> + *				- 5 = 1-3x gain
> >> + *				- 6 = 1-5x gain
> >> + *				- 7 = 1-9x gain
> >> + * @mesh_page_r:	Mesh page select for red colour plane [0..2]
> >> + * @mesh_page_g:	Mesh page select for green colour plane [0..2]
> >> + * @mesh_page_b:	Mesh page select for blue colour plane [0..2]
> >> + * @mesh_width:		Number of horizontal nodes minus 1 [15,31]
> >> + * @mesh_height:	Number of vertical nodes minus 1 [15,31]
> >> + * @mesh:		Mesh shading correction tables
> >> + */
> >> +struct mali_c55_params_mesh_shading_config {
> >> +	struct mali_c55_params_block_header header;
> >> +	bool mesh_show;
> >> +	__u8 mesh_scale;
> >> +	__u8 mesh_page_r;
> >> +	__u8 mesh_page_g;
> >> +	__u8 mesh_page_b;
> >> +	__u8 mesh_width;
> >> +	__u8 mesh_height;
> >> +	__u32 mesh[MALI_C55_NUM_MESH_SHADING_ELEMENTS];
> >> +};
> >> +
> >> +/** enum mali_c55_params_mesh_alpha_bank - Mesh shading table bank selection
> >> + * @MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS1 - Select Light Sources 0 and 1
> >> + * @MALI_C55_MESH_ALPHA_BANK_LS1_AND_LS2 - Select Light Sources 1 and 2
> >> + * @MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS2 - Select Light Sources 0 and 2
> >> + */
> >> +enum mali_c55_params_mesh_alpha_bank {
> >> +	MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS1 = 0,
> >> +	MALI_C55_MESH_ALPHA_BANK_LS1_AND_LS2 = 1,
> >> +	MALI_C55_MESH_ALPHA_BANK_LS0_AND_LS2 = 4
> >> +};
> >> +
> >> +/**
> >> + * struct mali_c55_params_mesh_shading_selection - Mesh table selection
> >> + *
> >> + * The module computes the final correction coefficients by blending the ones
> >> + * from two light source tables, which are selected (independently for each
> >> + * colour channel) by the @mesh_alpha_bank_r/g/b fields.
> >> + *
> >> + * The final blended coefficients for each node are calculated using the
> >> + * following equation:
> >> + *
> >> + *     Final coefficient = (a x LS\ :sub:`b`\ + (256 - a) x LS\ :sub:`a`\) / 256
> >> + *
> >> + * Where a is the @mesh_alpha_r/g/b value, and LS\ :sub:`a`\ and LS\ :sub:`b`\
> >> + * are the node cofficients for the two tables selected by the
> >> + * @mesh_alpha_bank_r/g/b value.
> >> + *
> >> + * The scale of the applied correction may also be controlled by tuning the
> >> + * @mesh_strength member. This is a modifier to the final coefficients which can
> >> + * be used to globally reduce the gains applied.
> >> + *
> >> + * @header:		The Mali-C55 parameters block header
> >> + * @mesh_alpha_bank_r:	Red mesh table select (c:type:`enum mali_c55_params_mesh_alpha_bank`)
> >> + * @mesh_alpha_bank_g:	Green mesh table select (c:type:`enum mali_c55_params_mesh_alpha_bank`)
> >> + * @mesh_alpha_bank_b:	Blue mesh table select (c:type:`enum mali_c55_params_mesh_alpha_bank`)
> >> + * @mesh_alpha_r:	Blend coefficient for R [0..255]
> >> + * @mesh_alpha_g:	Blend coefficient for G [0..255]
> >> + * @mesh_alpha_b:	Blend coefficient for B [0..255]
> >> + * @mesh_strength:	Mesh strength in Q4.12 format [0..4096]
> >> + */
> >> +struct mali_c55_params_mesh_shading_selection {
> >> +	struct mali_c55_params_block_header header;
> >> +	__u8 mesh_alpha_bank_r;
> >> +	__u8 mesh_alpha_bank_g;
> >> +	__u8 mesh_alpha_bank_b;
> >> +	__u8 mesh_alpha_r;
> >> +	__u8 mesh_alpha_g;
> >> +	__u8 mesh_alpha_b;
> >> +	__u16 mesh_strength;
> >> +};
> >> +
> >> +/**
> >> + * define MALI_C55_PARAMS_MAX_SIZE - Maximum size of all Mali C55 Parameters
> >> + *
> >> + * Though the parameters for the Mali-C55 are passed as optional blocks, the
> >> + * driver still needs to know the absolute maximum size so that it can allocate
> >> + * a buffer sized appropriately to accomodate userspace attempting to set all
> >> + * possible parameters in a single frame.
> >> + */
> >> +#define MALI_C55_PARAMS_MAX_SIZE				\
> >> +	sizeof(struct mali_c55_params_sensor_off_preshading) + 	\
> >> +	sizeof(struct mali_c55_params_aexp_hist) +		\
> >> +	sizeof(struct mali_c55_params_aexp_weights) +		\
> >> +	sizeof(struct mali_c55_params_aexp_hist) +		\
> >> +	sizeof(struct mali_c55_params_aexp_weights) +		\
> >> +	sizeof(struct mali_c55_params_digital_gain) +		\
> >> +	sizeof(struct mali_c55_params_awb_gains) +		\
> >> +	sizeof(struct mali_c55_params_awb_config) +		\
> >> +	sizeof(struct mali_c55_params_awb_gains) +		\
> >
> > Some data structures are duplicated, please explain why somewhere, or it
> > may appear to be a bug.
> 
> I'll add an explanatory comment
> 
> >> +	sizeof(struct mali_c55_params_mesh_shading_config) +	\
> >> +	sizeof(struct mali_c55_params_mesh_shading_selection)
> >
> > I think this will work fine now, but as soon as you add a block that
> > contains a 64-bit field, you'll have issues. Many CPUs expect 64-bit
> > accesses to be aligned. If some structures have a size that are not
> > multiples of 64-bit, you'll end up having unaligned blocks.
> >
> > I would recommend aligning the size of each data structure to 64 bits. A
> > simple way to do so would be to align the size of the header to 64 bits.
> > Please test the result to make sure I'm right though :-)
> >
> >> +
> >> +/**
> >> + * struct mali_c55_params_buffer - 3A configuration parameters
> >> + *
> >> + * This struct contains the configuration parameters of the Mali-C55 ISP
> >> + * algorithms, serialized by userspace into an opaque data buffer. Each
> >
> > It's not opaque, you've documented it fully :-)
> >
> > s/opaque //
> 
> Yeah wrong word...I can't think of a better one - I'll just remove it.
> 
> >> + * configuration parameter block is represented by a block-specific structure
> >> + * which contains a :c:type:`mali_c55_params_block_header` entry as first
> >> + * member. Userspace populates the @data buffer with configuration parameters
> >> + * for the blocks that it intends to configure. As a consequence, the data
> >> + * buffer effective size changes according to the number of ISP blocks that
> >> + * userspace intends to configure.
> >> + *
> >> + * The parameters buffer is versioned by the @version field to allow modifying
> >> + * and extending its definition. Userspace should populate the @version field to
> >
> > s/should/shall/
> >
> >> + * inform the driver about the version it intends to use. The driver will parse
> >> + * and handle the @data buffer according to the data layout specific to the
> >> + * indicated revision and return an error if the desired revision is not
> >
> > s/revision/version/g
> >
> >> + * supported.
> >> + *
> >> + * For each ISP block that userspace wants to configure, a block-specific
> >> + * structure is appended to the @data buffer, one after the other without gaps
> >> + * in between nor overlaps. Userspace shall populate the @total_size field with
> >> + * the effective size, in bytes, of the @data buffer.
> >> + *
> >> + * The expected memory layout of the parameters buffer is::
> >> + *
> >> + *	+-------------------- struct mali_c55_params_buffer ------------------+
> >> + *	| version = MALI_C55_PARAM_BUFFER_V0;                                 |
> >> + *	| total_size = sizeof(struct mali_c55_params_sensor_off_preshading)   |
> >> + *	|              sizeof(struct mali_c55_params_aexp_hist);              |
> >> + *	| +------------------------- data  ---------------------------------+ |
> >> + *	| | +--------- struct mali_c55_params_sensor_off_preshading ------+ | |
> >> + *	| | | +-------- struct mali_c55_params_block_header header -----+ | | |
> >> + *	| | | | type = MALI_C55_PARAM_BLOCK_SENSOR_OFFS;                | | | |
> >> + *	| | | | enabled = 1;                                            | | | |
> >> + *	| | | | size =                                                  | | | |
> >> + *	| | | |    sizeof(struct mali_c55_params_sensor_off_preshading);| | | |
> >> + *	| | | +---------------------------------------------------------+ | | |
> >> + *	| | | chan00 = ...;                                               | | |
> >> + *	| | | chan01 = ...;                                               | | |
> >> + *	| | | chan10 = ...;                                               | | |
> >> + *	| | | chan11 = ...;                                               | | |
> >> + *	| | +------------ struct mali_c55_params_aexp_hist ---------------+ | |
> >> + *	| | | +-------- struct mali_c55_params_block_header header -----+ | | |
> >> + *	| | | | type = MALI_C55_PARAM_BLOCK_AEXP_HIST;                  | | | |
> >> + *	| | | | enabled = 1;                                            | | | |
> >> + *	| | | | size = sizeof(struct mali_c55_params_aexp_hist);        | | | |
> >> + *	| | | +---------------------------------------------------------+ | | |
> >> + *	| | | skip_x = ...;                                               | | |
> >> + *	| | | offset_x = ...;                                             | | |
> >> + *	| | | skip_y = ...;                                               | | |
> >> + *	| | | offset_y = ...;                                             | | |
> >> + *	| | | scale_bottom = ...;                                         | | |
> >> + *	| | | scale_top = ...;                                            | | |
> >> + *	| | | plane_mode = ...;                                           | | |
> >> + *	| | | tap_point = ...;                                            | | |
> >> + *	| | +-------------------------------------------------------------+ | |
> >> + *	| +-----------------------------------------------------------------+ |
> >> + *	+---------------------------------------------------------------------+
> >> + *
> >> + * @version: The Mali-C55 parameters buffer version
> >> + * @total_size: The Mali-C55 configuration data effective size, excluding this
> >> + *		header
> >> + * @data: The Mali-C55 configuration blocks data
> >> + */
> >> +struct mali_c55_params_buffer {
> >> +	enum mali_c55_param_buffer_version version;
> >> +	size_t total_size;
> >> +	__u8 data[MALI_C55_PARAMS_MAX_SIZE];
> >> +};
> >> +
> >>   #endif /* __UAPI_MALI_C55_CONFIG_H */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Jacopo Mondi]

Hi Laurent

On Fri, May 31, 2024 at 12:43:48AM GMT, Laurent Pinchart wrote:
> And now the second part of the review, addressing mali-c55-capture.c and
> mali-c55-resizer.c. I've reviewed the code from the bottom up, so some
> messages may be repeated in an order that seems weird. Sorry about that.
>

[snip]

A few replies/questions on the resizer module

> >
> > > +
> > > +#endif /* _MALI_C55_RESIZER_COEFS_H */
> > > diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> > > new file mode 100644
> > > index 000000000000..0a5a2969d3ce
> > > --- /dev/null
> > > +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> > > @@ -0,0 +1,779 @@
> > > +// SPDX-License-Identifier: GPL-2.0
> > > +/*
> > > + * ARM Mali-C55 ISP Driver - Image signal processor
> > > + *
> > > + * Copyright (C) 2024 Ideas on Board Oy
> > > + */
> > > +
> > > +#include <linux/math.h>
> > > +#include <linux/minmax.h>
> > > +
> > > +#include <media/media-entity.h>
> > > +#include <media/v4l2-subdev.h>
> > > +
> > > +#include "mali-c55-common.h"
> > > +#include "mali-c55-registers.h"
> > > +#include "mali-c55-resizer-coefs.h"
> > > +
> > > +/* Scaling factor in Q4.20 format. */
> > > +#define MALI_C55_RZR_SCALER_FACTOR	(1U << 20)
> > > +
> > > +static const u32 rzr_non_bypass_src_fmts[] = {
> > > +	MEDIA_BUS_FMT_RGB121212_1X36,
> > > +	MEDIA_BUS_FMT_YUV10_1X30
> > > +};
> > > +
> > > +static const char * const mali_c55_resizer_names[] = {
> > > +	[MALI_C55_RZR_FR] = "resizer fr",
> > > +	[MALI_C55_RZR_DS] = "resizer ds",
> > > +};
> > > +
> > > +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
> > > +				     struct v4l2_subdev_state *state)
> > > +{
> > > +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
> > > +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> > > +	struct v4l2_mbus_framefmt *fmt;
> > > +	struct v4l2_rect *crop;
>
> const
>
> > > +
> > > +	/* Verify if crop should be enabled. */
> > > +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
> > > +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> > > +
> > > +	if (fmt->width == crop->width && fmt->height == crop->height)
> > > +		return MALI_C55_BYPASS_CROP;
> > > +
> > > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
> > > +		       crop->left);
> > > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
> > > +		       crop->top);
> > > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
> > > +		       crop->width);
> > > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
> > > +		       crop->height);
> > > +
> > > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
> > > +		       MALI_C55_CROP_ENABLE);
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
> > > +					struct v4l2_subdev_state *state)
> > > +{
> > > +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
> > > +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> > > +	struct v4l2_rect *crop, *scale;
>
> const
>
> Once "[PATCH v4 0/3] media: v4l2-subdev: Support const-awareness in
> state accessors" gets merged, the state argument to this function can be
> made const too. Same for other functions, as applicable.
>
> > > +	unsigned int h_bank, v_bank;
> > > +	u64 h_scale, v_scale;
> > > +
> > > +	/* Verify if scaling should be enabled. */
> > > +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> > > +	scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
> > > +
> > > +	if (crop->width == scale->width && crop->height == scale->height)
> > > +		return MALI_C55_BYPASS_SCALER;
> > > +
> > > +	/* Program the V/H scaling factor in Q4.20 format. */
> > > +	h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
> > > +	v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
> > > +
> > > +	do_div(h_scale, scale->width);
> > > +	do_div(v_scale, scale->height);
> > > +
> > > +	mali_c55_write(mali_c55,
> > > +		       MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
> > > +		       crop->width);
> > > +	mali_c55_write(mali_c55,
> > > +		       MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
> > > +		       crop->height);
> > > +
> > > +	mali_c55_write(mali_c55,
> > > +		       MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
> > > +		       scale->width);
> > > +	mali_c55_write(mali_c55,
> > > +		       MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
> > > +		       scale->height);
> > > +
> > > +	mali_c55_write(mali_c55,
> > > +		       MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
> > > +		       h_scale);
> > > +	mali_c55_write(mali_c55,
> > > +		       MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
> > > +		       v_scale);
> > > +
> > > +	h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
> > > +					     scale->width);
> > > +	mali_c55_write(mali_c55,
> > > +		       MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
> > > +		       h_bank);
> > > +
> > > +	v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
> > > +					     scale->height);
> > > +	mali_c55_write(mali_c55,
> > > +		       MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
> > > +		       v_bank);
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
> > > +				 struct v4l2_subdev_state *state)
> > > +{
> > > +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> > > +	u32 bypass = 0;
> > > +
> > > +	/* Verify if cropping and scaling should be enabled. */
> > > +	bypass |= mali_c55_rzr_program_crop(rzr, state);
> > > +	bypass |= mali_c55_rzr_program_resizer(rzr, state);
> > > +
> > > +	mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
> > > +			     MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
> > > +			     MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
> > > +			     bypass);
> > > +}
> > > +
> > > +/*
> > > + * Inspect the routing table to know which of the two (mutually exclusive)
> > > + * routes is enabled and return the sink pad id of the active route.
> > > + */
> > > +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
> > > +{
> > > +	struct v4l2_subdev_krouting *routing = &state->routing;
> > > +	struct v4l2_subdev_route *route;
> > > +
> > > +	/* A single route is enabled at a time. */
> > > +	for_each_active_route(routing, route)
> > > +		return route->sink_pad;
> > > +
> > > +	return MALI_C55_RZR_SINK_PAD;
> > > +}
> > > +
> > > +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
> > > +{
> > > +	u32 corrected_code = 0;
> > > +
> > > +	/*
> > > +	 * The ISP takes input in a 20-bit format, but can only output 16-bit
> > > +	 * RAW bayer data (with the 4 least significant bits from the input
> > > +	 * being lost). Return the 16-bit version of the 20-bit input formats.
> > > +	 */
> > > +	switch (mbus_code) {
> > > +	case MEDIA_BUS_FMT_SBGGR20_1X20:
> > > +		corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
> > > +		break;
> > > +	case MEDIA_BUS_FMT_SGBRG20_1X20:
> > > +		corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
> > > +		break;
> > > +	case MEDIA_BUS_FMT_SGRBG20_1X20:
> > > +		corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
> > > +		break;
> > > +	case MEDIA_BUS_FMT_SRGGB20_1X20:
> > > +		corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
> > > +		break;
>
> Would it make sense to add the shifted code to mali_c55_isp_fmt ?
>
> > > +	}
> > > +
> > > +	return corrected_code;
> > > +}
> > > +
> > > +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> > > +				      struct v4l2_subdev_state *state,
> > > +				      struct v4l2_subdev_krouting *routing)
>
> I think the last argument can be const.

If I have to adjust the routing table instead of refusing it, it can't

>
> > > +{
> > > +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> > > +						    sd);
>
> A to_mali_c55_resizer() static inline function would be useful. Same for
> other components, where applicable.
>
> > > +	unsigned int active_sink = UINT_MAX;
> > > +	struct v4l2_mbus_framefmt *src_fmt;
> > > +	struct v4l2_rect *crop, *compose;
> > > +	struct v4l2_subdev_route *route;
> > > +	unsigned int active_routes = 0;
> > > +	struct v4l2_mbus_framefmt *fmt;
> > > +	int ret;
> > > +
> > > +	ret = v4l2_subdev_routing_validate(sd, routing, 0);
> > > +	if (ret)
> > > +		return ret;
> > > +
> > > +	/* Only a single route can be enabled at a time. */
> > > +	for_each_active_route(routing, route) {
> > > +		if (++active_routes > 1) {
> > > +			dev_err(rzr->mali_c55->dev,
> > > +				"Only one route can be active");
>
> No kernel log message with a level higher than dev_dbg() from
> user-controlled paths please, here and where applicable. This is to
> avoid giving applications an easy way to flood the kernel log.
>
> > > +			return -EINVAL;
> > > +		}
> > > +
> > > +		active_sink = route->sink_pad;
> > > +	}
> > > +	if (active_sink == UINT_MAX) {
> > > +		dev_err(rzr->mali_c55->dev, "One route has to be active");
> > > +		return -EINVAL;
> > > +	}
>
> The recommended handling of invalid routing is to adjust the routing
> table, not to return errors.
>

How should I adjust it ? The error here is due to the fact multiple
routes are set as active, which one should I make active ? the first
one ? Should I go and reset the flags in the subdev_route for the one
that has to be made non-active ?

> > > +
> > > +	ret = v4l2_subdev_set_routing(sd, state, routing);
> > > +	if (ret) {
> > > +		dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
> > > +		return ret;
> > > +	}
> > > +
> > > +	fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
> > > +	crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
> > > +	compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
> > > +
> > > +	fmt->width = MALI_C55_DEFAULT_WIDTH;
> > > +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
> > > +	fmt->colorspace = V4L2_COLORSPACE_SRGB;
>
> There are other colorspace-related fields.
>
> > > +	fmt->field = V4L2_FIELD_NONE;
>
> I wonder if we should really update the sink pad format, or just
> propagate it. If we update it, I think it should be set to defaults on
> both sink pads, not just the active sink pad.
>

If only one route can be active, there will only be one state.stream_config
entry for the active sink, not for the other one (see
v4l2_subdev_init_stream_configs()), this mean I can't reset both sink
formats ?

> > > +
> > > +	if (active_sink == MALI_C55_RZR_SINK_PAD) {
> > > +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > > +
> > > +		crop->left = crop->top = 0;
>
> 		crop->left = 0;
> 		crop->top = 0;
>
> > > +		crop->width = MALI_C55_DEFAULT_WIDTH;
> > > +		crop->height = MALI_C55_DEFAULT_HEIGHT;
> > > +
> > > +		*compose = *crop;
> > > +	} else {
> > > +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> > > +	}
> > > +
> > > +	/* Propagate the format to the source pad */
> > > +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
> > > +					       0);
> > > +	*src_fmt = *fmt;
> > > +
> > > +	/* In the event this is the bypass pad the mbus code needs correcting */
> > > +	if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
> > > +		src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
> > > +				       struct v4l2_subdev_state *state,
> > > +				       struct v4l2_subdev_mbus_code_enum *code)
> > > +{
> > > +	struct v4l2_mbus_framefmt *sink_fmt;
> > > +	const struct mali_c55_isp_fmt *fmt;
> > > +	unsigned int index = 0;
> > > +	u32 sink_pad;
> > > +
> > > +	switch (code->pad) {
> > > +	case MALI_C55_RZR_SINK_PAD:
> > > +		if (code->index)
> > > +			return -EINVAL;
> > > +
> > > +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > > +
> > > +		return 0;
> > > +	case MALI_C55_RZR_SOURCE_PAD:
> > > +		sink_pad = mali_c55_rzr_get_active_sink(state);
> > > +		sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> > > +
> > > +		/*
> > > +		 * If the active route is from the Bypass sink pad, then the
> > > +		 * source pad is a simple passthrough of the sink format,
> > > +		 * downshifted to 16-bits.
> > > +		 */
> > > +
> > > +		if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> > > +			if (code->index)
> > > +				return -EINVAL;
> > > +
> > > +			code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> > > +			if (!code->code)
> > > +				return -EINVAL;
> > > +
> > > +			return 0;
> > > +		}
> > > +
> > > +		/*
> > > +		 * If the active route is from the non-bypass sink then we can
> > > +		 * select either RGB or conversion to YUV.
> > > +		 */
> > > +
> > > +		if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
> > > +			return -EINVAL;
> > > +
> > > +		code->code = rzr_non_bypass_src_fmts[code->index];
> > > +
> > > +		return 0;
> > > +	case MALI_C55_RZR_SINK_BYPASS_PAD:
> > > +		for_each_mali_isp_fmt(fmt) {
> > > +			if (index++ == code->index) {
> > > +				code->code = fmt->code;
> > > +				return 0;
> > > +			}
> > > +		}
> > > +
> > > +		break;
> > > +	}
> > > +
> > > +	return -EINVAL;
> > > +}
> > > +
> > > +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
> > > +					struct v4l2_subdev_state *state,
> > > +					struct v4l2_subdev_frame_size_enum *fse)
> > > +{
> > > +	if (fse->index)
> > > +		return -EINVAL;
> > > +
> > > +	fse->max_width = MALI_C55_MAX_WIDTH;
> > > +	fse->max_height = MALI_C55_MAX_HEIGHT;
> > > +	fse->min_width = MALI_C55_MIN_WIDTH;
> > > +	fse->min_height = MALI_C55_MIN_HEIGHT;
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
> > > +				     struct v4l2_subdev_state *state,
> > > +				     struct v4l2_subdev_format *format)
> > > +{
> > > +	struct v4l2_mbus_framefmt *fmt = &format->format;
> > > +	struct v4l2_rect *rect;
> > > +	unsigned int sink_pad;
> > > +
> > > +	/*
> > > +	 * Clamp to min/max and then reset crop and compose rectangles to the
> > > +	 * newly applied size.
> > > +	 */
> > > +	clamp_t(unsigned int, fmt->width,
> > > +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);

also, clamp_t doens't clamp in place

        fmt->width = clamp_t...

> > > +	clamp_t(unsigned int, fmt->height,
> > > +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>
> Please check comments for other components related to the colorspace
> fields, to decide how to handle them here.
>
> > > +
> > > +	sink_pad = mali_c55_rzr_get_active_sink(state);
> > > +	if (sink_pad == MALI_C55_RZR_SINK_PAD) {
>
> The selection here should depend on format->pad, not the active sink
> pad.
>
> > > +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > > +
> > > +		rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> > > +		rect->left = 0;
> > > +		rect->top = 0;
> > > +		rect->width = fmt->width;
> > > +		rect->height = fmt->height;
> > > +
> > > +		rect = v4l2_subdev_state_get_compose(state,
> > > +						     MALI_C55_RZR_SINK_PAD);
> > > +		rect->left = 0;
> > > +		rect->top = 0;
> > > +		rect->width = fmt->width;
> > > +		rect->height = fmt->height;
> > > +	} else {
> > > +		/*
> > > +		 * Make sure the media bus code is one of the supported
> > > +		 * ISP input media bus codes.
> > > +		 */
> > > +		if (!mali_c55_isp_is_format_supported(fmt->code))
> > > +			fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;

And DEFAULT_MEDIA_BUS_FMT is not one of the supported input media bus
codes

> > > +	}
> > > +
> > > +	*v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
> > > +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
>
> Propagation to the source pad, however, should depend on the active
> route. If format->pad is routed to the source pad, you should propagate,
> otherwise, you shouldn't.
>
> > > +
> > > +	return 0;

I ended up with

static int mali_c55_rsz_set_sink_fmt(struct v4l2_subdev *sd,
				     struct v4l2_subdev_state *state,
				     struct v4l2_subdev_format *format)
{
	struct v4l2_mbus_framefmt *fmt = &format->format;
	unsigned int active_sink;
	struct v4l2_rect *rect;

	/*
	 * Clamp to min/max and then reset crop and compose rectangles to the
	 * newly applied size.
	 */
	fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
			     MALI_C55_MAX_WIDTH);
	fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
			      MALI_C55_MAX_HEIGHT);

	rect = v4l2_subdev_state_get_crop(state, format->pad);
	rect->left = 0;
	rect->top = 0;
	rect->width = fmt->width;
	rect->height = fmt->height;

	rect = v4l2_subdev_state_get_compose(state, format->pad);
	rect->left = 0;
	rect->top = 0;
	rect->width = fmt->width;
	rect->height = fmt->height;

	if (format->pad == MALI_C55_RSZ_SINK_BYPASS_PAD) {
		/*
		 * Make sure the media bus code is one of the supported
		 * ISP input media bus codes. Default it to SRGGB otherwise.
		 */
		if (!mali_c55_isp_is_format_supported(fmt->code))
			fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
	} else {
		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
	}

	*v4l2_subdev_state_get_format(state, format->pad, 0) = *fmt;

	/* If format->pad is routed to the source pad, propagate the format. */
	active_sink = mali_c55_rsz_get_active_sink(state);
	if (active_sink == format->pad) {

		/* If the bypass route is used, downshift the code to 16bpp. */
		if (active_sink == MALI_C55_RSZ_SINK_BYPASS_PAD)
			fmt->code = mali_c55_rsz_shift_mbus_code(fmt->code);

		*v4l2_subdev_state_get_format(state,
					      MALI_C55_RSZ_SOURCE_PAD, 0) = *fmt;
	}

	return 0;
}
> > > +}
> > > +
> > > +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> > > +				       struct v4l2_subdev_state *state,
> > > +				       struct v4l2_subdev_format *format)
> > > +{
> > > +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> > > +						    sd);
> > > +	struct v4l2_mbus_framefmt *fmt = &format->format;
> > > +	struct v4l2_mbus_framefmt *sink_fmt;
> > > +	struct v4l2_rect *crop, *compose;
> > > +	unsigned int sink_pad;
> > > +	unsigned int i;
> > > +
> > > +	sink_pad = mali_c55_rzr_get_active_sink(state);
> > > +	sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> > > +	crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
> > > +	compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
> > > +
> > > +	/* FR Bypass pipe. */
> > > +
> > > +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> > > +		/*
> > > +		 * Format on the source pad is the same as the one on the
> > > +		 * sink pad, downshifted to 16-bits.
> > > +		 */
> > > +		fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> > > +		if (!fmt->code)
> > > +			return -EINVAL;
> > > +
> > > +		/* RAW bypass disables scaling and cropping. */
> > > +		crop->top = compose->top = 0;
> > > +		crop->left = compose->left = 0;
> > > +		fmt->width = crop->width = compose->width = sink_fmt->width;
> > > +		fmt->height = crop->height = compose->height = sink_fmt->height;
>
> I don't think this is right. This function sets the format on the source
> pad. Subdevs should propagate formats from the sink to the source, not
> the other way around.
>
> The only parameter that can be modified on the source pad (as far as I
> understand) is the media bus code. In the bypass path, I understand it's
> fixed, while in the other path, you can select between RGB and YUV. I
> think the following code is what you need to implement this function.
>
> static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> 				       struct v4l2_subdev_state *state,
> 				       struct v4l2_subdev_format *format)
> {
> 	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> 						    sd);
> 	struct v4l2_mbus_framefmt *fmt;
>
> 	fmt = v4l2_subdev_state_get_format(state, format->pad);
>
> 	/* In the non-bypass path the output format can be selected. */
> 	if (mali_c55_rzr_get_active_sink(state) == MALI_C55_RZR_SINK_PAD) {
> 		unsigned int i;
>
> 		fmt->code = format->format.code;
>
> 		for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> 			if (fmt->code == rzr_non_bypass_src_fmts[i])
> 				break;
> 		}
>
> 		if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts))
> 			fmt->code = rzr_non_bypass_src_fmts[0];
> 	}
>
> 	format->format = *fmt;
>
> 	return 0;
> }

Almost. Your proposal doesn't adjust format->format.width/height

I think the following is more appropriate

static int mali_c55_rsz_set_source_fmt(struct v4l2_subdev *sd,
				       struct v4l2_subdev_state *state,
				       struct v4l2_subdev_format *format)
{
	struct v4l2_mbus_framefmt *fmt = &format->format;
	struct v4l2_mbus_framefmt *sink_fmt;
	struct v4l2_rect *sink_compose;
	unsigned int active_sink;

	active_sink = mali_c55_rsz_get_active_sink(state);
	sink_fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
	sink_compose = v4l2_subdev_state_get_compose(state, active_sink, 0);

	/*
	 * The source pad format sizes come directly from the active sink pad
	 * compose rectangle.
	 */
	fmt->width = sink_compose->width;
	fmt->height = sink_compose->height;

	if (active_sink == MALI_C55_RSZ_SINK_PAD) {
		/*
		 * Regular processing pipe: RGB121212 can be color-space
		 * converted to YUV101010.
		 */
		unsigned int i;

		for (i = 0; i < ARRAY_SIZE(rsz_non_bypass_src_fmts); i++) {
			if (fmt->code == rsz_non_bypass_src_fmts[i])
				break;
		}

		if (i == ARRAY_SIZE(rsz_non_bypass_src_fmts))
			fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
	} else {
		/*
		 * Bypass pipe: the source format is the same as the bypass
		 * sink pad downshifted to 16bpp.
		 */
		fmt->code = mali_c55_rsz_shift_mbus_code(sink_fmt->code);
	}

	*v4l2_subdev_state_get_format(state, MALI_C55_RSZ_SOURCE_PAD) = *fmt;

	return 0;
}

I'll handle the colorspace fields as well

>
> > > +
> > > +		*v4l2_subdev_state_get_format(state,
> > > +					      MALI_C55_RZR_SOURCE_PAD) = *fmt;
> > > +
> > > +		return 0;
> > > +	}
> > > +
> > > +	/* Regular processing pipe. */
> > > +
> > > +	for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> > > +		if (fmt->code == rzr_non_bypass_src_fmts[i])
> > > +			break;
> > > +	}
> > > +
> > > +	if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
> > > +		dev_dbg(rzr->mali_c55->dev,
> > > +			"Unsupported mbus code 0x%x: using default\n",
> > > +			fmt->code);
>
> I think you can drop this message.
>
> > > +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > > +	}
> > > +
> > > +	/*
> > > +	 * The source pad format size comes directly from the sink pad
> > > +	 * compose rectangle.
> > > +	 */
> > > +	fmt->width = compose->width;
> > > +	fmt->height = compose->height;
> > > +
> > > +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
> > > +				struct v4l2_subdev_state *state,
> > > +				struct v4l2_subdev_format *format)
> > > +{
> > > +	/*
> > > +	 * On sink pads fmt is either fixed for the 'regular' processing
> > > +	 * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
> > > +	 * pad.
> > > +	 *
> > > +	 * On source pad sizes are the result of crop+compose on the sink
> > > +	 * pad sizes, while the format depends on the active route.
> > > +	 */
> > > +
> > > +	if (format->pad != MALI_C55_RZR_SOURCE_PAD)
> > > +		return mali_c55_rzr_set_sink_fmt(sd, state, format);
> > > +
> > > +	return mali_c55_rzr_set_source_fmt(sd, state, format);
>
> Nitpicking,
>
> 	if (format->pad == MALI_C55_RZR_SOURCE_PAD)
> 		return mali_c55_rzr_set_source_fmt(sd, state, format);
>
> 	return mali_c55_rzr_set_sink_fmt(sd, state, format);
>
> to match SOURCE_PAD and source_fmt.
>

Done at the expense a bit more verbose check

	if (format->pad == MALI_C55_RSZ_SINK_PAD ||
	    format->pad == MALI_C55_RSZ_SINK_BYPASS_PAD)
		return mali_c55_rsz_set_sink_fmt(sd, state, format);

> > > +}
> > > +
> > > +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
> > > +				      struct v4l2_subdev_state *state,
> > > +				      struct v4l2_subdev_selection *sel)
> > > +{
> > > +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
> > > +		return -EINVAL;
> > > +
> > > +	if (sel->target != V4L2_SEL_TGT_CROP &&
> > > +	    sel->target != V4L2_SEL_TGT_COMPOSE)
> > > +		return -EINVAL;
> > > +
> > > +	sel->r = sel->target == V4L2_SEL_TGT_CROP
> > > +	       ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
> > > +	       : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
> > > +				      struct v4l2_subdev_state *state,
> > > +				      struct v4l2_subdev_selection *sel)
> > > +{
> > > +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> > > +						    sd);
> > > +	struct v4l2_mbus_framefmt *source_fmt;
> > > +	struct v4l2_mbus_framefmt *sink_fmt;
> > > +	struct v4l2_rect *crop, *compose;
> > > +
> > > +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
> > > +		return -EINVAL;
> > > +
> > > +	if (sel->target != V4L2_SEL_TGT_CROP &&
> > > +	    sel->target != V4L2_SEL_TGT_COMPOSE)
> > > +		return -EINVAL;
> > > +
> > > +	source_fmt = v4l2_subdev_state_get_format(state,
> > > +						  MALI_C55_RZR_SOURCE_PAD);
> > > +	sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
> > > +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> > > +	compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> > > +
> > > +	/* RAW bypass disables crop/scaling. */
> > > +	if (mali_c55_format_is_raw(source_fmt->code)) {
> > > +		crop->top = compose->top = 0;
> > > +		crop->left = compose->left = 0;
> > > +		crop->width = compose->width = sink_fmt->width;
> > > +		crop->height = compose->height = sink_fmt->height;
> > > +
> > > +		sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> > > +
> > > +		return 0;
> > > +	}
> > > +
> > > +	/* During streaming, it is allowed to only change the crop rectangle. */
> > > +	if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
> > > +		return -EINVAL;
> > > +
> > > +	 /*
> > > +	  * Update the desired target and then clamp the crop rectangle to the
> > > +	  * sink format sizes and the compose size to the crop sizes.
> > > +	  */
> > > +	if (sel->target == V4L2_SEL_TGT_CROP)
> > > +		*crop = sel->r;
> > > +	else
> > > +		*compose = sel->r;
> > > +
> > > +	clamp_t(unsigned int, crop->left, 0,  sink_fmt->width);
> > > +	clamp_t(unsigned int, crop->top, 0,  sink_fmt->height);
> > > +	clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
> > > +		sink_fmt->width - crop->left);
> > > +	clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
> > > +		sink_fmt->height - crop->top);
> > > +
> > > +	if (rzr->streaming) {
> > > +		/*
> > > +		 * Apply at runtime a crop rectangle on the resizer's sink only
> > > +		 * if it doesn't require re-programming the scaler output sizes
> > > +		 * as it would require changing the output buffer sizes as well.
> > > +		 */
> > > +		if (sel->r.width < compose->width ||
> > > +		    sel->r.height < compose->height)
> > > +			return -EINVAL;
> > > +
> > > +		*crop = sel->r;
> > > +		mali_c55_rzr_program(rzr, state);
> > > +
> > > +		return 0;
> > > +	}
> > > +
> > > +	compose->left = 0;
> > > +	compose->top = 0;
> > > +	clamp_t(unsigned int, compose->left, 0,  sink_fmt->width);
> > > +	clamp_t(unsigned int, compose->top, 0,  sink_fmt->height);
> > > +	clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
> > > +	clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
> > > +
> > > +	sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> > > +				    struct v4l2_subdev_state *state,
> > > +				    enum v4l2_subdev_format_whence which,
> > > +				    struct v4l2_subdev_krouting *routing)
> > > +{
> > > +	if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
> > > +	    media_entity_is_streaming(&sd->entity))
> > > +		return -EBUSY;
> > > +
> > > +	return __mali_c55_rzr_set_routing(sd, state, routing);
> > > +}
> > > +
> > > +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
> > > +	.enum_mbus_code		= mali_c55_rzr_enum_mbus_code,
> > > +	.enum_frame_size	= mali_c55_rzr_enum_frame_size,
> > > +	.get_fmt		= v4l2_subdev_get_fmt,
> > > +	.set_fmt		= mali_c55_rzr_set_fmt,
> > > +	.get_selection		= mali_c55_rzr_get_selection,
> > > +	.set_selection		= mali_c55_rzr_set_selection,
> > > +	.set_routing		= mali_c55_rzr_set_routing,
> > > +};
> > > +
> > > +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
>
> Could this be handled through the .enable_streams() and
> .disable_streams() operations ? They ensure that the stream state stored
> internal is correct. That may not matter much today, but I think it will
> become increasingly important in the future for the V4L2 core.
>
> > > +{
> > > +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> > > +	struct v4l2_subdev *sd = &rzr->sd;
> > > +	struct v4l2_subdev_state *state;
> > > +	unsigned int sink_pad;
> > > +
> > > +	state = v4l2_subdev_lock_and_get_active_state(sd);
> > > +
> > > +	sink_pad = mali_c55_rzr_get_active_sink(state);
> > > +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> > > +		/* Bypass FR pipe processing if the bypass route is active. */
> > > +		mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> > > +				     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
> > > +				     MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
> > > +		goto unlock_state;
> > > +	}
> > > +
> > > +	/* Disable bypass and use regular processing. */
> > > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> > > +			     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
> > > +	mali_c55_rzr_program(rzr, state);
> > > +
> > > +unlock_state:
> > > +	rzr->streaming = true;
>
> And hopefully you'll be able to replace this with
> v4l2_subdev_is_streaming(), introduced in "[PATCH v6 00/11] media:
> subdev: Improve stream enable/disable machinery" (Sakari has sent a pull
> request for v6.11 yesterday).
>
> > > +	v4l2_subdev_unlock_state(state);
> > > +}
> > > +
> > > +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
> > > +{
> > > +	struct v4l2_subdev *sd = &rzr->sd;
> > > +	struct v4l2_subdev_state *state;
> > > +
> > > +	state = v4l2_subdev_lock_and_get_active_state(sd);
> > > +	rzr->streaming = false;
> > > +	v4l2_subdev_unlock_state(state);
> > > +}
> > > +
> > > +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
> > > +	.pad	= &mali_c55_resizer_pad_ops,
> > > +};
> > > +
> > > +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
> > > +				   struct v4l2_subdev_state *state)
> > > +{
> > > +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> > > +						    sd);
> > > +	struct v4l2_subdev_krouting routing = { };
> > > +	struct v4l2_subdev_route *routes;
> > > +	unsigned int i;
> > > +	int ret;
> > > +
> > > +	routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
> > > +	if (!routes)
> > > +		return -ENOMEM;
> > > +
> > > +	for (i = 0; i < rzr->num_routes; ++i) {
> > > +		struct v4l2_subdev_route *route = &routes[i];
> > > +
> > > +		route->sink_pad = i
> > > +				? MALI_C55_RZR_SINK_BYPASS_PAD
> > > +				: MALI_C55_RZR_SINK_PAD;
> > > +		route->source_pad = MALI_C55_RZR_SOURCE_PAD;
> > > +		if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
> > > +			route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
> > > +	}
> > > +
> > > +	routing.num_routes = rzr->num_routes;
> > > +	routing.routes = routes;
> > > +
> > > +	ret = __mali_c55_rzr_set_routing(sd, state, &routing);
> > > +	kfree(routes);
> > > +
> > > +	return ret;
>
> I think this could be simplified.
>
> 	struct v4l2_subdev_route routes[2] = {
> 		{
> 			.sink_pad = MALI_C55_RZR_SINK_PAD,
> 			.source_pad = MALI_C55_RZR_SOURCE_PAD,
> 			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> 		}, {
> 			.sink_pad = MALI_C55_RZR_SINK_BYPASS_PAD,
> 			.source_pad = MALI_C55_RZR_SOURCE_PAD,
> 		},
> 	};
> 	struct v4l2_subdev_krouting routing = {
> 		.num_routes = rzr->num_routes,
> 		.routes = routes,
> 	};
>
> 	return __mali_c55_rzr_set_routing(sd, state, &routing);
>
> > > +}
> > > +
> > > +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
> > > +	.init_state = mali_c55_rzr_init_state,
> > > +};
> > > +
> > > +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
> > > +						  unsigned int index)
> > > +{
> > > +	const unsigned int scaler_filt_coefmem_addrs[][2] = {
> > > +		[MALI_C55_RZR_FR] = {
> > > +			0x034A8, /* hfilt */
> > > +			0x044A8  /* vfilt */
> >
> > Lowercase hex constants.
>
> And addresses belong to the mali-c55-registers.h file.
>
> > > +		},
> > > +		[MALI_C55_RZR_DS] = {
> > > +			0x014A8, /* hfilt */
> > > +			0x024A8  /* vfilt */
> > > +		},
> > > +	};
> > > +	unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
> > > +	unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
> > > +	unsigned int i, j;
> > > +
> > > +	for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
> > > +		for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
> > > +			mali_c55_write(mali_c55, haddr,
> > > +				mali_c55_scaler_h_filter_coefficients[i][j]);
> > > +			mali_c55_write(mali_c55, vaddr,
> > > +				mali_c55_scaler_v_filter_coefficients[i][j]);
> > > +
> > > +			haddr += sizeof(u32);
> > > +			vaddr += sizeof(u32);
> > > +		}
> > > +	}
>
> How about memcpy_toio() ? I suppose this function isn't
> performance sensitive, so maybe usage of mali_c55_write() is better from
> a consistency point of view.
>
> > > +}
> > > +
> > > +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
> > > +{
> > > +	unsigned int i;
> > > +	int ret;
> > > +
> > > +	for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
>
> Moving the inner content to a separate mali_c55_register_resizer()
> function would increase readability I think, and remove usage of gotos.
> I would probably do the same for unregistration too, for consistency.
>
> > > +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> > > +		struct v4l2_subdev *sd = &rzr->sd;
> > > +		unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
> > > +
> > > +		rzr->id = i;
> > > +		rzr->streaming = false;
> > > +
> > > +		if (rzr->id == MALI_C55_RZR_FR)
> > > +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
> > > +		else
> > > +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
> > > +
> > > +		mali_c55_resizer_program_coefficients(mali_c55, i);
>
> Should this be done at stream start, given that power may be cut off
> between streaming sessions ?
>
> > > +
> > > +		v4l2_subdev_init(sd, &mali_c55_resizer_ops);
> > > +		sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
> > > +			     | V4L2_SUBDEV_FL_STREAMS;
> > > +		sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
> > > +		sd->internal_ops = &mali_c55_resizer_internal_ops;
> > > +		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
>
> 		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s resizer %s",
>
> and drop the "resizer " prefix from mali_c55_resizer_names. You can also
> make mali_c55_resizer_names a local static const variable.
>
> > > +			 MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
> > > +
> > > +		rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
> > > +		rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
> > > +
> > > +		/* Only the FR pipe has a bypass pad. */
> > > +		if (rzr->id == MALI_C55_RZR_FR) {
> > > +			rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
> > > +							MEDIA_PAD_FL_SINK;
> > > +			rzr->num_routes = 2;
> > > +		} else {
> > > +			num_pads -= 1;
> > > +			rzr->num_routes = 1;
> > > +		}
> > > +
> > > +		ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
> > > +		if (ret)
> > > +			return ret;
> > > +
> > > +		ret = v4l2_subdev_init_finalize(sd);
> > > +		if (ret)
> > > +			goto err_cleanup;
> > > +
> > > +		ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> > > +		if (ret)
> > > +			goto err_cleanup;
> > > +
> > > +		rzr->mali_c55 = mali_c55;
> > > +	}
> > > +
> > > +	return 0;
> > > +
> > > +err_cleanup:
> > > +	for (; i >= 0; --i) {
> > > +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> > > +		struct v4l2_subdev *sd = &rzr->sd;
> > > +
> > > +		v4l2_subdev_cleanup(sd);
> > > +		media_entity_cleanup(&sd->entity);
> > > +	}
> > > +
> > > +	return ret;
> > > +}
> > > +
> > > +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
> > > +{
> > > +	unsigned int i;
> > > +
> > > +	for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
> > > +		struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
> > > +
> > > +		if (!resizer->mali_c55)
> > > +			continue;
> > > +
> > > +		v4l2_device_unregister_subdev(&resizer->sd);
> > > +		v4l2_subdev_cleanup(&resizer->sd);
> > > +		media_entity_cleanup(&resizer->sd.entity);
> > > +	}
> > > +}

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Laurent Pinchart]

On Thu, Jun 06, 2024 at 02:47:08PM +0200, Jacopo Mondi wrote:
> Hi Laurent
> 
> On Fri, May 31, 2024 at 12:43:48AM GMT, Laurent Pinchart wrote:
> > And now the second part of the review, addressing mali-c55-capture.c and
> > mali-c55-resizer.c. I've reviewed the code from the bottom up, so some
> > messages may be repeated in an order that seems weird. Sorry about that.
> 
> [snip]
> 
> A few replies/questions on the resizer module
> 
> > > > +
> > > > +#endif /* _MALI_C55_RESIZER_COEFS_H */
> > > > diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> > > > new file mode 100644
> > > > index 000000000000..0a5a2969d3ce
> > > > --- /dev/null
> > > > +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> > > > @@ -0,0 +1,779 @@
> > > > +// SPDX-License-Identifier: GPL-2.0
> > > > +/*
> > > > + * ARM Mali-C55 ISP Driver - Image signal processor
> > > > + *
> > > > + * Copyright (C) 2024 Ideas on Board Oy
> > > > + */
> > > > +
> > > > +#include <linux/math.h>
> > > > +#include <linux/minmax.h>
> > > > +
> > > > +#include <media/media-entity.h>
> > > > +#include <media/v4l2-subdev.h>
> > > > +
> > > > +#include "mali-c55-common.h"
> > > > +#include "mali-c55-registers.h"
> > > > +#include "mali-c55-resizer-coefs.h"
> > > > +
> > > > +/* Scaling factor in Q4.20 format. */
> > > > +#define MALI_C55_RZR_SCALER_FACTOR	(1U << 20)
> > > > +
> > > > +static const u32 rzr_non_bypass_src_fmts[] = {
> > > > +	MEDIA_BUS_FMT_RGB121212_1X36,
> > > > +	MEDIA_BUS_FMT_YUV10_1X30
> > > > +};
> > > > +
> > > > +static const char * const mali_c55_resizer_names[] = {
> > > > +	[MALI_C55_RZR_FR] = "resizer fr",
> > > > +	[MALI_C55_RZR_DS] = "resizer ds",
> > > > +};
> > > > +
> > > > +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
> > > > +				     struct v4l2_subdev_state *state)
> > > > +{
> > > > +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
> > > > +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> > > > +	struct v4l2_mbus_framefmt *fmt;
> > > > +	struct v4l2_rect *crop;
> >
> > const
> >
> > > > +
> > > > +	/* Verify if crop should be enabled. */
> > > > +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
> > > > +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> > > > +
> > > > +	if (fmt->width == crop->width && fmt->height == crop->height)
> > > > +		return MALI_C55_BYPASS_CROP;
> > > > +
> > > > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
> > > > +		       crop->left);
> > > > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
> > > > +		       crop->top);
> > > > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
> > > > +		       crop->width);
> > > > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
> > > > +		       crop->height);
> > > > +
> > > > +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
> > > > +		       MALI_C55_CROP_ENABLE);
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
> > > > +					struct v4l2_subdev_state *state)
> > > > +{
> > > > +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
> > > > +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> > > > +	struct v4l2_rect *crop, *scale;
> >
> > const
> >
> > Once "[PATCH v4 0/3] media: v4l2-subdev: Support const-awareness in
> > state accessors" gets merged, the state argument to this function can be
> > made const too. Same for other functions, as applicable.
> >
> > > > +	unsigned int h_bank, v_bank;
> > > > +	u64 h_scale, v_scale;
> > > > +
> > > > +	/* Verify if scaling should be enabled. */
> > > > +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> > > > +	scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
> > > > +
> > > > +	if (crop->width == scale->width && crop->height == scale->height)
> > > > +		return MALI_C55_BYPASS_SCALER;
> > > > +
> > > > +	/* Program the V/H scaling factor in Q4.20 format. */
> > > > +	h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
> > > > +	v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
> > > > +
> > > > +	do_div(h_scale, scale->width);
> > > > +	do_div(v_scale, scale->height);
> > > > +
> > > > +	mali_c55_write(mali_c55,
> > > > +		       MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
> > > > +		       crop->width);
> > > > +	mali_c55_write(mali_c55,
> > > > +		       MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
> > > > +		       crop->height);
> > > > +
> > > > +	mali_c55_write(mali_c55,
> > > > +		       MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
> > > > +		       scale->width);
> > > > +	mali_c55_write(mali_c55,
> > > > +		       MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
> > > > +		       scale->height);
> > > > +
> > > > +	mali_c55_write(mali_c55,
> > > > +		       MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
> > > > +		       h_scale);
> > > > +	mali_c55_write(mali_c55,
> > > > +		       MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
> > > > +		       v_scale);
> > > > +
> > > > +	h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
> > > > +					     scale->width);
> > > > +	mali_c55_write(mali_c55,
> > > > +		       MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
> > > > +		       h_bank);
> > > > +
> > > > +	v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
> > > > +					     scale->height);
> > > > +	mali_c55_write(mali_c55,
> > > > +		       MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
> > > > +		       v_bank);
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
> > > > +				 struct v4l2_subdev_state *state)
> > > > +{
> > > > +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> > > > +	u32 bypass = 0;
> > > > +
> > > > +	/* Verify if cropping and scaling should be enabled. */
> > > > +	bypass |= mali_c55_rzr_program_crop(rzr, state);
> > > > +	bypass |= mali_c55_rzr_program_resizer(rzr, state);
> > > > +
> > > > +	mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
> > > > +			     MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
> > > > +			     MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
> > > > +			     bypass);
> > > > +}
> > > > +
> > > > +/*
> > > > + * Inspect the routing table to know which of the two (mutually exclusive)
> > > > + * routes is enabled and return the sink pad id of the active route.
> > > > + */
> > > > +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
> > > > +{
> > > > +	struct v4l2_subdev_krouting *routing = &state->routing;
> > > > +	struct v4l2_subdev_route *route;
> > > > +
> > > > +	/* A single route is enabled at a time. */
> > > > +	for_each_active_route(routing, route)
> > > > +		return route->sink_pad;
> > > > +
> > > > +	return MALI_C55_RZR_SINK_PAD;
> > > > +}
> > > > +
> > > > +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
> > > > +{
> > > > +	u32 corrected_code = 0;
> > > > +
> > > > +	/*
> > > > +	 * The ISP takes input in a 20-bit format, but can only output 16-bit
> > > > +	 * RAW bayer data (with the 4 least significant bits from the input
> > > > +	 * being lost). Return the 16-bit version of the 20-bit input formats.
> > > > +	 */
> > > > +	switch (mbus_code) {
> > > > +	case MEDIA_BUS_FMT_SBGGR20_1X20:
> > > > +		corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
> > > > +		break;
> > > > +	case MEDIA_BUS_FMT_SGBRG20_1X20:
> > > > +		corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
> > > > +		break;
> > > > +	case MEDIA_BUS_FMT_SGRBG20_1X20:
> > > > +		corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
> > > > +		break;
> > > > +	case MEDIA_BUS_FMT_SRGGB20_1X20:
> > > > +		corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
> > > > +		break;
> >
> > Would it make sense to add the shifted code to mali_c55_isp_fmt ?
> >
> > > > +	}
> > > > +
> > > > +	return corrected_code;
> > > > +}
> > > > +
> > > > +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> > > > +				      struct v4l2_subdev_state *state,
> > > > +				      struct v4l2_subdev_krouting *routing)
> >
> > I think the last argument can be const.
> 
> If I have to adjust the routing table instead of refusing it, it can't

Indeed, my bad. I wrote that based on the current implementation and
forgot to modify the comment after.

> > > > +{
> > > > +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> > > > +						    sd);
> >
> > A to_mali_c55_resizer() static inline function would be useful. Same for
> > other components, where applicable.
> >
> > > > +	unsigned int active_sink = UINT_MAX;
> > > > +	struct v4l2_mbus_framefmt *src_fmt;
> > > > +	struct v4l2_rect *crop, *compose;
> > > > +	struct v4l2_subdev_route *route;
> > > > +	unsigned int active_routes = 0;
> > > > +	struct v4l2_mbus_framefmt *fmt;
> > > > +	int ret;
> > > > +
> > > > +	ret = v4l2_subdev_routing_validate(sd, routing, 0);
> > > > +	if (ret)
> > > > +		return ret;
> > > > +
> > > > +	/* Only a single route can be enabled at a time. */
> > > > +	for_each_active_route(routing, route) {
> > > > +		if (++active_routes > 1) {
> > > > +			dev_err(rzr->mali_c55->dev,
> > > > +				"Only one route can be active");
> >
> > No kernel log message with a level higher than dev_dbg() from
> > user-controlled paths please, here and where applicable. This is to
> > avoid giving applications an easy way to flood the kernel log.
> >
> > > > +			return -EINVAL;
> > > > +		}
> > > > +
> > > > +		active_sink = route->sink_pad;
> > > > +	}
> > > > +	if (active_sink == UINT_MAX) {
> > > > +		dev_err(rzr->mali_c55->dev, "One route has to be active");
> > > > +		return -EINVAL;
> > > > +	}
> >
> > The recommended handling of invalid routing is to adjust the routing
> > table, not to return errors.
> 
> How should I adjust it ? The error here is due to the fact multiple
> routes are set as active, which one should I make active ? the first
> one ? Should I go and reset the flags in the subdev_route for the one
> that has to be made non-active ?

The same way you would adjust an invalid format, you can pick the route
you consider should be the default.

I'd like Sakari's and Tomi's opinions on this, as it's a new API and the
behaviour is still a bit in flux.

> > > > +
> > > > +	ret = v4l2_subdev_set_routing(sd, state, routing);
> > > > +	if (ret) {
> > > > +		dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
> > > > +		return ret;
> > > > +	}
> > > > +
> > > > +	fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
> > > > +	crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
> > > > +	compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
> > > > +
> > > > +	fmt->width = MALI_C55_DEFAULT_WIDTH;
> > > > +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
> > > > +	fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >
> > There are other colorspace-related fields.
> >
> > > > +	fmt->field = V4L2_FIELD_NONE;
> >
> > I wonder if we should really update the sink pad format, or just
> > propagate it. If we update it, I think it should be set to defaults on
> > both sink pads, not just the active sink pad.
> 
> If only one route can be active, there will only be one state.stream_config
> entry for the active sink, not for the other one (see
> v4l2_subdev_init_stream_configs()), this mean I can't reset both sink
> formats ?

I think you can change the format on pads not included in active routes.
Again, new API, so thoughts are appreciated :-)

> > > > +
> > > > +	if (active_sink == MALI_C55_RZR_SINK_PAD) {
> > > > +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > > > +
> > > > +		crop->left = crop->top = 0;
> >
> > 		crop->left = 0;
> > 		crop->top = 0;
> >
> > > > +		crop->width = MALI_C55_DEFAULT_WIDTH;
> > > > +		crop->height = MALI_C55_DEFAULT_HEIGHT;
> > > > +
> > > > +		*compose = *crop;
> > > > +	} else {
> > > > +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> > > > +	}
> > > > +
> > > > +	/* Propagate the format to the source pad */
> > > > +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
> > > > +					       0);
> > > > +	*src_fmt = *fmt;
> > > > +
> > > > +	/* In the event this is the bypass pad the mbus code needs correcting */
> > > > +	if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
> > > > +		src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
> > > > +				       struct v4l2_subdev_state *state,
> > > > +				       struct v4l2_subdev_mbus_code_enum *code)
> > > > +{
> > > > +	struct v4l2_mbus_framefmt *sink_fmt;
> > > > +	const struct mali_c55_isp_fmt *fmt;
> > > > +	unsigned int index = 0;
> > > > +	u32 sink_pad;
> > > > +
> > > > +	switch (code->pad) {
> > > > +	case MALI_C55_RZR_SINK_PAD:
> > > > +		if (code->index)
> > > > +			return -EINVAL;
> > > > +
> > > > +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > > > +
> > > > +		return 0;
> > > > +	case MALI_C55_RZR_SOURCE_PAD:
> > > > +		sink_pad = mali_c55_rzr_get_active_sink(state);
> > > > +		sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> > > > +
> > > > +		/*
> > > > +		 * If the active route is from the Bypass sink pad, then the
> > > > +		 * source pad is a simple passthrough of the sink format,
> > > > +		 * downshifted to 16-bits.
> > > > +		 */
> > > > +
> > > > +		if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> > > > +			if (code->index)
> > > > +				return -EINVAL;
> > > > +
> > > > +			code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> > > > +			if (!code->code)
> > > > +				return -EINVAL;
> > > > +
> > > > +			return 0;
> > > > +		}
> > > > +
> > > > +		/*
> > > > +		 * If the active route is from the non-bypass sink then we can
> > > > +		 * select either RGB or conversion to YUV.
> > > > +		 */
> > > > +
> > > > +		if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
> > > > +			return -EINVAL;
> > > > +
> > > > +		code->code = rzr_non_bypass_src_fmts[code->index];
> > > > +
> > > > +		return 0;
> > > > +	case MALI_C55_RZR_SINK_BYPASS_PAD:
> > > > +		for_each_mali_isp_fmt(fmt) {
> > > > +			if (index++ == code->index) {
> > > > +				code->code = fmt->code;
> > > > +				return 0;
> > > > +			}
> > > > +		}
> > > > +
> > > > +		break;
> > > > +	}
> > > > +
> > > > +	return -EINVAL;
> > > > +}
> > > > +
> > > > +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
> > > > +					struct v4l2_subdev_state *state,
> > > > +					struct v4l2_subdev_frame_size_enum *fse)
> > > > +{
> > > > +	if (fse->index)
> > > > +		return -EINVAL;
> > > > +
> > > > +	fse->max_width = MALI_C55_MAX_WIDTH;
> > > > +	fse->max_height = MALI_C55_MAX_HEIGHT;
> > > > +	fse->min_width = MALI_C55_MIN_WIDTH;
> > > > +	fse->min_height = MALI_C55_MIN_HEIGHT;
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
> > > > +				     struct v4l2_subdev_state *state,
> > > > +				     struct v4l2_subdev_format *format)
> > > > +{
> > > > +	struct v4l2_mbus_framefmt *fmt = &format->format;
> > > > +	struct v4l2_rect *rect;
> > > > +	unsigned int sink_pad;
> > > > +
> > > > +	/*
> > > > +	 * Clamp to min/max and then reset crop and compose rectangles to the
> > > > +	 * newly applied size.
> > > > +	 */
> > > > +	clamp_t(unsigned int, fmt->width,
> > > > +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> 
> also, clamp_t doens't clamp in place
> 
>         fmt->width = clamp_t...

Correct. I've commented on that in other places.

> > > > +	clamp_t(unsigned int, fmt->height,
> > > > +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >
> > Please check comments for other components related to the colorspace
> > fields, to decide how to handle them here.
> >
> > > > +
> > > > +	sink_pad = mali_c55_rzr_get_active_sink(state);
> > > > +	if (sink_pad == MALI_C55_RZR_SINK_PAD) {
> >
> > The selection here should depend on format->pad, not the active sink
> > pad.
> >
> > > > +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > > > +
> > > > +		rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> > > > +		rect->left = 0;
> > > > +		rect->top = 0;
> > > > +		rect->width = fmt->width;
> > > > +		rect->height = fmt->height;
> > > > +
> > > > +		rect = v4l2_subdev_state_get_compose(state,
> > > > +						     MALI_C55_RZR_SINK_PAD);
> > > > +		rect->left = 0;
> > > > +		rect->top = 0;
> > > > +		rect->width = fmt->width;
> > > > +		rect->height = fmt->height;
> > > > +	} else {
> > > > +		/*
> > > > +		 * Make sure the media bus code is one of the supported
> > > > +		 * ISP input media bus codes.
> > > > +		 */
> > > > +		if (!mali_c55_isp_is_format_supported(fmt->code))
> > > > +			fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
> 
> And DEFAULT_MEDIA_BUS_FMT is not one of the supported input media bus
> codes

That's not good :-)

> > > > +	}
> > > > +
> > > > +	*v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
> > > > +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
> >
> > Propagation to the source pad, however, should depend on the active
> > route. If format->pad is routed to the source pad, you should propagate,
> > otherwise, you shouldn't.
> >
> > > > +
> > > > +	return 0;
> 
> I ended up with
> 
> static int mali_c55_rsz_set_sink_fmt(struct v4l2_subdev *sd,
> 				     struct v4l2_subdev_state *state,
> 				     struct v4l2_subdev_format *format)
> {
> 	struct v4l2_mbus_framefmt *fmt = &format->format;
> 	unsigned int active_sink;
> 	struct v4l2_rect *rect;
> 
> 	/*
> 	 * Clamp to min/max and then reset crop and compose rectangles to the
> 	 * newly applied size.
> 	 */
> 	fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> 			     MALI_C55_MAX_WIDTH);
> 	fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> 			      MALI_C55_MAX_HEIGHT);
> 
> 	rect = v4l2_subdev_state_get_crop(state, format->pad);
> 	rect->left = 0;
> 	rect->top = 0;
> 	rect->width = fmt->width;
> 	rect->height = fmt->height;
> 
> 	rect = v4l2_subdev_state_get_compose(state, format->pad);
> 	rect->left = 0;
> 	rect->top = 0;
> 	rect->width = fmt->width;
> 	rect->height = fmt->height;
> 
> 	if (format->pad == MALI_C55_RSZ_SINK_BYPASS_PAD) {
> 		/*
> 		 * Make sure the media bus code is one of the supported
> 		 * ISP input media bus codes. Default it to SRGGB otherwise.
> 		 */
> 		if (!mali_c55_isp_is_format_supported(fmt->code))
> 			fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> 	} else {
> 		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> 	}

I think you need to handle colour spaces.

> 
> 	*v4l2_subdev_state_get_format(state, format->pad, 0) = *fmt;

You can drop the last argument to the function, it's optional and
defaults to 0. The recommendation is to not include it when the device
doesn't support streams. Same elsewhere in the file.

> 
> 	/* If format->pad is routed to the source pad, propagate the format. */
> 	active_sink = mali_c55_rsz_get_active_sink(state);
> 	if (active_sink == format->pad) {

You can also

	if (active_sink != format->pad)
		return 0;

if you want to reduce indentation. Up to you.

> 
> 		/* If the bypass route is used, downshift the code to 16bpp. */
> 		if (active_sink == MALI_C55_RSZ_SINK_BYPASS_PAD)
> 			fmt->code = mali_c55_rsz_shift_mbus_code(fmt->code);
> 
> 		*v4l2_subdev_state_get_format(state,
> 					      MALI_C55_RSZ_SOURCE_PAD, 0) = *fmt;
> 	}
> 
> 	return 0;
> }
> 
> > > > +}
> > > > +
> > > > +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> > > > +				       struct v4l2_subdev_state *state,
> > > > +				       struct v4l2_subdev_format *format)
> > > > +{
> > > > +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> > > > +						    sd);
> > > > +	struct v4l2_mbus_framefmt *fmt = &format->format;
> > > > +	struct v4l2_mbus_framefmt *sink_fmt;
> > > > +	struct v4l2_rect *crop, *compose;
> > > > +	unsigned int sink_pad;
> > > > +	unsigned int i;
> > > > +
> > > > +	sink_pad = mali_c55_rzr_get_active_sink(state);
> > > > +	sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> > > > +	crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
> > > > +	compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
> > > > +
> > > > +	/* FR Bypass pipe. */
> > > > +
> > > > +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> > > > +		/*
> > > > +		 * Format on the source pad is the same as the one on the
> > > > +		 * sink pad, downshifted to 16-bits.
> > > > +		 */
> > > > +		fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> > > > +		if (!fmt->code)
> > > > +			return -EINVAL;
> > > > +
> > > > +		/* RAW bypass disables scaling and cropping. */
> > > > +		crop->top = compose->top = 0;
> > > > +		crop->left = compose->left = 0;
> > > > +		fmt->width = crop->width = compose->width = sink_fmt->width;
> > > > +		fmt->height = crop->height = compose->height = sink_fmt->height;
> >
> > I don't think this is right. This function sets the format on the source
> > pad. Subdevs should propagate formats from the sink to the source, not
> > the other way around.
> >
> > The only parameter that can be modified on the source pad (as far as I
> > understand) is the media bus code. In the bypass path, I understand it's
> > fixed, while in the other path, you can select between RGB and YUV. I
> > think the following code is what you need to implement this function.
> >
> > static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> > 				       struct v4l2_subdev_state *state,
> > 				       struct v4l2_subdev_format *format)
> > {
> > 	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> > 						    sd);
> > 	struct v4l2_mbus_framefmt *fmt;
> >
> > 	fmt = v4l2_subdev_state_get_format(state, format->pad);
> >
> > 	/* In the non-bypass path the output format can be selected. */
> > 	if (mali_c55_rzr_get_active_sink(state) == MALI_C55_RZR_SINK_PAD) {
> > 		unsigned int i;
> >
> > 		fmt->code = format->format.code;
> >
> > 		for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> > 			if (fmt->code == rzr_non_bypass_src_fmts[i])
> > 				break;
> > 		}
> >
> > 		if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts))
> > 			fmt->code = rzr_non_bypass_src_fmts[0];
> > 	}
> >
> > 	format->format = *fmt;
> >
> > 	return 0;
> > }
> 
> Almost. Your proposal doesn't adjust format->format.width/height

It does, on the last line:

 	format->format = *fmt;

*fmt is the format taken from the subdev state, so it has been
initialized by .init_cfg() and updated by format propagation when
setting a format on the sink pad, or when setting a selection rectangle.

I like this structure better, where we take the format from the state,
update the fields that can be updated, and then copy the result to the
function argument to provide it to userspace. The alternative below
updates the fields from the userspace-provided format instead, and then
copies it to the state. That's more error-prone, as you need to
explicitly update everything that can't be changed by userspace, while
in the code above you update the fields that can be changed. A white
list approach is safer than a black list.

> I think the following is more appropriate
> 
> static int mali_c55_rsz_set_source_fmt(struct v4l2_subdev *sd,
> 				       struct v4l2_subdev_state *state,
> 				       struct v4l2_subdev_format *format)
> {
> 	struct v4l2_mbus_framefmt *fmt = &format->format;
> 	struct v4l2_mbus_framefmt *sink_fmt;
> 	struct v4l2_rect *sink_compose;
> 	unsigned int active_sink;
> 
> 	active_sink = mali_c55_rsz_get_active_sink(state);
> 	sink_fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
> 	sink_compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
> 
> 	/*
> 	 * The source pad format sizes come directly from the active sink pad
> 	 * compose rectangle.
> 	 */
> 	fmt->width = sink_compose->width;
> 	fmt->height = sink_compose->height;
> 
> 	if (active_sink == MALI_C55_RSZ_SINK_PAD) {
> 		/*
> 		 * Regular processing pipe: RGB121212 can be color-space
> 		 * converted to YUV101010.
> 		 */
> 		unsigned int i;
> 
> 		for (i = 0; i < ARRAY_SIZE(rsz_non_bypass_src_fmts); i++) {
> 			if (fmt->code == rsz_non_bypass_src_fmts[i])
> 				break;
> 		}
> 
> 		if (i == ARRAY_SIZE(rsz_non_bypass_src_fmts))
> 			fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> 	} else {
> 		/*
> 		 * Bypass pipe: the source format is the same as the bypass
> 		 * sink pad downshifted to 16bpp.
> 		 */
> 		fmt->code = mali_c55_rsz_shift_mbus_code(sink_fmt->code);
> 	}
> 
> 	*v4l2_subdev_state_get_format(state, MALI_C55_RSZ_SOURCE_PAD) = *fmt;
> 
> 	return 0;
> }
> 
> I'll handle the colorspace fields as well

Ah, yes :-)

> > > > +
> > > > +		*v4l2_subdev_state_get_format(state,
> > > > +					      MALI_C55_RZR_SOURCE_PAD) = *fmt;
> > > > +
> > > > +		return 0;
> > > > +	}
> > > > +
> > > > +	/* Regular processing pipe. */
> > > > +
> > > > +	for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> > > > +		if (fmt->code == rzr_non_bypass_src_fmts[i])
> > > > +			break;
> > > > +	}
> > > > +
> > > > +	if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
> > > > +		dev_dbg(rzr->mali_c55->dev,
> > > > +			"Unsupported mbus code 0x%x: using default\n",
> > > > +			fmt->code);
> >
> > I think you can drop this message.
> >
> > > > +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> > > > +	}
> > > > +
> > > > +	/*
> > > > +	 * The source pad format size comes directly from the sink pad
> > > > +	 * compose rectangle.
> > > > +	 */
> > > > +	fmt->width = compose->width;
> > > > +	fmt->height = compose->height;
> > > > +
> > > > +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
> > > > +				struct v4l2_subdev_state *state,
> > > > +				struct v4l2_subdev_format *format)
> > > > +{
> > > > +	/*
> > > > +	 * On sink pads fmt is either fixed for the 'regular' processing
> > > > +	 * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
> > > > +	 * pad.
> > > > +	 *
> > > > +	 * On source pad sizes are the result of crop+compose on the sink
> > > > +	 * pad sizes, while the format depends on the active route.
> > > > +	 */
> > > > +
> > > > +	if (format->pad != MALI_C55_RZR_SOURCE_PAD)
> > > > +		return mali_c55_rzr_set_sink_fmt(sd, state, format);
> > > > +
> > > > +	return mali_c55_rzr_set_source_fmt(sd, state, format);
> >
> > Nitpicking,
> >
> > 	if (format->pad == MALI_C55_RZR_SOURCE_PAD)
> > 		return mali_c55_rzr_set_source_fmt(sd, state, format);
> >
> > 	return mali_c55_rzr_set_sink_fmt(sd, state, format);
> >
> > to match SOURCE_PAD and source_fmt.
> >
> 
> Done at the expense a bit more verbose check
> 
> 	if (format->pad == MALI_C55_RSZ_SINK_PAD ||
> 	    format->pad == MALI_C55_RSZ_SINK_BYPASS_PAD)
> 		return mali_c55_rsz_set_sink_fmt(sd, state, format);
> 
> > > > +}
> > > > +
> > > > +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
> > > > +				      struct v4l2_subdev_state *state,
> > > > +				      struct v4l2_subdev_selection *sel)
> > > > +{
> > > > +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
> > > > +		return -EINVAL;
> > > > +
> > > > +	if (sel->target != V4L2_SEL_TGT_CROP &&
> > > > +	    sel->target != V4L2_SEL_TGT_COMPOSE)
> > > > +		return -EINVAL;
> > > > +
> > > > +	sel->r = sel->target == V4L2_SEL_TGT_CROP
> > > > +	       ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
> > > > +	       : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
> > > > +				      struct v4l2_subdev_state *state,
> > > > +				      struct v4l2_subdev_selection *sel)
> > > > +{
> > > > +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> > > > +						    sd);
> > > > +	struct v4l2_mbus_framefmt *source_fmt;
> > > > +	struct v4l2_mbus_framefmt *sink_fmt;
> > > > +	struct v4l2_rect *crop, *compose;
> > > > +
> > > > +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
> > > > +		return -EINVAL;
> > > > +
> > > > +	if (sel->target != V4L2_SEL_TGT_CROP &&
> > > > +	    sel->target != V4L2_SEL_TGT_COMPOSE)
> > > > +		return -EINVAL;
> > > > +
> > > > +	source_fmt = v4l2_subdev_state_get_format(state,
> > > > +						  MALI_C55_RZR_SOURCE_PAD);
> > > > +	sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
> > > > +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> > > > +	compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> > > > +
> > > > +	/* RAW bypass disables crop/scaling. */
> > > > +	if (mali_c55_format_is_raw(source_fmt->code)) {
> > > > +		crop->top = compose->top = 0;
> > > > +		crop->left = compose->left = 0;
> > > > +		crop->width = compose->width = sink_fmt->width;
> > > > +		crop->height = compose->height = sink_fmt->height;
> > > > +
> > > > +		sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> > > > +
> > > > +		return 0;
> > > > +	}
> > > > +
> > > > +	/* During streaming, it is allowed to only change the crop rectangle. */
> > > > +	if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
> > > > +		return -EINVAL;
> > > > +
> > > > +	 /*
> > > > +	  * Update the desired target and then clamp the crop rectangle to the
> > > > +	  * sink format sizes and the compose size to the crop sizes.
> > > > +	  */
> > > > +	if (sel->target == V4L2_SEL_TGT_CROP)
> > > > +		*crop = sel->r;
> > > > +	else
> > > > +		*compose = sel->r;
> > > > +
> > > > +	clamp_t(unsigned int, crop->left, 0,  sink_fmt->width);
> > > > +	clamp_t(unsigned int, crop->top, 0,  sink_fmt->height);
> > > > +	clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
> > > > +		sink_fmt->width - crop->left);
> > > > +	clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
> > > > +		sink_fmt->height - crop->top);
> > > > +
> > > > +	if (rzr->streaming) {
> > > > +		/*
> > > > +		 * Apply at runtime a crop rectangle on the resizer's sink only
> > > > +		 * if it doesn't require re-programming the scaler output sizes
> > > > +		 * as it would require changing the output buffer sizes as well.
> > > > +		 */
> > > > +		if (sel->r.width < compose->width ||
> > > > +		    sel->r.height < compose->height)
> > > > +			return -EINVAL;
> > > > +
> > > > +		*crop = sel->r;
> > > > +		mali_c55_rzr_program(rzr, state);
> > > > +
> > > > +		return 0;
> > > > +	}
> > > > +
> > > > +	compose->left = 0;
> > > > +	compose->top = 0;
> > > > +	clamp_t(unsigned int, compose->left, 0,  sink_fmt->width);
> > > > +	clamp_t(unsigned int, compose->top, 0,  sink_fmt->height);
> > > > +	clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
> > > > +	clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
> > > > +
> > > > +	sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> > > > +				    struct v4l2_subdev_state *state,
> > > > +				    enum v4l2_subdev_format_whence which,
> > > > +				    struct v4l2_subdev_krouting *routing)
> > > > +{
> > > > +	if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
> > > > +	    media_entity_is_streaming(&sd->entity))
> > > > +		return -EBUSY;
> > > > +
> > > > +	return __mali_c55_rzr_set_routing(sd, state, routing);
> > > > +}
> > > > +
> > > > +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
> > > > +	.enum_mbus_code		= mali_c55_rzr_enum_mbus_code,
> > > > +	.enum_frame_size	= mali_c55_rzr_enum_frame_size,
> > > > +	.get_fmt		= v4l2_subdev_get_fmt,
> > > > +	.set_fmt		= mali_c55_rzr_set_fmt,
> > > > +	.get_selection		= mali_c55_rzr_get_selection,
> > > > +	.set_selection		= mali_c55_rzr_set_selection,
> > > > +	.set_routing		= mali_c55_rzr_set_routing,
> > > > +};
> > > > +
> > > > +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
> >
> > Could this be handled through the .enable_streams() and
> > .disable_streams() operations ? They ensure that the stream state stored
> > internal is correct. That may not matter much today, but I think it will
> > become increasingly important in the future for the V4L2 core.
> >
> > > > +{
> > > > +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> > > > +	struct v4l2_subdev *sd = &rzr->sd;
> > > > +	struct v4l2_subdev_state *state;
> > > > +	unsigned int sink_pad;
> > > > +
> > > > +	state = v4l2_subdev_lock_and_get_active_state(sd);
> > > > +
> > > > +	sink_pad = mali_c55_rzr_get_active_sink(state);
> > > > +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> > > > +		/* Bypass FR pipe processing if the bypass route is active. */
> > > > +		mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> > > > +				     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
> > > > +				     MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
> > > > +		goto unlock_state;
> > > > +	}
> > > > +
> > > > +	/* Disable bypass and use regular processing. */
> > > > +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> > > > +			     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
> > > > +	mali_c55_rzr_program(rzr, state);
> > > > +
> > > > +unlock_state:
> > > > +	rzr->streaming = true;
> >
> > And hopefully you'll be able to replace this with
> > v4l2_subdev_is_streaming(), introduced in "[PATCH v6 00/11] media:
> > subdev: Improve stream enable/disable machinery" (Sakari has sent a pull
> > request for v6.11 yesterday).
> >
> > > > +	v4l2_subdev_unlock_state(state);
> > > > +}
> > > > +
> > > > +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
> > > > +{
> > > > +	struct v4l2_subdev *sd = &rzr->sd;
> > > > +	struct v4l2_subdev_state *state;
> > > > +
> > > > +	state = v4l2_subdev_lock_and_get_active_state(sd);
> > > > +	rzr->streaming = false;
> > > > +	v4l2_subdev_unlock_state(state);
> > > > +}
> > > > +
> > > > +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
> > > > +	.pad	= &mali_c55_resizer_pad_ops,
> > > > +};
> > > > +
> > > > +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
> > > > +				   struct v4l2_subdev_state *state)
> > > > +{
> > > > +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> > > > +						    sd);
> > > > +	struct v4l2_subdev_krouting routing = { };
> > > > +	struct v4l2_subdev_route *routes;
> > > > +	unsigned int i;
> > > > +	int ret;
> > > > +
> > > > +	routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
> > > > +	if (!routes)
> > > > +		return -ENOMEM;
> > > > +
> > > > +	for (i = 0; i < rzr->num_routes; ++i) {
> > > > +		struct v4l2_subdev_route *route = &routes[i];
> > > > +
> > > > +		route->sink_pad = i
> > > > +				? MALI_C55_RZR_SINK_BYPASS_PAD
> > > > +				: MALI_C55_RZR_SINK_PAD;
> > > > +		route->source_pad = MALI_C55_RZR_SOURCE_PAD;
> > > > +		if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
> > > > +			route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
> > > > +	}
> > > > +
> > > > +	routing.num_routes = rzr->num_routes;
> > > > +	routing.routes = routes;
> > > > +
> > > > +	ret = __mali_c55_rzr_set_routing(sd, state, &routing);
> > > > +	kfree(routes);
> > > > +
> > > > +	return ret;
> >
> > I think this could be simplified.
> >
> > 	struct v4l2_subdev_route routes[2] = {
> > 		{
> > 			.sink_pad = MALI_C55_RZR_SINK_PAD,
> > 			.source_pad = MALI_C55_RZR_SOURCE_PAD,
> > 			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > 		}, {
> > 			.sink_pad = MALI_C55_RZR_SINK_BYPASS_PAD,
> > 			.source_pad = MALI_C55_RZR_SOURCE_PAD,
> > 		},
> > 	};
> > 	struct v4l2_subdev_krouting routing = {
> > 		.num_routes = rzr->num_routes,
> > 		.routes = routes,
> > 	};
> >
> > 	return __mali_c55_rzr_set_routing(sd, state, &routing);
> >
> > > > +}
> > > > +
> > > > +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
> > > > +	.init_state = mali_c55_rzr_init_state,
> > > > +};
> > > > +
> > > > +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
> > > > +						  unsigned int index)
> > > > +{
> > > > +	const unsigned int scaler_filt_coefmem_addrs[][2] = {
> > > > +		[MALI_C55_RZR_FR] = {
> > > > +			0x034A8, /* hfilt */
> > > > +			0x044A8  /* vfilt */
> > >
> > > Lowercase hex constants.
> >
> > And addresses belong to the mali-c55-registers.h file.
> >
> > > > +		},
> > > > +		[MALI_C55_RZR_DS] = {
> > > > +			0x014A8, /* hfilt */
> > > > +			0x024A8  /* vfilt */
> > > > +		},
> > > > +	};
> > > > +	unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
> > > > +	unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
> > > > +	unsigned int i, j;
> > > > +
> > > > +	for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
> > > > +		for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
> > > > +			mali_c55_write(mali_c55, haddr,
> > > > +				mali_c55_scaler_h_filter_coefficients[i][j]);
> > > > +			mali_c55_write(mali_c55, vaddr,
> > > > +				mali_c55_scaler_v_filter_coefficients[i][j]);
> > > > +
> > > > +			haddr += sizeof(u32);
> > > > +			vaddr += sizeof(u32);
> > > > +		}
> > > > +	}
> >
> > How about memcpy_toio() ? I suppose this function isn't
> > performance sensitive, so maybe usage of mali_c55_write() is better from
> > a consistency point of view.
> >
> > > > +}
> > > > +
> > > > +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
> > > > +{
> > > > +	unsigned int i;
> > > > +	int ret;
> > > > +
> > > > +	for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
> >
> > Moving the inner content to a separate mali_c55_register_resizer()
> > function would increase readability I think, and remove usage of gotos.
> > I would probably do the same for unregistration too, for consistency.
> >
> > > > +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> > > > +		struct v4l2_subdev *sd = &rzr->sd;
> > > > +		unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
> > > > +
> > > > +		rzr->id = i;
> > > > +		rzr->streaming = false;
> > > > +
> > > > +		if (rzr->id == MALI_C55_RZR_FR)
> > > > +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
> > > > +		else
> > > > +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
> > > > +
> > > > +		mali_c55_resizer_program_coefficients(mali_c55, i);
> >
> > Should this be done at stream start, given that power may be cut off
> > between streaming sessions ?
> >
> > > > +
> > > > +		v4l2_subdev_init(sd, &mali_c55_resizer_ops);
> > > > +		sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
> > > > +			     | V4L2_SUBDEV_FL_STREAMS;
> > > > +		sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
> > > > +		sd->internal_ops = &mali_c55_resizer_internal_ops;
> > > > +		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
> >
> > 		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s resizer %s",
> >
> > and drop the "resizer " prefix from mali_c55_resizer_names. You can also
> > make mali_c55_resizer_names a local static const variable.
> >
> > > > +			 MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
> > > > +
> > > > +		rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
> > > > +		rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
> > > > +
> > > > +		/* Only the FR pipe has a bypass pad. */
> > > > +		if (rzr->id == MALI_C55_RZR_FR) {
> > > > +			rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
> > > > +							MEDIA_PAD_FL_SINK;
> > > > +			rzr->num_routes = 2;
> > > > +		} else {
> > > > +			num_pads -= 1;
> > > > +			rzr->num_routes = 1;
> > > > +		}
> > > > +
> > > > +		ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
> > > > +		if (ret)
> > > > +			return ret;
> > > > +
> > > > +		ret = v4l2_subdev_init_finalize(sd);
> > > > +		if (ret)
> > > > +			goto err_cleanup;
> > > > +
> > > > +		ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> > > > +		if (ret)
> > > > +			goto err_cleanup;
> > > > +
> > > > +		rzr->mali_c55 = mali_c55;
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +
> > > > +err_cleanup:
> > > > +	for (; i >= 0; --i) {
> > > > +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> > > > +		struct v4l2_subdev *sd = &rzr->sd;
> > > > +
> > > > +		v4l2_subdev_cleanup(sd);
> > > > +		media_entity_cleanup(&sd->entity);
> > > > +	}
> > > > +
> > > > +	return ret;
> > > > +}
> > > > +
> > > > +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
> > > > +{
> > > > +	unsigned int i;
> > > > +
> > > > +	for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
> > > > +		struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
> > > > +
> > > > +		if (!resizer->mali_c55)
> > > > +			continue;
> > > > +
> > > > +		v4l2_device_unregister_subdev(&resizer->sd);
> > > > +		v4l2_subdev_cleanup(&resizer->sd);
> > > > +		media_entity_cleanup(&resizer->sd.entity);
> > > > +	}
> > > > +}

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Tomi Valkeinen]

On 06/06/2024 20:53, Laurent Pinchart wrote:
>>>>> +			return -EINVAL;
>>>>> +		}
>>>>> +
>>>>> +		active_sink = route->sink_pad;
>>>>> +	}
>>>>> +	if (active_sink == UINT_MAX) {
>>>>> +		dev_err(rzr->mali_c55->dev, "One route has to be active");
>>>>> +		return -EINVAL;
>>>>> +	}
>>> The recommended handling of invalid routing is to adjust the routing
>>> table, not to return errors.
>> How should I adjust it ? The error here is due to the fact multiple
>> routes are set as active, which one should I make active ? the first
>> one ? Should I go and reset the flags in the subdev_route for the one
>> that has to be made non-active ?
> The same way you would adjust an invalid format, you can pick the route
> you consider should be the default.
> 
> I'd like Sakari's and Tomi's opinions on this, as it's a new API and the
> behaviour is still a bit in flux.

Well... My opinion is that the driver adjusting the given config 
parameters (for any ioctl) is awful and should be deprecated. If the 
user asks for X, and the driver adjusts it and returns Y, then the user 
has two options: fail, because it didn't get X (after possibly laborious 
field by field checks), or shrug it's virtual shoulders and accept Y and 
hope that things still work even though it wanted X.

But maybe that was an answer to a question you didn't really ask =).

I think setting it to default routing in case of an error is as fine as 
any other "fix" for the routing. It won't work anyway.

But if the function sets default routing and returns 0 here, why would 
it return an error from v4l2_subdev_routing_validate()? Should it just 
set default routing in that case too? So should set_routing() ever 
return an error, if we can just set the default routing?

In the VIDIOC_SUBDEV_S_ROUTING doc we do list some cases where EINVAL or 
E2BIG is returned. But only a few, and I think 
v4l2_subdev_routing_validate() will return errors for many other cases too.

For what it's worth, the drivers I have written just return an error. 
It's simple for the driver and the user and works. If the consensus is 
that the drivers should instead set the default routing, or somehow 
mangle the given routing to an acceptable form, I can update those 
drivers accordingly.

But we probably need to update the docs too to be a bit more clear what 
VIDIOC_SUBDEV_S_ROUTING will do (although are the other ioctls any 
clearer?).

All that said, I think it's still a bit case-by-case. I don't think the 
drivers should always return an error if they get a routing table that's 
not 100% perfect. E.g. if a device supports two static routes, but the 
second one can be enabled or disabled, the driver should still accept a 
routing table from the user with only the first route present. Etc.

For the specific case in this patch... I'd prefer returning an error, or 
if that's not ok, set default routing.

  Tomi

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Sakari Ailus]

Moi,

On Thu, Jun 06, 2024 at 10:10:14PM +0300, Tomi Valkeinen wrote:
> On 06/06/2024 20:53, Laurent Pinchart wrote:
> > > > > > +			return -EINVAL;
> > > > > > +		}
> > > > > > +
> > > > > > +		active_sink = route->sink_pad;
> > > > > > +	}
> > > > > > +	if (active_sink == UINT_MAX) {
> > > > > > +		dev_err(rzr->mali_c55->dev, "One route has to be active");
> > > > > > +		return -EINVAL;
> > > > > > +	}
> > > > The recommended handling of invalid routing is to adjust the routing
> > > > table, not to return errors.
> > > How should I adjust it ? The error here is due to the fact multiple
> > > routes are set as active, which one should I make active ? the first
> > > one ? Should I go and reset the flags in the subdev_route for the one
> > > that has to be made non-active ?
> > The same way you would adjust an invalid format, you can pick the route
> > you consider should be the default.
> > 
> > I'd like Sakari's and Tomi's opinions on this, as it's a new API and the
> > behaviour is still a bit in flux.
> 
> Well... My opinion is that the driver adjusting the given config parameters
> (for any ioctl) is awful and should be deprecated. If the user asks for X,
> and the driver adjusts it and returns Y, then the user has two options:
> fail, because it didn't get X (after possibly laborious field by field
> checks), or shrug it's virtual shoulders and accept Y and hope that things
> still work even though it wanted X.

This is still often the only way to tell what the hardware can do as the
limitations in different cases (cropping and scaling for instance) can be
arbitrary. The other option is that the user space has to know the hardware
capabilities without them being available from the kernel.

There could be cases of IOCTLs where returning an error if what was
requested can't be performed exactly is workable in general, but then again
having consistency across IOCTL behaviour is very beneficial as well.

If you need something exactly, then I think you should check after the
IOCTL that this is what you also got, beyond the IOCTL succeeding.

> 
> But maybe that was an answer to a question you didn't really ask =).
> 
> I think setting it to default routing in case of an error is as fine as any
> other "fix" for the routing. It won't work anyway.
> 
> But if the function sets default routing and returns 0 here, why would it
> return an error from v4l2_subdev_routing_validate()? Should it just set
> default routing in that case too? So should set_routing() ever return an
> error, if we can just set the default routing?

S_ROUTING is a bit special as it deals with multiple routes and the user
space does have a way to add them incrementally.

Perhaps we should document better what the driver is expected to to correct
the routes?

I'd think routes may be added by the driver (as some of them cannot be
disabled for instance) but if a requested route cannot be created, that
should probably be an error.

I've copied my current (with all the pending patches) documentation here
<URL:https://www.retiisi.eu/~sailus/v4l2/tmp/streams-doc/userspace-api/media/v4l/dev-subdev.html#streams-multiplexed-media-pads-and-internal-routing>.

The text does not elaborate what exactly a driver could or should do, apart
from specifying the condition for EINVAL. I think we should specify this in
greater detail. My original thought wws the adjustment would be done by
adding static routes omitted by the caller, not trying to come up with e.g.
valid pad/stream pairs when user provided invalid ones.

Could this correction functionality be limited to returning static routes?

> 
> In the VIDIOC_SUBDEV_S_ROUTING doc we do list some cases where EINVAL or
> E2BIG is returned. But only a few, and I think
> v4l2_subdev_routing_validate() will return errors for many other cases too.
> 
> For what it's worth, the drivers I have written just return an error. It's
> simple for the driver and the user and works. If the consensus is that the
> drivers should instead set the default routing, or somehow mangle the given
> routing to an acceptable form, I can update those drivers accordingly.
> 
> But we probably need to update the docs too to be a bit more clear what
> VIDIOC_SUBDEV_S_ROUTING will do (although are the other ioctls any
> clearer?).
> 
> All that said, I think it's still a bit case-by-case. I don't think the
> drivers should always return an error if they get a routing table that's not
> 100% perfect. E.g. if a device supports two static routes, but the second
> one can be enabled or disabled, the driver should still accept a routing
> table from the user with only the first route present. Etc.
> 
> For the specific case in this patch... I'd prefer returning an error, or if
> that's not ok, set default routing.

Not modifying the routing table is another option as well but it may
require separating validating user-provided routes and applying the routes
to the sub-device state. The default could be useful in principle, too, for
routing-unaware applications but they won't be calling S_ROUTING anyway.

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Dan Scally]

Hi Laurent

On 30/05/2024 01:15, Laurent Pinchart wrote:
> Hi Dan,
>
> Thank you for the patch.
>
> On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
>> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
>> V4L2 and Media Controller compliant and creates subdevices to manage
>> the ISP itself, its internal test pattern generator as well as the
>> crop, scaler and output format functionality for each of its two
>> output devices.
>>
>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>> ---
>> Changes in v5:
>>
>> 	- Reworked input formats - previously we allowed representing input data
>> 	  as any 8-16 bit format. Now we only allow input data to be represented
>> 	  by the new 20-bit bayer formats, which is corrected to the equivalent
>> 	  16-bit format in RAW bypass mode.
>> 	- Stopped bypassing blocks that we haven't added supporting parameters
>> 	  for yet.
>> 	- Addressed most of Sakari's comments from the list
>>
>> Changes not yet made in v5:
>>
>> 	- The output pipelines can still be started and stopped independently of
>> 	  one another - I'd like to discuss that more.
>> 	- the TPG subdev still uses .s_stream() - I need to rebase onto a tree
>> 	  with working .enable_streams() for a single-source-pad subdevice.
>>
>> Changes in v4:
>>
>> 	- Reworked mali_c55_update_bits() to internally perform the bit-shift
> I really don't like that, it makes the code very confusing, even more so
> as it differs from regmap_update_bits().
>
> Look at this for instance:
>
> 	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
>
> It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
> BIT(0).
>
> Sorry, I know it will be painful, but this change needs to be reverted.


I'd like to argue for keeping this, on the grounds that it's better. I got lazy in the change you 
reference there, and because BIT(0) is the same as 0x01 didn't bother changing it. I agree that 
that's confusing but I think it would be better to keep the change and just update all the call 
sites properly. The benefits as I see them are two:


1. This method ensures sane consistent calling of the function. Without the internal shift you have 
to shift the values at the call site, but there's no reason to do that if the value you're setting 
is 0x00 or if the field you're targeting in the register starts at bit 0, so I think writing code 
naturally we'd have a mix of situations like so:


#define REG_1 0xfc00

#define REG_2 0xff

mali_c55_update_bits(mali_c55, 0x1234, REG_1, 0x02 << 10);

mali_c55_update_bits(mali_c55, 0x1234, REG_1, 0x00);

mali_c55_update_bits(mali_c55, 0x1234, REG_2, 0x02);


And I think that the mixture is more confusing than the difference with regmap_update_bits(). We 
could include the bitshifting for consistencies sake despite it being unecessary, but it's extra 
work for no real reason and itself "looks wrong" if the field starts at bit(0)...it would look less 
wrong with an offset macro that defines the number of bits to shift as 0 but then we're on to 
advantage #2...

2. It makes the driver far cleaner. Without it we either have magic numbers scattered throughout 
(and sometimes have to calculate them with extra variables where the write can target different 
places conditionally) or have macros defining the number of bits to shift, or have to do (ffs(mask) 
- 1) everywhere, and that tends to make the call sites a lot messier - this was the original reason 
I moved it internal actually.


What do you think?


Thanks

Dan



>
>> 	- Reworked the resizer to allow cropping during streaming
>> 	- Fixed a bug in NV12 output
>>
>> Changes in v3:
>>
>> 	- Mostly minor fixes suggested by Sakari
>> 	- Fixed the sequencing of vb2 buffers to be synchronised across the two
>> 	  capture devices.
>>
>> Changes in v2:
>>
>> 	- Clock handling
>> 	- Fixed the warnings raised by the kernel test robot
>>
>>   drivers/media/platform/Kconfig                |   1 +
>>   drivers/media/platform/Makefile               |   1 +
>>   drivers/media/platform/arm/Kconfig            |   5 +
>>   drivers/media/platform/arm/Makefile           |   2 +
>>   drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
>>   drivers/media/platform/arm/mali-c55/Makefile  |   9 +
>>   .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
>>   .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
>>   .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
>>   .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
>>   .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
>>   .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
>>   .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
>>   .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
>>   14 files changed, 4452 insertions(+)
>>   create mode 100644 drivers/media/platform/arm/Kconfig
>>   create mode 100644 drivers/media/platform/arm/Makefile
>>   create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
>>   create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> I've skipped review of capture.c and resizer.c as I already have plenty
> of comments for the other files, and it's getting late. I'll try to
> review the rest tomorrow.
>
>> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
>> index 2d79bfc68c15..c929169766aa 100644
>> --- a/drivers/media/platform/Kconfig
>> +++ b/drivers/media/platform/Kconfig
>> @@ -65,6 +65,7 @@ config VIDEO_MUX
>>   source "drivers/media/platform/allegro-dvt/Kconfig"
>>   source "drivers/media/platform/amlogic/Kconfig"
>>   source "drivers/media/platform/amphion/Kconfig"
>> +source "drivers/media/platform/arm/Kconfig"
>>   source "drivers/media/platform/aspeed/Kconfig"
>>   source "drivers/media/platform/atmel/Kconfig"
>>   source "drivers/media/platform/broadcom/Kconfig"
>> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
>> index da17301f7439..9a647abd5218 100644
>> --- a/drivers/media/platform/Makefile
>> +++ b/drivers/media/platform/Makefile
>> @@ -8,6 +8,7 @@
>>   obj-y += allegro-dvt/
>>   obj-y += amlogic/
>>   obj-y += amphion/
>> +obj-y += arm/
>>   obj-y += aspeed/
>>   obj-y += atmel/
>>   obj-y += broadcom/
>> diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
>> new file mode 100644
>> index 000000000000..4f0764c329c7
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/Kconfig
>> @@ -0,0 +1,5 @@
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +
>> +comment "ARM media platform drivers"
>> +
>> +source "drivers/media/platform/arm/mali-c55/Kconfig"
>> diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
>> new file mode 100644
>> index 000000000000..8cc4918725ef
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/Makefile
>> @@ -0,0 +1,2 @@
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +obj-y += mali-c55/
>> diff --git a/drivers/media/platform/arm/mali-c55/Kconfig b/drivers/media/platform/arm/mali-c55/Kconfig
>> new file mode 100644
>> index 000000000000..602085e28b01
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/Kconfig
>> @@ -0,0 +1,18 @@
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +config VIDEO_MALI_C55
>> +	tristate "ARM Mali-C55 Image Signal Processor driver"
>> +	depends on V4L_PLATFORM_DRIVERS
>> +	depends on VIDEO_DEV && OF
>> +	depends on ARCH_VEXPRESS || COMPILE_TEST
>> +	select MEDIA_CONTROLLER
>> +	select VIDEO_V4L2_SUBDEV_API
>> +	select VIDEOBUF2_DMA_CONTIG
>> +	select VIDEOBUF2_VMALLOC
>> +	select V4L2_FWNODE
>> +	select GENERIC_PHY_MIPI_DPHY
> Alphabetical order ?
>
>> +	default n
> That's the default, you don't have to specify ti.
>
>> +	help
>> +	  Enable this to support Arm's Mali-C55 Image Signal Processor.
>> +
>> +	  To compile this driver as a module, choose M here: the module
>> +	  will be called mali-c55.
>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
>> new file mode 100644
>> index 000000000000..77dcb2fbf0f4
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
>> @@ -0,0 +1,9 @@
>> +# SPDX-License-Identifier: GPL-2.0
>> +
>> +mali-c55-y := mali-c55-capture.o \
>> +	      mali-c55-core.o \
>> +	      mali-c55-isp.o \
>> +	      mali-c55-tpg.o \
>> +	      mali-c55-resizer.o
> Alphabetical order here too.
>
>> +
>> +obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>> new file mode 100644
>> index 000000000000..1d539ac9c498
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>> @@ -0,0 +1,951 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ARM Mali-C55 ISP Driver - Video capture devices
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#include <linux/cleanup.h>
>> +#include <linux/minmax.h>
>> +#include <linux/pm_runtime.h>
>> +#include <linux/string.h>
>> +#include <linux/videodev2.h>
>> +
>> +#include <media/v4l2-dev.h>
>> +#include <media/v4l2-event.h>
>> +#include <media/v4l2-ioctl.h>
>> +#include <media/v4l2-subdev.h>
>> +#include <media/videobuf2-core.h>
>> +#include <media/videobuf2-dma-contig.h>
>> +
>> +#include "mali-c55-common.h"
>> +#include "mali-c55-registers.h"
>> +
>> +static const struct mali_c55_fmt mali_c55_fmts[] = {
>> +	/*
>> +	 * This table is missing some entries which need further work or
>> +	 * investigation:
>> +	 *
>> +	 * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
>> +	 * Base mode 5 is "Generic Data"
>> +	 * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
>> +	 * Base mode 9 seems to have no V4L2 equivalent
>> +	 * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
>> +	 * equivalent
>> +	 */
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_ARGB2101010,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_RGB121212_1X36,
>> +			MEDIA_BUS_FMT_RGB202020_1X60,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_A2R10G10B10,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_RGB565,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_RGB121212_1X36,
>> +			MEDIA_BUS_FMT_RGB202020_1X60,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_RGB565,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_BGR24,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_RGB121212_1X36,
>> +			MEDIA_BUS_FMT_RGB202020_1X60,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_RGB24,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_YUYV,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_YUV10_1X30,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_YUY2,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_UYVY,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_YUV10_1X30,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_UYVY,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_Y210,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_YUV10_1X30,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_Y210,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	/*
>> +	 * This is something of a hack, the ISP thinks it's running NV12M but
>> +	 * by setting uv_plane = 0 we simply discard that planes and only output
>> +	 * the Y-plane.
>> +	 */
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_GREY,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_YUV10_1X30,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_NV12M,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_YUV10_1X30,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_NV21M,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_YUV10_1X30,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
>> +		}
>> +	},
>> +	/*
>> +	 * RAW uncompressed formats are all packed in 16 bpp.
>> +	 * TODO: Expand this list to encompass all possible RAW formats.
>> +	 */
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_SRGGB16,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_SRGGB16_1X16,
>> +		},
>> +		.is_raw = true,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_SBGGR16,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_SBGGR16_1X16,
>> +		},
>> +		.is_raw = true,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_SGBRG16,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_SGBRG16_1X16,
>> +		},
>> +		.is_raw = true,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_SGRBG16,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_SGRBG16_1X16,
>> +		},
>> +		.is_raw = true,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +};
>> +
>> +static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
>> +					       u32 code)
>> +{
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
>> +		if (fmt->mbus_codes[i] == code)
>> +			return true;
>> +	}
>> +
>> +	return false;
>> +}
>> +
>> +bool mali_c55_format_is_raw(unsigned int mbus_code)
>> +{
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>> +		if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
>> +			return mali_c55_fmts[i].is_raw;
>> +	}
>> +
>> +	return false;
>> +}
>> +
>> +static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
>> +{
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>> +		if (mali_c55_fmts[i].fourcc == pixelformat)
>> +			return &mali_c55_fmts[i];
>> +	}
>> +
>> +	/*
>> +	 * If we find no matching pixelformat, we'll just default to the first
>> +	 * one for now.
>> +	 */
>> +
>> +	return &mali_c55_fmts[0];
>> +}
>> +
>> +static const char * const capture_device_names[] = {
>> +	"mali-c55 fr",
>> +	"mali-c55 ds",
>> +	"mali-c55 3a stats",
>> +	"mali-c55 params",
>> +};
>> +
>> +static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
>> +{
>> +	if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
>> +		return capture_device_names[0];
>> +
>> +	if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
>> +		return capture_device_names[1];
>> +
>> +	return "params/stat not supported yet";
>> +}
>> +
>> +static int mali_c55_link_validate(struct media_link *link)
>> +{
>> +	struct video_device *vdev =
>> +		media_entity_to_video_device(link->sink->entity);
>> +	struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
>> +	struct v4l2_subdev *sd =
>> +		media_entity_to_v4l2_subdev(link->source->entity);
>> +	const struct v4l2_pix_format_mplane *pix_mp;
>> +	const struct mali_c55_fmt *cap_fmt;
>> +	struct v4l2_subdev_format sd_fmt = {
>> +		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
>> +		.pad = link->source->index,
>> +	};
>> +	int ret;
>> +
>> +	ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
>> +	if (ret)
>> +		return ret;
>> +
>> +	pix_mp = &cap_dev->mode.pix_mp;
>> +	cap_fmt = cap_dev->mode.capture_fmt;
>> +
>> +	if (sd_fmt.format.width != pix_mp->width ||
>> +	    sd_fmt.format.height != pix_mp->height) {
>> +		dev_dbg(cap_dev->mali_c55->dev,
>> +			"link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
>> +			link->source->entity->name, link->source->index,
>> +			link->sink->entity->name, link->sink->index,
>> +			sd_fmt.format.width, sd_fmt.format.height,
>> +			pix_mp->width, pix_mp->height);
>> +		return -EPIPE;
>> +	}
>> +
>> +	if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
>> +		dev_dbg(cap_dev->mali_c55->dev,
>> +			"link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format %p4cc\n",
>> +			link->source->entity->name, link->source->index,
>> +			link->sink->entity->name, link->sink->index,
>> +			sd_fmt.format.code, &pix_mp->pixelformat);
>> +		return -EPIPE;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct media_entity_operations mali_c55_media_ops = {
>> +	.link_validate = mali_c55_link_validate,
>> +};
>> +
>> +static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
>> +				    unsigned int *num_planes, unsigned int sizes[],
>> +				    struct device *alloc_devs[])
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>> +	unsigned int i;
>> +
>> +	if (*num_planes) {
>> +		if (*num_planes != cap_dev->mode.pix_mp.num_planes)
>> +			return -EINVAL;
>> +
>> +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>> +			if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
>> +				return -EINVAL;
>> +	} else {
>> +		*num_planes = cap_dev->mode.pix_mp.num_planes;
>> +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>> +			sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static void mali_c55_buf_queue(struct vb2_buffer *vb)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
>> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>> +	struct mali_c55_buffer *buf = container_of(vbuf,
>> +						   struct mali_c55_buffer, vb);
>> +	unsigned int i;
>> +
>> +	buf->plane_done[MALI_C55_PLANE_Y] = false;
>> +
>> +	/*
>> +	 * If we're in a single-plane format we flag the other plane as done
>> +	 * already so it's dequeued appropriately later
>> +	 */
>> +	buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
>> +
>> +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
>> +		unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
>> +
>> +		vb2_set_plane_payload(vb, i, size);
>> +	}
>> +
>> +	spin_lock(&cap_dev->buffers.lock);
>> +	list_add_tail(&buf->queue, &cap_dev->buffers.queue);
>> +	spin_unlock(&cap_dev->buffers.lock);
>> +}
>> +
>> +static int mali_c55_buf_init(struct vb2_buffer *vb)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
>> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>> +	struct mali_c55_buffer *buf = container_of(vbuf,
>> +						   struct mali_c55_buffer, vb);
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>> +		buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
>> +
>> +	return 0;
>> +}
>> +
>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
>> +{
>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>> +
>> +	guard(spinlock)(&cap_dev->buffers.lock);
>> +
>> +	cap_dev->buffers.curr = cap_dev->buffers.next;
>> +	cap_dev->buffers.next = NULL;
>> +
>> +	if (!list_empty(&cap_dev->buffers.queue)) {
>> +		struct v4l2_pix_format_mplane *pix_mp;
>> +		const struct v4l2_format_info *info;
>> +		u32 *addrs;
>> +
>> +		pix_mp = &cap_dev->mode.pix_mp;
>> +		info = v4l2_format_info(pix_mp->pixelformat);
>> +
>> +		mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
>> +		if (cap_dev->mode.capture_fmt->registers.uv_plane)
>> +			mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
>> +
>> +		cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
>> +							 struct mali_c55_buffer,
>> +							 queue);
>> +		list_del(&cap_dev->buffers.next->queue);
>> +
>> +		addrs = cap_dev->buffers.next->addrs;
>> +		mali_c55_write(mali_c55,
>> +			MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
>> +			addrs[MALI_C55_PLANE_Y]);
>> +		mali_c55_write(mali_c55,
>> +			MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
>> +			addrs[MALI_C55_PLANE_UV]);
>> +		mali_c55_write(mali_c55,
>> +			MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
>> +			pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
>> +		mali_c55_write(mali_c55,
>> +			MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
>> +			pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
>> +			/ info->hdiv);
>> +	} else {
>> +		/*
>> +		 * If we underflow then we can tell the ISP that we don't want
>> +		 * to write out the next frame.
>> +		 */
>> +		mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>> +		mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>> +	}
>> +}
>> +
>> +static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
>> +				   unsigned int framecount)
>> +{
>> +	curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>> +	curr_buf->vb.field = V4L2_FIELD_NONE;
>> +	curr_buf->vb.sequence = framecount;
>> +	vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
>> +}
>> +
>> +/**
>> + * mali_c55_set_plane_done - mark the plane as written and process the buffer if
>> + *			     both planes are finished.
>> + * @cap_dev:  pointer to the fr or ds pipe output
>> + * @plane:    the plane to mark as completed
>> + *
>> + * The Mali C55 ISP has muliplanar outputs for some formats that come with two
>> + * separate "buffer write completed" interrupts - we need to flag each plane's
>> + * completion and check whether both planes are done - if so, complete the buf
>> + * in vb2.
>> + */
>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>> +			     enum mali_c55_planes plane)
>> +{
>> +	struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
>> +	struct mali_c55_buffer *curr_buf;
>> +
>> +	guard(spinlock)(&cap_dev->buffers.lock);
>> +	curr_buf = cap_dev->buffers.curr;
>> +
>> +	/*
>> +	 * This _should_ never happen. If no buffer was available from vb2 then
>> +	 * we tell the ISP not to bother writing the next frame, which means the
>> +	 * interrupts that call this function should never trigger. If it does
>> +	 * happen then one of our assumptions is horribly wrong - complain
>> +	 * loudly and do nothing.
>> +	 */
>> +	if (!curr_buf) {
>> +		dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
>> +			mali_c55_cap_dev_to_name(cap_dev), __func__);
>> +		return;
>> +	}
>> +
>> +	/* If the other plane is also done... */
>> +	if (curr_buf->plane_done[~plane & 1]) {
>> +		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
>> +		cap_dev->buffers.curr = NULL;
>> +		isp->frame_sequence++;
>> +	} else {
>> +		curr_buf->plane_done[plane] = true;
>> +	}
>> +}
>> +
>> +static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
>> +{
>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>> +
>> +	mali_c55_update_bits(mali_c55,
>> +			     MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>> +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>> +	mali_c55_update_bits(mali_c55,
>> +			     MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>> +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>> +}
>> +
>> +static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
>> +{
>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>> +
>> +	/*
>> +	 * The Mali ISP can hold up to 5 buffer addresses and simply cycle
>> +	 * through them, but it's not clear to me that the vb2 queue _guarantees_
>> +	 * it will queue buffers to the driver in a fixed order, and ensuring
>> +	 * we call vb2_buffer_done() for the right buffer seems to me to add
>> +	 * pointless complexity given in multi-context mode we'd need to
>> +	 * re-write those registers every frame anyway...so we tell the ISP to
>> +	 * use a single register and update it for each frame.
>> +	 */
>> +	mali_c55_update_bits(mali_c55,
>> +			MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
>> +			MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
>> +	mali_c55_update_bits(mali_c55,
>> +			MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
>> +			MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
>> +
>> +	/*
>> +	 * We only queue a buffer in the streamon path if this is the first of
>> +	 * the capture devices to start streaming. If the ISP is already running
>> +	 * then we rely on the ISP_START interrupt to queue the first buffer for
>> +	 * this capture device.
>> +	 */
>> +	if (mali_c55->pipe.start_count == 1)
>> +		mali_c55_set_next_buffer(cap_dev);
>> +}
>> +
>> +static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
>> +					    enum vb2_buffer_state state)
>> +{
>> +	struct mali_c55_buffer *buf, *tmp;
>> +
>> +	guard(spinlock)(&cap_dev->buffers.lock);
>> +
>> +	if (cap_dev->buffers.curr) {
>> +		vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
>> +				state);
>> +		cap_dev->buffers.curr = NULL;
>> +	}
>> +
>> +	if (cap_dev->buffers.next) {
>> +		vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
>> +				state);
>> +		cap_dev->buffers.next = NULL;
>> +	}
>> +
>> +	list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
>> +		list_del(&buf->queue);
>> +		vb2_buffer_done(&buf->vb.vb2_buf, state);
>> +	}
>> +}
>> +
>> +static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>> +	struct mali_c55_resizer *rzr = cap_dev->rzr;
>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>> +	int ret;
>> +
>> +	guard(mutex)(&isp->lock);
>> +
>> +	ret = pm_runtime_resume_and_get(mali_c55->dev);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = video_device_pipeline_start(&cap_dev->vdev,
>> +					  &cap_dev->mali_c55->pipe);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
>> +			mali_c55_cap_dev_to_name(cap_dev));
>> +		goto err_pm_put;
>> +	}
>> +
>> +	mali_c55_cap_dev_stream_enable(cap_dev);
>> +	mali_c55_rzr_start_stream(rzr);
>> +
>> +	/*
>> +	 * We only start the ISP if we're the only capture device that's
>> +	 * streaming. Otherwise, it'll already be active.
>> +	 */
>> +	if (mali_c55->pipe.start_count == 1) {
>> +		ret = mali_c55_isp_start_stream(isp);
>> +		if (ret)
>> +			goto err_disable_cap_dev;
>> +	}
>> +
>> +	return 0;
>> +
>> +err_disable_cap_dev:
>> +	mali_c55_cap_dev_stream_disable(cap_dev);
>> +	video_device_pipeline_stop(&cap_dev->vdev);
>> +err_pm_put:
>> +	pm_runtime_put(mali_c55->dev);
>> +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
>> +
>> +	return ret;
>> +}
>> +
>> +static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>> +	struct mali_c55_resizer *rzr = cap_dev->rzr;
>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>> +
>> +	guard(mutex)(&isp->lock);
>> +
>> +	/*
>> +	 * If one of the other capture nodes is streaming, we shouldn't
>> +	 * disable the ISP here.
>> +	 */
>> +	if (mali_c55->pipe.start_count == 1)
>> +		mali_c55_isp_stop_stream(&mali_c55->isp);
>> +
>> +	mali_c55_rzr_stop_stream(rzr);
>> +	mali_c55_cap_dev_stream_disable(cap_dev);
>> +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
>> +	video_device_pipeline_stop(&cap_dev->vdev);
>> +	pm_runtime_put(mali_c55->dev);
>> +}
>> +
>> +static const struct vb2_ops mali_c55_vb2_ops = {
>> +	.queue_setup		= &mali_c55_vb2_queue_setup,
>> +	.buf_queue		= &mali_c55_buf_queue,
>> +	.buf_init		= &mali_c55_buf_init,
>> +	.wait_prepare		= vb2_ops_wait_prepare,
>> +	.wait_finish		= vb2_ops_wait_finish,
>> +	.start_streaming	= &mali_c55_vb2_start_streaming,
>> +	.stop_streaming		= &mali_c55_vb2_stop_streaming,
>> +};
>> +
>> +static const struct v4l2_file_operations mali_c55_v4l2_fops = {
>> +	.owner = THIS_MODULE,
>> +	.unlocked_ioctl = video_ioctl2,
>> +	.open = v4l2_fh_open,
>> +	.release = vb2_fop_release,
>> +	.poll = vb2_fop_poll,
>> +	.mmap = vb2_fop_mmap,
>> +};
>> +
>> +static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
>> +{
>> +	const struct mali_c55_fmt *capture_format;
>> +	const struct v4l2_format_info *info;
>> +	struct v4l2_plane_pix_format *plane;
>> +	unsigned int i;
>> +
>> +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
>> +	pix_mp->pixelformat = capture_format->fourcc;
>> +
>> +	pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
>> +			      MALI_C55_MAX_WIDTH);
>> +	pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
>> +			       MALI_C55_MAX_HEIGHT);
>> +
>> +	pix_mp->field = V4L2_FIELD_NONE;
>> +	pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
>> +	pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>> +	pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
>> +
>> +	info = v4l2_format_info(pix_mp->pixelformat);
>> +	pix_mp->num_planes = info->mem_planes;
>> +	memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
>> +
>> +	pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;
>> +	pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
>> +				       * pix_mp->height;
>> +
>> +	for (i = 1; i < info->comp_planes; i++) {
>> +		plane = &pix_mp->plane_fmt[i];
>> +
>> +		plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
>> +						   info->hdiv);
>> +		plane->sizeimage = DIV_ROUND_UP(
>> +					plane->bytesperline * pix_mp->height,
>> +					info->vdiv);
>> +	}
>> +
>> +	if (info->mem_planes == 1) {
>> +		for (i = 1; i < info->comp_planes; i++) {
>> +			plane = &pix_mp->plane_fmt[i];
>> +			pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
>> +		}
>> +	}
>> +}
>> +
>> +static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
>> +					   struct v4l2_format *f)
>> +{
>> +	mali_c55_try_fmt(&f->fmt.pix_mp);
>> +
>> +	return 0;
>> +}
>> +
>> +static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
>> +				struct v4l2_pix_format_mplane *pix_mp)
>> +{
>> +	const struct mali_c55_fmt *capture_format;
>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>> +	const struct v4l2_format_info *info;
>> +
>> +	mali_c55_try_fmt(pix_mp);
>> +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
>> +	info = v4l2_format_info(pix_mp->pixelformat);
>> +	if (WARN_ON(!info))
>> +		return;
>> +
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>> +		       capture_format->registers.base_mode);
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
>> +		       MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
>> +		       MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
>> +
>> +	if (info->mem_planes > 1) {
>> +		mali_c55_write(mali_c55,
>> +			       MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>> +			       capture_format->registers.base_mode);
>> +		mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>> +				MALI_C55_WRITER_SUBMODE_MASK,
>> +				capture_format->registers.uv_plane);
>> +
>> +		mali_c55_write(mali_c55,
>> +			MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
>> +			MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
>> +			MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
>> +	}
>> +
>> +	if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
>> +		/*
>> +		 * TODO: Figure out the colour matrix coefficients and calculate
>> +		 * and write them here.
>> +		 */
>> +
>> +		mali_c55_write(mali_c55,
>> +			       MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>> +			       MALI_C55_CS_CONV_MATRIX_MASK);
>> +
>> +		if (info->hdiv > 1)
>> +			mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>> +				MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
>> +		if (info->vdiv > 1)
>> +			mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>> +				MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
>> +		if (info->hdiv > 1 || info->vdiv > 1)
>> +			mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>> +				MALI_C55_CS_CONV_FILTER_MASK, 0x01);
>> +	}
>> +
>> +	cap_dev->mode.pix_mp = *pix_mp;
>> +	cap_dev->mode.capture_fmt = capture_format;
>> +}
>> +
>> +static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
>> +					 struct v4l2_format *f)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>> +
>> +	if (vb2_is_busy(&cap_dev->queue))
>> +		return -EBUSY;
>> +
>> +	mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
>> +					 struct v4l2_format *f)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>> +
>> +	f->fmt.pix_mp = cap_dev->mode.pix_mp;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
>> +					    struct v4l2_fmtdesc *f)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>> +	unsigned int j = 0;
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>> +		if (f->mbus_code &&
>> +		    !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
>> +						       f->mbus_code))
>> +			continue;
>> +
>> +		/* Downscale pipe can't output RAW formats */
>> +		if (mali_c55_fmts[i].is_raw &&
>> +		    cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
>> +			continue;
>> +
>> +		if (j++ == f->index) {
>> +			f->pixelformat = mali_c55_fmts[i].fourcc;
>> +			return 0;
>> +		}
>> +	}
>> +
>> +	return -EINVAL;
>> +}
>> +
>> +static int mali_c55_querycap(struct file *file, void *fh,
>> +			     struct v4l2_capability *cap)
>> +{
>> +	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
>> +	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
>> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
>> +	.vidioc_querybuf = vb2_ioctl_querybuf,
>> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
>> +	.vidioc_qbuf = vb2_ioctl_qbuf,
>> +	.vidioc_expbuf = vb2_ioctl_expbuf,
>> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
>> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
>> +	.vidioc_streamon = vb2_ioctl_streamon,
>> +	.vidioc_streamoff = vb2_ioctl_streamoff,
>> +	.vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
>> +	.vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
>> +	.vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
>> +	.vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
>> +	.vidioc_querycap = mali_c55_querycap,
>> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
>> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
>> +};
>> +
>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
>> +{
>> +	struct v4l2_pix_format_mplane pix_mp;
>> +	struct mali_c55_cap_dev *cap_dev;
>> +	struct video_device *vdev;
>> +	struct vb2_queue *vb2q;
>> +	unsigned int i;
>> +	int ret;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
>> +		cap_dev = &mali_c55->cap_devs[i];
>> +		vdev = &cap_dev->vdev;
>> +		vb2q = &cap_dev->queue;
>> +
>> +		/*
>> +		 * The downscale output pipe is an optional block within the ISP
>> +		 * so we need to check whether it's actually been fitted or not.
>> +		 */
>> +
>> +		if (i == MALI_C55_CAP_DEV_DS &&
>> +		    !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
>> +			continue;
>> +
>> +		cap_dev->mali_c55 = mali_c55;
>> +		mutex_init(&cap_dev->lock);
>> +		INIT_LIST_HEAD(&cap_dev->buffers.queue);
>> +
>> +		switch (i) {
>> +		case MALI_C55_CAP_DEV_FR:
>> +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
>> +			cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
>> +			break;
>> +		case MALI_C55_CAP_DEV_DS:
>> +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
>> +			cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
>> +			break;
>> +		default:
>> +			mutex_destroy(&cap_dev->lock);
>> +			ret = -EINVAL;
>> +			goto err_destroy_mutex;
>> +		}
>> +
>> +		cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
>> +		ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
>> +		if (ret) {
>> +			mutex_destroy(&cap_dev->lock);
>> +			goto err_destroy_mutex;
>> +		}
>> +
>> +		vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>> +		vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
>> +		vb2q->drv_priv = cap_dev;
>> +		vb2q->mem_ops = &vb2_dma_contig_memops;
>> +		vb2q->ops = &mali_c55_vb2_ops;
>> +		vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
>> +		vb2q->min_queued_buffers = 1;
>> +		vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
>> +		vb2q->lock = &cap_dev->lock;
>> +		vb2q->dev = mali_c55->dev;
>> +
>> +		ret = vb2_queue_init(vb2q);
>> +		if (ret) {
>> +			dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
>> +				mali_c55_cap_dev_to_name(cap_dev));
>> +			goto err_cleanup_media_entity;
>> +		}
>> +
>> +		strscpy(cap_dev->vdev.name, capture_device_names[i],
>> +			sizeof(cap_dev->vdev.name));
>> +		vdev->release = video_device_release_empty;
>> +		vdev->fops = &mali_c55_v4l2_fops;
>> +		vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
>> +		vdev->lock = &cap_dev->lock;
>> +		vdev->v4l2_dev = &mali_c55->v4l2_dev;
>> +		vdev->queue = &cap_dev->queue;
>> +		vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +				    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
>> +		vdev->entity.ops = &mali_c55_media_ops;
>> +		video_set_drvdata(vdev, cap_dev);
>> +
>> +		memset(&pix_mp, 0, sizeof(pix_mp));
>> +		pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
>> +		pix_mp.width = MALI_C55_DEFAULT_WIDTH;
>> +		pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
>> +		mali_c55_set_format(cap_dev, &pix_mp);
>> +
>> +		ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
>> +		if (ret) {
>> +			dev_err(mali_c55->dev,
>> +				"%s failed to register video device\n",
>> +				mali_c55_cap_dev_to_name(cap_dev));
>> +			goto err_release_vb2q;
>> +		}
>> +	}
>> +
>> +	return 0;
>> +
>> +err_release_vb2q:
>> +	vb2_queue_release(vb2q);
>> +err_cleanup_media_entity:
>> +	media_entity_cleanup(&cap_dev->vdev.entity);
>> +err_destroy_mutex:
>> +	mutex_destroy(&cap_dev->lock);
>> +	mali_c55_unregister_capture_devs(mali_c55);
>> +
>> +	return ret;
>> +}
>> +
>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev;
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
>> +		cap_dev = &mali_c55->cap_devs[i];
>> +
>> +		if (!video_is_registered(&cap_dev->vdev))
>> +			continue;
>> +
>> +		vb2_video_unregister_device(&cap_dev->vdev);
>> +		media_entity_cleanup(&cap_dev->vdev.entity);
>> +		mutex_destroy(&cap_dev->lock);
>> +	}
>> +}
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>> new file mode 100644
>> index 000000000000..2d0c4d152beb
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>> @@ -0,0 +1,266 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * ARM Mali-C55 ISP Driver - Common definitions
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#ifndef _MALI_C55_COMMON_H
>> +#define _MALI_C55_COMMON_H
>> +
>> +#include <linux/clk.h>
>> +#include <linux/io.h>
>> +#include <linux/list.h>
>> +#include <linux/mutex.h>
>> +#include <linux/scatterlist.h>
> I don't think this is needed. You're however missing spinlock.h.
>
>> +#include <linux/videodev2.h>
>> +
>> +#include <media/media-device.h>
>> +#include <media/v4l2-async.h>
>> +#include <media/v4l2-ctrls.h>
>> +#include <media/v4l2-dev.h>
>> +#include <media/v4l2-device.h>
>> +#include <media/v4l2-subdev.h>
>> +#include <media/videobuf2-core.h>
>> +#include <media/videobuf2-v4l2.h>
>> +
>> +#define MALI_C55_DRIVER_NAME		"mali-c55"
>> +
>> +/* min and max values for the image sizes */
>> +#define MALI_C55_MIN_WIDTH		640U
>> +#define MALI_C55_MIN_HEIGHT		480U
>> +#define MALI_C55_MAX_WIDTH		8192U
>> +#define MALI_C55_MAX_HEIGHT		8192U
>> +#define MALI_C55_DEFAULT_WIDTH		1920U
>> +#define MALI_C55_DEFAULT_HEIGHT		1080U
>> +
>> +#define MALI_C55_DEFAULT_MEDIA_BUS_FMT	MEDIA_BUS_FMT_RGB121212_1X36
>> +
>> +struct mali_c55;
>> +struct mali_c55_cap_dev;
>> +struct platform_device;
> You should also forward-declare
>
> struct device;
> struct dma_chan;
> struct resource;
>
>> +
>> +static const char * const mali_c55_clk_names[] = {
>> +	"aclk",
>> +	"hclk",
>> +};
> This will end up duplicating the array in each compilation unit, not
> great. Move it to mali-c55-core.c. You use it in this file just for its
> size, replace that with a macro that defines the size, or allocate
> mali_c55.clks dynamically with devm_kcalloc().
>
>> +
>> +enum mali_c55_interrupts {
>> +	MALI_C55_IRQ_ISP_START,
>> +	MALI_C55_IRQ_ISP_DONE,
>> +	MALI_C55_IRQ_MCM_ERROR,
>> +	MALI_C55_IRQ_BROKEN_FRAME_ERROR,
>> +	MALI_C55_IRQ_MET_AF_DONE,
>> +	MALI_C55_IRQ_MET_AEXP_DONE,
>> +	MALI_C55_IRQ_MET_AWB_DONE,
>> +	MALI_C55_IRQ_AEXP_1024_DONE,
>> +	MALI_C55_IRQ_IRIDIX_MET_DONE,
>> +	MALI_C55_IRQ_LUT_INIT_DONE,
>> +	MALI_C55_IRQ_FR_Y_DONE,
>> +	MALI_C55_IRQ_FR_UV_DONE,
>> +	MALI_C55_IRQ_DS_Y_DONE,
>> +	MALI_C55_IRQ_DS_UV_DONE,
>> +	MALI_C55_IRQ_LINEARIZATION_DONE,
>> +	MALI_C55_IRQ_RAW_FRONTEND_DONE,
>> +	MALI_C55_IRQ_NOISE_REDUCTION_DONE,
>> +	MALI_C55_IRQ_IRIDIX_DONE,
>> +	MALI_C55_IRQ_BAYER2RGB_DONE,
>> +	MALI_C55_IRQ_WATCHDOG_TIMER,
>> +	MALI_C55_IRQ_FRAME_COLLISION,
>> +	MALI_C55_IRQ_UNUSED,
>> +	MALI_C55_IRQ_DMA_ERROR,
>> +	MALI_C55_IRQ_INPUT_STOPPED,
>> +	MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
>> +	MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
>> +	MALI_C55_NUM_IRQ_BITS
> Those are register bits, I think they belong to mali-c55-registers.h,
> and should probably be macros instead of an enum.
>
>> +};
>> +
>> +enum mali_c55_isp_pads {
>> +	MALI_C55_ISP_PAD_SINK_VIDEO,
> As there's a single sink pad, maybe MALI_C55_ISP_PAD_SINK ? Ah, you're
> probably preparing for ISP parameters support. It's fine.
>
>> +	MALI_C55_ISP_PAD_SOURCE,
> Then maybe this should be named MALI_C55_ISP_PAD_SOURCE_VIDEO as I
> assume there will be a stats source pad.
>
>> +	MALI_C55_ISP_PAD_SOURCE_BYPASS,
>> +	MALI_C55_ISP_NUM_PADS,
>> +};
>> +
>> +struct mali_c55_tpg {
>> +	struct mali_c55 *mali_c55;
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pad;
>> +	struct mutex lock;
>> +	struct mali_c55_tpg_ctrls {
>> +		struct v4l2_ctrl_handler handler;
>> +		struct v4l2_ctrl *test_pattern;
> Set but never used. You can drop it.
>
>> +		struct v4l2_ctrl *hblank;
> Set and used only once, in the same function. You can make it a local
> variable.
>
>> +		struct v4l2_ctrl *vblank;
>> +	} ctrls;
>> +};
> I wonder if this file should be split, with mali-c55-capture.h,
> mali-c55-core.h, mali-c55-isp.h, ... I think it could increase
> readability by clearly separating the different elements. Up to you.
>
>> +
>> +struct mali_c55_isp {
>> +	struct mali_c55 *mali_c55;
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pads[MALI_C55_ISP_NUM_PADS];
>> +	struct media_pad *remote_src;
>> +	struct v4l2_async_notifier notifier;
> I'm tempted to move the notifier to mali_c55, as it's related to
> components external to the whole ISP, not to the ISP subdev itself.
> Could you give it a try, to see if it could be done without any drawback
> ?
>
>> +	struct mutex lock;
> Locks require a comment to explain what they protect. Same below where
> applicable (for both mutexes and spinlocks).
>
>> +	unsigned int frame_sequence;
>> +};
>> +
>> +enum mali_c55_resizer_ids {
>> +	MALI_C55_RZR_FR,
>> +	MALI_C55_RZR_DS,
>> +	MALI_C55_NUM_RZRS,
> The usual abbreviation for "resizer" in drivers/media/ is "rsz", not
> "rzr". I would have said we can leave it as-is as changing it would be a
> bit annoying, but I then realized that "rzr" is not just unusual, it's
> actually not used at all. Would you mind applying a sed globally ? :-)
>
>> +};
>> +
>> +enum mali_c55_rzr_pads {
> Same enums/structs use abbreviations, some don't. Consistency would
> help.
>
>> +	MALI_C55_RZR_SINK_PAD,
>> +	MALI_C55_RZR_SOURCE_PAD,
>> +	MALI_C55_RZR_SINK_BYPASS_PAD,
>> +	MALI_C55_RZR_NUM_PADS
>> +};
>> +
>> +struct mali_c55_resizer {
>> +	struct mali_c55 *mali_c55;
>> +	struct mali_c55_cap_dev *cap_dev;
>> +	enum mali_c55_resizer_ids id;
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pads[MALI_C55_RZR_NUM_PADS];
>> +	unsigned int num_routes;
>> +	bool streaming;
>> +};
>> +
>> +enum mali_c55_cap_devs {
>> +	MALI_C55_CAP_DEV_FR,
>> +	MALI_C55_CAP_DEV_DS,
>> +	MALI_C55_NUM_CAP_DEVS
>> +};
>> +
>> +struct mali_c55_fmt {
> mali_c55_format_info would be a better name I think, as this stores
> format information, not formats.
>
>> +	u32 fourcc;
>> +	unsigned int mbus_codes[2];
> A comment to explain why we have two media bus codes would be useful.
> You can document the whole structure if desired :-)
>
>> +	bool is_raw;
>> +	struct mali_c55_fmt_registers {
> Make it an anonymous structure, it's never used anywhere else.
>
>> +		unsigned int base_mode;
>> +		unsigned int uv_plane;
> If those are register field values, use u32 instead of unsigned int.
>
>> +	} registers;
> It's funny, we tend to abbreviate different things, I would have used
> "regs" here but written "format" in full in the structure name :-)
>
>> +};
>> +
>> +enum mali_c55_isp_bayer_order {
>> +	MALI_C55_BAYER_ORDER_RGGB,
>> +	MALI_C55_BAYER_ORDER_GRBG,
>> +	MALI_C55_BAYER_ORDER_GBRG,
>> +	MALI_C55_BAYER_ORDER_BGGR
> These are registers values too, they belong to mali-c55-registers.h.
>
>> +};
>> +
>> +struct mali_c55_isp_fmt {
> mali_c55_isp_format_info
>
>> +	u32 code;
>> +	enum v4l2_pixel_encoding encoding;
> Here you use v4l2_pixel_encoding, for mali_c55_fmt you use is_raw. Maybe
> pick the same option for both structures ?
>
>> +	enum mali_c55_isp_bayer_order order;
>> +};
>> +
>> +enum mali_c55_planes {
>> +	MALI_C55_PLANE_Y,
>> +	MALI_C55_PLANE_UV,
>> +	MALI_C55_NUM_PLANES
>> +};
>> +
>> +struct mali_c55_buffer {
>> +	struct vb2_v4l2_buffer vb;
>> +	bool plane_done[MALI_C55_NUM_PLANES];
> I think tracking the pending state would simplify the logic in
> mali_c55_set_plane_done(), which would become
>
> 	curr_buf->plane_pending[plane] = false;
>
> 	if (!curr_buf->plane_pending[0] && !curr_buf->plane_pending[1]) {
> 		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> 		cap_dev->buffers.curr = NULL;
> 		isp->frame_sequence++;
> 	}
>
> Or a counter may be even easier (and would consume less memory).
>
>> +	struct list_head queue;
>> +	u32 addrs[MALI_C55_NUM_PLANES];
> This stores DMA addresses, use dma_addr_t.
>
>> +};
>> +
>> +struct mali_c55_cap_dev {
>> +	struct mali_c55 *mali_c55;
>> +	struct mali_c55_resizer *rzr;
>> +	struct video_device vdev;
>> +	struct media_pad pad;
>> +	struct vb2_queue queue;
>> +	struct mutex lock;
>> +	unsigned int reg_offset;
> Manual handling of the offset everywhere, with parametric macros for the
> resizer register addresses, isn't very nice. Introduce resizer-specific
> accessors and capture-specific accessors (mali_c55_rsz_write(), ...)
> that take a mali_c55_resizer or mali_c55_cap_dev pointer, and handle the
> offset there. The register macros should loose their offset parameter.
>
> You could also use a single set of accessors that would become
> path/output-specific (mali_c55_path_write() ? mali_c55_output_write()
> ?), that may make the code easier to read.
>
> You can also replace reg_offset with a void __iomem * base, which would
> avoid the computation at runtime.
>
>> +
>> +	struct mali_c55_mode {
> Make the structure anonymous.
>
>> +		const struct mali_c55_fmt *capture_fmt;
>> +		struct v4l2_pix_format_mplane pix_mp;
>> +	} mode;
> What's a "mode" ? I think I'd name this
>
> 	struct {
> 		const struct mali_c55_fmt *info;
> 		struct v4l2_pix_format_mplane format;
> 	} format;
>
> Or you could just drop the structure and have
>
> 	const struct mali_c55_fmt *format_info;
> 	struct v4l2_pix_format_mplane format;
>
> or something similar.
>
>> +
>> +	struct {
>> +		spinlock_t lock;
>> +		struct list_head queue;
>> +		struct mali_c55_buffer *curr;
>> +		struct mali_c55_buffer *next;
>> +	} buffers;
>> +
>> +	bool streaming;
>> +};
>> +
>> +enum mali_c55_config_spaces {
>> +	MALI_C55_CONFIG_PING,
>> +	MALI_C55_CONFIG_PONG,
>> +	MALI_C55_NUM_CONFIG_SPACES
> The last enumerator is not used.
>
>> +};
>> +
>> +struct mali_c55_ctx {
> mali_c55_context ?
>
>> +	struct mali_c55 *mali_c55;
>> +	void *registers;
> Please document this structure and explain that this field points to a
> copy of the register space in system memory, I was about to write you're
> missing __iomem :-)
>
>> +	phys_addr_t base;
>> +	spinlock_t lock;
>> +	struct list_head list;
>> +};
>> +
>> +struct mali_c55 {
>> +	struct device *dev;
>> +	struct resource *res;
> You could possibly drop this field by passing the physical address of
> the register space from mali_c55_probe() to mali_c55_init_context() as a
> function parameter.
>
>> +	void __iomem *base;
>> +	struct dma_chan *channel;
>> +	struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
>> +
>> +	u16 capabilities;
>> +	struct media_device media_dev;
>> +	struct v4l2_device v4l2_dev;
>> +	struct media_pipeline pipe;
>> +
>> +	struct mali_c55_tpg tpg;
>> +	struct mali_c55_isp isp;
>> +	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
>> +	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
>> +
>> +	struct list_head contexts;
>> +	enum mali_c55_config_spaces next_config;
>> +};
>> +
>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
>> +		  bool force_hardware);
>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
>> +			  u32 mask, u32 val);
>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
>> +			  enum mali_c55_config_spaces cfg_space);
>> +
>> +int mali_c55_register_isp(struct mali_c55 *mali_c55);
>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55);
>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55);
>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>> +			     enum mali_c55_planes plane);
>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
>> +
>> +bool mali_c55_format_is_raw(unsigned int mbus_code);
>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
>> +
>> +const struct mali_c55_isp_fmt *
>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
>> +#define for_each_mali_isp_fmt(fmt)\
> #define for_each_mali_isp_fmt(fmt) \
>
>> +	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
> Looks like parentheses were on sale :-)
>
> 	for ((fmt) = NULL; (fmt) = mali_c55_isp_fmt_next(fmt); )
>
> This macro is used in two places only, in the mali-c55-isp.c file where
> open-coding the loop without using mali_c55_isp_fmt_next() would be more
> efficient, and in mali-c55-resizer.c where a function to return format
> i'th would be more efficient. I think you can drop the macro and the
> mali_c55_isp_fmt_next() function.
>
>> +
>> +#endif /* _MALI_C55_COMMON_H */
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>> new file mode 100644
>> index 000000000000..50caf5ee7474
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>> @@ -0,0 +1,767 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ARM Mali-C55 ISP Driver - Core driver code
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#include <linux/bitops.h>
>> +#include <linux/cleanup.h>
>> +#include <linux/clk.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/dmaengine.h>
>> +#include <linux/dma-mapping.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/iopoll.h>
>> +#include <linux/ioport.h>
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/of.h>
>> +#include <linux/of_reserved_mem.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/pm_runtime.h>
>> +#include <linux/scatterlist.h>
> I don't think this is needed.
>
> Missing slab.h.
>
>> +#include <linux/string.h>
>> +
>> +#include <media/media-entity.h>
>> +#include <media/v4l2-device.h>
>> +#include <media/videobuf2-dma-contig.h>
>> +
>> +#include "mali-c55-common.h"
>> +#include "mali-c55-registers.h"
>> +
>> +static const char * const mali_c55_interrupt_names[] = {
>> +	[MALI_C55_IRQ_ISP_START] = "ISP start",
>> +	[MALI_C55_IRQ_ISP_DONE] = "ISP done",
>> +	[MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
>> +	[MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
>> +	[MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
>> +	[MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
>> +	[MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
>> +	[MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
>> +	[MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
>> +	[MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
>> +	[MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
>> +	[MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
>> +	[MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
>> +	[MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
>> +	[MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
>> +	[MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
>> +	[MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
>> +	[MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
>> +	[MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
>> +	[MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
>> +	[MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
>> +	[MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
>> +	[MALI_C55_IRQ_DMA_ERROR] = "DMA error",
>> +	[MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
>> +	[MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
>> +	[MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
>> +};
>> +
>> +static unsigned int config_space_addrs[] = {
> const
>
>> +	[MALI_C55_CONFIG_PING] = 0x0AB6C,
>> +	[MALI_C55_CONFIG_PONG] = 0x22B2C,
> Lowercase hex constants.
>
> Don't the values belong to mali-c55-registers.h ?
>
>> +};
>> +
>> +/* System IO
> /*
>   * System IO
>
>> + *
>> + * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
>> + * and 'pong'), with the  expectation that the 'active' space will be left
> s/the  /the /
>
>> + * untouched whilst a frame is being processed and the 'inactive' space
>> + * configured ready to be passed during the blanking period before the next
> s/to be passed/to be switched to/ ?
>
>> + * frame processing starts. These spaces should ideally be set via DMA transfer
>> + * from a buffer rather than through individual register set operations. There
>> + * is also a shared global register space which should be set normally. Of
>> + * course, the ISP might be included in a system which lacks a suitable DMA
>> + * engine, and the second configuration space might not be fitted at all, which
>> + * means we need to support four scenarios:
>> + *
>> + * 1. Multi config space, with DMA engine.
>> + * 2. Multi config space, no DMA engine.
>> + * 3. Single config space, with DMA engine.
>> + * 4. Single config space, no DMA engine.
>> + *
>> + * The first case is very easy, but the rest present annoying problems. The best
>> + * way to solve them seems to be simply to replicate the concept of DMAing over
>> + * the configuration buffer even if there's no DMA engine on the board, for
>> + * which we rely on memcpy. To facilitate this any read/write call that is made
>> + * to an address within those config spaces should infact be directed to a
>> + * buffer that was allocated to hold them rather than the IO memory itself. The
>> + * actual copy of that buffer to IO mem will happen on interrupt.
>> + */
>> +
>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
>> +{
>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>> +
>> +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
>> +		spin_lock(&ctx->lock);
>> +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
>> +		((u32 *)ctx->registers)[addr] = val;
>> +		spin_unlock(&ctx->lock);
>> +
>> +		return;
>> +	}
> Ouch. This is likely the second comment you really won't like (after the
> comment regarding mali_c55_update_bits() at the very top). I apologize
> in advance.
>
> I really don't like this. Directing writes either to hardware registers
> or to the shadow registers in the context makes the callers of the
> read/write accessors very hard to read. The probe code, for instance,
> mixes writes to hardware registers and writes to the context shadow
> registers to initialize the value of some of the shadow registers.
>
> I'd like to split the read/write accessors into functions that access
> the hardware registers (that's easy) and functions that access the
> shadow registers. I think the latter should receive a mali_c55_ctx
> pointer instead of a mali_c55 pointer to prepare for multi-context
> support.
>
> You can add WARN_ON() guards to the two sets of functions, to ensure
> that no register from the "other" space gets passed to the wrong
> function by mistake.
>
>> +
>> +	writel(val, mali_c55->base + addr);
>> +}
>> +
>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
>> +		  bool force_hardware)
> force_hardware is never set to true.
>
>> +{
>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>> +	u32 val;
>> +
>> +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
>> +		spin_lock(&ctx->lock);
>> +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
>> +		val = ((u32 *)ctx->registers)[addr];
>> +		spin_unlock(&ctx->lock);
>> +
>> +		return val;
>> +	}
>> +
>> +	return readl(mali_c55->base + addr);
>> +}
>> +
>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
>> +			  u32 mask, u32 val)
>> +{
>> +	u32 orig, tmp;
>> +
>> +	orig = mali_c55_read(mali_c55, addr, false);
>> +
>> +	tmp = orig & ~mask;
>> +	tmp |= (val << (ffs(mask) - 1)) & mask;
>> +
>> +	if (tmp != orig)
>> +		mali_c55_write(mali_c55, addr, tmp);
>> +}
>> +
>> +static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
>> +			     dma_addr_t dst, enum dma_data_direction dir)
>> +{
>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>> +	struct dma_async_tx_descriptor *tx;
>> +	enum dma_status status;
>> +	dma_cookie_t cookie;
>> +
>> +	tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
>> +				       MALI_C55_CONFIG_SPACE_SIZE, 0);
>> +	if (!tx) {
>> +		dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
>> +		return -EIO;
>> +	}
>> +
>> +	cookie = dmaengine_submit(tx);
>> +	if (dma_submit_error(cookie)) {
>> +		dev_err(mali_c55->dev, "error submitting dma transfer\n");
>> +		return -EIO;
>> +	}
>> +
>> +	status = dma_sync_wait(mali_c55->channel, cookie);
> I've just realized this performs a busy-wait :-S See the comment in the
> probe function about the threaded IRQ handler. I think we'll need to
> rework all this. It could be done on top though.
>
>> +	if (status != DMA_COMPLETE) {
>> +		dev_err(mali_c55->dev, "dma transfer failed\n");
>> +		return -EIO;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
>> +			     enum mali_c55_config_spaces cfg_space)
>> +{
>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>> +	struct device *dma_dev = mali_c55->channel->device->dev;
>> +	dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
>> +	dma_addr_t dst;
>> +	int ret;
>> +
>> +	guard(spinlock)(&ctx->lock);
>> +
>> +	dst = dma_map_single(dma_dev, ctx->registers,
>> +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
>> +	if (dma_mapping_error(dma_dev, dst)) {
>> +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
>> +		return -EIO;
>> +	}
>> +
>> +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
>> +	dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
>> +			 DMA_FROM_DEVICE);
>> +
>> +	return ret;
>> +}
>> +
>> +static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
>> +		       enum mali_c55_config_spaces cfg_space)
>> +{
>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>> +	struct device *dma_dev = mali_c55->channel->device->dev;
>> +	dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
>> +	dma_addr_t src;
>> +	int ret;
>> +
>> +	guard(spinlock)(&ctx->lock);
> The code below can take a large amount of time, holding a spinlock will
> disable interrupts on the local CPU, that's not good :-(
>
>> +
>> +	src = dma_map_single(dma_dev, ctx->registers,
>> +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
>> +	if (dma_mapping_error(dma_dev, src)) {
>> +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
>> +		return -EIO;
>> +	}
>> +
>> +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
>> +	dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
>> +			 DMA_TO_DEVICE);
>> +
>> +	return ret;
>> +}
>> +
>> +static int mali_c55_config_read(struct mali_c55_ctx *ctx,
>> +				enum mali_c55_config_spaces cfg_space)
>> +{
>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>> +
>> +	if (mali_c55->channel) {
>> +		return mali_c55_dma_read(ctx, cfg_space);
> As this function is used at probe time only, to initialize the context,
> I think DMA is overkill.
>
>> +	} else {
>> +		memcpy_fromio(ctx->registers,
>> +			      mali_c55->base + config_space_addrs[cfg_space],
>> +			      MALI_C55_CONFIG_SPACE_SIZE);
>> +		return 0;
>> +	}
>> +}
>> +
>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
>> +			  enum mali_c55_config_spaces cfg_space)
>> +{
>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>> +
>> +	if (mali_c55->channel) {
>> +		return mali_c55_dma_write(ctx, cfg_space);
>> +	} else {
>> +		memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
>> +			    ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
>> +		return 0;
>> +	}
> Could you measure the time it typically takes to write the registers
> using DMA compared to using memcpy_toio() ?
>
>> +}
>> +
>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
>> +{
>> +	return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);
> I think it's too early to tell how multi-context support will look like.
> I'm fine keeping mali_c55_get_active_context() as changing that would be
> very intrusive (even if I think it will need to be changed), but the
> list of contexts is neither the mechanism we'll use, nor something we
> need now. Drop the list, embed the context in struct mali_c55, and
> return the pointer to that single context from this function.
>
>> +}
>> +
>> +static void mali_c55_remove_links(struct mali_c55 *mali_c55)
>> +{
>> +	unsigned int i;
>> +
>> +	media_entity_remove_links(&mali_c55->tpg.sd.entity);
>> +	media_entity_remove_links(&mali_c55->isp.sd.entity);
>> +
>> +	for (i = 0; i < MALI_C55_NUM_RZRS; i++)
>> +		media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
>> +
>> +	for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
>> +		media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
>> +}
>> +
>> +static int mali_c55_create_links(struct mali_c55 *mali_c55)
>> +{
>> +	int ret;
>> +
>> +	/* Test pattern generator to ISP */
>> +	ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
>> +				    &mali_c55->isp.sd.entity,
>> +				    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
>> +		goto err_remove_links;
>> +	}
>> +
>> +	/* Full resolution resizer pipe. */
>> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>> +			MALI_C55_ISP_PAD_SOURCE,
>> +			&mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
>> +			MALI_C55_RZR_SINK_PAD,
>> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
>> +		goto err_remove_links;
>> +	}
>> +
>> +	/* Full resolution bypass. */
>> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>> +				    MALI_C55_ISP_PAD_SOURCE_BYPASS,
>> +				    &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
>> +				    MALI_C55_RZR_SINK_BYPASS_PAD,
>> +				    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
>> +		goto err_remove_links;
>> +	}
>> +
>> +	/* Resizer pipe to video capture nodes. */
>> +	ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
>> +			MALI_C55_RZR_SOURCE_PAD,
>> +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
>> +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev,
>> +			"failed to link FR resizer and video device\n");
>> +		goto err_remove_links;
>> +	}
>> +
>> +	/* The downscale pipe is an optional hardware block */
>> +	if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
>> +		ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>> +			MALI_C55_ISP_PAD_SOURCE,
>> +			&mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
>> +			MALI_C55_RZR_SINK_PAD,
>> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>> +		if (ret) {
>> +			dev_err(mali_c55->dev,
>> +				"failed to link ISP and DS resizer\n");
>> +			goto err_remove_links;
>> +		}
>> +
>> +		ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
>> +			MALI_C55_RZR_SOURCE_PAD,
>> +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
>> +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>> +		if (ret) {
>> +			dev_err(mali_c55->dev,
>> +				"failed to link DS resizer and video device\n");
>> +			goto err_remove_links;
>> +		}
>> +	}
>> +
>> +	return 0;
>> +
>> +err_remove_links:
>> +	mali_c55_remove_links(mali_c55);
>> +	return ret;
>> +}
>> +
>> +static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
>> +{
>> +	mali_c55_unregister_tpg(mali_c55);
>> +	mali_c55_unregister_isp(mali_c55);
>> +	mali_c55_unregister_resizers(mali_c55);
>> +	mali_c55_unregister_capture_devs(mali_c55);
>> +}
>> +
>> +static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>> +{
>> +	int ret;
>> +
>> +	ret = mali_c55_register_tpg(mali_c55);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = mali_c55_register_isp(mali_c55);
>> +	if (ret)
>> +		goto err_unregister_entities;
>> +
>> +	ret = mali_c55_register_resizers(mali_c55);
>> +	if (ret)
>> +		goto err_unregister_entities;
>> +
>> +	ret = mali_c55_register_capture_devs(mali_c55);
>> +	if (ret)
>> +		goto err_unregister_entities;
>> +
>> +	ret = mali_c55_create_links(mali_c55);
>> +	if (ret)
>> +		goto err_unregister_entities;
>> +
>> +	return 0;
>> +
>> +err_unregister_entities:
>> +	mali_c55_unregister_entities(mali_c55);
>> +
>> +	return ret;
>> +}
>> +
>> +static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
>> +{
>> +	u32 product, version, revision, capabilities;
>> +
>> +	product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
>> +	version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
>> +	revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
>> +
>> +	dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
>> +		 product, version, revision);
>> +
>> +	capabilities = mali_c55_read(mali_c55,
>> +				     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
>> +				     false);
>> +	mali_c55->capabilities = (capabilities & 0xffff);
>> +
>> +	/* TODO: Might as well start some debugfs */
> If it's just to expose the version and capabilities, I think that's
> overkill. It's not needed for debug purpose (you can get it from the
> kernel log already). debugfs isn't meant to be accessible in production,
> so an application that would need access to the information wouldn't be
> able to use it.
>
>> +	dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);
> Combine the two messages into one.
>
>> +	return version;
>> +}
>> +
>> +static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>> +	u32 curr_config, next_config;
>> +
>> +	curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
>> +	curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
>> +		      >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
>> +	next_config = curr_config ^ 1;
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>> +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
>> +	mali_c55_config_write(ctx, next_config ?
>> +			      MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
>> +}
>> +
>> +static irqreturn_t mali_c55_isr(int irq, void *context)
>> +{
>> +	struct device *dev = context;
>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>> +	u32 interrupt_status;
>> +	unsigned int i, j;
>> +
>> +	interrupt_status = mali_c55_read(mali_c55,
>> +					 MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
>> +					 false);
>> +	if (!interrupt_status)
>> +		return IRQ_NONE;
>> +
>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
>> +		       interrupt_status);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
>> +
>> +	for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
>> +		if (!(interrupt_status & (1 << i)))
>> +			continue;
>> +
>> +		switch (i) {
>> +		case MALI_C55_IRQ_ISP_START:
>> +			mali_c55_isp_queue_event_sof(mali_c55);
>> +
>> +			for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
>> +				mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
>> +
>> +			mali_c55_swap_next_config(mali_c55);
>> +
>> +			break;
>> +		case MALI_C55_IRQ_ISP_DONE:
>> +			/*
>> +			 * TODO: Where the ISP has no Pong config fitted, we'd
>> +			 * have to do the mali_c55_swap_next_config() call here.
>> +			 */
>> +			break;
>> +		case MALI_C55_IRQ_FR_Y_DONE:
>> +			mali_c55_set_plane_done(
>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
>> +				MALI_C55_PLANE_Y);
>> +			break;
>> +		case MALI_C55_IRQ_FR_UV_DONE:
>> +			mali_c55_set_plane_done(
>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
>> +				MALI_C55_PLANE_UV);
>> +			break;
>> +		case MALI_C55_IRQ_DS_Y_DONE:
>> +			mali_c55_set_plane_done(
>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
>> +				MALI_C55_PLANE_Y);
>> +			break;
>> +		case MALI_C55_IRQ_DS_UV_DONE:
>> +			mali_c55_set_plane_done(
>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
>> +				MALI_C55_PLANE_UV);
>> +			break;
>> +		default:
>> +			/*
>> +			 * Only the above interrupts are currently unmasked. If
>> +			 * we receive anything else here then something weird
>> +			 * has gone on.
>> +			 */
>> +			dev_err(dev, "masked interrupt %s triggered\n",
>> +				mali_c55_interrupt_names[i]);
>> +		}
>> +	}
>> +
>> +	return IRQ_HANDLED;
>> +}
>> +
>> +static int mali_c55_init_context(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_ctx *ctx;
>> +	int ret;
>> +
>> +	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
>> +	if (!ctx) {
>> +		dev_err(mali_c55->dev, "failed to allocate new context\n");
> No need for an error message when memory allocation fails.
>
>> +		return -ENOMEM;
>> +	}
>> +
>> +	ctx->base = mali_c55->res->start;
>> +	ctx->mali_c55 = mali_c55;
>> +
>> +	ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
>> +				 GFP_KERNEL | GFP_DMA);
>> +	if (!ctx->registers) {
>> +		ret = -ENOMEM;
>> +		goto err_free_ctx;
>> +	}
>> +
>> +	/*
>> +	 * The allocated memory is empty, we need to load the default
>> +	 * register settings. We just read Ping; it's identical to Pong.
>> +	 */
>> +	ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
>> +	if (ret)
>> +		goto err_free_registers;
>> +
>> +	list_add_tail(&ctx->list, &mali_c55->contexts);
>> +
>> +	/*
>> +	 * Some features of the ISP need to be disabled by default and only
>> +	 * enabled at the same time as they're configured by a parameters buffer
>> +	 */
>> +
>> +	/* Bypass the sqrt and square compression and expansion modules */
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
>> +			     MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
>> +			     MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
>> +
>> +	/* Bypass the temper module */
>> +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
>> +		       MALI_C55_REG_BYPASS_2_TEMPER);
>> +
>> +	/* Bypass the colour noise reduction  */
>> +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
>> +		       MALI_C55_REG_BYPASS_4_CNR);
>> +
>> +	/* Disable the sinter module */
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
>> +			     MALI_C55_SINTER_ENABLE_MASK, 0x00);
>> +
>> +	/* Disable the RGB Gamma module for each output */
>> +	mali_c55_write(
>> +		mali_c55,
>> +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
>> +		0x00);
>> +	mali_c55_write(
>> +		mali_c55,
>> +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
>> +		0x00);
>> +
>> +	/* Disable the colour correction matrix */
>> +	mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
>> +
>> +	return 0;
>> +
>> +err_free_registers:
>> +	kfree(ctx->registers);
>> +err_free_ctx:
>> +	kfree(ctx);
>> +
>> +	return ret;
>> +}
>> +
>> +static int mali_c55_runtime_resume(struct device *dev)
>> +{
>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>> +	int ret;
>> +
>> +	ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
>> +				      mali_c55->clks);
>> +	if (ret)
>> +		dev_err(mali_c55->dev, "failed to enable clocks\n");
>> +
>> +	return ret;
>> +}
>> +
>> +static int mali_c55_runtime_suspend(struct device *dev)
>> +{
>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>> +
>> +	clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
>> +	return 0;
>> +}
>> +
>> +static const struct dev_pm_ops mali_c55_pm_ops = {
>> +	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
>> +				pm_runtime_force_resume)
>> +	SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
>> +			   NULL)
>> +};
>> +
>> +static int mali_c55_probe(struct platform_device *pdev)
>> +{
>> +	struct device *dev = &pdev->dev;
>> +	struct mali_c55 *mali_c55;
>> +	dma_cap_mask_t mask;
>> +	u32 version;
>> +	int ret;
>> +	u32 val;
>> +
>> +	mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
>> +	if (!mali_c55)
>> +		return dev_err_probe(dev, -ENOMEM,
>> +				     "failed to allocate memory\n");
> 		return -ENOMEM;
>
> There's no need to print messages for memory allocation failures.
>
>> +
>> +	mali_c55->dev = dev;
>> +	platform_set_drvdata(pdev, mali_c55);
>> +
>> +	mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
>> +								&mali_c55->res);
>> +	if (IS_ERR(mali_c55->base))
>> +		return dev_err_probe(dev, PTR_ERR(mali_c55->base),
>> +				     "failed to map IO memory\n");
>> +
>> +	ret = platform_get_irq(pdev, 0);
>> +	if (ret < 0)
>> +		return dev_err_probe(dev, ret, "failed to get interrupt num\n");
> s/ num// or s/num/number/
>
>> +
>> +	ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
>> +					mali_c55_isr, IRQF_ONESHOT,
>> +					dev_driver_string(&pdev->dev),
>> +					&pdev->dev);
> Requested the IRQ should be done much lower, after you have initialized
> everything, or an IRQ that would fire early would have really bad
> consequences.
>
> A comment to explain why you need a threaded interrupt handler would be
> good. I assume it is due only to the need to transfer the registers
> using DMA. I wonder if we should then split the interrupt handler in
> two, with a non-threaded part for the operations that can run quickly,
> and a threaded part for the reprogramming.
>
> It may also be that we could just start the DMA transfer in the
> non-threaded handler without waiting synchronously for it to complete.
> That would be a bigger change, and would require checking race
> conditions carefully. On the other hand, I'm a bit concerned about the
> current implementation, have you tested what happens if the DMA transfer
> takes too long to complete, and spans frame boundaries ? This part could
> be addressed by a patch on top of this one.
>
>> +	if (ret)
>> +		return dev_err_probe(dev, ret, "failed to request irq\n");
>> +
>> +	for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
>> +		mali_c55->clks[i].id = mali_c55_clk_names[i];
>> +
>> +	ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
>> +	if (ret)
>> +		return dev_err_probe(dev, ret, "failed to acquire clocks\n");
>> +
>> +	pm_runtime_enable(&pdev->dev);
>> +
>> +	ret = pm_runtime_resume_and_get(&pdev->dev);
>> +	if (ret)
>> +		goto err_pm_runtime_disable;
>> +
>> +	of_reserved_mem_device_init(dev);
> I'd move this, the vb2_dma_contig_set_max_seg_size() call and the
> dma_cap_* calls before pm_runtime_enable() as they don't need the device
> to be powered.
>
>> +	version = mali_c55_check_hwcfg(mali_c55);
>> +	vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
>> +
>> +	/* Use "software only" context management. */
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>> +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
>> +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> You handle that in mali_c55_isp_start(), does the register have to be
> set here too ?
>
>> +
>> +	dma_cap_zero(mask);
>> +	dma_cap_set(DMA_MEMCPY, mask);
>> +
>> +	/*
>> +	 * No error check, because we will just fallback on memcpy if there is
>> +	 * no usable DMA channel on the system.
>> +	 */
>> +	mali_c55->channel = dma_request_channel(mask, NULL, NULL);
>> +
>> +	INIT_LIST_HEAD(&mali_c55->contexts);
>> +	ret = mali_c55_init_context(mali_c55);
>> +	if (ret)
>> +		goto err_release_dma_channel;
>> +
> I'd move all the code from here ...
>
>> +	mali_c55->media_dev.dev = dev;
>> +	strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
>> +		sizeof(mali_c55->media_dev.model));
>> +	mali_c55->media_dev.hw_revision = version;
>> +
>> +	media_device_init(&mali_c55->media_dev);
>> +	ret = media_device_register(&mali_c55->media_dev);
>> +	if (ret)
>> +		goto err_cleanup_media_device;
>> +
>> +	mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
>> +	ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
>> +	if (ret) {
>> +		dev_err(dev, "failed to register V4L2 device\n");
>> +		goto err_unregister_media_device;
>> +	};
>> +
>> +	ret = mali_c55_register_entities(mali_c55);
>> +	if (ret) {
>> +		dev_err(dev, "failed to register entities\n");
>> +		goto err_unregister_v4l2_device;
>> +	}
> ... to here to a separate function, or maybe fold it all in
> mali_c55_register_entities() (which should the be renamed). Same thing
> for the cleanup code.
>
>> +
>> +	/* Set safe stop to ensure we're in a non-streaming state */
>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>> +		       MALI_C55_INPUT_SAFE_STOP);
>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>> +
>> +	/*
>> +	 * We're ready to process interrupts. Clear any that are set and then
>> +	 * unmask them for processing.
>> +	 */
>> +	mali_c55_write(mali_c55, 0x30, 0xffffffff);
>> +	mali_c55_write(mali_c55, 0x34, 0xffffffff);
>> +	mali_c55_write(mali_c55, 0x40, 0x01);
>> +	mali_c55_write(mali_c55, 0x40, 0x00);
> Please replace the register addresses with macros.
>
>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);
> The value should use the interrupt bits macros.
>
>> +
>> +	pm_runtime_put(&pdev->dev);
> Once power gets cut, the registers your programmed above may be lost. I
> think you should programe them in the runtime PM resume handler.
>
>> +
>> +	return 0;
>> +
>> +err_unregister_v4l2_device:
>> +	v4l2_device_unregister(&mali_c55->v4l2_dev);
>> +err_unregister_media_device:
>> +	media_device_unregister(&mali_c55->media_dev);
>> +err_cleanup_media_device:
>> +	media_device_cleanup(&mali_c55->media_dev);
>> +err_release_dma_channel:
>> +	dma_release_channel(mali_c55->channel);
>> +err_pm_runtime_disable:
>> +	pm_runtime_disable(&pdev->dev);
>> +
>> +	return ret;
>> +}
>> +
>> +static void mali_c55_remove(struct platform_device *pdev)
>> +{
>> +	struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
>> +	struct mali_c55_ctx *ctx, *tmp;
>> +
>> +	list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
>> +		list_del(&ctx->list);
>> +		kfree(ctx->registers);
>> +		kfree(ctx);
>> +	}
>> +
>> +	mali_c55_remove_links(mali_c55);
>> +	mali_c55_unregister_entities(mali_c55);
>> +	v4l2_device_put(&mali_c55->v4l2_dev);
>> +	media_device_unregister(&mali_c55->media_dev);
>> +	media_device_cleanup(&mali_c55->media_dev);
>> +	dma_release_channel(mali_c55->channel);
>> +}
>> +
>> +static const struct of_device_id mali_c55_of_match[] = {
>> +	{ .compatible = "arm,mali-c55", },
>> +	{},
> 	{ /* Sentinel */ },
>
>> +};
>> +MODULE_DEVICE_TABLE(of, mali_c55_of_match);
>> +
>> +static struct platform_driver mali_c55_driver = {
>> +	.driver = {
>> +		.name = "mali-c55",
>> +		.of_match_table = of_match_ptr(mali_c55_of_match),
> Drop of_match_ptr().
>
>> +		.pm = &mali_c55_pm_ops,
>> +	},
>> +	.probe = mali_c55_probe,
>> +	.remove_new = mali_c55_remove,
>> +};
>> +
>> +module_platform_driver(mali_c55_driver);
> Blank line.
>
>> +MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
>> +MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
>> +MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
>> +MODULE_LICENSE("GPL");
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>> new file mode 100644
>> index 000000000000..ea8b7b866e7a
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>> @@ -0,0 +1,611 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ARM Mali-C55 ISP Driver - Image signal processor
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#include <linux/delay.h>
>> +#include <linux/iopoll.h>
>> +#include <linux/property.h>
>> +#include <linux/string.h>
>> +
>> +#include <media/media-entity.h>
>> +#include <media/v4l2-common.h>
>> +#include <media/v4l2-event.h>
>> +#include <media/v4l2-mc.h>
>> +#include <media/v4l2-subdev.h>
>> +
>> +#include "mali-c55-common.h"
>> +#include "mali-c55-registers.h"
>> +
>> +static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
>> +	{
>> +		.code = MEDIA_BUS_FMT_SRGGB20_1X20,
>> +		.order = MALI_C55_BAYER_ORDER_RGGB,
>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>> +	},
>> +	{
>> +		.code = MEDIA_BUS_FMT_SGRBG20_1X20,
>> +		.order = MALI_C55_BAYER_ORDER_GRBG,
>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>> +	},
>> +	{
>> +		.code = MEDIA_BUS_FMT_SGBRG20_1X20,
>> +		.order = MALI_C55_BAYER_ORDER_GBRG,
>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>> +	},
>> +	{
>> +		.code = MEDIA_BUS_FMT_SBGGR20_1X20,
>> +		.order = MALI_C55_BAYER_ORDER_BGGR,
>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>> +	},
>> +	{
>> +		.code = MEDIA_BUS_FMT_RGB202020_1X60,
>> +		.order = 0, /* Not relevant for this format */
>> +		.encoding = V4L2_PIXEL_ENC_RGB,
>> +	}
>> +	/*
>> +	 * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
>> +	 * also support YUV input from a sensor passed-through to the output. At
>> +	 * present we have no mechanism to test that though so it may have to
>> +	 * wait a while...
>> +	 */
>> +};
>> +
>> +const struct mali_c55_isp_fmt *
>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
>> +{
>> +	if (!fmt)
>> +		fmt = &mali_c55_isp_fmts[0];
>> +	else
>> +		fmt++;
>> +
>> +	for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
>> +		return fmt;
> That's peculiar.
>
> 	if (!fmt)
> 		fmt = &mali_c55_isp_fmts[0];
> 	else if (fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)])
> 		return ++fmt;
> 	else
> 		return NULL;
>
>> +
>> +	return NULL;
>> +}
>> +
>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
>> +{
>> +	const struct mali_c55_isp_fmt *isp_fmt;
>> +
>> +	for_each_mali_isp_fmt(isp_fmt) {
> I would open-code the loop instead of using the macro, like you do
> below. It will be more efficient.
>
>> +		if (isp_fmt->code == mbus_code)
>> +			return true;
>> +	}
>> +
>> +	return false;
>> +}
>> +
>> +static const struct mali_c55_isp_fmt *
>> +mali_c55_isp_get_mbus_config_by_code(u32 code)
>> +{
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
>> +		if (mali_c55_isp_fmts[i].code == code)
>> +			return &mali_c55_isp_fmts[i];
>> +	}
>> +
>> +	return NULL;
>> +}
>> +
>> +static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
>> +{
>> +	u32 val;
>> +
>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);
> 	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> 		       MALI_C55_INPUT_SAFE_STOP);
>
>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>> +}
>> +
>> +static int mali_c55_isp_start(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>> +	const struct mali_c55_isp_fmt *cfg;
>> +	struct v4l2_mbus_framefmt *format;
> const
>
>> +	struct v4l2_subdev_state *state;
>> +	struct v4l2_rect *crop;
> const
>
>> +	struct v4l2_subdev *sd;
>> +	u32 val;
>> +	int ret;
>> +
>> +	sd = &mali_c55->isp.sd;
> Assign when declaring the variable.
>
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>> +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
>> +
>> +	/* Apply input windowing */
>> +	state = v4l2_subdev_get_locked_active_state(sd);
> Using .enable_streams() (see below) you'll get this for free.
>
>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>> +	format = v4l2_subdev_state_get_format(state,
>> +					      MALI_C55_ISP_PAD_SINK_VIDEO);
>> +	cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
>> +
>> +	mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
>> +		       MALI_C55_HC_START(crop->left));
>> +	mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
>> +		       MALI_C55_HC_SIZE(crop->width));
>> +	mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
>> +		       MALI_C55_VC_START(crop->top) |
>> +		       MALI_C55_VC_SIZE(crop->height));
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
>> +			     MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
>> +			     MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
>> +			     MALI_C55_BAYER_ORDER_MASK, cfg->order);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
>> +			     MALI_C55_INPUT_WIDTH_MASK,
>> +			     MALI_C55_INPUT_WIDTH_20BIT);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>> +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
>> +			     cfg->encoding == V4L2_PIXEL_ENC_RGB ?
>> +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
>> +
>> +	ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "failed to DMA config\n");
>> +		return ret;
>> +	}
>> +
>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>> +		       MALI_C55_INPUT_SAFE_START);
>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> Should you return an error in case of timeout ?
>
>> +
>> +	return 0;
>> +}
>> +
>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)
> Why is this not handled wired to .s_stream() ? Or better,
> .enable_streams() and .disable_streams().
>
>> +{
>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>> +	struct v4l2_subdev *sd;
>> +
>> +	if (isp->remote_src) {
>> +		sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
>> +		v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
>> +	}
>> +	isp->remote_src = NULL;
>> +
>> +	mali_c55_isp_stop(mali_c55);
>> +}
>> +
>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
>> +{
>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>> +	struct media_pad *sink_pad;
>> +	struct v4l2_subdev *sd;
>> +	int ret;
>> +
>> +	sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
>> +	isp->remote_src = media_pad_remote_pad_unique(sink_pad);
>> +	if (IS_ERR(isp->remote_src)) {
> If you set the MUST_CONNECT flag for the MALI_C55_ISP_PAD_SINK_VIDEO pad
> I think you can drop this check.
>
>> +		dev_err(mali_c55->dev, "Failed to get source for ISP\n");
>> +		return PTR_ERR(isp->remote_src);
>> +	}
>> +
>> +	sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
>> +
>> +	isp->frame_sequence = 0;
>> +	ret = mali_c55_isp_start(mali_c55);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "Failed to start ISP\n");
>> +		isp->remote_src = NULL;
>> +		return ret;
>> +	}
>> +
>> +	/*
>> +	 * We only support a single input stream, so we can just enable the 1st
>> +	 * entry in the streams mask.
>> +	 */
>> +	ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "Failed to start ISP source\n");
>> +		mali_c55_isp_stop(mali_c55);
>> +		return ret;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
>> +				       struct v4l2_subdev_state *state,
>> +				       struct v4l2_subdev_mbus_code_enum *code)
>> +{
>> +	/*
>> +	 * Only the internal RGB processed format is allowed on the regular
>> +	 * processing source pad.
>> +	 */
>> +	if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
>> +		if (code->index)
>> +			return -EINVAL;
>> +
>> +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
>> +		return 0;
>> +	}
>> +
>> +	/* On the sink and bypass pads all the supported formats are allowed. */
>> +	if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
>> +		return -EINVAL;
>> +
>> +	code->code = mali_c55_isp_fmts[code->index].code;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
>> +					struct v4l2_subdev_state *state,
>> +					struct v4l2_subdev_frame_size_enum *fse)
>> +{
>> +	const struct mali_c55_isp_fmt *cfg;
>> +
>> +	if (fse->index > 0)
>> +		return -EINVAL;
>> +
>> +	/*
>> +	 * Only the internal RGB processed format is allowed on the regular
>> +	 * processing source pad.
>> +	 *
>> +	 * On the sink and bypass pads all the supported formats are allowed.
>> +	 */
>> +	if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
>> +		if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
>> +			return -EINVAL;
>> +	} else {
>> +		cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
>> +		if (!cfg)
>> +			return -EINVAL;
>> +	}
>> +
>> +	fse->min_width = MALI_C55_MIN_WIDTH;
>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
>> +	fse->max_width = MALI_C55_MAX_WIDTH;
>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
>> +				struct v4l2_subdev_state *state,
>> +				struct v4l2_subdev_format *format)
>> +{
>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>> +	struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
>> +	const struct mali_c55_isp_fmt *cfg;
>> +	struct v4l2_rect *crop;
>> +
>> +	/*
>> +	 * Disallow set_fmt on the source pads; format is fixed and the sizes
>> +	 * are the result of applying the sink crop rectangle to the sink
>> +	 * format.
>> +	 */
>> +	if (format->pad)
> 	if (format->pad != MALI_C55_ISP_PAD_SINK_VIDEO)
>
>> +		return v4l2_subdev_get_fmt(sd, state, format);
>> +
>> +	cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
>> +	if (!cfg)
>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>> +	fmt->field = V4L2_FIELD_NONE;
> Do you intentionally allow the colorspace fields to be overwritten to
> any value ? rkisp1_isp_set_src_fmt() and rkisp1_isp_set_sink_fmt() may
> show you how this could be handled.
>
>> +
>> +	/*
>> +	 * Clamp sizes in the accepted limits and clamp the crop rectangle in
>> +	 * the new sizes.
>> +	 */
>> +	clamp_t(unsigned int, fmt->width,
>> +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>> +	clamp_t(unsigned int, fmt->width,
>> +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> clamp_t() returns a value, which you ignore. Those are no-ops. You meant
>
> 	fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> 			     MALI_C55_MAX_WIDTH);
> 	fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> 			      MALI_C55_MAX_HEIGHT);
>
> Same for every use of clamp_t() through the whole driver.
>
> Also, do you need clamp_t() ? I think all values are unsigned int, you
> can use clamp().
>
> Are there any alignment constraints, such a multiples of two for bayer
> formats ? Same in all the other locations where applicable.
>
>> +
>> +	sink_fmt = v4l2_subdev_state_get_format(state,
>> +						MALI_C55_ISP_PAD_SINK_VIDEO);
>> +	*sink_fmt = *fmt;
>> +
>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>> +	crop->left = 0;
>> +	crop->top = 0;
>> +	crop->width = fmt->width;
>> +	crop->height = fmt->height;
>> +
>> +	/*
>> +	 * Propagate format to source pads. On the 'regular' output pad use
>> +	 * the internal RGB processed format, while on the bypass pad simply
>> +	 * replicate the ISP sink format. The sizes on both pads are the same as
>> +	 * the ISP sink crop rectangle.
>> +	 */
> Colorspace information will need to be propagated too.
>
>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
>> +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>> +	src_fmt->width = crop->width;
>> +	src_fmt->height = crop->height;
>> +
>> +	src_fmt = v4l2_subdev_state_get_format(state,
>> +					       MALI_C55_ISP_PAD_SOURCE_BYPASS);
>> +	src_fmt->code = fmt->code;
>> +	src_fmt->width = crop->width;
>> +	src_fmt->height = crop->height;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
>> +				      struct v4l2_subdev_state *state,
>> +				      struct v4l2_subdev_selection *sel)
>> +{
>> +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> 	sel->pad != MALI_C55_ISP_PAD_SINK_VIDEO
>
>> +		return -EINVAL;
>> +
>> +	sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
>> +				      struct v4l2_subdev_state *state,
>> +				      struct v4l2_subdev_selection *sel)
>> +{
>> +	struct v4l2_mbus_framefmt *src_fmt;
>> +	struct v4l2_mbus_framefmt *fmt;
> const
>
>> +	struct v4l2_rect *crop;
>> +
>> +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> Ditto.
>
>> +		return -EINVAL;
>> +
>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>> +
>> +	clamp_t(unsigned int, sel->r.left, 0, fmt->width);
>> +	clamp_t(unsigned int, sel->r.top, 0, fmt->height);
>> +	clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
>> +		fmt->width - sel->r.left);
>> +	clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
>> +		fmt->height - sel->r.top);
>> +
>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>> +	*crop = sel->r;
>> +
>> +	/* Propagate the crop rectangle sizes to the source pad format. */
>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
>> +	src_fmt->width = crop->width;
>> +	src_fmt->height = crop->height;
> Can you confirm that cropping doesn't affect the bypass path ? And maybe
> add a comment to mention it.
>
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
>> +	.enum_mbus_code		= mali_c55_isp_enum_mbus_code,
>> +	.enum_frame_size	= mali_c55_isp_enum_frame_size,
>> +	.get_fmt		= v4l2_subdev_get_fmt,
>> +	.set_fmt		= mali_c55_isp_set_fmt,
>> +	.get_selection		= mali_c55_isp_get_selection,
>> +	.set_selection		= mali_c55_isp_set_selection,
>> +	.link_validate		= v4l2_subdev_link_validate_default,
>> +};
>> +
>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
>> +{
>> +	struct v4l2_event event = {
>> +		.type = V4L2_EVENT_FRAME_SYNC,
>> +	};
>> +
>> +	event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
>> +	v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
>> +}
>> +
>> +static int
>> +mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
>> +			     struct v4l2_event_subscription *sub)
>> +{
>> +	if (sub->type != V4L2_EVENT_FRAME_SYNC)
>> +		return -EINVAL;
>> +
>> +	/* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
>> +	if (sub->id != 0)
>> +		return -EINVAL;
>> +
>> +	return v4l2_event_subscribe(fh, sub, 0, NULL);
>> +}
>> +
>> +static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
>> +	.subscribe_event = mali_c55_isp_subscribe_event,
>> +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
>> +};
>> +
>> +static const struct v4l2_subdev_ops mali_c55_isp_ops = {
>> +	.pad	= &mali_c55_isp_pad_ops,
>> +	.core	= &mali_c55_isp_core_ops,
>> +};
>> +
>> +static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
>> +				   struct v4l2_subdev_state *sd_state)
> You name this variable state in every other subdev operation handler.
>
>> +{
>> +	struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>> +	struct v4l2_rect *in_crop;
>> +
>> +	sink_fmt = v4l2_subdev_state_get_format(sd_state,
>> +						MALI_C55_ISP_PAD_SINK_VIDEO);
>> +	src_fmt = v4l2_subdev_state_get_format(sd_state,
>> +					       MALI_C55_ISP_PAD_SOURCE);
>> +	in_crop = v4l2_subdev_state_get_crop(sd_state,
>> +					     MALI_C55_ISP_PAD_SINK_VIDEO);
>> +
>> +	sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
>> +	sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
>> +	sink_fmt->field = V4L2_FIELD_NONE;
>> +	sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> You should initialize the colorspace fields too. Same below.
>
>> +
>> +	*v4l2_subdev_state_get_format(sd_state,
>> +			      MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
>> +
>> +	src_fmt->width = MALI_C55_DEFAULT_WIDTH;
>> +	src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
>> +	src_fmt->field = V4L2_FIELD_NONE;
>> +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>> +
>> +	in_crop->top = 0;
>> +	in_crop->left = 0;
>> +	in_crop->width = MALI_C55_DEFAULT_WIDTH;
>> +	in_crop->height = MALI_C55_DEFAULT_HEIGHT;
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
>> +	.init_state = mali_c55_isp_init_state,
>> +};
>> +
>> +static const struct media_entity_operations mali_c55_isp_media_ops = {
>> +	.link_validate		= v4l2_subdev_link_validate,
> 	.link_validate = v4l2_subdev_link_validate,
>
> to match mali_c55_isp_internal_ops.
>
>> +};
>> +
>> +static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
>> +				       struct v4l2_subdev *subdev,
>> +				       struct v4l2_async_connection *asc)
>> +{
>> +	struct mali_c55_isp *isp = container_of(notifier,
>> +						struct mali_c55_isp, notifier);
>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>> +	struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
>> +	int ret;
>> +
>> +	/*
>> +	 * By default we'll flag this link enabled and the TPG disabled, but
>> +	 * no immutable flag because we need to be able to switch between the
>> +	 * two.
>> +	 */
>> +	ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
>> +					      MEDIA_LNK_FL_ENABLED);
>> +	if (ret)
>> +		dev_err(mali_c55->dev, "failed to create link for %s\n",
>> +			subdev->name);
>> +
>> +	return ret;
>> +}
>> +
>> +static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
>> +{
>> +	struct mali_c55_isp *isp = container_of(notifier,
>> +						struct mali_c55_isp, notifier);
>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>> +
>> +	return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
>> +}
>> +
>> +static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
>> +	.bound = mali_c55_isp_notifier_bound,
>> +	.complete = mali_c55_isp_notifier_complete,
>> +};
>> +
>> +static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
>> +{
>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>> +	struct v4l2_async_connection *asc;
>> +	struct fwnode_handle *ep;
>> +	int ret;
>> +
>> +	v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
>> +
>> +	/*
>> +	 * The ISP should have a single endpoint pointing to some flavour of
>> +	 * CSI-2 receiver...but for now at least we do want everything to work
>> +	 * normally even with no sensors connected, as we have the TPG. If we
>> +	 * don't find a sensor just warn and return success.
>> +	 */
>> +	ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
>> +					     0, 0, 0);
>> +	if (!ep) {
>> +		dev_warn(mali_c55->dev, "no local endpoint found\n");
>> +		return 0;
>> +	}
>> +
>> +	asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
>> +					      struct v4l2_async_connection);
>> +	if (IS_ERR(asc)) {
>> +		dev_err(mali_c55->dev, "failed to add remote fwnode\n");
>> +		ret = PTR_ERR(asc);
>> +		goto err_put_ep;
>> +	}
>> +
>> +	isp->notifier.ops = &mali_c55_isp_notifier_ops;
>> +	ret = v4l2_async_nf_register(&isp->notifier);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "failed to register notifier\n");
>> +		goto err_cleanup_nf;
>> +	}
>> +
>> +	fwnode_handle_put(ep);
>> +
>> +	return 0;
>> +
>> +err_cleanup_nf:
>> +	v4l2_async_nf_cleanup(&isp->notifier);
>> +err_put_ep:
>> +	fwnode_handle_put(ep);
>> +
>> +	return ret;
>> +}
>> +
>> +int mali_c55_register_isp(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>> +	struct v4l2_subdev *sd = &isp->sd;
>> +	int ret;
>> +
>> +	isp->mali_c55 = mali_c55;
>> +
>> +	v4l2_subdev_init(sd, &mali_c55_isp_ops);
>> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
>> +	sd->entity.ops = &mali_c55_isp_media_ops;
>> +	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
>> +	sd->internal_ops = &mali_c55_isp_internal_ops;
>> +	strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
>> +
>> +	isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
> The MUST_CONNECT flag would make sense here.
>
>> +	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>> +	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
>> +
>> +	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
>> +				     isp->pads);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = v4l2_subdev_init_finalize(sd);
>> +	if (ret)
>> +		goto err_cleanup_media_entity;
>> +
>> +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>> +	if (ret)
>> +		goto err_cleanup_subdev;
>> +
>> +	ret = mali_c55_isp_parse_endpoint(isp);
>> +	if (ret)
>> +		goto err_cleanup_subdev;
> As noted elsewhere, I think this belongs to mali-c55-core.c.
>
>> +
>> +	mutex_init(&isp->lock);
> This lock is used in mali-c55-capture.c only, that seems weird.
>
>> +
>> +	return 0;
>> +
>> +err_cleanup_subdev:
>> +	v4l2_subdev_cleanup(sd);
>> +err_cleanup_media_entity:
>> +	media_entity_cleanup(&sd->entity);
>> +	isp->mali_c55 = NULL;
>> +
>> +	return ret;
>> +}
>> +
>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>> +
>> +	if (!isp->mali_c55)
>> +		return;
>> +
>> +	mutex_destroy(&isp->lock);
>> +	v4l2_async_nf_unregister(&isp->notifier);
>> +	v4l2_async_nf_cleanup(&isp->notifier);
>> +	v4l2_device_unregister_subdev(&isp->sd);
>> +	v4l2_subdev_cleanup(&isp->sd);
>> +	media_entity_cleanup(&isp->sd.entity);
>> +}
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>> new file mode 100644
>> index 000000000000..cb27abde2aa5
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>> @@ -0,0 +1,258 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * ARM Mali-C55 ISP Driver - Register definitions
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#ifndef _MALI_C55_REGISTERS_H
>> +#define _MALI_C55_REGISTERS_H
>> +
>> +#include <linux/bits.h>
>> +
>> +/* ISP Common 0x00000 - 0x000ff */
>> +
>> +#define MALI_C55_REG_API				0x00000
>> +#define MALI_C55_REG_PRODUCT				0x00004
>> +#define MALI_C55_REG_VERSION				0x00008
>> +#define MALI_C55_REG_REVISION				0x0000c
>> +#define MALI_C55_REG_PULSE_MODE				0x0003c
>> +#define MALI_C55_REG_INPUT_MODE_REQUEST			0x0009c
>> +#define MALI_C55_INPUT_SAFE_STOP			0x00
>> +#define MALI_C55_INPUT_SAFE_START			0x01
>> +#define MALI_C55_REG_MODE_STATUS			0x000a0
>> +#define MALI_C55_REG_INTERRUPT_MASK_VECTOR		0x00030
>> +#define MALI_C55_INTERRUPT_MASK_ALL			GENMASK(31, 0)
>> +
>> +#define MALI_C55_REG_GLOBAL_MONITOR			0x00050
>> +
>> +#define MALI_C55_REG_GEN_VIDEO				0x00080
>> +#define MALI_C55_REG_GEN_VIDEO_ON_MASK			BIT(0)
>> +#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK		BIT(1)
>> +#define MALI_C55_REG_GEN_PREFETCH_MASK			GENMASK(31, 16)
>> +
>> +#define MALI_C55_REG_MCU_CONFIG				0x00020
>> +#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK		BIT(0)
> #define MALI_C55_REG_MCU_CONFIG_OVERRIDE		BIT(0)
>
> Same in other places where applicable.
>
>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK		BIT(1)
>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PING		BIT(1)
>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG		0x00
>> +#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK		BIT(8)
>> +#define MALI_C55_REG_PING_PONG_READ			0x00024
>> +#define MALI_C55_REG_PING_PONG_READ_MASK		BIT(2)
>> +
>> +#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR		0x00034
>> +#define MALI_C55_REG_INTERRUPT_CLEAR			0x00040
>> +#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR		0x00044
>> +#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS		0x00068
>> +#define MALI_C55_GPS_PONG_FITTED			BIT(0)
>> +#define MALI_C55_GPS_WDR_FITTED				BIT(1)
>> +#define MALI_C55_GPS_COMPRESSION_FITTED			BIT(2)
>> +#define MALI_C55_GPS_TEMPER_FITTED			BIT(3)
>> +#define MALI_C55_GPS_SINTER_LITE_FITTED			BIT(4)
>> +#define MALI_C55_GPS_SINTER_FITTED			BIT(5)
>> +#define MALI_C55_GPS_IRIDIX_LTM_FITTED			BIT(6)
>> +#define MALI_C55_GPS_IRIDIX_GTM_FITTED			BIT(7)
>> +#define MALI_C55_GPS_CNR_FITTED				BIT(8)
>> +#define MALI_C55_GPS_FRSCALER_FITTED			BIT(9)
>> +#define MALI_C55_GPS_DS_PIPE_FITTED			BIT(10)
>> +
>> +#define MALI_C55_REG_BLANKING				0x00084
>> +#define MALI_C55_REG_HBLANK_MASK			GENMASK(15, 0)
>> +#define MALI_C55_REG_VBLANK_MASK			GENMASK(31, 16)
>> +
>> +#define MALI_C55_REG_HC_START				0x00088
>> +#define MALI_C55_HC_START(h)				(((h) & 0xffff) << 16)
>> +#define MALI_C55_REG_HC_SIZE				0x0008c
>> +#define MALI_C55_HC_SIZE(h)				((h) & 0xffff)
>> +#define MALI_C55_REG_VC_START_SIZE			0x00094
>> +#define MALI_C55_VC_START(v)				((v) & 0xffff)
>> +#define MALI_C55_VC_SIZE(v)				(((v) & 0xffff) << 16)
>> +
>> +/* Ping/Pong Configuration Space */
>> +#define MALI_C55_REG_BASE_ADDR				0x18e88
>> +#define MALI_C55_REG_BYPASS_0				0x18eac
>> +#define MALI_C55_REG_BYPASS_0_VIDEO_TEST		BIT(0)
>> +#define MALI_C55_REG_BYPASS_0_INPUT_FMT			BIT(1)
>> +#define MALI_C55_REG_BYPASS_0_DECOMPANDER		BIT(2)
>> +#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR		BIT(3)
>> +#define MALI_C55_REG_BYPASS_0_GAIN_WDR			BIT(4)
>> +#define MALI_C55_REG_BYPASS_0_FRAME_STITCH		BIT(5)
>> +#define MALI_C55_REG_BYPASS_1				0x18eb0
>> +#define MALI_C55_REG_BYPASS_1_DIGI_GAIN			BIT(0)
>> +#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS		BIT(1)
>> +#define MALI_C55_REG_BYPASS_1_FE_SQRT			BIT(2)
>> +#define MALI_C55_REG_BYPASS_1_RAW_FE			BIT(3)
>> +#define MALI_C55_REG_BYPASS_2				0x18eb8
>> +#define MALI_C55_REG_BYPASS_2_SINTER			BIT(0)
>> +#define MALI_C55_REG_BYPASS_2_TEMPER			BIT(1)
>> +#define MALI_C55_REG_BYPASS_3				0x18ebc
>> +#define MALI_C55_REG_BYPASS_3_SQUARE_BE			BIT(0)
>> +#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH	BIT(1)
>> +#define MALI_C55_REG_BYPASS_3_MESH_SHADING		BIT(3)
>> +#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE		BIT(4)
>> +#define MALI_C55_REG_BYPASS_3_IRIDIX			BIT(5)
>> +#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN		BIT(6)
>> +#define MALI_C55_REG_BYPASS_4				0x18ec0
>> +#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB		BIT(1)
>> +#define MALI_C55_REG_BYPASS_4_PF_CORRECTION		BIT(3)
>> +#define MALI_C55_REG_BYPASS_4_CCM			BIT(4)
>> +#define MALI_C55_REG_BYPASS_4_CNR			BIT(5)
>> +#define MALI_C55_REG_FR_BYPASS				0x18ec4
>> +#define MALI_C55_REG_DS_BYPASS				0x18ec8
>> +#define MALI_C55_BYPASS_CROP				BIT(0)
>> +#define MALI_C55_BYPASS_SCALER				BIT(1)
>> +#define MALI_C55_BYPASS_GAMMA_RGB			BIT(2)
>> +#define MALI_C55_BYPASS_SHARPEN				BIT(3)
>> +#define MALI_C55_BYPASS_CS_CONV				BIT(4)
>> +#define MALI_C55_REG_ISP_RAW_BYPASS			0x18ecc
>> +#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK		BIT(0)
>> +#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK		GENMASK(9, 8)
>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS		2
>> +#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS		1
>> +#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE		BIT(1)
>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS		BIT(0)
>> +
>> +#define MALI_C55_REG_ACTIVE_WIDTH_MASK			0xffff
>> +#define MALI_C55_REG_ACTIVE_HEIGHT_MASK			0xffff0000
>> +#define MALI_C55_REG_BAYER_ORDER			0x18e8c
>> +#define MALI_C55_BAYER_ORDER_MASK			GENMASK(1, 0)
>> +#define MALI_C55_REG_TPG_CH0				0x18ed8
>> +#define MALI_C55_TEST_PATTERN_ON_OFF			BIT(0)
>> +#define MALI_C55_TEST_PATTERN_RGB_MASK			BIT(1)
>> +#define MALI_C55_REG_TPG_R_BACKGROUND			0x18ee0
>> +#define MALI_C55_REG_TPG_G_BACKGROUND			0x18ee4
>> +#define MALI_C55_REG_TPG_B_BACKGROUND			0x18ee8
>> +#define MALI_C55_TPG_BACKGROUND_MAX			0xfffff
>> +#define MALI_C55_REG_INPUT_WIDTH			0x18f98
>> +#define MALI_C55_INPUT_WIDTH_MASK			GENMASK(18, 16)
>> +#define MALI_C55_INPUT_WIDTH_8BIT			0
>> +#define MALI_C55_INPUT_WIDTH_10BIT			1
>> +#define MALI_C55_INPUT_WIDTH_12BIT			2
>> +#define MALI_C55_INPUT_WIDTH_14BIT			3
>> +#define MALI_C55_INPUT_WIDTH_16BIT			4
>> +#define MALI_C55_INPUT_WIDTH_20BIT			5
>> +#define MALI_C55_REG_SPACE_SIZE				0x4000
>> +#define MALI_C55_REG_CONFIG_SPACES_OFFSET		0x0ab6c
>> +#define MALI_C55_CONFIG_SPACE_SIZE			0x1231c
>> +
>> +#define MALI_C55_REG_SINTER_CONFIG			0x19348
>> +#define MALI_C55_SINTER_VIEW_FILTER_MASK		GENMASK(1, 0)
>> +#define MALI_C55_SINTER_SCALE_MODE_MASK			GENMASK(3, 2)
>> +#define MALI_C55_SINTER_ENABLE_MASK			BIT(4)
>> +#define MALI_C55_SINTER_FILTER_SELECT_MASK		BIT(5)
>> +#define MALI_C55_SINTER_INT_SELECT_MASK			BIT(6)
>> +#define MALI_C55_SINTER_RM_ENABLE_MASK			BIT(7)
>> +
>> +/* Colour Correction Matrix Configuration */
>> +#define MALI_C55_REG_CCM_ENABLE				0x1b07c
>> +#define MALI_C55_CCM_ENABLE_MASK			BIT(0)
>> +#define MALI_C55_REG_CCM_COEF_R_R			0x1b080
>> +#define MALI_C55_REG_CCM_COEF_R_G			0x1b084
>> +#define MALI_C55_REG_CCM_COEF_R_B			0x1b088
>> +#define MALI_C55_REG_CCM_COEF_G_R			0x1b090
>> +#define MALI_C55_REG_CCM_COEF_G_G			0x1b094
>> +#define MALI_C55_REG_CCM_COEF_G_B			0x1b098
>> +#define MALI_C55_REG_CCM_COEF_B_R			0x1b0a0
>> +#define MALI_C55_REG_CCM_COEF_B_G			0x1b0a4
>> +#define MALI_C55_REG_CCM_COEF_B_B			0x1b0a8
>> +#define MALI_C55_CCM_COEF_MASK				GENMASK(12, 0)
>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R			0x1b0b0
>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G			0x1b0b4
>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B			0x1b0b8
>> +#define MALI_C55_CCM_ANTIFOG_GAIN_MASK			GENMASK(11, 0)
>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R		0x1b0c0
>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G		0x1b0c4
>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B		0x1b0c8
>> +#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK		GENMASK(11, 0)
>> +
>> +/*
>> + * The Mali-C55 ISP has up to two output pipes; known as full resolution and
>> + * down scaled. The register space for these is laid out identically, but offset
>> + * by 372 bytes.
>> + */
>> +#define MALI_C55_CAP_DEV_FR_REG_OFFSET		0x0
>> +#define MALI_C55_CAP_DEV_DS_REG_OFFSET		0x174
>> +
>> +#define MALI_C55_REG_CS_CONV_CONFIG(offset)		(0x1c098 + (offset))
>> +#define MALI_C55_CS_CONV_MATRIX_MASK			BIT(0)
>> +#define MALI_C55_CS_CONV_FILTER_MASK			BIT(1)
>> +#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK		BIT(2)
>> +#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK		BIT(3)
>> +#define MALI_C55_REG_Y_WRITER_MODE(offset)		(0x1c0ec + (offset))
>> +#define MALI_C55_REG_UV_WRITER_MODE(offset)		(0x1c144 + (offset))
>> +#define MALI_C55_WRITER_MODE_MASK			GENMASK(4, 0)
>> +#define MALI_C55_WRITER_SUBMODE_MASK			GENMASK(7, 6)
>> +#define MALI_C55_WRITER_FRAME_WRITE_MASK		BIT(9)
>> +#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset)		(0x1c0f0 + (offset))
>> +#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset)		(0x1c148 + (offset))
>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)		((w) << 0)
>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)		((h) << 16)
>> +#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset)	(0x1c0f4 + (offset))
>> +#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset)	(0x1c108 + (offset))
>> +#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
>> +#define MALI_C55_REG_Y_WRITER_BANKS_RESTART		BIT(3)
>> +#define MALI_C55_REG_Y_WRITER_OFFSET(offset)		(0x1c10c + (offset))
>> +#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset)	(0x1c14c + (offset))
>> +#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset)	(0x1c160 + (offset))
>> +#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
>> +#define MALI_C55_REG_UV_WRITER_BANKS_RESTART		BIT(3)
>> +#define MALI_C55_REG_UV_WRITER_OFFSET(offset)		(0x1c164 + (offset))
>> +
>> +#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
>> +#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE		0x18edc
>> +
>> +#define MALI_C55_REG_CROP_EN(offset)			(0x1c028 + (offset))
>> +#define MALI_C55_CROP_ENABLE				BIT(0)
>> +#define MALI_C55_REG_CROP_X_START(offset)		(0x1c02c + (offset))
>> +#define MALI_C55_REG_CROP_Y_START(offset)		(0x1c030 + (offset))
>> +#define MALI_C55_REG_CROP_X_SIZE(offset)		(0x1c034 + (offset))
>> +#define MALI_C55_REG_CROP_Y_SIZE(offset)		(0x1c038 + (offset))
>> +#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset)		(0x1c040 + (offset))
>> +#define MALI_C55_SCALER_TIMEOUT_EN			BIT(4)
>> +#define MALI_C55_SCALER_TIMEOUT(t)			((t) << 16)
>> +#define MALI_C55_REG_SCALER_IN_WIDTH(offset)		(0x1c044 + (offset))
>> +#define MALI_C55_REG_SCALER_IN_HEIGHT(offset)		(0x1c048 + (offset))
>> +#define MALI_C55_REG_SCALER_OUT_WIDTH(offset)		(0x1c04c + (offset))
>> +#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset)		(0x1c050 + (offset))
>> +#define MALI_C55_REG_SCALER_HFILT_TINC(offset)		(0x1c054 + (offset))
>> +#define MALI_C55_REG_SCALER_HFILT_COEF(offset)		(0x1c058 + (offset))
>> +#define MALI_C55_REG_SCALER_VFILT_TINC(offset)		(0x1c05c + (offset))
>> +#define MALI_C55_REG_SCALER_VFILT_COEF(offset)		(0x1c060 + (offset))
>> +
>> +#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset)		(0x1c064 + (offset))
>> +#define MALI_C55_GAMMA_ENABLE_MASK			BIT(0)
>> +#define MALI_C55_REG_GAMMA_GAINS_1(offset)		(0x1c068 + (offset))
>> +#define MALI_C55_GAMMA_GAIN_R_MASK			GENMASK(11, 0)
>> +#define MALI_C55_GAMMA_GAIN_G_MASK			GENMASK(27, 16)
>> +#define MALI_C55_REG_GAMMA_GAINS_2(offset)		(0x1c06c + (offset))
>> +#define MALI_C55_GAMMA_GAIN_B_MASK			GENMASK(11, 0)
>> +#define MALI_C55_REG_GAMMA_OFFSETS_1(offset)		(0x1c070 + (offset))
>> +#define MALI_C55_GAMMA_OFFSET_R_MASK			GENMASK(11, 0)
>> +#define MALI_C55_GAMMA_OFFSET_G_MASK			GENMASK(27, 16)
>> +#define MALI_C55_REG_GAMMA_OFFSETS_2(offset)		(0x1c074 + (offset))
>> +#define MALI_C55_GAMMA_OFFSET_B_MASK			GENMASK(11, 0)
>> +
>> +/* Output DMA Writer */
>> +
>> +#define MALI_C55_OUTPUT_DISABLED		0
>> +#define MALI_C55_OUTPUT_RGB32			1
>> +#define MALI_C55_OUTPUT_A2R10G10B10		2
>> +#define MALI_C55_OUTPUT_RGB565			3
>> +#define MALI_C55_OUTPUT_RGB24			4
>> +#define MALI_C55_OUTPUT_GEN32			5
>> +#define MALI_C55_OUTPUT_RAW16			6
>> +#define MALI_C55_OUTPUT_AYUV			8
>> +#define MALI_C55_OUTPUT_Y410			9
>> +#define MALI_C55_OUTPUT_YUY2			10
>> +#define MALI_C55_OUTPUT_UYVY			11
>> +#define MALI_C55_OUTPUT_Y210			12
>> +#define MALI_C55_OUTPUT_NV12_21			13
>> +#define MALI_C55_OUTPUT_YUV_420_422		17
>> +#define MALI_C55_OUTPUT_P210_P010		19
>> +#define MALI_C55_OUTPUT_YUV422			20
> I'd define this just below the MALI_C55_REG_UV_WRITER_MODE register
> macro.
>
>> +
>> +#define MALI_C55_OUTPUT_PLANE_ALT0		0
>> +#define MALI_C55_OUTPUT_PLANE_ALT1		1
>> +#define MALI_C55_OUTPUT_PLANE_ALT2		2
> Same here ?
>
>> +
>> +#endif /* _MALI_C55_REGISTERS_H */
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>> new file mode 100644
>> index 000000000000..8edae87f1e5f
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>> @@ -0,0 +1,382 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * ARM Mali-C55 ISP Driver - Resizer Coefficients
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#ifndef _MALI_C55_RESIZER_COEFS_H
>> +#define _MALI_C55_RESIZER_COEFS_H
>> +
>> +#include "mali-c55-common.h"
>> +
>> +#define MALI_C55_RESIZER_COEFS_NUM_BANKS	8
>> +#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES	64
> Do these belongs to mali-c55-registers.h ?
>
>> +
>> +static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
>> +	{	/* Bank 0 */
>> +		0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
>> +		0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
>> +		0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
>> +		0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
>> +		0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
>> +		0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
>> +		0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
>> +		0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
>> +		0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
>> +		0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
>> +		0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
>> +		0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
>> +		0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
>> +		0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
>> +		0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
>> +		0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
>> +	},
>> +	{	/* Bank 1 */
>> +		0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
>> +		0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
>> +		0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
>> +		0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
>> +		0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
>> +		0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
>> +		0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
>> +		0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
>> +		0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
>> +		0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
>> +		0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
>> +		0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
>> +		0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
>> +		0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
>> +		0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
>> +		0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
>> +	},
>> +	{	/* Bank 2 */
>> +		0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
>> +		0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
>> +		0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
>> +		0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
>> +		0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
>> +		0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
>> +		0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
>> +		0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
>> +		0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
>> +		0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
>> +		0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
>> +		0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
>> +		0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
>> +		0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
>> +		0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
>> +		0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
>> +	},
>> +	{	/* Bank 3 */
>> +		0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
>> +		0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
>> +		0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
>> +		0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
>> +		0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
>> +		0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
>> +		0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
>> +		0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
>> +		0x20100000, 0x00000010, 0x1f110000, 0x00000010,
>> +		0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
>> +		0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
>> +		0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
>> +		0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
>> +		0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
>> +		0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
>> +		0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
>> +	},
>> +	{	/* Bank 4 */
>> +		0x17090000, 0x00000917, 0x18090000, 0x00000916,
>> +		0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
>> +		0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
>> +		0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
>> +		0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
>> +		0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
>> +		0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
>> +		0x190f0300, 0x00000411, 0x18100300, 0x00000411,
>> +		0x1a100300, 0x00000310, 0x18110400, 0x00000310,
>> +		0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
>> +		0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
>> +		0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
>> +		0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
>> +		0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
>> +		0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
>> +		0x17160800, 0x0000010a, 0x18160900, 0x00000009,
>> +	},
>> +	{	/* Bank 5 */
>> +		0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
>> +		0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
>> +		0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
>> +		0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
>> +		0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
>> +		0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
>> +		0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
>> +		0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
>> +		0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
>> +		0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
>> +		0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
>> +		0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
>> +		0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
>> +		0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
>> +		0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
>> +		0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
>> +	},
>> +	{	/* Bank 6 */
>> +		0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
>> +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
>> +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
>> +		0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
>> +		0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>> +		0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
>> +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>> +		0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
>> +		0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
>> +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
>> +		0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>> +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>> +	},
>> +	{	/* Bank 7 */
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +	}
>> +};
>> +
>> +static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
>> +	{	/* Bank 0 */
>> +		0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
>> +		0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
>> +		0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
>> +		0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
>> +		0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
>> +		0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
>> +		0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
>> +		0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
>> +		0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
>> +		0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
>> +		0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
>> +		0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
>> +		0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
>> +		0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
>> +		0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
>> +		0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
>> +	},
>> +	{	/* Bank 1 */
>> +		0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
>> +		0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
>> +		0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
>> +		0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
>> +		0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
>> +		0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
>> +		0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
>> +		0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
>> +		0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
>> +		0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
>> +		0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
>> +		0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
>> +		0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
>> +		0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
>> +		0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
>> +		0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
>> +	},
>> +	{	/* Bank 2 */
>> +		0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
>> +		0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
>> +		0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
>> +		0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
>> +		0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
>> +		0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
>> +		0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
>> +		0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
>> +		0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
>> +		0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
>> +		0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
>> +		0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
>> +		0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
>> +		0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
>> +		0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
>> +		0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
>> +	},
>> +	{	/* Bank 3 */
>> +		0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
>> +		0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
>> +		0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
>> +		0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
>> +		0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
>> +		0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
>> +		0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
>> +		0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
>> +		0x20100000, 0x00000010, 0x1f100000, 0x00000011,
>> +		0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
>> +		0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
>> +		0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
>> +		0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
>> +		0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
>> +		0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
>> +		0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
>> +	},
>> +	{	/* Bank 4 */
>> +		0x17170900, 0x00000009, 0x18160900, 0x00000009,
>> +		0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
>> +		0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
>> +		0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
>> +		0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
>> +		0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
>> +		0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
>> +		0x19110400, 0x0000030f, 0x18110400, 0x00000310,
>> +		0x1a100300, 0x00000310, 0x18100300, 0x00000411,
>> +		0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
>> +		0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
>> +		0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
>> +		0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
>> +		0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
>> +		0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
>> +		0x170a0100, 0x00000816, 0x18090000, 0x00000916,
>> +	},
>> +	{	/* Bank 5 */
>> +		0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
>> +		0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
>> +		0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
>> +		0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
>> +		0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
>> +		0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
>> +		0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
>> +		0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
>> +		0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
>> +		0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
>> +		0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
>> +		0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
>> +		0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
>> +		0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
>> +		0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
>> +		0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
>> +	},
>> +	{	/* Bank 6 */
>> +		0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
>> +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>> +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
>> +		0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
>> +		0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>> +		0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
>> +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>> +		0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
>> +		0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
>> +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
>> +		0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
>> +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
>> +	},
>> +	{	/* Bank 7 */
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +	}
>> +};
>> +
>> +struct mali_c55_resizer_coef_bank {
>> +	unsigned int bank;
> This is always equal to the index of the entry in the
> mali_c55_coefficient_banks array, you can drop it.
>
>> +	unsigned int top;
>> +	unsigned int bottom;
> The bottom value of bank N is always equal to the top value of bank N+1.
> You can simplify this by storing a single value.
>
>> +};
>> +
>> +static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
>> +	{
>> +		.bank = 0,
>> +		.top = 1000,
>> +		.bottom = 770,
>> +	},
>> +	{
>> +		.bank = 1,
>> +		.top = 769,
>> +		.bottom = 600,
>> +	},
>> +	{
>> +		.bank = 2,
>> +		.top = 599,
>> +		.bottom = 460,
>> +	},
>> +	{
>> +		.bank = 3,
>> +		.top = 459,
>> +		.bottom = 354,
>> +	},
>> +	{
>> +		.bank = 4,
>> +		.top = 353,
>> +		.bottom = 273,
>> +	},
>> +	{
>> +		.bank = 5,
>> +		.top = 272,
>> +		.bottom = 210,
>> +	},
>> +	{
>> +		.bank = 6,
>> +		.top = 209,
>> +		.bottom = 162,
>> +	},
>> +	{
>> +		.bank = 7,
>> +		.top = 161,
>> +		.bottom = 125,
>> +	},
>> +};
>> +
> A small comment would be nice, such as
>
> /* Select a bank of resizer coefficients, based on the scaling ratio. */
>
>> +static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,
> This function is related to the resizers. Add "rsz" somewhere in the
> function name, and pass a resizer pointer.
>
>> +						unsigned int crop,
>> +						unsigned int scale)
> I think those are the input and output sizes to the scaler. Rename them
> to make it clearer.
>
>> +{
>> +	unsigned int tmp;
> tmp is almost always a bad variable name. Please use a more descriptive
> name, size as rsz_ratio.
>
>> +	unsigned int i;
>> +
>> +	tmp = (scale * 1000U) / crop;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
>> +		if (tmp >= mali_c55_coefficient_banks[i].bottom &&
>> +		    tmp <= mali_c55_coefficient_banks[i].top)
>> +			return mali_c55_coefficient_banks[i].bank;
>> +	}
>> +
>> +	/*
>> +	 * We shouldn't ever get here, in theory. As we have no good choices
>> +	 * simply warn the user and use the first bank of coefficients.
>> +	 */
>> +	dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
>> +	return 0;
>> +}
> And everything else belongs to mali-c55-resizer.c. Drop this header
> file.
>
>> +
>> +#endif /* _MALI_C55_RESIZER_COEFS_H */
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>> new file mode 100644
>> index 000000000000..0a5a2969d3ce
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>> @@ -0,0 +1,779 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ARM Mali-C55 ISP Driver - Image signal processor
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#include <linux/math.h>
>> +#include <linux/minmax.h>
>> +
>> +#include <media/media-entity.h>
>> +#include <media/v4l2-subdev.h>
>> +
>> +#include "mali-c55-common.h"
>> +#include "mali-c55-registers.h"
>> +#include "mali-c55-resizer-coefs.h"
>> +
>> +/* Scaling factor in Q4.20 format. */
>> +#define MALI_C55_RZR_SCALER_FACTOR	(1U << 20)
>> +
>> +static const u32 rzr_non_bypass_src_fmts[] = {
>> +	MEDIA_BUS_FMT_RGB121212_1X36,
>> +	MEDIA_BUS_FMT_YUV10_1X30
>> +};
>> +
>> +static const char * const mali_c55_resizer_names[] = {
>> +	[MALI_C55_RZR_FR] = "resizer fr",
>> +	[MALI_C55_RZR_DS] = "resizer ds",
>> +};
>> +
>> +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
>> +				     struct v4l2_subdev_state *state)
>> +{
>> +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>> +	struct v4l2_mbus_framefmt *fmt;
>> +	struct v4l2_rect *crop;
>> +
>> +	/* Verify if crop should be enabled. */
>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
>> +
>> +	if (fmt->width == crop->width && fmt->height == crop->height)
>> +		return MALI_C55_BYPASS_CROP;
>> +
>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
>> +		       crop->left);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
>> +		       crop->top);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
>> +		       crop->width);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
>> +		       crop->height);
>> +
>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
>> +		       MALI_C55_CROP_ENABLE);
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
>> +					struct v4l2_subdev_state *state)
>> +{
>> +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>> +	struct v4l2_rect *crop, *scale;
>> +	unsigned int h_bank, v_bank;
>> +	u64 h_scale, v_scale;
>> +
>> +	/* Verify if scaling should be enabled. */
>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
>> +	scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
>> +
>> +	if (crop->width == scale->width && crop->height == scale->height)
>> +		return MALI_C55_BYPASS_SCALER;
>> +
>> +	/* Program the V/H scaling factor in Q4.20 format. */
>> +	h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
>> +	v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
>> +
>> +	do_div(h_scale, scale->width);
>> +	do_div(v_scale, scale->height);
>> +
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
>> +		       crop->width);
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
>> +		       crop->height);
>> +
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
>> +		       scale->width);
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
>> +		       scale->height);
>> +
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
>> +		       h_scale);
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
>> +		       v_scale);
>> +
>> +	h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
>> +					     scale->width);
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
>> +		       h_bank);
>> +
>> +	v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
>> +					     scale->height);
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
>> +		       v_bank);
>> +
>> +	return 0;
>> +}
>> +
>> +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
>> +				 struct v4l2_subdev_state *state)
>> +{
>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>> +	u32 bypass = 0;
>> +
>> +	/* Verify if cropping and scaling should be enabled. */
>> +	bypass |= mali_c55_rzr_program_crop(rzr, state);
>> +	bypass |= mali_c55_rzr_program_resizer(rzr, state);
>> +
>> +	mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
>> +			     MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
>> +			     MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
>> +			     bypass);
>> +}
>> +
>> +/*
>> + * Inspect the routing table to know which of the two (mutually exclusive)
>> + * routes is enabled and return the sink pad id of the active route.
>> + */
>> +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
>> +{
>> +	struct v4l2_subdev_krouting *routing = &state->routing;
>> +	struct v4l2_subdev_route *route;
>> +
>> +	/* A single route is enabled at a time. */
>> +	for_each_active_route(routing, route)
>> +		return route->sink_pad;
>> +
>> +	return MALI_C55_RZR_SINK_PAD;
>> +}
>> +
>> +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
>> +{
>> +	u32 corrected_code = 0;
>> +
>> +	/*
>> +	 * The ISP takes input in a 20-bit format, but can only output 16-bit
>> +	 * RAW bayer data (with the 4 least significant bits from the input
>> +	 * being lost). Return the 16-bit version of the 20-bit input formats.
>> +	 */
>> +	switch (mbus_code) {
>> +	case MEDIA_BUS_FMT_SBGGR20_1X20:
>> +		corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
>> +		break;
>> +	case MEDIA_BUS_FMT_SGBRG20_1X20:
>> +		corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
>> +		break;
>> +	case MEDIA_BUS_FMT_SGRBG20_1X20:
>> +		corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
>> +		break;
>> +	case MEDIA_BUS_FMT_SRGGB20_1X20:
>> +		corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
>> +		break;
>> +	}
>> +
>> +	return corrected_code;
>> +}
>> +
>> +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
>> +				      struct v4l2_subdev_state *state,
>> +				      struct v4l2_subdev_krouting *routing)
>> +{
>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>> +						    sd);
>> +	unsigned int active_sink = UINT_MAX;
>> +	struct v4l2_mbus_framefmt *src_fmt;
>> +	struct v4l2_rect *crop, *compose;
>> +	struct v4l2_subdev_route *route;
>> +	unsigned int active_routes = 0;
>> +	struct v4l2_mbus_framefmt *fmt;
>> +	int ret;
>> +
>> +	ret = v4l2_subdev_routing_validate(sd, routing, 0);
>> +	if (ret)
>> +		return ret;
>> +
>> +	/* Only a single route can be enabled at a time. */
>> +	for_each_active_route(routing, route) {
>> +		if (++active_routes > 1) {
>> +			dev_err(rzr->mali_c55->dev,
>> +				"Only one route can be active");
>> +			return -EINVAL;
>> +		}
>> +
>> +		active_sink = route->sink_pad;
>> +	}
>> +	if (active_sink == UINT_MAX) {
>> +		dev_err(rzr->mali_c55->dev, "One route has to be active");
>> +		return -EINVAL;
>> +	}
>> +
>> +	ret = v4l2_subdev_set_routing(sd, state, routing);
>> +	if (ret) {
>> +		dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
>> +		return ret;
>> +	}
>> +
>> +	fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
>> +	crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
>> +	compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
>> +
>> +	fmt->width = MALI_C55_DEFAULT_WIDTH;
>> +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
>> +	fmt->colorspace = V4L2_COLORSPACE_SRGB;
>> +	fmt->field = V4L2_FIELD_NONE;
>> +
>> +	if (active_sink == MALI_C55_RZR_SINK_PAD) {
>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>> +
>> +		crop->left = crop->top = 0;
>> +		crop->width = MALI_C55_DEFAULT_WIDTH;
>> +		crop->height = MALI_C55_DEFAULT_HEIGHT;
>> +
>> +		*compose = *crop;
>> +	} else {
>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>> +	}
>> +
>> +	/* Propagate the format to the source pad */
>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
>> +					       0);
>> +	*src_fmt = *fmt;
>> +
>> +	/* In the event this is the bypass pad the mbus code needs correcting */
>> +	if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
>> +		src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
>> +				       struct v4l2_subdev_state *state,
>> +				       struct v4l2_subdev_mbus_code_enum *code)
>> +{
>> +	struct v4l2_mbus_framefmt *sink_fmt;
>> +	const struct mali_c55_isp_fmt *fmt;
>> +	unsigned int index = 0;
>> +	u32 sink_pad;
>> +
>> +	switch (code->pad) {
>> +	case MALI_C55_RZR_SINK_PAD:
>> +		if (code->index)
>> +			return -EINVAL;
>> +
>> +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
>> +
>> +		return 0;
>> +	case MALI_C55_RZR_SOURCE_PAD:
>> +		sink_pad = mali_c55_rzr_get_active_sink(state);
>> +		sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
>> +
>> +		/*
>> +		 * If the active route is from the Bypass sink pad, then the
>> +		 * source pad is a simple passthrough of the sink format,
>> +		 * downshifted to 16-bits.
>> +		 */
>> +
>> +		if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>> +			if (code->index)
>> +				return -EINVAL;
>> +
>> +			code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
>> +			if (!code->code)
>> +				return -EINVAL;
>> +
>> +			return 0;
>> +		}
>> +
>> +		/*
>> +		 * If the active route is from the non-bypass sink then we can
>> +		 * select either RGB or conversion to YUV.
>> +		 */
>> +
>> +		if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
>> +			return -EINVAL;
>> +
>> +		code->code = rzr_non_bypass_src_fmts[code->index];
>> +
>> +		return 0;
>> +	case MALI_C55_RZR_SINK_BYPASS_PAD:
>> +		for_each_mali_isp_fmt(fmt) {
>> +			if (index++ == code->index) {
>> +				code->code = fmt->code;
>> +				return 0;
>> +			}
>> +		}
>> +
>> +		break;
>> +	}
>> +
>> +	return -EINVAL;
>> +}
>> +
>> +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
>> +					struct v4l2_subdev_state *state,
>> +					struct v4l2_subdev_frame_size_enum *fse)
>> +{
>> +	if (fse->index)
>> +		return -EINVAL;
>> +
>> +	fse->max_width = MALI_C55_MAX_WIDTH;
>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
>> +	fse->min_width = MALI_C55_MIN_WIDTH;
>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
>> +				     struct v4l2_subdev_state *state,
>> +				     struct v4l2_subdev_format *format)
>> +{
>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>> +	struct v4l2_rect *rect;
>> +	unsigned int sink_pad;
>> +
>> +	/*
>> +	 * Clamp to min/max and then reset crop and compose rectangles to the
>> +	 * newly applied size.
>> +	 */
>> +	clamp_t(unsigned int, fmt->width,
>> +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>> +	clamp_t(unsigned int, fmt->height,
>> +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>> +
>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
>> +	if (sink_pad == MALI_C55_RZR_SINK_PAD) {
>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>> +
>> +		rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
>> +		rect->left = 0;
>> +		rect->top = 0;
>> +		rect->width = fmt->width;
>> +		rect->height = fmt->height;
>> +
>> +		rect = v4l2_subdev_state_get_compose(state,
>> +						     MALI_C55_RZR_SINK_PAD);
>> +		rect->left = 0;
>> +		rect->top = 0;
>> +		rect->width = fmt->width;
>> +		rect->height = fmt->height;
>> +	} else {
>> +		/*
>> +		 * Make sure the media bus code is one of the supported
>> +		 * ISP input media bus codes.
>> +		 */
>> +		if (!mali_c55_isp_is_format_supported(fmt->code))
>> +			fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
>> +	}
>> +
>> +	*v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
>> +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
>> +				       struct v4l2_subdev_state *state,
>> +				       struct v4l2_subdev_format *format)
>> +{
>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>> +						    sd);
>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>> +	struct v4l2_mbus_framefmt *sink_fmt;
>> +	struct v4l2_rect *crop, *compose;
>> +	unsigned int sink_pad;
>> +	unsigned int i;
>> +
>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
>> +	sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
>> +	crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
>> +	compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
>> +
>> +	/* FR Bypass pipe. */
>> +
>> +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>> +		/*
>> +		 * Format on the source pad is the same as the one on the
>> +		 * sink pad, downshifted to 16-bits.
>> +		 */
>> +		fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
>> +		if (!fmt->code)
>> +			return -EINVAL;
>> +
>> +		/* RAW bypass disables scaling and cropping. */
>> +		crop->top = compose->top = 0;
>> +		crop->left = compose->left = 0;
>> +		fmt->width = crop->width = compose->width = sink_fmt->width;
>> +		fmt->height = crop->height = compose->height = sink_fmt->height;
>> +
>> +		*v4l2_subdev_state_get_format(state,
>> +					      MALI_C55_RZR_SOURCE_PAD) = *fmt;
>> +
>> +		return 0;
>> +	}
>> +
>> +	/* Regular processing pipe. */
>> +
>> +	for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
>> +		if (fmt->code == rzr_non_bypass_src_fmts[i])
>> +			break;
>> +	}
>> +
>> +	if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
>> +		dev_dbg(rzr->mali_c55->dev,
>> +			"Unsupported mbus code 0x%x: using default\n",
>> +			fmt->code);
>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>> +	}
>> +
>> +	/*
>> +	 * The source pad format size comes directly from the sink pad
>> +	 * compose rectangle.
>> +	 */
>> +	fmt->width = compose->width;
>> +	fmt->height = compose->height;
>> +
>> +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
>> +				struct v4l2_subdev_state *state,
>> +				struct v4l2_subdev_format *format)
>> +{
>> +	/*
>> +	 * On sink pads fmt is either fixed for the 'regular' processing
>> +	 * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
>> +	 * pad.
>> +	 *
>> +	 * On source pad sizes are the result of crop+compose on the sink
>> +	 * pad sizes, while the format depends on the active route.
>> +	 */
>> +
>> +	if (format->pad != MALI_C55_RZR_SOURCE_PAD)
>> +		return mali_c55_rzr_set_sink_fmt(sd, state, format);
>> +
>> +	return mali_c55_rzr_set_source_fmt(sd, state, format);
>> +}
>> +
>> +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
>> +				      struct v4l2_subdev_state *state,
>> +				      struct v4l2_subdev_selection *sel)
>> +{
>> +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
>> +		return -EINVAL;
>> +
>> +	if (sel->target != V4L2_SEL_TGT_CROP &&
>> +	    sel->target != V4L2_SEL_TGT_COMPOSE)
>> +		return -EINVAL;
>> +
>> +	sel->r = sel->target == V4L2_SEL_TGT_CROP
>> +	       ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
>> +	       : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
>> +				      struct v4l2_subdev_state *state,
>> +				      struct v4l2_subdev_selection *sel)
>> +{
>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>> +						    sd);
>> +	struct v4l2_mbus_framefmt *source_fmt;
>> +	struct v4l2_mbus_framefmt *sink_fmt;
>> +	struct v4l2_rect *crop, *compose;
>> +
>> +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
>> +		return -EINVAL;
>> +
>> +	if (sel->target != V4L2_SEL_TGT_CROP &&
>> +	    sel->target != V4L2_SEL_TGT_COMPOSE)
>> +		return -EINVAL;
>> +
>> +	source_fmt = v4l2_subdev_state_get_format(state,
>> +						  MALI_C55_RZR_SOURCE_PAD);
>> +	sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
>> +	compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
>> +
>> +	/* RAW bypass disables crop/scaling. */
>> +	if (mali_c55_format_is_raw(source_fmt->code)) {
>> +		crop->top = compose->top = 0;
>> +		crop->left = compose->left = 0;
>> +		crop->width = compose->width = sink_fmt->width;
>> +		crop->height = compose->height = sink_fmt->height;
>> +
>> +		sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
>> +
>> +		return 0;
>> +	}
>> +
>> +	/* During streaming, it is allowed to only change the crop rectangle. */
>> +	if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
>> +		return -EINVAL;
>> +
>> +	 /*
>> +	  * Update the desired target and then clamp the crop rectangle to the
>> +	  * sink format sizes and the compose size to the crop sizes.
>> +	  */
>> +	if (sel->target == V4L2_SEL_TGT_CROP)
>> +		*crop = sel->r;
>> +	else
>> +		*compose = sel->r;
>> +
>> +	clamp_t(unsigned int, crop->left, 0,  sink_fmt->width);
>> +	clamp_t(unsigned int, crop->top, 0,  sink_fmt->height);
>> +	clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
>> +		sink_fmt->width - crop->left);
>> +	clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
>> +		sink_fmt->height - crop->top);
>> +
>> +	if (rzr->streaming) {
>> +		/*
>> +		 * Apply at runtime a crop rectangle on the resizer's sink only
>> +		 * if it doesn't require re-programming the scaler output sizes
>> +		 * as it would require changing the output buffer sizes as well.
>> +		 */
>> +		if (sel->r.width < compose->width ||
>> +		    sel->r.height < compose->height)
>> +			return -EINVAL;
>> +
>> +		*crop = sel->r;
>> +		mali_c55_rzr_program(rzr, state);
>> +
>> +		return 0;
>> +	}
>> +
>> +	compose->left = 0;
>> +	compose->top = 0;
>> +	clamp_t(unsigned int, compose->left, 0,  sink_fmt->width);
>> +	clamp_t(unsigned int, compose->top, 0,  sink_fmt->height);
>> +	clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
>> +	clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
>> +
>> +	sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
>> +				    struct v4l2_subdev_state *state,
>> +				    enum v4l2_subdev_format_whence which,
>> +				    struct v4l2_subdev_krouting *routing)
>> +{
>> +	if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
>> +	    media_entity_is_streaming(&sd->entity))
>> +		return -EBUSY;
>> +
>> +	return __mali_c55_rzr_set_routing(sd, state, routing);
>> +}
>> +
>> +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
>> +	.enum_mbus_code		= mali_c55_rzr_enum_mbus_code,
>> +	.enum_frame_size	= mali_c55_rzr_enum_frame_size,
>> +	.get_fmt		= v4l2_subdev_get_fmt,
>> +	.set_fmt		= mali_c55_rzr_set_fmt,
>> +	.get_selection		= mali_c55_rzr_get_selection,
>> +	.set_selection		= mali_c55_rzr_set_selection,
>> +	.set_routing		= mali_c55_rzr_set_routing,
>> +};
>> +
>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
>> +{
>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>> +	struct v4l2_subdev *sd = &rzr->sd;
>> +	struct v4l2_subdev_state *state;
>> +	unsigned int sink_pad;
>> +
>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>> +
>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
>> +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>> +		/* Bypass FR pipe processing if the bypass route is active. */
>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>> +				     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
>> +				     MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
>> +		goto unlock_state;
>> +	}
>> +
>> +	/* Disable bypass and use regular processing. */
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>> +			     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
>> +	mali_c55_rzr_program(rzr, state);
>> +
>> +unlock_state:
>> +	rzr->streaming = true;
>> +	v4l2_subdev_unlock_state(state);
>> +}
>> +
>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
>> +{
>> +	struct v4l2_subdev *sd = &rzr->sd;
>> +	struct v4l2_subdev_state *state;
>> +
>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>> +	rzr->streaming = false;
>> +	v4l2_subdev_unlock_state(state);
>> +}
>> +
>> +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
>> +	.pad	= &mali_c55_resizer_pad_ops,
>> +};
>> +
>> +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
>> +				   struct v4l2_subdev_state *state)
>> +{
>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>> +						    sd);
>> +	struct v4l2_subdev_krouting routing = { };
>> +	struct v4l2_subdev_route *routes;
>> +	unsigned int i;
>> +	int ret;
>> +
>> +	routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
>> +	if (!routes)
>> +		return -ENOMEM;
>> +
>> +	for (i = 0; i < rzr->num_routes; ++i) {
>> +		struct v4l2_subdev_route *route = &routes[i];
>> +
>> +		route->sink_pad = i
>> +				? MALI_C55_RZR_SINK_BYPASS_PAD
>> +				: MALI_C55_RZR_SINK_PAD;
>> +		route->source_pad = MALI_C55_RZR_SOURCE_PAD;
>> +		if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
>> +			route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
>> +	}
>> +
>> +	routing.num_routes = rzr->num_routes;
>> +	routing.routes = routes;
>> +
>> +	ret = __mali_c55_rzr_set_routing(sd, state, &routing);
>> +	kfree(routes);
>> +
>> +	return ret;
>> +}
>> +
>> +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
>> +	.init_state = mali_c55_rzr_init_state,
>> +};
>> +
>> +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
>> +						  unsigned int index)
>> +{
>> +	const unsigned int scaler_filt_coefmem_addrs[][2] = {
>> +		[MALI_C55_RZR_FR] = {
>> +			0x034A8, /* hfilt */
>> +			0x044A8  /* vfilt */
> Lowercase hex constants.
>
>> +		},
>> +		[MALI_C55_RZR_DS] = {
>> +			0x014A8, /* hfilt */
>> +			0x024A8  /* vfilt */
>> +		},
>> +	};
>> +	unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
>> +	unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
>> +	unsigned int i, j;
>> +
>> +	for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
>> +		for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
>> +			mali_c55_write(mali_c55, haddr,
>> +				mali_c55_scaler_h_filter_coefficients[i][j]);
>> +			mali_c55_write(mali_c55, vaddr,
>> +				mali_c55_scaler_v_filter_coefficients[i][j]);
>> +
>> +			haddr += sizeof(u32);
>> +			vaddr += sizeof(u32);
>> +		}
>> +	}
>> +}
>> +
>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
>> +{
>> +	unsigned int i;
>> +	int ret;
>> +
>> +	for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
>> +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
>> +		struct v4l2_subdev *sd = &rzr->sd;
>> +		unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
>> +
>> +		rzr->id = i;
>> +		rzr->streaming = false;
>> +
>> +		if (rzr->id == MALI_C55_RZR_FR)
>> +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
>> +		else
>> +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
>> +
>> +		mali_c55_resizer_program_coefficients(mali_c55, i);
>> +
>> +		v4l2_subdev_init(sd, &mali_c55_resizer_ops);
>> +		sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
>> +			     | V4L2_SUBDEV_FL_STREAMS;
>> +		sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
>> +		sd->internal_ops = &mali_c55_resizer_internal_ops;
>> +		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
>> +			 MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
>> +
>> +		rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
>> +		rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
>> +
>> +		/* Only the FR pipe has a bypass pad. */
>> +		if (rzr->id == MALI_C55_RZR_FR) {
>> +			rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
>> +							MEDIA_PAD_FL_SINK;
>> +			rzr->num_routes = 2;
>> +		} else {
>> +			num_pads -= 1;
>> +			rzr->num_routes = 1;
>> +		}
>> +
>> +		ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
>> +		if (ret)
>> +			return ret;
>> +
>> +		ret = v4l2_subdev_init_finalize(sd);
>> +		if (ret)
>> +			goto err_cleanup;
>> +
>> +		ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>> +		if (ret)
>> +			goto err_cleanup;
>> +
>> +		rzr->mali_c55 = mali_c55;
>> +	}
>> +
>> +	return 0;
>> +
>> +err_cleanup:
>> +	for (; i >= 0; --i) {
>> +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
>> +		struct v4l2_subdev *sd = &rzr->sd;
>> +
>> +		v4l2_subdev_cleanup(sd);
>> +		media_entity_cleanup(&sd->entity);
>> +	}
>> +
>> +	return ret;
>> +}
>> +
>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
>> +{
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
>> +		struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
>> +
>> +		if (!resizer->mali_c55)
>> +			continue;
>> +
>> +		v4l2_device_unregister_subdev(&resizer->sd);
>> +		v4l2_subdev_cleanup(&resizer->sd);
>> +		media_entity_cleanup(&resizer->sd.entity);
>> +	}
>> +}
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>> new file mode 100644
>> index 000000000000..c7e699741c6d
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>> @@ -0,0 +1,402 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ARM Mali-C55 ISP Driver - Test pattern generator
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#include <linux/minmax.h>
>> +#include <linux/string.h>
>> +
>> +#include <media/media-entity.h>
>> +#include <media/v4l2-ctrls.h>
>> +#include <media/v4l2-subdev.h>
>> +
>> +#include "mali-c55-common.h"
>> +#include "mali-c55-registers.h"
>> +
>> +#define MALI_C55_TPG_SRC_PAD		0
>> +#define MALI_C55_TPG_FIXED_HBLANK	0x20
>> +#define MALI_C55_TPG_MAX_VBLANK		0xFFFF
> Lowercase hex constants.
>
>> +#define MALI_C55_TPG_PIXEL_RATE		100000000
> This should be exposed to applications using the V4L2_CID_PIXEL_RATE
> control (read-only).
>
>> +
>> +static const char * const mali_c55_tpg_test_pattern_menu[] = {
>> +	"Flat field",
>> +	"Horizontal gradient",
>> +	"Vertical gradient",
>> +	"Vertical bars",
>> +	"Arbitrary rectangle",
>> +	"White frame on black field"
>> +};
>> +
>> +static const u32 mali_c55_tpg_mbus_codes[] = {
>> +	MEDIA_BUS_FMT_SRGGB20_1X20,
>> +	MEDIA_BUS_FMT_RGB202020_1X60,
>> +};
>> +
>> +static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
>> +				       int *def_vblank, int *min_vblank)
> unsigned int ?
>
>> +{
>> +	unsigned int hts;
>> +	int tgt_fps;
>> +	int vblank;
>> +
>> +	hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
>> +
>> +	/*
>> +	 * The ISP has minimum vertical blanking requirements that must be
>> +	 * adhered to by the TPG. The minimum is a function of the Iridix blocks
>> +	 * clocking requirements and the width of the image and horizontal
>> +	 * blanking, but if we assume the worst case iVariance and sVariance
>> +	 * values then it boils down to the below.
>> +	 */
>> +	*min_vblank = 15 + (120500 / hts);
> I wonder if this should round up.
>
>> +
>> +	/*
>> +	 * We need to set a sensible default vblank for whatever format height
>> +	 * we happen to be given from set_fmt(). This function just targets
>> +	 * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
>> +	 * If we can't get 5fps we'll take whatever the minimum vblank gives us.
>> +	 */
>> +	tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
>> +
>> +	if (tgt_fps < 5)
>> +		vblank = *min_vblank;
>> +	else
>> +		vblank = MALI_C55_TPG_PIXEL_RATE / hts
>> +		       / max(rounddown(tgt_fps, 15), 5);
>> +
>> +	*def_vblank = ALIGN_DOWN(vblank, 2) - format->height;
> "vblank = vblank - height" doesn't seem right. The "else" branch stores
> a vts in vblank, which doesn't seem right either. Maybe you meant
> something like
>
> 	if (tgt_fps < 5)
> 		def_vts = *min_vblank + format->height;
> 	else
> 		def_vts = MALI_C55_TPG_PIXEL_RATE / hts
> 			/ max(rounddown(tgt_fps, 15), 5);
>
> 	*def_vblank = ALIGN_DOWN(def_vts, 2) - format->height;
>
>> +}
>> +
>> +static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
>> +{
>> +	struct mali_c55_tpg *tpg = container_of(ctrl->handler,
>> +						struct mali_c55_tpg,
>> +						ctrls.handler);
>> +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
>> +
> Should you return here if the pipeline isn't streaming ?
>
>> +	switch (ctrl->id) {
>> +	case V4L2_CID_TEST_PATTERN:
>> +		mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
>> +			       ctrl->val);
>> +		break;
>> +	case V4L2_CID_VBLANK:
>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
>> +				     MALI_C55_REG_VBLANK_MASK, ctrl->val);
>> +		break;
>> +	default:
>> +		dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
>> +		return -EINVAL;
> Can this happen ?
>
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
>> +	.s_ctrl = &mali_c55_tpg_s_ctrl,
>> +};
>> +
>> +static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
>> +				   struct v4l2_subdev *sd)
>> +{
>> +	struct v4l2_subdev_state *state;
>> +	struct v4l2_mbus_framefmt *fmt;
>> +
>> +	/*
>> +	 * hblank needs setting, but is a read-only control and thus won't be
>> +	 * called during __v4l2_ctrl_handler_setup(). Do it here instead.
>> +	 */
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
>> +			     MALI_C55_REG_HBLANK_MASK,
>> +			     MALI_C55_TPG_FIXED_HBLANK);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>> +			     MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
>> +
>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>> +			     MALI_C55_TEST_PATTERN_RGB_MASK,
>> +			     fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
>> +					  0x01 : 0x0);
>> +
>> +	v4l2_subdev_unlock_state(state);
>> +}
>> +
>> +static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
>> +{
>> +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
>> +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
>> +
>> +	if (!enable) {
>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>> +				MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>> +				MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
>> +		return 0;
>> +	}
>> +
>> +	/*
>> +	 * One might reasonably expect the framesize to be set here
>> +	 * given it's configurable in .set_fmt(), but it's done in the
>> +	 * ISP subdevice's stream on func instead, as the same register
> s/func/function/
>
>> +	 * is also used to indicate the size of the data coming from the
>> +	 * sensor.
>> +	 */
>> +	mali_c55_tpg_configure(mali_c55, sd);
> 	mali_c55_tpg_configure(tpg);
>
>> +	__v4l2_ctrl_handler_setup(sd->ctrl_handler);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>> +			     MALI_C55_TEST_PATTERN_ON_OFF,
>> +			     MALI_C55_TEST_PATTERN_ON_OFF);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>> +			     MALI_C55_REG_GEN_VIDEO_ON_MASK,
>> +			     MALI_C55_REG_GEN_VIDEO_ON_MASK);
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
>> +	.s_stream = &mali_c55_tpg_s_stream,
> Can we use .enable_streams() and .disable_streams() ?
>
>> +};
>> +
>> +static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
>> +				       struct v4l2_subdev_state *state,
>> +				       struct v4l2_subdev_mbus_code_enum *code)
>> +{
>> +	if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>> +		return -EINVAL;
>> +
>> +	code->code = mali_c55_tpg_mbus_codes[code->index];
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
>> +					struct v4l2_subdev_state *state,
>> +					struct v4l2_subdev_frame_size_enum *fse)
>> +{
> You sohuld verify here that fse->code is a supported value and return
> -EINVAL otherwise.
>
>> +	if (fse->index > 0 || fse->pad > sd->entity.num_pads)
> Drop the pad check, it's done in the subdev core already.
>
>> +		return -EINVAL;
>> +
>> +	fse->min_width = MALI_C55_MIN_WIDTH;
>> +	fse->max_width = MALI_C55_MAX_WIDTH;
>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
>> +				struct v4l2_subdev_state *state,
>> +				struct v4l2_subdev_format *format)
>> +{
>> +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>> +	int vblank_def, vblank_min;
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>> +		if (fmt->code == mali_c55_tpg_mbus_codes[i])
>> +			break;
>> +	}
>> +
>> +	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>> +
>> +	/*
>> +	 * The TPG says that the test frame timing generation logic expects a
>> +	 * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
>> +	 * handle anything smaller than 128x128 it seems pointless to allow a
>> +	 * smaller frame.
>> +	 */
>> +	clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
>> +		MALI_C55_MAX_WIDTH);
>> +	clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
>> +		MALI_C55_MAX_HEIGHT);
>> +
>> +	*v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;
> You're allowing userspace to set fmt->field, as well as all the
> colorspace parameters, to random values. I would instead do something
> like
>
> 	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> 		if (format->format.code == mali_c55_tpg_mbus_codes[i])
> 			break;
> 	}
>
> 	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> 		format->format.code = MEDIA_BUS_FMT_SRGGB20_1X20;
>
> 	format->format.width = clamp(format->format.width,
> 				     MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> 	format->format.height = clamp(format->format.height,
> 				      MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>
> 	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> 	fmt->code = format->format.code;
> 	fmt->width = format->format.width;
> 	fmt->height = format->format.height;
>
> 	format->format = *fmt;
>
> Alternatively (which I think I like better),
>
> 	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>
> 	fmt->code = format->format.code;
>
> 	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> 		if (fmt->code == mali_c55_tpg_mbus_codes[i])
> 			break;
> 	}
>
> 	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> 		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>
> 	fmt->width = clamp(format->format.width,
> 			   MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> 	fmt->height = clamp(format->format.height,
> 			    MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>
> 	format->format = *fmt;
>
>> +
>> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY)
>> +		return 0;
>> +
>> +	__mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
>> +	__v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
>> +				 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
>> +	__v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);
> Move those three calls to a separate function, it will be reused below.
> I'd name is mali_c55_tpg_update_vblank(). You can fold
> __mali_c55_tpg_calc_vblank() in it.
>
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
>> +	.enum_mbus_code		= mali_c55_tpg_enum_mbus_code,
>> +	.enum_frame_size	= mali_c55_tpg_enum_frame_size,
>> +	.get_fmt		= v4l2_subdev_get_fmt,
>> +	.set_fmt		= mali_c55_tpg_set_fmt,
>> +};
>> +
>> +static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
>> +	.video	= &mali_c55_tpg_video_ops,
>> +	.pad	= &mali_c55_tpg_pad_ops,
>> +};
>> +
>> +static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
>> +				   struct v4l2_subdev_state *sd_state)
> You name this variable state in every other subdev operation handler.
>
>> +{
>> +	struct v4l2_mbus_framefmt *fmt =
>> +		v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
>> +
>> +	fmt->width = MALI_C55_DEFAULT_WIDTH;
>> +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
>> +	fmt->field = V4L2_FIELD_NONE;
>> +	fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> Initialize the colorspace fields too.
>
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
>> +	.init_state = mali_c55_tpg_init_state,
>> +};
>> +
>> +static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
>> +	struct v4l2_subdev *sd = &mali_c55->tpg.sd;
>> +	struct v4l2_mbus_framefmt *format;
>> +	struct v4l2_subdev_state *state;
>> +	int vblank_def, vblank_min;
>> +	int ret;
>> +
>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>> +	format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>> +
>> +	ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);
> You have 3 controls.
>
>> +	if (ret)
>> +		goto err_unlock;
>> +
>> +	ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
>> +				&mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
>> +				ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
>> +				0, 3, mali_c55_tpg_test_pattern_menu);
>> +
>> +	/*
>> +	 * We fix hblank at the minimum allowed value and control framerate
>> +	 * solely through the vblank control.
>> +	 */
>> +	ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
>> +				&mali_c55_tpg_ctrl_ops,
>> +				V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
>> +				MALI_C55_TPG_FIXED_HBLANK, 1,
>> +				MALI_C55_TPG_FIXED_HBLANK);
>> +	if (ctrls->hblank)
>> +		ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
>> +
>> +	__mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);
> Drop this and initialize the control with default values. You can then
> update the value by calling mali_c55_tpg_update_vblank() in
> mali_c55_register_tpg().
>
> The reason is to share the same mutex between the control handler and
> the subdev active state without having to add a separate mutex in the
> mali_c55_tpg structure. The simplest way to do so is to initialize the
> controls first, set sd->state_lock to point to the control handler lock,
> and call v4l2_subdev_init_finalize() as the last step. As a consequence,
> you can't access the active state when initializing controls.
>
> You can alternatively keep the lock in mali_c55_tpg and set
> sd->state_lock to point to it, but I think that's more complex.
>
>> +	ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
>> +					  &mali_c55_tpg_ctrl_ops,
>> +					  V4L2_CID_VBLANK, vblank_min,
>> +					  MALI_C55_TPG_MAX_VBLANK, 1,
>> +					  vblank_def);
>> +
>> +	if (ctrls->handler.error) {
>> +		dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
>> +		ret = ctrls->handler.error;
>> +		goto err_free_handler;
>> +	}
>> +
>> +	ctrls->handler.lock = &mali_c55->tpg.lock;
> Drop this and drop the mutex. The control handler will use its internal
> mutex.
>
>> +	mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
>> +
>> +	v4l2_subdev_unlock_state(state);
>> +
>> +	return 0;
>> +
>> +err_free_handler:
>> +	v4l2_ctrl_handler_free(&ctrls->handler);
>> +err_unlock:
>> +	v4l2_subdev_unlock_state(state);
>> +	return ret;
>> +}
>> +
>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
>> +	struct v4l2_subdev *sd = &tpg->sd;
>> +	struct media_pad *pad = &tpg->pad;
>> +	int ret;
>> +
>> +	mutex_init(&tpg->lock);
>> +
>> +	v4l2_subdev_init(sd, &mali_c55_tpg_ops);
>> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
>> +	sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
> Should we introduce a TPG function ?
>
>> +	sd->internal_ops = &mali_c55_tpg_internal_ops;
>> +	strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
>> +
>> +	pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;
> I don't think MEDIA_PAD_FL_MUST_CONNECT is right.
>
>> +	ret = media_entity_pads_init(&sd->entity, 1, pad);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev,
>> +			"Failed to initialize media entity pads\n");
>> +		goto err_destroy_mutex;
>> +	}
>> +
> 	sd->state_lock = sd->ctrl_handler->lock;
>
> to use the same lock for the controls and the active state. You need to
> move this line and the v4l2_subdev_init_finalize() call after
> mali_c55_tpg_init_controls() to get the control handler lock initialized
> first.
>
>> +	ret = v4l2_subdev_init_finalize(sd);
>> +	if (ret)
>> +		goto err_cleanup_media_entity;
>> +
>> +	ret = mali_c55_tpg_init_controls(mali_c55);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev,
>> +			"Error initialising controls\n");
>> +		goto err_cleanup_subdev;
>> +	}
>> +
>> +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
>> +		goto err_free_ctrl_handler;
>> +	}
>> +
>> +	/*
>> +	 * By default the colour settings lead to a very dim image that is
>> +	 * nearly indistinguishable from black on some monitor settings. Ramp
>> +	 * them up a bit so the image is brighter.
>> +	 */
>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
>> +		       MALI_C55_TPG_BACKGROUND_MAX);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
>> +		       MALI_C55_TPG_BACKGROUND_MAX);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
>> +		       MALI_C55_TPG_BACKGROUND_MAX);
>> +
>> +	tpg->mali_c55 = mali_c55;
>> +
>> +	return 0;
>> +
>> +err_free_ctrl_handler:
>> +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
>> +err_cleanup_subdev:
>> +	v4l2_subdev_cleanup(sd);
>> +err_cleanup_media_entity:
>> +	media_entity_cleanup(&sd->entity);
>> +err_destroy_mutex:
>> +	mutex_destroy(&tpg->lock);
>> +
>> +	return ret;
>> +}
>> +
>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
>> +
>> +	if (!tpg->mali_c55)
>> +		return;
>> +
>> +	v4l2_device_unregister_subdev(&tpg->sd);
>> +	v4l2_subdev_cleanup(&tpg->sd);
>> +	media_entity_cleanup(&tpg->sd.entity);
>> +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> Free the control handler just after v4l2_device_unregister_subdev() to
> match the order in mali_c55_register_tpg().
>
>> +	mutex_destroy(&tpg->lock);
>> +}

Re: [PATCH v5 15/16] media: platform: Add mali-c55 parameters video node

[Sakari Ailus]

Hi Jacopo, Dan,

Thanks for the patch. Please see my comments below.

On Wed, May 29, 2024 at 04:28:57PM +0100, Daniel Scally wrote:
> Add a new code file to the mali-c55 driver that registers an output
> video node for userspace to queue buffers of parameters to. Handlers
> are included to program the statistics generation plus the white
> balance, black level correction and mesh shading correction blocks.
> 
> Update the rest of the driver to register and link the new video node
> 
> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
> Changes in v5:
> 
> 	- New patch
> 
>  drivers/media/platform/arm/mali-c55/Makefile  |   1 +
>  .../platform/arm/mali-c55/mali-c55-common.h   |  18 +
>  .../platform/arm/mali-c55/mali-c55-core.c     |  24 +
>  .../platform/arm/mali-c55/mali-c55-isp.c      |  16 +-
>  .../platform/arm/mali-c55/mali-c55-params.c   | 615 ++++++++++++++++++
>  .../arm/mali-c55/mali-c55-registers.h         | 104 +++
>  6 files changed, 777 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-params.c
> 
> diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
> index cd5a64bf0c62..b2443f2d416a 100644
> --- a/drivers/media/platform/arm/mali-c55/Makefile
> +++ b/drivers/media/platform/arm/mali-c55/Makefile
> @@ -5,6 +5,7 @@ mali-c55-y := mali-c55-capture.o \
>  	      mali-c55-isp.o \
>  	      mali-c55-tpg.o \
>  	      mali-c55-resizer.o \
> +	      mali-c55-params.o \
>  	      mali-c55-stats.o
>  
>  obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> index 44119e04009b..565d98acfcdd 100644
> --- a/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> @@ -80,6 +80,7 @@ enum mali_c55_isp_pads {
>  	MALI_C55_ISP_PAD_SOURCE,
>  	MALI_C55_ISP_PAD_SOURCE_BYPASS,
>  	MALI_C55_ISP_PAD_SOURCE_3A,
> +	MALI_C55_ISP_PAD_SINK_PARAMS,
>  	MALI_C55_ISP_NUM_PADS,
>  };
>  
> @@ -217,6 +218,19 @@ struct mali_c55_stats {
>  	} buffers;
>  };
>  
> +struct mali_c55_params {
> +	struct mali_c55 *mali_c55;
> +	struct video_device vdev;
> +	struct vb2_queue queue;
> +	struct media_pad pad;
> +	struct mutex lock;
> +
> +	struct {
> +		spinlock_t lock;
> +		struct list_head queue;
> +	} buffers;
> +};
> +
>  enum mali_c55_config_spaces {
>  	MALI_C55_CONFIG_PING,
>  	MALI_C55_CONFIG_PONG,
> @@ -247,6 +261,7 @@ struct mali_c55 {
>  	struct mali_c55_isp isp;
>  	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
>  	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
> +	struct mali_c55_params params;
>  	struct mali_c55_stats stats;
>  
>  	struct list_head contexts;
> @@ -271,6 +286,8 @@ int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
>  void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
>  int mali_c55_register_stats(struct mali_c55 *mali_c55);
>  void mali_c55_unregister_stats(struct mali_c55 *mali_c55);
> +int mali_c55_register_params(struct mali_c55 *mali_c55);
> +void mali_c55_unregister_params(struct mali_c55 *mali_c55);
>  struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
>  void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>  			     enum mali_c55_planes plane);
> @@ -290,5 +307,6 @@ bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
>  	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
>  void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
>  				enum mali_c55_config_spaces cfg_space);
> +void mali_c55_params_write_config(struct mali_c55 *mali_c55);
>  
>  #endif /* _MALI_C55_COMMON_H */
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> index 2cf8b1169604..6acee3edd03f 100644
> --- a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> @@ -347,6 +347,17 @@ static int mali_c55_create_links(struct mali_c55 *mali_c55)
>  		goto err_remove_links;
>  	}
>  
> +	ret = media_create_pad_link(&mali_c55->params.vdev.entity, 0,
> +				    &mali_c55->isp.sd.entity,
> +				    MALI_C55_ISP_PAD_SINK_PARAMS,
> +				    MEDIA_LNK_FL_ENABLED |
> +				    MEDIA_LNK_FL_IMMUTABLE);
> +	if (ret) {
> +		dev_err(mali_c55->dev,
> +			"failed to link ISP and parameters video node\n");
> +		goto err_remove_links;
> +	}
> +
>  	return 0;
>  
>  err_remove_links:
> @@ -360,6 +371,7 @@ static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
>  	mali_c55_unregister_isp(mali_c55);
>  	mali_c55_unregister_resizers(mali_c55);
>  	mali_c55_unregister_capture_devs(mali_c55);
> +	mali_c55_unregister_params(mali_c55);
>  	mali_c55_unregister_stats(mali_c55);
>  }
>  
> @@ -383,6 +395,10 @@ static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>  	if (ret)
>  		goto err_unregister_entities;
>  
> +	ret = mali_c55_register_params(mali_c55);
> +	if (ret)
> +		goto err_unregister_entities;
> +
>  	ret = mali_c55_register_stats(mali_c55);
>  	if (ret)
>  		goto err_unregister_entities;
> @@ -474,6 +490,14 @@ static irqreturn_t mali_c55_isr(int irq, void *context)
>  			curr_config >>= ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1;
>  			next_config = curr_config ^ 1;
>  
> +			/*
> +			 * Write the configuration parameters received from
> +			 * userspace into the configuration buffer, which will
> +			 * be transferred to the 'next' active config space at
> +			 * by mali_c55_swap_next_config().
> +			 */
> +			mali_c55_params_write_config(mali_c55);
> +
>  			/*
>  			 * The ordering of these two is currently important as
>  			 * mali_c55_stats_fill_buffer() is asynchronous whereas
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> index 94876fba3353..8c2b45bfd82d 100644
> --- a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> @@ -146,6 +146,7 @@ static int mali_c55_isp_start(struct mali_c55 *mali_c55)
>  			     cfg->encoding == V4L2_PIXEL_ENC_RGB ?
>  			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
>  
> +	mali_c55_params_write_config(mali_c55);
>  	ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
>  	if (ret) {
>  		dev_err(mali_c55->dev, "failed to DMA config\n");
> @@ -455,8 +456,20 @@ static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
>  	.init_state = mali_c55_isp_init_state,
>  };
>  
> +static int mali_c55_subdev_link_validate(struct media_link *link)
> +{
> +	/*
> +	 * Skip validation for the parameters sink pad, as the source is not
> +	 * a subdevice.
> +	 */
> +	if (link->sink->index == MALI_C55_ISP_PAD_SINK_PARAMS)
> +		return 0;
> +
> +	return v4l2_subdev_link_validate(link);
> +}
> +
>  static const struct media_entity_operations mali_c55_isp_media_ops = {
> -	.link_validate		= v4l2_subdev_link_validate,
> +	.link_validate		= mali_c55_subdev_link_validate,
>  };
>  
>  static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
> @@ -565,6 +578,7 @@ int mali_c55_register_isp(struct mali_c55 *mali_c55)
>  	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>  	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
>  	isp->pads[MALI_C55_ISP_PAD_SOURCE_3A].flags = MEDIA_PAD_FL_SOURCE;
> +	isp->pads[MALI_C55_ISP_PAD_SINK_PARAMS].flags = MEDIA_PAD_FL_SINK;
>  
>  	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
>  				     isp->pads);
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-params.c b/drivers/media/platform/arm/mali-c55/mali-c55-params.c
> new file mode 100644
> index 000000000000..049a7b8e4861
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-params.c
> @@ -0,0 +1,615 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ARM Mali-C55 ISP Driver - Configuration parameters output device
> + *
> + * Copyright (C) 2024 Ideas on Board Oy
> + */
> +#include <linux/media/arm/mali-c55-config.h>
> +
> +#include <media/media-entity.h>
> +#include <media/v4l2-dev.h>
> +#include <media/v4l2-event.h>
> +#include <media/v4l2-fh.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/videobuf2-core.h>
> +#include <media/videobuf2-dma-contig.h>
> +
> +#include "mali-c55-common.h"
> +#include "mali-c55-registers.h"
> +
> +typedef void (*mali_c55_block_handler)(struct mali_c55 *mali_c55,

You can wrap after the return type (including typedef). Same elsewhere.

> +				       struct mali_c55_params_block_header *block);
> +
> +struct mali_c55_block_handler {
> +	size_t size;
> +	mali_c55_block_handler handler;
> +};
> +
> +static void mali_c55_params_sensor_offs(struct mali_c55 *mali_c55,
> +					struct mali_c55_params_block_header *block)
> +{
> +	struct mali_c55_params_sensor_off_preshading *p =
> +		(struct mali_c55_params_sensor_off_preshading *)block;

I wonder if an union could be used to make this a bit cleaner. You're doing
a lot of casting that I think could be avoided.

> +	__u32 global_offset;
> +
> +	if (!block->enabled)
> +		return;
> +
> +	if (!(p->chan00 || p->chan01 || p->chan10 || p->chan11))
> +		return;
> +
> +	mali_c55_write(mali_c55, MALI_C55_REG_SENSOR_OFF_PRE_SHA_00,
> +		       p->chan00 & MALI_C55_SENSOR_OFF_PRE_SHA_MASK);
> +	mali_c55_write(mali_c55, MALI_C55_REG_SENSOR_OFF_PRE_SHA_01,
> +		       p->chan01 & MALI_C55_SENSOR_OFF_PRE_SHA_MASK);
> +	mali_c55_write(mali_c55, MALI_C55_REG_SENSOR_OFF_PRE_SHA_10,
> +		       p->chan10 & MALI_C55_SENSOR_OFF_PRE_SHA_MASK);
> +	mali_c55_write(mali_c55, MALI_C55_REG_SENSOR_OFF_PRE_SHA_11,
> +		       p->chan11 & MALI_C55_SENSOR_OFF_PRE_SHA_MASK);
> +
> +	/*
> +	 * The average offset is applied as a global offset for the digital
> +	 * gain block
> +	 */
> +	global_offset = (p->chan00 + p->chan01 + p->chan10 + p->chan11) >> 2;
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_DIGITAL_GAIN_OFFSET,
> +			     MALI_C55_DIGITAL_GAIN_OFFSET_MASK, global_offset);
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
> +			     MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH, 0x00);
> +}
> +
> +static void mali_c55_params_aexp_hist(struct mali_c55 *mali_c55,
> +				struct mali_c55_params_block_header *block)
> +{
> +	u32 disable_mask = block->type == MALI_C55_PARAM_BLOCK_AEXP_HIST ?
> +					  MALI_C55_AEXP_HIST_DISABLE_MASK :
> +					  MALI_C55_AEXP_IHIST_DISABLE_MASK;
> +	u32 base = block->type == MALI_C55_PARAM_BLOCK_AEXP_HIST ?
> +				  MALI_C55_REG_AEXP_HIST_BASE :
> +				  MALI_C55_REG_AEXP_IHIST_BASE;
> +	struct mali_c55_params_aexp_hist *params =
> +		(struct mali_c55_params_aexp_hist *)block;
> +
> +	if (!block->enabled) {
> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
> +				     disable_mask, true);
> +		return;
> +	}
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
> +			     disable_mask, false);
> +
> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SKIP_OFFSET,
> +			     MALI_C55_AEXP_HIST_SKIP_X_MASK, params->skip_x);
> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SKIP_OFFSET,
> +			     MALI_C55_AEXP_HIST_OFFSET_X_MASK, params->offset_x);
> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SKIP_OFFSET,
> +			     MALI_C55_AEXP_HIST_SKIP_Y_MASK, params->skip_y);
> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SKIP_OFFSET,
> +			     MALI_C55_AEXP_HIST_OFFSET_Y_MASK, params->offset_y);
> +
> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SCALE_OFFSET,
> +			     MALI_C55_AEXP_HIST_SCALE_BOTTOM_MASK, params->scale_bottom);
> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SCALE_OFFSET,
> +			     MALI_C55_AEXP_HIST_SCALE_TOP_MASK, params->scale_top);
> +
> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_PLANE_MODE_OFFSET,
> +			     MALI_C55_AEXP_HIST_PLANE_MODE_MASK, params->plane_mode);
> +
> +	if (block->type == MALI_C55_PARAM_BLOCK_AEXP_HIST)
> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
> +				     MALI_C55_AEXP_HIST_SWITCH_MASK,
> +				     params->tap_point);
> +}
> +
> +static void
> +mali_c55_params_aexp_hist_weights(struct mali_c55 *mali_c55,
> +				  struct mali_c55_params_block_header *block)
> +{
> +	struct mali_c55_params_aexp_weights *params =
> +		(struct mali_c55_params_aexp_weights *)block;
> +	u32 base;
> +
> +	if (!block->enabled)
> +		return;
> +
> +	base = block->type == MALI_C55_PARAM_BLOCK_AEXP_HIST_WEIGHTS ?
> +			      MALI_C55_REG_AEXP_HIST_BASE :
> +			      MALI_C55_REG_AEXP_IHIST_BASE;
> +
> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_NODES_USED_OFFSET,
> +			     MALI_C55_AEXP_HIST_NODES_USED_HORIZ_MASK, params->nodes_used_horiz);
> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_NODES_USED_OFFSET,
> +			     MALI_C55_AEXP_HIST_NODES_USED_VERT_MASK, params->nodes_used_vert);
> +
> +	/*
> +	 * The zone weights array is a 225-element array of u8 values, but that
> +	 * is a bit annoying to handle given the ISP expects 32-bit writes. We
> +	 * just reinterpret it as a 57-element array of 32-bit values for the
> +	 * purposes of this transaction (the 3 bytes of additional space at the
> +	 * end of the write is just padding for the array of weights in the ISP
> +	 * memory space anyway, so there's no risk of overwriting other
> +	 * registers).
> +	 */
> +	for (unsigned int i = 0; i < 57; i++) {
> +		u32 val = ((u32 *)params->zone_weights)[i]
> +			    & MALI_C55_AEXP_HIST_ZONE_WEIGHT_MASK;
> +		u32 addr = base + MALI_C55_AEXP_HIST_ZONE_WEIGHTS_OFFSET + (4 * i);
> +
> +		mali_c55_write(mali_c55, addr, val);
> +	}
> +}
> +
> +static void mali_c55_params_digital_gain(struct mali_c55 *mali_c55,
> +					 struct mali_c55_params_block_header *block)
> +{
> +	struct mali_c55_params_digital_gain *dgain =
> +		(struct mali_c55_params_digital_gain *)block;
> +
> +	/*
> +	 * If the block is flagged as disabled we write a gain of 1.0, which in
> +	 * Q5.8 format is 256.
> +	 */
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_DIGITAL_GAIN,
> +			     MALI_C55_DIGITAL_GAIN_MASK,
> +			     block->enabled ? dgain->gain : 256);
> +}
> +
> +static void mali_c55_params_awb_gains(struct mali_c55 *mali_c55,
> +				      struct mali_c55_params_block_header *block)
> +{
> +	struct mali_c55_params_awb_gains *gains =
> +		(struct mali_c55_params_awb_gains *)block;
> +
> +	/*
> +	 * There are two places AWB gains can be set in the ISP; one affects the
> +	 * image output data and the other affects the statistics for the
> +	 * AEXP-0 tap point.
> +	 */
> +	u32 addr1 = block->type = MALI_C55_PARAM_BLOCK_AWB_GAINS ?
> +				  MALI_C55_REG_AWB_GAINS1 :
> +				  MALI_C55_REG_AWB_GAINS1_AEXP;
> +	u32 addr2 = block->type = MALI_C55_PARAM_BLOCK_AWB_GAINS ?
> +				  MALI_C55_REG_AWB_GAINS2 :
> +				  MALI_C55_REG_AWB_GAINS2_AEXP;
> +
> +	mali_c55_update_bits(mali_c55, addr1, MALI_C55_AWB_GAIN00_MASK,
> +			     gains->gain00);
> +	mali_c55_update_bits(mali_c55, addr1, MALI_C55_AWB_GAIN01_MASK,
> +			     gains->gain01);
> +	mali_c55_update_bits(mali_c55, addr2, MALI_C55_AWB_GAIN10_MASK,
> +			     gains->gain10);
> +	mali_c55_update_bits(mali_c55, addr2, MALI_C55_AWB_GAIN11_MASK,
> +			     gains->gain11);
> +}
> +
> +static void mali_c55_params_awb_config(struct mali_c55 *mali_c55,
> +				      struct mali_c55_params_block_header *block)
> +{
> +	struct mali_c55_params_awb_config *params =
> +		(struct mali_c55_params_awb_config *)block;
> +
> +	if (!block->enabled) {
> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
> +				     MALI_C55_AWB_DISABLE_MASK, true);
> +		return;
> +	}
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
> +			     MALI_C55_AWB_DISABLE_MASK, false);
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_STATS_MODE,
> +			     MALI_C55_AWB_STATS_MODE_MASK, params->stats_mode);
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_WHITE_LEVEL,
> +			     MALI_C55_AWB_WHITE_LEVEL_MASK, params->white_level);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_BLACK_LEVEL,
> +			     MALI_C55_AWB_BLACK_LEVEL_MASK, params->black_level);
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CR_MAX,
> +			     MALI_C55_AWB_CR_MAX_MASK, params->cr_max);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CR_MIN,
> +			     MALI_C55_AWB_CR_MIN_MASK, params->cr_min);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CB_MAX,
> +			     MALI_C55_AWB_CB_MAX_MASK, params->cb_max);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CB_MIN,
> +			     MALI_C55_AWB_CB_MIN_MASK, params->cb_min);
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_NODES_USED,
> +			     MALI_C55_AWB_NODES_USED_HORIZ_MASK,
> +			     params->nodes_used_horiz);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_NODES_USED,
> +			     MALI_C55_AWB_NODES_USED_VERT_MASK,
> +			     params->nodes_used_vert);
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CR_HIGH,
> +			     MALI_C55_AWB_CR_HIGH_MASK, params->cr_high);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CR_LOW,
> +			     MALI_C55_AWB_CR_LOW_MASK, params->cr_low);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CB_HIGH,
> +			     MALI_C55_AWB_CB_HIGH_MASK, params->cb_high);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CB_LOW,
> +			     MALI_C55_AWB_CB_LOW_MASK, params->cb_low);
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
> +			     MALI_C55_AWB_SWITCH_MASK, params->tap_point);
> +}
> +
> +static void mali_c55_params_lsc_config(struct mali_c55 *mali_c55,
> +				       struct mali_c55_params_block_header *block)
> +{
> +	struct mali_c55_params_mesh_shading_config *params =
> +		(struct mali_c55_params_mesh_shading_config *)block;
> +	unsigned int i;
> +	u32 addr;
> +
> +	if (!block->enabled) {
> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
> +				     MALI_C55_MESH_SHADING_ENABLE_MASK, false);
> +		return;
> +	}
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
> +			     MALI_C55_MESH_SHADING_ENABLE_MASK, true);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
> +			     MALI_C55_MESH_SHADING_MESH_SHOW, params->mesh_show);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
> +			     MALI_C55_MESH_SHADING_SCALE_MASK,
> +			     params->mesh_scale);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
> +			     MALI_C55_MESH_SHADING_PAGE_R_MASK,
> +			     params->mesh_page_r);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
> +			     MALI_C55_MESH_SHADING_PAGE_G_MASK,
> +			     params->mesh_page_g);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
> +			     MALI_C55_MESH_SHADING_PAGE_B_MASK,
> +			     params->mesh_page_b);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
> +			     MALI_C55_MESH_SHADING_MESH_WIDTH_MASK,
> +			     params->mesh_width);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
> +			     MALI_C55_MESH_SHADING_MESH_HEIGHT_MASK,
> +			     params->mesh_height);
> +
> +	for (i = 0; i < MALI_C55_NUM_MESH_SHADING_ELEMENTS; i++) {
> +		addr = MALI_C55_REG_MESH_SHADING_TABLES + (i * 4);
> +		mali_c55_write(mali_c55, addr, params->mesh[i]);
> +	}
> +}
> +
> +static void mali_c55_params_lsc_selection(struct mali_c55 *mali_c55,
> +					  struct mali_c55_params_block_header *block)
> +{
> +	struct mali_c55_params_mesh_shading_selection *params =
> +		(struct mali_c55_params_mesh_shading_selection *)block;
> +
> +	if (!block->enabled)
> +		return;
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA_BANK,
> +			     MALI_C55_MESH_SHADING_ALPHA_BANK_R_MASK,
> +			     params->mesh_alpha_bank_r);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA_BANK,
> +			     MALI_C55_MESH_SHADING_ALPHA_BANK_G_MASK,
> +			     params->mesh_alpha_bank_g);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA_BANK,
> +			     MALI_C55_MESH_SHADING_ALPHA_BANK_B_MASK,
> +			     params->mesh_alpha_bank_b);
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA,
> +			     MALI_C55_MESH_SHADING_ALPHA_R_MASK,
> +			     params->mesh_alpha_r);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA,
> +			     MALI_C55_MESH_SHADING_ALPHA_G_MASK,
> +			     params->mesh_alpha_g);
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA,
> +			     MALI_C55_MESH_SHADING_ALPHA_B_MASK,
> +			     params->mesh_alpha_b);
> +
> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_MESH_STRENGTH,
> +			     MALI_c55_MESH_STRENGTH_MASK,
> +			     params->mesh_strength);
> +}
> +
> +static const struct mali_c55_block_handler mali_c55_block_handlers[] = {
> +	[MALI_C55_PARAM_BLOCK_SENSOR_OFFS] = {
> +		.size = sizeof(struct mali_c55_params_sensor_off_preshading),
> +		.handler = &mali_c55_params_sensor_offs,
> +	},
> +	[MALI_C55_PARAM_BLOCK_AEXP_HIST] = {
> +		.size = sizeof(struct mali_c55_params_aexp_hist),
> +		.handler = &mali_c55_params_aexp_hist,
> +	},
> +	[MALI_C55_PARAM_BLOCK_AEXP_IHIST] = {
> +		.size = sizeof(struct mali_c55_params_aexp_hist),
> +		.handler = &mali_c55_params_aexp_hist,
> +	},
> +	[MALI_C55_PARAM_BLOCK_AEXP_HIST_WEIGHTS] = {
> +		.size = sizeof(struct mali_c55_params_aexp_weights),
> +		.handler = &mali_c55_params_aexp_hist_weights,
> +	},
> +	[MALI_C55_PARAM_BLOCK_AEXP_IHIST_WEIGHTS] = {
> +		.size = sizeof(struct mali_c55_params_aexp_weights),
> +		.handler = &mali_c55_params_aexp_hist_weights,
> +	},
> +	[MALI_C55_PARAM_BLOCK_DIGITAL_GAIN] = {
> +		.size = sizeof(struct mali_c55_params_digital_gain),
> +		.handler = &mali_c55_params_digital_gain,
> +	},
> +	[MALI_C55_PARAM_BLOCK_AWB_GAINS] = {
> +		.size = sizeof(struct mali_c55_params_awb_gains),
> +		.handler = &mali_c55_params_awb_gains,
> +	},
> +	[MALI_C55_PARAM_BLOCK_AWB_CONFIG] = {
> +		.size = sizeof(struct mali_c55_params_awb_config),
> +		.handler = &mali_c55_params_awb_config,
> +	},
> +	[MALI_C55_PARAM_BLOCK_AWB_GAINS_AEXP] = {
> +		.size = sizeof(struct mali_c55_params_awb_gains),
> +		.handler = &mali_c55_params_awb_gains,
> +	},
> +	[MALI_C55_PARAM_MESH_SHADING_CONFIG] = {
> +		.size = sizeof(struct mali_c55_params_mesh_shading_config),
> +		.handler = &mali_c55_params_lsc_config,
> +	},
> +	[MALI_C55_PARAM_MESH_SHADING_SELECTION] = {
> +		.size = sizeof(struct mali_c55_params_mesh_shading_selection),
> +		.handler = &mali_c55_params_lsc_selection,
> +	},
> +};
> +
> +static int mali_c55_params_enum_fmt_meta_out(struct file *file, void *fh,
> +					    struct v4l2_fmtdesc *f)
> +{
> +	if (f->index || f->type != V4L2_BUF_TYPE_META_OUTPUT)

The buffer type check has been done by the caller already.

> +		return -EINVAL;
> +
> +	f->pixelformat = V4L2_META_FMT_MALI_C55_PARAMS;
> +
> +	return 0;
> +}
> +
> +static int mali_c55_params_g_fmt_meta_out(struct file *file, void *fh,
> +					 struct v4l2_format *f)
> +{
> +	static const struct v4l2_meta_format mfmt = {
> +		.dataformat = V4L2_META_FMT_MALI_C55_PARAMS,
> +		.buffersize = sizeof(struct mali_c55_params_buffer),
> +	};
> +
> +	if (f->type != V4L2_BUF_TYPE_META_OUTPUT)
> +		return -EINVAL;

Ditto.

Maybe check the other instances of format access functions in the driver,
too?

> +
> +	f->fmt.meta = mfmt;
> +
> +	return 0;
> +}
> +
> +static int mali_c55_params_querycap(struct file *file,
> +				   void *priv, struct v4l2_capability *cap)
> +{
> +	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
> +	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_ioctl_ops mali_c55_params_v4l2_ioctl_ops = {
> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
> +	.vidioc_querybuf = vb2_ioctl_querybuf,
> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
> +	.vidioc_qbuf = vb2_ioctl_qbuf,
> +	.vidioc_expbuf = vb2_ioctl_expbuf,
> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> +	.vidioc_streamon = vb2_ioctl_streamon,
> +	.vidioc_streamoff = vb2_ioctl_streamoff,
> +	.vidioc_enum_fmt_meta_out = mali_c55_params_enum_fmt_meta_out,
> +	.vidioc_g_fmt_meta_out = mali_c55_params_g_fmt_meta_out,
> +	.vidioc_s_fmt_meta_out = mali_c55_params_g_fmt_meta_out,
> +	.vidioc_try_fmt_meta_out = mali_c55_params_g_fmt_meta_out,
> +	.vidioc_querycap = mali_c55_params_querycap,
> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
> +};
> +
> +static const struct v4l2_file_operations mali_c55_params_v4l2_fops = {
> +	.owner = THIS_MODULE,
> +	.unlocked_ioctl = video_ioctl2,
> +	.open = v4l2_fh_open,
> +	.release = vb2_fop_release,
> +	.poll = vb2_fop_poll,
> +	.mmap = vb2_fop_mmap,
> +};
> +
> +static int
> +mali_c55_params_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
> +			   unsigned int *num_planes, unsigned int sizes[],
> +			   struct device *alloc_devs[])
> +{
> +	if (*num_planes && *num_planes > 1)
> +		return -EINVAL;
> +
> +	if (sizes[0] && sizes[0] != sizeof(struct mali_c55_params_buffer))
> +		return -EINVAL;
> +
> +	*num_planes = 1;
> +	sizes[0] = sizeof(struct mali_c55_params_buffer);
> +
> +	return 0;
> +}
> +
> +static void mali_c55_params_buf_queue(struct vb2_buffer *vb)
> +{
> +	struct mali_c55_params *params = vb2_get_drv_priv(vb->vb2_queue);
> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> +	struct mali_c55_buffer *buf = container_of(vbuf,
> +						   struct mali_c55_buffer, vb);
> +
> +	vb2_set_plane_payload(vb, 0, sizeof(struct mali_c55_params_buffer));
> +
> +	spin_lock(&params->buffers.lock);
> +	list_add_tail(&buf->queue, &params->buffers.queue);
> +	spin_unlock(&params->buffers.lock);
> +}
> +
> +static void mali_c55_params_stop_streaming(struct vb2_queue *q)
> +{
> +	struct mali_c55_params *params = vb2_get_drv_priv(q);
> +	struct mali_c55_buffer *buf, *tmp;
> +
> +	spin_lock(&params->buffers.lock);
> +
> +	list_for_each_entry_safe(buf, tmp, &params->buffers.queue, queue) {
> +		list_del(&buf->queue);
> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
> +	}
> +
> +	spin_unlock(&params->buffers.lock);
> +}
> +
> +static const struct vb2_ops mali_c55_params_vb2_ops = {
> +	.queue_setup = mali_c55_params_queue_setup,
> +	.buf_queue = mali_c55_params_buf_queue,
> +	.wait_prepare = vb2_ops_wait_prepare,
> +	.wait_finish = vb2_ops_wait_finish,
> +	.stop_streaming = mali_c55_params_stop_streaming,
> +};
> +
> +void mali_c55_params_write_config(struct mali_c55 *mali_c55)
> +{
> +	struct mali_c55_params *params = &mali_c55->params;
> +	enum vb2_buffer_state state = VB2_BUF_STATE_DONE;
> +	struct mali_c55_params_buffer *config;
> +	struct mali_c55_buffer *buf;
> +	size_t block_offset = 0;
> +
> +	spin_lock(&params->buffers.lock);
> +
> +	buf = list_first_entry_or_null(&params->buffers.queue,
> +				       struct mali_c55_buffer, queue);
> +	if (buf)
> +		list_del(&buf->queue);
> +	spin_unlock(&params->buffers.lock);
> +
> +	if (!buf)
> +		return;
> +
> +	buf->vb.sequence = mali_c55->isp.frame_sequence;
> +	config = vb2_plane_vaddr(&buf->vb.vb2_buf, 0);
> +
> +	if (config->total_size > MALI_C55_PARAMS_MAX_SIZE) {
> +		dev_dbg(mali_c55->dev, "Invalid parameters buffer size %lu\n",
> +			config->total_size);
> +		state = VB2_BUF_STATE_ERROR;
> +		goto err_buffer_done;
> +	}
> +
> +	/* Walk the list of parameter blocks and process them. */
> +	while (block_offset < config->total_size) {
> +		const struct mali_c55_block_handler *block_handler;
> +		struct mali_c55_params_block_header *block;
> +
> +		block = (struct mali_c55_params_block_header *)
> +			 &config->data[block_offset];

How do you ensure config->data does hold a full struct
mali_c33_params_block_header at block_offset (i.e. that the struct does not
exceed the memory available for config->data)?

> +
> +		if (block->type >= MALI_C55_PARAM_BLOCK_SENTINEL) {
> +			dev_dbg(mali_c55->dev, "Invalid parameters block type\n");
> +			state = VB2_BUF_STATE_ERROR;
> +			goto err_buffer_done;
> +		}
> +
> +		block_handler = &mali_c55_block_handlers[block->type];
> +		if (block->size != block_handler->size) {

How do you ensure config->data has room for the block?

> +			dev_dbg(mali_c55->dev, "Invalid parameters block size\n");
> +			state = VB2_BUF_STATE_ERROR;
> +			goto err_buffer_done;
> +		}
> +
> +		block_handler->handler(mali_c55, block);
> +
> +		block_offset += block->size;
> +	}
> +
> +err_buffer_done:
> +	vb2_buffer_done(&buf->vb.vb2_buf, state);
> +}
> +
> +void mali_c55_unregister_params(struct mali_c55 *mali_c55)
> +{
> +	struct mali_c55_params *params = &mali_c55->params;
> +
> +	if (!video_is_registered(&params->vdev))
> +		return;
> +
> +	vb2_video_unregister_device(&params->vdev);
> +	media_entity_cleanup(&params->vdev.entity);
> +	mutex_destroy(&params->lock);
> +}
> +
> +int mali_c55_register_params(struct mali_c55 *mali_c55)
> +{
> +	struct mali_c55_params *params = &mali_c55->params;
> +	struct video_device *vdev = &params->vdev;
> +	struct vb2_queue *vb2q = &params->queue;
> +	int ret;
> +
> +	mutex_init(&params->lock);
> +	INIT_LIST_HEAD(&params->buffers.queue);
> +
> +	params->pad.flags = MEDIA_PAD_FL_SOURCE;
> +	ret = media_entity_pads_init(&params->vdev.entity, 1, &params->pad);
> +	if (ret)
> +		goto err_destroy_mutex;
> +
> +	vb2q->type = V4L2_BUF_TYPE_META_OUTPUT;
> +	vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
> +	vb2q->drv_priv = params;
> +	vb2q->mem_ops = &vb2_dma_contig_memops;
> +	vb2q->ops = &mali_c55_params_vb2_ops;
> +	vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
> +	vb2q->min_queued_buffers = 1;
> +	vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> +	vb2q->lock = &params->lock;
> +	vb2q->dev = mali_c55->dev;
> +
> +	ret = vb2_queue_init(vb2q);
> +	if (ret) {
> +		dev_err(mali_c55->dev, "params vb2 queue init failed\n");
> +		goto err_cleanup_entity;
> +	}
> +
> +	strscpy(params->vdev.name, "mali-c55 3a params",
> +		sizeof(params->vdev.name));
> +	vdev->release = video_device_release_empty;
> +	vdev->fops = &mali_c55_params_v4l2_fops;
> +	vdev->ioctl_ops = &mali_c55_params_v4l2_ioctl_ops;
> +	vdev->lock = &params->lock;
> +	vdev->v4l2_dev = &mali_c55->v4l2_dev;
> +	vdev->queue = &params->queue;
> +	vdev->device_caps = V4L2_CAP_META_OUTPUT | V4L2_CAP_STREAMING;
> +	vdev->vfl_dir = VFL_DIR_TX;
> +	video_set_drvdata(vdev, params);
> +
> +	ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
> +	if (ret) {
> +		dev_err(mali_c55->dev,
> +			"failed to register params video device\n");
> +		goto err_release_vb2q;
> +	}
> +
> +	params->mali_c55 = mali_c55;
> +
> +	return 0;
> +
> +err_release_vb2q:
> +	vb2_queue_release(vb2q);
> +err_cleanup_entity:
> +	media_entity_cleanup(&params->vdev.entity);
> +err_destroy_mutex:
> +	mutex_destroy(&params->lock);
> +
> +	return ret;
> +}
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> index eb3719245ec3..8e6a801077ed 100644
> --- a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> @@ -119,6 +119,19 @@
>  #define MALI_C55_REG_ACTIVE_HEIGHT_MASK			0xffff0000
>  #define MALI_C55_REG_BAYER_ORDER			0x18e8c
>  #define MALI_C55_BAYER_ORDER_MASK			GENMASK(1, 0)
> +
> +#define MALI_C55_REG_METERING_CONFIG			0x18ed0
> +#define MALI_C55_5BIN_HIST_DISABLE_MASK			BIT(0)
> +#define MALI_C55_5BIN_HIST_SWITCH_MASK			GENMASK(2, 1)
> +#define MALI_C55_AF_DISABLE_MASK			BIT(4)
> +#define MALI_C55_AF_SWITCH_MASK				BIT(5)
> +#define MALI_C55_AWB_DISABLE_MASK			BIT(8)
> +#define MALI_C55_AWB_SWITCH_MASK			BIT(9)
> +#define MALI_C55_AEXP_HIST_DISABLE_MASK			BIT(12)
> +#define MALI_C55_AEXP_HIST_SWITCH_MASK			GENMASK(14, 13)
> +#define MALI_C55_AEXP_IHIST_DISABLE_MASK		BIT(16)
> +#define MALI_C55_AEXP_SRC_MASK				BIT(24)
> +
>  #define MALI_C55_REG_TPG_CH0				0x18ed8
>  #define MALI_C55_TEST_PATTERN_ON_OFF			BIT(0)
>  #define MALI_C55_TEST_PATTERN_RGB_MASK			BIT(1)
> @@ -138,6 +151,11 @@
>  #define MALI_C55_REG_CONFIG_SPACES_OFFSET		0x0ab6c
>  #define MALI_C55_CONFIG_SPACE_SIZE			0x1231c
>  
> +#define MALI_C55_REG_DIGITAL_GAIN			0x1926c
> +#define MALI_C55_DIGITAL_GAIN_MASK			GENMASK(12, 0)
> +#define MALI_C55_REG_DIGITAL_GAIN_OFFSET		0x19270
> +#define MALI_C55_DIGITAL_GAIN_OFFSET_MASK		GENMASK(19, 0)
> +
>  #define MALI_C55_REG_SINTER_CONFIG			0x19348
>  #define MALI_C55_SINTER_VIEW_FILTER_MASK		GENMASK(1, 0)
>  #define MALI_C55_SINTER_SCALE_MODE_MASK			GENMASK(3, 2)
> @@ -146,6 +164,46 @@
>  #define MALI_C55_SINTER_INT_SELECT_MASK			BIT(6)
>  #define MALI_C55_SINTER_RM_ENABLE_MASK			BIT(7)
>  
> +/* Black Level Correction Configuration */
> +#define MALI_C55_REG_SENSOR_OFF_PRE_SHA_00		0x1abcc
> +#define MALI_C55_REG_SENSOR_OFF_PRE_SHA_01		0x1abd0
> +#define MALI_C55_REG_SENSOR_OFF_PRE_SHA_10		0x1abd4
> +#define MALI_C55_REG_SENSOR_OFF_PRE_SHA_11		0x1abd8
> +#define MALI_C55_SENSOR_OFF_PRE_SHA_MASK		0xfffff
> +
> +/* Lens Mesh Shading Configuration */
> +#define MALI_C55_REG_MESH_SHADING_TABLES		0x13074
> +#define MALI_C55_REG_MESH_SHADING_CONFIG		0x1abfc
> +#define MALI_C55_MESH_SHADING_ENABLE_MASK		BIT(0)
> +#define MALI_C55_MESH_SHADING_MESH_SHOW			BIT(1)
> +#define MALI_C55_MESH_SHADING_SCALE_MASK		GENMASK(4, 2)
> +#define MALI_C55_MESH_SHADING_PAGE_R_MASK		GENMASK(9, 8)
> +#define MALI_C55_MESH_SHADING_PAGE_G_MASK		GENMASK(11, 10)
> +#define MALI_C55_MESH_SHADING_PAGE_B_MASK		GENMASK(13, 12)
> +#define MALI_C55_MESH_SHADING_MESH_WIDTH_MASK		GENMASK(21, 16)
> +#define MALI_C55_MESH_SHADING_MESH_HEIGHT_MASK		GENMASK(29, 24)
> +
> +#define MALI_C55_REG_MESH_SHADING_ALPHA_BANK		0x1ac04
> +#define MALI_C55_MESH_SHADING_ALPHA_BANK_R_MASK		GENMASK(2, 0)
> +#define MALI_C55_MESH_SHADING_ALPHA_BANK_G_MASK		GENMASK(5, 3)
> +#define MALI_C55_MESH_SHADING_ALPHA_BANK_B_MASK		GENMASK(8, 6)
> +#define MALI_C55_REG_MESH_SHADING_ALPHA			0x1ac08
> +#define MALI_C55_MESH_SHADING_ALPHA_R_MASK		GENMASK(7, 0)
> +#define MALI_C55_MESH_SHADING_ALPHA_G_MASK		GENMASK(15, 8)
> +#define MALI_C55_MESH_SHADING_ALPHA_B_MASK		GENMASK(23, 16)
> +#define MALI_C55_REG_MESH_SHADING_MESH_STRENGTH		0x1ac0c
> +#define MALI_c55_MESH_STRENGTH_MASK			GENMASK(15, 0)
> +
> +/* AWB Gains Configuration */
> +#define MALI_C55_REG_AWB_GAINS1				0x1ac10
> +#define MALI_C55_AWB_GAIN00_MASK			GENMASK(11, 0)
> +#define MALI_C55_AWB_GAIN01_MASK			GENMASK(27, 16)
> +#define MALI_C55_REG_AWB_GAINS2				0x1ac14
> +#define MALI_C55_AWB_GAIN10_MASK			GENMASK(11, 0)
> +#define MALI_C55_AWB_GAIN11_MASK			GENMASK(27, 16)
> +#define MALI_C55_REG_AWB_GAINS1_AEXP			0x1ac18
> +#define MALI_C55_REG_AWB_GAINS2_AEXP			0x1ac1c
> +
>  /* Colour Correction Matrix Configuration */
>  #define MALI_C55_REG_CCM_ENABLE				0x1b07c
>  #define MALI_C55_CCM_ENABLE_MASK			BIT(0)
> @@ -168,6 +226,52 @@
>  #define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B		0x1b0c8
>  #define MALI_C55_CCM_ANTIFOG_OFFSET_MASK		GENMASK(11, 0)
>  
> +/* AWB Statistics Configuration */
> +#define MALI_C55_REG_AWB_STATS_MODE			0x1b29c
> +#define MALI_C55_AWB_STATS_MODE_MASK			BIT(0)
> +#define MALI_C55_REG_AWB_WHITE_LEVEL			0x1b2a0
> +#define MALI_C55_AWB_WHITE_LEVEL_MASK			GENMASK(9, 0)
> +#define MALI_C55_REG_AWB_BLACK_LEVEL			0x1b2a4
> +#define MALI_C55_AWB_BLACK_LEVEL_MASK			GENMASK(9, 0)
> +#define MALI_C55_REG_AWB_CR_MAX				0x1b2a8
> +#define MALI_C55_AWB_CR_MAX_MASK			GENMASK(11, 0)
> +#define MALI_C55_REG_AWB_CR_MIN				0x1b2ac
> +#define MALI_C55_AWB_CR_MIN_MASK			GENMASK(11, 0)
> +#define MALI_C55_REG_AWB_CB_MAX				0x1b2b0
> +#define MALI_C55_REG_AWB_CB_MIN				0x1b2b4
> +#define MALI_C55_AWB_CB_MIN_MASK			GENMASK(11, 0)
> +#define MALI_C55_REG_AWB_NODES_USED			0x1b2c4
> +#define MALI_C55_AWB_NODES_USED_HORIZ_MASK		GENMASK(7, 0)
> +#define MALI_C55_AWB_NODES_USED_VERT_MASK		GENMASK(15, 8)
> +#define MALI_C55_REG_AWB_CR_HIGH			0x1b2c8
> +#define MALI_C55_AWB_CR_HIGH_MASK			GENMASK(11, 0)
> +#define MALI_C55_REG_AWB_CR_LOW				0x1b2cc
> +#define MALI_C55_AWB_CR_LOW_MASK			GENMASK(11, 0)
> +#define MALI_C55_REG_AWB_CB_HIGH			0x1b2d0
> +#define MALI_C55_AWB_CB_HIGH_MASK			GENMASK(11, 0)
> +#define MALI_C55_REG_AWB_CB_LOW				0x1b2d4
> +#define MALI_C55_AWB_CB_LOW_MASK			GENMASK(11, 0)
> +
> +/* AEXP Metering Histogram Configuration */
> +#define MALI_C55_REG_AEXP_HIST_BASE			0x1b730
> +#define MALI_C55_REG_AEXP_IHIST_BASE			0x1bbac
> +#define MALI_C55_AEXP_HIST_SKIP_OFFSET			0
> +#define MALI_C55_AEXP_HIST_SKIP_X_MASK			GENMASK(2, 0)
> +#define MALI_C55_AEXP_HIST_OFFSET_X_MASK		BIT(3)
> +#define MALI_C55_AEXP_HIST_SKIP_Y_MASK			GENMASK(6, 4)
> +#define MALI_C55_AEXP_HIST_OFFSET_Y_MASK		BIT(7)
> +#define MALI_C55_AEXP_HIST_SCALE_OFFSET			4
> +#define MALI_C55_AEXP_HIST_SCALE_BOTTOM_MASK		GENMASK(3, 0)
> +#define MALI_C55_AEXP_HIST_SCALE_TOP_MASK		GENMASK(7, 4)
> +#define MALI_C55_AEXP_HIST_PLANE_MODE_OFFSET		16
> +#define MALI_C55_AEXP_HIST_PLANE_MODE_MASK		GENMASK(2, 0)
> +#define MALI_C55_AEXP_HIST_NODES_USED_OFFSET		52
> +#define MALI_C55_AEXP_HIST_NODES_USED_HORIZ_MASK	GENMASK(7, 0)
> +#define MALI_C55_AEXP_HIST_NODES_USED_VERT_MASK		GENMASK(15, 8)
> +#define MALI_C55_AEXP_HIST_ZONE_WEIGHTS_OFFSET		56
> +#define MALI_C55_AEXP_HIST_ZONE_WEIGHT_MASK		0x0f0f0f0f
> +
>  /*
>   * The Mali-C55 ISP has up to two output pipes; known as full resolution and
>   * down scaled. The register space for these is laid out identically, but offset

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v5 15/16] media: platform: Add mali-c55 parameters video node

[Dan Scally]

Hi Sakari - thanks for the comments

On 14/06/2024 19:53, Sakari Ailus wrote:
> Hi Jacopo, Dan,
>
> Thanks for the patch. Please see my comments below.
>
> On Wed, May 29, 2024 at 04:28:57PM +0100, Daniel Scally wrote:
>> Add a new code file to the mali-c55 driver that registers an output
>> video node for userspace to queue buffers of parameters to. Handlers
>> are included to program the statistics generation plus the white
>> balance, black level correction and mesh shading correction blocks.
>>
>> Update the rest of the driver to register and link the new video node
>>
>> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>> ---
>> Changes in v5:
>>
>> 	- New patch
>>
>>   drivers/media/platform/arm/mali-c55/Makefile  |   1 +
>>   .../platform/arm/mali-c55/mali-c55-common.h   |  18 +
>>   .../platform/arm/mali-c55/mali-c55-core.c     |  24 +
>>   .../platform/arm/mali-c55/mali-c55-isp.c      |  16 +-
>>   .../platform/arm/mali-c55/mali-c55-params.c   | 615 ++++++++++++++++++
>>   .../arm/mali-c55/mali-c55-registers.h         | 104 +++
>>   6 files changed, 777 insertions(+), 1 deletion(-)
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-params.c
>>
>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
>> index cd5a64bf0c62..b2443f2d416a 100644
>> --- a/drivers/media/platform/arm/mali-c55/Makefile
>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
>> @@ -5,6 +5,7 @@ mali-c55-y := mali-c55-capture.o \
>>   	      mali-c55-isp.o \
>>   	      mali-c55-tpg.o \
>>   	      mali-c55-resizer.o \
>> +	      mali-c55-params.o \
>>   	      mali-c55-stats.o
>>   
>>   obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>> index 44119e04009b..565d98acfcdd 100644
>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>> @@ -80,6 +80,7 @@ enum mali_c55_isp_pads {
>>   	MALI_C55_ISP_PAD_SOURCE,
>>   	MALI_C55_ISP_PAD_SOURCE_BYPASS,
>>   	MALI_C55_ISP_PAD_SOURCE_3A,
>> +	MALI_C55_ISP_PAD_SINK_PARAMS,
>>   	MALI_C55_ISP_NUM_PADS,
>>   };
>>   
>> @@ -217,6 +218,19 @@ struct mali_c55_stats {
>>   	} buffers;
>>   };
>>   
>> +struct mali_c55_params {
>> +	struct mali_c55 *mali_c55;
>> +	struct video_device vdev;
>> +	struct vb2_queue queue;
>> +	struct media_pad pad;
>> +	struct mutex lock;
>> +
>> +	struct {
>> +		spinlock_t lock;
>> +		struct list_head queue;
>> +	} buffers;
>> +};
>> +
>>   enum mali_c55_config_spaces {
>>   	MALI_C55_CONFIG_PING,
>>   	MALI_C55_CONFIG_PONG,
>> @@ -247,6 +261,7 @@ struct mali_c55 {
>>   	struct mali_c55_isp isp;
>>   	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
>>   	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
>> +	struct mali_c55_params params;
>>   	struct mali_c55_stats stats;
>>   
>>   	struct list_head contexts;
>> @@ -271,6 +286,8 @@ int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
>>   void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
>>   int mali_c55_register_stats(struct mali_c55 *mali_c55);
>>   void mali_c55_unregister_stats(struct mali_c55 *mali_c55);
>> +int mali_c55_register_params(struct mali_c55 *mali_c55);
>> +void mali_c55_unregister_params(struct mali_c55 *mali_c55);
>>   struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
>>   void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>>   			     enum mali_c55_planes plane);
>> @@ -290,5 +307,6 @@ bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
>>   	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
>>   void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
>>   				enum mali_c55_config_spaces cfg_space);
>> +void mali_c55_params_write_config(struct mali_c55 *mali_c55);
>>   
>>   #endif /* _MALI_C55_COMMON_H */
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>> index 2cf8b1169604..6acee3edd03f 100644
>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>> @@ -347,6 +347,17 @@ static int mali_c55_create_links(struct mali_c55 *mali_c55)
>>   		goto err_remove_links;
>>   	}
>>   
>> +	ret = media_create_pad_link(&mali_c55->params.vdev.entity, 0,
>> +				    &mali_c55->isp.sd.entity,
>> +				    MALI_C55_ISP_PAD_SINK_PARAMS,
>> +				    MEDIA_LNK_FL_ENABLED |
>> +				    MEDIA_LNK_FL_IMMUTABLE);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev,
>> +			"failed to link ISP and parameters video node\n");
>> +		goto err_remove_links;
>> +	}
>> +
>>   	return 0;
>>   
>>   err_remove_links:
>> @@ -360,6 +371,7 @@ static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
>>   	mali_c55_unregister_isp(mali_c55);
>>   	mali_c55_unregister_resizers(mali_c55);
>>   	mali_c55_unregister_capture_devs(mali_c55);
>> +	mali_c55_unregister_params(mali_c55);
>>   	mali_c55_unregister_stats(mali_c55);
>>   }
>>   
>> @@ -383,6 +395,10 @@ static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>>   	if (ret)
>>   		goto err_unregister_entities;
>>   
>> +	ret = mali_c55_register_params(mali_c55);
>> +	if (ret)
>> +		goto err_unregister_entities;
>> +
>>   	ret = mali_c55_register_stats(mali_c55);
>>   	if (ret)
>>   		goto err_unregister_entities;
>> @@ -474,6 +490,14 @@ static irqreturn_t mali_c55_isr(int irq, void *context)
>>   			curr_config >>= ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1;
>>   			next_config = curr_config ^ 1;
>>   
>> +			/*
>> +			 * Write the configuration parameters received from
>> +			 * userspace into the configuration buffer, which will
>> +			 * be transferred to the 'next' active config space at
>> +			 * by mali_c55_swap_next_config().
>> +			 */
>> +			mali_c55_params_write_config(mali_c55);
>> +
>>   			/*
>>   			 * The ordering of these two is currently important as
>>   			 * mali_c55_stats_fill_buffer() is asynchronous whereas
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>> index 94876fba3353..8c2b45bfd82d 100644
>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>> @@ -146,6 +146,7 @@ static int mali_c55_isp_start(struct mali_c55 *mali_c55)
>>   			     cfg->encoding == V4L2_PIXEL_ENC_RGB ?
>>   			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
>>   
>> +	mali_c55_params_write_config(mali_c55);
>>   	ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
>>   	if (ret) {
>>   		dev_err(mali_c55->dev, "failed to DMA config\n");
>> @@ -455,8 +456,20 @@ static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
>>   	.init_state = mali_c55_isp_init_state,
>>   };
>>   
>> +static int mali_c55_subdev_link_validate(struct media_link *link)
>> +{
>> +	/*
>> +	 * Skip validation for the parameters sink pad, as the source is not
>> +	 * a subdevice.
>> +	 */
>> +	if (link->sink->index == MALI_C55_ISP_PAD_SINK_PARAMS)
>> +		return 0;
>> +
>> +	return v4l2_subdev_link_validate(link);
>> +}
>> +
>>   static const struct media_entity_operations mali_c55_isp_media_ops = {
>> -	.link_validate		= v4l2_subdev_link_validate,
>> +	.link_validate		= mali_c55_subdev_link_validate,
>>   };
>>   
>>   static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
>> @@ -565,6 +578,7 @@ int mali_c55_register_isp(struct mali_c55 *mali_c55)
>>   	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>>   	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
>>   	isp->pads[MALI_C55_ISP_PAD_SOURCE_3A].flags = MEDIA_PAD_FL_SOURCE;
>> +	isp->pads[MALI_C55_ISP_PAD_SINK_PARAMS].flags = MEDIA_PAD_FL_SINK;
>>   
>>   	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
>>   				     isp->pads);
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-params.c b/drivers/media/platform/arm/mali-c55/mali-c55-params.c
>> new file mode 100644
>> index 000000000000..049a7b8e4861
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-params.c
>> @@ -0,0 +1,615 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ARM Mali-C55 ISP Driver - Configuration parameters output device
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +#include <linux/media/arm/mali-c55-config.h>
>> +
>> +#include <media/media-entity.h>
>> +#include <media/v4l2-dev.h>
>> +#include <media/v4l2-event.h>
>> +#include <media/v4l2-fh.h>
>> +#include <media/v4l2-ioctl.h>
>> +#include <media/videobuf2-core.h>
>> +#include <media/videobuf2-dma-contig.h>
>> +
>> +#include "mali-c55-common.h"
>> +#include "mali-c55-registers.h"
>> +
>> +typedef void (*mali_c55_block_handler)(struct mali_c55 *mali_c55,
> You can wrap after the return type (including typedef). Same elsewhere.
>
>> +				       struct mali_c55_params_block_header *block);
>> +
>> +struct mali_c55_block_handler {
>> +	size_t size;
>> +	mali_c55_block_handler handler;
>> +};
>> +
>> +static void mali_c55_params_sensor_offs(struct mali_c55 *mali_c55,
>> +					struct mali_c55_params_block_header *block)
>> +{
>> +	struct mali_c55_params_sensor_off_preshading *p =
>> +		(struct mali_c55_params_sensor_off_preshading *)block;
> I wonder if an union could be used to make this a bit cleaner. You're doing
> a lot of casting that I think could be avoided.


Interesting idea, thank you - let me give that a try and see how it looks

>
>> +	__u32 global_offset;
>> +
>> +	if (!block->enabled)
>> +		return;
>> +
>> +	if (!(p->chan00 || p->chan01 || p->chan10 || p->chan11))
>> +		return;
>> +
>> +	mali_c55_write(mali_c55, MALI_C55_REG_SENSOR_OFF_PRE_SHA_00,
>> +		       p->chan00 & MALI_C55_SENSOR_OFF_PRE_SHA_MASK);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_SENSOR_OFF_PRE_SHA_01,
>> +		       p->chan01 & MALI_C55_SENSOR_OFF_PRE_SHA_MASK);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_SENSOR_OFF_PRE_SHA_10,
>> +		       p->chan10 & MALI_C55_SENSOR_OFF_PRE_SHA_MASK);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_SENSOR_OFF_PRE_SHA_11,
>> +		       p->chan11 & MALI_C55_SENSOR_OFF_PRE_SHA_MASK);
>> +
>> +	/*
>> +	 * The average offset is applied as a global offset for the digital
>> +	 * gain block
>> +	 */
>> +	global_offset = (p->chan00 + p->chan01 + p->chan10 + p->chan11) >> 2;
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_DIGITAL_GAIN_OFFSET,
>> +			     MALI_C55_DIGITAL_GAIN_OFFSET_MASK, global_offset);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
>> +			     MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH, 0x00);
>> +}
>> +
>> +static void mali_c55_params_aexp_hist(struct mali_c55 *mali_c55,
>> +				struct mali_c55_params_block_header *block)
>> +{
>> +	u32 disable_mask = block->type == MALI_C55_PARAM_BLOCK_AEXP_HIST ?
>> +					  MALI_C55_AEXP_HIST_DISABLE_MASK :
>> +					  MALI_C55_AEXP_IHIST_DISABLE_MASK;
>> +	u32 base = block->type == MALI_C55_PARAM_BLOCK_AEXP_HIST ?
>> +				  MALI_C55_REG_AEXP_HIST_BASE :
>> +				  MALI_C55_REG_AEXP_IHIST_BASE;
>> +	struct mali_c55_params_aexp_hist *params =
>> +		(struct mali_c55_params_aexp_hist *)block;
>> +
>> +	if (!block->enabled) {
>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
>> +				     disable_mask, true);
>> +		return;
>> +	}
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
>> +			     disable_mask, false);
>> +
>> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SKIP_OFFSET,
>> +			     MALI_C55_AEXP_HIST_SKIP_X_MASK, params->skip_x);
>> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SKIP_OFFSET,
>> +			     MALI_C55_AEXP_HIST_OFFSET_X_MASK, params->offset_x);
>> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SKIP_OFFSET,
>> +			     MALI_C55_AEXP_HIST_SKIP_Y_MASK, params->skip_y);
>> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SKIP_OFFSET,
>> +			     MALI_C55_AEXP_HIST_OFFSET_Y_MASK, params->offset_y);
>> +
>> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SCALE_OFFSET,
>> +			     MALI_C55_AEXP_HIST_SCALE_BOTTOM_MASK, params->scale_bottom);
>> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_SCALE_OFFSET,
>> +			     MALI_C55_AEXP_HIST_SCALE_TOP_MASK, params->scale_top);
>> +
>> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_PLANE_MODE_OFFSET,
>> +			     MALI_C55_AEXP_HIST_PLANE_MODE_MASK, params->plane_mode);
>> +
>> +	if (block->type == MALI_C55_PARAM_BLOCK_AEXP_HIST)
>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
>> +				     MALI_C55_AEXP_HIST_SWITCH_MASK,
>> +				     params->tap_point);
>> +}
>> +
>> +static void
>> +mali_c55_params_aexp_hist_weights(struct mali_c55 *mali_c55,
>> +				  struct mali_c55_params_block_header *block)
>> +{
>> +	struct mali_c55_params_aexp_weights *params =
>> +		(struct mali_c55_params_aexp_weights *)block;
>> +	u32 base;
>> +
>> +	if (!block->enabled)
>> +		return;
>> +
>> +	base = block->type == MALI_C55_PARAM_BLOCK_AEXP_HIST_WEIGHTS ?
>> +			      MALI_C55_REG_AEXP_HIST_BASE :
>> +			      MALI_C55_REG_AEXP_IHIST_BASE;
>> +
>> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_NODES_USED_OFFSET,
>> +			     MALI_C55_AEXP_HIST_NODES_USED_HORIZ_MASK, params->nodes_used_horiz);
>> +	mali_c55_update_bits(mali_c55, base + MALI_C55_AEXP_HIST_NODES_USED_OFFSET,
>> +			     MALI_C55_AEXP_HIST_NODES_USED_VERT_MASK, params->nodes_used_vert);
>> +
>> +	/*
>> +	 * The zone weights array is a 225-element array of u8 values, but that
>> +	 * is a bit annoying to handle given the ISP expects 32-bit writes. We
>> +	 * just reinterpret it as a 57-element array of 32-bit values for the
>> +	 * purposes of this transaction (the 3 bytes of additional space at the
>> +	 * end of the write is just padding for the array of weights in the ISP
>> +	 * memory space anyway, so there's no risk of overwriting other
>> +	 * registers).
>> +	 */
>> +	for (unsigned int i = 0; i < 57; i++) {
>> +		u32 val = ((u32 *)params->zone_weights)[i]
>> +			    & MALI_C55_AEXP_HIST_ZONE_WEIGHT_MASK;
>> +		u32 addr = base + MALI_C55_AEXP_HIST_ZONE_WEIGHTS_OFFSET + (4 * i);
>> +
>> +		mali_c55_write(mali_c55, addr, val);
>> +	}
>> +}
>> +
>> +static void mali_c55_params_digital_gain(struct mali_c55 *mali_c55,
>> +					 struct mali_c55_params_block_header *block)
>> +{
>> +	struct mali_c55_params_digital_gain *dgain =
>> +		(struct mali_c55_params_digital_gain *)block;
>> +
>> +	/*
>> +	 * If the block is flagged as disabled we write a gain of 1.0, which in
>> +	 * Q5.8 format is 256.
>> +	 */
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_DIGITAL_GAIN,
>> +			     MALI_C55_DIGITAL_GAIN_MASK,
>> +			     block->enabled ? dgain->gain : 256);
>> +}
>> +
>> +static void mali_c55_params_awb_gains(struct mali_c55 *mali_c55,
>> +				      struct mali_c55_params_block_header *block)
>> +{
>> +	struct mali_c55_params_awb_gains *gains =
>> +		(struct mali_c55_params_awb_gains *)block;
>> +
>> +	/*
>> +	 * There are two places AWB gains can be set in the ISP; one affects the
>> +	 * image output data and the other affects the statistics for the
>> +	 * AEXP-0 tap point.
>> +	 */
>> +	u32 addr1 = block->type = MALI_C55_PARAM_BLOCK_AWB_GAINS ?
>> +				  MALI_C55_REG_AWB_GAINS1 :
>> +				  MALI_C55_REG_AWB_GAINS1_AEXP;
>> +	u32 addr2 = block->type = MALI_C55_PARAM_BLOCK_AWB_GAINS ?
>> +				  MALI_C55_REG_AWB_GAINS2 :
>> +				  MALI_C55_REG_AWB_GAINS2_AEXP;
>> +
>> +	mali_c55_update_bits(mali_c55, addr1, MALI_C55_AWB_GAIN00_MASK,
>> +			     gains->gain00);
>> +	mali_c55_update_bits(mali_c55, addr1, MALI_C55_AWB_GAIN01_MASK,
>> +			     gains->gain01);
>> +	mali_c55_update_bits(mali_c55, addr2, MALI_C55_AWB_GAIN10_MASK,
>> +			     gains->gain10);
>> +	mali_c55_update_bits(mali_c55, addr2, MALI_C55_AWB_GAIN11_MASK,
>> +			     gains->gain11);
>> +}
>> +
>> +static void mali_c55_params_awb_config(struct mali_c55 *mali_c55,
>> +				      struct mali_c55_params_block_header *block)
>> +{
>> +	struct mali_c55_params_awb_config *params =
>> +		(struct mali_c55_params_awb_config *)block;
>> +
>> +	if (!block->enabled) {
>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
>> +				     MALI_C55_AWB_DISABLE_MASK, true);
>> +		return;
>> +	}
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
>> +			     MALI_C55_AWB_DISABLE_MASK, false);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_STATS_MODE,
>> +			     MALI_C55_AWB_STATS_MODE_MASK, params->stats_mode);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_WHITE_LEVEL,
>> +			     MALI_C55_AWB_WHITE_LEVEL_MASK, params->white_level);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_BLACK_LEVEL,
>> +			     MALI_C55_AWB_BLACK_LEVEL_MASK, params->black_level);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CR_MAX,
>> +			     MALI_C55_AWB_CR_MAX_MASK, params->cr_max);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CR_MIN,
>> +			     MALI_C55_AWB_CR_MIN_MASK, params->cr_min);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CB_MAX,
>> +			     MALI_C55_AWB_CB_MAX_MASK, params->cb_max);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CB_MIN,
>> +			     MALI_C55_AWB_CB_MIN_MASK, params->cb_min);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_NODES_USED,
>> +			     MALI_C55_AWB_NODES_USED_HORIZ_MASK,
>> +			     params->nodes_used_horiz);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_NODES_USED,
>> +			     MALI_C55_AWB_NODES_USED_VERT_MASK,
>> +			     params->nodes_used_vert);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CR_HIGH,
>> +			     MALI_C55_AWB_CR_HIGH_MASK, params->cr_high);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CR_LOW,
>> +			     MALI_C55_AWB_CR_LOW_MASK, params->cr_low);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CB_HIGH,
>> +			     MALI_C55_AWB_CB_HIGH_MASK, params->cb_high);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_AWB_CB_LOW,
>> +			     MALI_C55_AWB_CB_LOW_MASK, params->cb_low);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_METERING_CONFIG,
>> +			     MALI_C55_AWB_SWITCH_MASK, params->tap_point);
>> +}
>> +
>> +static void mali_c55_params_lsc_config(struct mali_c55 *mali_c55,
>> +				       struct mali_c55_params_block_header *block)
>> +{
>> +	struct mali_c55_params_mesh_shading_config *params =
>> +		(struct mali_c55_params_mesh_shading_config *)block;
>> +	unsigned int i;
>> +	u32 addr;
>> +
>> +	if (!block->enabled) {
>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
>> +				     MALI_C55_MESH_SHADING_ENABLE_MASK, false);
>> +		return;
>> +	}
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
>> +			     MALI_C55_MESH_SHADING_ENABLE_MASK, true);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
>> +			     MALI_C55_MESH_SHADING_MESH_SHOW, params->mesh_show);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
>> +			     MALI_C55_MESH_SHADING_SCALE_MASK,
>> +			     params->mesh_scale);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
>> +			     MALI_C55_MESH_SHADING_PAGE_R_MASK,
>> +			     params->mesh_page_r);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
>> +			     MALI_C55_MESH_SHADING_PAGE_G_MASK,
>> +			     params->mesh_page_g);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
>> +			     MALI_C55_MESH_SHADING_PAGE_B_MASK,
>> +			     params->mesh_page_b);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
>> +			     MALI_C55_MESH_SHADING_MESH_WIDTH_MASK,
>> +			     params->mesh_width);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_CONFIG,
>> +			     MALI_C55_MESH_SHADING_MESH_HEIGHT_MASK,
>> +			     params->mesh_height);
>> +
>> +	for (i = 0; i < MALI_C55_NUM_MESH_SHADING_ELEMENTS; i++) {
>> +		addr = MALI_C55_REG_MESH_SHADING_TABLES + (i * 4);
>> +		mali_c55_write(mali_c55, addr, params->mesh[i]);
>> +	}
>> +}
>> +
>> +static void mali_c55_params_lsc_selection(struct mali_c55 *mali_c55,
>> +					  struct mali_c55_params_block_header *block)
>> +{
>> +	struct mali_c55_params_mesh_shading_selection *params =
>> +		(struct mali_c55_params_mesh_shading_selection *)block;
>> +
>> +	if (!block->enabled)
>> +		return;
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA_BANK,
>> +			     MALI_C55_MESH_SHADING_ALPHA_BANK_R_MASK,
>> +			     params->mesh_alpha_bank_r);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA_BANK,
>> +			     MALI_C55_MESH_SHADING_ALPHA_BANK_G_MASK,
>> +			     params->mesh_alpha_bank_g);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA_BANK,
>> +			     MALI_C55_MESH_SHADING_ALPHA_BANK_B_MASK,
>> +			     params->mesh_alpha_bank_b);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA,
>> +			     MALI_C55_MESH_SHADING_ALPHA_R_MASK,
>> +			     params->mesh_alpha_r);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA,
>> +			     MALI_C55_MESH_SHADING_ALPHA_G_MASK,
>> +			     params->mesh_alpha_g);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_ALPHA,
>> +			     MALI_C55_MESH_SHADING_ALPHA_B_MASK,
>> +			     params->mesh_alpha_b);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MESH_SHADING_MESH_STRENGTH,
>> +			     MALI_c55_MESH_STRENGTH_MASK,
>> +			     params->mesh_strength);
>> +}
>> +
>> +static const struct mali_c55_block_handler mali_c55_block_handlers[] = {
>> +	[MALI_C55_PARAM_BLOCK_SENSOR_OFFS] = {
>> +		.size = sizeof(struct mali_c55_params_sensor_off_preshading),
>> +		.handler = &mali_c55_params_sensor_offs,
>> +	},
>> +	[MALI_C55_PARAM_BLOCK_AEXP_HIST] = {
>> +		.size = sizeof(struct mali_c55_params_aexp_hist),
>> +		.handler = &mali_c55_params_aexp_hist,
>> +	},
>> +	[MALI_C55_PARAM_BLOCK_AEXP_IHIST] = {
>> +		.size = sizeof(struct mali_c55_params_aexp_hist),
>> +		.handler = &mali_c55_params_aexp_hist,
>> +	},
>> +	[MALI_C55_PARAM_BLOCK_AEXP_HIST_WEIGHTS] = {
>> +		.size = sizeof(struct mali_c55_params_aexp_weights),
>> +		.handler = &mali_c55_params_aexp_hist_weights,
>> +	},
>> +	[MALI_C55_PARAM_BLOCK_AEXP_IHIST_WEIGHTS] = {
>> +		.size = sizeof(struct mali_c55_params_aexp_weights),
>> +		.handler = &mali_c55_params_aexp_hist_weights,
>> +	},
>> +	[MALI_C55_PARAM_BLOCK_DIGITAL_GAIN] = {
>> +		.size = sizeof(struct mali_c55_params_digital_gain),
>> +		.handler = &mali_c55_params_digital_gain,
>> +	},
>> +	[MALI_C55_PARAM_BLOCK_AWB_GAINS] = {
>> +		.size = sizeof(struct mali_c55_params_awb_gains),
>> +		.handler = &mali_c55_params_awb_gains,
>> +	},
>> +	[MALI_C55_PARAM_BLOCK_AWB_CONFIG] = {
>> +		.size = sizeof(struct mali_c55_params_awb_config),
>> +		.handler = &mali_c55_params_awb_config,
>> +	},
>> +	[MALI_C55_PARAM_BLOCK_AWB_GAINS_AEXP] = {
>> +		.size = sizeof(struct mali_c55_params_awb_gains),
>> +		.handler = &mali_c55_params_awb_gains,
>> +	},
>> +	[MALI_C55_PARAM_MESH_SHADING_CONFIG] = {
>> +		.size = sizeof(struct mali_c55_params_mesh_shading_config),
>> +		.handler = &mali_c55_params_lsc_config,
>> +	},
>> +	[MALI_C55_PARAM_MESH_SHADING_SELECTION] = {
>> +		.size = sizeof(struct mali_c55_params_mesh_shading_selection),
>> +		.handler = &mali_c55_params_lsc_selection,
>> +	},
>> +};
>> +
>> +static int mali_c55_params_enum_fmt_meta_out(struct file *file, void *fh,
>> +					    struct v4l2_fmtdesc *f)
>> +{
>> +	if (f->index || f->type != V4L2_BUF_TYPE_META_OUTPUT)
> The buffer type check has been done by the caller already.
>
>> +		return -EINVAL;
>> +
>> +	f->pixelformat = V4L2_META_FMT_MALI_C55_PARAMS;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_params_g_fmt_meta_out(struct file *file, void *fh,
>> +					 struct v4l2_format *f)
>> +{
>> +	static const struct v4l2_meta_format mfmt = {
>> +		.dataformat = V4L2_META_FMT_MALI_C55_PARAMS,
>> +		.buffersize = sizeof(struct mali_c55_params_buffer),
>> +	};
>> +
>> +	if (f->type != V4L2_BUF_TYPE_META_OUTPUT)
>> +		return -EINVAL;
> Ditto.
>
> Maybe check the other instances of format access functions in the driver,
> too?

Ack
>
>> +
>> +	f->fmt.meta = mfmt;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_params_querycap(struct file *file,
>> +				   void *priv, struct v4l2_capability *cap)
>> +{
>> +	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
>> +	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_ioctl_ops mali_c55_params_v4l2_ioctl_ops = {
>> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
>> +	.vidioc_querybuf = vb2_ioctl_querybuf,
>> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
>> +	.vidioc_qbuf = vb2_ioctl_qbuf,
>> +	.vidioc_expbuf = vb2_ioctl_expbuf,
>> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
>> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
>> +	.vidioc_streamon = vb2_ioctl_streamon,
>> +	.vidioc_streamoff = vb2_ioctl_streamoff,
>> +	.vidioc_enum_fmt_meta_out = mali_c55_params_enum_fmt_meta_out,
>> +	.vidioc_g_fmt_meta_out = mali_c55_params_g_fmt_meta_out,
>> +	.vidioc_s_fmt_meta_out = mali_c55_params_g_fmt_meta_out,
>> +	.vidioc_try_fmt_meta_out = mali_c55_params_g_fmt_meta_out,
>> +	.vidioc_querycap = mali_c55_params_querycap,
>> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
>> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
>> +};
>> +
>> +static const struct v4l2_file_operations mali_c55_params_v4l2_fops = {
>> +	.owner = THIS_MODULE,
>> +	.unlocked_ioctl = video_ioctl2,
>> +	.open = v4l2_fh_open,
>> +	.release = vb2_fop_release,
>> +	.poll = vb2_fop_poll,
>> +	.mmap = vb2_fop_mmap,
>> +};
>> +
>> +static int
>> +mali_c55_params_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
>> +			   unsigned int *num_planes, unsigned int sizes[],
>> +			   struct device *alloc_devs[])
>> +{
>> +	if (*num_planes && *num_planes > 1)
>> +		return -EINVAL;
>> +
>> +	if (sizes[0] && sizes[0] != sizeof(struct mali_c55_params_buffer))
>> +		return -EINVAL;
>> +
>> +	*num_planes = 1;
>> +	sizes[0] = sizeof(struct mali_c55_params_buffer);
>> +
>> +	return 0;
>> +}
>> +
>> +static void mali_c55_params_buf_queue(struct vb2_buffer *vb)
>> +{
>> +	struct mali_c55_params *params = vb2_get_drv_priv(vb->vb2_queue);
>> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>> +	struct mali_c55_buffer *buf = container_of(vbuf,
>> +						   struct mali_c55_buffer, vb);
>> +
>> +	vb2_set_plane_payload(vb, 0, sizeof(struct mali_c55_params_buffer));
>> +
>> +	spin_lock(&params->buffers.lock);
>> +	list_add_tail(&buf->queue, &params->buffers.queue);
>> +	spin_unlock(&params->buffers.lock);
>> +}
>> +
>> +static void mali_c55_params_stop_streaming(struct vb2_queue *q)
>> +{
>> +	struct mali_c55_params *params = vb2_get_drv_priv(q);
>> +	struct mali_c55_buffer *buf, *tmp;
>> +
>> +	spin_lock(&params->buffers.lock);
>> +
>> +	list_for_each_entry_safe(buf, tmp, &params->buffers.queue, queue) {
>> +		list_del(&buf->queue);
>> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
>> +	}
>> +
>> +	spin_unlock(&params->buffers.lock);
>> +}
>> +
>> +static const struct vb2_ops mali_c55_params_vb2_ops = {
>> +	.queue_setup = mali_c55_params_queue_setup,
>> +	.buf_queue = mali_c55_params_buf_queue,
>> +	.wait_prepare = vb2_ops_wait_prepare,
>> +	.wait_finish = vb2_ops_wait_finish,
>> +	.stop_streaming = mali_c55_params_stop_streaming,
>> +};
>> +
>> +void mali_c55_params_write_config(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_params *params = &mali_c55->params;
>> +	enum vb2_buffer_state state = VB2_BUF_STATE_DONE;
>> +	struct mali_c55_params_buffer *config;
>> +	struct mali_c55_buffer *buf;
>> +	size_t block_offset = 0;
>> +
>> +	spin_lock(&params->buffers.lock);
>> +
>> +	buf = list_first_entry_or_null(&params->buffers.queue,
>> +				       struct mali_c55_buffer, queue);
>> +	if (buf)
>> +		list_del(&buf->queue);
>> +	spin_unlock(&params->buffers.lock);
>> +
>> +	if (!buf)
>> +		return;
>> +
>> +	buf->vb.sequence = mali_c55->isp.frame_sequence;
>> +	config = vb2_plane_vaddr(&buf->vb.vb2_buf, 0);
>> +
>> +	if (config->total_size > MALI_C55_PARAMS_MAX_SIZE) {
>> +		dev_dbg(mali_c55->dev, "Invalid parameters buffer size %lu\n",
>> +			config->total_size);
>> +		state = VB2_BUF_STATE_ERROR;
>> +		goto err_buffer_done;
>> +	}
>> +
>> +	/* Walk the list of parameter blocks and process them. */
>> +	while (block_offset < config->total_size) {
>> +		const struct mali_c55_block_handler *block_handler;
>> +		struct mali_c55_params_block_header *block;
>> +
>> +		block = (struct mali_c55_params_block_header *)
>> +			 &config->data[block_offset];
> How do you ensure config->data does hold a full struct
> mali_c33_params_block_header at block_offset (i.e. that the struct does not
> exceed the memory available for config->data)?


We don't currently...the data buffer is sized specifically to be large enough to accept a single 
instance of each possible struct that could be included, we could keep track of the blocks that we 
have seen already and ensure that none are seen twice...and that should guarantee that the remaining 
space is sufficient to hold whatever the last block is. Does that sound ok?

>
>> +
>> +		if (block->type >= MALI_C55_PARAM_BLOCK_SENTINEL) {
>> +			dev_dbg(mali_c55->dev, "Invalid parameters block type\n");
>> +			state = VB2_BUF_STATE_ERROR;
>> +			goto err_buffer_done;
>> +		}
>> +
>> +		block_handler = &mali_c55_block_handlers[block->type];
>> +		if (block->size != block_handler->size) {
> How do you ensure config->data has room for the block?
I think through the same proposal as above.
>
>> +			dev_dbg(mali_c55->dev, "Invalid parameters block size\n");
>> +			state = VB2_BUF_STATE_ERROR;
>> +			goto err_buffer_done;
>> +		}
>> +
>> +		block_handler->handler(mali_c55, block);
>> +
>> +		block_offset += block->size;
>> +	}
>> +
>> +err_buffer_done:
>> +	vb2_buffer_done(&buf->vb.vb2_buf, state);
>> +}
>> +
>> +void mali_c55_unregister_params(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_params *params = &mali_c55->params;
>> +
>> +	if (!video_is_registered(&params->vdev))
>> +		return;
>> +
>> +	vb2_video_unregister_device(&params->vdev);
>> +	media_entity_cleanup(&params->vdev.entity);
>> +	mutex_destroy(&params->lock);
>> +}
>> +
>> +int mali_c55_register_params(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_params *params = &mali_c55->params;
>> +	struct video_device *vdev = &params->vdev;
>> +	struct vb2_queue *vb2q = &params->queue;
>> +	int ret;
>> +
>> +	mutex_init(&params->lock);
>> +	INIT_LIST_HEAD(&params->buffers.queue);
>> +
>> +	params->pad.flags = MEDIA_PAD_FL_SOURCE;
>> +	ret = media_entity_pads_init(&params->vdev.entity, 1, &params->pad);
>> +	if (ret)
>> +		goto err_destroy_mutex;
>> +
>> +	vb2q->type = V4L2_BUF_TYPE_META_OUTPUT;
>> +	vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
>> +	vb2q->drv_priv = params;
>> +	vb2q->mem_ops = &vb2_dma_contig_memops;
>> +	vb2q->ops = &mali_c55_params_vb2_ops;
>> +	vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
>> +	vb2q->min_queued_buffers = 1;
>> +	vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
>> +	vb2q->lock = &params->lock;
>> +	vb2q->dev = mali_c55->dev;
>> +
>> +	ret = vb2_queue_init(vb2q);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "params vb2 queue init failed\n");
>> +		goto err_cleanup_entity;
>> +	}
>> +
>> +	strscpy(params->vdev.name, "mali-c55 3a params",
>> +		sizeof(params->vdev.name));
>> +	vdev->release = video_device_release_empty;
>> +	vdev->fops = &mali_c55_params_v4l2_fops;
>> +	vdev->ioctl_ops = &mali_c55_params_v4l2_ioctl_ops;
>> +	vdev->lock = &params->lock;
>> +	vdev->v4l2_dev = &mali_c55->v4l2_dev;
>> +	vdev->queue = &params->queue;
>> +	vdev->device_caps = V4L2_CAP_META_OUTPUT | V4L2_CAP_STREAMING;
>> +	vdev->vfl_dir = VFL_DIR_TX;
>> +	video_set_drvdata(vdev, params);
>> +
>> +	ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev,
>> +			"failed to register params video device\n");
>> +		goto err_release_vb2q;
>> +	}
>> +
>> +	params->mali_c55 = mali_c55;
>> +
>> +	return 0;
>> +
>> +err_release_vb2q:
>> +	vb2_queue_release(vb2q);
>> +err_cleanup_entity:
>> +	media_entity_cleanup(&params->vdev.entity);
>> +err_destroy_mutex:
>> +	mutex_destroy(&params->lock);
>> +
>> +	return ret;
>> +}
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>> index eb3719245ec3..8e6a801077ed 100644
>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>> @@ -119,6 +119,19 @@
>>   #define MALI_C55_REG_ACTIVE_HEIGHT_MASK			0xffff0000
>>   #define MALI_C55_REG_BAYER_ORDER			0x18e8c
>>   #define MALI_C55_BAYER_ORDER_MASK			GENMASK(1, 0)
>> +
>> +#define MALI_C55_REG_METERING_CONFIG			0x18ed0
>> +#define MALI_C55_5BIN_HIST_DISABLE_MASK			BIT(0)
>> +#define MALI_C55_5BIN_HIST_SWITCH_MASK			GENMASK(2, 1)
>> +#define MALI_C55_AF_DISABLE_MASK			BIT(4)
>> +#define MALI_C55_AF_SWITCH_MASK				BIT(5)
>> +#define MALI_C55_AWB_DISABLE_MASK			BIT(8)
>> +#define MALI_C55_AWB_SWITCH_MASK			BIT(9)
>> +#define MALI_C55_AEXP_HIST_DISABLE_MASK			BIT(12)
>> +#define MALI_C55_AEXP_HIST_SWITCH_MASK			GENMASK(14, 13)
>> +#define MALI_C55_AEXP_IHIST_DISABLE_MASK		BIT(16)
>> +#define MALI_C55_AEXP_SRC_MASK				BIT(24)
>> +
>>   #define MALI_C55_REG_TPG_CH0				0x18ed8
>>   #define MALI_C55_TEST_PATTERN_ON_OFF			BIT(0)
>>   #define MALI_C55_TEST_PATTERN_RGB_MASK			BIT(1)
>> @@ -138,6 +151,11 @@
>>   #define MALI_C55_REG_CONFIG_SPACES_OFFSET		0x0ab6c
>>   #define MALI_C55_CONFIG_SPACE_SIZE			0x1231c
>>   
>> +#define MALI_C55_REG_DIGITAL_GAIN			0x1926c
>> +#define MALI_C55_DIGITAL_GAIN_MASK			GENMASK(12, 0)
>> +#define MALI_C55_REG_DIGITAL_GAIN_OFFSET		0x19270
>> +#define MALI_C55_DIGITAL_GAIN_OFFSET_MASK		GENMASK(19, 0)
>> +
>>   #define MALI_C55_REG_SINTER_CONFIG			0x19348
>>   #define MALI_C55_SINTER_VIEW_FILTER_MASK		GENMASK(1, 0)
>>   #define MALI_C55_SINTER_SCALE_MODE_MASK			GENMASK(3, 2)
>> @@ -146,6 +164,46 @@
>>   #define MALI_C55_SINTER_INT_SELECT_MASK			BIT(6)
>>   #define MALI_C55_SINTER_RM_ENABLE_MASK			BIT(7)
>>   
>> +/* Black Level Correction Configuration */
>> +#define MALI_C55_REG_SENSOR_OFF_PRE_SHA_00		0x1abcc
>> +#define MALI_C55_REG_SENSOR_OFF_PRE_SHA_01		0x1abd0
>> +#define MALI_C55_REG_SENSOR_OFF_PRE_SHA_10		0x1abd4
>> +#define MALI_C55_REG_SENSOR_OFF_PRE_SHA_11		0x1abd8
>> +#define MALI_C55_SENSOR_OFF_PRE_SHA_MASK		0xfffff
>> +
>> +/* Lens Mesh Shading Configuration */
>> +#define MALI_C55_REG_MESH_SHADING_TABLES		0x13074
>> +#define MALI_C55_REG_MESH_SHADING_CONFIG		0x1abfc
>> +#define MALI_C55_MESH_SHADING_ENABLE_MASK		BIT(0)
>> +#define MALI_C55_MESH_SHADING_MESH_SHOW			BIT(1)
>> +#define MALI_C55_MESH_SHADING_SCALE_MASK		GENMASK(4, 2)
>> +#define MALI_C55_MESH_SHADING_PAGE_R_MASK		GENMASK(9, 8)
>> +#define MALI_C55_MESH_SHADING_PAGE_G_MASK		GENMASK(11, 10)
>> +#define MALI_C55_MESH_SHADING_PAGE_B_MASK		GENMASK(13, 12)
>> +#define MALI_C55_MESH_SHADING_MESH_WIDTH_MASK		GENMASK(21, 16)
>> +#define MALI_C55_MESH_SHADING_MESH_HEIGHT_MASK		GENMASK(29, 24)
>> +
>> +#define MALI_C55_REG_MESH_SHADING_ALPHA_BANK		0x1ac04
>> +#define MALI_C55_MESH_SHADING_ALPHA_BANK_R_MASK		GENMASK(2, 0)
>> +#define MALI_C55_MESH_SHADING_ALPHA_BANK_G_MASK		GENMASK(5, 3)
>> +#define MALI_C55_MESH_SHADING_ALPHA_BANK_B_MASK		GENMASK(8, 6)
>> +#define MALI_C55_REG_MESH_SHADING_ALPHA			0x1ac08
>> +#define MALI_C55_MESH_SHADING_ALPHA_R_MASK		GENMASK(7, 0)
>> +#define MALI_C55_MESH_SHADING_ALPHA_G_MASK		GENMASK(15, 8)
>> +#define MALI_C55_MESH_SHADING_ALPHA_B_MASK		GENMASK(23, 16)
>> +#define MALI_C55_REG_MESH_SHADING_MESH_STRENGTH		0x1ac0c
>> +#define MALI_c55_MESH_STRENGTH_MASK			GENMASK(15, 0)
>> +
>> +/* AWB Gains Configuration */
>> +#define MALI_C55_REG_AWB_GAINS1				0x1ac10
>> +#define MALI_C55_AWB_GAIN00_MASK			GENMASK(11, 0)
>> +#define MALI_C55_AWB_GAIN01_MASK			GENMASK(27, 16)
>> +#define MALI_C55_REG_AWB_GAINS2				0x1ac14
>> +#define MALI_C55_AWB_GAIN10_MASK			GENMASK(11, 0)
>> +#define MALI_C55_AWB_GAIN11_MASK			GENMASK(27, 16)
>> +#define MALI_C55_REG_AWB_GAINS1_AEXP			0x1ac18
>> +#define MALI_C55_REG_AWB_GAINS2_AEXP			0x1ac1c
>> +
>>   /* Colour Correction Matrix Configuration */
>>   #define MALI_C55_REG_CCM_ENABLE				0x1b07c
>>   #define MALI_C55_CCM_ENABLE_MASK			BIT(0)
>> @@ -168,6 +226,52 @@
>>   #define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B		0x1b0c8
>>   #define MALI_C55_CCM_ANTIFOG_OFFSET_MASK		GENMASK(11, 0)
>>   
>> +/* AWB Statistics Configuration */
>> +#define MALI_C55_REG_AWB_STATS_MODE			0x1b29c
>> +#define MALI_C55_AWB_STATS_MODE_MASK			BIT(0)
>> +#define MALI_C55_REG_AWB_WHITE_LEVEL			0x1b2a0
>> +#define MALI_C55_AWB_WHITE_LEVEL_MASK			GENMASK(9, 0)
>> +#define MALI_C55_REG_AWB_BLACK_LEVEL			0x1b2a4
>> +#define MALI_C55_AWB_BLACK_LEVEL_MASK			GENMASK(9, 0)
>> +#define MALI_C55_REG_AWB_CR_MAX				0x1b2a8
>> +#define MALI_C55_AWB_CR_MAX_MASK			GENMASK(11, 0)
>> +#define MALI_C55_REG_AWB_CR_MIN				0x1b2ac
>> +#define MALI_C55_AWB_CR_MIN_MASK			GENMASK(11, 0)
>> +#define MALI_C55_REG_AWB_CB_MAX				0x1b2b0
>> +#define MALI_C55_REG_AWB_CB_MIN				0x1b2b4
>> +#define MALI_C55_AWB_CB_MIN_MASK			GENMASK(11, 0)
>> +#define MALI_C55_REG_AWB_NODES_USED			0x1b2c4
>> +#define MALI_C55_AWB_NODES_USED_HORIZ_MASK		GENMASK(7, 0)
>> +#define MALI_C55_AWB_NODES_USED_VERT_MASK		GENMASK(15, 8)
>> +#define MALI_C55_REG_AWB_CR_HIGH			0x1b2c8
>> +#define MALI_C55_AWB_CR_HIGH_MASK			GENMASK(11, 0)
>> +#define MALI_C55_REG_AWB_CR_LOW				0x1b2cc
>> +#define MALI_C55_AWB_CR_LOW_MASK			GENMASK(11, 0)
>> +#define MALI_C55_REG_AWB_CB_HIGH			0x1b2d0
>> +#define MALI_C55_AWB_CB_HIGH_MASK			GENMASK(11, 0)
>> +#define MALI_C55_REG_AWB_CB_LOW				0x1b2d4
>> +#define MALI_C55_AWB_CB_LOW_MASK			GENMASK(11, 0)
>> +
>> +/* AEXP Metering Histogram Configuration */
>> +#define MALI_C55_REG_AEXP_HIST_BASE			0x1b730
>> +#define MALI_C55_REG_AEXP_IHIST_BASE			0x1bbac
>> +#define MALI_C55_AEXP_HIST_SKIP_OFFSET			0
>> +#define MALI_C55_AEXP_HIST_SKIP_X_MASK			GENMASK(2, 0)
>> +#define MALI_C55_AEXP_HIST_OFFSET_X_MASK		BIT(3)
>> +#define MALI_C55_AEXP_HIST_SKIP_Y_MASK			GENMASK(6, 4)
>> +#define MALI_C55_AEXP_HIST_OFFSET_Y_MASK		BIT(7)
>> +#define MALI_C55_AEXP_HIST_SCALE_OFFSET			4
>> +#define MALI_C55_AEXP_HIST_SCALE_BOTTOM_MASK		GENMASK(3, 0)
>> +#define MALI_C55_AEXP_HIST_SCALE_TOP_MASK		GENMASK(7, 4)
>> +#define MALI_C55_AEXP_HIST_PLANE_MODE_OFFSET		16
>> +#define MALI_C55_AEXP_HIST_PLANE_MODE_MASK		GENMASK(2, 0)
>> +#define MALI_C55_AEXP_HIST_NODES_USED_OFFSET		52
>> +#define MALI_C55_AEXP_HIST_NODES_USED_HORIZ_MASK	GENMASK(7, 0)
>> +#define MALI_C55_AEXP_HIST_NODES_USED_VERT_MASK		GENMASK(15, 8)
>> +#define MALI_C55_AEXP_HIST_ZONE_WEIGHTS_OFFSET		56
>> +#define MALI_C55_AEXP_HIST_ZONE_WEIGHT_MASK		0x0f0f0f0f
>> +
>>   /*
>>    * The Mali-C55 ISP has up to two output pipes; known as full resolution and
>>    * down scaled. The register space for these is laid out identically, but offset

Re: [PATCH v5 15/16] media: platform: Add mali-c55 parameters video node

[Sakari Ailus]

Hi Dan,

On Fri, Jun 14, 2024 at 09:15:07PM +0100, Dan Scally wrote:
> > > +void mali_c55_params_write_config(struct mali_c55 *mali_c55)
> > > +{
> > > +	struct mali_c55_params *params = &mali_c55->params;
> > > +	enum vb2_buffer_state state = VB2_BUF_STATE_DONE;
> > > +	struct mali_c55_params_buffer *config;
> > > +	struct mali_c55_buffer *buf;
> > > +	size_t block_offset = 0;
> > > +
> > > +	spin_lock(&params->buffers.lock);
> > > +
> > > +	buf = list_first_entry_or_null(&params->buffers.queue,
> > > +				       struct mali_c55_buffer, queue);
> > > +	if (buf)
> > > +		list_del(&buf->queue);
> > > +	spin_unlock(&params->buffers.lock);
> > > +
> > > +	if (!buf)
> > > +		return;
> > > +
> > > +	buf->vb.sequence = mali_c55->isp.frame_sequence;
> > > +	config = vb2_plane_vaddr(&buf->vb.vb2_buf, 0);
> > > +
> > > +	if (config->total_size > MALI_C55_PARAMS_MAX_SIZE) {
> > > +		dev_dbg(mali_c55->dev, "Invalid parameters buffer size %lu\n",
> > > +			config->total_size);
> > > +		state = VB2_BUF_STATE_ERROR;
> > > +		goto err_buffer_done;
> > > +	}
> > > +
> > > +	/* Walk the list of parameter blocks and process them. */
> > > +	while (block_offset < config->total_size) {
> > > +		const struct mali_c55_block_handler *block_handler;
> > > +		struct mali_c55_params_block_header *block;
> > > +
> > > +		block = (struct mali_c55_params_block_header *)
> > > +			 &config->data[block_offset];
> > How do you ensure config->data does hold a full struct
> > mali_c33_params_block_header at block_offset (i.e. that the struct does not
> > exceed the memory available for config->data)?
> 
> 
> We don't currently...the data buffer is sized specifically to be large
> enough to accept a single instance of each possible struct that could be
> included, we could keep track of the blocks that we have seen already and
> ensure that none are seen twice...and that should guarantee that the
> remaining space is sufficient to hold whatever the last block is. Does that
> sound ok?

Ḯ'd add an explicit check here. It's more simple way to ensure memory
safety here: relying on a complex machinery that can't be trivially
validated does risk having grave bugs, not only now but later on as well as
modifications to the code are done.

> 
> > 
> > > +
> > > +		if (block->type >= MALI_C55_PARAM_BLOCK_SENTINEL) {
> > > +			dev_dbg(mali_c55->dev, "Invalid parameters block type\n");
> > > +			state = VB2_BUF_STATE_ERROR;
> > > +			goto err_buffer_done;
> > > +		}
> > > +
> > > +		block_handler = &mali_c55_block_handlers[block->type];
> > > +		if (block->size != block_handler->size) {
> > How do you ensure config->data has room for the block?
> I think through the same proposal as above.

Similarly here. You already even have the size of the blocks available
here.

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v5 15/16] media: platform: Add mali-c55 parameters video node

[Dan Scally]

Hi Sakari

On 14/06/2024 22:11, Sakari Ailus wrote:
> Hi Dan,
>
> On Fri, Jun 14, 2024 at 09:15:07PM +0100, Dan Scally wrote:
>>>> +void mali_c55_params_write_config(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	struct mali_c55_params *params = &mali_c55->params;
>>>> +	enum vb2_buffer_state state = VB2_BUF_STATE_DONE;
>>>> +	struct mali_c55_params_buffer *config;
>>>> +	struct mali_c55_buffer *buf;
>>>> +	size_t block_offset = 0;
>>>> +
>>>> +	spin_lock(&params->buffers.lock);
>>>> +
>>>> +	buf = list_first_entry_or_null(&params->buffers.queue,
>>>> +				       struct mali_c55_buffer, queue);
>>>> +	if (buf)
>>>> +		list_del(&buf->queue);
>>>> +	spin_unlock(&params->buffers.lock);
>>>> +
>>>> +	if (!buf)
>>>> +		return;
>>>> +
>>>> +	buf->vb.sequence = mali_c55->isp.frame_sequence;
>>>> +	config = vb2_plane_vaddr(&buf->vb.vb2_buf, 0);
>>>> +
>>>> +	if (config->total_size > MALI_C55_PARAMS_MAX_SIZE) {
>>>> +		dev_dbg(mali_c55->dev, "Invalid parameters buffer size %lu\n",
>>>> +			config->total_size);
>>>> +		state = VB2_BUF_STATE_ERROR;
>>>> +		goto err_buffer_done;
>>>> +	}
>>>> +
>>>> +	/* Walk the list of parameter blocks and process them. */
>>>> +	while (block_offset < config->total_size) {
>>>> +		const struct mali_c55_block_handler *block_handler;
>>>> +		struct mali_c55_params_block_header *block;
>>>> +
>>>> +		block = (struct mali_c55_params_block_header *)
>>>> +			 &config->data[block_offset];
>>> How do you ensure config->data does hold a full struct
>>> mali_c33_params_block_header at block_offset (i.e. that the struct does not
>>> exceed the memory available for config->data)?
>>
>> We don't currently...the data buffer is sized specifically to be large
>> enough to accept a single instance of each possible struct that could be
>> included, we could keep track of the blocks that we have seen already and
>> ensure that none are seen twice...and that should guarantee that the
>> remaining space is sufficient to hold whatever the last block is. Does that
>> sound ok?
> Ḯ'd add an explicit check here.


How would you do the check, sorry?

> It's more simple way to ensure memory
> safety here: relying on a complex machinery that can't be trivially
> validated does risk having grave bugs, not only now but later on as well as
> modifications to the code are done.
>
>>>> +
>>>> +		if (block->type >= MALI_C55_PARAM_BLOCK_SENTINEL) {
>>>> +			dev_dbg(mali_c55->dev, "Invalid parameters block type\n");
>>>> +			state = VB2_BUF_STATE_ERROR;
>>>> +			goto err_buffer_done;
>>>> +		}
>>>> +
>>>> +		block_handler = &mali_c55_block_handlers[block->type];
>>>> +		if (block->size != block_handler->size) {
>>> How do you ensure config->data has room for the block?
>> I think through the same proposal as above.
> Similarly here. You already even have the size of the blocks available
> here.
>

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Laurent Pinchart]

Hi Dan,

On Fri, Jun 14, 2024 at 11:13:29AM +0100, Daniel Scally wrote:
> On 30/05/2024 01:15, Laurent Pinchart wrote:
> > On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
> >> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
> >> V4L2 and Media Controller compliant and creates subdevices to manage
> >> the ISP itself, its internal test pattern generator as well as the
> >> crop, scaler and output format functionality for each of its two
> >> output devices.
> >>
> >> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
> >> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> >> ---
> >> Changes in v5:
> >>
> >> 	- Reworked input formats - previously we allowed representing input data
> >> 	  as any 8-16 bit format. Now we only allow input data to be represented
> >> 	  by the new 20-bit bayer formats, which is corrected to the equivalent
> >> 	  16-bit format in RAW bypass mode.
> >> 	- Stopped bypassing blocks that we haven't added supporting parameters
> >> 	  for yet.
> >> 	- Addressed most of Sakari's comments from the list
> >>
> >> Changes not yet made in v5:
> >>
> >> 	- The output pipelines can still be started and stopped independently of
> >> 	  one another - I'd like to discuss that more.
> >> 	- the TPG subdev still uses .s_stream() - I need to rebase onto a tree
> >> 	  with working .enable_streams() for a single-source-pad subdevice.
> >>
> >> Changes in v4:
> >>
> >> 	- Reworked mali_c55_update_bits() to internally perform the bit-shift
> >
> > I really don't like that, it makes the code very confusing, even more so
> > as it differs from regmap_update_bits().
> >
> > Look at this for instance:
> >
> > 	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> > 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> > 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> >
> > It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
> > BIT(0).
> >
> > Sorry, I know it will be painful, but this change needs to be reverted.
> 
> I'd like to argue for keeping this, on the grounds that it's better. I got lazy in the change you 
> reference there, and because BIT(0) is the same as 0x01 didn't bother changing it. I agree that 
> that's confusing but I think it would be better to keep the change and just update all the call 
> sites properly. The benefits as I see them are two:
> 
> 
> 1. This method ensures sane consistent calling of the function. Without the internal shift you have 
> to shift the values at the call site, but there's no reason to do that if the value you're setting 
> is 0x00 or if the field you're targeting in the register starts at bit 0, so I think writing code 
> naturally we'd have a mix of situations like so:
> 
> 
> #define REG_1 0xfc00
> 
> #define REG_2 0xff
> 
> mali_c55_update_bits(mali_c55, 0x1234, REG_1, 0x02 << 10);
> 
> mali_c55_update_bits(mali_c55, 0x1234, REG_1, 0x00);
> 
> mali_c55_update_bits(mali_c55, 0x1234, REG_2, 0x02);
>
> And I think that the mixture is more confusing than the difference with regmap_update_bits(). We 
> could include the bitshifting for consistencies sake despite it being unecessary, but it's extra 
> work for no real reason and itself "looks wrong" if the field starts at bit(0)...it would look less 
> wrong with an offset macro that defines the number of bits to shift as 0 but then we're on to 
> advantage #2...
> 
> 2. It makes the driver far cleaner. Without it we either have magic numbers scattered throughout 
> (and sometimes have to calculate them with extra variables where the write can target different 
> places conditionally) or have macros defining the number of bits to shift, or have to do (ffs(mask) 
> - 1) everywhere, and that tends to make the call sites a lot messier - this was the original reason 
> I moved it internal actually.
> 
> What do you think?

All register values (possibly with the exception of 0) should have
macros. The callers will thus not need to perform shifts manually, they
will all be handled in the register fields macros. From a caller point
of view, not handling the shifts inside mali_c55_update_bits() will not
make a difference.

It's the first time I notice a driver trying to shift internally in its
update_bits function. I think that's really confusing, especially given
that it departs from how regmap operates. I still strongly favour
handling the shifts in the register macros.

> >> 	- Reworked the resizer to allow cropping during streaming
> >> 	- Fixed a bug in NV12 output
> >>
> >> Changes in v3:
> >>
> >> 	- Mostly minor fixes suggested by Sakari
> >> 	- Fixed the sequencing of vb2 buffers to be synchronised across the two
> >> 	  capture devices.
> >>
> >> Changes in v2:
> >>
> >> 	- Clock handling
> >> 	- Fixed the warnings raised by the kernel test robot
> >>
> >>   drivers/media/platform/Kconfig                |   1 +
> >>   drivers/media/platform/Makefile               |   1 +
> >>   drivers/media/platform/arm/Kconfig            |   5 +
> >>   drivers/media/platform/arm/Makefile           |   2 +
> >>   drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
> >>   drivers/media/platform/arm/mali-c55/Makefile  |   9 +
> >>   .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
> >>   .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
> >>   .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
> >>   .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
> >>   .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
> >>   .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
> >>   .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
> >>   .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
> >>   14 files changed, 4452 insertions(+)
> >>   create mode 100644 drivers/media/platform/arm/Kconfig
> >>   create mode 100644 drivers/media/platform/arm/Makefile
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c

[snip]

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Laurent Pinchart]

On Sun, Jun 09, 2024 at 06:21:52AM +0000, Sakari Ailus wrote:
> On Thu, Jun 06, 2024 at 10:10:14PM +0300, Tomi Valkeinen wrote:
> > On 06/06/2024 20:53, Laurent Pinchart wrote:
> > > > > > > +			return -EINVAL;
> > > > > > > +		}
> > > > > > > +
> > > > > > > +		active_sink = route->sink_pad;
> > > > > > > +	}
> > > > > > > +	if (active_sink == UINT_MAX) {
> > > > > > > +		dev_err(rzr->mali_c55->dev, "One route has to be active");
> > > > > > > +		return -EINVAL;
> > > > > > > +	}
> > > > >
> > > > > The recommended handling of invalid routing is to adjust the routing
> > > > > table, not to return errors.
> > > >
> > > > How should I adjust it ? The error here is due to the fact multiple
> > > > routes are set as active, which one should I make active ? the first
> > > > one ? Should I go and reset the flags in the subdev_route for the one
> > > > that has to be made non-active ?
> > >
> > > The same way you would adjust an invalid format, you can pick the route
> > > you consider should be the default.
> > > 
> > > I'd like Sakari's and Tomi's opinions on this, as it's a new API and the
> > > behaviour is still a bit in flux.
> > 
> > Well... My opinion is that the driver adjusting the given config parameters
> > (for any ioctl) is awful and should be deprecated. If the user asks for X,
> > and the driver adjusts it and returns Y, then the user has two options:
> > fail, because it didn't get X (after possibly laborious field by field
> > checks), or shrug it's virtual shoulders and accept Y and hope that things
> > still work even though it wanted X.
> 
> This is still often the only way to tell what the hardware can do as the
> limitations in different cases (cropping and scaling for instance) can be
> arbitrary. The other option is that the user space has to know the hardware
> capabilities without them being available from the kernel.

For some parameters that make sense (we don't have a try mechanism for
ISP parameters buffers for instance), but when it comes to configuring a
pipeline, I think a parameters adjustment model is needed when we don't
have means to expose constraints in a generic way to userspace. The
question is in which category routing falls.

> There could be cases of IOCTLs where returning an error if what was
> requested can't be performed exactly is workable in general, but then again
> having consistency across IOCTL behaviour is very beneficial as well.
> 
> If you need something exactly, then I think you should check after the
> IOCTL that this is what you also got, beyond the IOCTL succeeding.

I do agree with Tomi that this kind of check can be annoying for
applications. In cases where checking the result would be complex, and
where there is very little use case for receiving anything but the exact
configuration you asked for, adjusting the parameters could increase the
implementation complexity on both the kernel side and userspace side for
no or very little benefit.

> > But maybe that was an answer to a question you didn't really ask =).
> > 
> > I think setting it to default routing in case of an error is as fine as any
> > other "fix" for the routing. It won't work anyway.
> > 
> > But if the function sets default routing and returns 0 here, why would it
> > return an error from v4l2_subdev_routing_validate()? Should it just set
> > default routing in that case too? So should set_routing() ever return an
> > error, if we can just set the default routing?

That's a good point. I asked myself the same question after sending my
previous e-mail, and wondered if anyone else would notice too :-)

> S_ROUTING is a bit special as it deals with multiple routes and the user
> space does have a way to add them incrementally.
> 
> Perhaps we should document better what the driver is expected to to correct
> the routes?

We should document the expected behaviour clearly. After agreeing on the
expected behaviour, that is.

> I'd think routes may be added by the driver (as some of them cannot be
> disabled for instance) but if a requested route cannot be created, that
> should probably be an error.
> 
> I've copied my current (with all the pending patches) documentation here
> <URL:https://www.retiisi.eu/~sailus/v4l2/tmp/streams-doc/userspace-api/media/v4l/dev-subdev.html#streams-multiplexed-media-pads-and-internal-routing>.
>
> The text does not elaborate what exactly a driver could or should do, apart
> from specifying the condition for EINVAL. I think we should specify this in

I don't see mentions of EINVAL related to streams there, am I missing
something ?

> greater detail. My original thought wws the adjustment would be done by
> adding static routes omitted by the caller, not trying to come up with e.g.
> valid pad/stream pairs when user provided invalid ones.
> 
> Could this correction functionality be limited to returning static routes?

That would make userspace a tad simpler, and wouldn't be hard to do in
the kernel, but I wonder if departing from the rule that invalid routing
tables result in an error is worth it for such a small gain.

> > In the VIDIOC_SUBDEV_S_ROUTING doc we do list some cases where EINVAL or
> > E2BIG is returned. But only a few, and I think
> > v4l2_subdev_routing_validate() will return errors for many other cases too.
> > 
> > For what it's worth, the drivers I have written just return an error. It's
> > simple for the driver and the user and works. If the consensus is that the
> > drivers should instead set the default routing, or somehow mangle the given
> > routing to an acceptable form, I can update those drivers accordingly.
> > 
> > But we probably need to update the docs too to be a bit more clear what
> > VIDIOC_SUBDEV_S_ROUTING will do (although are the other ioctls any
> > clearer?).
> > 
> > All that said, I think it's still a bit case-by-case. I don't think the
> > drivers should always return an error if they get a routing table that's not
> > 100% perfect. E.g. if a device supports two static routes, but the second
> > one can be enabled or disabled, the driver should still accept a routing
> > table from the user with only the first route present. Etc.
> > 
> > For the specific case in this patch... I'd prefer returning an error, or if
> > that's not ok, set default routing.
> 
> Not modifying the routing table is another option as well but it may
> require separating validating user-provided routes and applying the routes

I'm not sure to follow you here. By not modifying the routing table, do
you mean returning an error ? Why would that require separation of
validation and configuration ?

> to the sub-device state. The default could be useful in principle, too, for
> routing-unaware applications but they won't be calling S_ROUTING anyway.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 10/16] media: platform: Add mali-c55 3a stats devnode

[Laurent Pinchart]

Hi Dan,

Thank you for the patch.

On Wed, May 29, 2024 at 04:28:52PM +0100, Daniel Scally wrote:
> Add a new code file to govern the 3a statistics capture node.
> 
> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> ---
> Changes in v5:
> 
> 	- New patch
> 
>  drivers/media/platform/arm/mali-c55/Makefile  |   3 +-
>  .../platform/arm/mali-c55/mali-c55-common.h   |  28 ++
>  .../platform/arm/mali-c55/mali-c55-core.c     |  15 +
>  .../platform/arm/mali-c55/mali-c55-isp.c      |   1 +
>  .../arm/mali-c55/mali-c55-registers.h         |   3 +
>  .../platform/arm/mali-c55/mali-c55-stats.c    | 350 ++++++++++++++++++
>  6 files changed, 399 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-stats.c
> 
> diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
> index 77dcb2fbf0f4..cd5a64bf0c62 100644
> --- a/drivers/media/platform/arm/mali-c55/Makefile
> +++ b/drivers/media/platform/arm/mali-c55/Makefile
> @@ -4,6 +4,7 @@ mali-c55-y := mali-c55-capture.o \
>  	      mali-c55-core.o \
>  	      mali-c55-isp.o \
>  	      mali-c55-tpg.o \
> -	      mali-c55-resizer.o
> +	      mali-c55-resizer.o \
> +	      mali-c55-stats.o
>  
>  obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> index 2d0c4d152beb..44119e04009b 100644
> --- a/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> @@ -79,6 +79,7 @@ enum mali_c55_isp_pads {
>  	MALI_C55_ISP_PAD_SINK_VIDEO,
>  	MALI_C55_ISP_PAD_SOURCE,
>  	MALI_C55_ISP_PAD_SOURCE_BYPASS,
> +	MALI_C55_ISP_PAD_SOURCE_3A,

Functions and structures are named with a "stats" suffix, let's call
this MALI_C55_ISP_PAD_SOURCE_STATS.

>  	MALI_C55_ISP_NUM_PADS,
>  };
>  
> @@ -194,6 +195,28 @@ struct mali_c55_cap_dev {
>  	bool streaming;
>  };
>  
> +struct mali_c55_stats_buf {
> +	struct vb2_v4l2_buffer vb;
> +	spinlock_t lock;

All locks require a comment to document what they protect. Same below.

> +	unsigned int segments_remaining;
> +	struct list_head queue;
> +	bool failed;
> +};
> +
> +struct mali_c55_stats {
> +	struct mali_c55 *mali_c55;
> +	struct video_device vdev;
> +	struct dma_chan *channel;
> +	struct vb2_queue queue;
> +	struct media_pad pad;
> +	struct mutex lock;
> +
> +	struct {
> +		spinlock_t lock;
> +		struct list_head queue;
> +	} buffers;
> +};
> +
>  enum mali_c55_config_spaces {
>  	MALI_C55_CONFIG_PING,
>  	MALI_C55_CONFIG_PONG,
> @@ -224,6 +247,7 @@ struct mali_c55 {
>  	struct mali_c55_isp isp;
>  	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
>  	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
> +	struct mali_c55_stats stats;
>  
>  	struct list_head contexts;
>  	enum mali_c55_config_spaces next_config;
> @@ -245,6 +269,8 @@ int mali_c55_register_resizers(struct mali_c55 *mali_c55);
>  void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
>  int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
>  void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
> +int mali_c55_register_stats(struct mali_c55 *mali_c55);
> +void mali_c55_unregister_stats(struct mali_c55 *mali_c55);
>  struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
>  void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>  			     enum mali_c55_planes plane);
> @@ -262,5 +288,7 @@ mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
>  bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
>  #define for_each_mali_isp_fmt(fmt)\
>  	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
> +void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
> +				enum mali_c55_config_spaces cfg_space);
>  
>  #endif /* _MALI_C55_COMMON_H */
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> index 50caf5ee7474..9ea70010876c 100644
> --- a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> @@ -337,6 +337,16 @@ static int mali_c55_create_links(struct mali_c55 *mali_c55)
>  		}
>  	}
>  
> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> +			MALI_C55_ISP_PAD_SOURCE_3A,
> +			&mali_c55->stats.vdev.entity, 0,
> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> +	if (ret) {
> +		dev_err(mali_c55->dev,
> +			"failed to link ISP and 3a stats node\n");

s/3a stats/stats/

> +		goto err_remove_links;
> +	}
> +
>  	return 0;
>  
>  err_remove_links:
> @@ -350,6 +360,7 @@ static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
>  	mali_c55_unregister_isp(mali_c55);
>  	mali_c55_unregister_resizers(mali_c55);
>  	mali_c55_unregister_capture_devs(mali_c55);
> +	mali_c55_unregister_stats(mali_c55);
>  }
>  
>  static int mali_c55_register_entities(struct mali_c55 *mali_c55)
> @@ -372,6 +383,10 @@ static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>  	if (ret)
>  		goto err_unregister_entities;
>  
> +	ret = mali_c55_register_stats(mali_c55);
> +	if (ret)
> +		goto err_unregister_entities;
> +
>  	ret = mali_c55_create_links(mali_c55);
>  	if (ret)
>  		goto err_unregister_entities;
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> index ea8b7b866e7a..94876fba3353 100644
> --- a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> @@ -564,6 +564,7 @@ int mali_c55_register_isp(struct mali_c55 *mali_c55)
>  	isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
>  	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>  	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
> +	isp->pads[MALI_C55_ISP_PAD_SOURCE_3A].flags = MEDIA_PAD_FL_SOURCE;
>  
>  	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
>  				     isp->pads);
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> index cb27abde2aa5..eb3719245ec3 100644
> --- a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> @@ -68,6 +68,9 @@
>  #define MALI_C55_VC_START(v)				((v) & 0xffff)
>  #define MALI_C55_VC_SIZE(v)				(((v) & 0xffff) << 16)
>  
> +#define MALI_C55_REG_1024BIN_HIST			0x054a8
> +#define MALI_C55_1024BIN_HIST_SIZE			4096
> +
>  /* Ping/Pong Configuration Space */
>  #define MALI_C55_REG_BASE_ADDR				0x18e88
>  #define MALI_C55_REG_BYPASS_0				0x18eac
> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-stats.c b/drivers/media/platform/arm/mali-c55/mali-c55-stats.c
> new file mode 100644
> index 000000000000..aa40480ed814
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-stats.c
> @@ -0,0 +1,350 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ARM Mali-C55 ISP Driver - 3A Statistics capture device
> + *
> + * Copyright (C) 2023 Ideas on Board Oy
> + */
> +
> +#include <linux/dmaengine.h>
> +#include <linux/media/arm/mali-c55-config.h>
> +#include <linux/spinlock.h>

You're missing some headers here, for

container_of()
dev_err()
list_*()
mutex_init()
strscpy()
strscpy()

> +
> +#include <media/media-entity.h>
> +#include <media/v4l2-dev.h>
> +#include <media/v4l2-event.h>
> +#include <media/v4l2-fh.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/videobuf2-core.h>
> +#include <media/videobuf2-dma-contig.h>
> +
> +#include "mali-c55-common.h"
> +#include "mali-c55-registers.h"
> +
> +static unsigned int metering_space_addrs[] = {

const

> +	[MALI_C55_CONFIG_PING] = 0x095AC,
> +	[MALI_C55_CONFIG_PONG] = 0x2156C,

Lower-case hex constants.

> +};
> +
> +static int mali_c55_stats_enum_fmt_meta_cap(struct file *file, void *fh,
> +					    struct v4l2_fmtdesc *f)
> +{
> +	if (f->index || f->type != V4L2_BUF_TYPE_META_CAPTURE)
> +		return -EINVAL;
> +
> +	f->pixelformat = V4L2_META_FMT_MALI_C55_3A_STATS;

The format could be called V4L2_META_FMT_MALI_C55_STATS. While most
statistics are related to one of the 3A algorithms, I think it would be
better to name this generically. It's name bikeshedding only of course.

> +
> +	return 0;
> +}
> +
> +static int mali_c55_stats_g_fmt_meta_cap(struct file *file, void *fh,
> +					 struct v4l2_format *f)
> +{
> +	static const struct v4l2_meta_format mfmt = {
> +		.dataformat = V4L2_META_FMT_MALI_C55_3A_STATS,
> +		.buffersize = sizeof(struct mali_c55_stats_buffer)
> +	};
> +
> +	if (f->type != V4L2_BUF_TYPE_META_CAPTURE)
> +		return -EINVAL;
> +
> +	f->fmt.meta = mfmt;
> +
> +	return 0;
> +}
> +
> +static int mali_c55_stats_querycap(struct file *file,
> +				   void *priv, struct v4l2_capability *cap)
> +{
> +	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
> +	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_ioctl_ops mali_c55_stats_v4l2_ioctl_ops = {
> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
> +	.vidioc_querybuf = vb2_ioctl_querybuf,
> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
> +	.vidioc_qbuf = vb2_ioctl_qbuf,
> +	.vidioc_expbuf = vb2_ioctl_expbuf,
> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> +	.vidioc_streamon = vb2_ioctl_streamon,
> +	.vidioc_streamoff = vb2_ioctl_streamoff,
> +	.vidioc_enum_fmt_meta_cap = mali_c55_stats_enum_fmt_meta_cap,
> +	.vidioc_g_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
> +	.vidioc_s_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
> +	.vidioc_try_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
> +	.vidioc_querycap = mali_c55_stats_querycap,
> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
> +};
> +
> +static const struct v4l2_file_operations mali_c55_stats_v4l2_fops = {
> +	.owner = THIS_MODULE,
> +	.unlocked_ioctl = video_ioctl2,
> +	.open = v4l2_fh_open,
> +	.release = vb2_fop_release,
> +	.poll = vb2_fop_poll,
> +	.mmap = vb2_fop_mmap,
> +};
> +
> +static int
> +mali_c55_stats_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
> +			   unsigned int *num_planes, unsigned int sizes[],
> +			   struct device *alloc_devs[])
> +{
> +	struct mali_c55_stats *stats = vb2_get_drv_priv(q);
> +
> +	if (*num_planes && *num_planes > 1)
> +		return -EINVAL;
> +
> +	if (sizes[0] && sizes[0] != sizeof(struct mali_c55_stats_buffer))
> +		return -EINVAL;
> +
> +	*num_planes = 1;
> +	sizes[0] = sizeof(struct mali_c55_stats_buffer);
> +
> +	if (stats->channel)
> +		alloc_devs[0] = stats->channel->device->dev;
> +
> +	return 0;
> +}
> +
> +static void mali_c55_stats_buf_queue(struct vb2_buffer *vb)
> +{
> +	struct mali_c55_stats *stats = vb2_get_drv_priv(vb->vb2_queue);
> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> +	struct mali_c55_stats_buf *buf = container_of(vbuf,
> +						struct mali_c55_stats_buf, vb);
> +
> +	vb2_set_plane_payload(vb, 0, sizeof(struct mali_c55_stats_buffer));
> +	buf->segments_remaining = 2;
> +	buf->failed = false;
> +
> +	spin_lock(&stats->buffers.lock);

Isn't the DMA completion handler run from IRQ context ? If so you'll
need to use spin_lock_irq() here and in the other function that are not
called with interrupts disabled.

> +	list_add_tail(&buf->queue, &stats->buffers.queue);
> +	spin_unlock(&stats->buffers.lock);
> +}
> +
> +static void mali_c55_stats_stop_streaming(struct vb2_queue *q)
> +{
> +	struct mali_c55_stats *stats = vb2_get_drv_priv(q);
> +	struct mali_c55_stats_buf *buf, *tmp;
> +
> +	spin_lock(&stats->buffers.lock);
> +
> +	list_for_each_entry_safe(buf, tmp, &stats->buffers.queue, queue) {
> +		list_del(&buf->queue);
> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
> +	}
> +
> +	spin_unlock(&stats->buffers.lock);
> +}
> +
> +static const struct vb2_ops mali_c55_stats_vb2_ops = {
> +	.queue_setup = mali_c55_stats_queue_setup,
> +	.buf_queue = mali_c55_stats_buf_queue,
> +	.wait_prepare = vb2_ops_wait_prepare,
> +	.wait_finish = vb2_ops_wait_finish,
> +	.stop_streaming = mali_c55_stats_stop_streaming,
> +};
> +
> +static void
> +mali_c55_stats_metering_complete(void *param,
> +				 const struct dmaengine_result *result)
> +{
> +	struct mali_c55_stats_buf *buf = param;
> +
> +	spin_lock(&buf->lock);

I wonder if this is needed. Can the DMA engine call the completion
handlers of two sequential DMA transfers in parallel ?

> +
> +	if (buf->failed)
> +		goto out_unlock;
> +
> +	buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> +
> +	if (result->result != DMA_TRANS_NOERROR) {
> +		buf->failed = true;
> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);

This will possibly return the buffer to userspace after the first DMA
transfer. Userspace could then requeue the buffer to the kernel before
the completion of the second DMA transfer. That will cause trouble. I
think you should instead do something like

	spin_lock(&buf->lock);

	if (result->result != DMA_TRANS_NOERROR)
		buf->failed = true;

	if (!--buf->segments_remaining) {
		buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
		vb2_buffer_done(&buf->vb.vb2_buf, buf->failed ?
				VB2_BUF_STATE_ERROR : VB2_BUF_STATE_DONE);
	}

	spin_unlock(&buf->lock);

The

	buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();

line could also be moved to mali_c55_stats_fill_buffer(), which would
make sure the timestamp is filled in case of DMA submission failures.

> +		goto out_unlock;
> +	}
> +
> +	if (!--buf->segments_remaining)
> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
> +
> +out_unlock:
> +	spin_unlock(&buf->lock);
> +}
> +
> +static int mali_c55_stats_dma_xfer(struct mali_c55_stats *stats, dma_addr_t src,
> +				   dma_addr_t dst,
> +				   struct mali_c55_stats_buf *buf,
> +				   size_t length,
> +				   void (*callback)(void *, const struct dmaengine_result *result))

The same callback is used for both invocations of this function, you can
drop the parameter and hardcode it below.

> +{
> +	struct dma_async_tx_descriptor *tx;
> +	dma_cookie_t cookie;
> +
> +	tx = dmaengine_prep_dma_memcpy(stats->channel, dst, src, length, 0);
> +	if (!tx) {
> +		dev_err(stats->mali_c55->dev, "failed to prep stats DMA\n");
> +		return -EIO;
> +	}
> +
> +	tx->callback_result = callback;
> +	tx->callback_param = buf;
> +
> +	cookie = dmaengine_submit(tx);
> +	if (dma_submit_error(cookie)) {
> +		dev_err(stats->mali_c55->dev, "failed to submit stats DMA\n");
> +		return -EIO;
> +	}
> +
> +	dma_async_issue_pending(stats->channel);
> +	return 0;
> +}
> +
> +void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
> +				enum mali_c55_config_spaces cfg_space)
> +{
> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> +	struct mali_c55_stats *stats = &mali_c55->stats;
> +	struct mali_c55_stats_buf *buf = NULL;
> +	dma_addr_t src, dst;
> +	int ret;
> +
> +	spin_lock(&stats->buffers.lock);
> +	if (!list_empty(&stats->buffers.queue)) {
> +		buf = list_first_entry(&stats->buffers.queue,
> +				       struct mali_c55_stats_buf, queue);
> +		list_del(&buf->queue);
> +	}
> +	spin_unlock(&stats->buffers.lock);
> +
> +	if (!buf)
> +		return;
> +
> +	buf->vb.sequence = mali_c55->isp.frame_sequence;
> +
> +	/*
> +	 * There are infact two noncontiguous sections of the ISP's

s/infact/in fact/

> +	 * memory space that hold statistics for 3a algorithms to use. A

s/use. A/use: a/

> +	 * section in each config space and a global section holding
> +	 * histograms which is double buffered and so holds data for the
> +	 * last frame. We need to read both.
> +	 */
> +	src = ctx->base + MALI_C55_REG_1024BIN_HIST;
> +	dst = vb2_dma_contig_plane_dma_addr(&buf->vb.vb2_buf, 0);
> +
> +	ret = mali_c55_stats_dma_xfer(stats, src, dst, buf,
> +				      MALI_C55_1024BIN_HIST_SIZE,
> +				      mali_c55_stats_metering_complete);
> +	if (ret)
> +		goto err_fail_buffer;
> +
> +	src = ctx->base + metering_space_addrs[cfg_space];
> +	dst += MALI_C55_1024BIN_HIST_SIZE;
> +
> +	ret = mali_c55_stats_dma_xfer(
> +		stats, src, dst, buf,
> +		sizeof(struct mali_c55_stats_buffer) - MALI_C55_1024BIN_HIST_SIZE,
> +		mali_c55_stats_metering_complete);
> +	if (ret) {
> +		dmaengine_terminate_sync(stats->channel);
> +		goto err_fail_buffer;
> +	}

I think you will need to terminate DMA transfers at stream off time.

> +
> +	return;
> +
> +err_fail_buffer:
> +	vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
> +}
> +
> +void mali_c55_unregister_stats(struct mali_c55 *mali_c55)
> +{
> +	struct mali_c55_stats *stats = &mali_c55->stats;
> +
> +	if (!video_is_registered(&stats->vdev))
> +		return;
> +
> +	vb2_video_unregister_device(&stats->vdev);
> +	media_entity_cleanup(&stats->vdev.entity);
> +	dma_release_channel(stats->channel);
> +	mutex_destroy(&stats->lock);
> +}
> +
> +int mali_c55_register_stats(struct mali_c55 *mali_c55)
> +{
> +	struct mali_c55_stats *stats = &mali_c55->stats;
> +	struct video_device *vdev = &stats->vdev;
> +	struct vb2_queue *vb2q = &stats->queue;
> +	dma_cap_mask_t mask;
> +	int ret;
> +
> +	mutex_init(&stats->lock);
> +	INIT_LIST_HEAD(&stats->buffers.queue);
> +
> +	dma_cap_zero(mask);
> +	dma_cap_set(DMA_MEMCPY, mask);
> +
> +	stats->channel = dma_request_channel(mask, 0, NULL);

Do we need a CPU fallback in case no DMA is available ?

I'm still very curious to know how long it takes to perform the DMA
transfer, compared to copying the data with the CPU, and especially
compared to the frame duration.

> +	if (!stats->channel) {
> +		ret = -ENODEV;
> +		goto err_destroy_mutex;
> +	}
> +
> +	stats->pad.flags = MEDIA_PAD_FL_SINK;
> +	ret = media_entity_pads_init(&stats->vdev.entity, 1, &stats->pad);
> +	if (ret)
> +		goto err_release_dma_channel;
> +
> +	vb2q->type = V4L2_BUF_TYPE_META_CAPTURE;
> +	vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
> +	vb2q->drv_priv = stats;
> +	vb2q->mem_ops = &vb2_dma_contig_memops;
> +	vb2q->ops = &mali_c55_stats_vb2_ops;
> +	vb2q->buf_struct_size = sizeof(struct mali_c55_stats_buf);
> +	vb2q->min_queued_buffers = 1;
> +	vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> +	vb2q->lock = &stats->lock;
> +	vb2q->dev = mali_c55->dev;

That's not the right device. The device that performs the DMA operation
is the DMA engine, and that's what you need to pass to vb2. Otherwise
the DMA address returned by vb2_dma_contig_plane_dma_addr() will be
mapped to the ISP device, not the DMA engine. In practice, if neither
are behind an IOMMU, things will likely work, but when that's not the
case, run into problems.

> +
> +	ret = vb2_queue_init(vb2q);
> +	if (ret) {
> +		dev_err(mali_c55->dev, "stats vb2 queue init failed\n");
> +		goto err_cleanup_entity;
> +	}
> +
> +	strscpy(stats->vdev.name, "mali-c55 3a stats", sizeof(stats->vdev.name));

s/3a //

> +	vdev->release = video_device_release_empty;

That's never right. You should refcount the data structures to ensure
proper lifetime management.

> +	vdev->fops = &mali_c55_stats_v4l2_fops;
> +	vdev->ioctl_ops = &mali_c55_stats_v4l2_ioctl_ops;
> +	vdev->lock = &stats->lock;
> +	vdev->v4l2_dev = &mali_c55->v4l2_dev;
> +	vdev->queue = &stats->queue;
> +	vdev->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING;
> +	vdev->vfl_dir = VFL_DIR_RX;
> +	video_set_drvdata(vdev, stats);
> +
> +	ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
> +	if (ret) {
> +		dev_err(mali_c55->dev,
> +			"failed to register stats video device\n");
> +		goto err_release_vb2q;
> +	}
> +
> +	stats->mali_c55 = mali_c55;
> +
> +	return 0;
> +
> +err_release_vb2q:
> +	vb2_queue_release(vb2q);
> +err_cleanup_entity:
> +	media_entity_cleanup(&stats->vdev.entity);
> +err_release_dma_channel:
> +	dma_release_channel(stats->channel);
> +err_destroy_mutex:
> +	mutex_destroy(&stats->lock);
> +
> +	return ret;
> +}

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 15/16] media: platform: Add mali-c55 parameters video node

[Laurent Pinchart]

On Fri, Jun 14, 2024 at 10:49:37PM +0100, Daniel Scally wrote:
> On 14/06/2024 22:11, Sakari Ailus wrote:
> > On Fri, Jun 14, 2024 at 09:15:07PM +0100, Dan Scally wrote:
> >>>> +void mali_c55_params_write_config(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	struct mali_c55_params *params = &mali_c55->params;
> >>>> +	enum vb2_buffer_state state = VB2_BUF_STATE_DONE;
> >>>> +	struct mali_c55_params_buffer *config;
> >>>> +	struct mali_c55_buffer *buf;
> >>>> +	size_t block_offset = 0;
> >>>> +
> >>>> +	spin_lock(&params->buffers.lock);
> >>>> +
> >>>> +	buf = list_first_entry_or_null(&params->buffers.queue,
> >>>> +				       struct mali_c55_buffer, queue);
> >>>> +	if (buf)
> >>>> +		list_del(&buf->queue);
> >>>> +	spin_unlock(&params->buffers.lock);
> >>>> +
> >>>> +	if (!buf)
> >>>> +		return;
> >>>> +
> >>>> +	buf->vb.sequence = mali_c55->isp.frame_sequence;
> >>>> +	config = vb2_plane_vaddr(&buf->vb.vb2_buf, 0);
> >>>> +
> >>>> +	if (config->total_size > MALI_C55_PARAMS_MAX_SIZE) {
> >>>> +		dev_dbg(mali_c55->dev, "Invalid parameters buffer size %lu\n",
> >>>> +			config->total_size);
> >>>> +		state = VB2_BUF_STATE_ERROR;
> >>>> +		goto err_buffer_done;
> >>>> +	}
> >>>> +
> >>>> +	/* Walk the list of parameter blocks and process them. */
> >>>> +	while (block_offset < config->total_size) {
> >>>> +		const struct mali_c55_block_handler *block_handler;
> >>>> +		struct mali_c55_params_block_header *block;
> >>>> +
> >>>> +		block = (struct mali_c55_params_block_header *)
> >>>> +			 &config->data[block_offset];
> >>>
> >>> How do you ensure config->data does hold a full struct
> >>> mali_c33_params_block_header at block_offset (i.e. that the struct does not
> >>> exceed the memory available for config->data)?
> >>
> >> We don't currently...the data buffer is sized specifically to be large
> >> enough to accept a single instance of each possible struct that could be
> >> included, we could keep track of the blocks that we have seen already and
> >> ensure that none are seen twice...and that should guarantee that the
> >> remaining space is sufficient to hold whatever the last block is. Does that
> >> sound ok?
> >
> > Ḯ'd add an explicit check here.
> 
> How would you do the check, sorry?

You could simply change the while() loop to

	max_offset = config->total_size - sizeof(mali_c55_params_block_header);
	while (block_offset <= max_offset) {

That would ensure that you always have enough space left for a header.
Within the loop, you will need to check that block->size doesn't go past
the end of the remaining space. Please also check the code for integer
overflows.

> > It's more simple way to ensure memory
> > safety here: relying on a complex machinery that can't be trivially
> > validated does risk having grave bugs, not only now but later on as well as
> > modifications to the code are done.
> >
> >>>> +
> >>>> +		if (block->type >= MALI_C55_PARAM_BLOCK_SENTINEL) {
> >>>> +			dev_dbg(mali_c55->dev, "Invalid parameters block type\n");
> >>>> +			state = VB2_BUF_STATE_ERROR;
> >>>> +			goto err_buffer_done;
> >>>> +		}
> >>>> +
> >>>> +		block_handler = &mali_c55_block_handlers[block->type];
> >>>> +		if (block->size != block_handler->size) {
> >>>
> >>> How do you ensure config->data has room for the block?
> >>
> >> I think through the same proposal as above.
> >
> > Similarly here. You already even have the size of the blocks available
> > here.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Dan Scally]

Morning Laurent

On 16/06/2024 20:39, Laurent Pinchart wrote:
> Hi Dan,
>
> On Fri, Jun 14, 2024 at 11:13:29AM +0100, Daniel Scally wrote:
>> On 30/05/2024 01:15, Laurent Pinchart wrote:
>>> On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
>>>> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
>>>> V4L2 and Media Controller compliant and creates subdevices to manage
>>>> the ISP itself, its internal test pattern generator as well as the
>>>> crop, scaler and output format functionality for each of its two
>>>> output devices.
>>>>
>>>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
>>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>>>> ---
>>>> Changes in v5:
>>>>
>>>> 	- Reworked input formats - previously we allowed representing input data
>>>> 	  as any 8-16 bit format. Now we only allow input data to be represented
>>>> 	  by the new 20-bit bayer formats, which is corrected to the equivalent
>>>> 	  16-bit format in RAW bypass mode.
>>>> 	- Stopped bypassing blocks that we haven't added supporting parameters
>>>> 	  for yet.
>>>> 	- Addressed most of Sakari's comments from the list
>>>>
>>>> Changes not yet made in v5:
>>>>
>>>> 	- The output pipelines can still be started and stopped independently of
>>>> 	  one another - I'd like to discuss that more.
>>>> 	- the TPG subdev still uses .s_stream() - I need to rebase onto a tree
>>>> 	  with working .enable_streams() for a single-source-pad subdevice.
>>>>
>>>> Changes in v4:
>>>>
>>>> 	- Reworked mali_c55_update_bits() to internally perform the bit-shift
>>> I really don't like that, it makes the code very confusing, even more so
>>> as it differs from regmap_update_bits().
>>>
>>> Look at this for instance:
>>>
>>> 	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>> 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
>>> 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
>>>
>>> It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
>>> BIT(0).
>>>
>>> Sorry, I know it will be painful, but this change needs to be reverted.
>> I'd like to argue for keeping this, on the grounds that it's better. I got lazy in the change you
>> reference there, and because BIT(0) is the same as 0x01 didn't bother changing it. I agree that
>> that's confusing but I think it would be better to keep the change and just update all the call
>> sites properly. The benefits as I see them are two:
>>
>>
>> 1. This method ensures sane consistent calling of the function. Without the internal shift you have
>> to shift the values at the call site, but there's no reason to do that if the value you're setting
>> is 0x00 or if the field you're targeting in the register starts at bit 0, so I think writing code
>> naturally we'd have a mix of situations like so:
>>
>>
>> #define REG_1 0xfc00
>>
>> #define REG_2 0xff
>>
>> mali_c55_update_bits(mali_c55, 0x1234, REG_1, 0x02 << 10);
>>
>> mali_c55_update_bits(mali_c55, 0x1234, REG_1, 0x00);
>>
>> mali_c55_update_bits(mali_c55, 0x1234, REG_2, 0x02);
>>
>> And I think that the mixture is more confusing than the difference with regmap_update_bits(). We
>> could include the bitshifting for consistencies sake despite it being unecessary, but it's extra
>> work for no real reason and itself "looks wrong" if the field starts at bit(0)...it would look less
>> wrong with an offset macro that defines the number of bits to shift as 0 but then we're on to
>> advantage #2...
>>
>> 2. It makes the driver far cleaner. Without it we either have magic numbers scattered throughout
>> (and sometimes have to calculate them with extra variables where the write can target different
>> places conditionally) or have macros defining the number of bits to shift, or have to do (ffs(mask)
>> - 1) everywhere, and that tends to make the call sites a lot messier - this was the original reason
>> I moved it internal actually.
>>
>> What do you think?
> All register values (possibly with the exception of 0) should have
> macros. The callers will thus not need to perform shifts manually, they
> will all be handled in the register fields macros. From a caller point
> of view, not handling the shifts inside mali_c55_update_bits() will not
> make a difference.
>
> It's the first time I notice a driver trying to shift internally in its
> update_bits function. I think that's really confusing, especially given
> that it departs from how regmap operates. I still strongly favour
> handling the shifts in the register macros.


Alright - I will handle it that way (in fact I already did). Thanks!

>
>>>> 	- Reworked the resizer to allow cropping during streaming
>>>> 	- Fixed a bug in NV12 output
>>>>
>>>> Changes in v3:
>>>>
>>>> 	- Mostly minor fixes suggested by Sakari
>>>> 	- Fixed the sequencing of vb2 buffers to be synchronised across the two
>>>> 	  capture devices.
>>>>
>>>> Changes in v2:
>>>>
>>>> 	- Clock handling
>>>> 	- Fixed the warnings raised by the kernel test robot
>>>>
>>>>    drivers/media/platform/Kconfig                |   1 +
>>>>    drivers/media/platform/Makefile               |   1 +
>>>>    drivers/media/platform/arm/Kconfig            |   5 +
>>>>    drivers/media/platform/arm/Makefile           |   2 +
>>>>    drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
>>>>    drivers/media/platform/arm/mali-c55/Makefile  |   9 +
>>>>    .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
>>>>    .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
>>>>    .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
>>>>    .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
>>>>    .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
>>>>    .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
>>>>    .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
>>>>    .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
>>>>    14 files changed, 4452 insertions(+)
>>>>    create mode 100644 drivers/media/platform/arm/Kconfig
>>>>    create mode 100644 drivers/media/platform/arm/Makefile
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> [snip]
>

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Sakari Ailus]

Hi Laurent,

On Sun, Jun 16, 2024 at 11:38:07PM +0300, Laurent Pinchart wrote:
> On Sun, Jun 09, 2024 at 06:21:52AM +0000, Sakari Ailus wrote:
> > On Thu, Jun 06, 2024 at 10:10:14PM +0300, Tomi Valkeinen wrote:
> > > On 06/06/2024 20:53, Laurent Pinchart wrote:
> > > > > > > > +			return -EINVAL;
> > > > > > > > +		}
> > > > > > > > +
> > > > > > > > +		active_sink = route->sink_pad;
> > > > > > > > +	}
> > > > > > > > +	if (active_sink == UINT_MAX) {
> > > > > > > > +		dev_err(rzr->mali_c55->dev, "One route has to be active");
> > > > > > > > +		return -EINVAL;
> > > > > > > > +	}
> > > > > >
> > > > > > The recommended handling of invalid routing is to adjust the routing
> > > > > > table, not to return errors.
> > > > >
> > > > > How should I adjust it ? The error here is due to the fact multiple
> > > > > routes are set as active, which one should I make active ? the first
> > > > > one ? Should I go and reset the flags in the subdev_route for the one
> > > > > that has to be made non-active ?
> > > >
> > > > The same way you would adjust an invalid format, you can pick the route
> > > > you consider should be the default.
> > > > 
> > > > I'd like Sakari's and Tomi's opinions on this, as it's a new API and the
> > > > behaviour is still a bit in flux.
> > > 
> > > Well... My opinion is that the driver adjusting the given config parameters
> > > (for any ioctl) is awful and should be deprecated. If the user asks for X,
> > > and the driver adjusts it and returns Y, then the user has two options:
> > > fail, because it didn't get X (after possibly laborious field by field
> > > checks), or shrug it's virtual shoulders and accept Y and hope that things
> > > still work even though it wanted X.
> > 
> > This is still often the only way to tell what the hardware can do as the
> > limitations in different cases (cropping and scaling for instance) can be
> > arbitrary. The other option is that the user space has to know the hardware
> > capabilities without them being available from the kernel.
> 
> For some parameters that make sense (we don't have a try mechanism for
> ISP parameters buffers for instance), but when it comes to configuring a
> pipeline, I think a parameters adjustment model is needed when we don't
> have means to expose constraints in a generic way to userspace. The
> question is in which category routing falls.
> 
> > There could be cases of IOCTLs where returning an error if what was
> > requested can't be performed exactly is workable in general, but then again
> > having consistency across IOCTL behaviour is very beneficial as well.
> > 
> > If you need something exactly, then I think you should check after the
> > IOCTL that this is what you also got, beyond the IOCTL succeeding.
> 
> I do agree with Tomi that this kind of check can be annoying for
> applications. In cases where checking the result would be complex, and
> where there is very little use case for receiving anything but the exact
> configuration you asked for, adjusting the parameters could increase the
> implementation complexity on both the kernel side and userspace side for
> no or very little benefit.
> 
> > > But maybe that was an answer to a question you didn't really ask =).
> > > 
> > > I think setting it to default routing in case of an error is as fine as any
> > > other "fix" for the routing. It won't work anyway.
> > > 
> > > But if the function sets default routing and returns 0 here, why would it
> > > return an error from v4l2_subdev_routing_validate()? Should it just set
> > > default routing in that case too? So should set_routing() ever return an
> > > error, if we can just set the default routing?
> 
> That's a good point. I asked myself the same question after sending my
> previous e-mail, and wondered if anyone else would notice too :-)
> 
> > S_ROUTING is a bit special as it deals with multiple routes and the user
> > space does have a way to add them incrementally.
> > 
> > Perhaps we should document better what the driver is expected to to correct
> > the routes?
> 
> We should document the expected behaviour clearly. After agreeing on the
> expected behaviour, that is.
> 
> > I'd think routes may be added by the driver (as some of them cannot be
> > disabled for instance) but if a requested route cannot be created, that
> > should probably be an error.
> > 
> > I've copied my current (with all the pending patches) documentation here
> > <URL:https://www.retiisi.eu/~sailus/v4l2/tmp/streams-doc/userspace-api/media/v4l/dev-subdev.html#streams-multiplexed-media-pads-and-internal-routing>.
> >
> > The text does not elaborate what exactly a driver could or should do, apart
> > from specifying the condition for EINVAL. I think we should specify this in
> 
> I don't see mentions of EINVAL related to streams there, am I missing
> something ?
> 
> > greater detail. My original thought wws the adjustment would be done by
> > adding static routes omitted by the caller, not trying to come up with e.g.
> > valid pad/stream pairs when user provided invalid ones.
> > 
> > Could this correction functionality be limited to returning static routes?
> 
> That would make userspace a tad simpler, and wouldn't be hard to do in
> the kernel, but I wonder if departing from the rule that invalid routing
> tables result in an error is worth it for such a small gain.

I'm just referring to our previous decision on the matter. :-)

Of course an application can do G_ROUTING, toggle the routes it needs to
and call S_ROUTING again, in order to be (fairly) certain it'll succeed.

Say, if an application wants to enable an embedded data route, then it'll
be required to supply the route for the image data as well, even if there's
no configuration that could be made for that route.

I'm thinking of fairly generic code here, if a device requires special
routing setup, it'll need the user space to be aware of it.

> 
> > > In the VIDIOC_SUBDEV_S_ROUTING doc we do list some cases where EINVAL or
> > > E2BIG is returned. But only a few, and I think
> > > v4l2_subdev_routing_validate() will return errors for many other cases too.
> > > 
> > > For what it's worth, the drivers I have written just return an error. It's
> > > simple for the driver and the user and works. If the consensus is that the
> > > drivers should instead set the default routing, or somehow mangle the given
> > > routing to an acceptable form, I can update those drivers accordingly.
> > > 
> > > But we probably need to update the docs too to be a bit more clear what
> > > VIDIOC_SUBDEV_S_ROUTING will do (although are the other ioctls any
> > > clearer?).
> > > 
> > > All that said, I think it's still a bit case-by-case. I don't think the
> > > drivers should always return an error if they get a routing table that's not
> > > 100% perfect. E.g. if a device supports two static routes, but the second
> > > one can be enabled or disabled, the driver should still accept a routing
> > > table from the user with only the first route present. Etc.
> > > 
> > > For the specific case in this patch... I'd prefer returning an error, or if
> > > that's not ok, set default routing.
> > 
> > Not modifying the routing table is another option as well but it may
> > require separating validating user-provided routes and applying the routes
> 
> I'm not sure to follow you here. By not modifying the routing table, do
> you mean returning an error ? Why would that require separation of
> validation and configuration ?

If a driver has already made changes to its routing table, it's a bad idea
to return an error to the user. In this case changes shouldn't be made.

> 
> > to the sub-device state. The default could be useful in principle, too, for
> > routing-unaware applications but they won't be calling S_ROUTING anyway.
> 

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Dan Scally]

Hi Laurent - sorry, should have included everything in the last reply rather than responding 
piecemeal. Some more responses and questions below

On 30/05/2024 01:15, Laurent Pinchart wrote:
> Hi Dan,
>
> Thank you for the patch.
>
> On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
>> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
>> V4L2 and Media Controller compliant and creates subdevices to manage
>> the ISP itself, its internal test pattern generator as well as the
>> crop, scaler and output format functionality for each of its two
>> output devices.
>>
>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>> ---
>> Changes in v5:
>>
>> 	- Reworked input formats - previously we allowed representing input data
>> 	  as any 8-16 bit format. Now we only allow input data to be represented
>> 	  by the new 20-bit bayer formats, which is corrected to the equivalent
>> 	  16-bit format in RAW bypass mode.
>> 	- Stopped bypassing blocks that we haven't added supporting parameters
>> 	  for yet.
>> 	- Addressed most of Sakari's comments from the list
>>
>> Changes not yet made in v5:
>>
>> 	- The output pipelines can still be started and stopped independently of
>> 	  one another - I'd like to discuss that more.
>> 	- the TPG subdev still uses .s_stream() - I need to rebase onto a tree
>> 	  with working .enable_streams() for a single-source-pad subdevice.
>>
>> Changes in v4:
>>
>> 	- Reworked mali_c55_update_bits() to internally perform the bit-shift
> I really don't like that, it makes the code very confusing, even more so
> as it differs from regmap_update_bits().
>
> Look at this for instance:
>
> 	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
>
> It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
> BIT(0).
>
> Sorry, I know it will be painful, but this change needs to be reverted.
>
>> 	- Reworked the resizer to allow cropping during streaming
>> 	- Fixed a bug in NV12 output
>>
>> Changes in v3:
>>
>> 	- Mostly minor fixes suggested by Sakari
>> 	- Fixed the sequencing of vb2 buffers to be synchronised across the two
>> 	  capture devices.
>>
>> Changes in v2:
>>
>> 	- Clock handling
>> 	- Fixed the warnings raised by the kernel test robot
>>
>>   drivers/media/platform/Kconfig                |   1 +
>>   drivers/media/platform/Makefile               |   1 +
>>   drivers/media/platform/arm/Kconfig            |   5 +
>>   drivers/media/platform/arm/Makefile           |   2 +
>>   drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
>>   drivers/media/platform/arm/mali-c55/Makefile  |   9 +
>>   .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
>>   .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
>>   .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
>>   .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
>>   .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
>>   .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
>>   .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
>>   .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
>>   14 files changed, 4452 insertions(+)
>>   create mode 100644 drivers/media/platform/arm/Kconfig
>>   create mode 100644 drivers/media/platform/arm/Makefile
>>   create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
>>   create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> I've skipped review of capture.c and resizer.c as I already have plenty
> of comments for the other files, and it's getting late. I'll try to
> review the rest tomorrow.
>
>> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
>> index 2d79bfc68c15..c929169766aa 100644
>> --- a/drivers/media/platform/Kconfig
>> +++ b/drivers/media/platform/Kconfig
>> @@ -65,6 +65,7 @@ config VIDEO_MUX
>>   source "drivers/media/platform/allegro-dvt/Kconfig"
>>   source "drivers/media/platform/amlogic/Kconfig"
>>   source "drivers/media/platform/amphion/Kconfig"
>> +source "drivers/media/platform/arm/Kconfig"
>>   source "drivers/media/platform/aspeed/Kconfig"
>>   source "drivers/media/platform/atmel/Kconfig"
>>   source "drivers/media/platform/broadcom/Kconfig"
>> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
>> index da17301f7439..9a647abd5218 100644
>> --- a/drivers/media/platform/Makefile
>> +++ b/drivers/media/platform/Makefile
>> @@ -8,6 +8,7 @@
>>   obj-y += allegro-dvt/
>>   obj-y += amlogic/
>>   obj-y += amphion/
>> +obj-y += arm/
>>   obj-y += aspeed/
>>   obj-y += atmel/
>>   obj-y += broadcom/
>> diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
>> new file mode 100644
>> index 000000000000..4f0764c329c7
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/Kconfig
>> @@ -0,0 +1,5 @@
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +
>> +comment "ARM media platform drivers"
>> +
>> +source "drivers/media/platform/arm/mali-c55/Kconfig"
>> diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
>> new file mode 100644
>> index 000000000000..8cc4918725ef
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/Makefile
>> @@ -0,0 +1,2 @@
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +obj-y += mali-c55/
>> diff --git a/drivers/media/platform/arm/mali-c55/Kconfig b/drivers/media/platform/arm/mali-c55/Kconfig
>> new file mode 100644
>> index 000000000000..602085e28b01
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/Kconfig
>> @@ -0,0 +1,18 @@
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +config VIDEO_MALI_C55
>> +	tristate "ARM Mali-C55 Image Signal Processor driver"
>> +	depends on V4L_PLATFORM_DRIVERS
>> +	depends on VIDEO_DEV && OF
>> +	depends on ARCH_VEXPRESS || COMPILE_TEST
>> +	select MEDIA_CONTROLLER
>> +	select VIDEO_V4L2_SUBDEV_API
>> +	select VIDEOBUF2_DMA_CONTIG
>> +	select VIDEOBUF2_VMALLOC
>> +	select V4L2_FWNODE
>> +	select GENERIC_PHY_MIPI_DPHY
> Alphabetical order ?
>
>> +	default n
> That's the default, you don't have to specify ti.
>
>> +	help
>> +	  Enable this to support Arm's Mali-C55 Image Signal Processor.
>> +
>> +	  To compile this driver as a module, choose M here: the module
>> +	  will be called mali-c55.
>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
>> new file mode 100644
>> index 000000000000..77dcb2fbf0f4
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
>> @@ -0,0 +1,9 @@
>> +# SPDX-License-Identifier: GPL-2.0
>> +
>> +mali-c55-y := mali-c55-capture.o \
>> +	      mali-c55-core.o \
>> +	      mali-c55-isp.o \
>> +	      mali-c55-tpg.o \
>> +	      mali-c55-resizer.o
> Alphabetical order here too.
>
>> +
>> +obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>> new file mode 100644
>> index 000000000000..1d539ac9c498
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>> @@ -0,0 +1,951 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ARM Mali-C55 ISP Driver - Video capture devices
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#include <linux/cleanup.h>
>> +#include <linux/minmax.h>
>> +#include <linux/pm_runtime.h>
>> +#include <linux/string.h>
>> +#include <linux/videodev2.h>
>> +
>> +#include <media/v4l2-dev.h>
>> +#include <media/v4l2-event.h>
>> +#include <media/v4l2-ioctl.h>
>> +#include <media/v4l2-subdev.h>
>> +#include <media/videobuf2-core.h>
>> +#include <media/videobuf2-dma-contig.h>
>> +
>> +#include "mali-c55-common.h"
>> +#include "mali-c55-registers.h"
>> +
>> +static const struct mali_c55_fmt mali_c55_fmts[] = {
>> +	/*
>> +	 * This table is missing some entries which need further work or
>> +	 * investigation:
>> +	 *
>> +	 * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
>> +	 * Base mode 5 is "Generic Data"
>> +	 * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
>> +	 * Base mode 9 seems to have no V4L2 equivalent
>> +	 * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
>> +	 * equivalent
>> +	 */
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_ARGB2101010,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_RGB121212_1X36,
>> +			MEDIA_BUS_FMT_RGB202020_1X60,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_A2R10G10B10,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_RGB565,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_RGB121212_1X36,
>> +			MEDIA_BUS_FMT_RGB202020_1X60,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_RGB565,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_BGR24,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_RGB121212_1X36,
>> +			MEDIA_BUS_FMT_RGB202020_1X60,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_RGB24,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_YUYV,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_YUV10_1X30,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_YUY2,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_UYVY,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_YUV10_1X30,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_UYVY,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_Y210,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_YUV10_1X30,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_Y210,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	/*
>> +	 * This is something of a hack, the ISP thinks it's running NV12M but
>> +	 * by setting uv_plane = 0 we simply discard that planes and only output
>> +	 * the Y-plane.
>> +	 */
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_GREY,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_YUV10_1X30,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_NV12M,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_YUV10_1X30,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_NV21M,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_YUV10_1X30,
>> +		},
>> +		.is_raw = false,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
>> +		}
>> +	},
>> +	/*
>> +	 * RAW uncompressed formats are all packed in 16 bpp.
>> +	 * TODO: Expand this list to encompass all possible RAW formats.
>> +	 */
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_SRGGB16,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_SRGGB16_1X16,
>> +		},
>> +		.is_raw = true,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_SBGGR16,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_SBGGR16_1X16,
>> +		},
>> +		.is_raw = true,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_SGBRG16,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_SGBRG16_1X16,
>> +		},
>> +		.is_raw = true,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +	{
>> +		.fourcc = V4L2_PIX_FMT_SGRBG16,
>> +		.mbus_codes = {
>> +			MEDIA_BUS_FMT_SGRBG16_1X16,
>> +		},
>> +		.is_raw = true,
>> +		.registers = {
>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>> +		}
>> +	},
>> +};
>> +
>> +static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
>> +					       u32 code)
>> +{
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
>> +		if (fmt->mbus_codes[i] == code)
>> +			return true;
>> +	}
>> +
>> +	return false;
>> +}
>> +
>> +bool mali_c55_format_is_raw(unsigned int mbus_code)
>> +{
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>> +		if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
>> +			return mali_c55_fmts[i].is_raw;
>> +	}
>> +
>> +	return false;
>> +}
>> +
>> +static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
>> +{
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>> +		if (mali_c55_fmts[i].fourcc == pixelformat)
>> +			return &mali_c55_fmts[i];
>> +	}
>> +
>> +	/*
>> +	 * If we find no matching pixelformat, we'll just default to the first
>> +	 * one for now.
>> +	 */
>> +
>> +	return &mali_c55_fmts[0];
>> +}
>> +
>> +static const char * const capture_device_names[] = {
>> +	"mali-c55 fr",
>> +	"mali-c55 ds",
>> +	"mali-c55 3a stats",
>> +	"mali-c55 params",
>> +};
>> +
>> +static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
>> +{
>> +	if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
>> +		return capture_device_names[0];
>> +
>> +	if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
>> +		return capture_device_names[1];
>> +
>> +	return "params/stat not supported yet";
>> +}
>> +
>> +static int mali_c55_link_validate(struct media_link *link)
>> +{
>> +	struct video_device *vdev =
>> +		media_entity_to_video_device(link->sink->entity);
>> +	struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
>> +	struct v4l2_subdev *sd =
>> +		media_entity_to_v4l2_subdev(link->source->entity);
>> +	const struct v4l2_pix_format_mplane *pix_mp;
>> +	const struct mali_c55_fmt *cap_fmt;
>> +	struct v4l2_subdev_format sd_fmt = {
>> +		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
>> +		.pad = link->source->index,
>> +	};
>> +	int ret;
>> +
>> +	ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
>> +	if (ret)
>> +		return ret;
>> +
>> +	pix_mp = &cap_dev->mode.pix_mp;
>> +	cap_fmt = cap_dev->mode.capture_fmt;
>> +
>> +	if (sd_fmt.format.width != pix_mp->width ||
>> +	    sd_fmt.format.height != pix_mp->height) {
>> +		dev_dbg(cap_dev->mali_c55->dev,
>> +			"link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
>> +			link->source->entity->name, link->source->index,
>> +			link->sink->entity->name, link->sink->index,
>> +			sd_fmt.format.width, sd_fmt.format.height,
>> +			pix_mp->width, pix_mp->height);
>> +		return -EPIPE;
>> +	}
>> +
>> +	if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
>> +		dev_dbg(cap_dev->mali_c55->dev,
>> +			"link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format %p4cc\n",
>> +			link->source->entity->name, link->source->index,
>> +			link->sink->entity->name, link->sink->index,
>> +			sd_fmt.format.code, &pix_mp->pixelformat);
>> +		return -EPIPE;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct media_entity_operations mali_c55_media_ops = {
>> +	.link_validate = mali_c55_link_validate,
>> +};
>> +
>> +static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
>> +				    unsigned int *num_planes, unsigned int sizes[],
>> +				    struct device *alloc_devs[])
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>> +	unsigned int i;
>> +
>> +	if (*num_planes) {
>> +		if (*num_planes != cap_dev->mode.pix_mp.num_planes)
>> +			return -EINVAL;
>> +
>> +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>> +			if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
>> +				return -EINVAL;
>> +	} else {
>> +		*num_planes = cap_dev->mode.pix_mp.num_planes;
>> +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>> +			sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static void mali_c55_buf_queue(struct vb2_buffer *vb)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
>> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>> +	struct mali_c55_buffer *buf = container_of(vbuf,
>> +						   struct mali_c55_buffer, vb);
>> +	unsigned int i;
>> +
>> +	buf->plane_done[MALI_C55_PLANE_Y] = false;
>> +
>> +	/*
>> +	 * If we're in a single-plane format we flag the other plane as done
>> +	 * already so it's dequeued appropriately later
>> +	 */
>> +	buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
>> +
>> +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
>> +		unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
>> +
>> +		vb2_set_plane_payload(vb, i, size);
>> +	}
>> +
>> +	spin_lock(&cap_dev->buffers.lock);
>> +	list_add_tail(&buf->queue, &cap_dev->buffers.queue);
>> +	spin_unlock(&cap_dev->buffers.lock);
>> +}
>> +
>> +static int mali_c55_buf_init(struct vb2_buffer *vb)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
>> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>> +	struct mali_c55_buffer *buf = container_of(vbuf,
>> +						   struct mali_c55_buffer, vb);
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>> +		buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
>> +
>> +	return 0;
>> +}
>> +
>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
>> +{
>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>> +
>> +	guard(spinlock)(&cap_dev->buffers.lock);
>> +
>> +	cap_dev->buffers.curr = cap_dev->buffers.next;
>> +	cap_dev->buffers.next = NULL;
>> +
>> +	if (!list_empty(&cap_dev->buffers.queue)) {
>> +		struct v4l2_pix_format_mplane *pix_mp;
>> +		const struct v4l2_format_info *info;
>> +		u32 *addrs;
>> +
>> +		pix_mp = &cap_dev->mode.pix_mp;
>> +		info = v4l2_format_info(pix_mp->pixelformat);
>> +
>> +		mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
>> +		if (cap_dev->mode.capture_fmt->registers.uv_plane)
>> +			mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
>> +
>> +		cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
>> +							 struct mali_c55_buffer,
>> +							 queue);
>> +		list_del(&cap_dev->buffers.next->queue);
>> +
>> +		addrs = cap_dev->buffers.next->addrs;
>> +		mali_c55_write(mali_c55,
>> +			MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
>> +			addrs[MALI_C55_PLANE_Y]);
>> +		mali_c55_write(mali_c55,
>> +			MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
>> +			addrs[MALI_C55_PLANE_UV]);
>> +		mali_c55_write(mali_c55,
>> +			MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
>> +			pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
>> +		mali_c55_write(mali_c55,
>> +			MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
>> +			pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
>> +			/ info->hdiv);
>> +	} else {
>> +		/*
>> +		 * If we underflow then we can tell the ISP that we don't want
>> +		 * to write out the next frame.
>> +		 */
>> +		mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>> +		mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>> +	}
>> +}
>> +
>> +static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
>> +				   unsigned int framecount)
>> +{
>> +	curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>> +	curr_buf->vb.field = V4L2_FIELD_NONE;
>> +	curr_buf->vb.sequence = framecount;
>> +	vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
>> +}
>> +
>> +/**
>> + * mali_c55_set_plane_done - mark the plane as written and process the buffer if
>> + *			     both planes are finished.
>> + * @cap_dev:  pointer to the fr or ds pipe output
>> + * @plane:    the plane to mark as completed
>> + *
>> + * The Mali C55 ISP has muliplanar outputs for some formats that come with two
>> + * separate "buffer write completed" interrupts - we need to flag each plane's
>> + * completion and check whether both planes are done - if so, complete the buf
>> + * in vb2.
>> + */
>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>> +			     enum mali_c55_planes plane)
>> +{
>> +	struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
>> +	struct mali_c55_buffer *curr_buf;
>> +
>> +	guard(spinlock)(&cap_dev->buffers.lock);
>> +	curr_buf = cap_dev->buffers.curr;
>> +
>> +	/*
>> +	 * This _should_ never happen. If no buffer was available from vb2 then
>> +	 * we tell the ISP not to bother writing the next frame, which means the
>> +	 * interrupts that call this function should never trigger. If it does
>> +	 * happen then one of our assumptions is horribly wrong - complain
>> +	 * loudly and do nothing.
>> +	 */
>> +	if (!curr_buf) {
>> +		dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
>> +			mali_c55_cap_dev_to_name(cap_dev), __func__);
>> +		return;
>> +	}
>> +
>> +	/* If the other plane is also done... */
>> +	if (curr_buf->plane_done[~plane & 1]) {
>> +		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
>> +		cap_dev->buffers.curr = NULL;
>> +		isp->frame_sequence++;
>> +	} else {
>> +		curr_buf->plane_done[plane] = true;
>> +	}
>> +}
>> +
>> +static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
>> +{
>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>> +
>> +	mali_c55_update_bits(mali_c55,
>> +			     MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>> +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>> +	mali_c55_update_bits(mali_c55,
>> +			     MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>> +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>> +}
>> +
>> +static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
>> +{
>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>> +
>> +	/*
>> +	 * The Mali ISP can hold up to 5 buffer addresses and simply cycle
>> +	 * through them, but it's not clear to me that the vb2 queue _guarantees_
>> +	 * it will queue buffers to the driver in a fixed order, and ensuring
>> +	 * we call vb2_buffer_done() for the right buffer seems to me to add
>> +	 * pointless complexity given in multi-context mode we'd need to
>> +	 * re-write those registers every frame anyway...so we tell the ISP to
>> +	 * use a single register and update it for each frame.
>> +	 */
>> +	mali_c55_update_bits(mali_c55,
>> +			MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
>> +			MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
>> +	mali_c55_update_bits(mali_c55,
>> +			MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
>> +			MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
>> +
>> +	/*
>> +	 * We only queue a buffer in the streamon path if this is the first of
>> +	 * the capture devices to start streaming. If the ISP is already running
>> +	 * then we rely on the ISP_START interrupt to queue the first buffer for
>> +	 * this capture device.
>> +	 */
>> +	if (mali_c55->pipe.start_count == 1)
>> +		mali_c55_set_next_buffer(cap_dev);
>> +}
>> +
>> +static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
>> +					    enum vb2_buffer_state state)
>> +{
>> +	struct mali_c55_buffer *buf, *tmp;
>> +
>> +	guard(spinlock)(&cap_dev->buffers.lock);
>> +
>> +	if (cap_dev->buffers.curr) {
>> +		vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
>> +				state);
>> +		cap_dev->buffers.curr = NULL;
>> +	}
>> +
>> +	if (cap_dev->buffers.next) {
>> +		vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
>> +				state);
>> +		cap_dev->buffers.next = NULL;
>> +	}
>> +
>> +	list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
>> +		list_del(&buf->queue);
>> +		vb2_buffer_done(&buf->vb.vb2_buf, state);
>> +	}
>> +}
>> +
>> +static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>> +	struct mali_c55_resizer *rzr = cap_dev->rzr;
>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>> +	int ret;
>> +
>> +	guard(mutex)(&isp->lock);
>> +
>> +	ret = pm_runtime_resume_and_get(mali_c55->dev);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = video_device_pipeline_start(&cap_dev->vdev,
>> +					  &cap_dev->mali_c55->pipe);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
>> +			mali_c55_cap_dev_to_name(cap_dev));
>> +		goto err_pm_put;
>> +	}
>> +
>> +	mali_c55_cap_dev_stream_enable(cap_dev);
>> +	mali_c55_rzr_start_stream(rzr);
>> +
>> +	/*
>> +	 * We only start the ISP if we're the only capture device that's
>> +	 * streaming. Otherwise, it'll already be active.
>> +	 */
>> +	if (mali_c55->pipe.start_count == 1) {
>> +		ret = mali_c55_isp_start_stream(isp);
>> +		if (ret)
>> +			goto err_disable_cap_dev;
>> +	}
>> +
>> +	return 0;
>> +
>> +err_disable_cap_dev:
>> +	mali_c55_cap_dev_stream_disable(cap_dev);
>> +	video_device_pipeline_stop(&cap_dev->vdev);
>> +err_pm_put:
>> +	pm_runtime_put(mali_c55->dev);
>> +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
>> +
>> +	return ret;
>> +}
>> +
>> +static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>> +	struct mali_c55_resizer *rzr = cap_dev->rzr;
>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>> +
>> +	guard(mutex)(&isp->lock);
>> +
>> +	/*
>> +	 * If one of the other capture nodes is streaming, we shouldn't
>> +	 * disable the ISP here.
>> +	 */
>> +	if (mali_c55->pipe.start_count == 1)
>> +		mali_c55_isp_stop_stream(&mali_c55->isp);
>> +
>> +	mali_c55_rzr_stop_stream(rzr);
>> +	mali_c55_cap_dev_stream_disable(cap_dev);
>> +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
>> +	video_device_pipeline_stop(&cap_dev->vdev);
>> +	pm_runtime_put(mali_c55->dev);
>> +}
>> +
>> +static const struct vb2_ops mali_c55_vb2_ops = {
>> +	.queue_setup		= &mali_c55_vb2_queue_setup,
>> +	.buf_queue		= &mali_c55_buf_queue,
>> +	.buf_init		= &mali_c55_buf_init,
>> +	.wait_prepare		= vb2_ops_wait_prepare,
>> +	.wait_finish		= vb2_ops_wait_finish,
>> +	.start_streaming	= &mali_c55_vb2_start_streaming,
>> +	.stop_streaming		= &mali_c55_vb2_stop_streaming,
>> +};
>> +
>> +static const struct v4l2_file_operations mali_c55_v4l2_fops = {
>> +	.owner = THIS_MODULE,
>> +	.unlocked_ioctl = video_ioctl2,
>> +	.open = v4l2_fh_open,
>> +	.release = vb2_fop_release,
>> +	.poll = vb2_fop_poll,
>> +	.mmap = vb2_fop_mmap,
>> +};
>> +
>> +static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
>> +{
>> +	const struct mali_c55_fmt *capture_format;
>> +	const struct v4l2_format_info *info;
>> +	struct v4l2_plane_pix_format *plane;
>> +	unsigned int i;
>> +
>> +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
>> +	pix_mp->pixelformat = capture_format->fourcc;
>> +
>> +	pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
>> +			      MALI_C55_MAX_WIDTH);
>> +	pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
>> +			       MALI_C55_MAX_HEIGHT);
>> +
>> +	pix_mp->field = V4L2_FIELD_NONE;
>> +	pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
>> +	pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>> +	pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
>> +
>> +	info = v4l2_format_info(pix_mp->pixelformat);
>> +	pix_mp->num_planes = info->mem_planes;
>> +	memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
>> +
>> +	pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;
>> +	pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
>> +				       * pix_mp->height;
>> +
>> +	for (i = 1; i < info->comp_planes; i++) {
>> +		plane = &pix_mp->plane_fmt[i];
>> +
>> +		plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
>> +						   info->hdiv);
>> +		plane->sizeimage = DIV_ROUND_UP(
>> +					plane->bytesperline * pix_mp->height,
>> +					info->vdiv);
>> +	}
>> +
>> +	if (info->mem_planes == 1) {
>> +		for (i = 1; i < info->comp_planes; i++) {
>> +			plane = &pix_mp->plane_fmt[i];
>> +			pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
>> +		}
>> +	}
>> +}
>> +
>> +static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
>> +					   struct v4l2_format *f)
>> +{
>> +	mali_c55_try_fmt(&f->fmt.pix_mp);
>> +
>> +	return 0;
>> +}
>> +
>> +static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
>> +				struct v4l2_pix_format_mplane *pix_mp)
>> +{
>> +	const struct mali_c55_fmt *capture_format;
>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>> +	const struct v4l2_format_info *info;
>> +
>> +	mali_c55_try_fmt(pix_mp);
>> +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
>> +	info = v4l2_format_info(pix_mp->pixelformat);
>> +	if (WARN_ON(!info))
>> +		return;
>> +
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>> +		       capture_format->registers.base_mode);
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
>> +		       MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
>> +		       MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
>> +
>> +	if (info->mem_planes > 1) {
>> +		mali_c55_write(mali_c55,
>> +			       MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>> +			       capture_format->registers.base_mode);
>> +		mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>> +				MALI_C55_WRITER_SUBMODE_MASK,
>> +				capture_format->registers.uv_plane);
>> +
>> +		mali_c55_write(mali_c55,
>> +			MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
>> +			MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
>> +			MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
>> +	}
>> +
>> +	if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
>> +		/*
>> +		 * TODO: Figure out the colour matrix coefficients and calculate
>> +		 * and write them here.
>> +		 */
>> +
>> +		mali_c55_write(mali_c55,
>> +			       MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>> +			       MALI_C55_CS_CONV_MATRIX_MASK);
>> +
>> +		if (info->hdiv > 1)
>> +			mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>> +				MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
>> +		if (info->vdiv > 1)
>> +			mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>> +				MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
>> +		if (info->hdiv > 1 || info->vdiv > 1)
>> +			mali_c55_update_bits(mali_c55,
>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>> +				MALI_C55_CS_CONV_FILTER_MASK, 0x01);
>> +	}
>> +
>> +	cap_dev->mode.pix_mp = *pix_mp;
>> +	cap_dev->mode.capture_fmt = capture_format;
>> +}
>> +
>> +static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
>> +					 struct v4l2_format *f)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>> +
>> +	if (vb2_is_busy(&cap_dev->queue))
>> +		return -EBUSY;
>> +
>> +	mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
>> +					 struct v4l2_format *f)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>> +
>> +	f->fmt.pix_mp = cap_dev->mode.pix_mp;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
>> +					    struct v4l2_fmtdesc *f)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>> +	unsigned int j = 0;
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>> +		if (f->mbus_code &&
>> +		    !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
>> +						       f->mbus_code))
>> +			continue;
>> +
>> +		/* Downscale pipe can't output RAW formats */
>> +		if (mali_c55_fmts[i].is_raw &&
>> +		    cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
>> +			continue;
>> +
>> +		if (j++ == f->index) {
>> +			f->pixelformat = mali_c55_fmts[i].fourcc;
>> +			return 0;
>> +		}
>> +	}
>> +
>> +	return -EINVAL;
>> +}
>> +
>> +static int mali_c55_querycap(struct file *file, void *fh,
>> +			     struct v4l2_capability *cap)
>> +{
>> +	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
>> +	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
>> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
>> +	.vidioc_querybuf = vb2_ioctl_querybuf,
>> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
>> +	.vidioc_qbuf = vb2_ioctl_qbuf,
>> +	.vidioc_expbuf = vb2_ioctl_expbuf,
>> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
>> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
>> +	.vidioc_streamon = vb2_ioctl_streamon,
>> +	.vidioc_streamoff = vb2_ioctl_streamoff,
>> +	.vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
>> +	.vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
>> +	.vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
>> +	.vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
>> +	.vidioc_querycap = mali_c55_querycap,
>> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
>> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
>> +};
>> +
>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
>> +{
>> +	struct v4l2_pix_format_mplane pix_mp;
>> +	struct mali_c55_cap_dev *cap_dev;
>> +	struct video_device *vdev;
>> +	struct vb2_queue *vb2q;
>> +	unsigned int i;
>> +	int ret;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
>> +		cap_dev = &mali_c55->cap_devs[i];
>> +		vdev = &cap_dev->vdev;
>> +		vb2q = &cap_dev->queue;
>> +
>> +		/*
>> +		 * The downscale output pipe is an optional block within the ISP
>> +		 * so we need to check whether it's actually been fitted or not.
>> +		 */
>> +
>> +		if (i == MALI_C55_CAP_DEV_DS &&
>> +		    !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
>> +			continue;
>> +
>> +		cap_dev->mali_c55 = mali_c55;
>> +		mutex_init(&cap_dev->lock);
>> +		INIT_LIST_HEAD(&cap_dev->buffers.queue);
>> +
>> +		switch (i) {
>> +		case MALI_C55_CAP_DEV_FR:
>> +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
>> +			cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
>> +			break;
>> +		case MALI_C55_CAP_DEV_DS:
>> +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
>> +			cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
>> +			break;
>> +		default:
>> +			mutex_destroy(&cap_dev->lock);
>> +			ret = -EINVAL;
>> +			goto err_destroy_mutex;
>> +		}
>> +
>> +		cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
>> +		ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
>> +		if (ret) {
>> +			mutex_destroy(&cap_dev->lock);
>> +			goto err_destroy_mutex;
>> +		}
>> +
>> +		vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>> +		vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
>> +		vb2q->drv_priv = cap_dev;
>> +		vb2q->mem_ops = &vb2_dma_contig_memops;
>> +		vb2q->ops = &mali_c55_vb2_ops;
>> +		vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
>> +		vb2q->min_queued_buffers = 1;
>> +		vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
>> +		vb2q->lock = &cap_dev->lock;
>> +		vb2q->dev = mali_c55->dev;
>> +
>> +		ret = vb2_queue_init(vb2q);
>> +		if (ret) {
>> +			dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
>> +				mali_c55_cap_dev_to_name(cap_dev));
>> +			goto err_cleanup_media_entity;
>> +		}
>> +
>> +		strscpy(cap_dev->vdev.name, capture_device_names[i],
>> +			sizeof(cap_dev->vdev.name));
>> +		vdev->release = video_device_release_empty;
>> +		vdev->fops = &mali_c55_v4l2_fops;
>> +		vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
>> +		vdev->lock = &cap_dev->lock;
>> +		vdev->v4l2_dev = &mali_c55->v4l2_dev;
>> +		vdev->queue = &cap_dev->queue;
>> +		vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +				    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
>> +		vdev->entity.ops = &mali_c55_media_ops;
>> +		video_set_drvdata(vdev, cap_dev);
>> +
>> +		memset(&pix_mp, 0, sizeof(pix_mp));
>> +		pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
>> +		pix_mp.width = MALI_C55_DEFAULT_WIDTH;
>> +		pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
>> +		mali_c55_set_format(cap_dev, &pix_mp);
>> +
>> +		ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
>> +		if (ret) {
>> +			dev_err(mali_c55->dev,
>> +				"%s failed to register video device\n",
>> +				mali_c55_cap_dev_to_name(cap_dev));
>> +			goto err_release_vb2q;
>> +		}
>> +	}
>> +
>> +	return 0;
>> +
>> +err_release_vb2q:
>> +	vb2_queue_release(vb2q);
>> +err_cleanup_media_entity:
>> +	media_entity_cleanup(&cap_dev->vdev.entity);
>> +err_destroy_mutex:
>> +	mutex_destroy(&cap_dev->lock);
>> +	mali_c55_unregister_capture_devs(mali_c55);
>> +
>> +	return ret;
>> +}
>> +
>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_cap_dev *cap_dev;
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
>> +		cap_dev = &mali_c55->cap_devs[i];
>> +
>> +		if (!video_is_registered(&cap_dev->vdev))
>> +			continue;
>> +
>> +		vb2_video_unregister_device(&cap_dev->vdev);
>> +		media_entity_cleanup(&cap_dev->vdev.entity);
>> +		mutex_destroy(&cap_dev->lock);
>> +	}
>> +}
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>> new file mode 100644
>> index 000000000000..2d0c4d152beb
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>> @@ -0,0 +1,266 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * ARM Mali-C55 ISP Driver - Common definitions
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#ifndef _MALI_C55_COMMON_H
>> +#define _MALI_C55_COMMON_H
>> +
>> +#include <linux/clk.h>
>> +#include <linux/io.h>
>> +#include <linux/list.h>
>> +#include <linux/mutex.h>
>> +#include <linux/scatterlist.h>
> I don't think this is needed. You're however missing spinlock.h.
>
>> +#include <linux/videodev2.h>
>> +
>> +#include <media/media-device.h>
>> +#include <media/v4l2-async.h>
>> +#include <media/v4l2-ctrls.h>
>> +#include <media/v4l2-dev.h>
>> +#include <media/v4l2-device.h>
>> +#include <media/v4l2-subdev.h>
>> +#include <media/videobuf2-core.h>
>> +#include <media/videobuf2-v4l2.h>
>> +
>> +#define MALI_C55_DRIVER_NAME		"mali-c55"
>> +
>> +/* min and max values for the image sizes */
>> +#define MALI_C55_MIN_WIDTH		640U
>> +#define MALI_C55_MIN_HEIGHT		480U
>> +#define MALI_C55_MAX_WIDTH		8192U
>> +#define MALI_C55_MAX_HEIGHT		8192U
>> +#define MALI_C55_DEFAULT_WIDTH		1920U
>> +#define MALI_C55_DEFAULT_HEIGHT		1080U
>> +
>> +#define MALI_C55_DEFAULT_MEDIA_BUS_FMT	MEDIA_BUS_FMT_RGB121212_1X36
>> +
>> +struct mali_c55;
>> +struct mali_c55_cap_dev;
>> +struct platform_device;
> You should also forward-declare
>
> struct device;
> struct dma_chan;
> struct resource;
>
>> +
>> +static const char * const mali_c55_clk_names[] = {
>> +	"aclk",
>> +	"hclk",
>> +};
> This will end up duplicating the array in each compilation unit, not
> great. Move it to mali-c55-core.c. You use it in this file just for its
> size, replace that with a macro that defines the size, or allocate
> mali_c55.clks dynamically with devm_kcalloc().
>
>> +
>> +enum mali_c55_interrupts {
>> +	MALI_C55_IRQ_ISP_START,
>> +	MALI_C55_IRQ_ISP_DONE,
>> +	MALI_C55_IRQ_MCM_ERROR,
>> +	MALI_C55_IRQ_BROKEN_FRAME_ERROR,
>> +	MALI_C55_IRQ_MET_AF_DONE,
>> +	MALI_C55_IRQ_MET_AEXP_DONE,
>> +	MALI_C55_IRQ_MET_AWB_DONE,
>> +	MALI_C55_IRQ_AEXP_1024_DONE,
>> +	MALI_C55_IRQ_IRIDIX_MET_DONE,
>> +	MALI_C55_IRQ_LUT_INIT_DONE,
>> +	MALI_C55_IRQ_FR_Y_DONE,
>> +	MALI_C55_IRQ_FR_UV_DONE,
>> +	MALI_C55_IRQ_DS_Y_DONE,
>> +	MALI_C55_IRQ_DS_UV_DONE,
>> +	MALI_C55_IRQ_LINEARIZATION_DONE,
>> +	MALI_C55_IRQ_RAW_FRONTEND_DONE,
>> +	MALI_C55_IRQ_NOISE_REDUCTION_DONE,
>> +	MALI_C55_IRQ_IRIDIX_DONE,
>> +	MALI_C55_IRQ_BAYER2RGB_DONE,
>> +	MALI_C55_IRQ_WATCHDOG_TIMER,
>> +	MALI_C55_IRQ_FRAME_COLLISION,
>> +	MALI_C55_IRQ_UNUSED,
>> +	MALI_C55_IRQ_DMA_ERROR,
>> +	MALI_C55_IRQ_INPUT_STOPPED,
>> +	MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
>> +	MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
>> +	MALI_C55_NUM_IRQ_BITS
> Those are register bits, I think they belong to mali-c55-registers.h,
> and should probably be macros instead of an enum.
>
>> +};
>> +
>> +enum mali_c55_isp_pads {
>> +	MALI_C55_ISP_PAD_SINK_VIDEO,
> As there's a single sink pad, maybe MALI_C55_ISP_PAD_SINK ? Ah, you're
> probably preparing for ISP parameters support. It's fine.
>
>> +	MALI_C55_ISP_PAD_SOURCE,
> Then maybe this should be named MALI_C55_ISP_PAD_SOURCE_VIDEO as I
> assume there will be a stats source pad.
>
>> +	MALI_C55_ISP_PAD_SOURCE_BYPASS,
>> +	MALI_C55_ISP_NUM_PADS,
>> +};
>> +
>> +struct mali_c55_tpg {
>> +	struct mali_c55 *mali_c55;
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pad;
>> +	struct mutex lock;
>> +	struct mali_c55_tpg_ctrls {
>> +		struct v4l2_ctrl_handler handler;
>> +		struct v4l2_ctrl *test_pattern;
> Set but never used. You can drop it.
>
>> +		struct v4l2_ctrl *hblank;
> Set and used only once, in the same function. You can make it a local
> variable.
>
>> +		struct v4l2_ctrl *vblank;
>> +	} ctrls;
>> +};
> I wonder if this file should be split, with mali-c55-capture.h,
> mali-c55-core.h, mali-c55-isp.h, ... I think it could increase
> readability by clearly separating the different elements. Up to you.
>
>> +
>> +struct mali_c55_isp {
>> +	struct mali_c55 *mali_c55;
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pads[MALI_C55_ISP_NUM_PADS];
>> +	struct media_pad *remote_src;
>> +	struct v4l2_async_notifier notifier;
> I'm tempted to move the notifier to mali_c55, as it's related to
> components external to the whole ISP, not to the ISP subdev itself.
> Could you give it a try, to see if it could be done without any drawback
> ?


This seems to work fine.

>
>> +	struct mutex lock;
> Locks require a comment to explain what they protect. Same below where
> applicable (for both mutexes and spinlocks).
>
>> +	unsigned int frame_sequence;
>> +};
>> +
>> +enum mali_c55_resizer_ids {
>> +	MALI_C55_RZR_FR,
>> +	MALI_C55_RZR_DS,
>> +	MALI_C55_NUM_RZRS,
> The usual abbreviation for "resizer" in drivers/media/ is "rsz", not
> "rzr". I would have said we can leave it as-is as changing it would be a
> bit annoying, but I then realized that "rzr" is not just unusual, it's
> actually not used at all. Would you mind applying a sed globally ? :-)
>
>> +};
>> +
>> +enum mali_c55_rzr_pads {
> Same enums/structs use abbreviations, some don't. Consistency would
> help.
>
>> +	MALI_C55_RZR_SINK_PAD,
>> +	MALI_C55_RZR_SOURCE_PAD,
>> +	MALI_C55_RZR_SINK_BYPASS_PAD,
>> +	MALI_C55_RZR_NUM_PADS
>> +};
>> +
>> +struct mali_c55_resizer {
>> +	struct mali_c55 *mali_c55;
>> +	struct mali_c55_cap_dev *cap_dev;
>> +	enum mali_c55_resizer_ids id;
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pads[MALI_C55_RZR_NUM_PADS];
>> +	unsigned int num_routes;
>> +	bool streaming;
>> +};
>> +
>> +enum mali_c55_cap_devs {
>> +	MALI_C55_CAP_DEV_FR,
>> +	MALI_C55_CAP_DEV_DS,
>> +	MALI_C55_NUM_CAP_DEVS
>> +};
>> +
>> +struct mali_c55_fmt {
> mali_c55_format_info would be a better name I think, as this stores
> format information, not formats.
>
>> +	u32 fourcc;
>> +	unsigned int mbus_codes[2];
> A comment to explain why we have two media bus codes would be useful.
> You can document the whole structure if desired :-)
>
>> +	bool is_raw;
>> +	struct mali_c55_fmt_registers {
> Make it an anonymous structure, it's never used anywhere else.
>
>> +		unsigned int base_mode;
>> +		unsigned int uv_plane;
> If those are register field values, use u32 instead of unsigned int.
>
>> +	} registers;
> It's funny, we tend to abbreviate different things, I would have used
> "regs" here but written "format" in full in the structure name :-)
>
>> +};
>> +
>> +enum mali_c55_isp_bayer_order {
>> +	MALI_C55_BAYER_ORDER_RGGB,
>> +	MALI_C55_BAYER_ORDER_GRBG,
>> +	MALI_C55_BAYER_ORDER_GBRG,
>> +	MALI_C55_BAYER_ORDER_BGGR
> These are registers values too, they belong to mali-c55-registers.h.
>
>> +};
>> +
>> +struct mali_c55_isp_fmt {
> mali_c55_isp_format_info
>
>> +	u32 code;
>> +	enum v4l2_pixel_encoding encoding;
> Here you use v4l2_pixel_encoding, for mali_c55_fmt you use is_raw. Maybe
> pick the same option for both structures ?
>
>> +	enum mali_c55_isp_bayer_order order;
>> +};
>> +
>> +enum mali_c55_planes {
>> +	MALI_C55_PLANE_Y,
>> +	MALI_C55_PLANE_UV,
>> +	MALI_C55_NUM_PLANES
>> +};
>> +
>> +struct mali_c55_buffer {
>> +	struct vb2_v4l2_buffer vb;
>> +	bool plane_done[MALI_C55_NUM_PLANES];
> I think tracking the pending state would simplify the logic in
> mali_c55_set_plane_done(), which would become
>
> 	curr_buf->plane_pending[plane] = false;
>
> 	if (!curr_buf->plane_pending[0] && !curr_buf->plane_pending[1]) {
> 		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> 		cap_dev->buffers.curr = NULL;
> 		isp->frame_sequence++;
> 	}
>
> Or a counter may be even easier (and would consume less memory).
I'll do the counter; a  similar function in the stats code does so already.
>
>> +	struct list_head queue;
>> +	u32 addrs[MALI_C55_NUM_PLANES];
> This stores DMA addresses, use dma_addr_t.
>
>> +};
>> +
>> +struct mali_c55_cap_dev {
>> +	struct mali_c55 *mali_c55;
>> +	struct mali_c55_resizer *rzr;
>> +	struct video_device vdev;
>> +	struct media_pad pad;
>> +	struct vb2_queue queue;
>> +	struct mutex lock;
>> +	unsigned int reg_offset;
> Manual handling of the offset everywhere, with parametric macros for the
> resizer register addresses, isn't very nice. Introduce resizer-specific
> accessors and capture-specific accessors (mali_c55_rsz_write(), ...)
> that take a mali_c55_resizer or mali_c55_cap_dev pointer, and handle the
> offset there. The register macros should loose their offset parameter.
>
> You could also use a single set of accessors that would become
> path/output-specific (mali_c55_path_write() ? mali_c55_output_write()
> ?), that may make the code easier to read.
>
> You can also replace reg_offset with a void __iomem * base, which would
> avoid the computation at runtime.
>
>> +
>> +	struct mali_c55_mode {
> Make the structure anonymous.
>
>> +		const struct mali_c55_fmt *capture_fmt;
>> +		struct v4l2_pix_format_mplane pix_mp;
>> +	} mode;
> What's a "mode" ? I think I'd name this
>
> 	struct {
> 		const struct mali_c55_fmt *info;
> 		struct v4l2_pix_format_mplane format;
> 	} format;
>
> Or you could just drop the structure and have
>
> 	const struct mali_c55_fmt *format_info;
> 	struct v4l2_pix_format_mplane format;
>
> or something similar.
>
>> +
>> +	struct {
>> +		spinlock_t lock;
>> +		struct list_head queue;
>> +		struct mali_c55_buffer *curr;
>> +		struct mali_c55_buffer *next;
>> +	} buffers;
>> +
>> +	bool streaming;
>> +};
>> +
>> +enum mali_c55_config_spaces {
>> +	MALI_C55_CONFIG_PING,
>> +	MALI_C55_CONFIG_PONG,
>> +	MALI_C55_NUM_CONFIG_SPACES
> The last enumerator is not used.
>
>> +};
>> +
>> +struct mali_c55_ctx {
> mali_c55_context ?
>
>> +	struct mali_c55 *mali_c55;
>> +	void *registers;
> Please document this structure and explain that this field points to a
> copy of the register space in system memory, I was about to write you're
> missing __iomem :-)
Will do
>
>> +	phys_addr_t base;
>> +	spinlock_t lock;
>> +	struct list_head list;
>> +};
>> +
>> +struct mali_c55 {
>> +	struct device *dev;
>> +	struct resource *res;
> You could possibly drop this field by passing the physical address of
> the register space from mali_c55_probe() to mali_c55_init_context() as a
> function parameter.
>
>> +	void __iomem *base;
>> +	struct dma_chan *channel;
>> +	struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
>> +
>> +	u16 capabilities;
>> +	struct media_device media_dev;
>> +	struct v4l2_device v4l2_dev;
>> +	struct media_pipeline pipe;
>> +
>> +	struct mali_c55_tpg tpg;
>> +	struct mali_c55_isp isp;
>> +	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
>> +	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
>> +
>> +	struct list_head contexts;
>> +	enum mali_c55_config_spaces next_config;
>> +};
>> +
>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
>> +		  bool force_hardware);
>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
>> +			  u32 mask, u32 val);
>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
>> +			  enum mali_c55_config_spaces cfg_space);
>> +
>> +int mali_c55_register_isp(struct mali_c55 *mali_c55);
>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55);
>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55);
>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>> +			     enum mali_c55_planes plane);
>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
>> +
>> +bool mali_c55_format_is_raw(unsigned int mbus_code);
>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
>> +
>> +const struct mali_c55_isp_fmt *
>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
>> +#define for_each_mali_isp_fmt(fmt)\
> #define for_each_mali_isp_fmt(fmt) \
>
>> +	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
> Looks like parentheses were on sale :-)
Hah
>
> 	for ((fmt) = NULL; (fmt) = mali_c55_isp_fmt_next(fmt); )
>
> This macro is used in two places only, in the mali-c55-isp.c file where
> open-coding the loop without using mali_c55_isp_fmt_next() would be more
> efficient, and in mali-c55-resizer.c where a function to return format
> i'th would be more efficient. I think you can drop the macro and the
> mali_c55_isp_fmt_next() function.
>
>> +
>> +#endif /* _MALI_C55_COMMON_H */
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>> new file mode 100644
>> index 000000000000..50caf5ee7474
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>> @@ -0,0 +1,767 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ARM Mali-C55 ISP Driver - Core driver code
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#include <linux/bitops.h>
>> +#include <linux/cleanup.h>
>> +#include <linux/clk.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/dmaengine.h>
>> +#include <linux/dma-mapping.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/iopoll.h>
>> +#include <linux/ioport.h>
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/of.h>
>> +#include <linux/of_reserved_mem.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/pm_runtime.h>
>> +#include <linux/scatterlist.h>
> I don't think this is needed.
>
> Missing slab.h.
>
>> +#include <linux/string.h>
>> +
>> +#include <media/media-entity.h>
>> +#include <media/v4l2-device.h>
>> +#include <media/videobuf2-dma-contig.h>
>> +
>> +#include "mali-c55-common.h"
>> +#include "mali-c55-registers.h"
>> +
>> +static const char * const mali_c55_interrupt_names[] = {
>> +	[MALI_C55_IRQ_ISP_START] = "ISP start",
>> +	[MALI_C55_IRQ_ISP_DONE] = "ISP done",
>> +	[MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
>> +	[MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
>> +	[MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
>> +	[MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
>> +	[MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
>> +	[MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
>> +	[MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
>> +	[MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
>> +	[MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
>> +	[MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
>> +	[MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
>> +	[MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
>> +	[MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
>> +	[MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
>> +	[MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
>> +	[MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
>> +	[MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
>> +	[MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
>> +	[MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
>> +	[MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
>> +	[MALI_C55_IRQ_DMA_ERROR] = "DMA error",
>> +	[MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
>> +	[MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
>> +	[MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
>> +};
>> +
>> +static unsigned int config_space_addrs[] = {
> const
>
>> +	[MALI_C55_CONFIG_PING] = 0x0AB6C,
>> +	[MALI_C55_CONFIG_PONG] = 0x22B2C,
> Lowercase hex constants.
>
> Don't the values belong to mali-c55-registers.h ?
>
>> +};
>> +
>> +/* System IO
> /*
>   * System IO
>
>> + *
>> + * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
>> + * and 'pong'), with the  expectation that the 'active' space will be left
> s/the  /the /
>
>> + * untouched whilst a frame is being processed and the 'inactive' space
>> + * configured ready to be passed during the blanking period before the next
> s/to be passed/to be switched to/ ?
>
>> + * frame processing starts. These spaces should ideally be set via DMA transfer
>> + * from a buffer rather than through individual register set operations. There
>> + * is also a shared global register space which should be set normally. Of
>> + * course, the ISP might be included in a system which lacks a suitable DMA
>> + * engine, and the second configuration space might not be fitted at all, which
>> + * means we need to support four scenarios:
>> + *
>> + * 1. Multi config space, with DMA engine.
>> + * 2. Multi config space, no DMA engine.
>> + * 3. Single config space, with DMA engine.
>> + * 4. Single config space, no DMA engine.
>> + *
>> + * The first case is very easy, but the rest present annoying problems. The best
>> + * way to solve them seems to be simply to replicate the concept of DMAing over
>> + * the configuration buffer even if there's no DMA engine on the board, for
>> + * which we rely on memcpy. To facilitate this any read/write call that is made
>> + * to an address within those config spaces should infact be directed to a
>> + * buffer that was allocated to hold them rather than the IO memory itself. The
>> + * actual copy of that buffer to IO mem will happen on interrupt.
>> + */
>> +
>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
>> +{
>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>> +
>> +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
>> +		spin_lock(&ctx->lock);
>> +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
>> +		((u32 *)ctx->registers)[addr] = val;
>> +		spin_unlock(&ctx->lock);
>> +
>> +		return;
>> +	}
> Ouch. This is likely the second comment you really won't like (after the
> comment regarding mali_c55_update_bits() at the very top). I apologize
> in advance.


All good - it's the thinking that is painful, changing the code is easy :)

>
> I really don't like this. Directing writes either to hardware registers
> or to the shadow registers in the context makes the callers of the
> read/write accessors very hard to read. The probe code, for instance,
> mixes writes to hardware registers and writes to the context shadow
> registers to initialize the value of some of the shadow registers.
>
> I'd like to split the read/write accessors into functions that access
> the hardware registers (that's easy) and functions that access the
> shadow registers. I think the latter should receive a mali_c55_ctx
> pointer instead of a mali_c55 pointer to prepare for multi-context
> support.
>
> You can add WARN_ON() guards to the two sets of functions, to ensure
> that no register from the "other" space gets passed to the wrong
> function by mistake.
>
>> +
>> +	writel(val, mali_c55->base + addr);
>> +}
>> +
>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
>> +		  bool force_hardware)
> force_hardware is never set to true.
>
>> +{
>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>> +	u32 val;
>> +
>> +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
>> +		spin_lock(&ctx->lock);
>> +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
>> +		val = ((u32 *)ctx->registers)[addr];
>> +		spin_unlock(&ctx->lock);
>> +
>> +		return val;
>> +	}
>> +
>> +	return readl(mali_c55->base + addr);
>> +}
>> +
>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
>> +			  u32 mask, u32 val)
>> +{
>> +	u32 orig, tmp;
>> +
>> +	orig = mali_c55_read(mali_c55, addr, false);
>> +
>> +	tmp = orig & ~mask;
>> +	tmp |= (val << (ffs(mask) - 1)) & mask;
>> +
>> +	if (tmp != orig)
>> +		mali_c55_write(mali_c55, addr, tmp);
>> +}
>> +
>> +static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
>> +			     dma_addr_t dst, enum dma_data_direction dir)
>> +{
>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>> +	struct dma_async_tx_descriptor *tx;
>> +	enum dma_status status;
>> +	dma_cookie_t cookie;
>> +
>> +	tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
>> +				       MALI_C55_CONFIG_SPACE_SIZE, 0);
>> +	if (!tx) {
>> +		dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
>> +		return -EIO;
>> +	}
>> +
>> +	cookie = dmaengine_submit(tx);
>> +	if (dma_submit_error(cookie)) {
>> +		dev_err(mali_c55->dev, "error submitting dma transfer\n");
>> +		return -EIO;
>> +	}
>> +
>> +	status = dma_sync_wait(mali_c55->channel, cookie);
> I've just realized this performs a busy-wait :-S See the comment in the
> probe function about the threaded IRQ handler. I think we'll need to
> rework all this. It could be done on top though.
It can be switched to an asynchronous transfer quite easily.
>
>> +	if (status != DMA_COMPLETE) {
>> +		dev_err(mali_c55->dev, "dma transfer failed\n");
>> +		return -EIO;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
>> +			     enum mali_c55_config_spaces cfg_space)
>> +{
>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>> +	struct device *dma_dev = mali_c55->channel->device->dev;
>> +	dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
>> +	dma_addr_t dst;
>> +	int ret;
>> +
>> +	guard(spinlock)(&ctx->lock);
>> +
>> +	dst = dma_map_single(dma_dev, ctx->registers,
>> +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
>> +	if (dma_mapping_error(dma_dev, dst)) {
>> +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
>> +		return -EIO;
>> +	}
>> +
>> +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
>> +	dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
>> +			 DMA_FROM_DEVICE);
>> +
>> +	return ret;
>> +}
>> +
>> +static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
>> +		       enum mali_c55_config_spaces cfg_space)
>> +{
>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>> +	struct device *dma_dev = mali_c55->channel->device->dev;
>> +	dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
>> +	dma_addr_t src;
>> +	int ret;
>> +
>> +	guard(spinlock)(&ctx->lock);
> The code below can take a large amount of time, holding a spinlock will
> disable interrupts on the local CPU, that's not good :-(


The intention here is just to prevent the rest of the driver writing into the register space whilst 
it's being DMA transferred to the hardware - perhaps a different means to signal it's safe is more 
appropriate?

>
>> +
>> +	src = dma_map_single(dma_dev, ctx->registers,
>> +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
>> +	if (dma_mapping_error(dma_dev, src)) {
>> +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
>> +		return -EIO;
>> +	}
>> +
>> +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
>> +	dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
>> +			 DMA_TO_DEVICE);
>> +
>> +	return ret;
>> +}
>> +
>> +static int mali_c55_config_read(struct mali_c55_ctx *ctx,
>> +				enum mali_c55_config_spaces cfg_space)
>> +{
>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>> +
>> +	if (mali_c55->channel) {
>> +		return mali_c55_dma_read(ctx, cfg_space);
> As this function is used at probe time only, to initialize the context,
> I think DMA is overkill.
Agreed - I'll switch this to memcpy_fromio()
>> +	} else {
>> +		memcpy_fromio(ctx->registers,
>> +			      mali_c55->base + config_space_addrs[cfg_space],
>> +			      MALI_C55_CONFIG_SPACE_SIZE);
>> +		return 0;
>> +	}
>> +}
>> +
>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
>> +			  enum mali_c55_config_spaces cfg_space)
>> +{
>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>> +
>> +	if (mali_c55->channel) {
>> +		return mali_c55_dma_write(ctx, cfg_space);
>> +	} else {
>> +		memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
>> +			    ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
>> +		return 0;
>> +	}
> Could you measure the time it typically takes to write the registers
> using DMA compared to using memcpy_toio() ?
I will test this and come back to you.
>
>> +}
>> +
>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
>> +{
>> +	return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);
> I think it's too early to tell how multi-context support will look like.
> I'm fine keeping mali_c55_get_active_context() as changing that would be
> very intrusive (even if I think it will need to be changed), but the
> list of contexts is neither the mechanism we'll use, nor something we
> need now. Drop the list, embed the context in struct mali_c55, and
> return the pointer to that single context from this function.
>
>> +}
>> +
>> +static void mali_c55_remove_links(struct mali_c55 *mali_c55)
>> +{
>> +	unsigned int i;
>> +
>> +	media_entity_remove_links(&mali_c55->tpg.sd.entity);
>> +	media_entity_remove_links(&mali_c55->isp.sd.entity);
>> +
>> +	for (i = 0; i < MALI_C55_NUM_RZRS; i++)
>> +		media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
>> +
>> +	for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
>> +		media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
>> +}
>> +
>> +static int mali_c55_create_links(struct mali_c55 *mali_c55)
>> +{
>> +	int ret;
>> +
>> +	/* Test pattern generator to ISP */
>> +	ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
>> +				    &mali_c55->isp.sd.entity,
>> +				    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
>> +		goto err_remove_links;
>> +	}
>> +
>> +	/* Full resolution resizer pipe. */
>> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>> +			MALI_C55_ISP_PAD_SOURCE,
>> +			&mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
>> +			MALI_C55_RZR_SINK_PAD,
>> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
>> +		goto err_remove_links;
>> +	}
>> +
>> +	/* Full resolution bypass. */
>> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>> +				    MALI_C55_ISP_PAD_SOURCE_BYPASS,
>> +				    &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
>> +				    MALI_C55_RZR_SINK_BYPASS_PAD,
>> +				    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
>> +		goto err_remove_links;
>> +	}
>> +
>> +	/* Resizer pipe to video capture nodes. */
>> +	ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
>> +			MALI_C55_RZR_SOURCE_PAD,
>> +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
>> +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev,
>> +			"failed to link FR resizer and video device\n");
>> +		goto err_remove_links;
>> +	}
>> +
>> +	/* The downscale pipe is an optional hardware block */
>> +	if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
>> +		ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>> +			MALI_C55_ISP_PAD_SOURCE,
>> +			&mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
>> +			MALI_C55_RZR_SINK_PAD,
>> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>> +		if (ret) {
>> +			dev_err(mali_c55->dev,
>> +				"failed to link ISP and DS resizer\n");
>> +			goto err_remove_links;
>> +		}
>> +
>> +		ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
>> +			MALI_C55_RZR_SOURCE_PAD,
>> +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
>> +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>> +		if (ret) {
>> +			dev_err(mali_c55->dev,
>> +				"failed to link DS resizer and video device\n");
>> +			goto err_remove_links;
>> +		}
>> +	}
>> +
>> +	return 0;
>> +
>> +err_remove_links:
>> +	mali_c55_remove_links(mali_c55);
>> +	return ret;
>> +}
>> +
>> +static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
>> +{
>> +	mali_c55_unregister_tpg(mali_c55);
>> +	mali_c55_unregister_isp(mali_c55);
>> +	mali_c55_unregister_resizers(mali_c55);
>> +	mali_c55_unregister_capture_devs(mali_c55);
>> +}
>> +
>> +static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>> +{
>> +	int ret;
>> +
>> +	ret = mali_c55_register_tpg(mali_c55);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = mali_c55_register_isp(mali_c55);
>> +	if (ret)
>> +		goto err_unregister_entities;
>> +
>> +	ret = mali_c55_register_resizers(mali_c55);
>> +	if (ret)
>> +		goto err_unregister_entities;
>> +
>> +	ret = mali_c55_register_capture_devs(mali_c55);
>> +	if (ret)
>> +		goto err_unregister_entities;
>> +
>> +	ret = mali_c55_create_links(mali_c55);
>> +	if (ret)
>> +		goto err_unregister_entities;
>> +
>> +	return 0;
>> +
>> +err_unregister_entities:
>> +	mali_c55_unregister_entities(mali_c55);
>> +
>> +	return ret;
>> +}
>> +
>> +static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
>> +{
>> +	u32 product, version, revision, capabilities;
>> +
>> +	product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
>> +	version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
>> +	revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
>> +
>> +	dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
>> +		 product, version, revision);
>> +
>> +	capabilities = mali_c55_read(mali_c55,
>> +				     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
>> +				     false);
>> +	mali_c55->capabilities = (capabilities & 0xffff);
>> +
>> +	/* TODO: Might as well start some debugfs */
> If it's just to expose the version and capabilities, I think that's
> overkill. It's not needed for debug purpose (you can get it from the
> kernel log already). debugfs isn't meant to be accessible in production,
> so an application that would need access to the information wouldn't be
> able to use it.
>
>> +	dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);
> Combine the two messages into one.
>
>> +	return version;
>> +}
>> +
>> +static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>> +	u32 curr_config, next_config;
>> +
>> +	curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
>> +	curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
>> +		      >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
>> +	next_config = curr_config ^ 1;
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>> +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
>> +	mali_c55_config_write(ctx, next_config ?
>> +			      MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
>> +}
>> +
>> +static irqreturn_t mali_c55_isr(int irq, void *context)
>> +{
>> +	struct device *dev = context;
>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>> +	u32 interrupt_status;
>> +	unsigned int i, j;
>> +
>> +	interrupt_status = mali_c55_read(mali_c55,
>> +					 MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
>> +					 false);
>> +	if (!interrupt_status)
>> +		return IRQ_NONE;
>> +
>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
>> +		       interrupt_status);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
>> +
>> +	for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
>> +		if (!(interrupt_status & (1 << i)))
>> +			continue;
>> +
>> +		switch (i) {
>> +		case MALI_C55_IRQ_ISP_START:
>> +			mali_c55_isp_queue_event_sof(mali_c55);
>> +
>> +			for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
>> +				mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
>> +
>> +			mali_c55_swap_next_config(mali_c55);
>> +
>> +			break;
>> +		case MALI_C55_IRQ_ISP_DONE:
>> +			/*
>> +			 * TODO: Where the ISP has no Pong config fitted, we'd
>> +			 * have to do the mali_c55_swap_next_config() call here.
>> +			 */
>> +			break;
>> +		case MALI_C55_IRQ_FR_Y_DONE:
>> +			mali_c55_set_plane_done(
>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
>> +				MALI_C55_PLANE_Y);
>> +			break;
>> +		case MALI_C55_IRQ_FR_UV_DONE:
>> +			mali_c55_set_plane_done(
>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
>> +				MALI_C55_PLANE_UV);
>> +			break;
>> +		case MALI_C55_IRQ_DS_Y_DONE:
>> +			mali_c55_set_plane_done(
>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
>> +				MALI_C55_PLANE_Y);
>> +			break;
>> +		case MALI_C55_IRQ_DS_UV_DONE:
>> +			mali_c55_set_plane_done(
>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
>> +				MALI_C55_PLANE_UV);
>> +			break;
>> +		default:
>> +			/*
>> +			 * Only the above interrupts are currently unmasked. If
>> +			 * we receive anything else here then something weird
>> +			 * has gone on.
>> +			 */
>> +			dev_err(dev, "masked interrupt %s triggered\n",
>> +				mali_c55_interrupt_names[i]);
>> +		}
>> +	}
>> +
>> +	return IRQ_HANDLED;
>> +}
>> +
>> +static int mali_c55_init_context(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_ctx *ctx;
>> +	int ret;
>> +
>> +	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
>> +	if (!ctx) {
>> +		dev_err(mali_c55->dev, "failed to allocate new context\n");
> No need for an error message when memory allocation fails.
>
>> +		return -ENOMEM;
>> +	}
>> +
>> +	ctx->base = mali_c55->res->start;
>> +	ctx->mali_c55 = mali_c55;
>> +
>> +	ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
>> +				 GFP_KERNEL | GFP_DMA);
>> +	if (!ctx->registers) {
>> +		ret = -ENOMEM;
>> +		goto err_free_ctx;
>> +	}
>> +
>> +	/*
>> +	 * The allocated memory is empty, we need to load the default
>> +	 * register settings. We just read Ping; it's identical to Pong.
>> +	 */
>> +	ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
>> +	if (ret)
>> +		goto err_free_registers;
>> +
>> +	list_add_tail(&ctx->list, &mali_c55->contexts);
>> +
>> +	/*
>> +	 * Some features of the ISP need to be disabled by default and only
>> +	 * enabled at the same time as they're configured by a parameters buffer
>> +	 */
>> +
>> +	/* Bypass the sqrt and square compression and expansion modules */
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
>> +			     MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
>> +			     MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
>> +
>> +	/* Bypass the temper module */
>> +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
>> +		       MALI_C55_REG_BYPASS_2_TEMPER);
>> +
>> +	/* Bypass the colour noise reduction  */
>> +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
>> +		       MALI_C55_REG_BYPASS_4_CNR);
>> +
>> +	/* Disable the sinter module */
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
>> +			     MALI_C55_SINTER_ENABLE_MASK, 0x00);
>> +
>> +	/* Disable the RGB Gamma module for each output */
>> +	mali_c55_write(
>> +		mali_c55,
>> +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
>> +		0x00);
>> +	mali_c55_write(
>> +		mali_c55,
>> +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
>> +		0x00);
>> +
>> +	/* Disable the colour correction matrix */
>> +	mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
>> +
>> +	return 0;
>> +
>> +err_free_registers:
>> +	kfree(ctx->registers);
>> +err_free_ctx:
>> +	kfree(ctx);
>> +
>> +	return ret;
>> +}
>> +
>> +static int mali_c55_runtime_resume(struct device *dev)
>> +{
>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>> +	int ret;
>> +
>> +	ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
>> +				      mali_c55->clks);
>> +	if (ret)
>> +		dev_err(mali_c55->dev, "failed to enable clocks\n");
>> +
>> +	return ret;
>> +}
>> +
>> +static int mali_c55_runtime_suspend(struct device *dev)
>> +{
>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>> +
>> +	clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
>> +	return 0;
>> +}
>> +
>> +static const struct dev_pm_ops mali_c55_pm_ops = {
>> +	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
>> +				pm_runtime_force_resume)
>> +	SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
>> +			   NULL)
>> +};
>> +
>> +static int mali_c55_probe(struct platform_device *pdev)
>> +{
>> +	struct device *dev = &pdev->dev;
>> +	struct mali_c55 *mali_c55;
>> +	dma_cap_mask_t mask;
>> +	u32 version;
>> +	int ret;
>> +	u32 val;
>> +
>> +	mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
>> +	if (!mali_c55)
>> +		return dev_err_probe(dev, -ENOMEM,
>> +				     "failed to allocate memory\n");
> 		return -ENOMEM;
>
> There's no need to print messages for memory allocation failures.
>
>> +
>> +	mali_c55->dev = dev;
>> +	platform_set_drvdata(pdev, mali_c55);
>> +
>> +	mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
>> +								&mali_c55->res);
>> +	if (IS_ERR(mali_c55->base))
>> +		return dev_err_probe(dev, PTR_ERR(mali_c55->base),
>> +				     "failed to map IO memory\n");
>> +
>> +	ret = platform_get_irq(pdev, 0);
>> +	if (ret < 0)
>> +		return dev_err_probe(dev, ret, "failed to get interrupt num\n");
> s/ num// or s/num/number/
>
>> +
>> +	ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
>> +					mali_c55_isr, IRQF_ONESHOT,
>> +					dev_driver_string(&pdev->dev),
>> +					&pdev->dev);
> Requested the IRQ should be done much lower, after you have initialized
> everything, or an IRQ that would fire early would have really bad
> consequences.
>
> A comment to explain why you need a threaded interrupt handler would be
> good. I assume it is due only to the need to transfer the registers
> using DMA. I wonder if we should then split the interrupt handler in
> two, with a non-threaded part for the operations that can run quickly,
> and a threaded part for the reprogramming.


Instead of passing NULL for the top handler you mean?

> It may also be that we could just start the DMA transfer in the
> non-threaded handler without waiting synchronously for it to complete.
> That would be a bigger change, and would require checking race
> conditions carefully. On the other hand, I'm a bit concerned about the
> current implementation, have you tested what happens if the DMA transfer
> takes too long to complete, and spans frame boundaries ?
No; I can force a delay in the DMA engine and see how it behaves. The stats buffers are currently 
DMAd asynchronously; I don't think it'd be a huge change to make the configuration buffer handling 
work that way too.
>
>> +	if (ret)
>> +		return dev_err_probe(dev, ret, "failed to request irq\n");
>> +
>> +	for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
>> +		mali_c55->clks[i].id = mali_c55_clk_names[i];
>> +
>> +	ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
>> +	if (ret)
>> +		return dev_err_probe(dev, ret, "failed to acquire clocks\n");
>> +
>> +	pm_runtime_enable(&pdev->dev);
>> +
>> +	ret = pm_runtime_resume_and_get(&pdev->dev);
>> +	if (ret)
>> +		goto err_pm_runtime_disable;
>> +
>> +	of_reserved_mem_device_init(dev);
> I'd move this, the vb2_dma_contig_set_max_seg_size() call and the
> dma_cap_* calls before pm_runtime_enable() as they don't need the device
> to be powered.
>
>> +	version = mali_c55_check_hwcfg(mali_c55);
>> +	vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
>> +
>> +	/* Use "software only" context management. */
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>> +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
>> +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> You handle that in mali_c55_isp_start(), does the register have to be
> set here too ?
>
>> +
>> +	dma_cap_zero(mask);
>> +	dma_cap_set(DMA_MEMCPY, mask);
>> +
>> +	/*
>> +	 * No error check, because we will just fallback on memcpy if there is
>> +	 * no usable DMA channel on the system.
>> +	 */
>> +	mali_c55->channel = dma_request_channel(mask, NULL, NULL);
>> +
>> +	INIT_LIST_HEAD(&mali_c55->contexts);
>> +	ret = mali_c55_init_context(mali_c55);
>> +	if (ret)
>> +		goto err_release_dma_channel;
>> +
> I'd move all the code from here ...
>
>> +	mali_c55->media_dev.dev = dev;
>> +	strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
>> +		sizeof(mali_c55->media_dev.model));
>> +	mali_c55->media_dev.hw_revision = version;
>> +
>> +	media_device_init(&mali_c55->media_dev);
>> +	ret = media_device_register(&mali_c55->media_dev);
>> +	if (ret)
>> +		goto err_cleanup_media_device;
>> +
>> +	mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
>> +	ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
>> +	if (ret) {
>> +		dev_err(dev, "failed to register V4L2 device\n");
>> +		goto err_unregister_media_device;
>> +	};
>> +
>> +	ret = mali_c55_register_entities(mali_c55);
>> +	if (ret) {
>> +		dev_err(dev, "failed to register entities\n");
>> +		goto err_unregister_v4l2_device;
>> +	}
> ... to here to a separate function, or maybe fold it all in
> mali_c55_register_entities() (which should the be renamed). Same thing
> for the cleanup code.
>
>> +
>> +	/* Set safe stop to ensure we're in a non-streaming state */
>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>> +		       MALI_C55_INPUT_SAFE_STOP);
>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>> +
>> +	/*
>> +	 * We're ready to process interrupts. Clear any that are set and then
>> +	 * unmask them for processing.
>> +	 */
>> +	mali_c55_write(mali_c55, 0x30, 0xffffffff);
>> +	mali_c55_write(mali_c55, 0x34, 0xffffffff);
>> +	mali_c55_write(mali_c55, 0x40, 0x01);
>> +	mali_c55_write(mali_c55, 0x40, 0x00);
> Please replace the register addresses with macros.
>
>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);
> The value should use the interrupt bits macros.
>
>> +
>> +	pm_runtime_put(&pdev->dev);
> Once power gets cut, the registers your programmed above may be lost. I
> think you should programe them in the runtime PM resume handler.
>
>> +
>> +	return 0;
>> +
>> +err_unregister_v4l2_device:
>> +	v4l2_device_unregister(&mali_c55->v4l2_dev);
>> +err_unregister_media_device:
>> +	media_device_unregister(&mali_c55->media_dev);
>> +err_cleanup_media_device:
>> +	media_device_cleanup(&mali_c55->media_dev);
>> +err_release_dma_channel:
>> +	dma_release_channel(mali_c55->channel);
>> +err_pm_runtime_disable:
>> +	pm_runtime_disable(&pdev->dev);
>> +
>> +	return ret;
>> +}
>> +
>> +static void mali_c55_remove(struct platform_device *pdev)
>> +{
>> +	struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
>> +	struct mali_c55_ctx *ctx, *tmp;
>> +
>> +	list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
>> +		list_del(&ctx->list);
>> +		kfree(ctx->registers);
>> +		kfree(ctx);
>> +	}
>> +
>> +	mali_c55_remove_links(mali_c55);
>> +	mali_c55_unregister_entities(mali_c55);
>> +	v4l2_device_put(&mali_c55->v4l2_dev);
>> +	media_device_unregister(&mali_c55->media_dev);
>> +	media_device_cleanup(&mali_c55->media_dev);
>> +	dma_release_channel(mali_c55->channel);
>> +}
>> +
>> +static const struct of_device_id mali_c55_of_match[] = {
>> +	{ .compatible = "arm,mali-c55", },
>> +	{},
> 	{ /* Sentinel */ },
>
>> +};
>> +MODULE_DEVICE_TABLE(of, mali_c55_of_match);
>> +
>> +static struct platform_driver mali_c55_driver = {
>> +	.driver = {
>> +		.name = "mali-c55",
>> +		.of_match_table = of_match_ptr(mali_c55_of_match),
> Drop of_match_ptr().
>
>> +		.pm = &mali_c55_pm_ops,
>> +	},
>> +	.probe = mali_c55_probe,
>> +	.remove_new = mali_c55_remove,
>> +};
>> +
>> +module_platform_driver(mali_c55_driver);
> Blank line.
>
>> +MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
>> +MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
>> +MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
>> +MODULE_LICENSE("GPL");
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>> new file mode 100644
>> index 000000000000..ea8b7b866e7a
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>> @@ -0,0 +1,611 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ARM Mali-C55 ISP Driver - Image signal processor
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#include <linux/delay.h>
>> +#include <linux/iopoll.h>
>> +#include <linux/property.h>
>> +#include <linux/string.h>
>> +
>> +#include <media/media-entity.h>
>> +#include <media/v4l2-common.h>
>> +#include <media/v4l2-event.h>
>> +#include <media/v4l2-mc.h>
>> +#include <media/v4l2-subdev.h>
>> +
>> +#include "mali-c55-common.h"
>> +#include "mali-c55-registers.h"
>> +
>> +static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
>> +	{
>> +		.code = MEDIA_BUS_FMT_SRGGB20_1X20,
>> +		.order = MALI_C55_BAYER_ORDER_RGGB,
>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>> +	},
>> +	{
>> +		.code = MEDIA_BUS_FMT_SGRBG20_1X20,
>> +		.order = MALI_C55_BAYER_ORDER_GRBG,
>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>> +	},
>> +	{
>> +		.code = MEDIA_BUS_FMT_SGBRG20_1X20,
>> +		.order = MALI_C55_BAYER_ORDER_GBRG,
>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>> +	},
>> +	{
>> +		.code = MEDIA_BUS_FMT_SBGGR20_1X20,
>> +		.order = MALI_C55_BAYER_ORDER_BGGR,
>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>> +	},
>> +	{
>> +		.code = MEDIA_BUS_FMT_RGB202020_1X60,
>> +		.order = 0, /* Not relevant for this format */
>> +		.encoding = V4L2_PIXEL_ENC_RGB,
>> +	}
>> +	/*
>> +	 * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
>> +	 * also support YUV input from a sensor passed-through to the output. At
>> +	 * present we have no mechanism to test that though so it may have to
>> +	 * wait a while...
>> +	 */
>> +};
>> +
>> +const struct mali_c55_isp_fmt *
>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
>> +{
>> +	if (!fmt)
>> +		fmt = &mali_c55_isp_fmts[0];
>> +	else
>> +		fmt++;
>> +
>> +	for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
>> +		return fmt;
> That's peculiar.
>
> 	if (!fmt)
> 		fmt = &mali_c55_isp_fmts[0];
> 	else if (fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)])
> 		return ++fmt;
> 	else
> 		return NULL;
>
>> +
>> +	return NULL;
>> +}
>> +
>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
>> +{
>> +	const struct mali_c55_isp_fmt *isp_fmt;
>> +
>> +	for_each_mali_isp_fmt(isp_fmt) {
> I would open-code the loop instead of using the macro, like you do
> below. It will be more efficient.
>
>> +		if (isp_fmt->code == mbus_code)
>> +			return true;
>> +	}
>> +
>> +	return false;
>> +}
>> +
>> +static const struct mali_c55_isp_fmt *
>> +mali_c55_isp_get_mbus_config_by_code(u32 code)
>> +{
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
>> +		if (mali_c55_isp_fmts[i].code == code)
>> +			return &mali_c55_isp_fmts[i];
>> +	}
>> +
>> +	return NULL;
>> +}
>> +
>> +static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
>> +{
>> +	u32 val;
>> +
>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);
> 	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> 		       MALI_C55_INPUT_SAFE_STOP);
>
>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>> +}
>> +
>> +static int mali_c55_isp_start(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>> +	const struct mali_c55_isp_fmt *cfg;
>> +	struct v4l2_mbus_framefmt *format;
> const
>
>> +	struct v4l2_subdev_state *state;
>> +	struct v4l2_rect *crop;
> const
>
>> +	struct v4l2_subdev *sd;
>> +	u32 val;
>> +	int ret;
>> +
>> +	sd = &mali_c55->isp.sd;
> Assign when declaring the variable.
>
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>> +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
>> +
>> +	/* Apply input windowing */
>> +	state = v4l2_subdev_get_locked_active_state(sd);
> Using .enable_streams() (see below) you'll get this for free.
>
>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>> +	format = v4l2_subdev_state_get_format(state,
>> +					      MALI_C55_ISP_PAD_SINK_VIDEO);
>> +	cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
>> +
>> +	mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
>> +		       MALI_C55_HC_START(crop->left));
>> +	mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
>> +		       MALI_C55_HC_SIZE(crop->width));
>> +	mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
>> +		       MALI_C55_VC_START(crop->top) |
>> +		       MALI_C55_VC_SIZE(crop->height));
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
>> +			     MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
>> +			     MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
>> +			     MALI_C55_BAYER_ORDER_MASK, cfg->order);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
>> +			     MALI_C55_INPUT_WIDTH_MASK,
>> +			     MALI_C55_INPUT_WIDTH_20BIT);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>> +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
>> +			     cfg->encoding == V4L2_PIXEL_ENC_RGB ?
>> +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
>> +
>> +	ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "failed to DMA config\n");
>> +		return ret;
>> +	}
>> +
>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>> +		       MALI_C55_INPUT_SAFE_START);
>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> Should you return an error in case of timeout ?
>
>> +
>> +	return 0;
>> +}
>> +
>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)
> Why is this not handled wired to .s_stream() ? Or better,
> .enable_streams() and .disable_streams().


There didn't really seem to be much point, since nothing outside this driver is ever going to start 
the subdev streaming...it's not like the sensor case where a separate driver might have to call some 
operation to start it.

>
>> +{
>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>> +	struct v4l2_subdev *sd;
>> +
>> +	if (isp->remote_src) {
>> +		sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
>> +		v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
>> +	}
>> +	isp->remote_src = NULL;
>> +
>> +	mali_c55_isp_stop(mali_c55);
>> +}
>> +
>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
>> +{
>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>> +	struct media_pad *sink_pad;
>> +	struct v4l2_subdev *sd;
>> +	int ret;
>> +
>> +	sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
>> +	isp->remote_src = media_pad_remote_pad_unique(sink_pad);
>> +	if (IS_ERR(isp->remote_src)) {
> If you set the MUST_CONNECT flag for the MALI_C55_ISP_PAD_SINK_VIDEO pad
> I think you can drop this check.
>
>> +		dev_err(mali_c55->dev, "Failed to get source for ISP\n");
>> +		return PTR_ERR(isp->remote_src);
>> +	}
>> +
>> +	sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
>> +
>> +	isp->frame_sequence = 0;
>> +	ret = mali_c55_isp_start(mali_c55);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "Failed to start ISP\n");
>> +		isp->remote_src = NULL;
>> +		return ret;
>> +	}
>> +
>> +	/*
>> +	 * We only support a single input stream, so we can just enable the 1st
>> +	 * entry in the streams mask.
>> +	 */
>> +	ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "Failed to start ISP source\n");
>> +		mali_c55_isp_stop(mali_c55);
>> +		return ret;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
>> +				       struct v4l2_subdev_state *state,
>> +				       struct v4l2_subdev_mbus_code_enum *code)
>> +{
>> +	/*
>> +	 * Only the internal RGB processed format is allowed on the regular
>> +	 * processing source pad.
>> +	 */
>> +	if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
>> +		if (code->index)
>> +			return -EINVAL;
>> +
>> +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
>> +		return 0;
>> +	}
>> +
>> +	/* On the sink and bypass pads all the supported formats are allowed. */
>> +	if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
>> +		return -EINVAL;
>> +
>> +	code->code = mali_c55_isp_fmts[code->index].code;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
>> +					struct v4l2_subdev_state *state,
>> +					struct v4l2_subdev_frame_size_enum *fse)
>> +{
>> +	const struct mali_c55_isp_fmt *cfg;
>> +
>> +	if (fse->index > 0)
>> +		return -EINVAL;
>> +
>> +	/*
>> +	 * Only the internal RGB processed format is allowed on the regular
>> +	 * processing source pad.
>> +	 *
>> +	 * On the sink and bypass pads all the supported formats are allowed.
>> +	 */
>> +	if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
>> +		if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
>> +			return -EINVAL;
>> +	} else {
>> +		cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
>> +		if (!cfg)
>> +			return -EINVAL;
>> +	}
>> +
>> +	fse->min_width = MALI_C55_MIN_WIDTH;
>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
>> +	fse->max_width = MALI_C55_MAX_WIDTH;
>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
>> +				struct v4l2_subdev_state *state,
>> +				struct v4l2_subdev_format *format)
>> +{
>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>> +	struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
>> +	const struct mali_c55_isp_fmt *cfg;
>> +	struct v4l2_rect *crop;
>> +
>> +	/*
>> +	 * Disallow set_fmt on the source pads; format is fixed and the sizes
>> +	 * are the result of applying the sink crop rectangle to the sink
>> +	 * format.
>> +	 */
>> +	if (format->pad)
> 	if (format->pad != MALI_C55_ISP_PAD_SINK_VIDEO)
>
>> +		return v4l2_subdev_get_fmt(sd, state, format);
>> +
>> +	cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
>> +	if (!cfg)
>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>> +	fmt->field = V4L2_FIELD_NONE;
> Do you intentionally allow the colorspace fields to be overwritten to
> any value ? rkisp1_isp_set_src_fmt() and rkisp1_isp_set_sink_fmt() may
> show you how this could be handled.
>
>> +
>> +	/*
>> +	 * Clamp sizes in the accepted limits and clamp the crop rectangle in
>> +	 * the new sizes.
>> +	 */
>> +	clamp_t(unsigned int, fmt->width,
>> +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>> +	clamp_t(unsigned int, fmt->width,
>> +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> clamp_t() returns a value, which you ignore. Those are no-ops. You meant
>
> 	fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> 			     MALI_C55_MAX_WIDTH);
> 	fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> 			      MALI_C55_MAX_HEIGHT);
>
> Same for every use of clamp_t() through the whole driver.
>
> Also, do you need clamp_t() ? I think all values are unsigned int, you
> can use clamp().
>
> Are there any alignment constraints, such a multiples of two for bayer
> formats ? Same in all the other locations where applicable.
>
>> +
>> +	sink_fmt = v4l2_subdev_state_get_format(state,
>> +						MALI_C55_ISP_PAD_SINK_VIDEO);
>> +	*sink_fmt = *fmt;
>> +
>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>> +	crop->left = 0;
>> +	crop->top = 0;
>> +	crop->width = fmt->width;
>> +	crop->height = fmt->height;
>> +
>> +	/*
>> +	 * Propagate format to source pads. On the 'regular' output pad use
>> +	 * the internal RGB processed format, while on the bypass pad simply
>> +	 * replicate the ISP sink format. The sizes on both pads are the same as
>> +	 * the ISP sink crop rectangle.
>> +	 */
> Colorspace information will need to be propagated too.
>
>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
>> +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>> +	src_fmt->width = crop->width;
>> +	src_fmt->height = crop->height;
>> +
>> +	src_fmt = v4l2_subdev_state_get_format(state,
>> +					       MALI_C55_ISP_PAD_SOURCE_BYPASS);
>> +	src_fmt->code = fmt->code;
>> +	src_fmt->width = crop->width;
>> +	src_fmt->height = crop->height;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
>> +				      struct v4l2_subdev_state *state,
>> +				      struct v4l2_subdev_selection *sel)
>> +{
>> +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> 	sel->pad != MALI_C55_ISP_PAD_SINK_VIDEO
>
>> +		return -EINVAL;
>> +
>> +	sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
>> +				      struct v4l2_subdev_state *state,
>> +				      struct v4l2_subdev_selection *sel)
>> +{
>> +	struct v4l2_mbus_framefmt *src_fmt;
>> +	struct v4l2_mbus_framefmt *fmt;
> const
>
>> +	struct v4l2_rect *crop;
>> +
>> +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> Ditto.
>
>> +		return -EINVAL;
>> +
>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>> +
>> +	clamp_t(unsigned int, sel->r.left, 0, fmt->width);
>> +	clamp_t(unsigned int, sel->r.top, 0, fmt->height);
>> +	clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
>> +		fmt->width - sel->r.left);
>> +	clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
>> +		fmt->height - sel->r.top);
>> +
>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>> +	*crop = sel->r;
>> +
>> +	/* Propagate the crop rectangle sizes to the source pad format. */
>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
>> +	src_fmt->width = crop->width;
>> +	src_fmt->height = crop->height;
> Can you confirm that cropping doesn't affect the bypass path ?


Yes

> And maybe
> add a comment to mention it.


Will

>
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
>> +	.enum_mbus_code		= mali_c55_isp_enum_mbus_code,
>> +	.enum_frame_size	= mali_c55_isp_enum_frame_size,
>> +	.get_fmt		= v4l2_subdev_get_fmt,
>> +	.set_fmt		= mali_c55_isp_set_fmt,
>> +	.get_selection		= mali_c55_isp_get_selection,
>> +	.set_selection		= mali_c55_isp_set_selection,
>> +	.link_validate		= v4l2_subdev_link_validate_default,
>> +};
>> +
>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
>> +{
>> +	struct v4l2_event event = {
>> +		.type = V4L2_EVENT_FRAME_SYNC,
>> +	};
>> +
>> +	event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
>> +	v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
>> +}
>> +
>> +static int
>> +mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
>> +			     struct v4l2_event_subscription *sub)
>> +{
>> +	if (sub->type != V4L2_EVENT_FRAME_SYNC)
>> +		return -EINVAL;
>> +
>> +	/* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
>> +	if (sub->id != 0)
>> +		return -EINVAL;
>> +
>> +	return v4l2_event_subscribe(fh, sub, 0, NULL);
>> +}
>> +
>> +static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
>> +	.subscribe_event = mali_c55_isp_subscribe_event,
>> +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
>> +};
>> +
>> +static const struct v4l2_subdev_ops mali_c55_isp_ops = {
>> +	.pad	= &mali_c55_isp_pad_ops,
>> +	.core	= &mali_c55_isp_core_ops,
>> +};
>> +
>> +static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
>> +				   struct v4l2_subdev_state *sd_state)
> You name this variable state in every other subdev operation handler.
>
>> +{
>> +	struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>> +	struct v4l2_rect *in_crop;
>> +
>> +	sink_fmt = v4l2_subdev_state_get_format(sd_state,
>> +						MALI_C55_ISP_PAD_SINK_VIDEO);
>> +	src_fmt = v4l2_subdev_state_get_format(sd_state,
>> +					       MALI_C55_ISP_PAD_SOURCE);
>> +	in_crop = v4l2_subdev_state_get_crop(sd_state,
>> +					     MALI_C55_ISP_PAD_SINK_VIDEO);
>> +
>> +	sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
>> +	sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
>> +	sink_fmt->field = V4L2_FIELD_NONE;
>> +	sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> You should initialize the colorspace fields too. Same below.
>
>> +
>> +	*v4l2_subdev_state_get_format(sd_state,
>> +			      MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
>> +
>> +	src_fmt->width = MALI_C55_DEFAULT_WIDTH;
>> +	src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
>> +	src_fmt->field = V4L2_FIELD_NONE;
>> +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>> +
>> +	in_crop->top = 0;
>> +	in_crop->left = 0;
>> +	in_crop->width = MALI_C55_DEFAULT_WIDTH;
>> +	in_crop->height = MALI_C55_DEFAULT_HEIGHT;
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
>> +	.init_state = mali_c55_isp_init_state,
>> +};
>> +
>> +static const struct media_entity_operations mali_c55_isp_media_ops = {
>> +	.link_validate		= v4l2_subdev_link_validate,
> 	.link_validate = v4l2_subdev_link_validate,
>
> to match mali_c55_isp_internal_ops.
>
>> +};
>> +
>> +static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
>> +				       struct v4l2_subdev *subdev,
>> +				       struct v4l2_async_connection *asc)
>> +{
>> +	struct mali_c55_isp *isp = container_of(notifier,
>> +						struct mali_c55_isp, notifier);
>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>> +	struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
>> +	int ret;
>> +
>> +	/*
>> +	 * By default we'll flag this link enabled and the TPG disabled, but
>> +	 * no immutable flag because we need to be able to switch between the
>> +	 * two.
>> +	 */
>> +	ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
>> +					      MEDIA_LNK_FL_ENABLED);
>> +	if (ret)
>> +		dev_err(mali_c55->dev, "failed to create link for %s\n",
>> +			subdev->name);
>> +
>> +	return ret;
>> +}
>> +
>> +static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
>> +{
>> +	struct mali_c55_isp *isp = container_of(notifier,
>> +						struct mali_c55_isp, notifier);
>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>> +
>> +	return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
>> +}
>> +
>> +static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
>> +	.bound = mali_c55_isp_notifier_bound,
>> +	.complete = mali_c55_isp_notifier_complete,
>> +};
>> +
>> +static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
>> +{
>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>> +	struct v4l2_async_connection *asc;
>> +	struct fwnode_handle *ep;
>> +	int ret;
>> +
>> +	v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
>> +
>> +	/*
>> +	 * The ISP should have a single endpoint pointing to some flavour of
>> +	 * CSI-2 receiver...but for now at least we do want everything to work
>> +	 * normally even with no sensors connected, as we have the TPG. If we
>> +	 * don't find a sensor just warn and return success.
>> +	 */
>> +	ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
>> +					     0, 0, 0);
>> +	if (!ep) {
>> +		dev_warn(mali_c55->dev, "no local endpoint found\n");
>> +		return 0;
>> +	}
>> +
>> +	asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
>> +					      struct v4l2_async_connection);
>> +	if (IS_ERR(asc)) {
>> +		dev_err(mali_c55->dev, "failed to add remote fwnode\n");
>> +		ret = PTR_ERR(asc);
>> +		goto err_put_ep;
>> +	}
>> +
>> +	isp->notifier.ops = &mali_c55_isp_notifier_ops;
>> +	ret = v4l2_async_nf_register(&isp->notifier);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "failed to register notifier\n");
>> +		goto err_cleanup_nf;
>> +	}
>> +
>> +	fwnode_handle_put(ep);
>> +
>> +	return 0;
>> +
>> +err_cleanup_nf:
>> +	v4l2_async_nf_cleanup(&isp->notifier);
>> +err_put_ep:
>> +	fwnode_handle_put(ep);
>> +
>> +	return ret;
>> +}
>> +
>> +int mali_c55_register_isp(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>> +	struct v4l2_subdev *sd = &isp->sd;
>> +	int ret;
>> +
>> +	isp->mali_c55 = mali_c55;
>> +
>> +	v4l2_subdev_init(sd, &mali_c55_isp_ops);
>> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
>> +	sd->entity.ops = &mali_c55_isp_media_ops;
>> +	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
>> +	sd->internal_ops = &mali_c55_isp_internal_ops;
>> +	strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
>> +
>> +	isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
> The MUST_CONNECT flag would make sense here.
>
>> +	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>> +	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
>> +
>> +	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
>> +				     isp->pads);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = v4l2_subdev_init_finalize(sd);
>> +	if (ret)
>> +		goto err_cleanup_media_entity;
>> +
>> +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>> +	if (ret)
>> +		goto err_cleanup_subdev;
>> +
>> +	ret = mali_c55_isp_parse_endpoint(isp);
>> +	if (ret)
>> +		goto err_cleanup_subdev;
> As noted elsewhere, I think this belongs to mali-c55-core.c.
>
>> +
>> +	mutex_init(&isp->lock);
> This lock is used in mali-c55-capture.c only, that seems weird.


It's because we have two separate capture devices, who's start/stop streaming path calls into the 
ISP subdevice's start streaming function, but has to do it after a bunch of other writes to the cap 
device or resizer specific sections...the intention was to delay doing any of that until the ISP was 
in a known state.

>
>> +
>> +	return 0;
>> +
>> +err_cleanup_subdev:
>> +	v4l2_subdev_cleanup(sd);
>> +err_cleanup_media_entity:
>> +	media_entity_cleanup(&sd->entity);
>> +	isp->mali_c55 = NULL;
>> +
>> +	return ret;
>> +}
>> +
>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>> +
>> +	if (!isp->mali_c55)
>> +		return;
>> +
>> +	mutex_destroy(&isp->lock);
>> +	v4l2_async_nf_unregister(&isp->notifier);
>> +	v4l2_async_nf_cleanup(&isp->notifier);
>> +	v4l2_device_unregister_subdev(&isp->sd);
>> +	v4l2_subdev_cleanup(&isp->sd);
>> +	media_entity_cleanup(&isp->sd.entity);
>> +}
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>> new file mode 100644
>> index 000000000000..cb27abde2aa5
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>> @@ -0,0 +1,258 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * ARM Mali-C55 ISP Driver - Register definitions
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#ifndef _MALI_C55_REGISTERS_H
>> +#define _MALI_C55_REGISTERS_H
>> +
>> +#include <linux/bits.h>
>> +
>> +/* ISP Common 0x00000 - 0x000ff */
>> +
>> +#define MALI_C55_REG_API				0x00000
>> +#define MALI_C55_REG_PRODUCT				0x00004
>> +#define MALI_C55_REG_VERSION				0x00008
>> +#define MALI_C55_REG_REVISION				0x0000c
>> +#define MALI_C55_REG_PULSE_MODE				0x0003c
>> +#define MALI_C55_REG_INPUT_MODE_REQUEST			0x0009c
>> +#define MALI_C55_INPUT_SAFE_STOP			0x00
>> +#define MALI_C55_INPUT_SAFE_START			0x01
>> +#define MALI_C55_REG_MODE_STATUS			0x000a0
>> +#define MALI_C55_REG_INTERRUPT_MASK_VECTOR		0x00030
>> +#define MALI_C55_INTERRUPT_MASK_ALL			GENMASK(31, 0)
>> +
>> +#define MALI_C55_REG_GLOBAL_MONITOR			0x00050
>> +
>> +#define MALI_C55_REG_GEN_VIDEO				0x00080
>> +#define MALI_C55_REG_GEN_VIDEO_ON_MASK			BIT(0)
>> +#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK		BIT(1)
>> +#define MALI_C55_REG_GEN_PREFETCH_MASK			GENMASK(31, 16)
>> +
>> +#define MALI_C55_REG_MCU_CONFIG				0x00020
>> +#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK		BIT(0)
> #define MALI_C55_REG_MCU_CONFIG_OVERRIDE		BIT(0)
>
> Same in other places where applicable.
>
>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK		BIT(1)
>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PING		BIT(1)
>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG		0x00
>> +#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK		BIT(8)
>> +#define MALI_C55_REG_PING_PONG_READ			0x00024
>> +#define MALI_C55_REG_PING_PONG_READ_MASK		BIT(2)
>> +
>> +#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR		0x00034
>> +#define MALI_C55_REG_INTERRUPT_CLEAR			0x00040
>> +#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR		0x00044
>> +#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS		0x00068
>> +#define MALI_C55_GPS_PONG_FITTED			BIT(0)
>> +#define MALI_C55_GPS_WDR_FITTED				BIT(1)
>> +#define MALI_C55_GPS_COMPRESSION_FITTED			BIT(2)
>> +#define MALI_C55_GPS_TEMPER_FITTED			BIT(3)
>> +#define MALI_C55_GPS_SINTER_LITE_FITTED			BIT(4)
>> +#define MALI_C55_GPS_SINTER_FITTED			BIT(5)
>> +#define MALI_C55_GPS_IRIDIX_LTM_FITTED			BIT(6)
>> +#define MALI_C55_GPS_IRIDIX_GTM_FITTED			BIT(7)
>> +#define MALI_C55_GPS_CNR_FITTED				BIT(8)
>> +#define MALI_C55_GPS_FRSCALER_FITTED			BIT(9)
>> +#define MALI_C55_GPS_DS_PIPE_FITTED			BIT(10)
>> +
>> +#define MALI_C55_REG_BLANKING				0x00084
>> +#define MALI_C55_REG_HBLANK_MASK			GENMASK(15, 0)
>> +#define MALI_C55_REG_VBLANK_MASK			GENMASK(31, 16)
>> +
>> +#define MALI_C55_REG_HC_START				0x00088
>> +#define MALI_C55_HC_START(h)				(((h) & 0xffff) << 16)
>> +#define MALI_C55_REG_HC_SIZE				0x0008c
>> +#define MALI_C55_HC_SIZE(h)				((h) & 0xffff)
>> +#define MALI_C55_REG_VC_START_SIZE			0x00094
>> +#define MALI_C55_VC_START(v)				((v) & 0xffff)
>> +#define MALI_C55_VC_SIZE(v)				(((v) & 0xffff) << 16)
>> +
>> +/* Ping/Pong Configuration Space */
>> +#define MALI_C55_REG_BASE_ADDR				0x18e88
>> +#define MALI_C55_REG_BYPASS_0				0x18eac
>> +#define MALI_C55_REG_BYPASS_0_VIDEO_TEST		BIT(0)
>> +#define MALI_C55_REG_BYPASS_0_INPUT_FMT			BIT(1)
>> +#define MALI_C55_REG_BYPASS_0_DECOMPANDER		BIT(2)
>> +#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR		BIT(3)
>> +#define MALI_C55_REG_BYPASS_0_GAIN_WDR			BIT(4)
>> +#define MALI_C55_REG_BYPASS_0_FRAME_STITCH		BIT(5)
>> +#define MALI_C55_REG_BYPASS_1				0x18eb0
>> +#define MALI_C55_REG_BYPASS_1_DIGI_GAIN			BIT(0)
>> +#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS		BIT(1)
>> +#define MALI_C55_REG_BYPASS_1_FE_SQRT			BIT(2)
>> +#define MALI_C55_REG_BYPASS_1_RAW_FE			BIT(3)
>> +#define MALI_C55_REG_BYPASS_2				0x18eb8
>> +#define MALI_C55_REG_BYPASS_2_SINTER			BIT(0)
>> +#define MALI_C55_REG_BYPASS_2_TEMPER			BIT(1)
>> +#define MALI_C55_REG_BYPASS_3				0x18ebc
>> +#define MALI_C55_REG_BYPASS_3_SQUARE_BE			BIT(0)
>> +#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH	BIT(1)
>> +#define MALI_C55_REG_BYPASS_3_MESH_SHADING		BIT(3)
>> +#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE		BIT(4)
>> +#define MALI_C55_REG_BYPASS_3_IRIDIX			BIT(5)
>> +#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN		BIT(6)
>> +#define MALI_C55_REG_BYPASS_4				0x18ec0
>> +#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB		BIT(1)
>> +#define MALI_C55_REG_BYPASS_4_PF_CORRECTION		BIT(3)
>> +#define MALI_C55_REG_BYPASS_4_CCM			BIT(4)
>> +#define MALI_C55_REG_BYPASS_4_CNR			BIT(5)
>> +#define MALI_C55_REG_FR_BYPASS				0x18ec4
>> +#define MALI_C55_REG_DS_BYPASS				0x18ec8
>> +#define MALI_C55_BYPASS_CROP				BIT(0)
>> +#define MALI_C55_BYPASS_SCALER				BIT(1)
>> +#define MALI_C55_BYPASS_GAMMA_RGB			BIT(2)
>> +#define MALI_C55_BYPASS_SHARPEN				BIT(3)
>> +#define MALI_C55_BYPASS_CS_CONV				BIT(4)
>> +#define MALI_C55_REG_ISP_RAW_BYPASS			0x18ecc
>> +#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK		BIT(0)
>> +#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK		GENMASK(9, 8)
>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS		2
>> +#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS		1
>> +#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE		BIT(1)
>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS		BIT(0)
>> +
>> +#define MALI_C55_REG_ACTIVE_WIDTH_MASK			0xffff
>> +#define MALI_C55_REG_ACTIVE_HEIGHT_MASK			0xffff0000
>> +#define MALI_C55_REG_BAYER_ORDER			0x18e8c
>> +#define MALI_C55_BAYER_ORDER_MASK			GENMASK(1, 0)
>> +#define MALI_C55_REG_TPG_CH0				0x18ed8
>> +#define MALI_C55_TEST_PATTERN_ON_OFF			BIT(0)
>> +#define MALI_C55_TEST_PATTERN_RGB_MASK			BIT(1)
>> +#define MALI_C55_REG_TPG_R_BACKGROUND			0x18ee0
>> +#define MALI_C55_REG_TPG_G_BACKGROUND			0x18ee4
>> +#define MALI_C55_REG_TPG_B_BACKGROUND			0x18ee8
>> +#define MALI_C55_TPG_BACKGROUND_MAX			0xfffff
>> +#define MALI_C55_REG_INPUT_WIDTH			0x18f98
>> +#define MALI_C55_INPUT_WIDTH_MASK			GENMASK(18, 16)
>> +#define MALI_C55_INPUT_WIDTH_8BIT			0
>> +#define MALI_C55_INPUT_WIDTH_10BIT			1
>> +#define MALI_C55_INPUT_WIDTH_12BIT			2
>> +#define MALI_C55_INPUT_WIDTH_14BIT			3
>> +#define MALI_C55_INPUT_WIDTH_16BIT			4
>> +#define MALI_C55_INPUT_WIDTH_20BIT			5
>> +#define MALI_C55_REG_SPACE_SIZE				0x4000
>> +#define MALI_C55_REG_CONFIG_SPACES_OFFSET		0x0ab6c
>> +#define MALI_C55_CONFIG_SPACE_SIZE			0x1231c
>> +
>> +#define MALI_C55_REG_SINTER_CONFIG			0x19348
>> +#define MALI_C55_SINTER_VIEW_FILTER_MASK		GENMASK(1, 0)
>> +#define MALI_C55_SINTER_SCALE_MODE_MASK			GENMASK(3, 2)
>> +#define MALI_C55_SINTER_ENABLE_MASK			BIT(4)
>> +#define MALI_C55_SINTER_FILTER_SELECT_MASK		BIT(5)
>> +#define MALI_C55_SINTER_INT_SELECT_MASK			BIT(6)
>> +#define MALI_C55_SINTER_RM_ENABLE_MASK			BIT(7)
>> +
>> +/* Colour Correction Matrix Configuration */
>> +#define MALI_C55_REG_CCM_ENABLE				0x1b07c
>> +#define MALI_C55_CCM_ENABLE_MASK			BIT(0)
>> +#define MALI_C55_REG_CCM_COEF_R_R			0x1b080
>> +#define MALI_C55_REG_CCM_COEF_R_G			0x1b084
>> +#define MALI_C55_REG_CCM_COEF_R_B			0x1b088
>> +#define MALI_C55_REG_CCM_COEF_G_R			0x1b090
>> +#define MALI_C55_REG_CCM_COEF_G_G			0x1b094
>> +#define MALI_C55_REG_CCM_COEF_G_B			0x1b098
>> +#define MALI_C55_REG_CCM_COEF_B_R			0x1b0a0
>> +#define MALI_C55_REG_CCM_COEF_B_G			0x1b0a4
>> +#define MALI_C55_REG_CCM_COEF_B_B			0x1b0a8
>> +#define MALI_C55_CCM_COEF_MASK				GENMASK(12, 0)
>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R			0x1b0b0
>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G			0x1b0b4
>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B			0x1b0b8
>> +#define MALI_C55_CCM_ANTIFOG_GAIN_MASK			GENMASK(11, 0)
>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R		0x1b0c0
>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G		0x1b0c4
>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B		0x1b0c8
>> +#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK		GENMASK(11, 0)
>> +
>> +/*
>> + * The Mali-C55 ISP has up to two output pipes; known as full resolution and
>> + * down scaled. The register space for these is laid out identically, but offset
>> + * by 372 bytes.
>> + */
>> +#define MALI_C55_CAP_DEV_FR_REG_OFFSET		0x0
>> +#define MALI_C55_CAP_DEV_DS_REG_OFFSET		0x174
>> +
>> +#define MALI_C55_REG_CS_CONV_CONFIG(offset)		(0x1c098 + (offset))
>> +#define MALI_C55_CS_CONV_MATRIX_MASK			BIT(0)
>> +#define MALI_C55_CS_CONV_FILTER_MASK			BIT(1)
>> +#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK		BIT(2)
>> +#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK		BIT(3)
>> +#define MALI_C55_REG_Y_WRITER_MODE(offset)		(0x1c0ec + (offset))
>> +#define MALI_C55_REG_UV_WRITER_MODE(offset)		(0x1c144 + (offset))
>> +#define MALI_C55_WRITER_MODE_MASK			GENMASK(4, 0)
>> +#define MALI_C55_WRITER_SUBMODE_MASK			GENMASK(7, 6)
>> +#define MALI_C55_WRITER_FRAME_WRITE_MASK		BIT(9)
>> +#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset)		(0x1c0f0 + (offset))
>> +#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset)		(0x1c148 + (offset))
>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)		((w) << 0)
>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)		((h) << 16)
>> +#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset)	(0x1c0f4 + (offset))
>> +#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset)	(0x1c108 + (offset))
>> +#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
>> +#define MALI_C55_REG_Y_WRITER_BANKS_RESTART		BIT(3)
>> +#define MALI_C55_REG_Y_WRITER_OFFSET(offset)		(0x1c10c + (offset))
>> +#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset)	(0x1c14c + (offset))
>> +#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset)	(0x1c160 + (offset))
>> +#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
>> +#define MALI_C55_REG_UV_WRITER_BANKS_RESTART		BIT(3)
>> +#define MALI_C55_REG_UV_WRITER_OFFSET(offset)		(0x1c164 + (offset))
>> +
>> +#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
>> +#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE		0x18edc
>> +
>> +#define MALI_C55_REG_CROP_EN(offset)			(0x1c028 + (offset))
>> +#define MALI_C55_CROP_ENABLE				BIT(0)
>> +#define MALI_C55_REG_CROP_X_START(offset)		(0x1c02c + (offset))
>> +#define MALI_C55_REG_CROP_Y_START(offset)		(0x1c030 + (offset))
>> +#define MALI_C55_REG_CROP_X_SIZE(offset)		(0x1c034 + (offset))
>> +#define MALI_C55_REG_CROP_Y_SIZE(offset)		(0x1c038 + (offset))
>> +#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset)		(0x1c040 + (offset))
>> +#define MALI_C55_SCALER_TIMEOUT_EN			BIT(4)
>> +#define MALI_C55_SCALER_TIMEOUT(t)			((t) << 16)
>> +#define MALI_C55_REG_SCALER_IN_WIDTH(offset)		(0x1c044 + (offset))
>> +#define MALI_C55_REG_SCALER_IN_HEIGHT(offset)		(0x1c048 + (offset))
>> +#define MALI_C55_REG_SCALER_OUT_WIDTH(offset)		(0x1c04c + (offset))
>> +#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset)		(0x1c050 + (offset))
>> +#define MALI_C55_REG_SCALER_HFILT_TINC(offset)		(0x1c054 + (offset))
>> +#define MALI_C55_REG_SCALER_HFILT_COEF(offset)		(0x1c058 + (offset))
>> +#define MALI_C55_REG_SCALER_VFILT_TINC(offset)		(0x1c05c + (offset))
>> +#define MALI_C55_REG_SCALER_VFILT_COEF(offset)		(0x1c060 + (offset))
>> +
>> +#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset)		(0x1c064 + (offset))
>> +#define MALI_C55_GAMMA_ENABLE_MASK			BIT(0)
>> +#define MALI_C55_REG_GAMMA_GAINS_1(offset)		(0x1c068 + (offset))
>> +#define MALI_C55_GAMMA_GAIN_R_MASK			GENMASK(11, 0)
>> +#define MALI_C55_GAMMA_GAIN_G_MASK			GENMASK(27, 16)
>> +#define MALI_C55_REG_GAMMA_GAINS_2(offset)		(0x1c06c + (offset))
>> +#define MALI_C55_GAMMA_GAIN_B_MASK			GENMASK(11, 0)
>> +#define MALI_C55_REG_GAMMA_OFFSETS_1(offset)		(0x1c070 + (offset))
>> +#define MALI_C55_GAMMA_OFFSET_R_MASK			GENMASK(11, 0)
>> +#define MALI_C55_GAMMA_OFFSET_G_MASK			GENMASK(27, 16)
>> +#define MALI_C55_REG_GAMMA_OFFSETS_2(offset)		(0x1c074 + (offset))
>> +#define MALI_C55_GAMMA_OFFSET_B_MASK			GENMASK(11, 0)
>> +
>> +/* Output DMA Writer */
>> +
>> +#define MALI_C55_OUTPUT_DISABLED		0
>> +#define MALI_C55_OUTPUT_RGB32			1
>> +#define MALI_C55_OUTPUT_A2R10G10B10		2
>> +#define MALI_C55_OUTPUT_RGB565			3
>> +#define MALI_C55_OUTPUT_RGB24			4
>> +#define MALI_C55_OUTPUT_GEN32			5
>> +#define MALI_C55_OUTPUT_RAW16			6
>> +#define MALI_C55_OUTPUT_AYUV			8
>> +#define MALI_C55_OUTPUT_Y410			9
>> +#define MALI_C55_OUTPUT_YUY2			10
>> +#define MALI_C55_OUTPUT_UYVY			11
>> +#define MALI_C55_OUTPUT_Y210			12
>> +#define MALI_C55_OUTPUT_NV12_21			13
>> +#define MALI_C55_OUTPUT_YUV_420_422		17
>> +#define MALI_C55_OUTPUT_P210_P010		19
>> +#define MALI_C55_OUTPUT_YUV422			20
> I'd define this just below the MALI_C55_REG_UV_WRITER_MODE register
> macro.
>
>> +
>> +#define MALI_C55_OUTPUT_PLANE_ALT0		0
>> +#define MALI_C55_OUTPUT_PLANE_ALT1		1
>> +#define MALI_C55_OUTPUT_PLANE_ALT2		2
> Same here ?
>
>> +
>> +#endif /* _MALI_C55_REGISTERS_H */
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>> new file mode 100644
>> index 000000000000..8edae87f1e5f
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>> @@ -0,0 +1,382 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * ARM Mali-C55 ISP Driver - Resizer Coefficients
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#ifndef _MALI_C55_RESIZER_COEFS_H
>> +#define _MALI_C55_RESIZER_COEFS_H
>> +
>> +#include "mali-c55-common.h"
>> +
>> +#define MALI_C55_RESIZER_COEFS_NUM_BANKS	8
>> +#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES	64
> Do these belongs to mali-c55-registers.h ?
>
>> +
>> +static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
>> +	{	/* Bank 0 */
>> +		0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
>> +		0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
>> +		0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
>> +		0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
>> +		0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
>> +		0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
>> +		0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
>> +		0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
>> +		0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
>> +		0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
>> +		0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
>> +		0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
>> +		0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
>> +		0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
>> +		0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
>> +		0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
>> +	},
>> +	{	/* Bank 1 */
>> +		0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
>> +		0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
>> +		0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
>> +		0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
>> +		0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
>> +		0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
>> +		0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
>> +		0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
>> +		0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
>> +		0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
>> +		0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
>> +		0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
>> +		0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
>> +		0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
>> +		0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
>> +		0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
>> +	},
>> +	{	/* Bank 2 */
>> +		0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
>> +		0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
>> +		0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
>> +		0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
>> +		0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
>> +		0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
>> +		0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
>> +		0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
>> +		0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
>> +		0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
>> +		0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
>> +		0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
>> +		0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
>> +		0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
>> +		0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
>> +		0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
>> +	},
>> +	{	/* Bank 3 */
>> +		0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
>> +		0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
>> +		0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
>> +		0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
>> +		0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
>> +		0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
>> +		0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
>> +		0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
>> +		0x20100000, 0x00000010, 0x1f110000, 0x00000010,
>> +		0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
>> +		0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
>> +		0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
>> +		0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
>> +		0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
>> +		0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
>> +		0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
>> +	},
>> +	{	/* Bank 4 */
>> +		0x17090000, 0x00000917, 0x18090000, 0x00000916,
>> +		0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
>> +		0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
>> +		0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
>> +		0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
>> +		0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
>> +		0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
>> +		0x190f0300, 0x00000411, 0x18100300, 0x00000411,
>> +		0x1a100300, 0x00000310, 0x18110400, 0x00000310,
>> +		0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
>> +		0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
>> +		0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
>> +		0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
>> +		0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
>> +		0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
>> +		0x17160800, 0x0000010a, 0x18160900, 0x00000009,
>> +	},
>> +	{	/* Bank 5 */
>> +		0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
>> +		0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
>> +		0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
>> +		0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
>> +		0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
>> +		0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
>> +		0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
>> +		0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
>> +		0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
>> +		0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
>> +		0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
>> +		0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
>> +		0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
>> +		0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
>> +		0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
>> +		0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
>> +	},
>> +	{	/* Bank 6 */
>> +		0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
>> +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
>> +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
>> +		0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
>> +		0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>> +		0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
>> +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>> +		0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
>> +		0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
>> +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
>> +		0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>> +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>> +	},
>> +	{	/* Bank 7 */
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>> +	}
>> +};
>> +
>> +static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
>> +	{	/* Bank 0 */
>> +		0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
>> +		0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
>> +		0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
>> +		0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
>> +		0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
>> +		0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
>> +		0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
>> +		0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
>> +		0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
>> +		0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
>> +		0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
>> +		0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
>> +		0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
>> +		0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
>> +		0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
>> +		0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
>> +	},
>> +	{	/* Bank 1 */
>> +		0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
>> +		0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
>> +		0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
>> +		0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
>> +		0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
>> +		0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
>> +		0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
>> +		0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
>> +		0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
>> +		0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
>> +		0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
>> +		0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
>> +		0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
>> +		0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
>> +		0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
>> +		0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
>> +	},
>> +	{	/* Bank 2 */
>> +		0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
>> +		0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
>> +		0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
>> +		0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
>> +		0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
>> +		0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
>> +		0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
>> +		0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
>> +		0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
>> +		0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
>> +		0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
>> +		0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
>> +		0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
>> +		0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
>> +		0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
>> +		0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
>> +	},
>> +	{	/* Bank 3 */
>> +		0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
>> +		0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
>> +		0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
>> +		0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
>> +		0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
>> +		0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
>> +		0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
>> +		0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
>> +		0x20100000, 0x00000010, 0x1f100000, 0x00000011,
>> +		0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
>> +		0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
>> +		0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
>> +		0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
>> +		0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
>> +		0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
>> +		0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
>> +	},
>> +	{	/* Bank 4 */
>> +		0x17170900, 0x00000009, 0x18160900, 0x00000009,
>> +		0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
>> +		0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
>> +		0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
>> +		0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
>> +		0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
>> +		0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
>> +		0x19110400, 0x0000030f, 0x18110400, 0x00000310,
>> +		0x1a100300, 0x00000310, 0x18100300, 0x00000411,
>> +		0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
>> +		0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
>> +		0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
>> +		0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
>> +		0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
>> +		0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
>> +		0x170a0100, 0x00000816, 0x18090000, 0x00000916,
>> +	},
>> +	{	/* Bank 5 */
>> +		0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
>> +		0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
>> +		0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
>> +		0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
>> +		0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
>> +		0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
>> +		0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
>> +		0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
>> +		0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
>> +		0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
>> +		0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
>> +		0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
>> +		0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
>> +		0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
>> +		0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
>> +		0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
>> +	},
>> +	{	/* Bank 6 */
>> +		0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
>> +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>> +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
>> +		0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
>> +		0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>> +		0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
>> +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>> +		0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
>> +		0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
>> +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
>> +		0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
>> +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
>> +	},
>> +	{	/* Bank 7 */
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>> +	}
>> +};
>> +
>> +struct mali_c55_resizer_coef_bank {
>> +	unsigned int bank;
> This is always equal to the index of the entry in the
> mali_c55_coefficient_banks array, you can drop it.
>
>> +	unsigned int top;
>> +	unsigned int bottom;
> The bottom value of bank N is always equal to the top value of bank N+1.
> You can simplify this by storing a single value.
>
>> +};
>> +
>> +static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
>> +	{
>> +		.bank = 0,
>> +		.top = 1000,
>> +		.bottom = 770,
>> +	},
>> +	{
>> +		.bank = 1,
>> +		.top = 769,
>> +		.bottom = 600,
>> +	},
>> +	{
>> +		.bank = 2,
>> +		.top = 599,
>> +		.bottom = 460,
>> +	},
>> +	{
>> +		.bank = 3,
>> +		.top = 459,
>> +		.bottom = 354,
>> +	},
>> +	{
>> +		.bank = 4,
>> +		.top = 353,
>> +		.bottom = 273,
>> +	},
>> +	{
>> +		.bank = 5,
>> +		.top = 272,
>> +		.bottom = 210,
>> +	},
>> +	{
>> +		.bank = 6,
>> +		.top = 209,
>> +		.bottom = 162,
>> +	},
>> +	{
>> +		.bank = 7,
>> +		.top = 161,
>> +		.bottom = 125,
>> +	},
>> +};
>> +
> A small comment would be nice, such as
>
> /* Select a bank of resizer coefficients, based on the scaling ratio. */
>
>> +static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,
> This function is related to the resizers. Add "rsz" somewhere in the
> function name, and pass a resizer pointer.
>
>> +						unsigned int crop,
>> +						unsigned int scale)
> I think those are the input and output sizes to the scaler. Rename them
> to make it clearer.
>
>> +{
>> +	unsigned int tmp;
> tmp is almost always a bad variable name. Please use a more descriptive
> name, size as rsz_ratio.
>
>> +	unsigned int i;
>> +
>> +	tmp = (scale * 1000U) / crop;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
>> +		if (tmp >= mali_c55_coefficient_banks[i].bottom &&
>> +		    tmp <= mali_c55_coefficient_banks[i].top)
>> +			return mali_c55_coefficient_banks[i].bank;
>> +	}
>> +
>> +	/*
>> +	 * We shouldn't ever get here, in theory. As we have no good choices
>> +	 * simply warn the user and use the first bank of coefficients.
>> +	 */
>> +	dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
>> +	return 0;
>> +}
> And everything else belongs to mali-c55-resizer.c. Drop this header
> file.
>
>> +
>> +#endif /* _MALI_C55_RESIZER_COEFS_H */
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>> new file mode 100644
>> index 000000000000..0a5a2969d3ce
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>> @@ -0,0 +1,779 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ARM Mali-C55 ISP Driver - Image signal processor
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#include <linux/math.h>
>> +#include <linux/minmax.h>
>> +
>> +#include <media/media-entity.h>
>> +#include <media/v4l2-subdev.h>
>> +
>> +#include "mali-c55-common.h"
>> +#include "mali-c55-registers.h"
>> +#include "mali-c55-resizer-coefs.h"
>> +
>> +/* Scaling factor in Q4.20 format. */
>> +#define MALI_C55_RZR_SCALER_FACTOR	(1U << 20)
>> +
>> +static const u32 rzr_non_bypass_src_fmts[] = {
>> +	MEDIA_BUS_FMT_RGB121212_1X36,
>> +	MEDIA_BUS_FMT_YUV10_1X30
>> +};
>> +
>> +static const char * const mali_c55_resizer_names[] = {
>> +	[MALI_C55_RZR_FR] = "resizer fr",
>> +	[MALI_C55_RZR_DS] = "resizer ds",
>> +};
>> +
>> +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
>> +				     struct v4l2_subdev_state *state)
>> +{
>> +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>> +	struct v4l2_mbus_framefmt *fmt;
>> +	struct v4l2_rect *crop;
>> +
>> +	/* Verify if crop should be enabled. */
>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
>> +
>> +	if (fmt->width == crop->width && fmt->height == crop->height)
>> +		return MALI_C55_BYPASS_CROP;
>> +
>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
>> +		       crop->left);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
>> +		       crop->top);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
>> +		       crop->width);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
>> +		       crop->height);
>> +
>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
>> +		       MALI_C55_CROP_ENABLE);
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
>> +					struct v4l2_subdev_state *state)
>> +{
>> +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>> +	struct v4l2_rect *crop, *scale;
>> +	unsigned int h_bank, v_bank;
>> +	u64 h_scale, v_scale;
>> +
>> +	/* Verify if scaling should be enabled. */
>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
>> +	scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
>> +
>> +	if (crop->width == scale->width && crop->height == scale->height)
>> +		return MALI_C55_BYPASS_SCALER;
>> +
>> +	/* Program the V/H scaling factor in Q4.20 format. */
>> +	h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
>> +	v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
>> +
>> +	do_div(h_scale, scale->width);
>> +	do_div(v_scale, scale->height);
>> +
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
>> +		       crop->width);
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
>> +		       crop->height);
>> +
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
>> +		       scale->width);
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
>> +		       scale->height);
>> +
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
>> +		       h_scale);
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
>> +		       v_scale);
>> +
>> +	h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
>> +					     scale->width);
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
>> +		       h_bank);
>> +
>> +	v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
>> +					     scale->height);
>> +	mali_c55_write(mali_c55,
>> +		       MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
>> +		       v_bank);
>> +
>> +	return 0;
>> +}
>> +
>> +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
>> +				 struct v4l2_subdev_state *state)
>> +{
>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>> +	u32 bypass = 0;
>> +
>> +	/* Verify if cropping and scaling should be enabled. */
>> +	bypass |= mali_c55_rzr_program_crop(rzr, state);
>> +	bypass |= mali_c55_rzr_program_resizer(rzr, state);
>> +
>> +	mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
>> +			     MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
>> +			     MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
>> +			     bypass);
>> +}
>> +
>> +/*
>> + * Inspect the routing table to know which of the two (mutually exclusive)
>> + * routes is enabled and return the sink pad id of the active route.
>> + */
>> +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
>> +{
>> +	struct v4l2_subdev_krouting *routing = &state->routing;
>> +	struct v4l2_subdev_route *route;
>> +
>> +	/* A single route is enabled at a time. */
>> +	for_each_active_route(routing, route)
>> +		return route->sink_pad;
>> +
>> +	return MALI_C55_RZR_SINK_PAD;
>> +}
>> +
>> +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
>> +{
>> +	u32 corrected_code = 0;
>> +
>> +	/*
>> +	 * The ISP takes input in a 20-bit format, but can only output 16-bit
>> +	 * RAW bayer data (with the 4 least significant bits from the input
>> +	 * being lost). Return the 16-bit version of the 20-bit input formats.
>> +	 */
>> +	switch (mbus_code) {
>> +	case MEDIA_BUS_FMT_SBGGR20_1X20:
>> +		corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
>> +		break;
>> +	case MEDIA_BUS_FMT_SGBRG20_1X20:
>> +		corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
>> +		break;
>> +	case MEDIA_BUS_FMT_SGRBG20_1X20:
>> +		corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
>> +		break;
>> +	case MEDIA_BUS_FMT_SRGGB20_1X20:
>> +		corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
>> +		break;
>> +	}
>> +
>> +	return corrected_code;
>> +}
>> +
>> +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
>> +				      struct v4l2_subdev_state *state,
>> +				      struct v4l2_subdev_krouting *routing)
>> +{
>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>> +						    sd);
>> +	unsigned int active_sink = UINT_MAX;
>> +	struct v4l2_mbus_framefmt *src_fmt;
>> +	struct v4l2_rect *crop, *compose;
>> +	struct v4l2_subdev_route *route;
>> +	unsigned int active_routes = 0;
>> +	struct v4l2_mbus_framefmt *fmt;
>> +	int ret;
>> +
>> +	ret = v4l2_subdev_routing_validate(sd, routing, 0);
>> +	if (ret)
>> +		return ret;
>> +
>> +	/* Only a single route can be enabled at a time. */
>> +	for_each_active_route(routing, route) {
>> +		if (++active_routes > 1) {
>> +			dev_err(rzr->mali_c55->dev,
>> +				"Only one route can be active");
>> +			return -EINVAL;
>> +		}
>> +
>> +		active_sink = route->sink_pad;
>> +	}
>> +	if (active_sink == UINT_MAX) {
>> +		dev_err(rzr->mali_c55->dev, "One route has to be active");
>> +		return -EINVAL;
>> +	}
>> +
>> +	ret = v4l2_subdev_set_routing(sd, state, routing);
>> +	if (ret) {
>> +		dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
>> +		return ret;
>> +	}
>> +
>> +	fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
>> +	crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
>> +	compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
>> +
>> +	fmt->width = MALI_C55_DEFAULT_WIDTH;
>> +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
>> +	fmt->colorspace = V4L2_COLORSPACE_SRGB;
>> +	fmt->field = V4L2_FIELD_NONE;
>> +
>> +	if (active_sink == MALI_C55_RZR_SINK_PAD) {
>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>> +
>> +		crop->left = crop->top = 0;
>> +		crop->width = MALI_C55_DEFAULT_WIDTH;
>> +		crop->height = MALI_C55_DEFAULT_HEIGHT;
>> +
>> +		*compose = *crop;
>> +	} else {
>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>> +	}
>> +
>> +	/* Propagate the format to the source pad */
>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
>> +					       0);
>> +	*src_fmt = *fmt;
>> +
>> +	/* In the event this is the bypass pad the mbus code needs correcting */
>> +	if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
>> +		src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
>> +				       struct v4l2_subdev_state *state,
>> +				       struct v4l2_subdev_mbus_code_enum *code)
>> +{
>> +	struct v4l2_mbus_framefmt *sink_fmt;
>> +	const struct mali_c55_isp_fmt *fmt;
>> +	unsigned int index = 0;
>> +	u32 sink_pad;
>> +
>> +	switch (code->pad) {
>> +	case MALI_C55_RZR_SINK_PAD:
>> +		if (code->index)
>> +			return -EINVAL;
>> +
>> +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
>> +
>> +		return 0;
>> +	case MALI_C55_RZR_SOURCE_PAD:
>> +		sink_pad = mali_c55_rzr_get_active_sink(state);
>> +		sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
>> +
>> +		/*
>> +		 * If the active route is from the Bypass sink pad, then the
>> +		 * source pad is a simple passthrough of the sink format,
>> +		 * downshifted to 16-bits.
>> +		 */
>> +
>> +		if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>> +			if (code->index)
>> +				return -EINVAL;
>> +
>> +			code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
>> +			if (!code->code)
>> +				return -EINVAL;
>> +
>> +			return 0;
>> +		}
>> +
>> +		/*
>> +		 * If the active route is from the non-bypass sink then we can
>> +		 * select either RGB or conversion to YUV.
>> +		 */
>> +
>> +		if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
>> +			return -EINVAL;
>> +
>> +		code->code = rzr_non_bypass_src_fmts[code->index];
>> +
>> +		return 0;
>> +	case MALI_C55_RZR_SINK_BYPASS_PAD:
>> +		for_each_mali_isp_fmt(fmt) {
>> +			if (index++ == code->index) {
>> +				code->code = fmt->code;
>> +				return 0;
>> +			}
>> +		}
>> +
>> +		break;
>> +	}
>> +
>> +	return -EINVAL;
>> +}
>> +
>> +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
>> +					struct v4l2_subdev_state *state,
>> +					struct v4l2_subdev_frame_size_enum *fse)
>> +{
>> +	if (fse->index)
>> +		return -EINVAL;
>> +
>> +	fse->max_width = MALI_C55_MAX_WIDTH;
>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
>> +	fse->min_width = MALI_C55_MIN_WIDTH;
>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
>> +				     struct v4l2_subdev_state *state,
>> +				     struct v4l2_subdev_format *format)
>> +{
>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>> +	struct v4l2_rect *rect;
>> +	unsigned int sink_pad;
>> +
>> +	/*
>> +	 * Clamp to min/max and then reset crop and compose rectangles to the
>> +	 * newly applied size.
>> +	 */
>> +	clamp_t(unsigned int, fmt->width,
>> +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>> +	clamp_t(unsigned int, fmt->height,
>> +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>> +
>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
>> +	if (sink_pad == MALI_C55_RZR_SINK_PAD) {
>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>> +
>> +		rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
>> +		rect->left = 0;
>> +		rect->top = 0;
>> +		rect->width = fmt->width;
>> +		rect->height = fmt->height;
>> +
>> +		rect = v4l2_subdev_state_get_compose(state,
>> +						     MALI_C55_RZR_SINK_PAD);
>> +		rect->left = 0;
>> +		rect->top = 0;
>> +		rect->width = fmt->width;
>> +		rect->height = fmt->height;
>> +	} else {
>> +		/*
>> +		 * Make sure the media bus code is one of the supported
>> +		 * ISP input media bus codes.
>> +		 */
>> +		if (!mali_c55_isp_is_format_supported(fmt->code))
>> +			fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
>> +	}
>> +
>> +	*v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
>> +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
>> +				       struct v4l2_subdev_state *state,
>> +				       struct v4l2_subdev_format *format)
>> +{
>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>> +						    sd);
>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>> +	struct v4l2_mbus_framefmt *sink_fmt;
>> +	struct v4l2_rect *crop, *compose;
>> +	unsigned int sink_pad;
>> +	unsigned int i;
>> +
>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
>> +	sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
>> +	crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
>> +	compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
>> +
>> +	/* FR Bypass pipe. */
>> +
>> +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>> +		/*
>> +		 * Format on the source pad is the same as the one on the
>> +		 * sink pad, downshifted to 16-bits.
>> +		 */
>> +		fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
>> +		if (!fmt->code)
>> +			return -EINVAL;
>> +
>> +		/* RAW bypass disables scaling and cropping. */
>> +		crop->top = compose->top = 0;
>> +		crop->left = compose->left = 0;
>> +		fmt->width = crop->width = compose->width = sink_fmt->width;
>> +		fmt->height = crop->height = compose->height = sink_fmt->height;
>> +
>> +		*v4l2_subdev_state_get_format(state,
>> +					      MALI_C55_RZR_SOURCE_PAD) = *fmt;
>> +
>> +		return 0;
>> +	}
>> +
>> +	/* Regular processing pipe. */
>> +
>> +	for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
>> +		if (fmt->code == rzr_non_bypass_src_fmts[i])
>> +			break;
>> +	}
>> +
>> +	if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
>> +		dev_dbg(rzr->mali_c55->dev,
>> +			"Unsupported mbus code 0x%x: using default\n",
>> +			fmt->code);
>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>> +	}
>> +
>> +	/*
>> +	 * The source pad format size comes directly from the sink pad
>> +	 * compose rectangle.
>> +	 */
>> +	fmt->width = compose->width;
>> +	fmt->height = compose->height;
>> +
>> +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
>> +				struct v4l2_subdev_state *state,
>> +				struct v4l2_subdev_format *format)
>> +{
>> +	/*
>> +	 * On sink pads fmt is either fixed for the 'regular' processing
>> +	 * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
>> +	 * pad.
>> +	 *
>> +	 * On source pad sizes are the result of crop+compose on the sink
>> +	 * pad sizes, while the format depends on the active route.
>> +	 */
>> +
>> +	if (format->pad != MALI_C55_RZR_SOURCE_PAD)
>> +		return mali_c55_rzr_set_sink_fmt(sd, state, format);
>> +
>> +	return mali_c55_rzr_set_source_fmt(sd, state, format);
>> +}
>> +
>> +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
>> +				      struct v4l2_subdev_state *state,
>> +				      struct v4l2_subdev_selection *sel)
>> +{
>> +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
>> +		return -EINVAL;
>> +
>> +	if (sel->target != V4L2_SEL_TGT_CROP &&
>> +	    sel->target != V4L2_SEL_TGT_COMPOSE)
>> +		return -EINVAL;
>> +
>> +	sel->r = sel->target == V4L2_SEL_TGT_CROP
>> +	       ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
>> +	       : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
>> +				      struct v4l2_subdev_state *state,
>> +				      struct v4l2_subdev_selection *sel)
>> +{
>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>> +						    sd);
>> +	struct v4l2_mbus_framefmt *source_fmt;
>> +	struct v4l2_mbus_framefmt *sink_fmt;
>> +	struct v4l2_rect *crop, *compose;
>> +
>> +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
>> +		return -EINVAL;
>> +
>> +	if (sel->target != V4L2_SEL_TGT_CROP &&
>> +	    sel->target != V4L2_SEL_TGT_COMPOSE)
>> +		return -EINVAL;
>> +
>> +	source_fmt = v4l2_subdev_state_get_format(state,
>> +						  MALI_C55_RZR_SOURCE_PAD);
>> +	sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
>> +	compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
>> +
>> +	/* RAW bypass disables crop/scaling. */
>> +	if (mali_c55_format_is_raw(source_fmt->code)) {
>> +		crop->top = compose->top = 0;
>> +		crop->left = compose->left = 0;
>> +		crop->width = compose->width = sink_fmt->width;
>> +		crop->height = compose->height = sink_fmt->height;
>> +
>> +		sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
>> +
>> +		return 0;
>> +	}
>> +
>> +	/* During streaming, it is allowed to only change the crop rectangle. */
>> +	if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
>> +		return -EINVAL;
>> +
>> +	 /*
>> +	  * Update the desired target and then clamp the crop rectangle to the
>> +	  * sink format sizes and the compose size to the crop sizes.
>> +	  */
>> +	if (sel->target == V4L2_SEL_TGT_CROP)
>> +		*crop = sel->r;
>> +	else
>> +		*compose = sel->r;
>> +
>> +	clamp_t(unsigned int, crop->left, 0,  sink_fmt->width);
>> +	clamp_t(unsigned int, crop->top, 0,  sink_fmt->height);
>> +	clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
>> +		sink_fmt->width - crop->left);
>> +	clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
>> +		sink_fmt->height - crop->top);
>> +
>> +	if (rzr->streaming) {
>> +		/*
>> +		 * Apply at runtime a crop rectangle on the resizer's sink only
>> +		 * if it doesn't require re-programming the scaler output sizes
>> +		 * as it would require changing the output buffer sizes as well.
>> +		 */
>> +		if (sel->r.width < compose->width ||
>> +		    sel->r.height < compose->height)
>> +			return -EINVAL;
>> +
>> +		*crop = sel->r;
>> +		mali_c55_rzr_program(rzr, state);
>> +
>> +		return 0;
>> +	}
>> +
>> +	compose->left = 0;
>> +	compose->top = 0;
>> +	clamp_t(unsigned int, compose->left, 0,  sink_fmt->width);
>> +	clamp_t(unsigned int, compose->top, 0,  sink_fmt->height);
>> +	clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
>> +	clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
>> +
>> +	sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
>> +				    struct v4l2_subdev_state *state,
>> +				    enum v4l2_subdev_format_whence which,
>> +				    struct v4l2_subdev_krouting *routing)
>> +{
>> +	if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
>> +	    media_entity_is_streaming(&sd->entity))
>> +		return -EBUSY;
>> +
>> +	return __mali_c55_rzr_set_routing(sd, state, routing);
>> +}
>> +
>> +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
>> +	.enum_mbus_code		= mali_c55_rzr_enum_mbus_code,
>> +	.enum_frame_size	= mali_c55_rzr_enum_frame_size,
>> +	.get_fmt		= v4l2_subdev_get_fmt,
>> +	.set_fmt		= mali_c55_rzr_set_fmt,
>> +	.get_selection		= mali_c55_rzr_get_selection,
>> +	.set_selection		= mali_c55_rzr_set_selection,
>> +	.set_routing		= mali_c55_rzr_set_routing,
>> +};
>> +
>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
>> +{
>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>> +	struct v4l2_subdev *sd = &rzr->sd;
>> +	struct v4l2_subdev_state *state;
>> +	unsigned int sink_pad;
>> +
>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>> +
>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
>> +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>> +		/* Bypass FR pipe processing if the bypass route is active. */
>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>> +				     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
>> +				     MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
>> +		goto unlock_state;
>> +	}
>> +
>> +	/* Disable bypass and use regular processing. */
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>> +			     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
>> +	mali_c55_rzr_program(rzr, state);
>> +
>> +unlock_state:
>> +	rzr->streaming = true;
>> +	v4l2_subdev_unlock_state(state);
>> +}
>> +
>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
>> +{
>> +	struct v4l2_subdev *sd = &rzr->sd;
>> +	struct v4l2_subdev_state *state;
>> +
>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>> +	rzr->streaming = false;
>> +	v4l2_subdev_unlock_state(state);
>> +}
>> +
>> +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
>> +	.pad	= &mali_c55_resizer_pad_ops,
>> +};
>> +
>> +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
>> +				   struct v4l2_subdev_state *state)
>> +{
>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>> +						    sd);
>> +	struct v4l2_subdev_krouting routing = { };
>> +	struct v4l2_subdev_route *routes;
>> +	unsigned int i;
>> +	int ret;
>> +
>> +	routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
>> +	if (!routes)
>> +		return -ENOMEM;
>> +
>> +	for (i = 0; i < rzr->num_routes; ++i) {
>> +		struct v4l2_subdev_route *route = &routes[i];
>> +
>> +		route->sink_pad = i
>> +				? MALI_C55_RZR_SINK_BYPASS_PAD
>> +				: MALI_C55_RZR_SINK_PAD;
>> +		route->source_pad = MALI_C55_RZR_SOURCE_PAD;
>> +		if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
>> +			route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
>> +	}
>> +
>> +	routing.num_routes = rzr->num_routes;
>> +	routing.routes = routes;
>> +
>> +	ret = __mali_c55_rzr_set_routing(sd, state, &routing);
>> +	kfree(routes);
>> +
>> +	return ret;
>> +}
>> +
>> +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
>> +	.init_state = mali_c55_rzr_init_state,
>> +};
>> +
>> +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
>> +						  unsigned int index)
>> +{
>> +	const unsigned int scaler_filt_coefmem_addrs[][2] = {
>> +		[MALI_C55_RZR_FR] = {
>> +			0x034A8, /* hfilt */
>> +			0x044A8  /* vfilt */
> Lowercase hex constants.
>
>> +		},
>> +		[MALI_C55_RZR_DS] = {
>> +			0x014A8, /* hfilt */
>> +			0x024A8  /* vfilt */
>> +		},
>> +	};
>> +	unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
>> +	unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
>> +	unsigned int i, j;
>> +
>> +	for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
>> +		for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
>> +			mali_c55_write(mali_c55, haddr,
>> +				mali_c55_scaler_h_filter_coefficients[i][j]);
>> +			mali_c55_write(mali_c55, vaddr,
>> +				mali_c55_scaler_v_filter_coefficients[i][j]);
>> +
>> +			haddr += sizeof(u32);
>> +			vaddr += sizeof(u32);
>> +		}
>> +	}
>> +}
>> +
>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
>> +{
>> +	unsigned int i;
>> +	int ret;
>> +
>> +	for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
>> +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
>> +		struct v4l2_subdev *sd = &rzr->sd;
>> +		unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
>> +
>> +		rzr->id = i;
>> +		rzr->streaming = false;
>> +
>> +		if (rzr->id == MALI_C55_RZR_FR)
>> +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
>> +		else
>> +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
>> +
>> +		mali_c55_resizer_program_coefficients(mali_c55, i);
>> +
>> +		v4l2_subdev_init(sd, &mali_c55_resizer_ops);
>> +		sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
>> +			     | V4L2_SUBDEV_FL_STREAMS;
>> +		sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
>> +		sd->internal_ops = &mali_c55_resizer_internal_ops;
>> +		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
>> +			 MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
>> +
>> +		rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
>> +		rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
>> +
>> +		/* Only the FR pipe has a bypass pad. */
>> +		if (rzr->id == MALI_C55_RZR_FR) {
>> +			rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
>> +							MEDIA_PAD_FL_SINK;
>> +			rzr->num_routes = 2;
>> +		} else {
>> +			num_pads -= 1;
>> +			rzr->num_routes = 1;
>> +		}
>> +
>> +		ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
>> +		if (ret)
>> +			return ret;
>> +
>> +		ret = v4l2_subdev_init_finalize(sd);
>> +		if (ret)
>> +			goto err_cleanup;
>> +
>> +		ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>> +		if (ret)
>> +			goto err_cleanup;
>> +
>> +		rzr->mali_c55 = mali_c55;
>> +	}
>> +
>> +	return 0;
>> +
>> +err_cleanup:
>> +	for (; i >= 0; --i) {
>> +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
>> +		struct v4l2_subdev *sd = &rzr->sd;
>> +
>> +		v4l2_subdev_cleanup(sd);
>> +		media_entity_cleanup(&sd->entity);
>> +	}
>> +
>> +	return ret;
>> +}
>> +
>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
>> +{
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
>> +		struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
>> +
>> +		if (!resizer->mali_c55)
>> +			continue;
>> +
>> +		v4l2_device_unregister_subdev(&resizer->sd);
>> +		v4l2_subdev_cleanup(&resizer->sd);
>> +		media_entity_cleanup(&resizer->sd.entity);
>> +	}
>> +}
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>> new file mode 100644
>> index 000000000000..c7e699741c6d
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>> @@ -0,0 +1,402 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ARM Mali-C55 ISP Driver - Test pattern generator
>> + *
>> + * Copyright (C) 2024 Ideas on Board Oy
>> + */
>> +
>> +#include <linux/minmax.h>
>> +#include <linux/string.h>
>> +
>> +#include <media/media-entity.h>
>> +#include <media/v4l2-ctrls.h>
>> +#include <media/v4l2-subdev.h>
>> +
>> +#include "mali-c55-common.h"
>> +#include "mali-c55-registers.h"
>> +
>> +#define MALI_C55_TPG_SRC_PAD		0
>> +#define MALI_C55_TPG_FIXED_HBLANK	0x20
>> +#define MALI_C55_TPG_MAX_VBLANK		0xFFFF
> Lowercase hex constants.
>
>> +#define MALI_C55_TPG_PIXEL_RATE		100000000
> This should be exposed to applications using the V4L2_CID_PIXEL_RATE
> control (read-only).
>
>> +
>> +static const char * const mali_c55_tpg_test_pattern_menu[] = {
>> +	"Flat field",
>> +	"Horizontal gradient",
>> +	"Vertical gradient",
>> +	"Vertical bars",
>> +	"Arbitrary rectangle",
>> +	"White frame on black field"
>> +};
>> +
>> +static const u32 mali_c55_tpg_mbus_codes[] = {
>> +	MEDIA_BUS_FMT_SRGGB20_1X20,
>> +	MEDIA_BUS_FMT_RGB202020_1X60,
>> +};
>> +
>> +static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
>> +				       int *def_vblank, int *min_vblank)
> unsigned int ?
>
>> +{
>> +	unsigned int hts;
>> +	int tgt_fps;
>> +	int vblank;
>> +
>> +	hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
>> +
>> +	/*
>> +	 * The ISP has minimum vertical blanking requirements that must be
>> +	 * adhered to by the TPG. The minimum is a function of the Iridix blocks
>> +	 * clocking requirements and the width of the image and horizontal
>> +	 * blanking, but if we assume the worst case iVariance and sVariance
>> +	 * values then it boils down to the below.
>> +	 */
>> +	*min_vblank = 15 + (120500 / hts);
> I wonder if this should round up.


Maybe? I think the difference is probably too minor to have a practical effect

>
>> +
>> +	/*
>> +	 * We need to set a sensible default vblank for whatever format height
>> +	 * we happen to be given from set_fmt(). This function just targets
>> +	 * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
>> +	 * If we can't get 5fps we'll take whatever the minimum vblank gives us.
>> +	 */
>> +	tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
>> +
>> +	if (tgt_fps < 5)
>> +		vblank = *min_vblank;
>> +	else
>> +		vblank = MALI_C55_TPG_PIXEL_RATE / hts
>> +		       / max(rounddown(tgt_fps, 15), 5);
>> +
>> +	*def_vblank = ALIGN_DOWN(vblank, 2) - format->height;
> "vblank = vblank - height" doesn't seem right. The "else" branch stores
> a vts in vblank, which doesn't seem right either. Maybe you meant
> something like
>
> 	if (tgt_fps < 5)
> 		def_vts = *min_vblank + format->height;
> 	else
> 		def_vts = MALI_C55_TPG_PIXEL_RATE / hts
> 			/ max(rounddown(tgt_fps, 15), 5);
>
> 	*def_vblank = ALIGN_DOWN(def_vts, 2) - format->height;


I did, thank you.

>
>> +}
>> +
>> +static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
>> +{
>> +	struct mali_c55_tpg *tpg = container_of(ctrl->handler,
>> +						struct mali_c55_tpg,
>> +						ctrls.handler);
>> +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
>> +
> Should you return here if the pipeline isn't streaming ?


Yes probably; or if (pm_runtime_get_if_in_use()) would be the usual model I guess.

>
>> +	switch (ctrl->id) {
>> +	case V4L2_CID_TEST_PATTERN:
>> +		mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
>> +			       ctrl->val);
>> +		break;
>> +	case V4L2_CID_VBLANK:
>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
>> +				     MALI_C55_REG_VBLANK_MASK, ctrl->val);
>> +		break;
>> +	default:
>> +		dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
>> +		return -EINVAL;
> Can this happen ?


Not unless somebody breaks something

>
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
>> +	.s_ctrl = &mali_c55_tpg_s_ctrl,
>> +};
>> +
>> +static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
>> +				   struct v4l2_subdev *sd)
>> +{
>> +	struct v4l2_subdev_state *state;
>> +	struct v4l2_mbus_framefmt *fmt;
>> +
>> +	/*
>> +	 * hblank needs setting, but is a read-only control and thus won't be
>> +	 * called during __v4l2_ctrl_handler_setup(). Do it here instead.
>> +	 */
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
>> +			     MALI_C55_REG_HBLANK_MASK,
>> +			     MALI_C55_TPG_FIXED_HBLANK);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>> +			     MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
>> +
>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>> +			     MALI_C55_TEST_PATTERN_RGB_MASK,
>> +			     fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
>> +					  0x01 : 0x0);
>> +
>> +	v4l2_subdev_unlock_state(state);
>> +}
>> +
>> +static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
>> +{
>> +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
>> +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
>> +
>> +	if (!enable) {
>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>> +				MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>> +				MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
>> +		return 0;
>> +	}
>> +
>> +	/*
>> +	 * One might reasonably expect the framesize to be set here
>> +	 * given it's configurable in .set_fmt(), but it's done in the
>> +	 * ISP subdevice's stream on func instead, as the same register
> s/func/function/
>
>> +	 * is also used to indicate the size of the data coming from the
>> +	 * sensor.
>> +	 */
>> +	mali_c55_tpg_configure(mali_c55, sd);
> 	mali_c55_tpg_configure(tpg);
>
>> +	__v4l2_ctrl_handler_setup(sd->ctrl_handler);
>> +
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>> +			     MALI_C55_TEST_PATTERN_ON_OFF,
>> +			     MALI_C55_TEST_PATTERN_ON_OFF);
>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>> +			     MALI_C55_REG_GEN_VIDEO_ON_MASK,
>> +			     MALI_C55_REG_GEN_VIDEO_ON_MASK);
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
>> +	.s_stream = &mali_c55_tpg_s_stream,
> Can we use .enable_streams() and .disable_streams() ?


Yes, with some extra patches from Tomi that Sakari is picking - I'll base on top of those for the v6.

>
>> +};
>> +
>> +static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
>> +				       struct v4l2_subdev_state *state,
>> +				       struct v4l2_subdev_mbus_code_enum *code)
>> +{
>> +	if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>> +		return -EINVAL;
>> +
>> +	code->code = mali_c55_tpg_mbus_codes[code->index];
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
>> +					struct v4l2_subdev_state *state,
>> +					struct v4l2_subdev_frame_size_enum *fse)
>> +{
> You sohuld verify here that fse->code is a supported value and return
> -EINVAL otherwise.
>
>> +	if (fse->index > 0 || fse->pad > sd->entity.num_pads)
> Drop the pad check, it's done in the subdev core already.
>
>> +		return -EINVAL;
>> +
>> +	fse->min_width = MALI_C55_MIN_WIDTH;
>> +	fse->max_width = MALI_C55_MAX_WIDTH;
>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
>> +				struct v4l2_subdev_state *state,
>> +				struct v4l2_subdev_format *format)
>> +{
>> +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>> +	int vblank_def, vblank_min;
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>> +		if (fmt->code == mali_c55_tpg_mbus_codes[i])
>> +			break;
>> +	}
>> +
>> +	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>> +
>> +	/*
>> +	 * The TPG says that the test frame timing generation logic expects a
>> +	 * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
>> +	 * handle anything smaller than 128x128 it seems pointless to allow a
>> +	 * smaller frame.
>> +	 */
>> +	clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
>> +		MALI_C55_MAX_WIDTH);
>> +	clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
>> +		MALI_C55_MAX_HEIGHT);
>> +
>> +	*v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;
> You're allowing userspace to set fmt->field, as well as all the
> colorspace parameters, to random values. I would instead do something
> like
>
> 	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> 		if (format->format.code == mali_c55_tpg_mbus_codes[i])
> 			break;
> 	}
>
> 	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> 		format->format.code = MEDIA_BUS_FMT_SRGGB20_1X20;
>
> 	format->format.width = clamp(format->format.width,
> 				     MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> 	format->format.height = clamp(format->format.height,
> 				      MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>
> 	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> 	fmt->code = format->format.code;
> 	fmt->width = format->format.width;
> 	fmt->height = format->format.height;
>
> 	format->format = *fmt;
>
> Alternatively (which I think I like better),
>
> 	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>
> 	fmt->code = format->format.code;
>
> 	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> 		if (fmt->code == mali_c55_tpg_mbus_codes[i])
> 			break;
> 	}
>
> 	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> 		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>
> 	fmt->width = clamp(format->format.width,
> 			   MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> 	fmt->height = clamp(format->format.height,
> 			    MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>
> 	format->format = *fmt;
>
>> +
>> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY)
>> +		return 0;
>> +
>> +	__mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
>> +	__v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
>> +				 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
>> +	__v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);
> Move those three calls to a separate function, it will be reused below.
> I'd name is mali_c55_tpg_update_vblank(). You can fold
> __mali_c55_tpg_calc_vblank() in it.
>
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
>> +	.enum_mbus_code		= mali_c55_tpg_enum_mbus_code,
>> +	.enum_frame_size	= mali_c55_tpg_enum_frame_size,
>> +	.get_fmt		= v4l2_subdev_get_fmt,
>> +	.set_fmt		= mali_c55_tpg_set_fmt,
>> +};
>> +
>> +static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
>> +	.video	= &mali_c55_tpg_video_ops,
>> +	.pad	= &mali_c55_tpg_pad_ops,
>> +};
>> +
>> +static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
>> +				   struct v4l2_subdev_state *sd_state)
> You name this variable state in every other subdev operation handler.
>
>> +{
>> +	struct v4l2_mbus_framefmt *fmt =
>> +		v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
>> +
>> +	fmt->width = MALI_C55_DEFAULT_WIDTH;
>> +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
>> +	fmt->field = V4L2_FIELD_NONE;
>> +	fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> Initialize the colorspace fields too.
>
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
>> +	.init_state = mali_c55_tpg_init_state,
>> +};
>> +
>> +static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
>> +	struct v4l2_subdev *sd = &mali_c55->tpg.sd;
>> +	struct v4l2_mbus_framefmt *format;
>> +	struct v4l2_subdev_state *state;
>> +	int vblank_def, vblank_min;
>> +	int ret;
>> +
>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>> +	format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>> +
>> +	ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);
> You have 3 controls.
>
>> +	if (ret)
>> +		goto err_unlock;
>> +
>> +	ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
>> +				&mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
>> +				ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
>> +				0, 3, mali_c55_tpg_test_pattern_menu);
>> +
>> +	/*
>> +	 * We fix hblank at the minimum allowed value and control framerate
>> +	 * solely through the vblank control.
>> +	 */
>> +	ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
>> +				&mali_c55_tpg_ctrl_ops,
>> +				V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
>> +				MALI_C55_TPG_FIXED_HBLANK, 1,
>> +				MALI_C55_TPG_FIXED_HBLANK);
>> +	if (ctrls->hblank)
>> +		ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
>> +
>> +	__mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);
> Drop this and initialize the control with default values. You can then
> update the value by calling mali_c55_tpg_update_vblank() in
> mali_c55_register_tpg().
>
> The reason is to share the same mutex between the control handler and
> the subdev active state without having to add a separate mutex in the
> mali_c55_tpg structure. The simplest way to do so is to initialize the
> controls first, set sd->state_lock to point to the control handler lock,
> and call v4l2_subdev_init_finalize() as the last step. As a consequence,
> you can't access the active state when initializing controls.
>
> You can alternatively keep the lock in mali_c55_tpg and set
> sd->state_lock to point to it, but I think that's more complex.
>
>> +	ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
>> +					  &mali_c55_tpg_ctrl_ops,
>> +					  V4L2_CID_VBLANK, vblank_min,
>> +					  MALI_C55_TPG_MAX_VBLANK, 1,
>> +					  vblank_def);
>> +
>> +	if (ctrls->handler.error) {
>> +		dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
>> +		ret = ctrls->handler.error;
>> +		goto err_free_handler;
>> +	}
>> +
>> +	ctrls->handler.lock = &mali_c55->tpg.lock;
> Drop this and drop the mutex. The control handler will use its internal
> mutex.
>
>> +	mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
>> +
>> +	v4l2_subdev_unlock_state(state);
>> +
>> +	return 0;
>> +
>> +err_free_handler:
>> +	v4l2_ctrl_handler_free(&ctrls->handler);
>> +err_unlock:
>> +	v4l2_subdev_unlock_state(state);
>> +	return ret;
>> +}
>> +
>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
>> +	struct v4l2_subdev *sd = &tpg->sd;
>> +	struct media_pad *pad = &tpg->pad;
>> +	int ret;
>> +
>> +	mutex_init(&tpg->lock);
>> +
>> +	v4l2_subdev_init(sd, &mali_c55_tpg_ops);
>> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
>> +	sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
> Should we introduce a TPG function ?


Hmmm I vacillate a bit. I don't see that it would hurt right now, but on the other hand I think 
there's some value in pretending they're sensors - on the grounds that they should be handled 
identically as much as possible. I'd be quite wary if we ever saw "if (sd->entity.function == 
MEDIA_ENT_F_TPG)" somewhere.

>
>> +	sd->internal_ops = &mali_c55_tpg_internal_ops;
>> +	strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
>> +
>> +	pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;
> I don't think MEDIA_PAD_FL_MUST_CONNECT is right.


The docs say "If this flag is set and the pad is linked to any other pad, then at least one of those 
links must be enabled for the entity to be able to stream.", and that's the case here right?

>
>> +	ret = media_entity_pads_init(&sd->entity, 1, pad);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev,
>> +			"Failed to initialize media entity pads\n");
>> +		goto err_destroy_mutex;
>> +	}
>> +
> 	sd->state_lock = sd->ctrl_handler->lock;
>
> to use the same lock for the controls and the active state. You need to
> move this line and the v4l2_subdev_init_finalize() call after
> mali_c55_tpg_init_controls() to get the control handler lock initialized
> first.
>
>> +	ret = v4l2_subdev_init_finalize(sd);
>> +	if (ret)
>> +		goto err_cleanup_media_entity;
>> +
>> +	ret = mali_c55_tpg_init_controls(mali_c55);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev,
>> +			"Error initialising controls\n");
>> +		goto err_cleanup_subdev;
>> +	}
>> +
>> +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
>> +		goto err_free_ctrl_handler;
>> +	}
>> +
>> +	/*
>> +	 * By default the colour settings lead to a very dim image that is
>> +	 * nearly indistinguishable from black on some monitor settings. Ramp
>> +	 * them up a bit so the image is brighter.
>> +	 */
>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
>> +		       MALI_C55_TPG_BACKGROUND_MAX);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
>> +		       MALI_C55_TPG_BACKGROUND_MAX);
>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
>> +		       MALI_C55_TPG_BACKGROUND_MAX);
>> +
>> +	tpg->mali_c55 = mali_c55;
>> +
>> +	return 0;
>> +
>> +err_free_ctrl_handler:
>> +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
>> +err_cleanup_subdev:
>> +	v4l2_subdev_cleanup(sd);
>> +err_cleanup_media_entity:
>> +	media_entity_cleanup(&sd->entity);
>> +err_destroy_mutex:
>> +	mutex_destroy(&tpg->lock);
>> +
>> +	return ret;
>> +}
>> +
>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
>> +
>> +	if (!tpg->mali_c55)
>> +		return;
>> +
>> +	v4l2_device_unregister_subdev(&tpg->sd);
>> +	v4l2_subdev_cleanup(&tpg->sd);
>> +	media_entity_cleanup(&tpg->sd.entity);
>> +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> Free the control handler just after v4l2_device_unregister_subdev() to
> match the order in mali_c55_register_tpg().
>
>> +	mutex_destroy(&tpg->lock);
>> +}

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Laurent Pinchart]

On Mon, Jun 17, 2024 at 06:53:07AM +0000, Sakari Ailus wrote:
> Hi Laurent,
> 
> On Sun, Jun 16, 2024 at 11:38:07PM +0300, Laurent Pinchart wrote:
> > On Sun, Jun 09, 2024 at 06:21:52AM +0000, Sakari Ailus wrote:
> > > On Thu, Jun 06, 2024 at 10:10:14PM +0300, Tomi Valkeinen wrote:
> > > > On 06/06/2024 20:53, Laurent Pinchart wrote:
> > > > > > > > > +			return -EINVAL;
> > > > > > > > > +		}
> > > > > > > > > +
> > > > > > > > > +		active_sink = route->sink_pad;
> > > > > > > > > +	}
> > > > > > > > > +	if (active_sink == UINT_MAX) {
> > > > > > > > > +		dev_err(rzr->mali_c55->dev, "One route has to be active");
> > > > > > > > > +		return -EINVAL;
> > > > > > > > > +	}
> > > > > > >
> > > > > > > The recommended handling of invalid routing is to adjust the routing
> > > > > > > table, not to return errors.
> > > > > >
> > > > > > How should I adjust it ? The error here is due to the fact multiple
> > > > > > routes are set as active, which one should I make active ? the first
> > > > > > one ? Should I go and reset the flags in the subdev_route for the one
> > > > > > that has to be made non-active ?
> > > > >
> > > > > The same way you would adjust an invalid format, you can pick the route
> > > > > you consider should be the default.
> > > > > 
> > > > > I'd like Sakari's and Tomi's opinions on this, as it's a new API and the
> > > > > behaviour is still a bit in flux.
> > > > 
> > > > Well... My opinion is that the driver adjusting the given config parameters
> > > > (for any ioctl) is awful and should be deprecated. If the user asks for X,
> > > > and the driver adjusts it and returns Y, then the user has two options:
> > > > fail, because it didn't get X (after possibly laborious field by field
> > > > checks), or shrug it's virtual shoulders and accept Y and hope that things
> > > > still work even though it wanted X.
> > > 
> > > This is still often the only way to tell what the hardware can do as the
> > > limitations in different cases (cropping and scaling for instance) can be
> > > arbitrary. The other option is that the user space has to know the hardware
> > > capabilities without them being available from the kernel.
> > 
> > For some parameters that make sense (we don't have a try mechanism for
> > ISP parameters buffers for instance), but when it comes to configuring a
> > pipeline, I think a parameters adjustment model is needed when we don't
> > have means to expose constraints in a generic way to userspace. The
> > question is in which category routing falls.
> > 
> > > There could be cases of IOCTLs where returning an error if what was
> > > requested can't be performed exactly is workable in general, but then again
> > > having consistency across IOCTL behaviour is very beneficial as well.
> > > 
> > > If you need something exactly, then I think you should check after the
> > > IOCTL that this is what you also got, beyond the IOCTL succeeding.
> > 
> > I do agree with Tomi that this kind of check can be annoying for
> > applications. In cases where checking the result would be complex, and
> > where there is very little use case for receiving anything but the exact
> > configuration you asked for, adjusting the parameters could increase the
> > implementation complexity on both the kernel side and userspace side for
> > no or very little benefit.
> > 
> > > > But maybe that was an answer to a question you didn't really ask =).
> > > > 
> > > > I think setting it to default routing in case of an error is as fine as any
> > > > other "fix" for the routing. It won't work anyway.
> > > > 
> > > > But if the function sets default routing and returns 0 here, why would it
> > > > return an error from v4l2_subdev_routing_validate()? Should it just set
> > > > default routing in that case too? So should set_routing() ever return an
> > > > error, if we can just set the default routing?
> > 
> > That's a good point. I asked myself the same question after sending my
> > previous e-mail, and wondered if anyone else would notice too :-)
> > 
> > > S_ROUTING is a bit special as it deals with multiple routes and the user
> > > space does have a way to add them incrementally.
> > > 
> > > Perhaps we should document better what the driver is expected to to correct
> > > the routes?
> > 
> > We should document the expected behaviour clearly. After agreeing on the
> > expected behaviour, that is.
> > 
> > > I'd think routes may be added by the driver (as some of them cannot be
> > > disabled for instance) but if a requested route cannot be created, that
> > > should probably be an error.
> > > 
> > > I've copied my current (with all the pending patches) documentation here
> > > <URL:https://www.retiisi.eu/~sailus/v4l2/tmp/streams-doc/userspace-api/media/v4l/dev-subdev.html#streams-multiplexed-media-pads-and-internal-routing>.
> > >
> > > The text does not elaborate what exactly a driver could or should do, apart
> > > from specifying the condition for EINVAL. I think we should specify this in
> > 
> > I don't see mentions of EINVAL related to streams there, am I missing
> > something ?
> > 
> > > greater detail. My original thought wws the adjustment would be done by
> > > adding static routes omitted by the caller, not trying to come up with e.g.
> > > valid pad/stream pairs when user provided invalid ones.
> > > 
> > > Could this correction functionality be limited to returning static routes?
> > 
> > That would make userspace a tad simpler, and wouldn't be hard to do in
> > the kernel, but I wonder if departing from the rule that invalid routing
> > tables result in an error is worth it for such a small gain.
> 
> I'm just referring to our previous decision on the matter. :-)
> 
> Of course an application can do G_ROUTING, toggle the routes it needs to
> and call S_ROUTING again, in order to be (fairly) certain it'll succeed.
> 
> Say, if an application wants to enable an embedded data route, then it'll
> be required to supply the route for the image data as well, even if there's
> no configuration that could be made for that route.
> 
> I'm thinking of fairly generic code here, if a device requires special
> routing setup, it'll need the user space to be aware of it.

I hope the libcamera implementation of the routing API will help us
figuring this out. We can leave the point open for now.

> > > > In the VIDIOC_SUBDEV_S_ROUTING doc we do list some cases where EINVAL or
> > > > E2BIG is returned. But only a few, and I think
> > > > v4l2_subdev_routing_validate() will return errors for many other cases too.
> > > > 
> > > > For what it's worth, the drivers I have written just return an error. It's
> > > > simple for the driver and the user and works. If the consensus is that the
> > > > drivers should instead set the default routing, or somehow mangle the given
> > > > routing to an acceptable form, I can update those drivers accordingly.
> > > > 
> > > > But we probably need to update the docs too to be a bit more clear what
> > > > VIDIOC_SUBDEV_S_ROUTING will do (although are the other ioctls any
> > > > clearer?).
> > > > 
> > > > All that said, I think it's still a bit case-by-case. I don't think the
> > > > drivers should always return an error if they get a routing table that's not
> > > > 100% perfect. E.g. if a device supports two static routes, but the second
> > > > one can be enabled or disabled, the driver should still accept a routing
> > > > table from the user with only the first route present. Etc.
> > > > 
> > > > For the specific case in this patch... I'd prefer returning an error, or if
> > > > that's not ok, set default routing.
> > > 
> > > Not modifying the routing table is another option as well but it may
> > > require separating validating user-provided routes and applying the routes
> > 
> > I'm not sure to follow you here. By not modifying the routing table, do
> > you mean returning an error ? Why would that require separation of
> > validation and configuration ?
> 
> If a driver has already made changes to its routing table, it's a bad idea
> to return an error to the user. In this case changes shouldn't be made.

Ah yes, that should be clearly documented. If an error is returned, no
change to the state shall occur.

> > > to the sub-device state. The default could be useful in principle, too, for
> > > routing-unaware applications but they won't be calling S_ROUTING anyway.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Laurent Pinchart]

Hi Dan,

On Mon, Jun 17, 2024 at 12:41:11PM +0100, Daniel Scally wrote:
> Hi Laurent - sorry, should have included everything in the last reply rather than responding 
> piecemeal. Some more responses and questions below

No worries.

> On 30/05/2024 01:15, Laurent Pinchart wrote:
> > Hi Dan,
> >
> > Thank you for the patch.
> >
> > On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
> >> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
> >> V4L2 and Media Controller compliant and creates subdevices to manage
> >> the ISP itself, its internal test pattern generator as well as the
> >> crop, scaler and output format functionality for each of its two
> >> output devices.
> >>
> >> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
> >> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> >> ---
> >> Changes in v5:
> >>
> >> 	- Reworked input formats - previously we allowed representing input data
> >> 	  as any 8-16 bit format. Now we only allow input data to be represented
> >> 	  by the new 20-bit bayer formats, which is corrected to the equivalent
> >> 	  16-bit format in RAW bypass mode.
> >> 	- Stopped bypassing blocks that we haven't added supporting parameters
> >> 	  for yet.
> >> 	- Addressed most of Sakari's comments from the list
> >>
> >> Changes not yet made in v5:
> >>
> >> 	- The output pipelines can still be started and stopped independently of
> >> 	  one another - I'd like to discuss that more.
> >> 	- the TPG subdev still uses .s_stream() - I need to rebase onto a tree
> >> 	  with working .enable_streams() for a single-source-pad subdevice.
> >>
> >> Changes in v4:
> >>
> >> 	- Reworked mali_c55_update_bits() to internally perform the bit-shift
> >
> > I really don't like that, it makes the code very confusing, even more so
> > as it differs from regmap_update_bits().
> >
> > Look at this for instance:
> >
> > 	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> > 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> > 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> >
> > It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
> > BIT(0).
> >
> > Sorry, I know it will be painful, but this change needs to be reverted.
> >
> >> 	- Reworked the resizer to allow cropping during streaming
> >> 	- Fixed a bug in NV12 output
> >>
> >> Changes in v3:
> >>
> >> 	- Mostly minor fixes suggested by Sakari
> >> 	- Fixed the sequencing of vb2 buffers to be synchronised across the two
> >> 	  capture devices.
> >>
> >> Changes in v2:
> >>
> >> 	- Clock handling
> >> 	- Fixed the warnings raised by the kernel test robot
> >>
> >>   drivers/media/platform/Kconfig                |   1 +
> >>   drivers/media/platform/Makefile               |   1 +
> >>   drivers/media/platform/arm/Kconfig            |   5 +
> >>   drivers/media/platform/arm/Makefile           |   2 +
> >>   drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
> >>   drivers/media/platform/arm/mali-c55/Makefile  |   9 +
> >>   .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
> >>   .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
> >>   .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
> >>   .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
> >>   .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
> >>   .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
> >>   .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
> >>   .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
> >>   14 files changed, 4452 insertions(+)
> >>   create mode 100644 drivers/media/platform/arm/Kconfig
> >>   create mode 100644 drivers/media/platform/arm/Makefile
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >
> > I've skipped review of capture.c and resizer.c as I already have plenty
> > of comments for the other files, and it's getting late. I'll try to
> > review the rest tomorrow.
> >
> >> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> >> index 2d79bfc68c15..c929169766aa 100644
> >> --- a/drivers/media/platform/Kconfig
> >> +++ b/drivers/media/platform/Kconfig
> >> @@ -65,6 +65,7 @@ config VIDEO_MUX
> >>   source "drivers/media/platform/allegro-dvt/Kconfig"
> >>   source "drivers/media/platform/amlogic/Kconfig"
> >>   source "drivers/media/platform/amphion/Kconfig"
> >> +source "drivers/media/platform/arm/Kconfig"
> >>   source "drivers/media/platform/aspeed/Kconfig"
> >>   source "drivers/media/platform/atmel/Kconfig"
> >>   source "drivers/media/platform/broadcom/Kconfig"
> >> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> >> index da17301f7439..9a647abd5218 100644
> >> --- a/drivers/media/platform/Makefile
> >> +++ b/drivers/media/platform/Makefile
> >> @@ -8,6 +8,7 @@
> >>   obj-y += allegro-dvt/
> >>   obj-y += amlogic/
> >>   obj-y += amphion/
> >> +obj-y += arm/
> >>   obj-y += aspeed/
> >>   obj-y += atmel/
> >>   obj-y += broadcom/
> >> diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
> >> new file mode 100644
> >> index 000000000000..4f0764c329c7
> >> --- /dev/null
> >> +++ b/drivers/media/platform/arm/Kconfig
> >> @@ -0,0 +1,5 @@
> >> +# SPDX-License-Identifier: GPL-2.0-only
> >> +
> >> +comment "ARM media platform drivers"
> >> +
> >> +source "drivers/media/platform/arm/mali-c55/Kconfig"
> >> diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
> >> new file mode 100644
> >> index 000000000000..8cc4918725ef
> >> --- /dev/null
> >> +++ b/drivers/media/platform/arm/Makefile
> >> @@ -0,0 +1,2 @@
> >> +# SPDX-License-Identifier: GPL-2.0-only
> >> +obj-y += mali-c55/
> >> diff --git a/drivers/media/platform/arm/mali-c55/Kconfig b/drivers/media/platform/arm/mali-c55/Kconfig
> >> new file mode 100644
> >> index 000000000000..602085e28b01
> >> --- /dev/null
> >> +++ b/drivers/media/platform/arm/mali-c55/Kconfig
> >> @@ -0,0 +1,18 @@
> >> +# SPDX-License-Identifier: GPL-2.0-only
> >> +config VIDEO_MALI_C55
> >> +	tristate "ARM Mali-C55 Image Signal Processor driver"
> >> +	depends on V4L_PLATFORM_DRIVERS
> >> +	depends on VIDEO_DEV && OF
> >> +	depends on ARCH_VEXPRESS || COMPILE_TEST
> >> +	select MEDIA_CONTROLLER
> >> +	select VIDEO_V4L2_SUBDEV_API
> >> +	select VIDEOBUF2_DMA_CONTIG
> >> +	select VIDEOBUF2_VMALLOC
> >> +	select V4L2_FWNODE
> >> +	select GENERIC_PHY_MIPI_DPHY
> >
> > Alphabetical order ?
> >
> >> +	default n
> >
> > That's the default, you don't have to specify ti.
> >
> >> +	help
> >> +	  Enable this to support Arm's Mali-C55 Image Signal Processor.
> >> +
> >> +	  To compile this driver as a module, choose M here: the module
> >> +	  will be called mali-c55.
> >> diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
> >> new file mode 100644
> >> index 000000000000..77dcb2fbf0f4
> >> --- /dev/null
> >> +++ b/drivers/media/platform/arm/mali-c55/Makefile
> >> @@ -0,0 +1,9 @@
> >> +# SPDX-License-Identifier: GPL-2.0
> >> +
> >> +mali-c55-y := mali-c55-capture.o \
> >> +	      mali-c55-core.o \
> >> +	      mali-c55-isp.o \
> >> +	      mali-c55-tpg.o \
> >> +	      mali-c55-resizer.o
> >
> > Alphabetical order here too.
> >
> >> +
> >> +obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
> >> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >> new file mode 100644
> >> index 000000000000..1d539ac9c498
> >> --- /dev/null
> >> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >> @@ -0,0 +1,951 @@
> >> +// SPDX-License-Identifier: GPL-2.0
> >> +/*
> >> + * ARM Mali-C55 ISP Driver - Video capture devices
> >> + *
> >> + * Copyright (C) 2024 Ideas on Board Oy
> >> + */
> >> +
> >> +#include <linux/cleanup.h>
> >> +#include <linux/minmax.h>
> >> +#include <linux/pm_runtime.h>
> >> +#include <linux/string.h>
> >> +#include <linux/videodev2.h>
> >> +
> >> +#include <media/v4l2-dev.h>
> >> +#include <media/v4l2-event.h>
> >> +#include <media/v4l2-ioctl.h>
> >> +#include <media/v4l2-subdev.h>
> >> +#include <media/videobuf2-core.h>
> >> +#include <media/videobuf2-dma-contig.h>
> >> +
> >> +#include "mali-c55-common.h"
> >> +#include "mali-c55-registers.h"
> >> +
> >> +static const struct mali_c55_fmt mali_c55_fmts[] = {
> >> +	/*
> >> +	 * This table is missing some entries which need further work or
> >> +	 * investigation:
> >> +	 *
> >> +	 * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
> >> +	 * Base mode 5 is "Generic Data"
> >> +	 * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
> >> +	 * Base mode 9 seems to have no V4L2 equivalent
> >> +	 * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
> >> +	 * equivalent
> >> +	 */
> >> +	{
> >> +		.fourcc = V4L2_PIX_FMT_ARGB2101010,
> >> +		.mbus_codes = {
> >> +			MEDIA_BUS_FMT_RGB121212_1X36,
> >> +			MEDIA_BUS_FMT_RGB202020_1X60,
> >> +		},
> >> +		.is_raw = false,
> >> +		.registers = {
> >> +			.base_mode = MALI_C55_OUTPUT_A2R10G10B10,
> >> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >> +		}
> >> +	},
> >> +	{
> >> +		.fourcc = V4L2_PIX_FMT_RGB565,
> >> +		.mbus_codes = {
> >> +			MEDIA_BUS_FMT_RGB121212_1X36,
> >> +			MEDIA_BUS_FMT_RGB202020_1X60,
> >> +		},
> >> +		.is_raw = false,
> >> +		.registers = {
> >> +			.base_mode = MALI_C55_OUTPUT_RGB565,
> >> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >> +		}
> >> +	},
> >> +	{
> >> +		.fourcc = V4L2_PIX_FMT_BGR24,
> >> +		.mbus_codes = {
> >> +			MEDIA_BUS_FMT_RGB121212_1X36,
> >> +			MEDIA_BUS_FMT_RGB202020_1X60,
> >> +		},
> >> +		.is_raw = false,
> >> +		.registers = {
> >> +			.base_mode = MALI_C55_OUTPUT_RGB24,
> >> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >> +		}
> >> +	},
> >> +	{
> >> +		.fourcc = V4L2_PIX_FMT_YUYV,
> >> +		.mbus_codes = {
> >> +			MEDIA_BUS_FMT_YUV10_1X30,
> >> +		},
> >> +		.is_raw = false,
> >> +		.registers = {
> >> +			.base_mode = MALI_C55_OUTPUT_YUY2,
> >> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >> +		}
> >> +	},
> >> +	{
> >> +		.fourcc = V4L2_PIX_FMT_UYVY,
> >> +		.mbus_codes = {
> >> +			MEDIA_BUS_FMT_YUV10_1X30,
> >> +		},
> >> +		.is_raw = false,
> >> +		.registers = {
> >> +			.base_mode = MALI_C55_OUTPUT_UYVY,
> >> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >> +		}
> >> +	},
> >> +	{
> >> +		.fourcc = V4L2_PIX_FMT_Y210,
> >> +		.mbus_codes = {
> >> +			MEDIA_BUS_FMT_YUV10_1X30,
> >> +		},
> >> +		.is_raw = false,
> >> +		.registers = {
> >> +			.base_mode = MALI_C55_OUTPUT_Y210,
> >> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >> +		}
> >> +	},
> >> +	/*
> >> +	 * This is something of a hack, the ISP thinks it's running NV12M but
> >> +	 * by setting uv_plane = 0 we simply discard that planes and only output
> >> +	 * the Y-plane.
> >> +	 */
> >> +	{
> >> +		.fourcc = V4L2_PIX_FMT_GREY,
> >> +		.mbus_codes = {
> >> +			MEDIA_BUS_FMT_YUV10_1X30,
> >> +		},
> >> +		.is_raw = false,
> >> +		.registers = {
> >> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
> >> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >> +		}
> >> +	},
> >> +	{
> >> +		.fourcc = V4L2_PIX_FMT_NV12M,
> >> +		.mbus_codes = {
> >> +			MEDIA_BUS_FMT_YUV10_1X30,
> >> +		},
> >> +		.is_raw = false,
> >> +		.registers = {
> >> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
> >> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
> >> +		}
> >> +	},
> >> +	{
> >> +		.fourcc = V4L2_PIX_FMT_NV21M,
> >> +		.mbus_codes = {
> >> +			MEDIA_BUS_FMT_YUV10_1X30,
> >> +		},
> >> +		.is_raw = false,
> >> +		.registers = {
> >> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
> >> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
> >> +		}
> >> +	},
> >> +	/*
> >> +	 * RAW uncompressed formats are all packed in 16 bpp.
> >> +	 * TODO: Expand this list to encompass all possible RAW formats.
> >> +	 */
> >> +	{
> >> +		.fourcc = V4L2_PIX_FMT_SRGGB16,
> >> +		.mbus_codes = {
> >> +			MEDIA_BUS_FMT_SRGGB16_1X16,
> >> +		},
> >> +		.is_raw = true,
> >> +		.registers = {
> >> +			.base_mode = MALI_C55_OUTPUT_RAW16,
> >> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >> +		}
> >> +	},
> >> +	{
> >> +		.fourcc = V4L2_PIX_FMT_SBGGR16,
> >> +		.mbus_codes = {
> >> +			MEDIA_BUS_FMT_SBGGR16_1X16,
> >> +		},
> >> +		.is_raw = true,
> >> +		.registers = {
> >> +			.base_mode = MALI_C55_OUTPUT_RAW16,
> >> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >> +		}
> >> +	},
> >> +	{
> >> +		.fourcc = V4L2_PIX_FMT_SGBRG16,
> >> +		.mbus_codes = {
> >> +			MEDIA_BUS_FMT_SGBRG16_1X16,
> >> +		},
> >> +		.is_raw = true,
> >> +		.registers = {
> >> +			.base_mode = MALI_C55_OUTPUT_RAW16,
> >> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >> +		}
> >> +	},
> >> +	{
> >> +		.fourcc = V4L2_PIX_FMT_SGRBG16,
> >> +		.mbus_codes = {
> >> +			MEDIA_BUS_FMT_SGRBG16_1X16,
> >> +		},
> >> +		.is_raw = true,
> >> +		.registers = {
> >> +			.base_mode = MALI_C55_OUTPUT_RAW16,
> >> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >> +		}
> >> +	},
> >> +};
> >> +
> >> +static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
> >> +					       u32 code)
> >> +{
> >> +	unsigned int i;
> >> +
> >> +	for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
> >> +		if (fmt->mbus_codes[i] == code)
> >> +			return true;
> >> +	}
> >> +
> >> +	return false;
> >> +}
> >> +
> >> +bool mali_c55_format_is_raw(unsigned int mbus_code)
> >> +{
> >> +	unsigned int i;
> >> +
> >> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >> +		if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
> >> +			return mali_c55_fmts[i].is_raw;
> >> +	}
> >> +
> >> +	return false;
> >> +}
> >> +
> >> +static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
> >> +{
> >> +	unsigned int i;
> >> +
> >> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >> +		if (mali_c55_fmts[i].fourcc == pixelformat)
> >> +			return &mali_c55_fmts[i];
> >> +	}
> >> +
> >> +	/*
> >> +	 * If we find no matching pixelformat, we'll just default to the first
> >> +	 * one for now.
> >> +	 */
> >> +
> >> +	return &mali_c55_fmts[0];
> >> +}
> >> +
> >> +static const char * const capture_device_names[] = {
> >> +	"mali-c55 fr",
> >> +	"mali-c55 ds",
> >> +	"mali-c55 3a stats",
> >> +	"mali-c55 params",
> >> +};
> >> +
> >> +static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
> >> +{
> >> +	if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
> >> +		return capture_device_names[0];
> >> +
> >> +	if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
> >> +		return capture_device_names[1];
> >> +
> >> +	return "params/stat not supported yet";
> >> +}
> >> +
> >> +static int mali_c55_link_validate(struct media_link *link)
> >> +{
> >> +	struct video_device *vdev =
> >> +		media_entity_to_video_device(link->sink->entity);
> >> +	struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
> >> +	struct v4l2_subdev *sd =
> >> +		media_entity_to_v4l2_subdev(link->source->entity);
> >> +	const struct v4l2_pix_format_mplane *pix_mp;
> >> +	const struct mali_c55_fmt *cap_fmt;
> >> +	struct v4l2_subdev_format sd_fmt = {
> >> +		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
> >> +		.pad = link->source->index,
> >> +	};
> >> +	int ret;
> >> +
> >> +	ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	pix_mp = &cap_dev->mode.pix_mp;
> >> +	cap_fmt = cap_dev->mode.capture_fmt;
> >> +
> >> +	if (sd_fmt.format.width != pix_mp->width ||
> >> +	    sd_fmt.format.height != pix_mp->height) {
> >> +		dev_dbg(cap_dev->mali_c55->dev,
> >> +			"link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
> >> +			link->source->entity->name, link->source->index,
> >> +			link->sink->entity->name, link->sink->index,
> >> +			sd_fmt.format.width, sd_fmt.format.height,
> >> +			pix_mp->width, pix_mp->height);
> >> +		return -EPIPE;
> >> +	}
> >> +
> >> +	if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
> >> +		dev_dbg(cap_dev->mali_c55->dev,
> >> +			"link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format %p4cc\n",
> >> +			link->source->entity->name, link->source->index,
> >> +			link->sink->entity->name, link->sink->index,
> >> +			sd_fmt.format.code, &pix_mp->pixelformat);
> >> +		return -EPIPE;
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static const struct media_entity_operations mali_c55_media_ops = {
> >> +	.link_validate = mali_c55_link_validate,
> >> +};
> >> +
> >> +static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
> >> +				    unsigned int *num_planes, unsigned int sizes[],
> >> +				    struct device *alloc_devs[])
> >> +{
> >> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >> +	unsigned int i;
> >> +
> >> +	if (*num_planes) {
> >> +		if (*num_planes != cap_dev->mode.pix_mp.num_planes)
> >> +			return -EINVAL;
> >> +
> >> +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >> +			if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
> >> +				return -EINVAL;
> >> +	} else {
> >> +		*num_planes = cap_dev->mode.pix_mp.num_planes;
> >> +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >> +			sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static void mali_c55_buf_queue(struct vb2_buffer *vb)
> >> +{
> >> +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
> >> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> >> +	struct mali_c55_buffer *buf = container_of(vbuf,
> >> +						   struct mali_c55_buffer, vb);
> >> +	unsigned int i;
> >> +
> >> +	buf->plane_done[MALI_C55_PLANE_Y] = false;
> >> +
> >> +	/*
> >> +	 * If we're in a single-plane format we flag the other plane as done
> >> +	 * already so it's dequeued appropriately later
> >> +	 */
> >> +	buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
> >> +
> >> +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
> >> +		unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
> >> +
> >> +		vb2_set_plane_payload(vb, i, size);
> >> +	}
> >> +
> >> +	spin_lock(&cap_dev->buffers.lock);
> >> +	list_add_tail(&buf->queue, &cap_dev->buffers.queue);
> >> +	spin_unlock(&cap_dev->buffers.lock);
> >> +}
> >> +
> >> +static int mali_c55_buf_init(struct vb2_buffer *vb)
> >> +{
> >> +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
> >> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> >> +	struct mali_c55_buffer *buf = container_of(vbuf,
> >> +						   struct mali_c55_buffer, vb);
> >> +	unsigned int i;
> >> +
> >> +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >> +		buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
> >> +{
> >> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >> +
> >> +	guard(spinlock)(&cap_dev->buffers.lock);
> >> +
> >> +	cap_dev->buffers.curr = cap_dev->buffers.next;
> >> +	cap_dev->buffers.next = NULL;
> >> +
> >> +	if (!list_empty(&cap_dev->buffers.queue)) {
> >> +		struct v4l2_pix_format_mplane *pix_mp;
> >> +		const struct v4l2_format_info *info;
> >> +		u32 *addrs;
> >> +
> >> +		pix_mp = &cap_dev->mode.pix_mp;
> >> +		info = v4l2_format_info(pix_mp->pixelformat);
> >> +
> >> +		mali_c55_update_bits(mali_c55,
> >> +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
> >> +		if (cap_dev->mode.capture_fmt->registers.uv_plane)
> >> +			mali_c55_update_bits(mali_c55,
> >> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
> >> +
> >> +		cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
> >> +							 struct mali_c55_buffer,
> >> +							 queue);
> >> +		list_del(&cap_dev->buffers.next->queue);
> >> +
> >> +		addrs = cap_dev->buffers.next->addrs;
> >> +		mali_c55_write(mali_c55,
> >> +			MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
> >> +			addrs[MALI_C55_PLANE_Y]);
> >> +		mali_c55_write(mali_c55,
> >> +			MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
> >> +			addrs[MALI_C55_PLANE_UV]);
> >> +		mali_c55_write(mali_c55,
> >> +			MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
> >> +			pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
> >> +		mali_c55_write(mali_c55,
> >> +			MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
> >> +			pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
> >> +			/ info->hdiv);
> >> +	} else {
> >> +		/*
> >> +		 * If we underflow then we can tell the ISP that we don't want
> >> +		 * to write out the next frame.
> >> +		 */
> >> +		mali_c55_update_bits(mali_c55,
> >> +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >> +		mali_c55_update_bits(mali_c55,
> >> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >> +	}
> >> +}
> >> +
> >> +static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
> >> +				   unsigned int framecount)
> >> +{
> >> +	curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> >> +	curr_buf->vb.field = V4L2_FIELD_NONE;
> >> +	curr_buf->vb.sequence = framecount;
> >> +	vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
> >> +}
> >> +
> >> +/**
> >> + * mali_c55_set_plane_done - mark the plane as written and process the buffer if
> >> + *			     both planes are finished.
> >> + * @cap_dev:  pointer to the fr or ds pipe output
> >> + * @plane:    the plane to mark as completed
> >> + *
> >> + * The Mali C55 ISP has muliplanar outputs for some formats that come with two
> >> + * separate "buffer write completed" interrupts - we need to flag each plane's
> >> + * completion and check whether both planes are done - if so, complete the buf
> >> + * in vb2.
> >> + */
> >> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> >> +			     enum mali_c55_planes plane)
> >> +{
> >> +	struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
> >> +	struct mali_c55_buffer *curr_buf;
> >> +
> >> +	guard(spinlock)(&cap_dev->buffers.lock);
> >> +	curr_buf = cap_dev->buffers.curr;
> >> +
> >> +	/*
> >> +	 * This _should_ never happen. If no buffer was available from vb2 then
> >> +	 * we tell the ISP not to bother writing the next frame, which means the
> >> +	 * interrupts that call this function should never trigger. If it does
> >> +	 * happen then one of our assumptions is horribly wrong - complain
> >> +	 * loudly and do nothing.
> >> +	 */
> >> +	if (!curr_buf) {
> >> +		dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
> >> +			mali_c55_cap_dev_to_name(cap_dev), __func__);
> >> +		return;
> >> +	}
> >> +
> >> +	/* If the other plane is also done... */
> >> +	if (curr_buf->plane_done[~plane & 1]) {
> >> +		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> >> +		cap_dev->buffers.curr = NULL;
> >> +		isp->frame_sequence++;
> >> +	} else {
> >> +		curr_buf->plane_done[plane] = true;
> >> +	}
> >> +}
> >> +
> >> +static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
> >> +{
> >> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >> +
> >> +	mali_c55_update_bits(mali_c55,
> >> +			     MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >> +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >> +	mali_c55_update_bits(mali_c55,
> >> +			     MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >> +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >> +}
> >> +
> >> +static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
> >> +{
> >> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >> +
> >> +	/*
> >> +	 * The Mali ISP can hold up to 5 buffer addresses and simply cycle
> >> +	 * through them, but it's not clear to me that the vb2 queue _guarantees_
> >> +	 * it will queue buffers to the driver in a fixed order, and ensuring
> >> +	 * we call vb2_buffer_done() for the right buffer seems to me to add
> >> +	 * pointless complexity given in multi-context mode we'd need to
> >> +	 * re-write those registers every frame anyway...so we tell the ISP to
> >> +	 * use a single register and update it for each frame.
> >> +	 */
> >> +	mali_c55_update_bits(mali_c55,
> >> +			MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
> >> +			MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
> >> +	mali_c55_update_bits(mali_c55,
> >> +			MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
> >> +			MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
> >> +
> >> +	/*
> >> +	 * We only queue a buffer in the streamon path if this is the first of
> >> +	 * the capture devices to start streaming. If the ISP is already running
> >> +	 * then we rely on the ISP_START interrupt to queue the first buffer for
> >> +	 * this capture device.
> >> +	 */
> >> +	if (mali_c55->pipe.start_count == 1)
> >> +		mali_c55_set_next_buffer(cap_dev);
> >> +}
> >> +
> >> +static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
> >> +					    enum vb2_buffer_state state)
> >> +{
> >> +	struct mali_c55_buffer *buf, *tmp;
> >> +
> >> +	guard(spinlock)(&cap_dev->buffers.lock);
> >> +
> >> +	if (cap_dev->buffers.curr) {
> >> +		vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
> >> +				state);
> >> +		cap_dev->buffers.curr = NULL;
> >> +	}
> >> +
> >> +	if (cap_dev->buffers.next) {
> >> +		vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
> >> +				state);
> >> +		cap_dev->buffers.next = NULL;
> >> +	}
> >> +
> >> +	list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
> >> +		list_del(&buf->queue);
> >> +		vb2_buffer_done(&buf->vb.vb2_buf, state);
> >> +	}
> >> +}
> >> +
> >> +static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
> >> +{
> >> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >> +	struct mali_c55_resizer *rzr = cap_dev->rzr;
> >> +	struct mali_c55_isp *isp = &mali_c55->isp;
> >> +	int ret;
> >> +
> >> +	guard(mutex)(&isp->lock);
> >> +
> >> +	ret = pm_runtime_resume_and_get(mali_c55->dev);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	ret = video_device_pipeline_start(&cap_dev->vdev,
> >> +					  &cap_dev->mali_c55->pipe);
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
> >> +			mali_c55_cap_dev_to_name(cap_dev));
> >> +		goto err_pm_put;
> >> +	}
> >> +
> >> +	mali_c55_cap_dev_stream_enable(cap_dev);
> >> +	mali_c55_rzr_start_stream(rzr);
> >> +
> >> +	/*
> >> +	 * We only start the ISP if we're the only capture device that's
> >> +	 * streaming. Otherwise, it'll already be active.
> >> +	 */
> >> +	if (mali_c55->pipe.start_count == 1) {
> >> +		ret = mali_c55_isp_start_stream(isp);
> >> +		if (ret)
> >> +			goto err_disable_cap_dev;
> >> +	}
> >> +
> >> +	return 0;
> >> +
> >> +err_disable_cap_dev:
> >> +	mali_c55_cap_dev_stream_disable(cap_dev);
> >> +	video_device_pipeline_stop(&cap_dev->vdev);
> >> +err_pm_put:
> >> +	pm_runtime_put(mali_c55->dev);
> >> +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
> >> +{
> >> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >> +	struct mali_c55_resizer *rzr = cap_dev->rzr;
> >> +	struct mali_c55_isp *isp = &mali_c55->isp;
> >> +
> >> +	guard(mutex)(&isp->lock);
> >> +
> >> +	/*
> >> +	 * If one of the other capture nodes is streaming, we shouldn't
> >> +	 * disable the ISP here.
> >> +	 */
> >> +	if (mali_c55->pipe.start_count == 1)
> >> +		mali_c55_isp_stop_stream(&mali_c55->isp);
> >> +
> >> +	mali_c55_rzr_stop_stream(rzr);
> >> +	mali_c55_cap_dev_stream_disable(cap_dev);
> >> +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
> >> +	video_device_pipeline_stop(&cap_dev->vdev);
> >> +	pm_runtime_put(mali_c55->dev);
> >> +}
> >> +
> >> +static const struct vb2_ops mali_c55_vb2_ops = {
> >> +	.queue_setup		= &mali_c55_vb2_queue_setup,
> >> +	.buf_queue		= &mali_c55_buf_queue,
> >> +	.buf_init		= &mali_c55_buf_init,
> >> +	.wait_prepare		= vb2_ops_wait_prepare,
> >> +	.wait_finish		= vb2_ops_wait_finish,
> >> +	.start_streaming	= &mali_c55_vb2_start_streaming,
> >> +	.stop_streaming		= &mali_c55_vb2_stop_streaming,
> >> +};
> >> +
> >> +static const struct v4l2_file_operations mali_c55_v4l2_fops = {
> >> +	.owner = THIS_MODULE,
> >> +	.unlocked_ioctl = video_ioctl2,
> >> +	.open = v4l2_fh_open,
> >> +	.release = vb2_fop_release,
> >> +	.poll = vb2_fop_poll,
> >> +	.mmap = vb2_fop_mmap,
> >> +};
> >> +
> >> +static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
> >> +{
> >> +	const struct mali_c55_fmt *capture_format;
> >> +	const struct v4l2_format_info *info;
> >> +	struct v4l2_plane_pix_format *plane;
> >> +	unsigned int i;
> >> +
> >> +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
> >> +	pix_mp->pixelformat = capture_format->fourcc;
> >> +
> >> +	pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
> >> +			      MALI_C55_MAX_WIDTH);
> >> +	pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
> >> +			       MALI_C55_MAX_HEIGHT);
> >> +
> >> +	pix_mp->field = V4L2_FIELD_NONE;
> >> +	pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
> >> +	pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >> +	pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
> >> +
> >> +	info = v4l2_format_info(pix_mp->pixelformat);
> >> +	pix_mp->num_planes = info->mem_planes;
> >> +	memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
> >> +
> >> +	pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;
> >> +	pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
> >> +				       * pix_mp->height;
> >> +
> >> +	for (i = 1; i < info->comp_planes; i++) {
> >> +		plane = &pix_mp->plane_fmt[i];
> >> +
> >> +		plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
> >> +						   info->hdiv);
> >> +		plane->sizeimage = DIV_ROUND_UP(
> >> +					plane->bytesperline * pix_mp->height,
> >> +					info->vdiv);
> >> +	}
> >> +
> >> +	if (info->mem_planes == 1) {
> >> +		for (i = 1; i < info->comp_planes; i++) {
> >> +			plane = &pix_mp->plane_fmt[i];
> >> +			pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
> >> +		}
> >> +	}
> >> +}
> >> +
> >> +static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
> >> +					   struct v4l2_format *f)
> >> +{
> >> +	mali_c55_try_fmt(&f->fmt.pix_mp);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
> >> +				struct v4l2_pix_format_mplane *pix_mp)
> >> +{
> >> +	const struct mali_c55_fmt *capture_format;
> >> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >> +	const struct v4l2_format_info *info;
> >> +
> >> +	mali_c55_try_fmt(pix_mp);
> >> +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
> >> +	info = v4l2_format_info(pix_mp->pixelformat);
> >> +	if (WARN_ON(!info))
> >> +		return;
> >> +
> >> +	mali_c55_write(mali_c55,
> >> +		       MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >> +		       capture_format->registers.base_mode);
> >> +	mali_c55_write(mali_c55,
> >> +		       MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
> >> +		       MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
> >> +		       MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
> >> +
> >> +	if (info->mem_planes > 1) {
> >> +		mali_c55_write(mali_c55,
> >> +			       MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >> +			       capture_format->registers.base_mode);
> >> +		mali_c55_update_bits(mali_c55,
> >> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >> +				MALI_C55_WRITER_SUBMODE_MASK,
> >> +				capture_format->registers.uv_plane);
> >> +
> >> +		mali_c55_write(mali_c55,
> >> +			MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
> >> +			MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
> >> +			MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
> >> +	}
> >> +
> >> +	if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
> >> +		/*
> >> +		 * TODO: Figure out the colour matrix coefficients and calculate
> >> +		 * and write them here.
> >> +		 */
> >> +
> >> +		mali_c55_write(mali_c55,
> >> +			       MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >> +			       MALI_C55_CS_CONV_MATRIX_MASK);
> >> +
> >> +		if (info->hdiv > 1)
> >> +			mali_c55_update_bits(mali_c55,
> >> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >> +				MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
> >> +		if (info->vdiv > 1)
> >> +			mali_c55_update_bits(mali_c55,
> >> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >> +				MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
> >> +		if (info->hdiv > 1 || info->vdiv > 1)
> >> +			mali_c55_update_bits(mali_c55,
> >> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >> +				MALI_C55_CS_CONV_FILTER_MASK, 0x01);
> >> +	}
> >> +
> >> +	cap_dev->mode.pix_mp = *pix_mp;
> >> +	cap_dev->mode.capture_fmt = capture_format;
> >> +}
> >> +
> >> +static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
> >> +					 struct v4l2_format *f)
> >> +{
> >> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >> +
> >> +	if (vb2_is_busy(&cap_dev->queue))
> >> +		return -EBUSY;
> >> +
> >> +	mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
> >> +					 struct v4l2_format *f)
> >> +{
> >> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >> +
> >> +	f->fmt.pix_mp = cap_dev->mode.pix_mp;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
> >> +					    struct v4l2_fmtdesc *f)
> >> +{
> >> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >> +	unsigned int j = 0;
> >> +	unsigned int i;
> >> +
> >> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >> +		if (f->mbus_code &&
> >> +		    !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
> >> +						       f->mbus_code))
> >> +			continue;
> >> +
> >> +		/* Downscale pipe can't output RAW formats */
> >> +		if (mali_c55_fmts[i].is_raw &&
> >> +		    cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
> >> +			continue;
> >> +
> >> +		if (j++ == f->index) {
> >> +			f->pixelformat = mali_c55_fmts[i].fourcc;
> >> +			return 0;
> >> +		}
> >> +	}
> >> +
> >> +	return -EINVAL;
> >> +}
> >> +
> >> +static int mali_c55_querycap(struct file *file, void *fh,
> >> +			     struct v4l2_capability *cap)
> >> +{
> >> +	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
> >> +	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
> >> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
> >> +	.vidioc_querybuf = vb2_ioctl_querybuf,
> >> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
> >> +	.vidioc_qbuf = vb2_ioctl_qbuf,
> >> +	.vidioc_expbuf = vb2_ioctl_expbuf,
> >> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
> >> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> >> +	.vidioc_streamon = vb2_ioctl_streamon,
> >> +	.vidioc_streamoff = vb2_ioctl_streamoff,
> >> +	.vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
> >> +	.vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
> >> +	.vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
> >> +	.vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
> >> +	.vidioc_querycap = mali_c55_querycap,
> >> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
> >> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
> >> +};
> >> +
> >> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
> >> +{
> >> +	struct v4l2_pix_format_mplane pix_mp;
> >> +	struct mali_c55_cap_dev *cap_dev;
> >> +	struct video_device *vdev;
> >> +	struct vb2_queue *vb2q;
> >> +	unsigned int i;
> >> +	int ret;
> >> +
> >> +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
> >> +		cap_dev = &mali_c55->cap_devs[i];
> >> +		vdev = &cap_dev->vdev;
> >> +		vb2q = &cap_dev->queue;
> >> +
> >> +		/*
> >> +		 * The downscale output pipe is an optional block within the ISP
> >> +		 * so we need to check whether it's actually been fitted or not.
> >> +		 */
> >> +
> >> +		if (i == MALI_C55_CAP_DEV_DS &&
> >> +		    !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
> >> +			continue;
> >> +
> >> +		cap_dev->mali_c55 = mali_c55;
> >> +		mutex_init(&cap_dev->lock);
> >> +		INIT_LIST_HEAD(&cap_dev->buffers.queue);
> >> +
> >> +		switch (i) {
> >> +		case MALI_C55_CAP_DEV_FR:
> >> +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
> >> +			cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
> >> +			break;
> >> +		case MALI_C55_CAP_DEV_DS:
> >> +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
> >> +			cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
> >> +			break;
> >> +		default:
> >> +			mutex_destroy(&cap_dev->lock);
> >> +			ret = -EINVAL;
> >> +			goto err_destroy_mutex;
> >> +		}
> >> +
> >> +		cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
> >> +		ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
> >> +		if (ret) {
> >> +			mutex_destroy(&cap_dev->lock);
> >> +			goto err_destroy_mutex;
> >> +		}
> >> +
> >> +		vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> >> +		vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
> >> +		vb2q->drv_priv = cap_dev;
> >> +		vb2q->mem_ops = &vb2_dma_contig_memops;
> >> +		vb2q->ops = &mali_c55_vb2_ops;
> >> +		vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
> >> +		vb2q->min_queued_buffers = 1;
> >> +		vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> >> +		vb2q->lock = &cap_dev->lock;
> >> +		vb2q->dev = mali_c55->dev;
> >> +
> >> +		ret = vb2_queue_init(vb2q);
> >> +		if (ret) {
> >> +			dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
> >> +				mali_c55_cap_dev_to_name(cap_dev));
> >> +			goto err_cleanup_media_entity;
> >> +		}
> >> +
> >> +		strscpy(cap_dev->vdev.name, capture_device_names[i],
> >> +			sizeof(cap_dev->vdev.name));
> >> +		vdev->release = video_device_release_empty;
> >> +		vdev->fops = &mali_c55_v4l2_fops;
> >> +		vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
> >> +		vdev->lock = &cap_dev->lock;
> >> +		vdev->v4l2_dev = &mali_c55->v4l2_dev;
> >> +		vdev->queue = &cap_dev->queue;
> >> +		vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> >> +				    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
> >> +		vdev->entity.ops = &mali_c55_media_ops;
> >> +		video_set_drvdata(vdev, cap_dev);
> >> +
> >> +		memset(&pix_mp, 0, sizeof(pix_mp));
> >> +		pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
> >> +		pix_mp.width = MALI_C55_DEFAULT_WIDTH;
> >> +		pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
> >> +		mali_c55_set_format(cap_dev, &pix_mp);
> >> +
> >> +		ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
> >> +		if (ret) {
> >> +			dev_err(mali_c55->dev,
> >> +				"%s failed to register video device\n",
> >> +				mali_c55_cap_dev_to_name(cap_dev));
> >> +			goto err_release_vb2q;
> >> +		}
> >> +	}
> >> +
> >> +	return 0;
> >> +
> >> +err_release_vb2q:
> >> +	vb2_queue_release(vb2q);
> >> +err_cleanup_media_entity:
> >> +	media_entity_cleanup(&cap_dev->vdev.entity);
> >> +err_destroy_mutex:
> >> +	mutex_destroy(&cap_dev->lock);
> >> +	mali_c55_unregister_capture_devs(mali_c55);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
> >> +{
> >> +	struct mali_c55_cap_dev *cap_dev;
> >> +	unsigned int i;
> >> +
> >> +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
> >> +		cap_dev = &mali_c55->cap_devs[i];
> >> +
> >> +		if (!video_is_registered(&cap_dev->vdev))
> >> +			continue;
> >> +
> >> +		vb2_video_unregister_device(&cap_dev->vdev);
> >> +		media_entity_cleanup(&cap_dev->vdev.entity);
> >> +		mutex_destroy(&cap_dev->lock);
> >> +	}
> >> +}
> >> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >> new file mode 100644
> >> index 000000000000..2d0c4d152beb
> >> --- /dev/null
> >> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >> @@ -0,0 +1,266 @@
> >> +/* SPDX-License-Identifier: GPL-2.0 */
> >> +/*
> >> + * ARM Mali-C55 ISP Driver - Common definitions
> >> + *
> >> + * Copyright (C) 2024 Ideas on Board Oy
> >> + */
> >> +
> >> +#ifndef _MALI_C55_COMMON_H
> >> +#define _MALI_C55_COMMON_H
> >> +
> >> +#include <linux/clk.h>
> >> +#include <linux/io.h>
> >> +#include <linux/list.h>
> >> +#include <linux/mutex.h>
> >> +#include <linux/scatterlist.h>
> >
> > I don't think this is needed. You're however missing spinlock.h.
> >
> >> +#include <linux/videodev2.h>
> >> +
> >> +#include <media/media-device.h>
> >> +#include <media/v4l2-async.h>
> >> +#include <media/v4l2-ctrls.h>
> >> +#include <media/v4l2-dev.h>
> >> +#include <media/v4l2-device.h>
> >> +#include <media/v4l2-subdev.h>
> >> +#include <media/videobuf2-core.h>
> >> +#include <media/videobuf2-v4l2.h>
> >> +
> >> +#define MALI_C55_DRIVER_NAME		"mali-c55"
> >> +
> >> +/* min and max values for the image sizes */
> >> +#define MALI_C55_MIN_WIDTH		640U
> >> +#define MALI_C55_MIN_HEIGHT		480U
> >> +#define MALI_C55_MAX_WIDTH		8192U
> >> +#define MALI_C55_MAX_HEIGHT		8192U
> >> +#define MALI_C55_DEFAULT_WIDTH		1920U
> >> +#define MALI_C55_DEFAULT_HEIGHT		1080U
> >> +
> >> +#define MALI_C55_DEFAULT_MEDIA_BUS_FMT	MEDIA_BUS_FMT_RGB121212_1X36
> >> +
> >> +struct mali_c55;
> >> +struct mali_c55_cap_dev;
> >> +struct platform_device;
> >
> > You should also forward-declare
> >
> > struct device;
> > struct dma_chan;
> > struct resource;
> >
> >> +
> >> +static const char * const mali_c55_clk_names[] = {
> >> +	"aclk",
> >> +	"hclk",
> >> +};
> >
> > This will end up duplicating the array in each compilation unit, not
> > great. Move it to mali-c55-core.c. You use it in this file just for its
> > size, replace that with a macro that defines the size, or allocate
> > mali_c55.clks dynamically with devm_kcalloc().
> >
> >> +
> >> +enum mali_c55_interrupts {
> >> +	MALI_C55_IRQ_ISP_START,
> >> +	MALI_C55_IRQ_ISP_DONE,
> >> +	MALI_C55_IRQ_MCM_ERROR,
> >> +	MALI_C55_IRQ_BROKEN_FRAME_ERROR,
> >> +	MALI_C55_IRQ_MET_AF_DONE,
> >> +	MALI_C55_IRQ_MET_AEXP_DONE,
> >> +	MALI_C55_IRQ_MET_AWB_DONE,
> >> +	MALI_C55_IRQ_AEXP_1024_DONE,
> >> +	MALI_C55_IRQ_IRIDIX_MET_DONE,
> >> +	MALI_C55_IRQ_LUT_INIT_DONE,
> >> +	MALI_C55_IRQ_FR_Y_DONE,
> >> +	MALI_C55_IRQ_FR_UV_DONE,
> >> +	MALI_C55_IRQ_DS_Y_DONE,
> >> +	MALI_C55_IRQ_DS_UV_DONE,
> >> +	MALI_C55_IRQ_LINEARIZATION_DONE,
> >> +	MALI_C55_IRQ_RAW_FRONTEND_DONE,
> >> +	MALI_C55_IRQ_NOISE_REDUCTION_DONE,
> >> +	MALI_C55_IRQ_IRIDIX_DONE,
> >> +	MALI_C55_IRQ_BAYER2RGB_DONE,
> >> +	MALI_C55_IRQ_WATCHDOG_TIMER,
> >> +	MALI_C55_IRQ_FRAME_COLLISION,
> >> +	MALI_C55_IRQ_UNUSED,
> >> +	MALI_C55_IRQ_DMA_ERROR,
> >> +	MALI_C55_IRQ_INPUT_STOPPED,
> >> +	MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
> >> +	MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
> >> +	MALI_C55_NUM_IRQ_BITS
> >
> > Those are register bits, I think they belong to mali-c55-registers.h,
> > and should probably be macros instead of an enum.
> >
> >> +};
> >> +
> >> +enum mali_c55_isp_pads {
> >> +	MALI_C55_ISP_PAD_SINK_VIDEO,
> >
> > As there's a single sink pad, maybe MALI_C55_ISP_PAD_SINK ? Ah, you're
> > probably preparing for ISP parameters support. It's fine.
> >
> >> +	MALI_C55_ISP_PAD_SOURCE,
> >
> > Then maybe this should be named MALI_C55_ISP_PAD_SOURCE_VIDEO as I
> > assume there will be a stats source pad.
> >
> >> +	MALI_C55_ISP_PAD_SOURCE_BYPASS,
> >> +	MALI_C55_ISP_NUM_PADS,
> >> +};
> >> +
> >> +struct mali_c55_tpg {
> >> +	struct mali_c55 *mali_c55;
> >> +	struct v4l2_subdev sd;
> >> +	struct media_pad pad;
> >> +	struct mutex lock;
> >> +	struct mali_c55_tpg_ctrls {
> >> +		struct v4l2_ctrl_handler handler;
> >> +		struct v4l2_ctrl *test_pattern;
> >
> > Set but never used. You can drop it.
> >
> >> +		struct v4l2_ctrl *hblank;
> >
> > Set and used only once, in the same function. You can make it a local
> > variable.
> >
> >> +		struct v4l2_ctrl *vblank;
> >> +	} ctrls;
> >> +};
> >
> > I wonder if this file should be split, with mali-c55-capture.h,
> > mali-c55-core.h, mali-c55-isp.h, ... I think it could increase
> > readability by clearly separating the different elements. Up to you.
> >
> >> +
> >> +struct mali_c55_isp {
> >> +	struct mali_c55 *mali_c55;
> >> +	struct v4l2_subdev sd;
> >> +	struct media_pad pads[MALI_C55_ISP_NUM_PADS];
> >> +	struct media_pad *remote_src;
> >> +	struct v4l2_async_notifier notifier;
> >
> > I'm tempted to move the notifier to mali_c55, as it's related to
> > components external to the whole ISP, not to the ISP subdev itself.
> > Could you give it a try, to see if it could be done without any drawback
> > ?
> 
> This seems to work fine.
> 
> >
> >> +	struct mutex lock;
> >
> > Locks require a comment to explain what they protect. Same below where
> > applicable (for both mutexes and spinlocks).
> >
> >> +	unsigned int frame_sequence;
> >> +};
> >> +
> >> +enum mali_c55_resizer_ids {
> >> +	MALI_C55_RZR_FR,
> >> +	MALI_C55_RZR_DS,
> >> +	MALI_C55_NUM_RZRS,
> >
> > The usual abbreviation for "resizer" in drivers/media/ is "rsz", not
> > "rzr". I would have said we can leave it as-is as changing it would be a
> > bit annoying, but I then realized that "rzr" is not just unusual, it's
> > actually not used at all. Would you mind applying a sed globally ? :-)
> >
> >> +};
> >> +
> >> +enum mali_c55_rzr_pads {
> >
> > Same enums/structs use abbreviations, some don't. Consistency would
> > help.
> >
> >> +	MALI_C55_RZR_SINK_PAD,
> >> +	MALI_C55_RZR_SOURCE_PAD,
> >> +	MALI_C55_RZR_SINK_BYPASS_PAD,
> >> +	MALI_C55_RZR_NUM_PADS
> >> +};
> >> +
> >> +struct mali_c55_resizer {
> >> +	struct mali_c55 *mali_c55;
> >> +	struct mali_c55_cap_dev *cap_dev;
> >> +	enum mali_c55_resizer_ids id;
> >> +	struct v4l2_subdev sd;
> >> +	struct media_pad pads[MALI_C55_RZR_NUM_PADS];
> >> +	unsigned int num_routes;
> >> +	bool streaming;
> >> +};
> >> +
> >> +enum mali_c55_cap_devs {
> >> +	MALI_C55_CAP_DEV_FR,
> >> +	MALI_C55_CAP_DEV_DS,
> >> +	MALI_C55_NUM_CAP_DEVS
> >> +};
> >> +
> >> +struct mali_c55_fmt {
> >
> > mali_c55_format_info would be a better name I think, as this stores
> > format information, not formats.
> >
> >> +	u32 fourcc;
> >> +	unsigned int mbus_codes[2];
> >
> > A comment to explain why we have two media bus codes would be useful.
> > You can document the whole structure if desired :-)
> >
> >> +	bool is_raw;
> >> +	struct mali_c55_fmt_registers {
> >
> > Make it an anonymous structure, it's never used anywhere else.
> >
> >> +		unsigned int base_mode;
> >> +		unsigned int uv_plane;
> >
> > If those are register field values, use u32 instead of unsigned int.
> >
> >> +	} registers;
> >
> > It's funny, we tend to abbreviate different things, I would have used
> > "regs" here but written "format" in full in the structure name :-)
> >
> >> +};
> >> +
> >> +enum mali_c55_isp_bayer_order {
> >> +	MALI_C55_BAYER_ORDER_RGGB,
> >> +	MALI_C55_BAYER_ORDER_GRBG,
> >> +	MALI_C55_BAYER_ORDER_GBRG,
> >> +	MALI_C55_BAYER_ORDER_BGGR
> >
> > These are registers values too, they belong to mali-c55-registers.h.
> >
> >> +};
> >> +
> >> +struct mali_c55_isp_fmt {
> >
> > mali_c55_isp_format_info
> >
> >> +	u32 code;
> >> +	enum v4l2_pixel_encoding encoding;
> >
> > Here you use v4l2_pixel_encoding, for mali_c55_fmt you use is_raw. Maybe
> > pick the same option for both structures ?
> >
> >> +	enum mali_c55_isp_bayer_order order;
> >> +};
> >> +
> >> +enum mali_c55_planes {
> >> +	MALI_C55_PLANE_Y,
> >> +	MALI_C55_PLANE_UV,
> >> +	MALI_C55_NUM_PLANES
> >> +};
> >> +
> >> +struct mali_c55_buffer {
> >> +	struct vb2_v4l2_buffer vb;
> >> +	bool plane_done[MALI_C55_NUM_PLANES];
> >
> > I think tracking the pending state would simplify the logic in
> > mali_c55_set_plane_done(), which would become
> >
> > 	curr_buf->plane_pending[plane] = false;
> >
> > 	if (!curr_buf->plane_pending[0] && !curr_buf->plane_pending[1]) {
> > 		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> > 		cap_dev->buffers.curr = NULL;
> > 		isp->frame_sequence++;
> > 	}
> >
> > Or a counter may be even easier (and would consume less memory).
>
> I'll do the counter; a  similar function in the stats code does so already.
>
> >> +	struct list_head queue;
> >> +	u32 addrs[MALI_C55_NUM_PLANES];
> >
> > This stores DMA addresses, use dma_addr_t.
> >
> >> +};
> >> +
> >> +struct mali_c55_cap_dev {
> >> +	struct mali_c55 *mali_c55;
> >> +	struct mali_c55_resizer *rzr;
> >> +	struct video_device vdev;
> >> +	struct media_pad pad;
> >> +	struct vb2_queue queue;
> >> +	struct mutex lock;
> >> +	unsigned int reg_offset;
> >
> > Manual handling of the offset everywhere, with parametric macros for the
> > resizer register addresses, isn't very nice. Introduce resizer-specific
> > accessors and capture-specific accessors (mali_c55_rsz_write(), ...)
> > that take a mali_c55_resizer or mali_c55_cap_dev pointer, and handle the
> > offset there. The register macros should loose their offset parameter.
> >
> > You could also use a single set of accessors that would become
> > path/output-specific (mali_c55_path_write() ? mali_c55_output_write()
> > ?), that may make the code easier to read.
> >
> > You can also replace reg_offset with a void __iomem * base, which would
> > avoid the computation at runtime.
> >
> >> +
> >> +	struct mali_c55_mode {
> >
> > Make the structure anonymous.
> >
> >> +		const struct mali_c55_fmt *capture_fmt;
> >> +		struct v4l2_pix_format_mplane pix_mp;
> >> +	} mode;
> >
> > What's a "mode" ? I think I'd name this
> >
> > 	struct {
> > 		const struct mali_c55_fmt *info;
> > 		struct v4l2_pix_format_mplane format;
> > 	} format;
> >
> > Or you could just drop the structure and have
> >
> > 	const struct mali_c55_fmt *format_info;
> > 	struct v4l2_pix_format_mplane format;
> >
> > or something similar.
> >
> >> +
> >> +	struct {
> >> +		spinlock_t lock;
> >> +		struct list_head queue;
> >> +		struct mali_c55_buffer *curr;
> >> +		struct mali_c55_buffer *next;
> >> +	} buffers;
> >> +
> >> +	bool streaming;
> >> +};
> >> +
> >> +enum mali_c55_config_spaces {
> >> +	MALI_C55_CONFIG_PING,
> >> +	MALI_C55_CONFIG_PONG,
> >> +	MALI_C55_NUM_CONFIG_SPACES
> >
> > The last enumerator is not used.
> >
> >> +};
> >> +
> >> +struct mali_c55_ctx {
> >
> > mali_c55_context ?
> >
> >> +	struct mali_c55 *mali_c55;
> >> +	void *registers;
> >
> > Please document this structure and explain that this field points to a
> > copy of the register space in system memory, I was about to write you're
> > missing __iomem :-)
>
> Will do
>
> >> +	phys_addr_t base;
> >> +	spinlock_t lock;
> >> +	struct list_head list;
> >> +};
> >> +
> >> +struct mali_c55 {
> >> +	struct device *dev;
> >> +	struct resource *res;
> >
> > You could possibly drop this field by passing the physical address of
> > the register space from mali_c55_probe() to mali_c55_init_context() as a
> > function parameter.
> >
> >> +	void __iomem *base;
> >> +	struct dma_chan *channel;
> >> +	struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
> >> +
> >> +	u16 capabilities;
> >> +	struct media_device media_dev;
> >> +	struct v4l2_device v4l2_dev;
> >> +	struct media_pipeline pipe;
> >> +
> >> +	struct mali_c55_tpg tpg;
> >> +	struct mali_c55_isp isp;
> >> +	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
> >> +	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
> >> +
> >> +	struct list_head contexts;
> >> +	enum mali_c55_config_spaces next_config;
> >> +};
> >> +
> >> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
> >> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
> >> +		  bool force_hardware);
> >> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
> >> +			  u32 mask, u32 val);
> >> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
> >> +			  enum mali_c55_config_spaces cfg_space);
> >> +
> >> +int mali_c55_register_isp(struct mali_c55 *mali_c55);
> >> +int mali_c55_register_tpg(struct mali_c55 *mali_c55);
> >> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
> >> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
> >> +int mali_c55_register_resizers(struct mali_c55 *mali_c55);
> >> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
> >> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
> >> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
> >> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
> >> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> >> +			     enum mali_c55_planes plane);
> >> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
> >> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
> >> +
> >> +bool mali_c55_format_is_raw(unsigned int mbus_code);
> >> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
> >> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
> >> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
> >> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
> >> +
> >> +const struct mali_c55_isp_fmt *
> >> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
> >> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
> >> +#define for_each_mali_isp_fmt(fmt)\
> >
> > #define for_each_mali_isp_fmt(fmt) \
> >
> >> +	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
> >
> > Looks like parentheses were on sale :-)
>
> Hah
>
> > 	for ((fmt) = NULL; (fmt) = mali_c55_isp_fmt_next(fmt); )
> >
> > This macro is used in two places only, in the mali-c55-isp.c file where
> > open-coding the loop without using mali_c55_isp_fmt_next() would be more
> > efficient, and in mali-c55-resizer.c where a function to return format
> > i'th would be more efficient. I think you can drop the macro and the
> > mali_c55_isp_fmt_next() function.
> >
> >> +
> >> +#endif /* _MALI_C55_COMMON_H */
> >> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >> new file mode 100644
> >> index 000000000000..50caf5ee7474
> >> --- /dev/null
> >> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >> @@ -0,0 +1,767 @@
> >> +// SPDX-License-Identifier: GPL-2.0
> >> +/*
> >> + * ARM Mali-C55 ISP Driver - Core driver code
> >> + *
> >> + * Copyright (C) 2024 Ideas on Board Oy
> >> + */
> >> +
> >> +#include <linux/bitops.h>
> >> +#include <linux/cleanup.h>
> >> +#include <linux/clk.h>
> >> +#include <linux/delay.h>
> >> +#include <linux/device.h>
> >> +#include <linux/dmaengine.h>
> >> +#include <linux/dma-mapping.h>
> >> +#include <linux/interrupt.h>
> >> +#include <linux/iopoll.h>
> >> +#include <linux/ioport.h>
> >> +#include <linux/mod_devicetable.h>
> >> +#include <linux/of.h>
> >> +#include <linux/of_reserved_mem.h>
> >> +#include <linux/platform_device.h>
> >> +#include <linux/pm_runtime.h>
> >> +#include <linux/scatterlist.h>
> >
> > I don't think this is needed.
> >
> > Missing slab.h.
> >
> >> +#include <linux/string.h>
> >> +
> >> +#include <media/media-entity.h>
> >> +#include <media/v4l2-device.h>
> >> +#include <media/videobuf2-dma-contig.h>
> >> +
> >> +#include "mali-c55-common.h"
> >> +#include "mali-c55-registers.h"
> >> +
> >> +static const char * const mali_c55_interrupt_names[] = {
> >> +	[MALI_C55_IRQ_ISP_START] = "ISP start",
> >> +	[MALI_C55_IRQ_ISP_DONE] = "ISP done",
> >> +	[MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
> >> +	[MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
> >> +	[MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
> >> +	[MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
> >> +	[MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
> >> +	[MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
> >> +	[MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
> >> +	[MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
> >> +	[MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
> >> +	[MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
> >> +	[MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
> >> +	[MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
> >> +	[MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
> >> +	[MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
> >> +	[MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
> >> +	[MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
> >> +	[MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
> >> +	[MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
> >> +	[MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
> >> +	[MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
> >> +	[MALI_C55_IRQ_DMA_ERROR] = "DMA error",
> >> +	[MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
> >> +	[MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
> >> +	[MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
> >> +};
> >> +
> >> +static unsigned int config_space_addrs[] = {
> >
> > const
> >
> >> +	[MALI_C55_CONFIG_PING] = 0x0AB6C,
> >> +	[MALI_C55_CONFIG_PONG] = 0x22B2C,
> >
> > Lowercase hex constants.
> >
> > Don't the values belong to mali-c55-registers.h ?
> >
> >> +};
> >> +
> >> +/* System IO
> >
> > /*
> >   * System IO
> >
> >> + *
> >> + * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
> >> + * and 'pong'), with the  expectation that the 'active' space will be left
> >
> > s/the  /the /
> >
> >> + * untouched whilst a frame is being processed and the 'inactive' space
> >> + * configured ready to be passed during the blanking period before the next
> >
> > s/to be passed/to be switched to/ ?
> >
> >> + * frame processing starts. These spaces should ideally be set via DMA transfer
> >> + * from a buffer rather than through individual register set operations. There
> >> + * is also a shared global register space which should be set normally. Of
> >> + * course, the ISP might be included in a system which lacks a suitable DMA
> >> + * engine, and the second configuration space might not be fitted at all, which
> >> + * means we need to support four scenarios:
> >> + *
> >> + * 1. Multi config space, with DMA engine.
> >> + * 2. Multi config space, no DMA engine.
> >> + * 3. Single config space, with DMA engine.
> >> + * 4. Single config space, no DMA engine.
> >> + *
> >> + * The first case is very easy, but the rest present annoying problems. The best
> >> + * way to solve them seems to be simply to replicate the concept of DMAing over
> >> + * the configuration buffer even if there's no DMA engine on the board, for
> >> + * which we rely on memcpy. To facilitate this any read/write call that is made
> >> + * to an address within those config spaces should infact be directed to a
> >> + * buffer that was allocated to hold them rather than the IO memory itself. The
> >> + * actual copy of that buffer to IO mem will happen on interrupt.
> >> + */
> >> +
> >> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
> >> +{
> >> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >> +
> >> +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
> >> +		spin_lock(&ctx->lock);
> >> +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
> >> +		((u32 *)ctx->registers)[addr] = val;
> >> +		spin_unlock(&ctx->lock);
> >> +
> >> +		return;
> >> +	}
> >
> > Ouch. This is likely the second comment you really won't like (after the
> > comment regarding mali_c55_update_bits() at the very top). I apologize
> > in advance.
> 
> All good - it's the thinking that is painful, changing the code is easy :)
> 
> > I really don't like this. Directing writes either to hardware registers
> > or to the shadow registers in the context makes the callers of the
> > read/write accessors very hard to read. The probe code, for instance,
> > mixes writes to hardware registers and writes to the context shadow
> > registers to initialize the value of some of the shadow registers.
> >
> > I'd like to split the read/write accessors into functions that access
> > the hardware registers (that's easy) and functions that access the
> > shadow registers. I think the latter should receive a mali_c55_ctx
> > pointer instead of a mali_c55 pointer to prepare for multi-context
> > support.
> >
> > You can add WARN_ON() guards to the two sets of functions, to ensure
> > that no register from the "other" space gets passed to the wrong
> > function by mistake.
> >
> >> +
> >> +	writel(val, mali_c55->base + addr);
> >> +}
> >> +
> >> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
> >> +		  bool force_hardware)
> >
> > force_hardware is never set to true.
> >
> >> +{
> >> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >> +	u32 val;
> >> +
> >> +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
> >> +		spin_lock(&ctx->lock);
> >> +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
> >> +		val = ((u32 *)ctx->registers)[addr];
> >> +		spin_unlock(&ctx->lock);
> >> +
> >> +		return val;
> >> +	}
> >> +
> >> +	return readl(mali_c55->base + addr);
> >> +}
> >> +
> >> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
> >> +			  u32 mask, u32 val)
> >> +{
> >> +	u32 orig, tmp;
> >> +
> >> +	orig = mali_c55_read(mali_c55, addr, false);
> >> +
> >> +	tmp = orig & ~mask;
> >> +	tmp |= (val << (ffs(mask) - 1)) & mask;
> >> +
> >> +	if (tmp != orig)
> >> +		mali_c55_write(mali_c55, addr, tmp);
> >> +}
> >> +
> >> +static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
> >> +			     dma_addr_t dst, enum dma_data_direction dir)
> >> +{
> >> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> >> +	struct dma_async_tx_descriptor *tx;
> >> +	enum dma_status status;
> >> +	dma_cookie_t cookie;
> >> +
> >> +	tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
> >> +				       MALI_C55_CONFIG_SPACE_SIZE, 0);
> >> +	if (!tx) {
> >> +		dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
> >> +		return -EIO;
> >> +	}
> >> +
> >> +	cookie = dmaengine_submit(tx);
> >> +	if (dma_submit_error(cookie)) {
> >> +		dev_err(mali_c55->dev, "error submitting dma transfer\n");
> >> +		return -EIO;
> >> +	}
> >> +
> >> +	status = dma_sync_wait(mali_c55->channel, cookie);
> >
> > I've just realized this performs a busy-wait :-S See the comment in the
> > probe function about the threaded IRQ handler. I think we'll need to
> > rework all this. It could be done on top though.
>
> It can be switched to an asynchronous transfer quite easily.
> >
> >> +	if (status != DMA_COMPLETE) {
> >> +		dev_err(mali_c55->dev, "dma transfer failed\n");
> >> +		return -EIO;
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
> >> +			     enum mali_c55_config_spaces cfg_space)
> >> +{
> >> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> >> +	struct device *dma_dev = mali_c55->channel->device->dev;
> >> +	dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
> >> +	dma_addr_t dst;
> >> +	int ret;
> >> +
> >> +	guard(spinlock)(&ctx->lock);
> >> +
> >> +	dst = dma_map_single(dma_dev, ctx->registers,
> >> +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
> >> +	if (dma_mapping_error(dma_dev, dst)) {
> >> +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
> >> +		return -EIO;
> >> +	}
> >> +
> >> +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
> >> +	dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
> >> +			 DMA_FROM_DEVICE);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
> >> +		       enum mali_c55_config_spaces cfg_space)
> >> +{
> >> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> >> +	struct device *dma_dev = mali_c55->channel->device->dev;
> >> +	dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
> >> +	dma_addr_t src;
> >> +	int ret;
> >> +
> >> +	guard(spinlock)(&ctx->lock);
> >
> > The code below can take a large amount of time, holding a spinlock will
> > disable interrupts on the local CPU, that's not good :-(
> 
> The intention here is just to prevent the rest of the driver writing into the register space whilst 
> it's being DMA transferred to the hardware - perhaps a different means to signal it's safe is more 
> appropriate?

Are there other parts of the driver that can write to the config space,
or do all writes to the config space go through this function ?

> >> +
> >> +	src = dma_map_single(dma_dev, ctx->registers,
> >> +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
> >> +	if (dma_mapping_error(dma_dev, src)) {
> >> +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
> >> +		return -EIO;
> >> +	}
> >> +
> >> +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
> >> +	dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
> >> +			 DMA_TO_DEVICE);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static int mali_c55_config_read(struct mali_c55_ctx *ctx,
> >> +				enum mali_c55_config_spaces cfg_space)
> >> +{
> >> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> >> +
> >> +	if (mali_c55->channel) {
> >> +		return mali_c55_dma_read(ctx, cfg_space);
> >
> > As this function is used at probe time only, to initialize the context,
> > I think DMA is overkill.
>
> Agreed - I'll switch this to memcpy_fromio()
>
> >> +	} else {
> >> +		memcpy_fromio(ctx->registers,
> >> +			      mali_c55->base + config_space_addrs[cfg_space],
> >> +			      MALI_C55_CONFIG_SPACE_SIZE);
> >> +		return 0;
> >> +	}
> >> +}
> >> +
> >> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
> >> +			  enum mali_c55_config_spaces cfg_space)
> >> +{
> >> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> >> +
> >> +	if (mali_c55->channel) {
> >> +		return mali_c55_dma_write(ctx, cfg_space);
> >> +	} else {
> >> +		memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
> >> +			    ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
> >> +		return 0;
> >> +	}
> >
> > Could you measure the time it typically takes to write the registers
> > using DMA compared to using memcpy_toio() ?
>
> I will test this and come back to you.
>
> >> +}
> >> +
> >> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
> >> +{
> >> +	return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);
> >
> > I think it's too early to tell how multi-context support will look like.
> > I'm fine keeping mali_c55_get_active_context() as changing that would be
> > very intrusive (even if I think it will need to be changed), but the
> > list of contexts is neither the mechanism we'll use, nor something we
> > need now. Drop the list, embed the context in struct mali_c55, and
> > return the pointer to that single context from this function.
> >
> >> +}
> >> +
> >> +static void mali_c55_remove_links(struct mali_c55 *mali_c55)
> >> +{
> >> +	unsigned int i;
> >> +
> >> +	media_entity_remove_links(&mali_c55->tpg.sd.entity);
> >> +	media_entity_remove_links(&mali_c55->isp.sd.entity);
> >> +
> >> +	for (i = 0; i < MALI_C55_NUM_RZRS; i++)
> >> +		media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
> >> +
> >> +	for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
> >> +		media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
> >> +}
> >> +
> >> +static int mali_c55_create_links(struct mali_c55 *mali_c55)
> >> +{
> >> +	int ret;
> >> +
> >> +	/* Test pattern generator to ISP */
> >> +	ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
> >> +				    &mali_c55->isp.sd.entity,
> >> +				    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
> >> +		goto err_remove_links;
> >> +	}
> >> +
> >> +	/* Full resolution resizer pipe. */
> >> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >> +			MALI_C55_ISP_PAD_SOURCE,
> >> +			&mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
> >> +			MALI_C55_RZR_SINK_PAD,
> >> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
> >> +		goto err_remove_links;
> >> +	}
> >> +
> >> +	/* Full resolution bypass. */
> >> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >> +				    MALI_C55_ISP_PAD_SOURCE_BYPASS,
> >> +				    &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
> >> +				    MALI_C55_RZR_SINK_BYPASS_PAD,
> >> +				    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
> >> +		goto err_remove_links;
> >> +	}
> >> +
> >> +	/* Resizer pipe to video capture nodes. */
> >> +	ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
> >> +			MALI_C55_RZR_SOURCE_PAD,
> >> +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
> >> +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev,
> >> +			"failed to link FR resizer and video device\n");
> >> +		goto err_remove_links;
> >> +	}
> >> +
> >> +	/* The downscale pipe is an optional hardware block */
> >> +	if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
> >> +		ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >> +			MALI_C55_ISP_PAD_SOURCE,
> >> +			&mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
> >> +			MALI_C55_RZR_SINK_PAD,
> >> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >> +		if (ret) {
> >> +			dev_err(mali_c55->dev,
> >> +				"failed to link ISP and DS resizer\n");
> >> +			goto err_remove_links;
> >> +		}
> >> +
> >> +		ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
> >> +			MALI_C55_RZR_SOURCE_PAD,
> >> +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
> >> +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >> +		if (ret) {
> >> +			dev_err(mali_c55->dev,
> >> +				"failed to link DS resizer and video device\n");
> >> +			goto err_remove_links;
> >> +		}
> >> +	}
> >> +
> >> +	return 0;
> >> +
> >> +err_remove_links:
> >> +	mali_c55_remove_links(mali_c55);
> >> +	return ret;
> >> +}
> >> +
> >> +static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
> >> +{
> >> +	mali_c55_unregister_tpg(mali_c55);
> >> +	mali_c55_unregister_isp(mali_c55);
> >> +	mali_c55_unregister_resizers(mali_c55);
> >> +	mali_c55_unregister_capture_devs(mali_c55);
> >> +}
> >> +
> >> +static int mali_c55_register_entities(struct mali_c55 *mali_c55)
> >> +{
> >> +	int ret;
> >> +
> >> +	ret = mali_c55_register_tpg(mali_c55);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	ret = mali_c55_register_isp(mali_c55);
> >> +	if (ret)
> >> +		goto err_unregister_entities;
> >> +
> >> +	ret = mali_c55_register_resizers(mali_c55);
> >> +	if (ret)
> >> +		goto err_unregister_entities;
> >> +
> >> +	ret = mali_c55_register_capture_devs(mali_c55);
> >> +	if (ret)
> >> +		goto err_unregister_entities;
> >> +
> >> +	ret = mali_c55_create_links(mali_c55);
> >> +	if (ret)
> >> +		goto err_unregister_entities;
> >> +
> >> +	return 0;
> >> +
> >> +err_unregister_entities:
> >> +	mali_c55_unregister_entities(mali_c55);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
> >> +{
> >> +	u32 product, version, revision, capabilities;
> >> +
> >> +	product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
> >> +	version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
> >> +	revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
> >> +
> >> +	dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
> >> +		 product, version, revision);
> >> +
> >> +	capabilities = mali_c55_read(mali_c55,
> >> +				     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
> >> +				     false);
> >> +	mali_c55->capabilities = (capabilities & 0xffff);
> >> +
> >> +	/* TODO: Might as well start some debugfs */
> >
> > If it's just to expose the version and capabilities, I think that's
> > overkill. It's not needed for debug purpose (you can get it from the
> > kernel log already). debugfs isn't meant to be accessible in production,
> > so an application that would need access to the information wouldn't be
> > able to use it.
> >
> >> +	dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);
> >
> > Combine the two messages into one.
> >
> >> +	return version;
> >> +}
> >> +
> >> +static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
> >> +{
> >> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >> +	u32 curr_config, next_config;
> >> +
> >> +	curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
> >> +	curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
> >> +		      >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
> >> +	next_config = curr_config ^ 1;
> >> +
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >> +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
> >> +	mali_c55_config_write(ctx, next_config ?
> >> +			      MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
> >> +}
> >> +
> >> +static irqreturn_t mali_c55_isr(int irq, void *context)
> >> +{
> >> +	struct device *dev = context;
> >> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >> +	u32 interrupt_status;
> >> +	unsigned int i, j;
> >> +
> >> +	interrupt_status = mali_c55_read(mali_c55,
> >> +					 MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
> >> +					 false);
> >> +	if (!interrupt_status)
> >> +		return IRQ_NONE;
> >> +
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
> >> +		       interrupt_status);
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
> >> +
> >> +	for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
> >> +		if (!(interrupt_status & (1 << i)))
> >> +			continue;
> >> +
> >> +		switch (i) {
> >> +		case MALI_C55_IRQ_ISP_START:
> >> +			mali_c55_isp_queue_event_sof(mali_c55);
> >> +
> >> +			for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
> >> +				mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
> >> +
> >> +			mali_c55_swap_next_config(mali_c55);
> >> +
> >> +			break;
> >> +		case MALI_C55_IRQ_ISP_DONE:
> >> +			/*
> >> +			 * TODO: Where the ISP has no Pong config fitted, we'd
> >> +			 * have to do the mali_c55_swap_next_config() call here.
> >> +			 */
> >> +			break;
> >> +		case MALI_C55_IRQ_FR_Y_DONE:
> >> +			mali_c55_set_plane_done(
> >> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
> >> +				MALI_C55_PLANE_Y);
> >> +			break;
> >> +		case MALI_C55_IRQ_FR_UV_DONE:
> >> +			mali_c55_set_plane_done(
> >> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
> >> +				MALI_C55_PLANE_UV);
> >> +			break;
> >> +		case MALI_C55_IRQ_DS_Y_DONE:
> >> +			mali_c55_set_plane_done(
> >> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
> >> +				MALI_C55_PLANE_Y);
> >> +			break;
> >> +		case MALI_C55_IRQ_DS_UV_DONE:
> >> +			mali_c55_set_plane_done(
> >> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
> >> +				MALI_C55_PLANE_UV);
> >> +			break;
> >> +		default:
> >> +			/*
> >> +			 * Only the above interrupts are currently unmasked. If
> >> +			 * we receive anything else here then something weird
> >> +			 * has gone on.
> >> +			 */
> >> +			dev_err(dev, "masked interrupt %s triggered\n",
> >> +				mali_c55_interrupt_names[i]);
> >> +		}
> >> +	}
> >> +
> >> +	return IRQ_HANDLED;
> >> +}
> >> +
> >> +static int mali_c55_init_context(struct mali_c55 *mali_c55)
> >> +{
> >> +	struct mali_c55_ctx *ctx;
> >> +	int ret;
> >> +
> >> +	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
> >> +	if (!ctx) {
> >> +		dev_err(mali_c55->dev, "failed to allocate new context\n");
> >
> > No need for an error message when memory allocation fails.
> >
> >> +		return -ENOMEM;
> >> +	}
> >> +
> >> +	ctx->base = mali_c55->res->start;
> >> +	ctx->mali_c55 = mali_c55;
> >> +
> >> +	ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
> >> +				 GFP_KERNEL | GFP_DMA);
> >> +	if (!ctx->registers) {
> >> +		ret = -ENOMEM;
> >> +		goto err_free_ctx;
> >> +	}
> >> +
> >> +	/*
> >> +	 * The allocated memory is empty, we need to load the default
> >> +	 * register settings. We just read Ping; it's identical to Pong.
> >> +	 */
> >> +	ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
> >> +	if (ret)
> >> +		goto err_free_registers;
> >> +
> >> +	list_add_tail(&ctx->list, &mali_c55->contexts);
> >> +
> >> +	/*
> >> +	 * Some features of the ISP need to be disabled by default and only
> >> +	 * enabled at the same time as they're configured by a parameters buffer
> >> +	 */
> >> +
> >> +	/* Bypass the sqrt and square compression and expansion modules */
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
> >> +			     MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
> >> +			     MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
> >> +
> >> +	/* Bypass the temper module */
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
> >> +		       MALI_C55_REG_BYPASS_2_TEMPER);
> >> +
> >> +	/* Bypass the colour noise reduction  */
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
> >> +		       MALI_C55_REG_BYPASS_4_CNR);
> >> +
> >> +	/* Disable the sinter module */
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
> >> +			     MALI_C55_SINTER_ENABLE_MASK, 0x00);
> >> +
> >> +	/* Disable the RGB Gamma module for each output */
> >> +	mali_c55_write(
> >> +		mali_c55,
> >> +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
> >> +		0x00);
> >> +	mali_c55_write(
> >> +		mali_c55,
> >> +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
> >> +		0x00);
> >> +
> >> +	/* Disable the colour correction matrix */
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
> >> +
> >> +	return 0;
> >> +
> >> +err_free_registers:
> >> +	kfree(ctx->registers);
> >> +err_free_ctx:
> >> +	kfree(ctx);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static int mali_c55_runtime_resume(struct device *dev)
> >> +{
> >> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >> +	int ret;
> >> +
> >> +	ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
> >> +				      mali_c55->clks);
> >> +	if (ret)
> >> +		dev_err(mali_c55->dev, "failed to enable clocks\n");
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static int mali_c55_runtime_suspend(struct device *dev)
> >> +{
> >> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >> +
> >> +	clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
> >> +	return 0;
> >> +}
> >> +
> >> +static const struct dev_pm_ops mali_c55_pm_ops = {
> >> +	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> >> +				pm_runtime_force_resume)
> >> +	SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
> >> +			   NULL)
> >> +};
> >> +
> >> +static int mali_c55_probe(struct platform_device *pdev)
> >> +{
> >> +	struct device *dev = &pdev->dev;
> >> +	struct mali_c55 *mali_c55;
> >> +	dma_cap_mask_t mask;
> >> +	u32 version;
> >> +	int ret;
> >> +	u32 val;
> >> +
> >> +	mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
> >> +	if (!mali_c55)
> >> +		return dev_err_probe(dev, -ENOMEM,
> >> +				     "failed to allocate memory\n");
> >
> > 		return -ENOMEM;
> >
> > There's no need to print messages for memory allocation failures.
> >
> >> +
> >> +	mali_c55->dev = dev;
> >> +	platform_set_drvdata(pdev, mali_c55);
> >> +
> >> +	mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
> >> +								&mali_c55->res);
> >> +	if (IS_ERR(mali_c55->base))
> >> +		return dev_err_probe(dev, PTR_ERR(mali_c55->base),
> >> +				     "failed to map IO memory\n");
> >> +
> >> +	ret = platform_get_irq(pdev, 0);
> >> +	if (ret < 0)
> >> +		return dev_err_probe(dev, ret, "failed to get interrupt num\n");
> >
> > s/ num// or s/num/number/
> >
> >> +
> >> +	ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
> >> +					mali_c55_isr, IRQF_ONESHOT,
> >> +					dev_driver_string(&pdev->dev),
> >> +					&pdev->dev);
> >
> > Requested the IRQ should be done much lower, after you have initialized
> > everything, or an IRQ that would fire early would have really bad
> > consequences.
> >
> > A comment to explain why you need a threaded interrupt handler would be
> > good. I assume it is due only to the need to transfer the registers
> > using DMA. I wonder if we should then split the interrupt handler in
> > two, with a non-threaded part for the operations that can run quickly,
> > and a threaded part for the reprogramming.
> 
> Instead of passing NULL for the top handler you mean?

Yes.

> > It may also be that we could just start the DMA transfer in the
> > non-threaded handler without waiting synchronously for it to complete.
> > That would be a bigger change, and would require checking race
> > conditions carefully. On the other hand, I'm a bit concerned about the
> > current implementation, have you tested what happens if the DMA transfer
> > takes too long to complete, and spans frame boundaries ?
> 
> No; I can force a delay in the DMA engine and see how it behaves. The stats buffers are currently 
> DMAd asynchronously; I don't think it'd be a huge change to make the configuration buffer handling 
> work that way too.

That would be nice. Race conditions will need to be carefully considered
(but they should, anyway).

> >> +	if (ret)
> >> +		return dev_err_probe(dev, ret, "failed to request irq\n");
> >> +
> >> +	for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
> >> +		mali_c55->clks[i].id = mali_c55_clk_names[i];
> >> +
> >> +	ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
> >> +	if (ret)
> >> +		return dev_err_probe(dev, ret, "failed to acquire clocks\n");
> >> +
> >> +	pm_runtime_enable(&pdev->dev);
> >> +
> >> +	ret = pm_runtime_resume_and_get(&pdev->dev);
> >> +	if (ret)
> >> +		goto err_pm_runtime_disable;
> >> +
> >> +	of_reserved_mem_device_init(dev);
> >
> > I'd move this, the vb2_dma_contig_set_max_seg_size() call and the
> > dma_cap_* calls before pm_runtime_enable() as they don't need the device
> > to be powered.
> >
> >> +	version = mali_c55_check_hwcfg(mali_c55);
> >> +	vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
> >> +
> >> +	/* Use "software only" context management. */
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >> +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> >> +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> >
> > You handle that in mali_c55_isp_start(), does the register have to be
> > set here too ?
> >
> >> +
> >> +	dma_cap_zero(mask);
> >> +	dma_cap_set(DMA_MEMCPY, mask);
> >> +
> >> +	/*
> >> +	 * No error check, because we will just fallback on memcpy if there is
> >> +	 * no usable DMA channel on the system.
> >> +	 */
> >> +	mali_c55->channel = dma_request_channel(mask, NULL, NULL);
> >> +
> >> +	INIT_LIST_HEAD(&mali_c55->contexts);
> >> +	ret = mali_c55_init_context(mali_c55);
> >> +	if (ret)
> >> +		goto err_release_dma_channel;
> >> +
> >
> > I'd move all the code from here ...
> >
> >> +	mali_c55->media_dev.dev = dev;
> >> +	strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
> >> +		sizeof(mali_c55->media_dev.model));
> >> +	mali_c55->media_dev.hw_revision = version;
> >> +
> >> +	media_device_init(&mali_c55->media_dev);
> >> +	ret = media_device_register(&mali_c55->media_dev);
> >> +	if (ret)
> >> +		goto err_cleanup_media_device;
> >> +
> >> +	mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
> >> +	ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
> >> +	if (ret) {
> >> +		dev_err(dev, "failed to register V4L2 device\n");
> >> +		goto err_unregister_media_device;
> >> +	};
> >> +
> >> +	ret = mali_c55_register_entities(mali_c55);
> >> +	if (ret) {
> >> +		dev_err(dev, "failed to register entities\n");
> >> +		goto err_unregister_v4l2_device;
> >> +	}
> >
> > ... to here to a separate function, or maybe fold it all in
> > mali_c55_register_entities() (which should the be renamed). Same thing
> > for the cleanup code.
> >
> >> +
> >> +	/* Set safe stop to ensure we're in a non-streaming state */
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> >> +		       MALI_C55_INPUT_SAFE_STOP);
> >> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >> +
> >> +	/*
> >> +	 * We're ready to process interrupts. Clear any that are set and then
> >> +	 * unmask them for processing.
> >> +	 */
> >> +	mali_c55_write(mali_c55, 0x30, 0xffffffff);
> >> +	mali_c55_write(mali_c55, 0x34, 0xffffffff);
> >> +	mali_c55_write(mali_c55, 0x40, 0x01);
> >> +	mali_c55_write(mali_c55, 0x40, 0x00);
> >
> > Please replace the register addresses with macros.
> >
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);
> >
> > The value should use the interrupt bits macros.
> >
> >> +
> >> +	pm_runtime_put(&pdev->dev);
> >
> > Once power gets cut, the registers your programmed above may be lost. I
> > think you should programe them in the runtime PM resume handler.
> >
> >> +
> >> +	return 0;
> >> +
> >> +err_unregister_v4l2_device:
> >> +	v4l2_device_unregister(&mali_c55->v4l2_dev);
> >> +err_unregister_media_device:
> >> +	media_device_unregister(&mali_c55->media_dev);
> >> +err_cleanup_media_device:
> >> +	media_device_cleanup(&mali_c55->media_dev);
> >> +err_release_dma_channel:
> >> +	dma_release_channel(mali_c55->channel);
> >> +err_pm_runtime_disable:
> >> +	pm_runtime_disable(&pdev->dev);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static void mali_c55_remove(struct platform_device *pdev)
> >> +{
> >> +	struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
> >> +	struct mali_c55_ctx *ctx, *tmp;
> >> +
> >> +	list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
> >> +		list_del(&ctx->list);
> >> +		kfree(ctx->registers);
> >> +		kfree(ctx);
> >> +	}
> >> +
> >> +	mali_c55_remove_links(mali_c55);
> >> +	mali_c55_unregister_entities(mali_c55);
> >> +	v4l2_device_put(&mali_c55->v4l2_dev);
> >> +	media_device_unregister(&mali_c55->media_dev);
> >> +	media_device_cleanup(&mali_c55->media_dev);
> >> +	dma_release_channel(mali_c55->channel);
> >> +}
> >> +
> >> +static const struct of_device_id mali_c55_of_match[] = {
> >> +	{ .compatible = "arm,mali-c55", },
> >> +	{},
> >
> > 	{ /* Sentinel */ },
> >
> >> +};
> >> +MODULE_DEVICE_TABLE(of, mali_c55_of_match);
> >> +
> >> +static struct platform_driver mali_c55_driver = {
> >> +	.driver = {
> >> +		.name = "mali-c55",
> >> +		.of_match_table = of_match_ptr(mali_c55_of_match),
> >
> > Drop of_match_ptr().
> >
> >> +		.pm = &mali_c55_pm_ops,
> >> +	},
> >> +	.probe = mali_c55_probe,
> >> +	.remove_new = mali_c55_remove,
> >> +};
> >> +
> >> +module_platform_driver(mali_c55_driver);
> >
> > Blank line.
> >
> >> +MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
> >> +MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
> >> +MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
> >> +MODULE_LICENSE("GPL");
> >> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >> new file mode 100644
> >> index 000000000000..ea8b7b866e7a
> >> --- /dev/null
> >> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >> @@ -0,0 +1,611 @@
> >> +// SPDX-License-Identifier: GPL-2.0
> >> +/*
> >> + * ARM Mali-C55 ISP Driver - Image signal processor
> >> + *
> >> + * Copyright (C) 2024 Ideas on Board Oy
> >> + */
> >> +
> >> +#include <linux/delay.h>
> >> +#include <linux/iopoll.h>
> >> +#include <linux/property.h>
> >> +#include <linux/string.h>
> >> +
> >> +#include <media/media-entity.h>
> >> +#include <media/v4l2-common.h>
> >> +#include <media/v4l2-event.h>
> >> +#include <media/v4l2-mc.h>
> >> +#include <media/v4l2-subdev.h>
> >> +
> >> +#include "mali-c55-common.h"
> >> +#include "mali-c55-registers.h"
> >> +
> >> +static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
> >> +	{
> >> +		.code = MEDIA_BUS_FMT_SRGGB20_1X20,
> >> +		.order = MALI_C55_BAYER_ORDER_RGGB,
> >> +		.encoding = V4L2_PIXEL_ENC_BAYER,
> >> +	},
> >> +	{
> >> +		.code = MEDIA_BUS_FMT_SGRBG20_1X20,
> >> +		.order = MALI_C55_BAYER_ORDER_GRBG,
> >> +		.encoding = V4L2_PIXEL_ENC_BAYER,
> >> +	},
> >> +	{
> >> +		.code = MEDIA_BUS_FMT_SGBRG20_1X20,
> >> +		.order = MALI_C55_BAYER_ORDER_GBRG,
> >> +		.encoding = V4L2_PIXEL_ENC_BAYER,
> >> +	},
> >> +	{
> >> +		.code = MEDIA_BUS_FMT_SBGGR20_1X20,
> >> +		.order = MALI_C55_BAYER_ORDER_BGGR,
> >> +		.encoding = V4L2_PIXEL_ENC_BAYER,
> >> +	},
> >> +	{
> >> +		.code = MEDIA_BUS_FMT_RGB202020_1X60,
> >> +		.order = 0, /* Not relevant for this format */
> >> +		.encoding = V4L2_PIXEL_ENC_RGB,
> >> +	}
> >> +	/*
> >> +	 * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
> >> +	 * also support YUV input from a sensor passed-through to the output. At
> >> +	 * present we have no mechanism to test that though so it may have to
> >> +	 * wait a while...
> >> +	 */
> >> +};
> >> +
> >> +const struct mali_c55_isp_fmt *
> >> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
> >> +{
> >> +	if (!fmt)
> >> +		fmt = &mali_c55_isp_fmts[0];
> >> +	else
> >> +		fmt++;
> >> +
> >> +	for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
> >> +		return fmt;
> >
> > That's peculiar.
> >
> > 	if (!fmt)
> > 		fmt = &mali_c55_isp_fmts[0];
> > 	else if (fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)])
> > 		return ++fmt;
> > 	else
> > 		return NULL;
> >
> >> +
> >> +	return NULL;
> >> +}
> >> +
> >> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
> >> +{
> >> +	const struct mali_c55_isp_fmt *isp_fmt;
> >> +
> >> +	for_each_mali_isp_fmt(isp_fmt) {
> >
> > I would open-code the loop instead of using the macro, like you do
> > below. It will be more efficient.
> >
> >> +		if (isp_fmt->code == mbus_code)
> >> +			return true;
> >> +	}
> >> +
> >> +	return false;
> >> +}
> >> +
> >> +static const struct mali_c55_isp_fmt *
> >> +mali_c55_isp_get_mbus_config_by_code(u32 code)
> >> +{
> >> +	unsigned int i;
> >> +
> >> +	for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
> >> +		if (mali_c55_isp_fmts[i].code == code)
> >> +			return &mali_c55_isp_fmts[i];
> >> +	}
> >> +
> >> +	return NULL;
> >> +}
> >> +
> >> +static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
> >> +{
> >> +	u32 val;
> >> +
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);
> >
> > 	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> > 		       MALI_C55_INPUT_SAFE_STOP);
> >
> >> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >> +}
> >> +
> >> +static int mali_c55_isp_start(struct mali_c55 *mali_c55)
> >> +{
> >> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >> +	const struct mali_c55_isp_fmt *cfg;
> >> +	struct v4l2_mbus_framefmt *format;
> >
> > const
> >
> >> +	struct v4l2_subdev_state *state;
> >> +	struct v4l2_rect *crop;
> >
> > const
> >
> >> +	struct v4l2_subdev *sd;
> >> +	u32 val;
> >> +	int ret;
> >> +
> >> +	sd = &mali_c55->isp.sd;
> >
> > Assign when declaring the variable.
> >
> >> +
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >> +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
> >> +
> >> +	/* Apply input windowing */
> >> +	state = v4l2_subdev_get_locked_active_state(sd);
> >
> > Using .enable_streams() (see below) you'll get this for free.
> >
> >> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >> +	format = v4l2_subdev_state_get_format(state,
> >> +					      MALI_C55_ISP_PAD_SINK_VIDEO);
> >> +	cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
> >> +
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
> >> +		       MALI_C55_HC_START(crop->left));
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
> >> +		       MALI_C55_HC_SIZE(crop->width));
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
> >> +		       MALI_C55_VC_START(crop->top) |
> >> +		       MALI_C55_VC_SIZE(crop->height));
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
> >> +			     MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
> >> +			     MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
> >> +			     MALI_C55_BAYER_ORDER_MASK, cfg->order);
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
> >> +			     MALI_C55_INPUT_WIDTH_MASK,
> >> +			     MALI_C55_INPUT_WIDTH_20BIT);
> >> +
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >> +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
> >> +			     cfg->encoding == V4L2_PIXEL_ENC_RGB ?
> >> +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
> >> +
> >> +	ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev, "failed to DMA config\n");
> >> +		return ret;
> >> +	}
> >> +
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> >> +		       MALI_C55_INPUT_SAFE_START);
> >> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >
> > Should you return an error in case of timeout ?
> >
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)
> >
> > Why is this not handled wired to .s_stream() ? Or better,
> > .enable_streams() and .disable_streams().
> 
> There didn't really seem to be much point, since nothing outside this driver is ever going to start 
> the subdev streaming...it's not like the sensor case where a separate driver might have to call some 
> operation to start it.

True, but we're moving towards recording per-stream and per-pad
streaming state in the subdev through v4l2_subdev_enable_streams() and
v4l2_subdev_disable_streams(). We will likely expand usage of that
information, so it would be nice to keep it up-to-date already.

> >> +{
> >> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> >> +	struct v4l2_subdev *sd;
> >> +
> >> +	if (isp->remote_src) {
> >> +		sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
> >> +		v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
> >> +	}
> >> +	isp->remote_src = NULL;
> >> +
> >> +	mali_c55_isp_stop(mali_c55);
> >> +}
> >> +
> >> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
> >> +{
> >> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> >> +	struct media_pad *sink_pad;
> >> +	struct v4l2_subdev *sd;
> >> +	int ret;
> >> +
> >> +	sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
> >> +	isp->remote_src = media_pad_remote_pad_unique(sink_pad);
> >> +	if (IS_ERR(isp->remote_src)) {
> >
> > If you set the MUST_CONNECT flag for the MALI_C55_ISP_PAD_SINK_VIDEO pad
> > I think you can drop this check.
> >
> >> +		dev_err(mali_c55->dev, "Failed to get source for ISP\n");
> >> +		return PTR_ERR(isp->remote_src);
> >> +	}
> >> +
> >> +	sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
> >> +
> >> +	isp->frame_sequence = 0;
> >> +	ret = mali_c55_isp_start(mali_c55);
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev, "Failed to start ISP\n");
> >> +		isp->remote_src = NULL;
> >> +		return ret;
> >> +	}
> >> +
> >> +	/*
> >> +	 * We only support a single input stream, so we can just enable the 1st
> >> +	 * entry in the streams mask.
> >> +	 */
> >> +	ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev, "Failed to start ISP source\n");
> >> +		mali_c55_isp_stop(mali_c55);
> >> +		return ret;
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >> +				       struct v4l2_subdev_state *state,
> >> +				       struct v4l2_subdev_mbus_code_enum *code)
> >> +{
> >> +	/*
> >> +	 * Only the internal RGB processed format is allowed on the regular
> >> +	 * processing source pad.
> >> +	 */
> >> +	if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
> >> +		if (code->index)
> >> +			return -EINVAL;
> >> +
> >> +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >> +		return 0;
> >> +	}
> >> +
> >> +	/* On the sink and bypass pads all the supported formats are allowed. */
> >> +	if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
> >> +		return -EINVAL;
> >> +
> >> +	code->code = mali_c55_isp_fmts[code->index].code;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
> >> +					struct v4l2_subdev_state *state,
> >> +					struct v4l2_subdev_frame_size_enum *fse)
> >> +{
> >> +	const struct mali_c55_isp_fmt *cfg;
> >> +
> >> +	if (fse->index > 0)
> >> +		return -EINVAL;
> >> +
> >> +	/*
> >> +	 * Only the internal RGB processed format is allowed on the regular
> >> +	 * processing source pad.
> >> +	 *
> >> +	 * On the sink and bypass pads all the supported formats are allowed.
> >> +	 */
> >> +	if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
> >> +		if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
> >> +			return -EINVAL;
> >> +	} else {
> >> +		cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
> >> +		if (!cfg)
> >> +			return -EINVAL;
> >> +	}
> >> +
> >> +	fse->min_width = MALI_C55_MIN_WIDTH;
> >> +	fse->min_height = MALI_C55_MIN_HEIGHT;
> >> +	fse->max_width = MALI_C55_MAX_WIDTH;
> >> +	fse->max_height = MALI_C55_MAX_HEIGHT;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
> >> +				struct v4l2_subdev_state *state,
> >> +				struct v4l2_subdev_format *format)
> >> +{
> >> +	struct v4l2_mbus_framefmt *fmt = &format->format;
> >> +	struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
> >> +	const struct mali_c55_isp_fmt *cfg;
> >> +	struct v4l2_rect *crop;
> >> +
> >> +	/*
> >> +	 * Disallow set_fmt on the source pads; format is fixed and the sizes
> >> +	 * are the result of applying the sink crop rectangle to the sink
> >> +	 * format.
> >> +	 */
> >> +	if (format->pad)
> >
> > 	if (format->pad != MALI_C55_ISP_PAD_SINK_VIDEO)
> >
> >> +		return v4l2_subdev_get_fmt(sd, state, format);
> >> +
> >> +	cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
> >> +	if (!cfg)
> >> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >> +	fmt->field = V4L2_FIELD_NONE;
> >
> > Do you intentionally allow the colorspace fields to be overwritten to
> > any value ? rkisp1_isp_set_src_fmt() and rkisp1_isp_set_sink_fmt() may
> > show you how this could be handled.
> >
> >> +
> >> +	/*
> >> +	 * Clamp sizes in the accepted limits and clamp the crop rectangle in
> >> +	 * the new sizes.
> >> +	 */
> >> +	clamp_t(unsigned int, fmt->width,
> >> +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >> +	clamp_t(unsigned int, fmt->width,
> >> +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >
> > clamp_t() returns a value, which you ignore. Those are no-ops. You meant
> >
> > 	fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> > 			     MALI_C55_MAX_WIDTH);
> > 	fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> > 			      MALI_C55_MAX_HEIGHT);
> >
> > Same for every use of clamp_t() through the whole driver.
> >
> > Also, do you need clamp_t() ? I think all values are unsigned int, you
> > can use clamp().
> >
> > Are there any alignment constraints, such a multiples of two for bayer
> > formats ? Same in all the other locations where applicable.
> >
> >> +
> >> +	sink_fmt = v4l2_subdev_state_get_format(state,
> >> +						MALI_C55_ISP_PAD_SINK_VIDEO);
> >> +	*sink_fmt = *fmt;
> >> +
> >> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >> +	crop->left = 0;
> >> +	crop->top = 0;
> >> +	crop->width = fmt->width;
> >> +	crop->height = fmt->height;
> >> +
> >> +	/*
> >> +	 * Propagate format to source pads. On the 'regular' output pad use
> >> +	 * the internal RGB processed format, while on the bypass pad simply
> >> +	 * replicate the ISP sink format. The sizes on both pads are the same as
> >> +	 * the ISP sink crop rectangle.
> >> +	 */
> >
> > Colorspace information will need to be propagated too.
> >
> >> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
> >> +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >> +	src_fmt->width = crop->width;
> >> +	src_fmt->height = crop->height;
> >> +
> >> +	src_fmt = v4l2_subdev_state_get_format(state,
> >> +					       MALI_C55_ISP_PAD_SOURCE_BYPASS);
> >> +	src_fmt->code = fmt->code;
> >> +	src_fmt->width = crop->width;
> >> +	src_fmt->height = crop->height;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
> >> +				      struct v4l2_subdev_state *state,
> >> +				      struct v4l2_subdev_selection *sel)
> >> +{
> >> +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> >
> > 	sel->pad != MALI_C55_ISP_PAD_SINK_VIDEO
> >
> >> +		return -EINVAL;
> >> +
> >> +	sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
> >> +				      struct v4l2_subdev_state *state,
> >> +				      struct v4l2_subdev_selection *sel)
> >> +{
> >> +	struct v4l2_mbus_framefmt *src_fmt;
> >> +	struct v4l2_mbus_framefmt *fmt;
> >
> > const
> >
> >> +	struct v4l2_rect *crop;
> >> +
> >> +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> >
> > Ditto.
> >
> >> +		return -EINVAL;
> >> +
> >> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >> +
> >> +	clamp_t(unsigned int, sel->r.left, 0, fmt->width);
> >> +	clamp_t(unsigned int, sel->r.top, 0, fmt->height);
> >> +	clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
> >> +		fmt->width - sel->r.left);
> >> +	clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
> >> +		fmt->height - sel->r.top);
> >> +
> >> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >> +	*crop = sel->r;
> >> +
> >> +	/* Propagate the crop rectangle sizes to the source pad format. */
> >> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
> >> +	src_fmt->width = crop->width;
> >> +	src_fmt->height = crop->height;
> >
> > Can you confirm that cropping doesn't affect the bypass path ?
> 
> Yes
> 
> > And maybe
> > add a comment to mention it.
> 
> Will
> 
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
> >> +	.enum_mbus_code		= mali_c55_isp_enum_mbus_code,
> >> +	.enum_frame_size	= mali_c55_isp_enum_frame_size,
> >> +	.get_fmt		= v4l2_subdev_get_fmt,
> >> +	.set_fmt		= mali_c55_isp_set_fmt,
> >> +	.get_selection		= mali_c55_isp_get_selection,
> >> +	.set_selection		= mali_c55_isp_set_selection,
> >> +	.link_validate		= v4l2_subdev_link_validate_default,
> >> +};
> >> +
> >> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
> >> +{
> >> +	struct v4l2_event event = {
> >> +		.type = V4L2_EVENT_FRAME_SYNC,
> >> +	};
> >> +
> >> +	event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
> >> +	v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
> >> +}
> >> +
> >> +static int
> >> +mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
> >> +			     struct v4l2_event_subscription *sub)
> >> +{
> >> +	if (sub->type != V4L2_EVENT_FRAME_SYNC)
> >> +		return -EINVAL;
> >> +
> >> +	/* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
> >> +	if (sub->id != 0)
> >> +		return -EINVAL;
> >> +
> >> +	return v4l2_event_subscribe(fh, sub, 0, NULL);
> >> +}
> >> +
> >> +static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
> >> +	.subscribe_event = mali_c55_isp_subscribe_event,
> >> +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
> >> +};
> >> +
> >> +static const struct v4l2_subdev_ops mali_c55_isp_ops = {
> >> +	.pad	= &mali_c55_isp_pad_ops,
> >> +	.core	= &mali_c55_isp_core_ops,
> >> +};
> >> +
> >> +static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
> >> +				   struct v4l2_subdev_state *sd_state)
> >
> > You name this variable state in every other subdev operation handler.
> >
> >> +{
> >> +	struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >> +	struct v4l2_rect *in_crop;
> >> +
> >> +	sink_fmt = v4l2_subdev_state_get_format(sd_state,
> >> +						MALI_C55_ISP_PAD_SINK_VIDEO);
> >> +	src_fmt = v4l2_subdev_state_get_format(sd_state,
> >> +					       MALI_C55_ISP_PAD_SOURCE);
> >> +	in_crop = v4l2_subdev_state_get_crop(sd_state,
> >> +					     MALI_C55_ISP_PAD_SINK_VIDEO);
> >> +
> >> +	sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
> >> +	sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >> +	sink_fmt->field = V4L2_FIELD_NONE;
> >> +	sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >
> > You should initialize the colorspace fields too. Same below.
> >
> >> +
> >> +	*v4l2_subdev_state_get_format(sd_state,
> >> +			      MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
> >> +
> >> +	src_fmt->width = MALI_C55_DEFAULT_WIDTH;
> >> +	src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >> +	src_fmt->field = V4L2_FIELD_NONE;
> >> +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >> +
> >> +	in_crop->top = 0;
> >> +	in_crop->left = 0;
> >> +	in_crop->width = MALI_C55_DEFAULT_WIDTH;
> >> +	in_crop->height = MALI_C55_DEFAULT_HEIGHT;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
> >> +	.init_state = mali_c55_isp_init_state,
> >> +};
> >> +
> >> +static const struct media_entity_operations mali_c55_isp_media_ops = {
> >> +	.link_validate		= v4l2_subdev_link_validate,
> >
> > 	.link_validate = v4l2_subdev_link_validate,
> >
> > to match mali_c55_isp_internal_ops.
> >
> >> +};
> >> +
> >> +static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
> >> +				       struct v4l2_subdev *subdev,
> >> +				       struct v4l2_async_connection *asc)
> >> +{
> >> +	struct mali_c55_isp *isp = container_of(notifier,
> >> +						struct mali_c55_isp, notifier);
> >> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> >> +	struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
> >> +	int ret;
> >> +
> >> +	/*
> >> +	 * By default we'll flag this link enabled and the TPG disabled, but
> >> +	 * no immutable flag because we need to be able to switch between the
> >> +	 * two.
> >> +	 */
> >> +	ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
> >> +					      MEDIA_LNK_FL_ENABLED);
> >> +	if (ret)
> >> +		dev_err(mali_c55->dev, "failed to create link for %s\n",
> >> +			subdev->name);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
> >> +{
> >> +	struct mali_c55_isp *isp = container_of(notifier,
> >> +						struct mali_c55_isp, notifier);
> >> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> >> +
> >> +	return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
> >> +}
> >> +
> >> +static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
> >> +	.bound = mali_c55_isp_notifier_bound,
> >> +	.complete = mali_c55_isp_notifier_complete,
> >> +};
> >> +
> >> +static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
> >> +{
> >> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> >> +	struct v4l2_async_connection *asc;
> >> +	struct fwnode_handle *ep;
> >> +	int ret;
> >> +
> >> +	v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
> >> +
> >> +	/*
> >> +	 * The ISP should have a single endpoint pointing to some flavour of
> >> +	 * CSI-2 receiver...but for now at least we do want everything to work
> >> +	 * normally even with no sensors connected, as we have the TPG. If we
> >> +	 * don't find a sensor just warn and return success.
> >> +	 */
> >> +	ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
> >> +					     0, 0, 0);
> >> +	if (!ep) {
> >> +		dev_warn(mali_c55->dev, "no local endpoint found\n");
> >> +		return 0;
> >> +	}
> >> +
> >> +	asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
> >> +					      struct v4l2_async_connection);
> >> +	if (IS_ERR(asc)) {
> >> +		dev_err(mali_c55->dev, "failed to add remote fwnode\n");
> >> +		ret = PTR_ERR(asc);
> >> +		goto err_put_ep;
> >> +	}
> >> +
> >> +	isp->notifier.ops = &mali_c55_isp_notifier_ops;
> >> +	ret = v4l2_async_nf_register(&isp->notifier);
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev, "failed to register notifier\n");
> >> +		goto err_cleanup_nf;
> >> +	}
> >> +
> >> +	fwnode_handle_put(ep);
> >> +
> >> +	return 0;
> >> +
> >> +err_cleanup_nf:
> >> +	v4l2_async_nf_cleanup(&isp->notifier);
> >> +err_put_ep:
> >> +	fwnode_handle_put(ep);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +int mali_c55_register_isp(struct mali_c55 *mali_c55)
> >> +{
> >> +	struct mali_c55_isp *isp = &mali_c55->isp;
> >> +	struct v4l2_subdev *sd = &isp->sd;
> >> +	int ret;
> >> +
> >> +	isp->mali_c55 = mali_c55;
> >> +
> >> +	v4l2_subdev_init(sd, &mali_c55_isp_ops);
> >> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> >> +	sd->entity.ops = &mali_c55_isp_media_ops;
> >> +	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
> >> +	sd->internal_ops = &mali_c55_isp_internal_ops;
> >> +	strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
> >> +
> >> +	isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
> >
> > The MUST_CONNECT flag would make sense here.
> >
> >> +	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> >> +	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
> >> +
> >> +	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
> >> +				     isp->pads);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	ret = v4l2_subdev_init_finalize(sd);
> >> +	if (ret)
> >> +		goto err_cleanup_media_entity;
> >> +
> >> +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >> +	if (ret)
> >> +		goto err_cleanup_subdev;
> >> +
> >> +	ret = mali_c55_isp_parse_endpoint(isp);
> >> +	if (ret)
> >> +		goto err_cleanup_subdev;
> >
> > As noted elsewhere, I think this belongs to mali-c55-core.c.
> >
> >> +
> >> +	mutex_init(&isp->lock);
> >
> > This lock is used in mali-c55-capture.c only, that seems weird.
> 
> It's because we have two separate capture devices, who's start/stop streaming path calls into the 
> ISP subdevice's start streaming function, but has to do it after a bunch of other writes to the cap 
> device or resizer specific sections...the intention was to delay doing any of that until the ISP was 
> in a known state.

OK. Naming it capture_lock or something similar, with a comment, would
help.

> >> +
> >> +	return 0;
> >> +
> >> +err_cleanup_subdev:
> >> +	v4l2_subdev_cleanup(sd);
> >> +err_cleanup_media_entity:
> >> +	media_entity_cleanup(&sd->entity);
> >> +	isp->mali_c55 = NULL;
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
> >> +{
> >> +	struct mali_c55_isp *isp = &mali_c55->isp;
> >> +
> >> +	if (!isp->mali_c55)
> >> +		return;
> >> +
> >> +	mutex_destroy(&isp->lock);
> >> +	v4l2_async_nf_unregister(&isp->notifier);
> >> +	v4l2_async_nf_cleanup(&isp->notifier);
> >> +	v4l2_device_unregister_subdev(&isp->sd);
> >> +	v4l2_subdev_cleanup(&isp->sd);
> >> +	media_entity_cleanup(&isp->sd.entity);
> >> +}
> >> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >> new file mode 100644
> >> index 000000000000..cb27abde2aa5
> >> --- /dev/null
> >> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >> @@ -0,0 +1,258 @@
> >> +/* SPDX-License-Identifier: GPL-2.0 */
> >> +/*
> >> + * ARM Mali-C55 ISP Driver - Register definitions
> >> + *
> >> + * Copyright (C) 2024 Ideas on Board Oy
> >> + */
> >> +
> >> +#ifndef _MALI_C55_REGISTERS_H
> >> +#define _MALI_C55_REGISTERS_H
> >> +
> >> +#include <linux/bits.h>
> >> +
> >> +/* ISP Common 0x00000 - 0x000ff */
> >> +
> >> +#define MALI_C55_REG_API				0x00000
> >> +#define MALI_C55_REG_PRODUCT				0x00004
> >> +#define MALI_C55_REG_VERSION				0x00008
> >> +#define MALI_C55_REG_REVISION				0x0000c
> >> +#define MALI_C55_REG_PULSE_MODE				0x0003c
> >> +#define MALI_C55_REG_INPUT_MODE_REQUEST			0x0009c
> >> +#define MALI_C55_INPUT_SAFE_STOP			0x00
> >> +#define MALI_C55_INPUT_SAFE_START			0x01
> >> +#define MALI_C55_REG_MODE_STATUS			0x000a0
> >> +#define MALI_C55_REG_INTERRUPT_MASK_VECTOR		0x00030
> >> +#define MALI_C55_INTERRUPT_MASK_ALL			GENMASK(31, 0)
> >> +
> >> +#define MALI_C55_REG_GLOBAL_MONITOR			0x00050
> >> +
> >> +#define MALI_C55_REG_GEN_VIDEO				0x00080
> >> +#define MALI_C55_REG_GEN_VIDEO_ON_MASK			BIT(0)
> >> +#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK		BIT(1)
> >> +#define MALI_C55_REG_GEN_PREFETCH_MASK			GENMASK(31, 16)
> >> +
> >> +#define MALI_C55_REG_MCU_CONFIG				0x00020
> >> +#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK		BIT(0)
> >
> > #define MALI_C55_REG_MCU_CONFIG_OVERRIDE		BIT(0)
> >
> > Same in other places where applicable.
> >
> >> +#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK		BIT(1)
> >> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PING		BIT(1)
> >> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG		0x00
> >> +#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK		BIT(8)
> >> +#define MALI_C55_REG_PING_PONG_READ			0x00024
> >> +#define MALI_C55_REG_PING_PONG_READ_MASK		BIT(2)
> >> +
> >> +#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR		0x00034
> >> +#define MALI_C55_REG_INTERRUPT_CLEAR			0x00040
> >> +#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR		0x00044
> >> +#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS		0x00068
> >> +#define MALI_C55_GPS_PONG_FITTED			BIT(0)
> >> +#define MALI_C55_GPS_WDR_FITTED				BIT(1)
> >> +#define MALI_C55_GPS_COMPRESSION_FITTED			BIT(2)
> >> +#define MALI_C55_GPS_TEMPER_FITTED			BIT(3)
> >> +#define MALI_C55_GPS_SINTER_LITE_FITTED			BIT(4)
> >> +#define MALI_C55_GPS_SINTER_FITTED			BIT(5)
> >> +#define MALI_C55_GPS_IRIDIX_LTM_FITTED			BIT(6)
> >> +#define MALI_C55_GPS_IRIDIX_GTM_FITTED			BIT(7)
> >> +#define MALI_C55_GPS_CNR_FITTED				BIT(8)
> >> +#define MALI_C55_GPS_FRSCALER_FITTED			BIT(9)
> >> +#define MALI_C55_GPS_DS_PIPE_FITTED			BIT(10)
> >> +
> >> +#define MALI_C55_REG_BLANKING				0x00084
> >> +#define MALI_C55_REG_HBLANK_MASK			GENMASK(15, 0)
> >> +#define MALI_C55_REG_VBLANK_MASK			GENMASK(31, 16)
> >> +
> >> +#define MALI_C55_REG_HC_START				0x00088
> >> +#define MALI_C55_HC_START(h)				(((h) & 0xffff) << 16)
> >> +#define MALI_C55_REG_HC_SIZE				0x0008c
> >> +#define MALI_C55_HC_SIZE(h)				((h) & 0xffff)
> >> +#define MALI_C55_REG_VC_START_SIZE			0x00094
> >> +#define MALI_C55_VC_START(v)				((v) & 0xffff)
> >> +#define MALI_C55_VC_SIZE(v)				(((v) & 0xffff) << 16)
> >> +
> >> +/* Ping/Pong Configuration Space */
> >> +#define MALI_C55_REG_BASE_ADDR				0x18e88
> >> +#define MALI_C55_REG_BYPASS_0				0x18eac
> >> +#define MALI_C55_REG_BYPASS_0_VIDEO_TEST		BIT(0)
> >> +#define MALI_C55_REG_BYPASS_0_INPUT_FMT			BIT(1)
> >> +#define MALI_C55_REG_BYPASS_0_DECOMPANDER		BIT(2)
> >> +#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR		BIT(3)
> >> +#define MALI_C55_REG_BYPASS_0_GAIN_WDR			BIT(4)
> >> +#define MALI_C55_REG_BYPASS_0_FRAME_STITCH		BIT(5)
> >> +#define MALI_C55_REG_BYPASS_1				0x18eb0
> >> +#define MALI_C55_REG_BYPASS_1_DIGI_GAIN			BIT(0)
> >> +#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS		BIT(1)
> >> +#define MALI_C55_REG_BYPASS_1_FE_SQRT			BIT(2)
> >> +#define MALI_C55_REG_BYPASS_1_RAW_FE			BIT(3)
> >> +#define MALI_C55_REG_BYPASS_2				0x18eb8
> >> +#define MALI_C55_REG_BYPASS_2_SINTER			BIT(0)
> >> +#define MALI_C55_REG_BYPASS_2_TEMPER			BIT(1)
> >> +#define MALI_C55_REG_BYPASS_3				0x18ebc
> >> +#define MALI_C55_REG_BYPASS_3_SQUARE_BE			BIT(0)
> >> +#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH	BIT(1)
> >> +#define MALI_C55_REG_BYPASS_3_MESH_SHADING		BIT(3)
> >> +#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE		BIT(4)
> >> +#define MALI_C55_REG_BYPASS_3_IRIDIX			BIT(5)
> >> +#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN		BIT(6)
> >> +#define MALI_C55_REG_BYPASS_4				0x18ec0
> >> +#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB		BIT(1)
> >> +#define MALI_C55_REG_BYPASS_4_PF_CORRECTION		BIT(3)
> >> +#define MALI_C55_REG_BYPASS_4_CCM			BIT(4)
> >> +#define MALI_C55_REG_BYPASS_4_CNR			BIT(5)
> >> +#define MALI_C55_REG_FR_BYPASS				0x18ec4
> >> +#define MALI_C55_REG_DS_BYPASS				0x18ec8
> >> +#define MALI_C55_BYPASS_CROP				BIT(0)
> >> +#define MALI_C55_BYPASS_SCALER				BIT(1)
> >> +#define MALI_C55_BYPASS_GAMMA_RGB			BIT(2)
> >> +#define MALI_C55_BYPASS_SHARPEN				BIT(3)
> >> +#define MALI_C55_BYPASS_CS_CONV				BIT(4)
> >> +#define MALI_C55_REG_ISP_RAW_BYPASS			0x18ecc
> >> +#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK		BIT(0)
> >> +#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK		GENMASK(9, 8)
> >> +#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS		2
> >> +#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS		1
> >> +#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE		BIT(1)
> >> +#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS		BIT(0)
> >> +
> >> +#define MALI_C55_REG_ACTIVE_WIDTH_MASK			0xffff
> >> +#define MALI_C55_REG_ACTIVE_HEIGHT_MASK			0xffff0000
> >> +#define MALI_C55_REG_BAYER_ORDER			0x18e8c
> >> +#define MALI_C55_BAYER_ORDER_MASK			GENMASK(1, 0)
> >> +#define MALI_C55_REG_TPG_CH0				0x18ed8
> >> +#define MALI_C55_TEST_PATTERN_ON_OFF			BIT(0)
> >> +#define MALI_C55_TEST_PATTERN_RGB_MASK			BIT(1)
> >> +#define MALI_C55_REG_TPG_R_BACKGROUND			0x18ee0
> >> +#define MALI_C55_REG_TPG_G_BACKGROUND			0x18ee4
> >> +#define MALI_C55_REG_TPG_B_BACKGROUND			0x18ee8
> >> +#define MALI_C55_TPG_BACKGROUND_MAX			0xfffff
> >> +#define MALI_C55_REG_INPUT_WIDTH			0x18f98
> >> +#define MALI_C55_INPUT_WIDTH_MASK			GENMASK(18, 16)
> >> +#define MALI_C55_INPUT_WIDTH_8BIT			0
> >> +#define MALI_C55_INPUT_WIDTH_10BIT			1
> >> +#define MALI_C55_INPUT_WIDTH_12BIT			2
> >> +#define MALI_C55_INPUT_WIDTH_14BIT			3
> >> +#define MALI_C55_INPUT_WIDTH_16BIT			4
> >> +#define MALI_C55_INPUT_WIDTH_20BIT			5
> >> +#define MALI_C55_REG_SPACE_SIZE				0x4000
> >> +#define MALI_C55_REG_CONFIG_SPACES_OFFSET		0x0ab6c
> >> +#define MALI_C55_CONFIG_SPACE_SIZE			0x1231c
> >> +
> >> +#define MALI_C55_REG_SINTER_CONFIG			0x19348
> >> +#define MALI_C55_SINTER_VIEW_FILTER_MASK		GENMASK(1, 0)
> >> +#define MALI_C55_SINTER_SCALE_MODE_MASK			GENMASK(3, 2)
> >> +#define MALI_C55_SINTER_ENABLE_MASK			BIT(4)
> >> +#define MALI_C55_SINTER_FILTER_SELECT_MASK		BIT(5)
> >> +#define MALI_C55_SINTER_INT_SELECT_MASK			BIT(6)
> >> +#define MALI_C55_SINTER_RM_ENABLE_MASK			BIT(7)
> >> +
> >> +/* Colour Correction Matrix Configuration */
> >> +#define MALI_C55_REG_CCM_ENABLE				0x1b07c
> >> +#define MALI_C55_CCM_ENABLE_MASK			BIT(0)
> >> +#define MALI_C55_REG_CCM_COEF_R_R			0x1b080
> >> +#define MALI_C55_REG_CCM_COEF_R_G			0x1b084
> >> +#define MALI_C55_REG_CCM_COEF_R_B			0x1b088
> >> +#define MALI_C55_REG_CCM_COEF_G_R			0x1b090
> >> +#define MALI_C55_REG_CCM_COEF_G_G			0x1b094
> >> +#define MALI_C55_REG_CCM_COEF_G_B			0x1b098
> >> +#define MALI_C55_REG_CCM_COEF_B_R			0x1b0a0
> >> +#define MALI_C55_REG_CCM_COEF_B_G			0x1b0a4
> >> +#define MALI_C55_REG_CCM_COEF_B_B			0x1b0a8
> >> +#define MALI_C55_CCM_COEF_MASK				GENMASK(12, 0)
> >> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R			0x1b0b0
> >> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G			0x1b0b4
> >> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B			0x1b0b8
> >> +#define MALI_C55_CCM_ANTIFOG_GAIN_MASK			GENMASK(11, 0)
> >> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R		0x1b0c0
> >> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G		0x1b0c4
> >> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B		0x1b0c8
> >> +#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK		GENMASK(11, 0)
> >> +
> >> +/*
> >> + * The Mali-C55 ISP has up to two output pipes; known as full resolution and
> >> + * down scaled. The register space for these is laid out identically, but offset
> >> + * by 372 bytes.
> >> + */
> >> +#define MALI_C55_CAP_DEV_FR_REG_OFFSET		0x0
> >> +#define MALI_C55_CAP_DEV_DS_REG_OFFSET		0x174
> >> +
> >> +#define MALI_C55_REG_CS_CONV_CONFIG(offset)		(0x1c098 + (offset))
> >> +#define MALI_C55_CS_CONV_MATRIX_MASK			BIT(0)
> >> +#define MALI_C55_CS_CONV_FILTER_MASK			BIT(1)
> >> +#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK		BIT(2)
> >> +#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK		BIT(3)
> >> +#define MALI_C55_REG_Y_WRITER_MODE(offset)		(0x1c0ec + (offset))
> >> +#define MALI_C55_REG_UV_WRITER_MODE(offset)		(0x1c144 + (offset))
> >> +#define MALI_C55_WRITER_MODE_MASK			GENMASK(4, 0)
> >> +#define MALI_C55_WRITER_SUBMODE_MASK			GENMASK(7, 6)
> >> +#define MALI_C55_WRITER_FRAME_WRITE_MASK		BIT(9)
> >> +#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset)		(0x1c0f0 + (offset))
> >> +#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset)		(0x1c148 + (offset))
> >> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)		((w) << 0)
> >> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)		((h) << 16)
> >> +#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset)	(0x1c0f4 + (offset))
> >> +#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset)	(0x1c108 + (offset))
> >> +#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
> >> +#define MALI_C55_REG_Y_WRITER_BANKS_RESTART		BIT(3)
> >> +#define MALI_C55_REG_Y_WRITER_OFFSET(offset)		(0x1c10c + (offset))
> >> +#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset)	(0x1c14c + (offset))
> >> +#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset)	(0x1c160 + (offset))
> >> +#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
> >> +#define MALI_C55_REG_UV_WRITER_BANKS_RESTART		BIT(3)
> >> +#define MALI_C55_REG_UV_WRITER_OFFSET(offset)		(0x1c164 + (offset))
> >> +
> >> +#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
> >> +#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE		0x18edc
> >> +
> >> +#define MALI_C55_REG_CROP_EN(offset)			(0x1c028 + (offset))
> >> +#define MALI_C55_CROP_ENABLE				BIT(0)
> >> +#define MALI_C55_REG_CROP_X_START(offset)		(0x1c02c + (offset))
> >> +#define MALI_C55_REG_CROP_Y_START(offset)		(0x1c030 + (offset))
> >> +#define MALI_C55_REG_CROP_X_SIZE(offset)		(0x1c034 + (offset))
> >> +#define MALI_C55_REG_CROP_Y_SIZE(offset)		(0x1c038 + (offset))
> >> +#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset)		(0x1c040 + (offset))
> >> +#define MALI_C55_SCALER_TIMEOUT_EN			BIT(4)
> >> +#define MALI_C55_SCALER_TIMEOUT(t)			((t) << 16)
> >> +#define MALI_C55_REG_SCALER_IN_WIDTH(offset)		(0x1c044 + (offset))
> >> +#define MALI_C55_REG_SCALER_IN_HEIGHT(offset)		(0x1c048 + (offset))
> >> +#define MALI_C55_REG_SCALER_OUT_WIDTH(offset)		(0x1c04c + (offset))
> >> +#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset)		(0x1c050 + (offset))
> >> +#define MALI_C55_REG_SCALER_HFILT_TINC(offset)		(0x1c054 + (offset))
> >> +#define MALI_C55_REG_SCALER_HFILT_COEF(offset)		(0x1c058 + (offset))
> >> +#define MALI_C55_REG_SCALER_VFILT_TINC(offset)		(0x1c05c + (offset))
> >> +#define MALI_C55_REG_SCALER_VFILT_COEF(offset)		(0x1c060 + (offset))
> >> +
> >> +#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset)		(0x1c064 + (offset))
> >> +#define MALI_C55_GAMMA_ENABLE_MASK			BIT(0)
> >> +#define MALI_C55_REG_GAMMA_GAINS_1(offset)		(0x1c068 + (offset))
> >> +#define MALI_C55_GAMMA_GAIN_R_MASK			GENMASK(11, 0)
> >> +#define MALI_C55_GAMMA_GAIN_G_MASK			GENMASK(27, 16)
> >> +#define MALI_C55_REG_GAMMA_GAINS_2(offset)		(0x1c06c + (offset))
> >> +#define MALI_C55_GAMMA_GAIN_B_MASK			GENMASK(11, 0)
> >> +#define MALI_C55_REG_GAMMA_OFFSETS_1(offset)		(0x1c070 + (offset))
> >> +#define MALI_C55_GAMMA_OFFSET_R_MASK			GENMASK(11, 0)
> >> +#define MALI_C55_GAMMA_OFFSET_G_MASK			GENMASK(27, 16)
> >> +#define MALI_C55_REG_GAMMA_OFFSETS_2(offset)		(0x1c074 + (offset))
> >> +#define MALI_C55_GAMMA_OFFSET_B_MASK			GENMASK(11, 0)
> >> +
> >> +/* Output DMA Writer */
> >> +
> >> +#define MALI_C55_OUTPUT_DISABLED		0
> >> +#define MALI_C55_OUTPUT_RGB32			1
> >> +#define MALI_C55_OUTPUT_A2R10G10B10		2
> >> +#define MALI_C55_OUTPUT_RGB565			3
> >> +#define MALI_C55_OUTPUT_RGB24			4
> >> +#define MALI_C55_OUTPUT_GEN32			5
> >> +#define MALI_C55_OUTPUT_RAW16			6
> >> +#define MALI_C55_OUTPUT_AYUV			8
> >> +#define MALI_C55_OUTPUT_Y410			9
> >> +#define MALI_C55_OUTPUT_YUY2			10
> >> +#define MALI_C55_OUTPUT_UYVY			11
> >> +#define MALI_C55_OUTPUT_Y210			12
> >> +#define MALI_C55_OUTPUT_NV12_21			13
> >> +#define MALI_C55_OUTPUT_YUV_420_422		17
> >> +#define MALI_C55_OUTPUT_P210_P010		19
> >> +#define MALI_C55_OUTPUT_YUV422			20
> >
> > I'd define this just below the MALI_C55_REG_UV_WRITER_MODE register
> > macro.
> >
> >> +
> >> +#define MALI_C55_OUTPUT_PLANE_ALT0		0
> >> +#define MALI_C55_OUTPUT_PLANE_ALT1		1
> >> +#define MALI_C55_OUTPUT_PLANE_ALT2		2
> >
> > Same here ?
> >
> >> +
> >> +#endif /* _MALI_C55_REGISTERS_H */
> >> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >> new file mode 100644
> >> index 000000000000..8edae87f1e5f
> >> --- /dev/null
> >> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >> @@ -0,0 +1,382 @@
> >> +/* SPDX-License-Identifier: GPL-2.0 */
> >> +/*
> >> + * ARM Mali-C55 ISP Driver - Resizer Coefficients
> >> + *
> >> + * Copyright (C) 2024 Ideas on Board Oy
> >> + */
> >> +
> >> +#ifndef _MALI_C55_RESIZER_COEFS_H
> >> +#define _MALI_C55_RESIZER_COEFS_H
> >> +
> >> +#include "mali-c55-common.h"
> >> +
> >> +#define MALI_C55_RESIZER_COEFS_NUM_BANKS	8
> >> +#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES	64
> >
> > Do these belongs to mali-c55-registers.h ?
> >
> >> +
> >> +static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
> >> +	{	/* Bank 0 */
> >> +		0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
> >> +		0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
> >> +		0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
> >> +		0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
> >> +		0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
> >> +		0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
> >> +		0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
> >> +		0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
> >> +		0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
> >> +		0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
> >> +		0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
> >> +		0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
> >> +		0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
> >> +		0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
> >> +		0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
> >> +		0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
> >> +	},
> >> +	{	/* Bank 1 */
> >> +		0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
> >> +		0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
> >> +		0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
> >> +		0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
> >> +		0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
> >> +		0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
> >> +		0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
> >> +		0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
> >> +		0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
> >> +		0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
> >> +		0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
> >> +		0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
> >> +		0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
> >> +		0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
> >> +		0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
> >> +		0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
> >> +	},
> >> +	{	/* Bank 2 */
> >> +		0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
> >> +		0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
> >> +		0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
> >> +		0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
> >> +		0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
> >> +		0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
> >> +		0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
> >> +		0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
> >> +		0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
> >> +		0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
> >> +		0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
> >> +		0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
> >> +		0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
> >> +		0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
> >> +		0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
> >> +		0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
> >> +	},
> >> +	{	/* Bank 3 */
> >> +		0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
> >> +		0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
> >> +		0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
> >> +		0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
> >> +		0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
> >> +		0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
> >> +		0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
> >> +		0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
> >> +		0x20100000, 0x00000010, 0x1f110000, 0x00000010,
> >> +		0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
> >> +		0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
> >> +		0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
> >> +		0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
> >> +		0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
> >> +		0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
> >> +		0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
> >> +	},
> >> +	{	/* Bank 4 */
> >> +		0x17090000, 0x00000917, 0x18090000, 0x00000916,
> >> +		0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
> >> +		0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
> >> +		0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
> >> +		0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
> >> +		0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
> >> +		0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
> >> +		0x190f0300, 0x00000411, 0x18100300, 0x00000411,
> >> +		0x1a100300, 0x00000310, 0x18110400, 0x00000310,
> >> +		0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
> >> +		0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
> >> +		0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
> >> +		0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
> >> +		0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
> >> +		0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
> >> +		0x17160800, 0x0000010a, 0x18160900, 0x00000009,
> >> +	},
> >> +	{	/* Bank 5 */
> >> +		0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
> >> +		0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
> >> +		0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
> >> +		0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
> >> +		0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
> >> +		0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
> >> +		0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
> >> +		0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
> >> +		0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
> >> +		0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
> >> +		0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
> >> +		0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
> >> +		0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
> >> +		0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
> >> +		0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
> >> +		0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
> >> +	},
> >> +	{	/* Bank 6 */
> >> +		0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
> >> +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
> >> +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
> >> +		0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
> >> +		0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
> >> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >> +		0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
> >> +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >> +		0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
> >> +		0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
> >> +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
> >> +		0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >> +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >> +	},
> >> +	{	/* Bank 7 */
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >> +	}
> >> +};
> >> +
> >> +static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
> >> +	{	/* Bank 0 */
> >> +		0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
> >> +		0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
> >> +		0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
> >> +		0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
> >> +		0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
> >> +		0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
> >> +		0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
> >> +		0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
> >> +		0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
> >> +		0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
> >> +		0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
> >> +		0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
> >> +		0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
> >> +		0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
> >> +		0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
> >> +		0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
> >> +	},
> >> +	{	/* Bank 1 */
> >> +		0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
> >> +		0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
> >> +		0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
> >> +		0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
> >> +		0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
> >> +		0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
> >> +		0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
> >> +		0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
> >> +		0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
> >> +		0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
> >> +		0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
> >> +		0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
> >> +		0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
> >> +		0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
> >> +		0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
> >> +		0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
> >> +	},
> >> +	{	/* Bank 2 */
> >> +		0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
> >> +		0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
> >> +		0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
> >> +		0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
> >> +		0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
> >> +		0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
> >> +		0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
> >> +		0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
> >> +		0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
> >> +		0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
> >> +		0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
> >> +		0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
> >> +		0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
> >> +		0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
> >> +		0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
> >> +		0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
> >> +	},
> >> +	{	/* Bank 3 */
> >> +		0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
> >> +		0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
> >> +		0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
> >> +		0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
> >> +		0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
> >> +		0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
> >> +		0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
> >> +		0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
> >> +		0x20100000, 0x00000010, 0x1f100000, 0x00000011,
> >> +		0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
> >> +		0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
> >> +		0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
> >> +		0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
> >> +		0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
> >> +		0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
> >> +		0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
> >> +	},
> >> +	{	/* Bank 4 */
> >> +		0x17170900, 0x00000009, 0x18160900, 0x00000009,
> >> +		0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
> >> +		0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
> >> +		0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
> >> +		0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
> >> +		0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
> >> +		0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
> >> +		0x19110400, 0x0000030f, 0x18110400, 0x00000310,
> >> +		0x1a100300, 0x00000310, 0x18100300, 0x00000411,
> >> +		0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
> >> +		0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
> >> +		0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
> >> +		0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
> >> +		0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
> >> +		0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
> >> +		0x170a0100, 0x00000816, 0x18090000, 0x00000916,
> >> +	},
> >> +	{	/* Bank 5 */
> >> +		0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
> >> +		0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
> >> +		0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
> >> +		0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
> >> +		0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
> >> +		0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
> >> +		0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
> >> +		0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
> >> +		0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
> >> +		0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
> >> +		0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
> >> +		0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
> >> +		0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
> >> +		0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
> >> +		0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
> >> +		0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
> >> +	},
> >> +	{	/* Bank 6 */
> >> +		0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
> >> +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >> +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
> >> +		0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
> >> +		0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
> >> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >> +		0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >> +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >> +		0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
> >> +		0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
> >> +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
> >> +		0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
> >> +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
> >> +	},
> >> +	{	/* Bank 7 */
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >> +	}
> >> +};
> >> +
> >> +struct mali_c55_resizer_coef_bank {
> >> +	unsigned int bank;
> >
> > This is always equal to the index of the entry in the
> > mali_c55_coefficient_banks array, you can drop it.
> >
> >> +	unsigned int top;
> >> +	unsigned int bottom;
> >
> > The bottom value of bank N is always equal to the top value of bank N+1.
> > You can simplify this by storing a single value.
> >
> >> +};
> >> +
> >> +static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
> >> +	{
> >> +		.bank = 0,
> >> +		.top = 1000,
> >> +		.bottom = 770,
> >> +	},
> >> +	{
> >> +		.bank = 1,
> >> +		.top = 769,
> >> +		.bottom = 600,
> >> +	},
> >> +	{
> >> +		.bank = 2,
> >> +		.top = 599,
> >> +		.bottom = 460,
> >> +	},
> >> +	{
> >> +		.bank = 3,
> >> +		.top = 459,
> >> +		.bottom = 354,
> >> +	},
> >> +	{
> >> +		.bank = 4,
> >> +		.top = 353,
> >> +		.bottom = 273,
> >> +	},
> >> +	{
> >> +		.bank = 5,
> >> +		.top = 272,
> >> +		.bottom = 210,
> >> +	},
> >> +	{
> >> +		.bank = 6,
> >> +		.top = 209,
> >> +		.bottom = 162,
> >> +	},
> >> +	{
> >> +		.bank = 7,
> >> +		.top = 161,
> >> +		.bottom = 125,
> >> +	},
> >> +};
> >> +
> >
> > A small comment would be nice, such as
> >
> > /* Select a bank of resizer coefficients, based on the scaling ratio. */
> >
> >> +static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,
> >
> > This function is related to the resizers. Add "rsz" somewhere in the
> > function name, and pass a resizer pointer.
> >
> >> +						unsigned int crop,
> >> +						unsigned int scale)
> >
> > I think those are the input and output sizes to the scaler. Rename them
> > to make it clearer.
> >
> >> +{
> >> +	unsigned int tmp;
> >
> > tmp is almost always a bad variable name. Please use a more descriptive
> > name, size as rsz_ratio.
> >
> >> +	unsigned int i;
> >> +
> >> +	tmp = (scale * 1000U) / crop;
> >> +
> >> +	for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
> >> +		if (tmp >= mali_c55_coefficient_banks[i].bottom &&
> >> +		    tmp <= mali_c55_coefficient_banks[i].top)
> >> +			return mali_c55_coefficient_banks[i].bank;
> >> +	}
> >> +
> >> +	/*
> >> +	 * We shouldn't ever get here, in theory. As we have no good choices
> >> +	 * simply warn the user and use the first bank of coefficients.
> >> +	 */
> >> +	dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
> >> +	return 0;
> >> +}
> >
> > And everything else belongs to mali-c55-resizer.c. Drop this header
> > file.
> >
> >> +
> >> +#endif /* _MALI_C55_RESIZER_COEFS_H */
> >> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >> new file mode 100644
> >> index 000000000000..0a5a2969d3ce
> >> --- /dev/null
> >> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >> @@ -0,0 +1,779 @@
> >> +// SPDX-License-Identifier: GPL-2.0
> >> +/*
> >> + * ARM Mali-C55 ISP Driver - Image signal processor
> >> + *
> >> + * Copyright (C) 2024 Ideas on Board Oy
> >> + */
> >> +
> >> +#include <linux/math.h>
> >> +#include <linux/minmax.h>
> >> +
> >> +#include <media/media-entity.h>
> >> +#include <media/v4l2-subdev.h>
> >> +
> >> +#include "mali-c55-common.h"
> >> +#include "mali-c55-registers.h"
> >> +#include "mali-c55-resizer-coefs.h"
> >> +
> >> +/* Scaling factor in Q4.20 format. */
> >> +#define MALI_C55_RZR_SCALER_FACTOR	(1U << 20)
> >> +
> >> +static const u32 rzr_non_bypass_src_fmts[] = {
> >> +	MEDIA_BUS_FMT_RGB121212_1X36,
> >> +	MEDIA_BUS_FMT_YUV10_1X30
> >> +};
> >> +
> >> +static const char * const mali_c55_resizer_names[] = {
> >> +	[MALI_C55_RZR_FR] = "resizer fr",
> >> +	[MALI_C55_RZR_DS] = "resizer ds",
> >> +};
> >> +
> >> +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
> >> +				     struct v4l2_subdev_state *state)
> >> +{
> >> +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
> >> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> >> +	struct v4l2_mbus_framefmt *fmt;
> >> +	struct v4l2_rect *crop;
> >> +
> >> +	/* Verify if crop should be enabled. */
> >> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
> >> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> >> +
> >> +	if (fmt->width == crop->width && fmt->height == crop->height)
> >> +		return MALI_C55_BYPASS_CROP;
> >> +
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
> >> +		       crop->left);
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
> >> +		       crop->top);
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
> >> +		       crop->width);
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
> >> +		       crop->height);
> >> +
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
> >> +		       MALI_C55_CROP_ENABLE);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
> >> +					struct v4l2_subdev_state *state)
> >> +{
> >> +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
> >> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> >> +	struct v4l2_rect *crop, *scale;
> >> +	unsigned int h_bank, v_bank;
> >> +	u64 h_scale, v_scale;
> >> +
> >> +	/* Verify if scaling should be enabled. */
> >> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> >> +	scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
> >> +
> >> +	if (crop->width == scale->width && crop->height == scale->height)
> >> +		return MALI_C55_BYPASS_SCALER;
> >> +
> >> +	/* Program the V/H scaling factor in Q4.20 format. */
> >> +	h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
> >> +	v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
> >> +
> >> +	do_div(h_scale, scale->width);
> >> +	do_div(v_scale, scale->height);
> >> +
> >> +	mali_c55_write(mali_c55,
> >> +		       MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
> >> +		       crop->width);
> >> +	mali_c55_write(mali_c55,
> >> +		       MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
> >> +		       crop->height);
> >> +
> >> +	mali_c55_write(mali_c55,
> >> +		       MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
> >> +		       scale->width);
> >> +	mali_c55_write(mali_c55,
> >> +		       MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
> >> +		       scale->height);
> >> +
> >> +	mali_c55_write(mali_c55,
> >> +		       MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
> >> +		       h_scale);
> >> +	mali_c55_write(mali_c55,
> >> +		       MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
> >> +		       v_scale);
> >> +
> >> +	h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
> >> +					     scale->width);
> >> +	mali_c55_write(mali_c55,
> >> +		       MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
> >> +		       h_bank);
> >> +
> >> +	v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
> >> +					     scale->height);
> >> +	mali_c55_write(mali_c55,
> >> +		       MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
> >> +		       v_bank);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
> >> +				 struct v4l2_subdev_state *state)
> >> +{
> >> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> >> +	u32 bypass = 0;
> >> +
> >> +	/* Verify if cropping and scaling should be enabled. */
> >> +	bypass |= mali_c55_rzr_program_crop(rzr, state);
> >> +	bypass |= mali_c55_rzr_program_resizer(rzr, state);
> >> +
> >> +	mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
> >> +			     MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
> >> +			     MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
> >> +			     bypass);
> >> +}
> >> +
> >> +/*
> >> + * Inspect the routing table to know which of the two (mutually exclusive)
> >> + * routes is enabled and return the sink pad id of the active route.
> >> + */
> >> +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
> >> +{
> >> +	struct v4l2_subdev_krouting *routing = &state->routing;
> >> +	struct v4l2_subdev_route *route;
> >> +
> >> +	/* A single route is enabled at a time. */
> >> +	for_each_active_route(routing, route)
> >> +		return route->sink_pad;
> >> +
> >> +	return MALI_C55_RZR_SINK_PAD;
> >> +}
> >> +
> >> +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
> >> +{
> >> +	u32 corrected_code = 0;
> >> +
> >> +	/*
> >> +	 * The ISP takes input in a 20-bit format, but can only output 16-bit
> >> +	 * RAW bayer data (with the 4 least significant bits from the input
> >> +	 * being lost). Return the 16-bit version of the 20-bit input formats.
> >> +	 */
> >> +	switch (mbus_code) {
> >> +	case MEDIA_BUS_FMT_SBGGR20_1X20:
> >> +		corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
> >> +		break;
> >> +	case MEDIA_BUS_FMT_SGBRG20_1X20:
> >> +		corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
> >> +		break;
> >> +	case MEDIA_BUS_FMT_SGRBG20_1X20:
> >> +		corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
> >> +		break;
> >> +	case MEDIA_BUS_FMT_SRGGB20_1X20:
> >> +		corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
> >> +		break;
> >> +	}
> >> +
> >> +	return corrected_code;
> >> +}
> >> +
> >> +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> >> +				      struct v4l2_subdev_state *state,
> >> +				      struct v4l2_subdev_krouting *routing)
> >> +{
> >> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >> +						    sd);
> >> +	unsigned int active_sink = UINT_MAX;
> >> +	struct v4l2_mbus_framefmt *src_fmt;
> >> +	struct v4l2_rect *crop, *compose;
> >> +	struct v4l2_subdev_route *route;
> >> +	unsigned int active_routes = 0;
> >> +	struct v4l2_mbus_framefmt *fmt;
> >> +	int ret;
> >> +
> >> +	ret = v4l2_subdev_routing_validate(sd, routing, 0);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	/* Only a single route can be enabled at a time. */
> >> +	for_each_active_route(routing, route) {
> >> +		if (++active_routes > 1) {
> >> +			dev_err(rzr->mali_c55->dev,
> >> +				"Only one route can be active");
> >> +			return -EINVAL;
> >> +		}
> >> +
> >> +		active_sink = route->sink_pad;
> >> +	}
> >> +	if (active_sink == UINT_MAX) {
> >> +		dev_err(rzr->mali_c55->dev, "One route has to be active");
> >> +		return -EINVAL;
> >> +	}
> >> +
> >> +	ret = v4l2_subdev_set_routing(sd, state, routing);
> >> +	if (ret) {
> >> +		dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
> >> +		return ret;
> >> +	}
> >> +
> >> +	fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
> >> +	crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
> >> +	compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
> >> +
> >> +	fmt->width = MALI_C55_DEFAULT_WIDTH;
> >> +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >> +	fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >> +	fmt->field = V4L2_FIELD_NONE;
> >> +
> >> +	if (active_sink == MALI_C55_RZR_SINK_PAD) {
> >> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >> +
> >> +		crop->left = crop->top = 0;
> >> +		crop->width = MALI_C55_DEFAULT_WIDTH;
> >> +		crop->height = MALI_C55_DEFAULT_HEIGHT;
> >> +
> >> +		*compose = *crop;
> >> +	} else {
> >> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >> +	}
> >> +
> >> +	/* Propagate the format to the source pad */
> >> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
> >> +					       0);
> >> +	*src_fmt = *fmt;
> >> +
> >> +	/* In the event this is the bypass pad the mbus code needs correcting */
> >> +	if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
> >> +		src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
> >> +				       struct v4l2_subdev_state *state,
> >> +				       struct v4l2_subdev_mbus_code_enum *code)
> >> +{
> >> +	struct v4l2_mbus_framefmt *sink_fmt;
> >> +	const struct mali_c55_isp_fmt *fmt;
> >> +	unsigned int index = 0;
> >> +	u32 sink_pad;
> >> +
> >> +	switch (code->pad) {
> >> +	case MALI_C55_RZR_SINK_PAD:
> >> +		if (code->index)
> >> +			return -EINVAL;
> >> +
> >> +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >> +
> >> +		return 0;
> >> +	case MALI_C55_RZR_SOURCE_PAD:
> >> +		sink_pad = mali_c55_rzr_get_active_sink(state);
> >> +		sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> >> +
> >> +		/*
> >> +		 * If the active route is from the Bypass sink pad, then the
> >> +		 * source pad is a simple passthrough of the sink format,
> >> +		 * downshifted to 16-bits.
> >> +		 */
> >> +
> >> +		if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >> +			if (code->index)
> >> +				return -EINVAL;
> >> +
> >> +			code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> >> +			if (!code->code)
> >> +				return -EINVAL;
> >> +
> >> +			return 0;
> >> +		}
> >> +
> >> +		/*
> >> +		 * If the active route is from the non-bypass sink then we can
> >> +		 * select either RGB or conversion to YUV.
> >> +		 */
> >> +
> >> +		if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
> >> +			return -EINVAL;
> >> +
> >> +		code->code = rzr_non_bypass_src_fmts[code->index];
> >> +
> >> +		return 0;
> >> +	case MALI_C55_RZR_SINK_BYPASS_PAD:
> >> +		for_each_mali_isp_fmt(fmt) {
> >> +			if (index++ == code->index) {
> >> +				code->code = fmt->code;
> >> +				return 0;
> >> +			}
> >> +		}
> >> +
> >> +		break;
> >> +	}
> >> +
> >> +	return -EINVAL;
> >> +}
> >> +
> >> +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
> >> +					struct v4l2_subdev_state *state,
> >> +					struct v4l2_subdev_frame_size_enum *fse)
> >> +{
> >> +	if (fse->index)
> >> +		return -EINVAL;
> >> +
> >> +	fse->max_width = MALI_C55_MAX_WIDTH;
> >> +	fse->max_height = MALI_C55_MAX_HEIGHT;
> >> +	fse->min_width = MALI_C55_MIN_WIDTH;
> >> +	fse->min_height = MALI_C55_MIN_HEIGHT;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
> >> +				     struct v4l2_subdev_state *state,
> >> +				     struct v4l2_subdev_format *format)
> >> +{
> >> +	struct v4l2_mbus_framefmt *fmt = &format->format;
> >> +	struct v4l2_rect *rect;
> >> +	unsigned int sink_pad;
> >> +
> >> +	/*
> >> +	 * Clamp to min/max and then reset crop and compose rectangles to the
> >> +	 * newly applied size.
> >> +	 */
> >> +	clamp_t(unsigned int, fmt->width,
> >> +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >> +	clamp_t(unsigned int, fmt->height,
> >> +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >> +
> >> +	sink_pad = mali_c55_rzr_get_active_sink(state);
> >> +	if (sink_pad == MALI_C55_RZR_SINK_PAD) {
> >> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >> +
> >> +		rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> >> +		rect->left = 0;
> >> +		rect->top = 0;
> >> +		rect->width = fmt->width;
> >> +		rect->height = fmt->height;
> >> +
> >> +		rect = v4l2_subdev_state_get_compose(state,
> >> +						     MALI_C55_RZR_SINK_PAD);
> >> +		rect->left = 0;
> >> +		rect->top = 0;
> >> +		rect->width = fmt->width;
> >> +		rect->height = fmt->height;
> >> +	} else {
> >> +		/*
> >> +		 * Make sure the media bus code is one of the supported
> >> +		 * ISP input media bus codes.
> >> +		 */
> >> +		if (!mali_c55_isp_is_format_supported(fmt->code))
> >> +			fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
> >> +	}
> >> +
> >> +	*v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
> >> +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> >> +				       struct v4l2_subdev_state *state,
> >> +				       struct v4l2_subdev_format *format)
> >> +{
> >> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >> +						    sd);
> >> +	struct v4l2_mbus_framefmt *fmt = &format->format;
> >> +	struct v4l2_mbus_framefmt *sink_fmt;
> >> +	struct v4l2_rect *crop, *compose;
> >> +	unsigned int sink_pad;
> >> +	unsigned int i;
> >> +
> >> +	sink_pad = mali_c55_rzr_get_active_sink(state);
> >> +	sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> >> +	crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
> >> +	compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
> >> +
> >> +	/* FR Bypass pipe. */
> >> +
> >> +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >> +		/*
> >> +		 * Format on the source pad is the same as the one on the
> >> +		 * sink pad, downshifted to 16-bits.
> >> +		 */
> >> +		fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> >> +		if (!fmt->code)
> >> +			return -EINVAL;
> >> +
> >> +		/* RAW bypass disables scaling and cropping. */
> >> +		crop->top = compose->top = 0;
> >> +		crop->left = compose->left = 0;
> >> +		fmt->width = crop->width = compose->width = sink_fmt->width;
> >> +		fmt->height = crop->height = compose->height = sink_fmt->height;
> >> +
> >> +		*v4l2_subdev_state_get_format(state,
> >> +					      MALI_C55_RZR_SOURCE_PAD) = *fmt;
> >> +
> >> +		return 0;
> >> +	}
> >> +
> >> +	/* Regular processing pipe. */
> >> +
> >> +	for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> >> +		if (fmt->code == rzr_non_bypass_src_fmts[i])
> >> +			break;
> >> +	}
> >> +
> >> +	if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
> >> +		dev_dbg(rzr->mali_c55->dev,
> >> +			"Unsupported mbus code 0x%x: using default\n",
> >> +			fmt->code);
> >> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >> +	}
> >> +
> >> +	/*
> >> +	 * The source pad format size comes directly from the sink pad
> >> +	 * compose rectangle.
> >> +	 */
> >> +	fmt->width = compose->width;
> >> +	fmt->height = compose->height;
> >> +
> >> +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
> >> +				struct v4l2_subdev_state *state,
> >> +				struct v4l2_subdev_format *format)
> >> +{
> >> +	/*
> >> +	 * On sink pads fmt is either fixed for the 'regular' processing
> >> +	 * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
> >> +	 * pad.
> >> +	 *
> >> +	 * On source pad sizes are the result of crop+compose on the sink
> >> +	 * pad sizes, while the format depends on the active route.
> >> +	 */
> >> +
> >> +	if (format->pad != MALI_C55_RZR_SOURCE_PAD)
> >> +		return mali_c55_rzr_set_sink_fmt(sd, state, format);
> >> +
> >> +	return mali_c55_rzr_set_source_fmt(sd, state, format);
> >> +}
> >> +
> >> +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
> >> +				      struct v4l2_subdev_state *state,
> >> +				      struct v4l2_subdev_selection *sel)
> >> +{
> >> +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
> >> +		return -EINVAL;
> >> +
> >> +	if (sel->target != V4L2_SEL_TGT_CROP &&
> >> +	    sel->target != V4L2_SEL_TGT_COMPOSE)
> >> +		return -EINVAL;
> >> +
> >> +	sel->r = sel->target == V4L2_SEL_TGT_CROP
> >> +	       ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
> >> +	       : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
> >> +				      struct v4l2_subdev_state *state,
> >> +				      struct v4l2_subdev_selection *sel)
> >> +{
> >> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >> +						    sd);
> >> +	struct v4l2_mbus_framefmt *source_fmt;
> >> +	struct v4l2_mbus_framefmt *sink_fmt;
> >> +	struct v4l2_rect *crop, *compose;
> >> +
> >> +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
> >> +		return -EINVAL;
> >> +
> >> +	if (sel->target != V4L2_SEL_TGT_CROP &&
> >> +	    sel->target != V4L2_SEL_TGT_COMPOSE)
> >> +		return -EINVAL;
> >> +
> >> +	source_fmt = v4l2_subdev_state_get_format(state,
> >> +						  MALI_C55_RZR_SOURCE_PAD);
> >> +	sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
> >> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> >> +	compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> >> +
> >> +	/* RAW bypass disables crop/scaling. */
> >> +	if (mali_c55_format_is_raw(source_fmt->code)) {
> >> +		crop->top = compose->top = 0;
> >> +		crop->left = compose->left = 0;
> >> +		crop->width = compose->width = sink_fmt->width;
> >> +		crop->height = compose->height = sink_fmt->height;
> >> +
> >> +		sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> >> +
> >> +		return 0;
> >> +	}
> >> +
> >> +	/* During streaming, it is allowed to only change the crop rectangle. */
> >> +	if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
> >> +		return -EINVAL;
> >> +
> >> +	 /*
> >> +	  * Update the desired target and then clamp the crop rectangle to the
> >> +	  * sink format sizes and the compose size to the crop sizes.
> >> +	  */
> >> +	if (sel->target == V4L2_SEL_TGT_CROP)
> >> +		*crop = sel->r;
> >> +	else
> >> +		*compose = sel->r;
> >> +
> >> +	clamp_t(unsigned int, crop->left, 0,  sink_fmt->width);
> >> +	clamp_t(unsigned int, crop->top, 0,  sink_fmt->height);
> >> +	clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
> >> +		sink_fmt->width - crop->left);
> >> +	clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
> >> +		sink_fmt->height - crop->top);
> >> +
> >> +	if (rzr->streaming) {
> >> +		/*
> >> +		 * Apply at runtime a crop rectangle on the resizer's sink only
> >> +		 * if it doesn't require re-programming the scaler output sizes
> >> +		 * as it would require changing the output buffer sizes as well.
> >> +		 */
> >> +		if (sel->r.width < compose->width ||
> >> +		    sel->r.height < compose->height)
> >> +			return -EINVAL;
> >> +
> >> +		*crop = sel->r;
> >> +		mali_c55_rzr_program(rzr, state);
> >> +
> >> +		return 0;
> >> +	}
> >> +
> >> +	compose->left = 0;
> >> +	compose->top = 0;
> >> +	clamp_t(unsigned int, compose->left, 0,  sink_fmt->width);
> >> +	clamp_t(unsigned int, compose->top, 0,  sink_fmt->height);
> >> +	clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
> >> +	clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
> >> +
> >> +	sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> >> +				    struct v4l2_subdev_state *state,
> >> +				    enum v4l2_subdev_format_whence which,
> >> +				    struct v4l2_subdev_krouting *routing)
> >> +{
> >> +	if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
> >> +	    media_entity_is_streaming(&sd->entity))
> >> +		return -EBUSY;
> >> +
> >> +	return __mali_c55_rzr_set_routing(sd, state, routing);
> >> +}
> >> +
> >> +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
> >> +	.enum_mbus_code		= mali_c55_rzr_enum_mbus_code,
> >> +	.enum_frame_size	= mali_c55_rzr_enum_frame_size,
> >> +	.get_fmt		= v4l2_subdev_get_fmt,
> >> +	.set_fmt		= mali_c55_rzr_set_fmt,
> >> +	.get_selection		= mali_c55_rzr_get_selection,
> >> +	.set_selection		= mali_c55_rzr_set_selection,
> >> +	.set_routing		= mali_c55_rzr_set_routing,
> >> +};
> >> +
> >> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
> >> +{
> >> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> >> +	struct v4l2_subdev *sd = &rzr->sd;
> >> +	struct v4l2_subdev_state *state;
> >> +	unsigned int sink_pad;
> >> +
> >> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> >> +
> >> +	sink_pad = mali_c55_rzr_get_active_sink(state);
> >> +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >> +		/* Bypass FR pipe processing if the bypass route is active. */
> >> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >> +				     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
> >> +				     MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
> >> +		goto unlock_state;
> >> +	}
> >> +
> >> +	/* Disable bypass and use regular processing. */
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >> +			     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
> >> +	mali_c55_rzr_program(rzr, state);
> >> +
> >> +unlock_state:
> >> +	rzr->streaming = true;
> >> +	v4l2_subdev_unlock_state(state);
> >> +}
> >> +
> >> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
> >> +{
> >> +	struct v4l2_subdev *sd = &rzr->sd;
> >> +	struct v4l2_subdev_state *state;
> >> +
> >> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> >> +	rzr->streaming = false;
> >> +	v4l2_subdev_unlock_state(state);
> >> +}
> >> +
> >> +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
> >> +	.pad	= &mali_c55_resizer_pad_ops,
> >> +};
> >> +
> >> +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
> >> +				   struct v4l2_subdev_state *state)
> >> +{
> >> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >> +						    sd);
> >> +	struct v4l2_subdev_krouting routing = { };
> >> +	struct v4l2_subdev_route *routes;
> >> +	unsigned int i;
> >> +	int ret;
> >> +
> >> +	routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
> >> +	if (!routes)
> >> +		return -ENOMEM;
> >> +
> >> +	for (i = 0; i < rzr->num_routes; ++i) {
> >> +		struct v4l2_subdev_route *route = &routes[i];
> >> +
> >> +		route->sink_pad = i
> >> +				? MALI_C55_RZR_SINK_BYPASS_PAD
> >> +				: MALI_C55_RZR_SINK_PAD;
> >> +		route->source_pad = MALI_C55_RZR_SOURCE_PAD;
> >> +		if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
> >> +			route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
> >> +	}
> >> +
> >> +	routing.num_routes = rzr->num_routes;
> >> +	routing.routes = routes;
> >> +
> >> +	ret = __mali_c55_rzr_set_routing(sd, state, &routing);
> >> +	kfree(routes);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
> >> +	.init_state = mali_c55_rzr_init_state,
> >> +};
> >> +
> >> +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
> >> +						  unsigned int index)
> >> +{
> >> +	const unsigned int scaler_filt_coefmem_addrs[][2] = {
> >> +		[MALI_C55_RZR_FR] = {
> >> +			0x034A8, /* hfilt */
> >> +			0x044A8  /* vfilt */
> >
> > Lowercase hex constants.
> >
> >> +		},
> >> +		[MALI_C55_RZR_DS] = {
> >> +			0x014A8, /* hfilt */
> >> +			0x024A8  /* vfilt */
> >> +		},
> >> +	};
> >> +	unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
> >> +	unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
> >> +	unsigned int i, j;
> >> +
> >> +	for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
> >> +		for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
> >> +			mali_c55_write(mali_c55, haddr,
> >> +				mali_c55_scaler_h_filter_coefficients[i][j]);
> >> +			mali_c55_write(mali_c55, vaddr,
> >> +				mali_c55_scaler_v_filter_coefficients[i][j]);
> >> +
> >> +			haddr += sizeof(u32);
> >> +			vaddr += sizeof(u32);
> >> +		}
> >> +	}
> >> +}
> >> +
> >> +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
> >> +{
> >> +	unsigned int i;
> >> +	int ret;
> >> +
> >> +	for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
> >> +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> >> +		struct v4l2_subdev *sd = &rzr->sd;
> >> +		unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
> >> +
> >> +		rzr->id = i;
> >> +		rzr->streaming = false;
> >> +
> >> +		if (rzr->id == MALI_C55_RZR_FR)
> >> +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
> >> +		else
> >> +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
> >> +
> >> +		mali_c55_resizer_program_coefficients(mali_c55, i);
> >> +
> >> +		v4l2_subdev_init(sd, &mali_c55_resizer_ops);
> >> +		sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
> >> +			     | V4L2_SUBDEV_FL_STREAMS;
> >> +		sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
> >> +		sd->internal_ops = &mali_c55_resizer_internal_ops;
> >> +		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
> >> +			 MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
> >> +
> >> +		rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
> >> +		rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
> >> +
> >> +		/* Only the FR pipe has a bypass pad. */
> >> +		if (rzr->id == MALI_C55_RZR_FR) {
> >> +			rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
> >> +							MEDIA_PAD_FL_SINK;
> >> +			rzr->num_routes = 2;
> >> +		} else {
> >> +			num_pads -= 1;
> >> +			rzr->num_routes = 1;
> >> +		}
> >> +
> >> +		ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
> >> +		if (ret)
> >> +			return ret;
> >> +
> >> +		ret = v4l2_subdev_init_finalize(sd);
> >> +		if (ret)
> >> +			goto err_cleanup;
> >> +
> >> +		ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >> +		if (ret)
> >> +			goto err_cleanup;
> >> +
> >> +		rzr->mali_c55 = mali_c55;
> >> +	}
> >> +
> >> +	return 0;
> >> +
> >> +err_cleanup:
> >> +	for (; i >= 0; --i) {
> >> +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> >> +		struct v4l2_subdev *sd = &rzr->sd;
> >> +
> >> +		v4l2_subdev_cleanup(sd);
> >> +		media_entity_cleanup(&sd->entity);
> >> +	}
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
> >> +{
> >> +	unsigned int i;
> >> +
> >> +	for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
> >> +		struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
> >> +
> >> +		if (!resizer->mali_c55)
> >> +			continue;
> >> +
> >> +		v4l2_device_unregister_subdev(&resizer->sd);
> >> +		v4l2_subdev_cleanup(&resizer->sd);
> >> +		media_entity_cleanup(&resizer->sd.entity);
> >> +	}
> >> +}
> >> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >> new file mode 100644
> >> index 000000000000..c7e699741c6d
> >> --- /dev/null
> >> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >> @@ -0,0 +1,402 @@
> >> +// SPDX-License-Identifier: GPL-2.0
> >> +/*
> >> + * ARM Mali-C55 ISP Driver - Test pattern generator
> >> + *
> >> + * Copyright (C) 2024 Ideas on Board Oy
> >> + */
> >> +
> >> +#include <linux/minmax.h>
> >> +#include <linux/string.h>
> >> +
> >> +#include <media/media-entity.h>
> >> +#include <media/v4l2-ctrls.h>
> >> +#include <media/v4l2-subdev.h>
> >> +
> >> +#include "mali-c55-common.h"
> >> +#include "mali-c55-registers.h"
> >> +
> >> +#define MALI_C55_TPG_SRC_PAD		0
> >> +#define MALI_C55_TPG_FIXED_HBLANK	0x20
> >> +#define MALI_C55_TPG_MAX_VBLANK		0xFFFF
> >
> > Lowercase hex constants.
> >
> >> +#define MALI_C55_TPG_PIXEL_RATE		100000000
> >
> > This should be exposed to applications using the V4L2_CID_PIXEL_RATE
> > control (read-only).
> >
> >> +
> >> +static const char * const mali_c55_tpg_test_pattern_menu[] = {
> >> +	"Flat field",
> >> +	"Horizontal gradient",
> >> +	"Vertical gradient",
> >> +	"Vertical bars",
> >> +	"Arbitrary rectangle",
> >> +	"White frame on black field"
> >> +};
> >> +
> >> +static const u32 mali_c55_tpg_mbus_codes[] = {
> >> +	MEDIA_BUS_FMT_SRGGB20_1X20,
> >> +	MEDIA_BUS_FMT_RGB202020_1X60,
> >> +};
> >> +
> >> +static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
> >> +				       int *def_vblank, int *min_vblank)
> >
> > unsigned int ?
> >
> >> +{
> >> +	unsigned int hts;
> >> +	int tgt_fps;
> >> +	int vblank;
> >> +
> >> +	hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
> >> +
> >> +	/*
> >> +	 * The ISP has minimum vertical blanking requirements that must be
> >> +	 * adhered to by the TPG. The minimum is a function of the Iridix blocks
> >> +	 * clocking requirements and the width of the image and horizontal
> >> +	 * blanking, but if we assume the worst case iVariance and sVariance
> >> +	 * values then it boils down to the below.
> >> +	 */
> >> +	*min_vblank = 15 + (120500 / hts);
> >
> > I wonder if this should round up.
> 
> Maybe? I think the difference is probably too minor to have a practical effect

Just to make sure we won't have any problem by having a just too short
minimum vblank when rounding down.

> >> +
> >> +	/*
> >> +	 * We need to set a sensible default vblank for whatever format height
> >> +	 * we happen to be given from set_fmt(). This function just targets
> >> +	 * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
> >> +	 * If we can't get 5fps we'll take whatever the minimum vblank gives us.
> >> +	 */
> >> +	tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
> >> +
> >> +	if (tgt_fps < 5)
> >> +		vblank = *min_vblank;
> >> +	else
> >> +		vblank = MALI_C55_TPG_PIXEL_RATE / hts
> >> +		       / max(rounddown(tgt_fps, 15), 5);
> >> +
> >> +	*def_vblank = ALIGN_DOWN(vblank, 2) - format->height;
> >
> > "vblank = vblank - height" doesn't seem right. The "else" branch stores
> > a vts in vblank, which doesn't seem right either. Maybe you meant
> > something like
> >
> > 	if (tgt_fps < 5)
> > 		def_vts = *min_vblank + format->height;
> > 	else
> > 		def_vts = MALI_C55_TPG_PIXEL_RATE / hts
> > 			/ max(rounddown(tgt_fps, 15), 5);
> >
> > 	*def_vblank = ALIGN_DOWN(def_vts, 2) - format->height;
> 
> I did, thank you.
> 
> >> +}
> >> +
> >> +static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
> >> +{
> >> +	struct mali_c55_tpg *tpg = container_of(ctrl->handler,
> >> +						struct mali_c55_tpg,
> >> +						ctrls.handler);
> >> +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
> >> +
> >
> > Should you return here if the pipeline isn't streaming ?
> 
> Yes probably; or if (pm_runtime_get_if_in_use()) would be the usual model I guess.

I think so yes.

> >> +	switch (ctrl->id) {
> >> +	case V4L2_CID_TEST_PATTERN:
> >> +		mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
> >> +			       ctrl->val);
> >> +		break;
> >> +	case V4L2_CID_VBLANK:
> >> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
> >> +				     MALI_C55_REG_VBLANK_MASK, ctrl->val);
> >> +		break;
> >> +	default:
> >> +		dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
> >> +		return -EINVAL;
> >
> > Can this happen ?
> 
> Not unless somebody breaks something

Then I think you can drop the error message.

> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
> >> +	.s_ctrl = &mali_c55_tpg_s_ctrl,
> >> +};
> >> +
> >> +static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
> >> +				   struct v4l2_subdev *sd)
> >> +{
> >> +	struct v4l2_subdev_state *state;
> >> +	struct v4l2_mbus_framefmt *fmt;
> >> +
> >> +	/*
> >> +	 * hblank needs setting, but is a read-only control and thus won't be
> >> +	 * called during __v4l2_ctrl_handler_setup(). Do it here instead.
> >> +	 */
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
> >> +			     MALI_C55_REG_HBLANK_MASK,
> >> +			     MALI_C55_TPG_FIXED_HBLANK);
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >> +			     MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
> >> +
> >> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> >> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >> +
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >> +			     MALI_C55_TEST_PATTERN_RGB_MASK,
> >> +			     fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
> >> +					  0x01 : 0x0);
> >> +
> >> +	v4l2_subdev_unlock_state(state);
> >> +}
> >> +
> >> +static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
> >> +{
> >> +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
> >> +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
> >> +
> >> +	if (!enable) {
> >> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >> +				MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
> >> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >> +				MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
> >> +		return 0;
> >> +	}
> >> +
> >> +	/*
> >> +	 * One might reasonably expect the framesize to be set here
> >> +	 * given it's configurable in .set_fmt(), but it's done in the
> >> +	 * ISP subdevice's stream on func instead, as the same register
> >
> > s/func/function/
> >
> >> +	 * is also used to indicate the size of the data coming from the
> >> +	 * sensor.
> >> +	 */
> >> +	mali_c55_tpg_configure(mali_c55, sd);
> >
> > 	mali_c55_tpg_configure(tpg);
> >
> >> +	__v4l2_ctrl_handler_setup(sd->ctrl_handler);
> >> +
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >> +			     MALI_C55_TEST_PATTERN_ON_OFF,
> >> +			     MALI_C55_TEST_PATTERN_ON_OFF);
> >> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >> +			     MALI_C55_REG_GEN_VIDEO_ON_MASK,
> >> +			     MALI_C55_REG_GEN_VIDEO_ON_MASK);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
> >> +	.s_stream = &mali_c55_tpg_s_stream,
> >
> > Can we use .enable_streams() and .disable_streams() ?
> 
> Yes, with some extra patches from Tomi that Sakari is picking - I'll base on top of those for the v6.
> 
> >> +};
> >> +
> >> +static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
> >> +				       struct v4l2_subdev_state *state,
> >> +				       struct v4l2_subdev_mbus_code_enum *code)
> >> +{
> >> +	if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >> +		return -EINVAL;
> >> +
> >> +	code->code = mali_c55_tpg_mbus_codes[code->index];
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
> >> +					struct v4l2_subdev_state *state,
> >> +					struct v4l2_subdev_frame_size_enum *fse)
> >> +{
> >
> > You sohuld verify here that fse->code is a supported value and return
> > -EINVAL otherwise.
> >
> >> +	if (fse->index > 0 || fse->pad > sd->entity.num_pads)
> >
> > Drop the pad check, it's done in the subdev core already.
> >
> >> +		return -EINVAL;
> >> +
> >> +	fse->min_width = MALI_C55_MIN_WIDTH;
> >> +	fse->max_width = MALI_C55_MAX_WIDTH;
> >> +	fse->min_height = MALI_C55_MIN_HEIGHT;
> >> +	fse->max_height = MALI_C55_MAX_HEIGHT;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
> >> +				struct v4l2_subdev_state *state,
> >> +				struct v4l2_subdev_format *format)
> >> +{
> >> +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
> >> +	struct v4l2_mbus_framefmt *fmt = &format->format;
> >> +	int vblank_def, vblank_min;
> >> +	unsigned int i;
> >> +
> >> +	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> >> +		if (fmt->code == mali_c55_tpg_mbus_codes[i])
> >> +			break;
> >> +	}
> >> +
> >> +	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >> +
> >> +	/*
> >> +	 * The TPG says that the test frame timing generation logic expects a
> >> +	 * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
> >> +	 * handle anything smaller than 128x128 it seems pointless to allow a
> >> +	 * smaller frame.
> >> +	 */
> >> +	clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> >> +		MALI_C55_MAX_WIDTH);
> >> +	clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> >> +		MALI_C55_MAX_HEIGHT);
> >> +
> >> +	*v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;
> >
> > You're allowing userspace to set fmt->field, as well as all the
> > colorspace parameters, to random values. I would instead do something
> > like
> >
> > 	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> > 		if (format->format.code == mali_c55_tpg_mbus_codes[i])
> > 			break;
> > 	}
> >
> > 	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> > 		format->format.code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >
> > 	format->format.width = clamp(format->format.width,
> > 				     MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> > 	format->format.height = clamp(format->format.height,
> > 				      MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >
> > 	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> > 	fmt->code = format->format.code;
> > 	fmt->width = format->format.width;
> > 	fmt->height = format->format.height;
> >
> > 	format->format = *fmt;
> >
> > Alternatively (which I think I like better),
> >
> > 	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >
> > 	fmt->code = format->format.code;
> >
> > 	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> > 		if (fmt->code == mali_c55_tpg_mbus_codes[i])
> > 			break;
> > 	}
> >
> > 	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> > 		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >
> > 	fmt->width = clamp(format->format.width,
> > 			   MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> > 	fmt->height = clamp(format->format.height,
> > 			    MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >
> > 	format->format = *fmt;
> >
> >> +
> >> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY)
> >> +		return 0;
> >> +
> >> +	__mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
> >> +	__v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
> >> +				 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
> >> +	__v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);
> >
> > Move those three calls to a separate function, it will be reused below.
> > I'd name is mali_c55_tpg_update_vblank(). You can fold
> > __mali_c55_tpg_calc_vblank() in it.
> >
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
> >> +	.enum_mbus_code		= mali_c55_tpg_enum_mbus_code,
> >> +	.enum_frame_size	= mali_c55_tpg_enum_frame_size,
> >> +	.get_fmt		= v4l2_subdev_get_fmt,
> >> +	.set_fmt		= mali_c55_tpg_set_fmt,
> >> +};
> >> +
> >> +static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
> >> +	.video	= &mali_c55_tpg_video_ops,
> >> +	.pad	= &mali_c55_tpg_pad_ops,
> >> +};
> >> +
> >> +static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
> >> +				   struct v4l2_subdev_state *sd_state)
> >
> > You name this variable state in every other subdev operation handler.
> >
> >> +{
> >> +	struct v4l2_mbus_framefmt *fmt =
> >> +		v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
> >> +
> >> +	fmt->width = MALI_C55_DEFAULT_WIDTH;
> >> +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >> +	fmt->field = V4L2_FIELD_NONE;
> >> +	fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >
> > Initialize the colorspace fields too.
> >
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
> >> +	.init_state = mali_c55_tpg_init_state,
> >> +};
> >> +
> >> +static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
> >> +{
> >> +	struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
> >> +	struct v4l2_subdev *sd = &mali_c55->tpg.sd;
> >> +	struct v4l2_mbus_framefmt *format;
> >> +	struct v4l2_subdev_state *state;
> >> +	int vblank_def, vblank_min;
> >> +	int ret;
> >> +
> >> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> >> +	format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >> +
> >> +	ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);
> >
> > You have 3 controls.
> >
> >> +	if (ret)
> >> +		goto err_unlock;
> >> +
> >> +	ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
> >> +				&mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
> >> +				ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
> >> +				0, 3, mali_c55_tpg_test_pattern_menu);
> >> +
> >> +	/*
> >> +	 * We fix hblank at the minimum allowed value and control framerate
> >> +	 * solely through the vblank control.
> >> +	 */
> >> +	ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
> >> +				&mali_c55_tpg_ctrl_ops,
> >> +				V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
> >> +				MALI_C55_TPG_FIXED_HBLANK, 1,
> >> +				MALI_C55_TPG_FIXED_HBLANK);
> >> +	if (ctrls->hblank)
> >> +		ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
> >> +
> >> +	__mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);
> >
> > Drop this and initialize the control with default values. You can then
> > update the value by calling mali_c55_tpg_update_vblank() in
> > mali_c55_register_tpg().
> >
> > The reason is to share the same mutex between the control handler and
> > the subdev active state without having to add a separate mutex in the
> > mali_c55_tpg structure. The simplest way to do so is to initialize the
> > controls first, set sd->state_lock to point to the control handler lock,
> > and call v4l2_subdev_init_finalize() as the last step. As a consequence,
> > you can't access the active state when initializing controls.
> >
> > You can alternatively keep the lock in mali_c55_tpg and set
> > sd->state_lock to point to it, but I think that's more complex.
> >
> >> +	ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
> >> +					  &mali_c55_tpg_ctrl_ops,
> >> +					  V4L2_CID_VBLANK, vblank_min,
> >> +					  MALI_C55_TPG_MAX_VBLANK, 1,
> >> +					  vblank_def);
> >> +
> >> +	if (ctrls->handler.error) {
> >> +		dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
> >> +		ret = ctrls->handler.error;
> >> +		goto err_free_handler;
> >> +	}
> >> +
> >> +	ctrls->handler.lock = &mali_c55->tpg.lock;
> >
> > Drop this and drop the mutex. The control handler will use its internal
> > mutex.
> >
> >> +	mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
> >> +
> >> +	v4l2_subdev_unlock_state(state);
> >> +
> >> +	return 0;
> >> +
> >> +err_free_handler:
> >> +	v4l2_ctrl_handler_free(&ctrls->handler);
> >> +err_unlock:
> >> +	v4l2_subdev_unlock_state(state);
> >> +	return ret;
> >> +}
> >> +
> >> +int mali_c55_register_tpg(struct mali_c55 *mali_c55)
> >> +{
> >> +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
> >> +	struct v4l2_subdev *sd = &tpg->sd;
> >> +	struct media_pad *pad = &tpg->pad;
> >> +	int ret;
> >> +
> >> +	mutex_init(&tpg->lock);
> >> +
> >> +	v4l2_subdev_init(sd, &mali_c55_tpg_ops);
> >> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> >> +	sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
> >
> > Should we introduce a TPG function ?
> 
> Hmmm I vacillate a bit. I don't see that it would hurt right now, but on the other hand I think 
> there's some value in pretending they're sensors - on the grounds that they should be handled 
> identically as much as possible. I'd be quite wary if we ever saw "if (sd->entity.function == 
> MEDIA_ENT_F_TPG)" somewhere.

The TPG doesn't expose all the API elements we expect from raw sensors,
so we'll have userspace code that is TPG-specific. I suppose it will
look for th TPG entity by name, so we don't need a separate function.
The other part to consider is generic code looking for a raw sensor that
would pick the TPG by mistake. I'm not sure how big of a risk that would
be.

> >> +	sd->internal_ops = &mali_c55_tpg_internal_ops;
> >> +	strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
> >> +
> >> +	pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;
> >
> > I don't think MEDIA_PAD_FL_MUST_CONNECT is right.
> 
> The docs say "If this flag is set and the pad is linked to any other pad, then at least one of those 
> links must be enabled for the entity to be able to stream.", and that's the case here right?

The MEDIA_PAD_FL_MUST_CONNECT flag indicates that at least one link must
be enabled for that pad in order for the entity to be included in a
pipeline. The intent is to check in the core if, for instance, a CSI-2
RX input is connected to something, and refuse streaming if it isn't.

The TPG output needs to be connected to something for the TPG to be used
in a pipeline, but the TPG won't be in the pipeline in the first place
if its output is not connected. The flag is therefore not necessary.

> >> +	ret = media_entity_pads_init(&sd->entity, 1, pad);
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev,
> >> +			"Failed to initialize media entity pads\n");
> >> +		goto err_destroy_mutex;
> >> +	}
> >> +
> >
> > 	sd->state_lock = sd->ctrl_handler->lock;
> >
> > to use the same lock for the controls and the active state. You need to
> > move this line and the v4l2_subdev_init_finalize() call after
> > mali_c55_tpg_init_controls() to get the control handler lock initialized
> > first.
> >
> >> +	ret = v4l2_subdev_init_finalize(sd);
> >> +	if (ret)
> >> +		goto err_cleanup_media_entity;
> >> +
> >> +	ret = mali_c55_tpg_init_controls(mali_c55);
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev,
> >> +			"Error initialising controls\n");
> >> +		goto err_cleanup_subdev;
> >> +	}
> >> +
> >> +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
> >> +		goto err_free_ctrl_handler;
> >> +	}
> >> +
> >> +	/*
> >> +	 * By default the colour settings lead to a very dim image that is
> >> +	 * nearly indistinguishable from black on some monitor settings. Ramp
> >> +	 * them up a bit so the image is brighter.
> >> +	 */
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
> >> +		       MALI_C55_TPG_BACKGROUND_MAX);
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
> >> +		       MALI_C55_TPG_BACKGROUND_MAX);
> >> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
> >> +		       MALI_C55_TPG_BACKGROUND_MAX);
> >> +
> >> +	tpg->mali_c55 = mali_c55;
> >> +
> >> +	return 0;
> >> +
> >> +err_free_ctrl_handler:
> >> +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> >> +err_cleanup_subdev:
> >> +	v4l2_subdev_cleanup(sd);
> >> +err_cleanup_media_entity:
> >> +	media_entity_cleanup(&sd->entity);
> >> +err_destroy_mutex:
> >> +	mutex_destroy(&tpg->lock);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
> >> +{
> >> +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
> >> +
> >> +	if (!tpg->mali_c55)
> >> +		return;
> >> +
> >> +	v4l2_device_unregister_subdev(&tpg->sd);
> >> +	v4l2_subdev_cleanup(&tpg->sd);
> >> +	media_entity_cleanup(&tpg->sd.entity);
> >> +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> >
> > Free the control handler just after v4l2_device_unregister_subdev() to
> > match the order in mali_c55_register_tpg().
> >
> >> +	mutex_destroy(&tpg->lock);
> >> +}

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Dan Scally]

Hi Laurent

On 18/06/2024 00:04, Laurent Pinchart wrote:
> Hi Dan,
>
> On Mon, Jun 17, 2024 at 12:41:11PM +0100, Daniel Scally wrote:
>> Hi Laurent - sorry, should have included everything in the last reply rather than responding
>> piecemeal. Some more responses and questions below
> No worries.
>
>> On 30/05/2024 01:15, Laurent Pinchart wrote:
>>> Hi Dan,
>>>
>>> Thank you for the patch.
>>>
>>> On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
>>>> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
>>>> V4L2 and Media Controller compliant and creates subdevices to manage
>>>> the ISP itself, its internal test pattern generator as well as the
>>>> crop, scaler and output format functionality for each of its two
>>>> output devices.
>>>>
>>>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
>>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>>>> ---
>>>> Changes in v5:
>>>>
>>>> 	- Reworked input formats - previously we allowed representing input data
>>>> 	  as any 8-16 bit format. Now we only allow input data to be represented
>>>> 	  by the new 20-bit bayer formats, which is corrected to the equivalent
>>>> 	  16-bit format in RAW bypass mode.
>>>> 	- Stopped bypassing blocks that we haven't added supporting parameters
>>>> 	  for yet.
>>>> 	- Addressed most of Sakari's comments from the list
>>>>
>>>> Changes not yet made in v5:
>>>>
>>>> 	- The output pipelines can still be started and stopped independently of
>>>> 	  one another - I'd like to discuss that more.
>>>> 	- the TPG subdev still uses .s_stream() - I need to rebase onto a tree
>>>> 	  with working .enable_streams() for a single-source-pad subdevice.
>>>>
>>>> Changes in v4:
>>>>
>>>> 	- Reworked mali_c55_update_bits() to internally perform the bit-shift
>>> I really don't like that, it makes the code very confusing, even more so
>>> as it differs from regmap_update_bits().
>>>
>>> Look at this for instance:
>>>
>>> 	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>> 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
>>> 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
>>>
>>> It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
>>> BIT(0).
>>>
>>> Sorry, I know it will be painful, but this change needs to be reverted.
>>>
>>>> 	- Reworked the resizer to allow cropping during streaming
>>>> 	- Fixed a bug in NV12 output
>>>>
>>>> Changes in v3:
>>>>
>>>> 	- Mostly minor fixes suggested by Sakari
>>>> 	- Fixed the sequencing of vb2 buffers to be synchronised across the two
>>>> 	  capture devices.
>>>>
>>>> Changes in v2:
>>>>
>>>> 	- Clock handling
>>>> 	- Fixed the warnings raised by the kernel test robot
>>>>
>>>>    drivers/media/platform/Kconfig                |   1 +
>>>>    drivers/media/platform/Makefile               |   1 +
>>>>    drivers/media/platform/arm/Kconfig            |   5 +
>>>>    drivers/media/platform/arm/Makefile           |   2 +
>>>>    drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
>>>>    drivers/media/platform/arm/mali-c55/Makefile  |   9 +
>>>>    .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
>>>>    .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
>>>>    .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
>>>>    .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
>>>>    .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
>>>>    .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
>>>>    .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
>>>>    .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
>>>>    14 files changed, 4452 insertions(+)
>>>>    create mode 100644 drivers/media/platform/arm/Kconfig
>>>>    create mode 100644 drivers/media/platform/arm/Makefile
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>> I've skipped review of capture.c and resizer.c as I already have plenty
>>> of comments for the other files, and it's getting late. I'll try to
>>> review the rest tomorrow.
>>>
>>>> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
>>>> index 2d79bfc68c15..c929169766aa 100644
>>>> --- a/drivers/media/platform/Kconfig
>>>> +++ b/drivers/media/platform/Kconfig
>>>> @@ -65,6 +65,7 @@ config VIDEO_MUX
>>>>    source "drivers/media/platform/allegro-dvt/Kconfig"
>>>>    source "drivers/media/platform/amlogic/Kconfig"
>>>>    source "drivers/media/platform/amphion/Kconfig"
>>>> +source "drivers/media/platform/arm/Kconfig"
>>>>    source "drivers/media/platform/aspeed/Kconfig"
>>>>    source "drivers/media/platform/atmel/Kconfig"
>>>>    source "drivers/media/platform/broadcom/Kconfig"
>>>> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
>>>> index da17301f7439..9a647abd5218 100644
>>>> --- a/drivers/media/platform/Makefile
>>>> +++ b/drivers/media/platform/Makefile
>>>> @@ -8,6 +8,7 @@
>>>>    obj-y += allegro-dvt/
>>>>    obj-y += amlogic/
>>>>    obj-y += amphion/
>>>> +obj-y += arm/
>>>>    obj-y += aspeed/
>>>>    obj-y += atmel/
>>>>    obj-y += broadcom/
>>>> diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
>>>> new file mode 100644
>>>> index 000000000000..4f0764c329c7
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/Kconfig
>>>> @@ -0,0 +1,5 @@
>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>> +
>>>> +comment "ARM media platform drivers"
>>>> +
>>>> +source "drivers/media/platform/arm/mali-c55/Kconfig"
>>>> diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
>>>> new file mode 100644
>>>> index 000000000000..8cc4918725ef
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/Makefile
>>>> @@ -0,0 +1,2 @@
>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>> +obj-y += mali-c55/
>>>> diff --git a/drivers/media/platform/arm/mali-c55/Kconfig b/drivers/media/platform/arm/mali-c55/Kconfig
>>>> new file mode 100644
>>>> index 000000000000..602085e28b01
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/Kconfig
>>>> @@ -0,0 +1,18 @@
>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>> +config VIDEO_MALI_C55
>>>> +	tristate "ARM Mali-C55 Image Signal Processor driver"
>>>> +	depends on V4L_PLATFORM_DRIVERS
>>>> +	depends on VIDEO_DEV && OF
>>>> +	depends on ARCH_VEXPRESS || COMPILE_TEST
>>>> +	select MEDIA_CONTROLLER
>>>> +	select VIDEO_V4L2_SUBDEV_API
>>>> +	select VIDEOBUF2_DMA_CONTIG
>>>> +	select VIDEOBUF2_VMALLOC
>>>> +	select V4L2_FWNODE
>>>> +	select GENERIC_PHY_MIPI_DPHY
>>> Alphabetical order ?
>>>
>>>> +	default n
>>> That's the default, you don't have to specify ti.
>>>
>>>> +	help
>>>> +	  Enable this to support Arm's Mali-C55 Image Signal Processor.
>>>> +
>>>> +	  To compile this driver as a module, choose M here: the module
>>>> +	  will be called mali-c55.
>>>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
>>>> new file mode 100644
>>>> index 000000000000..77dcb2fbf0f4
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
>>>> @@ -0,0 +1,9 @@
>>>> +# SPDX-License-Identifier: GPL-2.0
>>>> +
>>>> +mali-c55-y := mali-c55-capture.o \
>>>> +	      mali-c55-core.o \
>>>> +	      mali-c55-isp.o \
>>>> +	      mali-c55-tpg.o \
>>>> +	      mali-c55-resizer.o
>>> Alphabetical order here too.
>>>
>>>> +
>>>> +obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>> new file mode 100644
>>>> index 000000000000..1d539ac9c498
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>> @@ -0,0 +1,951 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Video capture devices
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#include <linux/cleanup.h>
>>>> +#include <linux/minmax.h>
>>>> +#include <linux/pm_runtime.h>
>>>> +#include <linux/string.h>
>>>> +#include <linux/videodev2.h>
>>>> +
>>>> +#include <media/v4l2-dev.h>
>>>> +#include <media/v4l2-event.h>
>>>> +#include <media/v4l2-ioctl.h>
>>>> +#include <media/v4l2-subdev.h>
>>>> +#include <media/videobuf2-core.h>
>>>> +#include <media/videobuf2-dma-contig.h>
>>>> +
>>>> +#include "mali-c55-common.h"
>>>> +#include "mali-c55-registers.h"
>>>> +
>>>> +static const struct mali_c55_fmt mali_c55_fmts[] = {
>>>> +	/*
>>>> +	 * This table is missing some entries which need further work or
>>>> +	 * investigation:
>>>> +	 *
>>>> +	 * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
>>>> +	 * Base mode 5 is "Generic Data"
>>>> +	 * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
>>>> +	 * Base mode 9 seems to have no V4L2 equivalent
>>>> +	 * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
>>>> +	 * equivalent
>>>> +	 */
>>>> +	{
>>>> +		.fourcc = V4L2_PIX_FMT_ARGB2101010,
>>>> +		.mbus_codes = {
>>>> +			MEDIA_BUS_FMT_RGB121212_1X36,
>>>> +			MEDIA_BUS_FMT_RGB202020_1X60,
>>>> +		},
>>>> +		.is_raw = false,
>>>> +		.registers = {
>>>> +			.base_mode = MALI_C55_OUTPUT_A2R10G10B10,
>>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +		}
>>>> +	},
>>>> +	{
>>>> +		.fourcc = V4L2_PIX_FMT_RGB565,
>>>> +		.mbus_codes = {
>>>> +			MEDIA_BUS_FMT_RGB121212_1X36,
>>>> +			MEDIA_BUS_FMT_RGB202020_1X60,
>>>> +		},
>>>> +		.is_raw = false,
>>>> +		.registers = {
>>>> +			.base_mode = MALI_C55_OUTPUT_RGB565,
>>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +		}
>>>> +	},
>>>> +	{
>>>> +		.fourcc = V4L2_PIX_FMT_BGR24,
>>>> +		.mbus_codes = {
>>>> +			MEDIA_BUS_FMT_RGB121212_1X36,
>>>> +			MEDIA_BUS_FMT_RGB202020_1X60,
>>>> +		},
>>>> +		.is_raw = false,
>>>> +		.registers = {
>>>> +			.base_mode = MALI_C55_OUTPUT_RGB24,
>>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +		}
>>>> +	},
>>>> +	{
>>>> +		.fourcc = V4L2_PIX_FMT_YUYV,
>>>> +		.mbus_codes = {
>>>> +			MEDIA_BUS_FMT_YUV10_1X30,
>>>> +		},
>>>> +		.is_raw = false,
>>>> +		.registers = {
>>>> +			.base_mode = MALI_C55_OUTPUT_YUY2,
>>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +		}
>>>> +	},
>>>> +	{
>>>> +		.fourcc = V4L2_PIX_FMT_UYVY,
>>>> +		.mbus_codes = {
>>>> +			MEDIA_BUS_FMT_YUV10_1X30,
>>>> +		},
>>>> +		.is_raw = false,
>>>> +		.registers = {
>>>> +			.base_mode = MALI_C55_OUTPUT_UYVY,
>>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +		}
>>>> +	},
>>>> +	{
>>>> +		.fourcc = V4L2_PIX_FMT_Y210,
>>>> +		.mbus_codes = {
>>>> +			MEDIA_BUS_FMT_YUV10_1X30,
>>>> +		},
>>>> +		.is_raw = false,
>>>> +		.registers = {
>>>> +			.base_mode = MALI_C55_OUTPUT_Y210,
>>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +		}
>>>> +	},
>>>> +	/*
>>>> +	 * This is something of a hack, the ISP thinks it's running NV12M but
>>>> +	 * by setting uv_plane = 0 we simply discard that planes and only output
>>>> +	 * the Y-plane.
>>>> +	 */
>>>> +	{
>>>> +		.fourcc = V4L2_PIX_FMT_GREY,
>>>> +		.mbus_codes = {
>>>> +			MEDIA_BUS_FMT_YUV10_1X30,
>>>> +		},
>>>> +		.is_raw = false,
>>>> +		.registers = {
>>>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
>>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +		}
>>>> +	},
>>>> +	{
>>>> +		.fourcc = V4L2_PIX_FMT_NV12M,
>>>> +		.mbus_codes = {
>>>> +			MEDIA_BUS_FMT_YUV10_1X30,
>>>> +		},
>>>> +		.is_raw = false,
>>>> +		.registers = {
>>>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
>>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
>>>> +		}
>>>> +	},
>>>> +	{
>>>> +		.fourcc = V4L2_PIX_FMT_NV21M,
>>>> +		.mbus_codes = {
>>>> +			MEDIA_BUS_FMT_YUV10_1X30,
>>>> +		},
>>>> +		.is_raw = false,
>>>> +		.registers = {
>>>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
>>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
>>>> +		}
>>>> +	},
>>>> +	/*
>>>> +	 * RAW uncompressed formats are all packed in 16 bpp.
>>>> +	 * TODO: Expand this list to encompass all possible RAW formats.
>>>> +	 */
>>>> +	{
>>>> +		.fourcc = V4L2_PIX_FMT_SRGGB16,
>>>> +		.mbus_codes = {
>>>> +			MEDIA_BUS_FMT_SRGGB16_1X16,
>>>> +		},
>>>> +		.is_raw = true,
>>>> +		.registers = {
>>>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +		}
>>>> +	},
>>>> +	{
>>>> +		.fourcc = V4L2_PIX_FMT_SBGGR16,
>>>> +		.mbus_codes = {
>>>> +			MEDIA_BUS_FMT_SBGGR16_1X16,
>>>> +		},
>>>> +		.is_raw = true,
>>>> +		.registers = {
>>>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +		}
>>>> +	},
>>>> +	{
>>>> +		.fourcc = V4L2_PIX_FMT_SGBRG16,
>>>> +		.mbus_codes = {
>>>> +			MEDIA_BUS_FMT_SGBRG16_1X16,
>>>> +		},
>>>> +		.is_raw = true,
>>>> +		.registers = {
>>>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +		}
>>>> +	},
>>>> +	{
>>>> +		.fourcc = V4L2_PIX_FMT_SGRBG16,
>>>> +		.mbus_codes = {
>>>> +			MEDIA_BUS_FMT_SGRBG16_1X16,
>>>> +		},
>>>> +		.is_raw = true,
>>>> +		.registers = {
>>>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +		}
>>>> +	},
>>>> +};
>>>> +
>>>> +static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
>>>> +					       u32 code)
>>>> +{
>>>> +	unsigned int i;
>>>> +
>>>> +	for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
>>>> +		if (fmt->mbus_codes[i] == code)
>>>> +			return true;
>>>> +	}
>>>> +
>>>> +	return false;
>>>> +}
>>>> +
>>>> +bool mali_c55_format_is_raw(unsigned int mbus_code)
>>>> +{
>>>> +	unsigned int i;
>>>> +
>>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>>> +		if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
>>>> +			return mali_c55_fmts[i].is_raw;
>>>> +	}
>>>> +
>>>> +	return false;
>>>> +}
>>>> +
>>>> +static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
>>>> +{
>>>> +	unsigned int i;
>>>> +
>>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>>> +		if (mali_c55_fmts[i].fourcc == pixelformat)
>>>> +			return &mali_c55_fmts[i];
>>>> +	}
>>>> +
>>>> +	/*
>>>> +	 * If we find no matching pixelformat, we'll just default to the first
>>>> +	 * one for now.
>>>> +	 */
>>>> +
>>>> +	return &mali_c55_fmts[0];
>>>> +}
>>>> +
>>>> +static const char * const capture_device_names[] = {
>>>> +	"mali-c55 fr",
>>>> +	"mali-c55 ds",
>>>> +	"mali-c55 3a stats",
>>>> +	"mali-c55 params",
>>>> +};
>>>> +
>>>> +static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
>>>> +{
>>>> +	if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
>>>> +		return capture_device_names[0];
>>>> +
>>>> +	if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
>>>> +		return capture_device_names[1];
>>>> +
>>>> +	return "params/stat not supported yet";
>>>> +}
>>>> +
>>>> +static int mali_c55_link_validate(struct media_link *link)
>>>> +{
>>>> +	struct video_device *vdev =
>>>> +		media_entity_to_video_device(link->sink->entity);
>>>> +	struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
>>>> +	struct v4l2_subdev *sd =
>>>> +		media_entity_to_v4l2_subdev(link->source->entity);
>>>> +	const struct v4l2_pix_format_mplane *pix_mp;
>>>> +	const struct mali_c55_fmt *cap_fmt;
>>>> +	struct v4l2_subdev_format sd_fmt = {
>>>> +		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
>>>> +		.pad = link->source->index,
>>>> +	};
>>>> +	int ret;
>>>> +
>>>> +	ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	pix_mp = &cap_dev->mode.pix_mp;
>>>> +	cap_fmt = cap_dev->mode.capture_fmt;
>>>> +
>>>> +	if (sd_fmt.format.width != pix_mp->width ||
>>>> +	    sd_fmt.format.height != pix_mp->height) {
>>>> +		dev_dbg(cap_dev->mali_c55->dev,
>>>> +			"link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
>>>> +			link->source->entity->name, link->source->index,
>>>> +			link->sink->entity->name, link->sink->index,
>>>> +			sd_fmt.format.width, sd_fmt.format.height,
>>>> +			pix_mp->width, pix_mp->height);
>>>> +		return -EPIPE;
>>>> +	}
>>>> +
>>>> +	if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
>>>> +		dev_dbg(cap_dev->mali_c55->dev,
>>>> +			"link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format %p4cc\n",
>>>> +			link->source->entity->name, link->source->index,
>>>> +			link->sink->entity->name, link->sink->index,
>>>> +			sd_fmt.format.code, &pix_mp->pixelformat);
>>>> +		return -EPIPE;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static const struct media_entity_operations mali_c55_media_ops = {
>>>> +	.link_validate = mali_c55_link_validate,
>>>> +};
>>>> +
>>>> +static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
>>>> +				    unsigned int *num_planes, unsigned int sizes[],
>>>> +				    struct device *alloc_devs[])
>>>> +{
>>>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>>> +	unsigned int i;
>>>> +
>>>> +	if (*num_planes) {
>>>> +		if (*num_planes != cap_dev->mode.pix_mp.num_planes)
>>>> +			return -EINVAL;
>>>> +
>>>> +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>>> +			if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
>>>> +				return -EINVAL;
>>>> +	} else {
>>>> +		*num_planes = cap_dev->mode.pix_mp.num_planes;
>>>> +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>>> +			sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static void mali_c55_buf_queue(struct vb2_buffer *vb)
>>>> +{
>>>> +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
>>>> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>>>> +	struct mali_c55_buffer *buf = container_of(vbuf,
>>>> +						   struct mali_c55_buffer, vb);
>>>> +	unsigned int i;
>>>> +
>>>> +	buf->plane_done[MALI_C55_PLANE_Y] = false;
>>>> +
>>>> +	/*
>>>> +	 * If we're in a single-plane format we flag the other plane as done
>>>> +	 * already so it's dequeued appropriately later
>>>> +	 */
>>>> +	buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
>>>> +
>>>> +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
>>>> +		unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
>>>> +
>>>> +		vb2_set_plane_payload(vb, i, size);
>>>> +	}
>>>> +
>>>> +	spin_lock(&cap_dev->buffers.lock);
>>>> +	list_add_tail(&buf->queue, &cap_dev->buffers.queue);
>>>> +	spin_unlock(&cap_dev->buffers.lock);
>>>> +}
>>>> +
>>>> +static int mali_c55_buf_init(struct vb2_buffer *vb)
>>>> +{
>>>> +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
>>>> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>>>> +	struct mali_c55_buffer *buf = container_of(vbuf,
>>>> +						   struct mali_c55_buffer, vb);
>>>> +	unsigned int i;
>>>> +
>>>> +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>>> +		buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>> +
>>>> +	guard(spinlock)(&cap_dev->buffers.lock);
>>>> +
>>>> +	cap_dev->buffers.curr = cap_dev->buffers.next;
>>>> +	cap_dev->buffers.next = NULL;
>>>> +
>>>> +	if (!list_empty(&cap_dev->buffers.queue)) {
>>>> +		struct v4l2_pix_format_mplane *pix_mp;
>>>> +		const struct v4l2_format_info *info;
>>>> +		u32 *addrs;
>>>> +
>>>> +		pix_mp = &cap_dev->mode.pix_mp;
>>>> +		info = v4l2_format_info(pix_mp->pixelformat);
>>>> +
>>>> +		mali_c55_update_bits(mali_c55,
>>>> +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
>>>> +		if (cap_dev->mode.capture_fmt->registers.uv_plane)
>>>> +			mali_c55_update_bits(mali_c55,
>>>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
>>>> +
>>>> +		cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
>>>> +							 struct mali_c55_buffer,
>>>> +							 queue);
>>>> +		list_del(&cap_dev->buffers.next->queue);
>>>> +
>>>> +		addrs = cap_dev->buffers.next->addrs;
>>>> +		mali_c55_write(mali_c55,
>>>> +			MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
>>>> +			addrs[MALI_C55_PLANE_Y]);
>>>> +		mali_c55_write(mali_c55,
>>>> +			MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
>>>> +			addrs[MALI_C55_PLANE_UV]);
>>>> +		mali_c55_write(mali_c55,
>>>> +			MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
>>>> +			pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
>>>> +		mali_c55_write(mali_c55,
>>>> +			MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
>>>> +			pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
>>>> +			/ info->hdiv);
>>>> +	} else {
>>>> +		/*
>>>> +		 * If we underflow then we can tell the ISP that we don't want
>>>> +		 * to write out the next frame.
>>>> +		 */
>>>> +		mali_c55_update_bits(mali_c55,
>>>> +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>> +		mali_c55_update_bits(mali_c55,
>>>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>> +	}
>>>> +}
>>>> +
>>>> +static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
>>>> +				   unsigned int framecount)
>>>> +{
>>>> +	curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>>>> +	curr_buf->vb.field = V4L2_FIELD_NONE;
>>>> +	curr_buf->vb.sequence = framecount;
>>>> +	vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
>>>> +}
>>>> +
>>>> +/**
>>>> + * mali_c55_set_plane_done - mark the plane as written and process the buffer if
>>>> + *			     both planes are finished.
>>>> + * @cap_dev:  pointer to the fr or ds pipe output
>>>> + * @plane:    the plane to mark as completed
>>>> + *
>>>> + * The Mali C55 ISP has muliplanar outputs for some formats that come with two
>>>> + * separate "buffer write completed" interrupts - we need to flag each plane's
>>>> + * completion and check whether both planes are done - if so, complete the buf
>>>> + * in vb2.
>>>> + */
>>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>>>> +			     enum mali_c55_planes plane)
>>>> +{
>>>> +	struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
>>>> +	struct mali_c55_buffer *curr_buf;
>>>> +
>>>> +	guard(spinlock)(&cap_dev->buffers.lock);
>>>> +	curr_buf = cap_dev->buffers.curr;
>>>> +
>>>> +	/*
>>>> +	 * This _should_ never happen. If no buffer was available from vb2 then
>>>> +	 * we tell the ISP not to bother writing the next frame, which means the
>>>> +	 * interrupts that call this function should never trigger. If it does
>>>> +	 * happen then one of our assumptions is horribly wrong - complain
>>>> +	 * loudly and do nothing.
>>>> +	 */
>>>> +	if (!curr_buf) {
>>>> +		dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
>>>> +			mali_c55_cap_dev_to_name(cap_dev), __func__);
>>>> +		return;
>>>> +	}
>>>> +
>>>> +	/* If the other plane is also done... */
>>>> +	if (curr_buf->plane_done[~plane & 1]) {
>>>> +		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
>>>> +		cap_dev->buffers.curr = NULL;
>>>> +		isp->frame_sequence++;
>>>> +	} else {
>>>> +		curr_buf->plane_done[plane] = true;
>>>> +	}
>>>> +}
>>>> +
>>>> +static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>> +
>>>> +	mali_c55_update_bits(mali_c55,
>>>> +			     MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>> +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>> +	mali_c55_update_bits(mali_c55,
>>>> +			     MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>> +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>> +}
>>>> +
>>>> +static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>> +
>>>> +	/*
>>>> +	 * The Mali ISP can hold up to 5 buffer addresses and simply cycle
>>>> +	 * through them, but it's not clear to me that the vb2 queue _guarantees_
>>>> +	 * it will queue buffers to the driver in a fixed order, and ensuring
>>>> +	 * we call vb2_buffer_done() for the right buffer seems to me to add
>>>> +	 * pointless complexity given in multi-context mode we'd need to
>>>> +	 * re-write those registers every frame anyway...so we tell the ISP to
>>>> +	 * use a single register and update it for each frame.
>>>> +	 */
>>>> +	mali_c55_update_bits(mali_c55,
>>>> +			MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
>>>> +			MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
>>>> +	mali_c55_update_bits(mali_c55,
>>>> +			MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
>>>> +			MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
>>>> +
>>>> +	/*
>>>> +	 * We only queue a buffer in the streamon path if this is the first of
>>>> +	 * the capture devices to start streaming. If the ISP is already running
>>>> +	 * then we rely on the ISP_START interrupt to queue the first buffer for
>>>> +	 * this capture device.
>>>> +	 */
>>>> +	if (mali_c55->pipe.start_count == 1)
>>>> +		mali_c55_set_next_buffer(cap_dev);
>>>> +}
>>>> +
>>>> +static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
>>>> +					    enum vb2_buffer_state state)
>>>> +{
>>>> +	struct mali_c55_buffer *buf, *tmp;
>>>> +
>>>> +	guard(spinlock)(&cap_dev->buffers.lock);
>>>> +
>>>> +	if (cap_dev->buffers.curr) {
>>>> +		vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
>>>> +				state);
>>>> +		cap_dev->buffers.curr = NULL;
>>>> +	}
>>>> +
>>>> +	if (cap_dev->buffers.next) {
>>>> +		vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
>>>> +				state);
>>>> +		cap_dev->buffers.next = NULL;
>>>> +	}
>>>> +
>>>> +	list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
>>>> +		list_del(&buf->queue);
>>>> +		vb2_buffer_done(&buf->vb.vb2_buf, state);
>>>> +	}
>>>> +}
>>>> +
>>>> +static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
>>>> +{
>>>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>> +	struct mali_c55_resizer *rzr = cap_dev->rzr;
>>>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>>>> +	int ret;
>>>> +
>>>> +	guard(mutex)(&isp->lock);
>>>> +
>>>> +	ret = pm_runtime_resume_and_get(mali_c55->dev);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	ret = video_device_pipeline_start(&cap_dev->vdev,
>>>> +					  &cap_dev->mali_c55->pipe);
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
>>>> +			mali_c55_cap_dev_to_name(cap_dev));
>>>> +		goto err_pm_put;
>>>> +	}
>>>> +
>>>> +	mali_c55_cap_dev_stream_enable(cap_dev);
>>>> +	mali_c55_rzr_start_stream(rzr);
>>>> +
>>>> +	/*
>>>> +	 * We only start the ISP if we're the only capture device that's
>>>> +	 * streaming. Otherwise, it'll already be active.
>>>> +	 */
>>>> +	if (mali_c55->pipe.start_count == 1) {
>>>> +		ret = mali_c55_isp_start_stream(isp);
>>>> +		if (ret)
>>>> +			goto err_disable_cap_dev;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +
>>>> +err_disable_cap_dev:
>>>> +	mali_c55_cap_dev_stream_disable(cap_dev);
>>>> +	video_device_pipeline_stop(&cap_dev->vdev);
>>>> +err_pm_put:
>>>> +	pm_runtime_put(mali_c55->dev);
>>>> +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
>>>> +{
>>>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>> +	struct mali_c55_resizer *rzr = cap_dev->rzr;
>>>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>>>> +
>>>> +	guard(mutex)(&isp->lock);
>>>> +
>>>> +	/*
>>>> +	 * If one of the other capture nodes is streaming, we shouldn't
>>>> +	 * disable the ISP here.
>>>> +	 */
>>>> +	if (mali_c55->pipe.start_count == 1)
>>>> +		mali_c55_isp_stop_stream(&mali_c55->isp);
>>>> +
>>>> +	mali_c55_rzr_stop_stream(rzr);
>>>> +	mali_c55_cap_dev_stream_disable(cap_dev);
>>>> +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
>>>> +	video_device_pipeline_stop(&cap_dev->vdev);
>>>> +	pm_runtime_put(mali_c55->dev);
>>>> +}
>>>> +
>>>> +static const struct vb2_ops mali_c55_vb2_ops = {
>>>> +	.queue_setup		= &mali_c55_vb2_queue_setup,
>>>> +	.buf_queue		= &mali_c55_buf_queue,
>>>> +	.buf_init		= &mali_c55_buf_init,
>>>> +	.wait_prepare		= vb2_ops_wait_prepare,
>>>> +	.wait_finish		= vb2_ops_wait_finish,
>>>> +	.start_streaming	= &mali_c55_vb2_start_streaming,
>>>> +	.stop_streaming		= &mali_c55_vb2_stop_streaming,
>>>> +};
>>>> +
>>>> +static const struct v4l2_file_operations mali_c55_v4l2_fops = {
>>>> +	.owner = THIS_MODULE,
>>>> +	.unlocked_ioctl = video_ioctl2,
>>>> +	.open = v4l2_fh_open,
>>>> +	.release = vb2_fop_release,
>>>> +	.poll = vb2_fop_poll,
>>>> +	.mmap = vb2_fop_mmap,
>>>> +};
>>>> +
>>>> +static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
>>>> +{
>>>> +	const struct mali_c55_fmt *capture_format;
>>>> +	const struct v4l2_format_info *info;
>>>> +	struct v4l2_plane_pix_format *plane;
>>>> +	unsigned int i;
>>>> +
>>>> +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
>>>> +	pix_mp->pixelformat = capture_format->fourcc;
>>>> +
>>>> +	pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
>>>> +			      MALI_C55_MAX_WIDTH);
>>>> +	pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
>>>> +			       MALI_C55_MAX_HEIGHT);
>>>> +
>>>> +	pix_mp->field = V4L2_FIELD_NONE;
>>>> +	pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>> +	pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>> +	pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
>>>> +
>>>> +	info = v4l2_format_info(pix_mp->pixelformat);
>>>> +	pix_mp->num_planes = info->mem_planes;
>>>> +	memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
>>>> +
>>>> +	pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;
>>>> +	pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
>>>> +				       * pix_mp->height;
>>>> +
>>>> +	for (i = 1; i < info->comp_planes; i++) {
>>>> +		plane = &pix_mp->plane_fmt[i];
>>>> +
>>>> +		plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
>>>> +						   info->hdiv);
>>>> +		plane->sizeimage = DIV_ROUND_UP(
>>>> +					plane->bytesperline * pix_mp->height,
>>>> +					info->vdiv);
>>>> +	}
>>>> +
>>>> +	if (info->mem_planes == 1) {
>>>> +		for (i = 1; i < info->comp_planes; i++) {
>>>> +			plane = &pix_mp->plane_fmt[i];
>>>> +			pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
>>>> +		}
>>>> +	}
>>>> +}
>>>> +
>>>> +static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>> +					   struct v4l2_format *f)
>>>> +{
>>>> +	mali_c55_try_fmt(&f->fmt.pix_mp);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
>>>> +				struct v4l2_pix_format_mplane *pix_mp)
>>>> +{
>>>> +	const struct mali_c55_fmt *capture_format;
>>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>> +	const struct v4l2_format_info *info;
>>>> +
>>>> +	mali_c55_try_fmt(pix_mp);
>>>> +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
>>>> +	info = v4l2_format_info(pix_mp->pixelformat);
>>>> +	if (WARN_ON(!info))
>>>> +		return;
>>>> +
>>>> +	mali_c55_write(mali_c55,
>>>> +		       MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>> +		       capture_format->registers.base_mode);
>>>> +	mali_c55_write(mali_c55,
>>>> +		       MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
>>>> +		       MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
>>>> +		       MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
>>>> +
>>>> +	if (info->mem_planes > 1) {
>>>> +		mali_c55_write(mali_c55,
>>>> +			       MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>> +			       capture_format->registers.base_mode);
>>>> +		mali_c55_update_bits(mali_c55,
>>>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>> +				MALI_C55_WRITER_SUBMODE_MASK,
>>>> +				capture_format->registers.uv_plane);
>>>> +
>>>> +		mali_c55_write(mali_c55,
>>>> +			MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
>>>> +			MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
>>>> +			MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
>>>> +	}
>>>> +
>>>> +	if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
>>>> +		/*
>>>> +		 * TODO: Figure out the colour matrix coefficients and calculate
>>>> +		 * and write them here.
>>>> +		 */
>>>> +
>>>> +		mali_c55_write(mali_c55,
>>>> +			       MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>> +			       MALI_C55_CS_CONV_MATRIX_MASK);
>>>> +
>>>> +		if (info->hdiv > 1)
>>>> +			mali_c55_update_bits(mali_c55,
>>>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>> +				MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
>>>> +		if (info->vdiv > 1)
>>>> +			mali_c55_update_bits(mali_c55,
>>>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>> +				MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
>>>> +		if (info->hdiv > 1 || info->vdiv > 1)
>>>> +			mali_c55_update_bits(mali_c55,
>>>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>> +				MALI_C55_CS_CONV_FILTER_MASK, 0x01);
>>>> +	}
>>>> +
>>>> +	cap_dev->mode.pix_mp = *pix_mp;
>>>> +	cap_dev->mode.capture_fmt = capture_format;
>>>> +}
>>>> +
>>>> +static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>> +					 struct v4l2_format *f)
>>>> +{
>>>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>>> +
>>>> +	if (vb2_is_busy(&cap_dev->queue))
>>>> +		return -EBUSY;
>>>> +
>>>> +	mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>> +					 struct v4l2_format *f)
>>>> +{
>>>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>>> +
>>>> +	f->fmt.pix_mp = cap_dev->mode.pix_mp;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>> +					    struct v4l2_fmtdesc *f)
>>>> +{
>>>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>>> +	unsigned int j = 0;
>>>> +	unsigned int i;
>>>> +
>>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>>> +		if (f->mbus_code &&
>>>> +		    !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
>>>> +						       f->mbus_code))
>>>> +			continue;
>>>> +
>>>> +		/* Downscale pipe can't output RAW formats */
>>>> +		if (mali_c55_fmts[i].is_raw &&
>>>> +		    cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
>>>> +			continue;
>>>> +
>>>> +		if (j++ == f->index) {
>>>> +			f->pixelformat = mali_c55_fmts[i].fourcc;
>>>> +			return 0;
>>>> +		}
>>>> +	}
>>>> +
>>>> +	return -EINVAL;
>>>> +}
>>>> +
>>>> +static int mali_c55_querycap(struct file *file, void *fh,
>>>> +			     struct v4l2_capability *cap)
>>>> +{
>>>> +	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
>>>> +	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
>>>> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
>>>> +	.vidioc_querybuf = vb2_ioctl_querybuf,
>>>> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
>>>> +	.vidioc_qbuf = vb2_ioctl_qbuf,
>>>> +	.vidioc_expbuf = vb2_ioctl_expbuf,
>>>> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
>>>> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
>>>> +	.vidioc_streamon = vb2_ioctl_streamon,
>>>> +	.vidioc_streamoff = vb2_ioctl_streamoff,
>>>> +	.vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
>>>> +	.vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
>>>> +	.vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
>>>> +	.vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
>>>> +	.vidioc_querycap = mali_c55_querycap,
>>>> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
>>>> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
>>>> +};
>>>> +
>>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	struct v4l2_pix_format_mplane pix_mp;
>>>> +	struct mali_c55_cap_dev *cap_dev;
>>>> +	struct video_device *vdev;
>>>> +	struct vb2_queue *vb2q;
>>>> +	unsigned int i;
>>>> +	int ret;
>>>> +
>>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
>>>> +		cap_dev = &mali_c55->cap_devs[i];
>>>> +		vdev = &cap_dev->vdev;
>>>> +		vb2q = &cap_dev->queue;
>>>> +
>>>> +		/*
>>>> +		 * The downscale output pipe is an optional block within the ISP
>>>> +		 * so we need to check whether it's actually been fitted or not.
>>>> +		 */
>>>> +
>>>> +		if (i == MALI_C55_CAP_DEV_DS &&
>>>> +		    !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
>>>> +			continue;
>>>> +
>>>> +		cap_dev->mali_c55 = mali_c55;
>>>> +		mutex_init(&cap_dev->lock);
>>>> +		INIT_LIST_HEAD(&cap_dev->buffers.queue);
>>>> +
>>>> +		switch (i) {
>>>> +		case MALI_C55_CAP_DEV_FR:
>>>> +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
>>>> +			cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
>>>> +			break;
>>>> +		case MALI_C55_CAP_DEV_DS:
>>>> +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
>>>> +			cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
>>>> +			break;
>>>> +		default:
>>>> +			mutex_destroy(&cap_dev->lock);
>>>> +			ret = -EINVAL;
>>>> +			goto err_destroy_mutex;
>>>> +		}
>>>> +
>>>> +		cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
>>>> +		ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
>>>> +		if (ret) {
>>>> +			mutex_destroy(&cap_dev->lock);
>>>> +			goto err_destroy_mutex;
>>>> +		}
>>>> +
>>>> +		vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>>>> +		vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
>>>> +		vb2q->drv_priv = cap_dev;
>>>> +		vb2q->mem_ops = &vb2_dma_contig_memops;
>>>> +		vb2q->ops = &mali_c55_vb2_ops;
>>>> +		vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
>>>> +		vb2q->min_queued_buffers = 1;
>>>> +		vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
>>>> +		vb2q->lock = &cap_dev->lock;
>>>> +		vb2q->dev = mali_c55->dev;
>>>> +
>>>> +		ret = vb2_queue_init(vb2q);
>>>> +		if (ret) {
>>>> +			dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
>>>> +				mali_c55_cap_dev_to_name(cap_dev));
>>>> +			goto err_cleanup_media_entity;
>>>> +		}
>>>> +
>>>> +		strscpy(cap_dev->vdev.name, capture_device_names[i],
>>>> +			sizeof(cap_dev->vdev.name));
>>>> +		vdev->release = video_device_release_empty;
>>>> +		vdev->fops = &mali_c55_v4l2_fops;
>>>> +		vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
>>>> +		vdev->lock = &cap_dev->lock;
>>>> +		vdev->v4l2_dev = &mali_c55->v4l2_dev;
>>>> +		vdev->queue = &cap_dev->queue;
>>>> +		vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>>>> +				    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
>>>> +		vdev->entity.ops = &mali_c55_media_ops;
>>>> +		video_set_drvdata(vdev, cap_dev);
>>>> +
>>>> +		memset(&pix_mp, 0, sizeof(pix_mp));
>>>> +		pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
>>>> +		pix_mp.width = MALI_C55_DEFAULT_WIDTH;
>>>> +		pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
>>>> +		mali_c55_set_format(cap_dev, &pix_mp);
>>>> +
>>>> +		ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
>>>> +		if (ret) {
>>>> +			dev_err(mali_c55->dev,
>>>> +				"%s failed to register video device\n",
>>>> +				mali_c55_cap_dev_to_name(cap_dev));
>>>> +			goto err_release_vb2q;
>>>> +		}
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +
>>>> +err_release_vb2q:
>>>> +	vb2_queue_release(vb2q);
>>>> +err_cleanup_media_entity:
>>>> +	media_entity_cleanup(&cap_dev->vdev.entity);
>>>> +err_destroy_mutex:
>>>> +	mutex_destroy(&cap_dev->lock);
>>>> +	mali_c55_unregister_capture_devs(mali_c55);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	struct mali_c55_cap_dev *cap_dev;
>>>> +	unsigned int i;
>>>> +
>>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
>>>> +		cap_dev = &mali_c55->cap_devs[i];
>>>> +
>>>> +		if (!video_is_registered(&cap_dev->vdev))
>>>> +			continue;
>>>> +
>>>> +		vb2_video_unregister_device(&cap_dev->vdev);
>>>> +		media_entity_cleanup(&cap_dev->vdev.entity);
>>>> +		mutex_destroy(&cap_dev->lock);
>>>> +	}
>>>> +}
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>> new file mode 100644
>>>> index 000000000000..2d0c4d152beb
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>> @@ -0,0 +1,266 @@
>>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Common definitions
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#ifndef _MALI_C55_COMMON_H
>>>> +#define _MALI_C55_COMMON_H
>>>> +
>>>> +#include <linux/clk.h>
>>>> +#include <linux/io.h>
>>>> +#include <linux/list.h>
>>>> +#include <linux/mutex.h>
>>>> +#include <linux/scatterlist.h>
>>> I don't think this is needed. You're however missing spinlock.h.
>>>
>>>> +#include <linux/videodev2.h>
>>>> +
>>>> +#include <media/media-device.h>
>>>> +#include <media/v4l2-async.h>
>>>> +#include <media/v4l2-ctrls.h>
>>>> +#include <media/v4l2-dev.h>
>>>> +#include <media/v4l2-device.h>
>>>> +#include <media/v4l2-subdev.h>
>>>> +#include <media/videobuf2-core.h>
>>>> +#include <media/videobuf2-v4l2.h>
>>>> +
>>>> +#define MALI_C55_DRIVER_NAME		"mali-c55"
>>>> +
>>>> +/* min and max values for the image sizes */
>>>> +#define MALI_C55_MIN_WIDTH		640U
>>>> +#define MALI_C55_MIN_HEIGHT		480U
>>>> +#define MALI_C55_MAX_WIDTH		8192U
>>>> +#define MALI_C55_MAX_HEIGHT		8192U
>>>> +#define MALI_C55_DEFAULT_WIDTH		1920U
>>>> +#define MALI_C55_DEFAULT_HEIGHT		1080U
>>>> +
>>>> +#define MALI_C55_DEFAULT_MEDIA_BUS_FMT	MEDIA_BUS_FMT_RGB121212_1X36
>>>> +
>>>> +struct mali_c55;
>>>> +struct mali_c55_cap_dev;
>>>> +struct platform_device;
>>> You should also forward-declare
>>>
>>> struct device;
>>> struct dma_chan;
>>> struct resource;
>>>
>>>> +
>>>> +static const char * const mali_c55_clk_names[] = {
>>>> +	"aclk",
>>>> +	"hclk",
>>>> +};
>>> This will end up duplicating the array in each compilation unit, not
>>> great. Move it to mali-c55-core.c. You use it in this file just for its
>>> size, replace that with a macro that defines the size, or allocate
>>> mali_c55.clks dynamically with devm_kcalloc().
>>>
>>>> +
>>>> +enum mali_c55_interrupts {
>>>> +	MALI_C55_IRQ_ISP_START,
>>>> +	MALI_C55_IRQ_ISP_DONE,
>>>> +	MALI_C55_IRQ_MCM_ERROR,
>>>> +	MALI_C55_IRQ_BROKEN_FRAME_ERROR,
>>>> +	MALI_C55_IRQ_MET_AF_DONE,
>>>> +	MALI_C55_IRQ_MET_AEXP_DONE,
>>>> +	MALI_C55_IRQ_MET_AWB_DONE,
>>>> +	MALI_C55_IRQ_AEXP_1024_DONE,
>>>> +	MALI_C55_IRQ_IRIDIX_MET_DONE,
>>>> +	MALI_C55_IRQ_LUT_INIT_DONE,
>>>> +	MALI_C55_IRQ_FR_Y_DONE,
>>>> +	MALI_C55_IRQ_FR_UV_DONE,
>>>> +	MALI_C55_IRQ_DS_Y_DONE,
>>>> +	MALI_C55_IRQ_DS_UV_DONE,
>>>> +	MALI_C55_IRQ_LINEARIZATION_DONE,
>>>> +	MALI_C55_IRQ_RAW_FRONTEND_DONE,
>>>> +	MALI_C55_IRQ_NOISE_REDUCTION_DONE,
>>>> +	MALI_C55_IRQ_IRIDIX_DONE,
>>>> +	MALI_C55_IRQ_BAYER2RGB_DONE,
>>>> +	MALI_C55_IRQ_WATCHDOG_TIMER,
>>>> +	MALI_C55_IRQ_FRAME_COLLISION,
>>>> +	MALI_C55_IRQ_UNUSED,
>>>> +	MALI_C55_IRQ_DMA_ERROR,
>>>> +	MALI_C55_IRQ_INPUT_STOPPED,
>>>> +	MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
>>>> +	MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
>>>> +	MALI_C55_NUM_IRQ_BITS
>>> Those are register bits, I think they belong to mali-c55-registers.h,
>>> and should probably be macros instead of an enum.
>>>
>>>> +};
>>>> +
>>>> +enum mali_c55_isp_pads {
>>>> +	MALI_C55_ISP_PAD_SINK_VIDEO,
>>> As there's a single sink pad, maybe MALI_C55_ISP_PAD_SINK ? Ah, you're
>>> probably preparing for ISP parameters support. It's fine.
>>>
>>>> +	MALI_C55_ISP_PAD_SOURCE,
>>> Then maybe this should be named MALI_C55_ISP_PAD_SOURCE_VIDEO as I
>>> assume there will be a stats source pad.
>>>
>>>> +	MALI_C55_ISP_PAD_SOURCE_BYPASS,
>>>> +	MALI_C55_ISP_NUM_PADS,
>>>> +};
>>>> +
>>>> +struct mali_c55_tpg {
>>>> +	struct mali_c55 *mali_c55;
>>>> +	struct v4l2_subdev sd;
>>>> +	struct media_pad pad;
>>>> +	struct mutex lock;
>>>> +	struct mali_c55_tpg_ctrls {
>>>> +		struct v4l2_ctrl_handler handler;
>>>> +		struct v4l2_ctrl *test_pattern;
>>> Set but never used. You can drop it.
>>>
>>>> +		struct v4l2_ctrl *hblank;
>>> Set and used only once, in the same function. You can make it a local
>>> variable.
>>>
>>>> +		struct v4l2_ctrl *vblank;
>>>> +	} ctrls;
>>>> +};
>>> I wonder if this file should be split, with mali-c55-capture.h,
>>> mali-c55-core.h, mali-c55-isp.h, ... I think it could increase
>>> readability by clearly separating the different elements. Up to you.
>>>
>>>> +
>>>> +struct mali_c55_isp {
>>>> +	struct mali_c55 *mali_c55;
>>>> +	struct v4l2_subdev sd;
>>>> +	struct media_pad pads[MALI_C55_ISP_NUM_PADS];
>>>> +	struct media_pad *remote_src;
>>>> +	struct v4l2_async_notifier notifier;
>>> I'm tempted to move the notifier to mali_c55, as it's related to
>>> components external to the whole ISP, not to the ISP subdev itself.
>>> Could you give it a try, to see if it could be done without any drawback
>>> ?
>> This seems to work fine.
>>
>>>> +	struct mutex lock;
>>> Locks require a comment to explain what they protect. Same below where
>>> applicable (for both mutexes and spinlocks).
>>>
>>>> +	unsigned int frame_sequence;
>>>> +};
>>>> +
>>>> +enum mali_c55_resizer_ids {
>>>> +	MALI_C55_RZR_FR,
>>>> +	MALI_C55_RZR_DS,
>>>> +	MALI_C55_NUM_RZRS,
>>> The usual abbreviation for "resizer" in drivers/media/ is "rsz", not
>>> "rzr". I would have said we can leave it as-is as changing it would be a
>>> bit annoying, but I then realized that "rzr" is not just unusual, it's
>>> actually not used at all. Would you mind applying a sed globally ? :-)
>>>
>>>> +};
>>>> +
>>>> +enum mali_c55_rzr_pads {
>>> Same enums/structs use abbreviations, some don't. Consistency would
>>> help.
>>>
>>>> +	MALI_C55_RZR_SINK_PAD,
>>>> +	MALI_C55_RZR_SOURCE_PAD,
>>>> +	MALI_C55_RZR_SINK_BYPASS_PAD,
>>>> +	MALI_C55_RZR_NUM_PADS
>>>> +};
>>>> +
>>>> +struct mali_c55_resizer {
>>>> +	struct mali_c55 *mali_c55;
>>>> +	struct mali_c55_cap_dev *cap_dev;
>>>> +	enum mali_c55_resizer_ids id;
>>>> +	struct v4l2_subdev sd;
>>>> +	struct media_pad pads[MALI_C55_RZR_NUM_PADS];
>>>> +	unsigned int num_routes;
>>>> +	bool streaming;
>>>> +};
>>>> +
>>>> +enum mali_c55_cap_devs {
>>>> +	MALI_C55_CAP_DEV_FR,
>>>> +	MALI_C55_CAP_DEV_DS,
>>>> +	MALI_C55_NUM_CAP_DEVS
>>>> +};
>>>> +
>>>> +struct mali_c55_fmt {
>>> mali_c55_format_info would be a better name I think, as this stores
>>> format information, not formats.
>>>
>>>> +	u32 fourcc;
>>>> +	unsigned int mbus_codes[2];
>>> A comment to explain why we have two media bus codes would be useful.
>>> You can document the whole structure if desired :-)
>>>
>>>> +	bool is_raw;
>>>> +	struct mali_c55_fmt_registers {
>>> Make it an anonymous structure, it's never used anywhere else.
>>>
>>>> +		unsigned int base_mode;
>>>> +		unsigned int uv_plane;
>>> If those are register field values, use u32 instead of unsigned int.
>>>
>>>> +	} registers;
>>> It's funny, we tend to abbreviate different things, I would have used
>>> "regs" here but written "format" in full in the structure name :-)
>>>
>>>> +};
>>>> +
>>>> +enum mali_c55_isp_bayer_order {
>>>> +	MALI_C55_BAYER_ORDER_RGGB,
>>>> +	MALI_C55_BAYER_ORDER_GRBG,
>>>> +	MALI_C55_BAYER_ORDER_GBRG,
>>>> +	MALI_C55_BAYER_ORDER_BGGR
>>> These are registers values too, they belong to mali-c55-registers.h.
>>>
>>>> +};
>>>> +
>>>> +struct mali_c55_isp_fmt {
>>> mali_c55_isp_format_info
>>>
>>>> +	u32 code;
>>>> +	enum v4l2_pixel_encoding encoding;
>>> Here you use v4l2_pixel_encoding, for mali_c55_fmt you use is_raw. Maybe
>>> pick the same option for both structures ?
>>>
>>>> +	enum mali_c55_isp_bayer_order order;
>>>> +};
>>>> +
>>>> +enum mali_c55_planes {
>>>> +	MALI_C55_PLANE_Y,
>>>> +	MALI_C55_PLANE_UV,
>>>> +	MALI_C55_NUM_PLANES
>>>> +};
>>>> +
>>>> +struct mali_c55_buffer {
>>>> +	struct vb2_v4l2_buffer vb;
>>>> +	bool plane_done[MALI_C55_NUM_PLANES];
>>> I think tracking the pending state would simplify the logic in
>>> mali_c55_set_plane_done(), which would become
>>>
>>> 	curr_buf->plane_pending[plane] = false;
>>>
>>> 	if (!curr_buf->plane_pending[0] && !curr_buf->plane_pending[1]) {
>>> 		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
>>> 		cap_dev->buffers.curr = NULL;
>>> 		isp->frame_sequence++;
>>> 	}
>>>
>>> Or a counter may be even easier (and would consume less memory).
>> I'll do the counter; a  similar function in the stats code does so already.
>>
>>>> +	struct list_head queue;
>>>> +	u32 addrs[MALI_C55_NUM_PLANES];
>>> This stores DMA addresses, use dma_addr_t.
>>>
>>>> +};
>>>> +
>>>> +struct mali_c55_cap_dev {
>>>> +	struct mali_c55 *mali_c55;
>>>> +	struct mali_c55_resizer *rzr;
>>>> +	struct video_device vdev;
>>>> +	struct media_pad pad;
>>>> +	struct vb2_queue queue;
>>>> +	struct mutex lock;
>>>> +	unsigned int reg_offset;
>>> Manual handling of the offset everywhere, with parametric macros for the
>>> resizer register addresses, isn't very nice. Introduce resizer-specific
>>> accessors and capture-specific accessors (mali_c55_rsz_write(), ...)
>>> that take a mali_c55_resizer or mali_c55_cap_dev pointer, and handle the
>>> offset there. The register macros should loose their offset parameter.
>>>
>>> You could also use a single set of accessors that would become
>>> path/output-specific (mali_c55_path_write() ? mali_c55_output_write()
>>> ?), that may make the code easier to read.
>>>
>>> You can also replace reg_offset with a void __iomem * base, which would
>>> avoid the computation at runtime.
>>>
>>>> +
>>>> +	struct mali_c55_mode {
>>> Make the structure anonymous.
>>>
>>>> +		const struct mali_c55_fmt *capture_fmt;
>>>> +		struct v4l2_pix_format_mplane pix_mp;
>>>> +	} mode;
>>> What's a "mode" ? I think I'd name this
>>>
>>> 	struct {
>>> 		const struct mali_c55_fmt *info;
>>> 		struct v4l2_pix_format_mplane format;
>>> 	} format;
>>>
>>> Or you could just drop the structure and have
>>>
>>> 	const struct mali_c55_fmt *format_info;
>>> 	struct v4l2_pix_format_mplane format;
>>>
>>> or something similar.
>>>
>>>> +
>>>> +	struct {
>>>> +		spinlock_t lock;
>>>> +		struct list_head queue;
>>>> +		struct mali_c55_buffer *curr;
>>>> +		struct mali_c55_buffer *next;
>>>> +	} buffers;
>>>> +
>>>> +	bool streaming;
>>>> +};
>>>> +
>>>> +enum mali_c55_config_spaces {
>>>> +	MALI_C55_CONFIG_PING,
>>>> +	MALI_C55_CONFIG_PONG,
>>>> +	MALI_C55_NUM_CONFIG_SPACES
>>> The last enumerator is not used.
>>>
>>>> +};
>>>> +
>>>> +struct mali_c55_ctx {
>>> mali_c55_context ?
>>>
>>>> +	struct mali_c55 *mali_c55;
>>>> +	void *registers;
>>> Please document this structure and explain that this field points to a
>>> copy of the register space in system memory, I was about to write you're
>>> missing __iomem :-)
>> Will do
>>
>>>> +	phys_addr_t base;
>>>> +	spinlock_t lock;
>>>> +	struct list_head list;
>>>> +};
>>>> +
>>>> +struct mali_c55 {
>>>> +	struct device *dev;
>>>> +	struct resource *res;
>>> You could possibly drop this field by passing the physical address of
>>> the register space from mali_c55_probe() to mali_c55_init_context() as a
>>> function parameter.
>>>
>>>> +	void __iomem *base;
>>>> +	struct dma_chan *channel;
>>>> +	struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
>>>> +
>>>> +	u16 capabilities;
>>>> +	struct media_device media_dev;
>>>> +	struct v4l2_device v4l2_dev;
>>>> +	struct media_pipeline pipe;
>>>> +
>>>> +	struct mali_c55_tpg tpg;
>>>> +	struct mali_c55_isp isp;
>>>> +	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
>>>> +	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
>>>> +
>>>> +	struct list_head contexts;
>>>> +	enum mali_c55_config_spaces next_config;
>>>> +};
>>>> +
>>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
>>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
>>>> +		  bool force_hardware);
>>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
>>>> +			  u32 mask, u32 val);
>>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
>>>> +			  enum mali_c55_config_spaces cfg_space);
>>>> +
>>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55);
>>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55);
>>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
>>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
>>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55);
>>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
>>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
>>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
>>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
>>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>>>> +			     enum mali_c55_planes plane);
>>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
>>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
>>>> +
>>>> +bool mali_c55_format_is_raw(unsigned int mbus_code);
>>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
>>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
>>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
>>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
>>>> +
>>>> +const struct mali_c55_isp_fmt *
>>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
>>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
>>>> +#define for_each_mali_isp_fmt(fmt)\
>>> #define for_each_mali_isp_fmt(fmt) \
>>>
>>>> +	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
>>> Looks like parentheses were on sale :-)
>> Hah
>>
>>> 	for ((fmt) = NULL; (fmt) = mali_c55_isp_fmt_next(fmt); )
>>>
>>> This macro is used in two places only, in the mali-c55-isp.c file where
>>> open-coding the loop without using mali_c55_isp_fmt_next() would be more
>>> efficient, and in mali-c55-resizer.c where a function to return format
>>> i'th would be more efficient. I think you can drop the macro and the
>>> mali_c55_isp_fmt_next() function.
>>>
>>>> +
>>>> +#endif /* _MALI_C55_COMMON_H */
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>> new file mode 100644
>>>> index 000000000000..50caf5ee7474
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>> @@ -0,0 +1,767 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Core driver code
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#include <linux/bitops.h>
>>>> +#include <linux/cleanup.h>
>>>> +#include <linux/clk.h>
>>>> +#include <linux/delay.h>
>>>> +#include <linux/device.h>
>>>> +#include <linux/dmaengine.h>
>>>> +#include <linux/dma-mapping.h>
>>>> +#include <linux/interrupt.h>
>>>> +#include <linux/iopoll.h>
>>>> +#include <linux/ioport.h>
>>>> +#include <linux/mod_devicetable.h>
>>>> +#include <linux/of.h>
>>>> +#include <linux/of_reserved_mem.h>
>>>> +#include <linux/platform_device.h>
>>>> +#include <linux/pm_runtime.h>
>>>> +#include <linux/scatterlist.h>
>>> I don't think this is needed.
>>>
>>> Missing slab.h.
>>>
>>>> +#include <linux/string.h>
>>>> +
>>>> +#include <media/media-entity.h>
>>>> +#include <media/v4l2-device.h>
>>>> +#include <media/videobuf2-dma-contig.h>
>>>> +
>>>> +#include "mali-c55-common.h"
>>>> +#include "mali-c55-registers.h"
>>>> +
>>>> +static const char * const mali_c55_interrupt_names[] = {
>>>> +	[MALI_C55_IRQ_ISP_START] = "ISP start",
>>>> +	[MALI_C55_IRQ_ISP_DONE] = "ISP done",
>>>> +	[MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
>>>> +	[MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
>>>> +	[MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
>>>> +	[MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
>>>> +	[MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
>>>> +	[MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
>>>> +	[MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
>>>> +	[MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
>>>> +	[MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
>>>> +	[MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
>>>> +	[MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
>>>> +	[MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
>>>> +	[MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
>>>> +	[MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
>>>> +	[MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
>>>> +	[MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
>>>> +	[MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
>>>> +	[MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
>>>> +	[MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
>>>> +	[MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
>>>> +	[MALI_C55_IRQ_DMA_ERROR] = "DMA error",
>>>> +	[MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
>>>> +	[MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
>>>> +	[MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
>>>> +};
>>>> +
>>>> +static unsigned int config_space_addrs[] = {
>>> const
>>>
>>>> +	[MALI_C55_CONFIG_PING] = 0x0AB6C,
>>>> +	[MALI_C55_CONFIG_PONG] = 0x22B2C,
>>> Lowercase hex constants.
>>>
>>> Don't the values belong to mali-c55-registers.h ?
>>>
>>>> +};
>>>> +
>>>> +/* System IO
>>> /*
>>>    * System IO
>>>
>>>> + *
>>>> + * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
>>>> + * and 'pong'), with the  expectation that the 'active' space will be left
>>> s/the  /the /
>>>
>>>> + * untouched whilst a frame is being processed and the 'inactive' space
>>>> + * configured ready to be passed during the blanking period before the next
>>> s/to be passed/to be switched to/ ?
>>>
>>>> + * frame processing starts. These spaces should ideally be set via DMA transfer
>>>> + * from a buffer rather than through individual register set operations. There
>>>> + * is also a shared global register space which should be set normally. Of
>>>> + * course, the ISP might be included in a system which lacks a suitable DMA
>>>> + * engine, and the second configuration space might not be fitted at all, which
>>>> + * means we need to support four scenarios:
>>>> + *
>>>> + * 1. Multi config space, with DMA engine.
>>>> + * 2. Multi config space, no DMA engine.
>>>> + * 3. Single config space, with DMA engine.
>>>> + * 4. Single config space, no DMA engine.
>>>> + *
>>>> + * The first case is very easy, but the rest present annoying problems. The best
>>>> + * way to solve them seems to be simply to replicate the concept of DMAing over
>>>> + * the configuration buffer even if there's no DMA engine on the board, for
>>>> + * which we rely on memcpy. To facilitate this any read/write call that is made
>>>> + * to an address within those config spaces should infact be directed to a
>>>> + * buffer that was allocated to hold them rather than the IO memory itself. The
>>>> + * actual copy of that buffer to IO mem will happen on interrupt.
>>>> + */
>>>> +
>>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
>>>> +{
>>>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>> +
>>>> +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
>>>> +		spin_lock(&ctx->lock);
>>>> +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
>>>> +		((u32 *)ctx->registers)[addr] = val;
>>>> +		spin_unlock(&ctx->lock);
>>>> +
>>>> +		return;
>>>> +	}
>>> Ouch. This is likely the second comment you really won't like (after the
>>> comment regarding mali_c55_update_bits() at the very top). I apologize
>>> in advance.
>> All good - it's the thinking that is painful, changing the code is easy :)
>>
>>> I really don't like this. Directing writes either to hardware registers
>>> or to the shadow registers in the context makes the callers of the
>>> read/write accessors very hard to read. The probe code, for instance,
>>> mixes writes to hardware registers and writes to the context shadow
>>> registers to initialize the value of some of the shadow registers.
>>>
>>> I'd like to split the read/write accessors into functions that access
>>> the hardware registers (that's easy) and functions that access the
>>> shadow registers. I think the latter should receive a mali_c55_ctx
>>> pointer instead of a mali_c55 pointer to prepare for multi-context
>>> support.
>>>
>>> You can add WARN_ON() guards to the two sets of functions, to ensure
>>> that no register from the "other" space gets passed to the wrong
>>> function by mistake.
>>>
>>>> +
>>>> +	writel(val, mali_c55->base + addr);
>>>> +}
>>>> +
>>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
>>>> +		  bool force_hardware)
>>> force_hardware is never set to true.
>>>
>>>> +{
>>>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>> +	u32 val;
>>>> +
>>>> +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
>>>> +		spin_lock(&ctx->lock);
>>>> +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
>>>> +		val = ((u32 *)ctx->registers)[addr];
>>>> +		spin_unlock(&ctx->lock);
>>>> +
>>>> +		return val;
>>>> +	}
>>>> +
>>>> +	return readl(mali_c55->base + addr);
>>>> +}
>>>> +
>>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
>>>> +			  u32 mask, u32 val)
>>>> +{
>>>> +	u32 orig, tmp;
>>>> +
>>>> +	orig = mali_c55_read(mali_c55, addr, false);
>>>> +
>>>> +	tmp = orig & ~mask;
>>>> +	tmp |= (val << (ffs(mask) - 1)) & mask;
>>>> +
>>>> +	if (tmp != orig)
>>>> +		mali_c55_write(mali_c55, addr, tmp);
>>>> +}
>>>> +
>>>> +static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
>>>> +			     dma_addr_t dst, enum dma_data_direction dir)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>> +	struct dma_async_tx_descriptor *tx;
>>>> +	enum dma_status status;
>>>> +	dma_cookie_t cookie;
>>>> +
>>>> +	tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
>>>> +				       MALI_C55_CONFIG_SPACE_SIZE, 0);
>>>> +	if (!tx) {
>>>> +		dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
>>>> +		return -EIO;
>>>> +	}
>>>> +
>>>> +	cookie = dmaengine_submit(tx);
>>>> +	if (dma_submit_error(cookie)) {
>>>> +		dev_err(mali_c55->dev, "error submitting dma transfer\n");
>>>> +		return -EIO;
>>>> +	}
>>>> +
>>>> +	status = dma_sync_wait(mali_c55->channel, cookie);
>>> I've just realized this performs a busy-wait :-S See the comment in the
>>> probe function about the threaded IRQ handler. I think we'll need to
>>> rework all this. It could be done on top though.
>> It can be switched to an asynchronous transfer quite easily.
>>>> +	if (status != DMA_COMPLETE) {
>>>> +		dev_err(mali_c55->dev, "dma transfer failed\n");
>>>> +		return -EIO;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
>>>> +			     enum mali_c55_config_spaces cfg_space)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>> +	struct device *dma_dev = mali_c55->channel->device->dev;
>>>> +	dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
>>>> +	dma_addr_t dst;
>>>> +	int ret;
>>>> +
>>>> +	guard(spinlock)(&ctx->lock);
>>>> +
>>>> +	dst = dma_map_single(dma_dev, ctx->registers,
>>>> +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
>>>> +	if (dma_mapping_error(dma_dev, dst)) {
>>>> +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
>>>> +		return -EIO;
>>>> +	}
>>>> +
>>>> +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
>>>> +	dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
>>>> +			 DMA_FROM_DEVICE);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
>>>> +		       enum mali_c55_config_spaces cfg_space)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>> +	struct device *dma_dev = mali_c55->channel->device->dev;
>>>> +	dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
>>>> +	dma_addr_t src;
>>>> +	int ret;
>>>> +
>>>> +	guard(spinlock)(&ctx->lock);
>>> The code below can take a large amount of time, holding a spinlock will
>>> disable interrupts on the local CPU, that's not good :-(
>> The intention here is just to prevent the rest of the driver writing into the register space whilst
>> it's being DMA transferred to the hardware - perhaps a different means to signal it's safe is more
>> appropriate?
> Are there other parts of the driver that can write to the config space,
> or do all writes to the config space go through this function ?


All writes to the hardware go through this function, but what I meant was to prevent other parts of 
the driver writing into the buffer at ctx->registers whilst this write is occurring

>
>>>> +
>>>> +	src = dma_map_single(dma_dev, ctx->registers,
>>>> +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
>>>> +	if (dma_mapping_error(dma_dev, src)) {
>>>> +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
>>>> +		return -EIO;
>>>> +	}
>>>> +
>>>> +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
>>>> +	dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
>>>> +			 DMA_TO_DEVICE);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static int mali_c55_config_read(struct mali_c55_ctx *ctx,
>>>> +				enum mali_c55_config_spaces cfg_space)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>> +
>>>> +	if (mali_c55->channel) {
>>>> +		return mali_c55_dma_read(ctx, cfg_space);
>>> As this function is used at probe time only, to initialize the context,
>>> I think DMA is overkill.
>> Agreed - I'll switch this to memcpy_fromio()
>>
>>>> +	} else {
>>>> +		memcpy_fromio(ctx->registers,
>>>> +			      mali_c55->base + config_space_addrs[cfg_space],
>>>> +			      MALI_C55_CONFIG_SPACE_SIZE);
>>>> +		return 0;
>>>> +	}
>>>> +}
>>>> +
>>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
>>>> +			  enum mali_c55_config_spaces cfg_space)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>> +
>>>> +	if (mali_c55->channel) {
>>>> +		return mali_c55_dma_write(ctx, cfg_space);
>>>> +	} else {
>>>> +		memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
>>>> +			    ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
>>>> +		return 0;
>>>> +	}
>>> Could you measure the time it typically takes to write the registers
>>> using DMA compared to using memcpy_toio() ?
>> I will test this and come back to you.
>>
>>>> +}
>>>> +
>>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);
>>> I think it's too early to tell how multi-context support will look like.
>>> I'm fine keeping mali_c55_get_active_context() as changing that would be
>>> very intrusive (even if I think it will need to be changed), but the
>>> list of contexts is neither the mechanism we'll use, nor something we
>>> need now. Drop the list, embed the context in struct mali_c55, and
>>> return the pointer to that single context from this function.
>>>
>>>> +}
>>>> +
>>>> +static void mali_c55_remove_links(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	unsigned int i;
>>>> +
>>>> +	media_entity_remove_links(&mali_c55->tpg.sd.entity);
>>>> +	media_entity_remove_links(&mali_c55->isp.sd.entity);
>>>> +
>>>> +	for (i = 0; i < MALI_C55_NUM_RZRS; i++)
>>>> +		media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
>>>> +
>>>> +	for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
>>>> +		media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
>>>> +}
>>>> +
>>>> +static int mali_c55_create_links(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	int ret;
>>>> +
>>>> +	/* Test pattern generator to ISP */
>>>> +	ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
>>>> +				    &mali_c55->isp.sd.entity,
>>>> +				    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
>>>> +		goto err_remove_links;
>>>> +	}
>>>> +
>>>> +	/* Full resolution resizer pipe. */
>>>> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>>> +			MALI_C55_ISP_PAD_SOURCE,
>>>> +			&mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
>>>> +			MALI_C55_RZR_SINK_PAD,
>>>> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
>>>> +		goto err_remove_links;
>>>> +	}
>>>> +
>>>> +	/* Full resolution bypass. */
>>>> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>>> +				    MALI_C55_ISP_PAD_SOURCE_BYPASS,
>>>> +				    &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
>>>> +				    MALI_C55_RZR_SINK_BYPASS_PAD,
>>>> +				    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
>>>> +		goto err_remove_links;
>>>> +	}
>>>> +
>>>> +	/* Resizer pipe to video capture nodes. */
>>>> +	ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
>>>> +			MALI_C55_RZR_SOURCE_PAD,
>>>> +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
>>>> +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev,
>>>> +			"failed to link FR resizer and video device\n");
>>>> +		goto err_remove_links;
>>>> +	}
>>>> +
>>>> +	/* The downscale pipe is an optional hardware block */
>>>> +	if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
>>>> +		ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>>> +			MALI_C55_ISP_PAD_SOURCE,
>>>> +			&mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
>>>> +			MALI_C55_RZR_SINK_PAD,
>>>> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>> +		if (ret) {
>>>> +			dev_err(mali_c55->dev,
>>>> +				"failed to link ISP and DS resizer\n");
>>>> +			goto err_remove_links;
>>>> +		}
>>>> +
>>>> +		ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
>>>> +			MALI_C55_RZR_SOURCE_PAD,
>>>> +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
>>>> +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>> +		if (ret) {
>>>> +			dev_err(mali_c55->dev,
>>>> +				"failed to link DS resizer and video device\n");
>>>> +			goto err_remove_links;
>>>> +		}
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +
>>>> +err_remove_links:
>>>> +	mali_c55_remove_links(mali_c55);
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	mali_c55_unregister_tpg(mali_c55);
>>>> +	mali_c55_unregister_isp(mali_c55);
>>>> +	mali_c55_unregister_resizers(mali_c55);
>>>> +	mali_c55_unregister_capture_devs(mali_c55);
>>>> +}
>>>> +
>>>> +static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	int ret;
>>>> +
>>>> +	ret = mali_c55_register_tpg(mali_c55);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	ret = mali_c55_register_isp(mali_c55);
>>>> +	if (ret)
>>>> +		goto err_unregister_entities;
>>>> +
>>>> +	ret = mali_c55_register_resizers(mali_c55);
>>>> +	if (ret)
>>>> +		goto err_unregister_entities;
>>>> +
>>>> +	ret = mali_c55_register_capture_devs(mali_c55);
>>>> +	if (ret)
>>>> +		goto err_unregister_entities;
>>>> +
>>>> +	ret = mali_c55_create_links(mali_c55);
>>>> +	if (ret)
>>>> +		goto err_unregister_entities;
>>>> +
>>>> +	return 0;
>>>> +
>>>> +err_unregister_entities:
>>>> +	mali_c55_unregister_entities(mali_c55);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	u32 product, version, revision, capabilities;
>>>> +
>>>> +	product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
>>>> +	version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
>>>> +	revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
>>>> +
>>>> +	dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
>>>> +		 product, version, revision);
>>>> +
>>>> +	capabilities = mali_c55_read(mali_c55,
>>>> +				     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
>>>> +				     false);
>>>> +	mali_c55->capabilities = (capabilities & 0xffff);
>>>> +
>>>> +	/* TODO: Might as well start some debugfs */
>>> If it's just to expose the version and capabilities, I think that's
>>> overkill. It's not needed for debug purpose (you can get it from the
>>> kernel log already). debugfs isn't meant to be accessible in production,
>>> so an application that would need access to the information wouldn't be
>>> able to use it.
>>>
>>>> +	dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);
>>> Combine the two messages into one.
>>>
>>>> +	return version;
>>>> +}
>>>> +
>>>> +static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>> +	u32 curr_config, next_config;
>>>> +
>>>> +	curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
>>>> +	curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
>>>> +		      >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
>>>> +	next_config = curr_config ^ 1;
>>>> +
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>> +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
>>>> +	mali_c55_config_write(ctx, next_config ?
>>>> +			      MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
>>>> +}
>>>> +
>>>> +static irqreturn_t mali_c55_isr(int irq, void *context)
>>>> +{
>>>> +	struct device *dev = context;
>>>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>>> +	u32 interrupt_status;
>>>> +	unsigned int i, j;
>>>> +
>>>> +	interrupt_status = mali_c55_read(mali_c55,
>>>> +					 MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
>>>> +					 false);
>>>> +	if (!interrupt_status)
>>>> +		return IRQ_NONE;
>>>> +
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
>>>> +		       interrupt_status);
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
>>>> +
>>>> +	for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
>>>> +		if (!(interrupt_status & (1 << i)))
>>>> +			continue;
>>>> +
>>>> +		switch (i) {
>>>> +		case MALI_C55_IRQ_ISP_START:
>>>> +			mali_c55_isp_queue_event_sof(mali_c55);
>>>> +
>>>> +			for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
>>>> +				mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
>>>> +
>>>> +			mali_c55_swap_next_config(mali_c55);
>>>> +
>>>> +			break;
>>>> +		case MALI_C55_IRQ_ISP_DONE:
>>>> +			/*
>>>> +			 * TODO: Where the ISP has no Pong config fitted, we'd
>>>> +			 * have to do the mali_c55_swap_next_config() call here.
>>>> +			 */
>>>> +			break;
>>>> +		case MALI_C55_IRQ_FR_Y_DONE:
>>>> +			mali_c55_set_plane_done(
>>>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
>>>> +				MALI_C55_PLANE_Y);
>>>> +			break;
>>>> +		case MALI_C55_IRQ_FR_UV_DONE:
>>>> +			mali_c55_set_plane_done(
>>>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
>>>> +				MALI_C55_PLANE_UV);
>>>> +			break;
>>>> +		case MALI_C55_IRQ_DS_Y_DONE:
>>>> +			mali_c55_set_plane_done(
>>>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
>>>> +				MALI_C55_PLANE_Y);
>>>> +			break;
>>>> +		case MALI_C55_IRQ_DS_UV_DONE:
>>>> +			mali_c55_set_plane_done(
>>>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
>>>> +				MALI_C55_PLANE_UV);
>>>> +			break;
>>>> +		default:
>>>> +			/*
>>>> +			 * Only the above interrupts are currently unmasked. If
>>>> +			 * we receive anything else here then something weird
>>>> +			 * has gone on.
>>>> +			 */
>>>> +			dev_err(dev, "masked interrupt %s triggered\n",
>>>> +				mali_c55_interrupt_names[i]);
>>>> +		}
>>>> +	}
>>>> +
>>>> +	return IRQ_HANDLED;
>>>> +}
>>>> +
>>>> +static int mali_c55_init_context(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	struct mali_c55_ctx *ctx;
>>>> +	int ret;
>>>> +
>>>> +	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
>>>> +	if (!ctx) {
>>>> +		dev_err(mali_c55->dev, "failed to allocate new context\n");
>>> No need for an error message when memory allocation fails.
>>>
>>>> +		return -ENOMEM;
>>>> +	}
>>>> +
>>>> +	ctx->base = mali_c55->res->start;
>>>> +	ctx->mali_c55 = mali_c55;
>>>> +
>>>> +	ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
>>>> +				 GFP_KERNEL | GFP_DMA);
>>>> +	if (!ctx->registers) {
>>>> +		ret = -ENOMEM;
>>>> +		goto err_free_ctx;
>>>> +	}
>>>> +
>>>> +	/*
>>>> +	 * The allocated memory is empty, we need to load the default
>>>> +	 * register settings. We just read Ping; it's identical to Pong.
>>>> +	 */
>>>> +	ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
>>>> +	if (ret)
>>>> +		goto err_free_registers;
>>>> +
>>>> +	list_add_tail(&ctx->list, &mali_c55->contexts);
>>>> +
>>>> +	/*
>>>> +	 * Some features of the ISP need to be disabled by default and only
>>>> +	 * enabled at the same time as they're configured by a parameters buffer
>>>> +	 */
>>>> +
>>>> +	/* Bypass the sqrt and square compression and expansion modules */
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
>>>> +			     MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
>>>> +			     MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
>>>> +
>>>> +	/* Bypass the temper module */
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
>>>> +		       MALI_C55_REG_BYPASS_2_TEMPER);
>>>> +
>>>> +	/* Bypass the colour noise reduction  */
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
>>>> +		       MALI_C55_REG_BYPASS_4_CNR);
>>>> +
>>>> +	/* Disable the sinter module */
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
>>>> +			     MALI_C55_SINTER_ENABLE_MASK, 0x00);
>>>> +
>>>> +	/* Disable the RGB Gamma module for each output */
>>>> +	mali_c55_write(
>>>> +		mali_c55,
>>>> +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
>>>> +		0x00);
>>>> +	mali_c55_write(
>>>> +		mali_c55,
>>>> +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
>>>> +		0x00);
>>>> +
>>>> +	/* Disable the colour correction matrix */
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
>>>> +
>>>> +	return 0;
>>>> +
>>>> +err_free_registers:
>>>> +	kfree(ctx->registers);
>>>> +err_free_ctx:
>>>> +	kfree(ctx);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static int mali_c55_runtime_resume(struct device *dev)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>>> +	int ret;
>>>> +
>>>> +	ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
>>>> +				      mali_c55->clks);
>>>> +	if (ret)
>>>> +		dev_err(mali_c55->dev, "failed to enable clocks\n");
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static int mali_c55_runtime_suspend(struct device *dev)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>>> +
>>>> +	clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static const struct dev_pm_ops mali_c55_pm_ops = {
>>>> +	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
>>>> +				pm_runtime_force_resume)
>>>> +	SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
>>>> +			   NULL)
>>>> +};
>>>> +
>>>> +static int mali_c55_probe(struct platform_device *pdev)
>>>> +{
>>>> +	struct device *dev = &pdev->dev;
>>>> +	struct mali_c55 *mali_c55;
>>>> +	dma_cap_mask_t mask;
>>>> +	u32 version;
>>>> +	int ret;
>>>> +	u32 val;
>>>> +
>>>> +	mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
>>>> +	if (!mali_c55)
>>>> +		return dev_err_probe(dev, -ENOMEM,
>>>> +				     "failed to allocate memory\n");
>>> 		return -ENOMEM;
>>>
>>> There's no need to print messages for memory allocation failures.
>>>
>>>> +
>>>> +	mali_c55->dev = dev;
>>>> +	platform_set_drvdata(pdev, mali_c55);
>>>> +
>>>> +	mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
>>>> +								&mali_c55->res);
>>>> +	if (IS_ERR(mali_c55->base))
>>>> +		return dev_err_probe(dev, PTR_ERR(mali_c55->base),
>>>> +				     "failed to map IO memory\n");
>>>> +
>>>> +	ret = platform_get_irq(pdev, 0);
>>>> +	if (ret < 0)
>>>> +		return dev_err_probe(dev, ret, "failed to get interrupt num\n");
>>> s/ num// or s/num/number/
>>>
>>>> +
>>>> +	ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
>>>> +					mali_c55_isr, IRQF_ONESHOT,
>>>> +					dev_driver_string(&pdev->dev),
>>>> +					&pdev->dev);
>>> Requested the IRQ should be done much lower, after you have initialized
>>> everything, or an IRQ that would fire early would have really bad
>>> consequences.
>>>
>>> A comment to explain why you need a threaded interrupt handler would be
>>> good. I assume it is due only to the need to transfer the registers
>>> using DMA. I wonder if we should then split the interrupt handler in
>>> two, with a non-threaded part for the operations that can run quickly,
>>> and a threaded part for the reprogramming.
>> Instead of passing NULL for the top handler you mean?
> Yes.
>
>>> It may also be that we could just start the DMA transfer in the
>>> non-threaded handler without waiting synchronously for it to complete.
>>> That would be a bigger change, and would require checking race
>>> conditions carefully. On the other hand, I'm a bit concerned about the
>>> current implementation, have you tested what happens if the DMA transfer
>>> takes too long to complete, and spans frame boundaries ?
>> No; I can force a delay in the DMA engine and see how it behaves. The stats buffers are currently
>> DMAd asynchronously; I don't think it'd be a huge change to make the configuration buffer handling
>> work that way too.
> That would be nice. Race conditions will need to be carefully considered
> (but they should, anyway).
>
>>>> +	if (ret)
>>>> +		return dev_err_probe(dev, ret, "failed to request irq\n");
>>>> +
>>>> +	for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
>>>> +		mali_c55->clks[i].id = mali_c55_clk_names[i];
>>>> +
>>>> +	ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
>>>> +	if (ret)
>>>> +		return dev_err_probe(dev, ret, "failed to acquire clocks\n");
>>>> +
>>>> +	pm_runtime_enable(&pdev->dev);
>>>> +
>>>> +	ret = pm_runtime_resume_and_get(&pdev->dev);
>>>> +	if (ret)
>>>> +		goto err_pm_runtime_disable;
>>>> +
>>>> +	of_reserved_mem_device_init(dev);
>>> I'd move this, the vb2_dma_contig_set_max_seg_size() call and the
>>> dma_cap_* calls before pm_runtime_enable() as they don't need the device
>>> to be powered.
>>>
>>>> +	version = mali_c55_check_hwcfg(mali_c55);
>>>> +	vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
>>>> +
>>>> +	/* Use "software only" context management. */
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>> +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
>>>> +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
>>> You handle that in mali_c55_isp_start(), does the register have to be
>>> set here too ?
>>>
>>>> +
>>>> +	dma_cap_zero(mask);
>>>> +	dma_cap_set(DMA_MEMCPY, mask);
>>>> +
>>>> +	/*
>>>> +	 * No error check, because we will just fallback on memcpy if there is
>>>> +	 * no usable DMA channel on the system.
>>>> +	 */
>>>> +	mali_c55->channel = dma_request_channel(mask, NULL, NULL);
>>>> +
>>>> +	INIT_LIST_HEAD(&mali_c55->contexts);
>>>> +	ret = mali_c55_init_context(mali_c55);
>>>> +	if (ret)
>>>> +		goto err_release_dma_channel;
>>>> +
>>> I'd move all the code from here ...
>>>
>>>> +	mali_c55->media_dev.dev = dev;
>>>> +	strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
>>>> +		sizeof(mali_c55->media_dev.model));
>>>> +	mali_c55->media_dev.hw_revision = version;
>>>> +
>>>> +	media_device_init(&mali_c55->media_dev);
>>>> +	ret = media_device_register(&mali_c55->media_dev);
>>>> +	if (ret)
>>>> +		goto err_cleanup_media_device;
>>>> +
>>>> +	mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
>>>> +	ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
>>>> +	if (ret) {
>>>> +		dev_err(dev, "failed to register V4L2 device\n");
>>>> +		goto err_unregister_media_device;
>>>> +	};
>>>> +
>>>> +	ret = mali_c55_register_entities(mali_c55);
>>>> +	if (ret) {
>>>> +		dev_err(dev, "failed to register entities\n");
>>>> +		goto err_unregister_v4l2_device;
>>>> +	}
>>> ... to here to a separate function, or maybe fold it all in
>>> mali_c55_register_entities() (which should the be renamed). Same thing
>>> for the cleanup code.
>>>
>>>> +
>>>> +	/* Set safe stop to ensure we're in a non-streaming state */
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>>>> +		       MALI_C55_INPUT_SAFE_STOP);
>>>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>>>> +
>>>> +	/*
>>>> +	 * We're ready to process interrupts. Clear any that are set and then
>>>> +	 * unmask them for processing.
>>>> +	 */
>>>> +	mali_c55_write(mali_c55, 0x30, 0xffffffff);
>>>> +	mali_c55_write(mali_c55, 0x34, 0xffffffff);
>>>> +	mali_c55_write(mali_c55, 0x40, 0x01);
>>>> +	mali_c55_write(mali_c55, 0x40, 0x00);
>>> Please replace the register addresses with macros.
>>>
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);
>>> The value should use the interrupt bits macros.
>>>
>>>> +
>>>> +	pm_runtime_put(&pdev->dev);
>>> Once power gets cut, the registers your programmed above may be lost. I
>>> think you should programe them in the runtime PM resume handler.
>>>
>>>> +
>>>> +	return 0;
>>>> +
>>>> +err_unregister_v4l2_device:
>>>> +	v4l2_device_unregister(&mali_c55->v4l2_dev);
>>>> +err_unregister_media_device:
>>>> +	media_device_unregister(&mali_c55->media_dev);
>>>> +err_cleanup_media_device:
>>>> +	media_device_cleanup(&mali_c55->media_dev);
>>>> +err_release_dma_channel:
>>>> +	dma_release_channel(mali_c55->channel);
>>>> +err_pm_runtime_disable:
>>>> +	pm_runtime_disable(&pdev->dev);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static void mali_c55_remove(struct platform_device *pdev)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
>>>> +	struct mali_c55_ctx *ctx, *tmp;
>>>> +
>>>> +	list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
>>>> +		list_del(&ctx->list);
>>>> +		kfree(ctx->registers);
>>>> +		kfree(ctx);
>>>> +	}
>>>> +
>>>> +	mali_c55_remove_links(mali_c55);
>>>> +	mali_c55_unregister_entities(mali_c55);
>>>> +	v4l2_device_put(&mali_c55->v4l2_dev);
>>>> +	media_device_unregister(&mali_c55->media_dev);
>>>> +	media_device_cleanup(&mali_c55->media_dev);
>>>> +	dma_release_channel(mali_c55->channel);
>>>> +}
>>>> +
>>>> +static const struct of_device_id mali_c55_of_match[] = {
>>>> +	{ .compatible = "arm,mali-c55", },
>>>> +	{},
>>> 	{ /* Sentinel */ },
>>>
>>>> +};
>>>> +MODULE_DEVICE_TABLE(of, mali_c55_of_match);
>>>> +
>>>> +static struct platform_driver mali_c55_driver = {
>>>> +	.driver = {
>>>> +		.name = "mali-c55",
>>>> +		.of_match_table = of_match_ptr(mali_c55_of_match),
>>> Drop of_match_ptr().
>>>
>>>> +		.pm = &mali_c55_pm_ops,
>>>> +	},
>>>> +	.probe = mali_c55_probe,
>>>> +	.remove_new = mali_c55_remove,
>>>> +};
>>>> +
>>>> +module_platform_driver(mali_c55_driver);
>>> Blank line.
>>>
>>>> +MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
>>>> +MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
>>>> +MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
>>>> +MODULE_LICENSE("GPL");
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>> new file mode 100644
>>>> index 000000000000..ea8b7b866e7a
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>> @@ -0,0 +1,611 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Image signal processor
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#include <linux/delay.h>
>>>> +#include <linux/iopoll.h>
>>>> +#include <linux/property.h>
>>>> +#include <linux/string.h>
>>>> +
>>>> +#include <media/media-entity.h>
>>>> +#include <media/v4l2-common.h>
>>>> +#include <media/v4l2-event.h>
>>>> +#include <media/v4l2-mc.h>
>>>> +#include <media/v4l2-subdev.h>
>>>> +
>>>> +#include "mali-c55-common.h"
>>>> +#include "mali-c55-registers.h"
>>>> +
>>>> +static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
>>>> +	{
>>>> +		.code = MEDIA_BUS_FMT_SRGGB20_1X20,
>>>> +		.order = MALI_C55_BAYER_ORDER_RGGB,
>>>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>>>> +	},
>>>> +	{
>>>> +		.code = MEDIA_BUS_FMT_SGRBG20_1X20,
>>>> +		.order = MALI_C55_BAYER_ORDER_GRBG,
>>>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>>>> +	},
>>>> +	{
>>>> +		.code = MEDIA_BUS_FMT_SGBRG20_1X20,
>>>> +		.order = MALI_C55_BAYER_ORDER_GBRG,
>>>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>>>> +	},
>>>> +	{
>>>> +		.code = MEDIA_BUS_FMT_SBGGR20_1X20,
>>>> +		.order = MALI_C55_BAYER_ORDER_BGGR,
>>>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>>>> +	},
>>>> +	{
>>>> +		.code = MEDIA_BUS_FMT_RGB202020_1X60,
>>>> +		.order = 0, /* Not relevant for this format */
>>>> +		.encoding = V4L2_PIXEL_ENC_RGB,
>>>> +	}
>>>> +	/*
>>>> +	 * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
>>>> +	 * also support YUV input from a sensor passed-through to the output. At
>>>> +	 * present we have no mechanism to test that though so it may have to
>>>> +	 * wait a while...
>>>> +	 */
>>>> +};
>>>> +
>>>> +const struct mali_c55_isp_fmt *
>>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
>>>> +{
>>>> +	if (!fmt)
>>>> +		fmt = &mali_c55_isp_fmts[0];
>>>> +	else
>>>> +		fmt++;
>>>> +
>>>> +	for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
>>>> +		return fmt;
>>> That's peculiar.
>>>
>>> 	if (!fmt)
>>> 		fmt = &mali_c55_isp_fmts[0];
>>> 	else if (fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)])
>>> 		return ++fmt;
>>> 	else
>>> 		return NULL;
>>>
>>>> +
>>>> +	return NULL;
>>>> +}
>>>> +
>>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
>>>> +{
>>>> +	const struct mali_c55_isp_fmt *isp_fmt;
>>>> +
>>>> +	for_each_mali_isp_fmt(isp_fmt) {
>>> I would open-code the loop instead of using the macro, like you do
>>> below. It will be more efficient.
>>>
>>>> +		if (isp_fmt->code == mbus_code)
>>>> +			return true;
>>>> +	}
>>>> +
>>>> +	return false;
>>>> +}
>>>> +
>>>> +static const struct mali_c55_isp_fmt *
>>>> +mali_c55_isp_get_mbus_config_by_code(u32 code)
>>>> +{
>>>> +	unsigned int i;
>>>> +
>>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
>>>> +		if (mali_c55_isp_fmts[i].code == code)
>>>> +			return &mali_c55_isp_fmts[i];
>>>> +	}
>>>> +
>>>> +	return NULL;
>>>> +}
>>>> +
>>>> +static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	u32 val;
>>>> +
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);
>>> 	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>>> 		       MALI_C55_INPUT_SAFE_STOP);
>>>
>>>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>>>> +}
>>>> +
>>>> +static int mali_c55_isp_start(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>> +	const struct mali_c55_isp_fmt *cfg;
>>>> +	struct v4l2_mbus_framefmt *format;
>>> const
>>>
>>>> +	struct v4l2_subdev_state *state;
>>>> +	struct v4l2_rect *crop;
>>> const
>>>
>>>> +	struct v4l2_subdev *sd;
>>>> +	u32 val;
>>>> +	int ret;
>>>> +
>>>> +	sd = &mali_c55->isp.sd;
>>> Assign when declaring the variable.
>>>
>>>> +
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>> +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
>>>> +
>>>> +	/* Apply input windowing */
>>>> +	state = v4l2_subdev_get_locked_active_state(sd);
>>> Using .enable_streams() (see below) you'll get this for free.
>>>
>>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +	format = v4l2_subdev_state_get_format(state,
>>>> +					      MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +	cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
>>>> +
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
>>>> +		       MALI_C55_HC_START(crop->left));
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
>>>> +		       MALI_C55_HC_SIZE(crop->width));
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
>>>> +		       MALI_C55_VC_START(crop->top) |
>>>> +		       MALI_C55_VC_SIZE(crop->height));
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
>>>> +			     MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
>>>> +			     MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
>>>> +			     MALI_C55_BAYER_ORDER_MASK, cfg->order);
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
>>>> +			     MALI_C55_INPUT_WIDTH_MASK,
>>>> +			     MALI_C55_INPUT_WIDTH_20BIT);
>>>> +
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>>> +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
>>>> +			     cfg->encoding == V4L2_PIXEL_ENC_RGB ?
>>>> +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
>>>> +
>>>> +	ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev, "failed to DMA config\n");
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>>>> +		       MALI_C55_INPUT_SAFE_START);
>>>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>>> Should you return an error in case of timeout ?
>>>
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)
>>> Why is this not handled wired to .s_stream() ? Or better,
>>> .enable_streams() and .disable_streams().
>> There didn't really seem to be much point, since nothing outside this driver is ever going to start
>> the subdev streaming...it's not like the sensor case where a separate driver might have to call some
>> operation to start it.
> True, but we're moving towards recording per-stream and per-pad
> streaming state in the subdev through v4l2_subdev_enable_streams() and
> v4l2_subdev_disable_streams(). We will likely expand usage of that
> information, so it would be nice to keep it up-to-date already.


I'll move all the subdevs in the driver to .enable_streams() and .disable_streams()

>
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>>>> +	struct v4l2_subdev *sd;
>>>> +
>>>> +	if (isp->remote_src) {
>>>> +		sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
>>>> +		v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
>>>> +	}
>>>> +	isp->remote_src = NULL;
>>>> +
>>>> +	mali_c55_isp_stop(mali_c55);
>>>> +}
>>>> +
>>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>>>> +	struct media_pad *sink_pad;
>>>> +	struct v4l2_subdev *sd;
>>>> +	int ret;
>>>> +
>>>> +	sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
>>>> +	isp->remote_src = media_pad_remote_pad_unique(sink_pad);
>>>> +	if (IS_ERR(isp->remote_src)) {
>>> If you set the MUST_CONNECT flag for the MALI_C55_ISP_PAD_SINK_VIDEO pad
>>> I think you can drop this check.
>>>
>>>> +		dev_err(mali_c55->dev, "Failed to get source for ISP\n");
>>>> +		return PTR_ERR(isp->remote_src);
>>>> +	}
>>>> +
>>>> +	sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
>>>> +
>>>> +	isp->frame_sequence = 0;
>>>> +	ret = mali_c55_isp_start(mali_c55);
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev, "Failed to start ISP\n");
>>>> +		isp->remote_src = NULL;
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	/*
>>>> +	 * We only support a single input stream, so we can just enable the 1st
>>>> +	 * entry in the streams mask.
>>>> +	 */
>>>> +	ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev, "Failed to start ISP source\n");
>>>> +		mali_c55_isp_stop(mali_c55);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>> +				       struct v4l2_subdev_state *state,
>>>> +				       struct v4l2_subdev_mbus_code_enum *code)
>>>> +{
>>>> +	/*
>>>> +	 * Only the internal RGB processed format is allowed on the regular
>>>> +	 * processing source pad.
>>>> +	 */
>>>> +	if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
>>>> +		if (code->index)
>>>> +			return -EINVAL;
>>>> +
>>>> +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	/* On the sink and bypass pads all the supported formats are allowed. */
>>>> +	if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
>>>> +		return -EINVAL;
>>>> +
>>>> +	code->code = mali_c55_isp_fmts[code->index].code;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
>>>> +					struct v4l2_subdev_state *state,
>>>> +					struct v4l2_subdev_frame_size_enum *fse)
>>>> +{
>>>> +	const struct mali_c55_isp_fmt *cfg;
>>>> +
>>>> +	if (fse->index > 0)
>>>> +		return -EINVAL;
>>>> +
>>>> +	/*
>>>> +	 * Only the internal RGB processed format is allowed on the regular
>>>> +	 * processing source pad.
>>>> +	 *
>>>> +	 * On the sink and bypass pads all the supported formats are allowed.
>>>> +	 */
>>>> +	if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
>>>> +		if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
>>>> +			return -EINVAL;
>>>> +	} else {
>>>> +		cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
>>>> +		if (!cfg)
>>>> +			return -EINVAL;
>>>> +	}
>>>> +
>>>> +	fse->min_width = MALI_C55_MIN_WIDTH;
>>>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
>>>> +	fse->max_width = MALI_C55_MAX_WIDTH;
>>>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
>>>> +				struct v4l2_subdev_state *state,
>>>> +				struct v4l2_subdev_format *format)
>>>> +{
>>>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>>>> +	struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
>>>> +	const struct mali_c55_isp_fmt *cfg;
>>>> +	struct v4l2_rect *crop;
>>>> +
>>>> +	/*
>>>> +	 * Disallow set_fmt on the source pads; format is fixed and the sizes
>>>> +	 * are the result of applying the sink crop rectangle to the sink
>>>> +	 * format.
>>>> +	 */
>>>> +	if (format->pad)
>>> 	if (format->pad != MALI_C55_ISP_PAD_SINK_VIDEO)
>>>
>>>> +		return v4l2_subdev_get_fmt(sd, state, format);
>>>> +
>>>> +	cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
>>>> +	if (!cfg)
>>>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>> +	fmt->field = V4L2_FIELD_NONE;
>>> Do you intentionally allow the colorspace fields to be overwritten to
>>> any value ? rkisp1_isp_set_src_fmt() and rkisp1_isp_set_sink_fmt() may
>>> show you how this could be handled.
>>>
>>>> +
>>>> +	/*
>>>> +	 * Clamp sizes in the accepted limits and clamp the crop rectangle in
>>>> +	 * the new sizes.
>>>> +	 */
>>>> +	clamp_t(unsigned int, fmt->width,
>>>> +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>>> +	clamp_t(unsigned int, fmt->width,
>>>> +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>> clamp_t() returns a value, which you ignore. Those are no-ops. You meant
>>>
>>> 	fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
>>> 			     MALI_C55_MAX_WIDTH);
>>> 	fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
>>> 			      MALI_C55_MAX_HEIGHT);
>>>
>>> Same for every use of clamp_t() through the whole driver.
>>>
>>> Also, do you need clamp_t() ? I think all values are unsigned int, you
>>> can use clamp().
>>>
>>> Are there any alignment constraints, such a multiples of two for bayer
>>> formats ? Same in all the other locations where applicable.
>>>
>>>> +
>>>> +	sink_fmt = v4l2_subdev_state_get_format(state,
>>>> +						MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +	*sink_fmt = *fmt;
>>>> +
>>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +	crop->left = 0;
>>>> +	crop->top = 0;
>>>> +	crop->width = fmt->width;
>>>> +	crop->height = fmt->height;
>>>> +
>>>> +	/*
>>>> +	 * Propagate format to source pads. On the 'regular' output pad use
>>>> +	 * the internal RGB processed format, while on the bypass pad simply
>>>> +	 * replicate the ISP sink format. The sizes on both pads are the same as
>>>> +	 * the ISP sink crop rectangle.
>>>> +	 */
>>> Colorspace information will need to be propagated too.
>>>
>>>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
>>>> +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>> +	src_fmt->width = crop->width;
>>>> +	src_fmt->height = crop->height;
>>>> +
>>>> +	src_fmt = v4l2_subdev_state_get_format(state,
>>>> +					       MALI_C55_ISP_PAD_SOURCE_BYPASS);
>>>> +	src_fmt->code = fmt->code;
>>>> +	src_fmt->width = crop->width;
>>>> +	src_fmt->height = crop->height;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
>>>> +				      struct v4l2_subdev_state *state,
>>>> +				      struct v4l2_subdev_selection *sel)
>>>> +{
>>>> +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
>>> 	sel->pad != MALI_C55_ISP_PAD_SINK_VIDEO
>>>
>>>> +		return -EINVAL;
>>>> +
>>>> +	sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
>>>> +				      struct v4l2_subdev_state *state,
>>>> +				      struct v4l2_subdev_selection *sel)
>>>> +{
>>>> +	struct v4l2_mbus_framefmt *src_fmt;
>>>> +	struct v4l2_mbus_framefmt *fmt;
>>> const
>>>
>>>> +	struct v4l2_rect *crop;
>>>> +
>>>> +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
>>> Ditto.
>>>
>>>> +		return -EINVAL;
>>>> +
>>>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +
>>>> +	clamp_t(unsigned int, sel->r.left, 0, fmt->width);
>>>> +	clamp_t(unsigned int, sel->r.top, 0, fmt->height);
>>>> +	clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
>>>> +		fmt->width - sel->r.left);
>>>> +	clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
>>>> +		fmt->height - sel->r.top);
>>>> +
>>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +	*crop = sel->r;
>>>> +
>>>> +	/* Propagate the crop rectangle sizes to the source pad format. */
>>>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
>>>> +	src_fmt->width = crop->width;
>>>> +	src_fmt->height = crop->height;
>>> Can you confirm that cropping doesn't affect the bypass path ?
>> Yes
>>
>>> And maybe
>>> add a comment to mention it.
>> Will
>>
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
>>>> +	.enum_mbus_code		= mali_c55_isp_enum_mbus_code,
>>>> +	.enum_frame_size	= mali_c55_isp_enum_frame_size,
>>>> +	.get_fmt		= v4l2_subdev_get_fmt,
>>>> +	.set_fmt		= mali_c55_isp_set_fmt,
>>>> +	.get_selection		= mali_c55_isp_get_selection,
>>>> +	.set_selection		= mali_c55_isp_set_selection,
>>>> +	.link_validate		= v4l2_subdev_link_validate_default,
>>>> +};
>>>> +
>>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	struct v4l2_event event = {
>>>> +		.type = V4L2_EVENT_FRAME_SYNC,
>>>> +	};
>>>> +
>>>> +	event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
>>>> +	v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
>>>> +}
>>>> +
>>>> +static int
>>>> +mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
>>>> +			     struct v4l2_event_subscription *sub)
>>>> +{
>>>> +	if (sub->type != V4L2_EVENT_FRAME_SYNC)
>>>> +		return -EINVAL;
>>>> +
>>>> +	/* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
>>>> +	if (sub->id != 0)
>>>> +		return -EINVAL;
>>>> +
>>>> +	return v4l2_event_subscribe(fh, sub, 0, NULL);
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
>>>> +	.subscribe_event = mali_c55_isp_subscribe_event,
>>>> +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
>>>> +};
>>>> +
>>>> +static const struct v4l2_subdev_ops mali_c55_isp_ops = {
>>>> +	.pad	= &mali_c55_isp_pad_ops,
>>>> +	.core	= &mali_c55_isp_core_ops,
>>>> +};
>>>> +
>>>> +static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
>>>> +				   struct v4l2_subdev_state *sd_state)
>>> You name this variable state in every other subdev operation handler.
>>>
>>>> +{
>>>> +	struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>> +	struct v4l2_rect *in_crop;
>>>> +
>>>> +	sink_fmt = v4l2_subdev_state_get_format(sd_state,
>>>> +						MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +	src_fmt = v4l2_subdev_state_get_format(sd_state,
>>>> +					       MALI_C55_ISP_PAD_SOURCE);
>>>> +	in_crop = v4l2_subdev_state_get_crop(sd_state,
>>>> +					     MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +
>>>> +	sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>> +	sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>> +	sink_fmt->field = V4L2_FIELD_NONE;
>>>> +	sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>> You should initialize the colorspace fields too. Same below.
>>>
>>>> +
>>>> +	*v4l2_subdev_state_get_format(sd_state,
>>>> +			      MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
>>>> +
>>>> +	src_fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>> +	src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>> +	src_fmt->field = V4L2_FIELD_NONE;
>>>> +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>> +
>>>> +	in_crop->top = 0;
>>>> +	in_crop->left = 0;
>>>> +	in_crop->width = MALI_C55_DEFAULT_WIDTH;
>>>> +	in_crop->height = MALI_C55_DEFAULT_HEIGHT;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
>>>> +	.init_state = mali_c55_isp_init_state,
>>>> +};
>>>> +
>>>> +static const struct media_entity_operations mali_c55_isp_media_ops = {
>>>> +	.link_validate		= v4l2_subdev_link_validate,
>>> 	.link_validate = v4l2_subdev_link_validate,
>>>
>>> to match mali_c55_isp_internal_ops.
>>>
>>>> +};
>>>> +
>>>> +static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
>>>> +				       struct v4l2_subdev *subdev,
>>>> +				       struct v4l2_async_connection *asc)
>>>> +{
>>>> +	struct mali_c55_isp *isp = container_of(notifier,
>>>> +						struct mali_c55_isp, notifier);
>>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>>>> +	struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
>>>> +	int ret;
>>>> +
>>>> +	/*
>>>> +	 * By default we'll flag this link enabled and the TPG disabled, but
>>>> +	 * no immutable flag because we need to be able to switch between the
>>>> +	 * two.
>>>> +	 */
>>>> +	ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
>>>> +					      MEDIA_LNK_FL_ENABLED);
>>>> +	if (ret)
>>>> +		dev_err(mali_c55->dev, "failed to create link for %s\n",
>>>> +			subdev->name);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
>>>> +{
>>>> +	struct mali_c55_isp *isp = container_of(notifier,
>>>> +						struct mali_c55_isp, notifier);
>>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>>>> +
>>>> +	return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
>>>> +}
>>>> +
>>>> +static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
>>>> +	.bound = mali_c55_isp_notifier_bound,
>>>> +	.complete = mali_c55_isp_notifier_complete,
>>>> +};
>>>> +
>>>> +static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>>>> +	struct v4l2_async_connection *asc;
>>>> +	struct fwnode_handle *ep;
>>>> +	int ret;
>>>> +
>>>> +	v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
>>>> +
>>>> +	/*
>>>> +	 * The ISP should have a single endpoint pointing to some flavour of
>>>> +	 * CSI-2 receiver...but for now at least we do want everything to work
>>>> +	 * normally even with no sensors connected, as we have the TPG. If we
>>>> +	 * don't find a sensor just warn and return success.
>>>> +	 */
>>>> +	ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
>>>> +					     0, 0, 0);
>>>> +	if (!ep) {
>>>> +		dev_warn(mali_c55->dev, "no local endpoint found\n");
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
>>>> +					      struct v4l2_async_connection);
>>>> +	if (IS_ERR(asc)) {
>>>> +		dev_err(mali_c55->dev, "failed to add remote fwnode\n");
>>>> +		ret = PTR_ERR(asc);
>>>> +		goto err_put_ep;
>>>> +	}
>>>> +
>>>> +	isp->notifier.ops = &mali_c55_isp_notifier_ops;
>>>> +	ret = v4l2_async_nf_register(&isp->notifier);
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev, "failed to register notifier\n");
>>>> +		goto err_cleanup_nf;
>>>> +	}
>>>> +
>>>> +	fwnode_handle_put(ep);
>>>> +
>>>> +	return 0;
>>>> +
>>>> +err_cleanup_nf:
>>>> +	v4l2_async_nf_cleanup(&isp->notifier);
>>>> +err_put_ep:
>>>> +	fwnode_handle_put(ep);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>>>> +	struct v4l2_subdev *sd = &isp->sd;
>>>> +	int ret;
>>>> +
>>>> +	isp->mali_c55 = mali_c55;
>>>> +
>>>> +	v4l2_subdev_init(sd, &mali_c55_isp_ops);
>>>> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
>>>> +	sd->entity.ops = &mali_c55_isp_media_ops;
>>>> +	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
>>>> +	sd->internal_ops = &mali_c55_isp_internal_ops;
>>>> +	strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
>>>> +
>>>> +	isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
>>> The MUST_CONNECT flag would make sense here.
>>>
>>>> +	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>>>> +	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
>>>> +
>>>> +	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
>>>> +				     isp->pads);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	ret = v4l2_subdev_init_finalize(sd);
>>>> +	if (ret)
>>>> +		goto err_cleanup_media_entity;
>>>> +
>>>> +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>>> +	if (ret)
>>>> +		goto err_cleanup_subdev;
>>>> +
>>>> +	ret = mali_c55_isp_parse_endpoint(isp);
>>>> +	if (ret)
>>>> +		goto err_cleanup_subdev;
>>> As noted elsewhere, I think this belongs to mali-c55-core.c.
>>>
>>>> +
>>>> +	mutex_init(&isp->lock);
>>> This lock is used in mali-c55-capture.c only, that seems weird.
>> It's because we have two separate capture devices, who's start/stop streaming path calls into the
>> ISP subdevice's start streaming function, but has to do it after a bunch of other writes to the cap
>> device or resizer specific sections...the intention was to delay doing any of that until the ISP was
>> in a known state.
> OK. Naming it capture_lock or something similar, with a comment, would
> help.


Ack

>>>> +
>>>> +	return 0;
>>>> +
>>>> +err_cleanup_subdev:
>>>> +	v4l2_subdev_cleanup(sd);
>>>> +err_cleanup_media_entity:
>>>> +	media_entity_cleanup(&sd->entity);
>>>> +	isp->mali_c55 = NULL;
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>>>> +
>>>> +	if (!isp->mali_c55)
>>>> +		return;
>>>> +
>>>> +	mutex_destroy(&isp->lock);
>>>> +	v4l2_async_nf_unregister(&isp->notifier);
>>>> +	v4l2_async_nf_cleanup(&isp->notifier);
>>>> +	v4l2_device_unregister_subdev(&isp->sd);
>>>> +	v4l2_subdev_cleanup(&isp->sd);
>>>> +	media_entity_cleanup(&isp->sd.entity);
>>>> +}
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>> new file mode 100644
>>>> index 000000000000..cb27abde2aa5
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>> @@ -0,0 +1,258 @@
>>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Register definitions
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#ifndef _MALI_C55_REGISTERS_H
>>>> +#define _MALI_C55_REGISTERS_H
>>>> +
>>>> +#include <linux/bits.h>
>>>> +
>>>> +/* ISP Common 0x00000 - 0x000ff */
>>>> +
>>>> +#define MALI_C55_REG_API				0x00000
>>>> +#define MALI_C55_REG_PRODUCT				0x00004
>>>> +#define MALI_C55_REG_VERSION				0x00008
>>>> +#define MALI_C55_REG_REVISION				0x0000c
>>>> +#define MALI_C55_REG_PULSE_MODE				0x0003c
>>>> +#define MALI_C55_REG_INPUT_MODE_REQUEST			0x0009c
>>>> +#define MALI_C55_INPUT_SAFE_STOP			0x00
>>>> +#define MALI_C55_INPUT_SAFE_START			0x01
>>>> +#define MALI_C55_REG_MODE_STATUS			0x000a0
>>>> +#define MALI_C55_REG_INTERRUPT_MASK_VECTOR		0x00030
>>>> +#define MALI_C55_INTERRUPT_MASK_ALL			GENMASK(31, 0)
>>>> +
>>>> +#define MALI_C55_REG_GLOBAL_MONITOR			0x00050
>>>> +
>>>> +#define MALI_C55_REG_GEN_VIDEO				0x00080
>>>> +#define MALI_C55_REG_GEN_VIDEO_ON_MASK			BIT(0)
>>>> +#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK		BIT(1)
>>>> +#define MALI_C55_REG_GEN_PREFETCH_MASK			GENMASK(31, 16)
>>>> +
>>>> +#define MALI_C55_REG_MCU_CONFIG				0x00020
>>>> +#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK		BIT(0)
>>> #define MALI_C55_REG_MCU_CONFIG_OVERRIDE		BIT(0)
>>>
>>> Same in other places where applicable.
>>>
>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK		BIT(1)
>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PING		BIT(1)
>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG		0x00
>>>> +#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK		BIT(8)
>>>> +#define MALI_C55_REG_PING_PONG_READ			0x00024
>>>> +#define MALI_C55_REG_PING_PONG_READ_MASK		BIT(2)
>>>> +
>>>> +#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR		0x00034
>>>> +#define MALI_C55_REG_INTERRUPT_CLEAR			0x00040
>>>> +#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR		0x00044
>>>> +#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS		0x00068
>>>> +#define MALI_C55_GPS_PONG_FITTED			BIT(0)
>>>> +#define MALI_C55_GPS_WDR_FITTED				BIT(1)
>>>> +#define MALI_C55_GPS_COMPRESSION_FITTED			BIT(2)
>>>> +#define MALI_C55_GPS_TEMPER_FITTED			BIT(3)
>>>> +#define MALI_C55_GPS_SINTER_LITE_FITTED			BIT(4)
>>>> +#define MALI_C55_GPS_SINTER_FITTED			BIT(5)
>>>> +#define MALI_C55_GPS_IRIDIX_LTM_FITTED			BIT(6)
>>>> +#define MALI_C55_GPS_IRIDIX_GTM_FITTED			BIT(7)
>>>> +#define MALI_C55_GPS_CNR_FITTED				BIT(8)
>>>> +#define MALI_C55_GPS_FRSCALER_FITTED			BIT(9)
>>>> +#define MALI_C55_GPS_DS_PIPE_FITTED			BIT(10)
>>>> +
>>>> +#define MALI_C55_REG_BLANKING				0x00084
>>>> +#define MALI_C55_REG_HBLANK_MASK			GENMASK(15, 0)
>>>> +#define MALI_C55_REG_VBLANK_MASK			GENMASK(31, 16)
>>>> +
>>>> +#define MALI_C55_REG_HC_START				0x00088
>>>> +#define MALI_C55_HC_START(h)				(((h) & 0xffff) << 16)
>>>> +#define MALI_C55_REG_HC_SIZE				0x0008c
>>>> +#define MALI_C55_HC_SIZE(h)				((h) & 0xffff)
>>>> +#define MALI_C55_REG_VC_START_SIZE			0x00094
>>>> +#define MALI_C55_VC_START(v)				((v) & 0xffff)
>>>> +#define MALI_C55_VC_SIZE(v)				(((v) & 0xffff) << 16)
>>>> +
>>>> +/* Ping/Pong Configuration Space */
>>>> +#define MALI_C55_REG_BASE_ADDR				0x18e88
>>>> +#define MALI_C55_REG_BYPASS_0				0x18eac
>>>> +#define MALI_C55_REG_BYPASS_0_VIDEO_TEST		BIT(0)
>>>> +#define MALI_C55_REG_BYPASS_0_INPUT_FMT			BIT(1)
>>>> +#define MALI_C55_REG_BYPASS_0_DECOMPANDER		BIT(2)
>>>> +#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR		BIT(3)
>>>> +#define MALI_C55_REG_BYPASS_0_GAIN_WDR			BIT(4)
>>>> +#define MALI_C55_REG_BYPASS_0_FRAME_STITCH		BIT(5)
>>>> +#define MALI_C55_REG_BYPASS_1				0x18eb0
>>>> +#define MALI_C55_REG_BYPASS_1_DIGI_GAIN			BIT(0)
>>>> +#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS		BIT(1)
>>>> +#define MALI_C55_REG_BYPASS_1_FE_SQRT			BIT(2)
>>>> +#define MALI_C55_REG_BYPASS_1_RAW_FE			BIT(3)
>>>> +#define MALI_C55_REG_BYPASS_2				0x18eb8
>>>> +#define MALI_C55_REG_BYPASS_2_SINTER			BIT(0)
>>>> +#define MALI_C55_REG_BYPASS_2_TEMPER			BIT(1)
>>>> +#define MALI_C55_REG_BYPASS_3				0x18ebc
>>>> +#define MALI_C55_REG_BYPASS_3_SQUARE_BE			BIT(0)
>>>> +#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH	BIT(1)
>>>> +#define MALI_C55_REG_BYPASS_3_MESH_SHADING		BIT(3)
>>>> +#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE		BIT(4)
>>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX			BIT(5)
>>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN		BIT(6)
>>>> +#define MALI_C55_REG_BYPASS_4				0x18ec0
>>>> +#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB		BIT(1)
>>>> +#define MALI_C55_REG_BYPASS_4_PF_CORRECTION		BIT(3)
>>>> +#define MALI_C55_REG_BYPASS_4_CCM			BIT(4)
>>>> +#define MALI_C55_REG_BYPASS_4_CNR			BIT(5)
>>>> +#define MALI_C55_REG_FR_BYPASS				0x18ec4
>>>> +#define MALI_C55_REG_DS_BYPASS				0x18ec8
>>>> +#define MALI_C55_BYPASS_CROP				BIT(0)
>>>> +#define MALI_C55_BYPASS_SCALER				BIT(1)
>>>> +#define MALI_C55_BYPASS_GAMMA_RGB			BIT(2)
>>>> +#define MALI_C55_BYPASS_SHARPEN				BIT(3)
>>>> +#define MALI_C55_BYPASS_CS_CONV				BIT(4)
>>>> +#define MALI_C55_REG_ISP_RAW_BYPASS			0x18ecc
>>>> +#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK		BIT(0)
>>>> +#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK		GENMASK(9, 8)
>>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS		2
>>>> +#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS		1
>>>> +#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE		BIT(1)
>>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS		BIT(0)
>>>> +
>>>> +#define MALI_C55_REG_ACTIVE_WIDTH_MASK			0xffff
>>>> +#define MALI_C55_REG_ACTIVE_HEIGHT_MASK			0xffff0000
>>>> +#define MALI_C55_REG_BAYER_ORDER			0x18e8c
>>>> +#define MALI_C55_BAYER_ORDER_MASK			GENMASK(1, 0)
>>>> +#define MALI_C55_REG_TPG_CH0				0x18ed8
>>>> +#define MALI_C55_TEST_PATTERN_ON_OFF			BIT(0)
>>>> +#define MALI_C55_TEST_PATTERN_RGB_MASK			BIT(1)
>>>> +#define MALI_C55_REG_TPG_R_BACKGROUND			0x18ee0
>>>> +#define MALI_C55_REG_TPG_G_BACKGROUND			0x18ee4
>>>> +#define MALI_C55_REG_TPG_B_BACKGROUND			0x18ee8
>>>> +#define MALI_C55_TPG_BACKGROUND_MAX			0xfffff
>>>> +#define MALI_C55_REG_INPUT_WIDTH			0x18f98
>>>> +#define MALI_C55_INPUT_WIDTH_MASK			GENMASK(18, 16)
>>>> +#define MALI_C55_INPUT_WIDTH_8BIT			0
>>>> +#define MALI_C55_INPUT_WIDTH_10BIT			1
>>>> +#define MALI_C55_INPUT_WIDTH_12BIT			2
>>>> +#define MALI_C55_INPUT_WIDTH_14BIT			3
>>>> +#define MALI_C55_INPUT_WIDTH_16BIT			4
>>>> +#define MALI_C55_INPUT_WIDTH_20BIT			5
>>>> +#define MALI_C55_REG_SPACE_SIZE				0x4000
>>>> +#define MALI_C55_REG_CONFIG_SPACES_OFFSET		0x0ab6c
>>>> +#define MALI_C55_CONFIG_SPACE_SIZE			0x1231c
>>>> +
>>>> +#define MALI_C55_REG_SINTER_CONFIG			0x19348
>>>> +#define MALI_C55_SINTER_VIEW_FILTER_MASK		GENMASK(1, 0)
>>>> +#define MALI_C55_SINTER_SCALE_MODE_MASK			GENMASK(3, 2)
>>>> +#define MALI_C55_SINTER_ENABLE_MASK			BIT(4)
>>>> +#define MALI_C55_SINTER_FILTER_SELECT_MASK		BIT(5)
>>>> +#define MALI_C55_SINTER_INT_SELECT_MASK			BIT(6)
>>>> +#define MALI_C55_SINTER_RM_ENABLE_MASK			BIT(7)
>>>> +
>>>> +/* Colour Correction Matrix Configuration */
>>>> +#define MALI_C55_REG_CCM_ENABLE				0x1b07c
>>>> +#define MALI_C55_CCM_ENABLE_MASK			BIT(0)
>>>> +#define MALI_C55_REG_CCM_COEF_R_R			0x1b080
>>>> +#define MALI_C55_REG_CCM_COEF_R_G			0x1b084
>>>> +#define MALI_C55_REG_CCM_COEF_R_B			0x1b088
>>>> +#define MALI_C55_REG_CCM_COEF_G_R			0x1b090
>>>> +#define MALI_C55_REG_CCM_COEF_G_G			0x1b094
>>>> +#define MALI_C55_REG_CCM_COEF_G_B			0x1b098
>>>> +#define MALI_C55_REG_CCM_COEF_B_R			0x1b0a0
>>>> +#define MALI_C55_REG_CCM_COEF_B_G			0x1b0a4
>>>> +#define MALI_C55_REG_CCM_COEF_B_B			0x1b0a8
>>>> +#define MALI_C55_CCM_COEF_MASK				GENMASK(12, 0)
>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R			0x1b0b0
>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G			0x1b0b4
>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B			0x1b0b8
>>>> +#define MALI_C55_CCM_ANTIFOG_GAIN_MASK			GENMASK(11, 0)
>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R		0x1b0c0
>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G		0x1b0c4
>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B		0x1b0c8
>>>> +#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK		GENMASK(11, 0)
>>>> +
>>>> +/*
>>>> + * The Mali-C55 ISP has up to two output pipes; known as full resolution and
>>>> + * down scaled. The register space for these is laid out identically, but offset
>>>> + * by 372 bytes.
>>>> + */
>>>> +#define MALI_C55_CAP_DEV_FR_REG_OFFSET		0x0
>>>> +#define MALI_C55_CAP_DEV_DS_REG_OFFSET		0x174
>>>> +
>>>> +#define MALI_C55_REG_CS_CONV_CONFIG(offset)		(0x1c098 + (offset))
>>>> +#define MALI_C55_CS_CONV_MATRIX_MASK			BIT(0)
>>>> +#define MALI_C55_CS_CONV_FILTER_MASK			BIT(1)
>>>> +#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK		BIT(2)
>>>> +#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK		BIT(3)
>>>> +#define MALI_C55_REG_Y_WRITER_MODE(offset)		(0x1c0ec + (offset))
>>>> +#define MALI_C55_REG_UV_WRITER_MODE(offset)		(0x1c144 + (offset))
>>>> +#define MALI_C55_WRITER_MODE_MASK			GENMASK(4, 0)
>>>> +#define MALI_C55_WRITER_SUBMODE_MASK			GENMASK(7, 6)
>>>> +#define MALI_C55_WRITER_FRAME_WRITE_MASK		BIT(9)
>>>> +#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset)		(0x1c0f0 + (offset))
>>>> +#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset)		(0x1c148 + (offset))
>>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)		((w) << 0)
>>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)		((h) << 16)
>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset)	(0x1c0f4 + (offset))
>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset)	(0x1c108 + (offset))
>>>> +#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_RESTART		BIT(3)
>>>> +#define MALI_C55_REG_Y_WRITER_OFFSET(offset)		(0x1c10c + (offset))
>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset)	(0x1c14c + (offset))
>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset)	(0x1c160 + (offset))
>>>> +#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_RESTART		BIT(3)
>>>> +#define MALI_C55_REG_UV_WRITER_OFFSET(offset)		(0x1c164 + (offset))
>>>> +
>>>> +#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
>>>> +#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE		0x18edc
>>>> +
>>>> +#define MALI_C55_REG_CROP_EN(offset)			(0x1c028 + (offset))
>>>> +#define MALI_C55_CROP_ENABLE				BIT(0)
>>>> +#define MALI_C55_REG_CROP_X_START(offset)		(0x1c02c + (offset))
>>>> +#define MALI_C55_REG_CROP_Y_START(offset)		(0x1c030 + (offset))
>>>> +#define MALI_C55_REG_CROP_X_SIZE(offset)		(0x1c034 + (offset))
>>>> +#define MALI_C55_REG_CROP_Y_SIZE(offset)		(0x1c038 + (offset))
>>>> +#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset)		(0x1c040 + (offset))
>>>> +#define MALI_C55_SCALER_TIMEOUT_EN			BIT(4)
>>>> +#define MALI_C55_SCALER_TIMEOUT(t)			((t) << 16)
>>>> +#define MALI_C55_REG_SCALER_IN_WIDTH(offset)		(0x1c044 + (offset))
>>>> +#define MALI_C55_REG_SCALER_IN_HEIGHT(offset)		(0x1c048 + (offset))
>>>> +#define MALI_C55_REG_SCALER_OUT_WIDTH(offset)		(0x1c04c + (offset))
>>>> +#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset)		(0x1c050 + (offset))
>>>> +#define MALI_C55_REG_SCALER_HFILT_TINC(offset)		(0x1c054 + (offset))
>>>> +#define MALI_C55_REG_SCALER_HFILT_COEF(offset)		(0x1c058 + (offset))
>>>> +#define MALI_C55_REG_SCALER_VFILT_TINC(offset)		(0x1c05c + (offset))
>>>> +#define MALI_C55_REG_SCALER_VFILT_COEF(offset)		(0x1c060 + (offset))
>>>> +
>>>> +#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset)		(0x1c064 + (offset))
>>>> +#define MALI_C55_GAMMA_ENABLE_MASK			BIT(0)
>>>> +#define MALI_C55_REG_GAMMA_GAINS_1(offset)		(0x1c068 + (offset))
>>>> +#define MALI_C55_GAMMA_GAIN_R_MASK			GENMASK(11, 0)
>>>> +#define MALI_C55_GAMMA_GAIN_G_MASK			GENMASK(27, 16)
>>>> +#define MALI_C55_REG_GAMMA_GAINS_2(offset)		(0x1c06c + (offset))
>>>> +#define MALI_C55_GAMMA_GAIN_B_MASK			GENMASK(11, 0)
>>>> +#define MALI_C55_REG_GAMMA_OFFSETS_1(offset)		(0x1c070 + (offset))
>>>> +#define MALI_C55_GAMMA_OFFSET_R_MASK			GENMASK(11, 0)
>>>> +#define MALI_C55_GAMMA_OFFSET_G_MASK			GENMASK(27, 16)
>>>> +#define MALI_C55_REG_GAMMA_OFFSETS_2(offset)		(0x1c074 + (offset))
>>>> +#define MALI_C55_GAMMA_OFFSET_B_MASK			GENMASK(11, 0)
>>>> +
>>>> +/* Output DMA Writer */
>>>> +
>>>> +#define MALI_C55_OUTPUT_DISABLED		0
>>>> +#define MALI_C55_OUTPUT_RGB32			1
>>>> +#define MALI_C55_OUTPUT_A2R10G10B10		2
>>>> +#define MALI_C55_OUTPUT_RGB565			3
>>>> +#define MALI_C55_OUTPUT_RGB24			4
>>>> +#define MALI_C55_OUTPUT_GEN32			5
>>>> +#define MALI_C55_OUTPUT_RAW16			6
>>>> +#define MALI_C55_OUTPUT_AYUV			8
>>>> +#define MALI_C55_OUTPUT_Y410			9
>>>> +#define MALI_C55_OUTPUT_YUY2			10
>>>> +#define MALI_C55_OUTPUT_UYVY			11
>>>> +#define MALI_C55_OUTPUT_Y210			12
>>>> +#define MALI_C55_OUTPUT_NV12_21			13
>>>> +#define MALI_C55_OUTPUT_YUV_420_422		17
>>>> +#define MALI_C55_OUTPUT_P210_P010		19
>>>> +#define MALI_C55_OUTPUT_YUV422			20
>>> I'd define this just below the MALI_C55_REG_UV_WRITER_MODE register
>>> macro.
>>>
>>>> +
>>>> +#define MALI_C55_OUTPUT_PLANE_ALT0		0
>>>> +#define MALI_C55_OUTPUT_PLANE_ALT1		1
>>>> +#define MALI_C55_OUTPUT_PLANE_ALT2		2
>>> Same here ?
>>>
>>>> +
>>>> +#endif /* _MALI_C55_REGISTERS_H */
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>> new file mode 100644
>>>> index 000000000000..8edae87f1e5f
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>> @@ -0,0 +1,382 @@
>>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Resizer Coefficients
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#ifndef _MALI_C55_RESIZER_COEFS_H
>>>> +#define _MALI_C55_RESIZER_COEFS_H
>>>> +
>>>> +#include "mali-c55-common.h"
>>>> +
>>>> +#define MALI_C55_RESIZER_COEFS_NUM_BANKS	8
>>>> +#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES	64
>>> Do these belongs to mali-c55-registers.h ?
>>>
>>>> +
>>>> +static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
>>>> +	{	/* Bank 0 */
>>>> +		0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
>>>> +		0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
>>>> +		0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
>>>> +		0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
>>>> +		0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
>>>> +		0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
>>>> +		0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
>>>> +		0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
>>>> +		0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
>>>> +		0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
>>>> +		0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
>>>> +		0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
>>>> +		0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
>>>> +		0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
>>>> +		0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
>>>> +		0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
>>>> +	},
>>>> +	{	/* Bank 1 */
>>>> +		0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
>>>> +		0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
>>>> +		0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
>>>> +		0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
>>>> +		0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
>>>> +		0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
>>>> +		0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
>>>> +		0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
>>>> +		0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
>>>> +		0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
>>>> +		0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
>>>> +		0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
>>>> +		0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
>>>> +		0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
>>>> +		0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
>>>> +		0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
>>>> +	},
>>>> +	{	/* Bank 2 */
>>>> +		0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
>>>> +		0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
>>>> +		0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
>>>> +		0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
>>>> +		0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
>>>> +		0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
>>>> +		0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
>>>> +		0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
>>>> +		0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
>>>> +		0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
>>>> +		0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
>>>> +		0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
>>>> +		0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
>>>> +		0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
>>>> +		0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
>>>> +		0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
>>>> +	},
>>>> +	{	/* Bank 3 */
>>>> +		0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
>>>> +		0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
>>>> +		0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
>>>> +		0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
>>>> +		0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
>>>> +		0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
>>>> +		0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
>>>> +		0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
>>>> +		0x20100000, 0x00000010, 0x1f110000, 0x00000010,
>>>> +		0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
>>>> +		0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
>>>> +		0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
>>>> +		0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
>>>> +		0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
>>>> +		0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
>>>> +		0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
>>>> +	},
>>>> +	{	/* Bank 4 */
>>>> +		0x17090000, 0x00000917, 0x18090000, 0x00000916,
>>>> +		0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
>>>> +		0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
>>>> +		0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
>>>> +		0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
>>>> +		0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
>>>> +		0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
>>>> +		0x190f0300, 0x00000411, 0x18100300, 0x00000411,
>>>> +		0x1a100300, 0x00000310, 0x18110400, 0x00000310,
>>>> +		0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
>>>> +		0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
>>>> +		0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
>>>> +		0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
>>>> +		0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
>>>> +		0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
>>>> +		0x17160800, 0x0000010a, 0x18160900, 0x00000009,
>>>> +	},
>>>> +	{	/* Bank 5 */
>>>> +		0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
>>>> +		0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
>>>> +		0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
>>>> +		0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
>>>> +		0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
>>>> +		0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
>>>> +		0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
>>>> +		0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
>>>> +		0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
>>>> +		0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
>>>> +		0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
>>>> +		0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
>>>> +		0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
>>>> +		0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
>>>> +		0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
>>>> +		0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
>>>> +	},
>>>> +	{	/* Bank 6 */
>>>> +		0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
>>>> +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
>>>> +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
>>>> +		0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
>>>> +		0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
>>>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>> +		0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
>>>> +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>> +		0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
>>>> +		0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
>>>> +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
>>>> +		0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>>> +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>>> +	},
>>>> +	{	/* Bank 7 */
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +	}
>>>> +};
>>>> +
>>>> +static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
>>>> +	{	/* Bank 0 */
>>>> +		0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
>>>> +		0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
>>>> +		0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
>>>> +		0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
>>>> +		0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
>>>> +		0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
>>>> +		0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
>>>> +		0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
>>>> +		0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
>>>> +		0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
>>>> +		0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
>>>> +		0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
>>>> +		0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
>>>> +		0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
>>>> +		0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
>>>> +		0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
>>>> +	},
>>>> +	{	/* Bank 1 */
>>>> +		0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
>>>> +		0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
>>>> +		0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
>>>> +		0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
>>>> +		0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
>>>> +		0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
>>>> +		0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
>>>> +		0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
>>>> +		0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
>>>> +		0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
>>>> +		0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
>>>> +		0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
>>>> +		0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
>>>> +		0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
>>>> +		0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
>>>> +		0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
>>>> +	},
>>>> +	{	/* Bank 2 */
>>>> +		0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
>>>> +		0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
>>>> +		0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
>>>> +		0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
>>>> +		0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
>>>> +		0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
>>>> +		0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
>>>> +		0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
>>>> +		0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
>>>> +		0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
>>>> +		0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
>>>> +		0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
>>>> +		0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
>>>> +		0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
>>>> +		0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
>>>> +		0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
>>>> +	},
>>>> +	{	/* Bank 3 */
>>>> +		0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
>>>> +		0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
>>>> +		0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
>>>> +		0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
>>>> +		0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
>>>> +		0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
>>>> +		0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
>>>> +		0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
>>>> +		0x20100000, 0x00000010, 0x1f100000, 0x00000011,
>>>> +		0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
>>>> +		0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
>>>> +		0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
>>>> +		0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
>>>> +		0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
>>>> +		0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
>>>> +		0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
>>>> +	},
>>>> +	{	/* Bank 4 */
>>>> +		0x17170900, 0x00000009, 0x18160900, 0x00000009,
>>>> +		0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
>>>> +		0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
>>>> +		0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
>>>> +		0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
>>>> +		0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
>>>> +		0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
>>>> +		0x19110400, 0x0000030f, 0x18110400, 0x00000310,
>>>> +		0x1a100300, 0x00000310, 0x18100300, 0x00000411,
>>>> +		0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
>>>> +		0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
>>>> +		0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
>>>> +		0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
>>>> +		0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
>>>> +		0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
>>>> +		0x170a0100, 0x00000816, 0x18090000, 0x00000916,
>>>> +	},
>>>> +	{	/* Bank 5 */
>>>> +		0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
>>>> +		0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
>>>> +		0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
>>>> +		0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
>>>> +		0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
>>>> +		0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
>>>> +		0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
>>>> +		0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
>>>> +		0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
>>>> +		0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
>>>> +		0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
>>>> +		0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
>>>> +		0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
>>>> +		0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
>>>> +		0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
>>>> +		0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
>>>> +	},
>>>> +	{	/* Bank 6 */
>>>> +		0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
>>>> +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>>> +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
>>>> +		0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
>>>> +		0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
>>>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>> +		0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>>> +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>> +		0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
>>>> +		0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
>>>> +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
>>>> +		0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
>>>> +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
>>>> +	},
>>>> +	{	/* Bank 7 */
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +	}
>>>> +};
>>>> +
>>>> +struct mali_c55_resizer_coef_bank {
>>>> +	unsigned int bank;
>>> This is always equal to the index of the entry in the
>>> mali_c55_coefficient_banks array, you can drop it.
>>>
>>>> +	unsigned int top;
>>>> +	unsigned int bottom;
>>> The bottom value of bank N is always equal to the top value of bank N+1.
>>> You can simplify this by storing a single value.
>>>
>>>> +};
>>>> +
>>>> +static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
>>>> +	{
>>>> +		.bank = 0,
>>>> +		.top = 1000,
>>>> +		.bottom = 770,
>>>> +	},
>>>> +	{
>>>> +		.bank = 1,
>>>> +		.top = 769,
>>>> +		.bottom = 600,
>>>> +	},
>>>> +	{
>>>> +		.bank = 2,
>>>> +		.top = 599,
>>>> +		.bottom = 460,
>>>> +	},
>>>> +	{
>>>> +		.bank = 3,
>>>> +		.top = 459,
>>>> +		.bottom = 354,
>>>> +	},
>>>> +	{
>>>> +		.bank = 4,
>>>> +		.top = 353,
>>>> +		.bottom = 273,
>>>> +	},
>>>> +	{
>>>> +		.bank = 5,
>>>> +		.top = 272,
>>>> +		.bottom = 210,
>>>> +	},
>>>> +	{
>>>> +		.bank = 6,
>>>> +		.top = 209,
>>>> +		.bottom = 162,
>>>> +	},
>>>> +	{
>>>> +		.bank = 7,
>>>> +		.top = 161,
>>>> +		.bottom = 125,
>>>> +	},
>>>> +};
>>>> +
>>> A small comment would be nice, such as
>>>
>>> /* Select a bank of resizer coefficients, based on the scaling ratio. */
>>>
>>>> +static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,
>>> This function is related to the resizers. Add "rsz" somewhere in the
>>> function name, and pass a resizer pointer.
>>>
>>>> +						unsigned int crop,
>>>> +						unsigned int scale)
>>> I think those are the input and output sizes to the scaler. Rename them
>>> to make it clearer.
>>>
>>>> +{
>>>> +	unsigned int tmp;
>>> tmp is almost always a bad variable name. Please use a more descriptive
>>> name, size as rsz_ratio.
>>>
>>>> +	unsigned int i;
>>>> +
>>>> +	tmp = (scale * 1000U) / crop;
>>>> +
>>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
>>>> +		if (tmp >= mali_c55_coefficient_banks[i].bottom &&
>>>> +		    tmp <= mali_c55_coefficient_banks[i].top)
>>>> +			return mali_c55_coefficient_banks[i].bank;
>>>> +	}
>>>> +
>>>> +	/*
>>>> +	 * We shouldn't ever get here, in theory. As we have no good choices
>>>> +	 * simply warn the user and use the first bank of coefficients.
>>>> +	 */
>>>> +	dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
>>>> +	return 0;
>>>> +}
>>> And everything else belongs to mali-c55-resizer.c. Drop this header
>>> file.
>>>
>>>> +
>>>> +#endif /* _MALI_C55_RESIZER_COEFS_H */
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>> new file mode 100644
>>>> index 000000000000..0a5a2969d3ce
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>> @@ -0,0 +1,779 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Image signal processor
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#include <linux/math.h>
>>>> +#include <linux/minmax.h>
>>>> +
>>>> +#include <media/media-entity.h>
>>>> +#include <media/v4l2-subdev.h>
>>>> +
>>>> +#include "mali-c55-common.h"
>>>> +#include "mali-c55-registers.h"
>>>> +#include "mali-c55-resizer-coefs.h"
>>>> +
>>>> +/* Scaling factor in Q4.20 format. */
>>>> +#define MALI_C55_RZR_SCALER_FACTOR	(1U << 20)
>>>> +
>>>> +static const u32 rzr_non_bypass_src_fmts[] = {
>>>> +	MEDIA_BUS_FMT_RGB121212_1X36,
>>>> +	MEDIA_BUS_FMT_YUV10_1X30
>>>> +};
>>>> +
>>>> +static const char * const mali_c55_resizer_names[] = {
>>>> +	[MALI_C55_RZR_FR] = "resizer fr",
>>>> +	[MALI_C55_RZR_DS] = "resizer ds",
>>>> +};
>>>> +
>>>> +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
>>>> +				     struct v4l2_subdev_state *state)
>>>> +{
>>>> +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
>>>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>> +	struct v4l2_mbus_framefmt *fmt;
>>>> +	struct v4l2_rect *crop;
>>>> +
>>>> +	/* Verify if crop should be enabled. */
>>>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
>>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
>>>> +
>>>> +	if (fmt->width == crop->width && fmt->height == crop->height)
>>>> +		return MALI_C55_BYPASS_CROP;
>>>> +
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
>>>> +		       crop->left);
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
>>>> +		       crop->top);
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
>>>> +		       crop->width);
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
>>>> +		       crop->height);
>>>> +
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
>>>> +		       MALI_C55_CROP_ENABLE);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
>>>> +					struct v4l2_subdev_state *state)
>>>> +{
>>>> +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
>>>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>> +	struct v4l2_rect *crop, *scale;
>>>> +	unsigned int h_bank, v_bank;
>>>> +	u64 h_scale, v_scale;
>>>> +
>>>> +	/* Verify if scaling should be enabled. */
>>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
>>>> +	scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
>>>> +
>>>> +	if (crop->width == scale->width && crop->height == scale->height)
>>>> +		return MALI_C55_BYPASS_SCALER;
>>>> +
>>>> +	/* Program the V/H scaling factor in Q4.20 format. */
>>>> +	h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
>>>> +	v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
>>>> +
>>>> +	do_div(h_scale, scale->width);
>>>> +	do_div(v_scale, scale->height);
>>>> +
>>>> +	mali_c55_write(mali_c55,
>>>> +		       MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
>>>> +		       crop->width);
>>>> +	mali_c55_write(mali_c55,
>>>> +		       MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
>>>> +		       crop->height);
>>>> +
>>>> +	mali_c55_write(mali_c55,
>>>> +		       MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
>>>> +		       scale->width);
>>>> +	mali_c55_write(mali_c55,
>>>> +		       MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
>>>> +		       scale->height);
>>>> +
>>>> +	mali_c55_write(mali_c55,
>>>> +		       MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
>>>> +		       h_scale);
>>>> +	mali_c55_write(mali_c55,
>>>> +		       MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
>>>> +		       v_scale);
>>>> +
>>>> +	h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
>>>> +					     scale->width);
>>>> +	mali_c55_write(mali_c55,
>>>> +		       MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
>>>> +		       h_bank);
>>>> +
>>>> +	v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
>>>> +					     scale->height);
>>>> +	mali_c55_write(mali_c55,
>>>> +		       MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
>>>> +		       v_bank);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
>>>> +				 struct v4l2_subdev_state *state)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>> +	u32 bypass = 0;
>>>> +
>>>> +	/* Verify if cropping and scaling should be enabled. */
>>>> +	bypass |= mali_c55_rzr_program_crop(rzr, state);
>>>> +	bypass |= mali_c55_rzr_program_resizer(rzr, state);
>>>> +
>>>> +	mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
>>>> +			     MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
>>>> +			     MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
>>>> +			     bypass);
>>>> +}
>>>> +
>>>> +/*
>>>> + * Inspect the routing table to know which of the two (mutually exclusive)
>>>> + * routes is enabled and return the sink pad id of the active route.
>>>> + */
>>>> +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
>>>> +{
>>>> +	struct v4l2_subdev_krouting *routing = &state->routing;
>>>> +	struct v4l2_subdev_route *route;
>>>> +
>>>> +	/* A single route is enabled at a time. */
>>>> +	for_each_active_route(routing, route)
>>>> +		return route->sink_pad;
>>>> +
>>>> +	return MALI_C55_RZR_SINK_PAD;
>>>> +}
>>>> +
>>>> +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
>>>> +{
>>>> +	u32 corrected_code = 0;
>>>> +
>>>> +	/*
>>>> +	 * The ISP takes input in a 20-bit format, but can only output 16-bit
>>>> +	 * RAW bayer data (with the 4 least significant bits from the input
>>>> +	 * being lost). Return the 16-bit version of the 20-bit input formats.
>>>> +	 */
>>>> +	switch (mbus_code) {
>>>> +	case MEDIA_BUS_FMT_SBGGR20_1X20:
>>>> +		corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
>>>> +		break;
>>>> +	case MEDIA_BUS_FMT_SGBRG20_1X20:
>>>> +		corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
>>>> +		break;
>>>> +	case MEDIA_BUS_FMT_SGRBG20_1X20:
>>>> +		corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
>>>> +		break;
>>>> +	case MEDIA_BUS_FMT_SRGGB20_1X20:
>>>> +		corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
>>>> +		break;
>>>> +	}
>>>> +
>>>> +	return corrected_code;
>>>> +}
>>>> +
>>>> +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
>>>> +				      struct v4l2_subdev_state *state,
>>>> +				      struct v4l2_subdev_krouting *routing)
>>>> +{
>>>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>> +						    sd);
>>>> +	unsigned int active_sink = UINT_MAX;
>>>> +	struct v4l2_mbus_framefmt *src_fmt;
>>>> +	struct v4l2_rect *crop, *compose;
>>>> +	struct v4l2_subdev_route *route;
>>>> +	unsigned int active_routes = 0;
>>>> +	struct v4l2_mbus_framefmt *fmt;
>>>> +	int ret;
>>>> +
>>>> +	ret = v4l2_subdev_routing_validate(sd, routing, 0);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	/* Only a single route can be enabled at a time. */
>>>> +	for_each_active_route(routing, route) {
>>>> +		if (++active_routes > 1) {
>>>> +			dev_err(rzr->mali_c55->dev,
>>>> +				"Only one route can be active");
>>>> +			return -EINVAL;
>>>> +		}
>>>> +
>>>> +		active_sink = route->sink_pad;
>>>> +	}
>>>> +	if (active_sink == UINT_MAX) {
>>>> +		dev_err(rzr->mali_c55->dev, "One route has to be active");
>>>> +		return -EINVAL;
>>>> +	}
>>>> +
>>>> +	ret = v4l2_subdev_set_routing(sd, state, routing);
>>>> +	if (ret) {
>>>> +		dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
>>>> +	crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
>>>> +	compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
>>>> +
>>>> +	fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>> +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>> +	fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>> +	fmt->field = V4L2_FIELD_NONE;
>>>> +
>>>> +	if (active_sink == MALI_C55_RZR_SINK_PAD) {
>>>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>> +
>>>> +		crop->left = crop->top = 0;
>>>> +		crop->width = MALI_C55_DEFAULT_WIDTH;
>>>> +		crop->height = MALI_C55_DEFAULT_HEIGHT;
>>>> +
>>>> +		*compose = *crop;
>>>> +	} else {
>>>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>> +	}
>>>> +
>>>> +	/* Propagate the format to the source pad */
>>>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
>>>> +					       0);
>>>> +	*src_fmt = *fmt;
>>>> +
>>>> +	/* In the event this is the bypass pad the mbus code needs correcting */
>>>> +	if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
>>>> +		src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
>>>> +				       struct v4l2_subdev_state *state,
>>>> +				       struct v4l2_subdev_mbus_code_enum *code)
>>>> +{
>>>> +	struct v4l2_mbus_framefmt *sink_fmt;
>>>> +	const struct mali_c55_isp_fmt *fmt;
>>>> +	unsigned int index = 0;
>>>> +	u32 sink_pad;
>>>> +
>>>> +	switch (code->pad) {
>>>> +	case MALI_C55_RZR_SINK_PAD:
>>>> +		if (code->index)
>>>> +			return -EINVAL;
>>>> +
>>>> +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>> +
>>>> +		return 0;
>>>> +	case MALI_C55_RZR_SOURCE_PAD:
>>>> +		sink_pad = mali_c55_rzr_get_active_sink(state);
>>>> +		sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
>>>> +
>>>> +		/*
>>>> +		 * If the active route is from the Bypass sink pad, then the
>>>> +		 * source pad is a simple passthrough of the sink format,
>>>> +		 * downshifted to 16-bits.
>>>> +		 */
>>>> +
>>>> +		if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>>> +			if (code->index)
>>>> +				return -EINVAL;
>>>> +
>>>> +			code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
>>>> +			if (!code->code)
>>>> +				return -EINVAL;
>>>> +
>>>> +			return 0;
>>>> +		}
>>>> +
>>>> +		/*
>>>> +		 * If the active route is from the non-bypass sink then we can
>>>> +		 * select either RGB or conversion to YUV.
>>>> +		 */
>>>> +
>>>> +		if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
>>>> +			return -EINVAL;
>>>> +
>>>> +		code->code = rzr_non_bypass_src_fmts[code->index];
>>>> +
>>>> +		return 0;
>>>> +	case MALI_C55_RZR_SINK_BYPASS_PAD:
>>>> +		for_each_mali_isp_fmt(fmt) {
>>>> +			if (index++ == code->index) {
>>>> +				code->code = fmt->code;
>>>> +				return 0;
>>>> +			}
>>>> +		}
>>>> +
>>>> +		break;
>>>> +	}
>>>> +
>>>> +	return -EINVAL;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
>>>> +					struct v4l2_subdev_state *state,
>>>> +					struct v4l2_subdev_frame_size_enum *fse)
>>>> +{
>>>> +	if (fse->index)
>>>> +		return -EINVAL;
>>>> +
>>>> +	fse->max_width = MALI_C55_MAX_WIDTH;
>>>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
>>>> +	fse->min_width = MALI_C55_MIN_WIDTH;
>>>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
>>>> +				     struct v4l2_subdev_state *state,
>>>> +				     struct v4l2_subdev_format *format)
>>>> +{
>>>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>>>> +	struct v4l2_rect *rect;
>>>> +	unsigned int sink_pad;
>>>> +
>>>> +	/*
>>>> +	 * Clamp to min/max and then reset crop and compose rectangles to the
>>>> +	 * newly applied size.
>>>> +	 */
>>>> +	clamp_t(unsigned int, fmt->width,
>>>> +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>>> +	clamp_t(unsigned int, fmt->height,
>>>> +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>>> +
>>>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
>>>> +	if (sink_pad == MALI_C55_RZR_SINK_PAD) {
>>>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>> +
>>>> +		rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
>>>> +		rect->left = 0;
>>>> +		rect->top = 0;
>>>> +		rect->width = fmt->width;
>>>> +		rect->height = fmt->height;
>>>> +
>>>> +		rect = v4l2_subdev_state_get_compose(state,
>>>> +						     MALI_C55_RZR_SINK_PAD);
>>>> +		rect->left = 0;
>>>> +		rect->top = 0;
>>>> +		rect->width = fmt->width;
>>>> +		rect->height = fmt->height;
>>>> +	} else {
>>>> +		/*
>>>> +		 * Make sure the media bus code is one of the supported
>>>> +		 * ISP input media bus codes.
>>>> +		 */
>>>> +		if (!mali_c55_isp_is_format_supported(fmt->code))
>>>> +			fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
>>>> +	}
>>>> +
>>>> +	*v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
>>>> +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
>>>> +				       struct v4l2_subdev_state *state,
>>>> +				       struct v4l2_subdev_format *format)
>>>> +{
>>>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>> +						    sd);
>>>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>>>> +	struct v4l2_mbus_framefmt *sink_fmt;
>>>> +	struct v4l2_rect *crop, *compose;
>>>> +	unsigned int sink_pad;
>>>> +	unsigned int i;
>>>> +
>>>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
>>>> +	sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
>>>> +	crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
>>>> +	compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
>>>> +
>>>> +	/* FR Bypass pipe. */
>>>> +
>>>> +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>>> +		/*
>>>> +		 * Format on the source pad is the same as the one on the
>>>> +		 * sink pad, downshifted to 16-bits.
>>>> +		 */
>>>> +		fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
>>>> +		if (!fmt->code)
>>>> +			return -EINVAL;
>>>> +
>>>> +		/* RAW bypass disables scaling and cropping. */
>>>> +		crop->top = compose->top = 0;
>>>> +		crop->left = compose->left = 0;
>>>> +		fmt->width = crop->width = compose->width = sink_fmt->width;
>>>> +		fmt->height = crop->height = compose->height = sink_fmt->height;
>>>> +
>>>> +		*v4l2_subdev_state_get_format(state,
>>>> +					      MALI_C55_RZR_SOURCE_PAD) = *fmt;
>>>> +
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	/* Regular processing pipe. */
>>>> +
>>>> +	for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
>>>> +		if (fmt->code == rzr_non_bypass_src_fmts[i])
>>>> +			break;
>>>> +	}
>>>> +
>>>> +	if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
>>>> +		dev_dbg(rzr->mali_c55->dev,
>>>> +			"Unsupported mbus code 0x%x: using default\n",
>>>> +			fmt->code);
>>>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>> +	}
>>>> +
>>>> +	/*
>>>> +	 * The source pad format size comes directly from the sink pad
>>>> +	 * compose rectangle.
>>>> +	 */
>>>> +	fmt->width = compose->width;
>>>> +	fmt->height = compose->height;
>>>> +
>>>> +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
>>>> +				struct v4l2_subdev_state *state,
>>>> +				struct v4l2_subdev_format *format)
>>>> +{
>>>> +	/*
>>>> +	 * On sink pads fmt is either fixed for the 'regular' processing
>>>> +	 * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
>>>> +	 * pad.
>>>> +	 *
>>>> +	 * On source pad sizes are the result of crop+compose on the sink
>>>> +	 * pad sizes, while the format depends on the active route.
>>>> +	 */
>>>> +
>>>> +	if (format->pad != MALI_C55_RZR_SOURCE_PAD)
>>>> +		return mali_c55_rzr_set_sink_fmt(sd, state, format);
>>>> +
>>>> +	return mali_c55_rzr_set_source_fmt(sd, state, format);
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
>>>> +				      struct v4l2_subdev_state *state,
>>>> +				      struct v4l2_subdev_selection *sel)
>>>> +{
>>>> +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
>>>> +		return -EINVAL;
>>>> +
>>>> +	if (sel->target != V4L2_SEL_TGT_CROP &&
>>>> +	    sel->target != V4L2_SEL_TGT_COMPOSE)
>>>> +		return -EINVAL;
>>>> +
>>>> +	sel->r = sel->target == V4L2_SEL_TGT_CROP
>>>> +	       ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
>>>> +	       : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
>>>> +				      struct v4l2_subdev_state *state,
>>>> +				      struct v4l2_subdev_selection *sel)
>>>> +{
>>>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>> +						    sd);
>>>> +	struct v4l2_mbus_framefmt *source_fmt;
>>>> +	struct v4l2_mbus_framefmt *sink_fmt;
>>>> +	struct v4l2_rect *crop, *compose;
>>>> +
>>>> +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
>>>> +		return -EINVAL;
>>>> +
>>>> +	if (sel->target != V4L2_SEL_TGT_CROP &&
>>>> +	    sel->target != V4L2_SEL_TGT_COMPOSE)
>>>> +		return -EINVAL;
>>>> +
>>>> +	source_fmt = v4l2_subdev_state_get_format(state,
>>>> +						  MALI_C55_RZR_SOURCE_PAD);
>>>> +	sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
>>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
>>>> +	compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
>>>> +
>>>> +	/* RAW bypass disables crop/scaling. */
>>>> +	if (mali_c55_format_is_raw(source_fmt->code)) {
>>>> +		crop->top = compose->top = 0;
>>>> +		crop->left = compose->left = 0;
>>>> +		crop->width = compose->width = sink_fmt->width;
>>>> +		crop->height = compose->height = sink_fmt->height;
>>>> +
>>>> +		sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
>>>> +
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	/* During streaming, it is allowed to only change the crop rectangle. */
>>>> +	if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
>>>> +		return -EINVAL;
>>>> +
>>>> +	 /*
>>>> +	  * Update the desired target and then clamp the crop rectangle to the
>>>> +	  * sink format sizes and the compose size to the crop sizes.
>>>> +	  */
>>>> +	if (sel->target == V4L2_SEL_TGT_CROP)
>>>> +		*crop = sel->r;
>>>> +	else
>>>> +		*compose = sel->r;
>>>> +
>>>> +	clamp_t(unsigned int, crop->left, 0,  sink_fmt->width);
>>>> +	clamp_t(unsigned int, crop->top, 0,  sink_fmt->height);
>>>> +	clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
>>>> +		sink_fmt->width - crop->left);
>>>> +	clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
>>>> +		sink_fmt->height - crop->top);
>>>> +
>>>> +	if (rzr->streaming) {
>>>> +		/*
>>>> +		 * Apply at runtime a crop rectangle on the resizer's sink only
>>>> +		 * if it doesn't require re-programming the scaler output sizes
>>>> +		 * as it would require changing the output buffer sizes as well.
>>>> +		 */
>>>> +		if (sel->r.width < compose->width ||
>>>> +		    sel->r.height < compose->height)
>>>> +			return -EINVAL;
>>>> +
>>>> +		*crop = sel->r;
>>>> +		mali_c55_rzr_program(rzr, state);
>>>> +
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	compose->left = 0;
>>>> +	compose->top = 0;
>>>> +	clamp_t(unsigned int, compose->left, 0,  sink_fmt->width);
>>>> +	clamp_t(unsigned int, compose->top, 0,  sink_fmt->height);
>>>> +	clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
>>>> +	clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
>>>> +
>>>> +	sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
>>>> +				    struct v4l2_subdev_state *state,
>>>> +				    enum v4l2_subdev_format_whence which,
>>>> +				    struct v4l2_subdev_krouting *routing)
>>>> +{
>>>> +	if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
>>>> +	    media_entity_is_streaming(&sd->entity))
>>>> +		return -EBUSY;
>>>> +
>>>> +	return __mali_c55_rzr_set_routing(sd, state, routing);
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
>>>> +	.enum_mbus_code		= mali_c55_rzr_enum_mbus_code,
>>>> +	.enum_frame_size	= mali_c55_rzr_enum_frame_size,
>>>> +	.get_fmt		= v4l2_subdev_get_fmt,
>>>> +	.set_fmt		= mali_c55_rzr_set_fmt,
>>>> +	.get_selection		= mali_c55_rzr_get_selection,
>>>> +	.set_selection		= mali_c55_rzr_set_selection,
>>>> +	.set_routing		= mali_c55_rzr_set_routing,
>>>> +};
>>>> +
>>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
>>>> +{
>>>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>> +	struct v4l2_subdev *sd = &rzr->sd;
>>>> +	struct v4l2_subdev_state *state;
>>>> +	unsigned int sink_pad;
>>>> +
>>>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>>>> +
>>>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
>>>> +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>>> +		/* Bypass FR pipe processing if the bypass route is active. */
>>>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>>> +				     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
>>>> +				     MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
>>>> +		goto unlock_state;
>>>> +	}
>>>> +
>>>> +	/* Disable bypass and use regular processing. */
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>>> +			     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
>>>> +	mali_c55_rzr_program(rzr, state);
>>>> +
>>>> +unlock_state:
>>>> +	rzr->streaming = true;
>>>> +	v4l2_subdev_unlock_state(state);
>>>> +}
>>>> +
>>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
>>>> +{
>>>> +	struct v4l2_subdev *sd = &rzr->sd;
>>>> +	struct v4l2_subdev_state *state;
>>>> +
>>>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>>>> +	rzr->streaming = false;
>>>> +	v4l2_subdev_unlock_state(state);
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
>>>> +	.pad	= &mali_c55_resizer_pad_ops,
>>>> +};
>>>> +
>>>> +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
>>>> +				   struct v4l2_subdev_state *state)
>>>> +{
>>>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>> +						    sd);
>>>> +	struct v4l2_subdev_krouting routing = { };
>>>> +	struct v4l2_subdev_route *routes;
>>>> +	unsigned int i;
>>>> +	int ret;
>>>> +
>>>> +	routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
>>>> +	if (!routes)
>>>> +		return -ENOMEM;
>>>> +
>>>> +	for (i = 0; i < rzr->num_routes; ++i) {
>>>> +		struct v4l2_subdev_route *route = &routes[i];
>>>> +
>>>> +		route->sink_pad = i
>>>> +				? MALI_C55_RZR_SINK_BYPASS_PAD
>>>> +				: MALI_C55_RZR_SINK_PAD;
>>>> +		route->source_pad = MALI_C55_RZR_SOURCE_PAD;
>>>> +		if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
>>>> +			route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
>>>> +	}
>>>> +
>>>> +	routing.num_routes = rzr->num_routes;
>>>> +	routing.routes = routes;
>>>> +
>>>> +	ret = __mali_c55_rzr_set_routing(sd, state, &routing);
>>>> +	kfree(routes);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
>>>> +	.init_state = mali_c55_rzr_init_state,
>>>> +};
>>>> +
>>>> +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
>>>> +						  unsigned int index)
>>>> +{
>>>> +	const unsigned int scaler_filt_coefmem_addrs[][2] = {
>>>> +		[MALI_C55_RZR_FR] = {
>>>> +			0x034A8, /* hfilt */
>>>> +			0x044A8  /* vfilt */
>>> Lowercase hex constants.
>>>
>>>> +		},
>>>> +		[MALI_C55_RZR_DS] = {
>>>> +			0x014A8, /* hfilt */
>>>> +			0x024A8  /* vfilt */
>>>> +		},
>>>> +	};
>>>> +	unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
>>>> +	unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
>>>> +	unsigned int i, j;
>>>> +
>>>> +	for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
>>>> +		for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
>>>> +			mali_c55_write(mali_c55, haddr,
>>>> +				mali_c55_scaler_h_filter_coefficients[i][j]);
>>>> +			mali_c55_write(mali_c55, vaddr,
>>>> +				mali_c55_scaler_v_filter_coefficients[i][j]);
>>>> +
>>>> +			haddr += sizeof(u32);
>>>> +			vaddr += sizeof(u32);
>>>> +		}
>>>> +	}
>>>> +}
>>>> +
>>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	unsigned int i;
>>>> +	int ret;
>>>> +
>>>> +	for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
>>>> +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
>>>> +		struct v4l2_subdev *sd = &rzr->sd;
>>>> +		unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
>>>> +
>>>> +		rzr->id = i;
>>>> +		rzr->streaming = false;
>>>> +
>>>> +		if (rzr->id == MALI_C55_RZR_FR)
>>>> +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
>>>> +		else
>>>> +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
>>>> +
>>>> +		mali_c55_resizer_program_coefficients(mali_c55, i);
>>>> +
>>>> +		v4l2_subdev_init(sd, &mali_c55_resizer_ops);
>>>> +		sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
>>>> +			     | V4L2_SUBDEV_FL_STREAMS;
>>>> +		sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
>>>> +		sd->internal_ops = &mali_c55_resizer_internal_ops;
>>>> +		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
>>>> +			 MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
>>>> +
>>>> +		rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
>>>> +		rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
>>>> +
>>>> +		/* Only the FR pipe has a bypass pad. */
>>>> +		if (rzr->id == MALI_C55_RZR_FR) {
>>>> +			rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
>>>> +							MEDIA_PAD_FL_SINK;
>>>> +			rzr->num_routes = 2;
>>>> +		} else {
>>>> +			num_pads -= 1;
>>>> +			rzr->num_routes = 1;
>>>> +		}
>>>> +
>>>> +		ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
>>>> +		if (ret)
>>>> +			return ret;
>>>> +
>>>> +		ret = v4l2_subdev_init_finalize(sd);
>>>> +		if (ret)
>>>> +			goto err_cleanup;
>>>> +
>>>> +		ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>>> +		if (ret)
>>>> +			goto err_cleanup;
>>>> +
>>>> +		rzr->mali_c55 = mali_c55;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +
>>>> +err_cleanup:
>>>> +	for (; i >= 0; --i) {
>>>> +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
>>>> +		struct v4l2_subdev *sd = &rzr->sd;
>>>> +
>>>> +		v4l2_subdev_cleanup(sd);
>>>> +		media_entity_cleanup(&sd->entity);
>>>> +	}
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	unsigned int i;
>>>> +
>>>> +	for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
>>>> +		struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
>>>> +
>>>> +		if (!resizer->mali_c55)
>>>> +			continue;
>>>> +
>>>> +		v4l2_device_unregister_subdev(&resizer->sd);
>>>> +		v4l2_subdev_cleanup(&resizer->sd);
>>>> +		media_entity_cleanup(&resizer->sd.entity);
>>>> +	}
>>>> +}
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>>> new file mode 100644
>>>> index 000000000000..c7e699741c6d
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>>> @@ -0,0 +1,402 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Test pattern generator
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#include <linux/minmax.h>
>>>> +#include <linux/string.h>
>>>> +
>>>> +#include <media/media-entity.h>
>>>> +#include <media/v4l2-ctrls.h>
>>>> +#include <media/v4l2-subdev.h>
>>>> +
>>>> +#include "mali-c55-common.h"
>>>> +#include "mali-c55-registers.h"
>>>> +
>>>> +#define MALI_C55_TPG_SRC_PAD		0
>>>> +#define MALI_C55_TPG_FIXED_HBLANK	0x20
>>>> +#define MALI_C55_TPG_MAX_VBLANK		0xFFFF
>>> Lowercase hex constants.
>>>
>>>> +#define MALI_C55_TPG_PIXEL_RATE		100000000
>>> This should be exposed to applications using the V4L2_CID_PIXEL_RATE
>>> control (read-only).
>>>
>>>> +
>>>> +static const char * const mali_c55_tpg_test_pattern_menu[] = {
>>>> +	"Flat field",
>>>> +	"Horizontal gradient",
>>>> +	"Vertical gradient",
>>>> +	"Vertical bars",
>>>> +	"Arbitrary rectangle",
>>>> +	"White frame on black field"
>>>> +};
>>>> +
>>>> +static const u32 mali_c55_tpg_mbus_codes[] = {
>>>> +	MEDIA_BUS_FMT_SRGGB20_1X20,
>>>> +	MEDIA_BUS_FMT_RGB202020_1X60,
>>>> +};
>>>> +
>>>> +static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
>>>> +				       int *def_vblank, int *min_vblank)
>>> unsigned int ?
>>>
>>>> +{
>>>> +	unsigned int hts;
>>>> +	int tgt_fps;
>>>> +	int vblank;
>>>> +
>>>> +	hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
>>>> +
>>>> +	/*
>>>> +	 * The ISP has minimum vertical blanking requirements that must be
>>>> +	 * adhered to by the TPG. The minimum is a function of the Iridix blocks
>>>> +	 * clocking requirements and the width of the image and horizontal
>>>> +	 * blanking, but if we assume the worst case iVariance and sVariance
>>>> +	 * values then it boils down to the below.
>>>> +	 */
>>>> +	*min_vblank = 15 + (120500 / hts);
>>> I wonder if this should round up.
>> Maybe? I think the difference is probably too minor to have a practical effect
> Just to make sure we won't have any problem by having a just too short
> minimum vblank when rounding down.
>
>>>> +
>>>> +	/*
>>>> +	 * We need to set a sensible default vblank for whatever format height
>>>> +	 * we happen to be given from set_fmt(). This function just targets
>>>> +	 * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
>>>> +	 * If we can't get 5fps we'll take whatever the minimum vblank gives us.
>>>> +	 */
>>>> +	tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
>>>> +
>>>> +	if (tgt_fps < 5)
>>>> +		vblank = *min_vblank;
>>>> +	else
>>>> +		vblank = MALI_C55_TPG_PIXEL_RATE / hts
>>>> +		       / max(rounddown(tgt_fps, 15), 5);
>>>> +
>>>> +	*def_vblank = ALIGN_DOWN(vblank, 2) - format->height;
>>> "vblank = vblank - height" doesn't seem right. The "else" branch stores
>>> a vts in vblank, which doesn't seem right either. Maybe you meant
>>> something like
>>>
>>> 	if (tgt_fps < 5)
>>> 		def_vts = *min_vblank + format->height;
>>> 	else
>>> 		def_vts = MALI_C55_TPG_PIXEL_RATE / hts
>>> 			/ max(rounddown(tgt_fps, 15), 5);
>>>
>>> 	*def_vblank = ALIGN_DOWN(def_vts, 2) - format->height;
>> I did, thank you.
>>
>>>> +}
>>>> +
>>>> +static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
>>>> +{
>>>> +	struct mali_c55_tpg *tpg = container_of(ctrl->handler,
>>>> +						struct mali_c55_tpg,
>>>> +						ctrls.handler);
>>>> +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
>>>> +
>>> Should you return here if the pipeline isn't streaming ?
>> Yes probably; or if (pm_runtime_get_if_in_use()) would be the usual model I guess.
> I think so yes.
>
>>>> +	switch (ctrl->id) {
>>>> +	case V4L2_CID_TEST_PATTERN:
>>>> +		mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
>>>> +			       ctrl->val);
>>>> +		break;
>>>> +	case V4L2_CID_VBLANK:
>>>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
>>>> +				     MALI_C55_REG_VBLANK_MASK, ctrl->val);
>>>> +		break;
>>>> +	default:
>>>> +		dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
>>>> +		return -EINVAL;
>>> Can this happen ?
>> Not unless somebody breaks something
> Then I think you can drop the error message.


Ack

>
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
>>>> +	.s_ctrl = &mali_c55_tpg_s_ctrl,
>>>> +};
>>>> +
>>>> +static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
>>>> +				   struct v4l2_subdev *sd)
>>>> +{
>>>> +	struct v4l2_subdev_state *state;
>>>> +	struct v4l2_mbus_framefmt *fmt;
>>>> +
>>>> +	/*
>>>> +	 * hblank needs setting, but is a read-only control and thus won't be
>>>> +	 * called during __v4l2_ctrl_handler_setup(). Do it here instead.
>>>> +	 */
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
>>>> +			     MALI_C55_REG_HBLANK_MASK,
>>>> +			     MALI_C55_TPG_FIXED_HBLANK);
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>>> +			     MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
>>>> +
>>>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>>>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>> +
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>>> +			     MALI_C55_TEST_PATTERN_RGB_MASK,
>>>> +			     fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
>>>> +					  0x01 : 0x0);
>>>> +
>>>> +	v4l2_subdev_unlock_state(state);
>>>> +}
>>>> +
>>>> +static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
>>>> +{
>>>> +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
>>>> +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
>>>> +
>>>> +	if (!enable) {
>>>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>>> +				MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
>>>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>>> +				MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	/*
>>>> +	 * One might reasonably expect the framesize to be set here
>>>> +	 * given it's configurable in .set_fmt(), but it's done in the
>>>> +	 * ISP subdevice's stream on func instead, as the same register
>>> s/func/function/
>>>
>>>> +	 * is also used to indicate the size of the data coming from the
>>>> +	 * sensor.
>>>> +	 */
>>>> +	mali_c55_tpg_configure(mali_c55, sd);
>>> 	mali_c55_tpg_configure(tpg);
>>>
>>>> +	__v4l2_ctrl_handler_setup(sd->ctrl_handler);
>>>> +
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>>> +			     MALI_C55_TEST_PATTERN_ON_OFF,
>>>> +			     MALI_C55_TEST_PATTERN_ON_OFF);
>>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>>> +			     MALI_C55_REG_GEN_VIDEO_ON_MASK,
>>>> +			     MALI_C55_REG_GEN_VIDEO_ON_MASK);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
>>>> +	.s_stream = &mali_c55_tpg_s_stream,
>>> Can we use .enable_streams() and .disable_streams() ?
>> Yes, with some extra patches from Tomi that Sakari is picking - I'll base on top of those for the v6.
>>
>>>> +};
>>>> +
>>>> +static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
>>>> +				       struct v4l2_subdev_state *state,
>>>> +				       struct v4l2_subdev_mbus_code_enum *code)
>>>> +{
>>>> +	if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>>> +		return -EINVAL;
>>>> +
>>>> +	code->code = mali_c55_tpg_mbus_codes[code->index];
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
>>>> +					struct v4l2_subdev_state *state,
>>>> +					struct v4l2_subdev_frame_size_enum *fse)
>>>> +{
>>> You sohuld verify here that fse->code is a supported value and return
>>> -EINVAL otherwise.
>>>
>>>> +	if (fse->index > 0 || fse->pad > sd->entity.num_pads)
>>> Drop the pad check, it's done in the subdev core already.
>>>
>>>> +		return -EINVAL;
>>>> +
>>>> +	fse->min_width = MALI_C55_MIN_WIDTH;
>>>> +	fse->max_width = MALI_C55_MAX_WIDTH;
>>>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
>>>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
>>>> +				struct v4l2_subdev_state *state,
>>>> +				struct v4l2_subdev_format *format)
>>>> +{
>>>> +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
>>>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>>>> +	int vblank_def, vblank_min;
>>>> +	unsigned int i;
>>>> +
>>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>>>> +		if (fmt->code == mali_c55_tpg_mbus_codes[i])
>>>> +			break;
>>>> +	}
>>>> +
>>>> +	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>> +
>>>> +	/*
>>>> +	 * The TPG says that the test frame timing generation logic expects a
>>>> +	 * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
>>>> +	 * handle anything smaller than 128x128 it seems pointless to allow a
>>>> +	 * smaller frame.
>>>> +	 */
>>>> +	clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
>>>> +		MALI_C55_MAX_WIDTH);
>>>> +	clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
>>>> +		MALI_C55_MAX_HEIGHT);
>>>> +
>>>> +	*v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;
>>> You're allowing userspace to set fmt->field, as well as all the
>>> colorspace parameters, to random values. I would instead do something
>>> like
>>>
>>> 	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>>> 		if (format->format.code == mali_c55_tpg_mbus_codes[i])
>>> 			break;
>>> 	}
>>>
>>> 	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>> 		format->format.code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>
>>> 	format->format.width = clamp(format->format.width,
>>> 				     MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>> 	format->format.height = clamp(format->format.height,
>>> 				      MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>>
>>> 	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>> 	fmt->code = format->format.code;
>>> 	fmt->width = format->format.width;
>>> 	fmt->height = format->format.height;
>>>
>>> 	format->format = *fmt;
>>>
>>> Alternatively (which I think I like better),
>>>
>>> 	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>
>>> 	fmt->code = format->format.code;
>>>
>>> 	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>>> 		if (fmt->code == mali_c55_tpg_mbus_codes[i])
>>> 			break;
>>> 	}
>>>
>>> 	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>> 		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>
>>> 	fmt->width = clamp(format->format.width,
>>> 			   MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>> 	fmt->height = clamp(format->format.height,
>>> 			    MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>>
>>> 	format->format = *fmt;
>>>
>>>> +
>>>> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY)
>>>> +		return 0;
>>>> +
>>>> +	__mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
>>>> +	__v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
>>>> +				 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
>>>> +	__v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);
>>> Move those three calls to a separate function, it will be reused below.
>>> I'd name is mali_c55_tpg_update_vblank(). You can fold
>>> __mali_c55_tpg_calc_vblank() in it.
>>>
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
>>>> +	.enum_mbus_code		= mali_c55_tpg_enum_mbus_code,
>>>> +	.enum_frame_size	= mali_c55_tpg_enum_frame_size,
>>>> +	.get_fmt		= v4l2_subdev_get_fmt,
>>>> +	.set_fmt		= mali_c55_tpg_set_fmt,
>>>> +};
>>>> +
>>>> +static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
>>>> +	.video	= &mali_c55_tpg_video_ops,
>>>> +	.pad	= &mali_c55_tpg_pad_ops,
>>>> +};
>>>> +
>>>> +static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
>>>> +				   struct v4l2_subdev_state *sd_state)
>>> You name this variable state in every other subdev operation handler.
>>>
>>>> +{
>>>> +	struct v4l2_mbus_framefmt *fmt =
>>>> +		v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
>>>> +
>>>> +	fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>> +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>> +	fmt->field = V4L2_FIELD_NONE;
>>>> +	fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>> Initialize the colorspace fields too.
>>>
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
>>>> +	.init_state = mali_c55_tpg_init_state,
>>>> +};
>>>> +
>>>> +static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
>>>> +	struct v4l2_subdev *sd = &mali_c55->tpg.sd;
>>>> +	struct v4l2_mbus_framefmt *format;
>>>> +	struct v4l2_subdev_state *state;
>>>> +	int vblank_def, vblank_min;
>>>> +	int ret;
>>>> +
>>>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>>>> +	format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>> +
>>>> +	ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);
>>> You have 3 controls.
>>>
>>>> +	if (ret)
>>>> +		goto err_unlock;
>>>> +
>>>> +	ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
>>>> +				&mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
>>>> +				ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
>>>> +				0, 3, mali_c55_tpg_test_pattern_menu);
>>>> +
>>>> +	/*
>>>> +	 * We fix hblank at the minimum allowed value and control framerate
>>>> +	 * solely through the vblank control.
>>>> +	 */
>>>> +	ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
>>>> +				&mali_c55_tpg_ctrl_ops,
>>>> +				V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
>>>> +				MALI_C55_TPG_FIXED_HBLANK, 1,
>>>> +				MALI_C55_TPG_FIXED_HBLANK);
>>>> +	if (ctrls->hblank)
>>>> +		ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
>>>> +
>>>> +	__mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);
>>> Drop this and initialize the control with default values. You can then
>>> update the value by calling mali_c55_tpg_update_vblank() in
>>> mali_c55_register_tpg().
>>>
>>> The reason is to share the same mutex between the control handler and
>>> the subdev active state without having to add a separate mutex in the
>>> mali_c55_tpg structure. The simplest way to do so is to initialize the
>>> controls first, set sd->state_lock to point to the control handler lock,
>>> and call v4l2_subdev_init_finalize() as the last step. As a consequence,
>>> you can't access the active state when initializing controls.
>>>
>>> You can alternatively keep the lock in mali_c55_tpg and set
>>> sd->state_lock to point to it, but I think that's more complex.
>>>
>>>> +	ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
>>>> +					  &mali_c55_tpg_ctrl_ops,
>>>> +					  V4L2_CID_VBLANK, vblank_min,
>>>> +					  MALI_C55_TPG_MAX_VBLANK, 1,
>>>> +					  vblank_def);
>>>> +
>>>> +	if (ctrls->handler.error) {
>>>> +		dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
>>>> +		ret = ctrls->handler.error;
>>>> +		goto err_free_handler;
>>>> +	}
>>>> +
>>>> +	ctrls->handler.lock = &mali_c55->tpg.lock;
>>> Drop this and drop the mutex. The control handler will use its internal
>>> mutex.
>>>
>>>> +	mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
>>>> +
>>>> +	v4l2_subdev_unlock_state(state);
>>>> +
>>>> +	return 0;
>>>> +
>>>> +err_free_handler:
>>>> +	v4l2_ctrl_handler_free(&ctrls->handler);
>>>> +err_unlock:
>>>> +	v4l2_subdev_unlock_state(state);
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
>>>> +	struct v4l2_subdev *sd = &tpg->sd;
>>>> +	struct media_pad *pad = &tpg->pad;
>>>> +	int ret;
>>>> +
>>>> +	mutex_init(&tpg->lock);
>>>> +
>>>> +	v4l2_subdev_init(sd, &mali_c55_tpg_ops);
>>>> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
>>>> +	sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
>>> Should we introduce a TPG function ?
>> Hmmm I vacillate a bit. I don't see that it would hurt right now, but on the other hand I think
>> there's some value in pretending they're sensors - on the grounds that they should be handled
>> identically as much as possible. I'd be quite wary if we ever saw "if (sd->entity.function ==
>> MEDIA_ENT_F_TPG)" somewhere.
> The TPG doesn't expose all the API elements we expect from raw sensors,
> so we'll have userspace code that is TPG-specific. I suppose it will
> look for th TPG entity by name, so we don't need a separate function.
> The other part to consider is generic code looking for a raw sensor that
> would pick the TPG by mistake. I'm not sure how big of a risk that would
> be.

That'll always be a risk though, unless they specifically search by function, which isn't 
guaranteed...we could provide a parameter to prevent the TPG from being enumerated if that was a worry?
>
>>>> +	sd->internal_ops = &mali_c55_tpg_internal_ops;
>>>> +	strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
>>>> +
>>>> +	pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;
>>> I don't think MEDIA_PAD_FL_MUST_CONNECT is right.
>> The docs say "If this flag is set and the pad is linked to any other pad, then at least one of those
>> links must be enabled for the entity to be able to stream.", and that's the case here right?
> The MEDIA_PAD_FL_MUST_CONNECT flag indicates that at least one link must
> be enabled for that pad in order for the entity to be included in a
> pipeline. The intent is to check in the core if, for instance, a CSI-2
> RX input is connected to something, and refuse streaming if it isn't.
>
> The TPG output needs to be connected to something for the TPG to be used
> in a pipeline, but the TPG won't be in the pipeline in the first place
> if its output is not connected. The flag is therefore not necessary.

Okedokey - thanks
>
>>>> +	ret = media_entity_pads_init(&sd->entity, 1, pad);
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev,
>>>> +			"Failed to initialize media entity pads\n");
>>>> +		goto err_destroy_mutex;
>>>> +	}
>>>> +
>>> 	sd->state_lock = sd->ctrl_handler->lock;
>>>
>>> to use the same lock for the controls and the active state. You need to
>>> move this line and the v4l2_subdev_init_finalize() call after
>>> mali_c55_tpg_init_controls() to get the control handler lock initialized
>>> first.
>>>
>>>> +	ret = v4l2_subdev_init_finalize(sd);
>>>> +	if (ret)
>>>> +		goto err_cleanup_media_entity;
>>>> +
>>>> +	ret = mali_c55_tpg_init_controls(mali_c55);
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev,
>>>> +			"Error initialising controls\n");
>>>> +		goto err_cleanup_subdev;
>>>> +	}
>>>> +
>>>> +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
>>>> +		goto err_free_ctrl_handler;
>>>> +	}
>>>> +
>>>> +	/*
>>>> +	 * By default the colour settings lead to a very dim image that is
>>>> +	 * nearly indistinguishable from black on some monitor settings. Ramp
>>>> +	 * them up a bit so the image is brighter.
>>>> +	 */
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
>>>> +		       MALI_C55_TPG_BACKGROUND_MAX);
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
>>>> +		       MALI_C55_TPG_BACKGROUND_MAX);
>>>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
>>>> +		       MALI_C55_TPG_BACKGROUND_MAX);
>>>> +
>>>> +	tpg->mali_c55 = mali_c55;
>>>> +
>>>> +	return 0;
>>>> +
>>>> +err_free_ctrl_handler:
>>>> +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
>>>> +err_cleanup_subdev:
>>>> +	v4l2_subdev_cleanup(sd);
>>>> +err_cleanup_media_entity:
>>>> +	media_entity_cleanup(&sd->entity);
>>>> +err_destroy_mutex:
>>>> +	mutex_destroy(&tpg->lock);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
>>>> +
>>>> +	if (!tpg->mali_c55)
>>>> +		return;
>>>> +
>>>> +	v4l2_device_unregister_subdev(&tpg->sd);
>>>> +	v4l2_subdev_cleanup(&tpg->sd);
>>>> +	media_entity_cleanup(&tpg->sd.entity);
>>>> +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
>>> Free the control handler just after v4l2_device_unregister_subdev() to
>>> match the order in mali_c55_register_tpg().
>>>
>>>> +	mutex_destroy(&tpg->lock);
>>>> +}

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Dan Scally]

Hi Laurent, thanks for the comments

On 30/05/2024 22:43, Laurent Pinchart wrote:
> And now the second part of the review, addressing mali-c55-capture.c and
> mali-c55-resizer.c. I've reviewed the code from the bottom up, so some
> messages may be repeated in an order that seems weird. Sorry about that.
>
> On Thu, May 30, 2024 at 03:15:10AM +0300, Laurent Pinchart wrote:
>> On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
>>> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
>>> V4L2 and Media Controller compliant and creates subdevices to manage
>>> the ISP itself, its internal test pattern generator as well as the
>>> crop, scaler and output format functionality for each of its two
>>> output devices.
>>>
>>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>>> ---
>>> Changes in v5:
>>>
>>> 	- Reworked input formats - previously we allowed representing input data
>>> 	  as any 8-16 bit format. Now we only allow input data to be represented
>>> 	  by the new 20-bit bayer formats, which is corrected to the equivalent
>>> 	  16-bit format in RAW bypass mode.
>>> 	- Stopped bypassing blocks that we haven't added supporting parameters
>>> 	  for yet.
>>> 	- Addressed most of Sakari's comments from the list
>>>
>>> Changes not yet made in v5:
>>>
>>> 	- The output pipelines can still be started and stopped independently of
>>> 	  one another - I'd like to discuss that more.
>>> 	- the TPG subdev still uses .s_stream() - I need to rebase onto a tree
>>> 	  with working .enable_streams() for a single-source-pad subdevice.
>>>
>>> Changes in v4:
>>>
>>> 	- Reworked mali_c55_update_bits() to internally perform the bit-shift
>> I really don't like that, it makes the code very confusing, even more so
>> as it differs from regmap_update_bits().
>>
>> Look at this for instance:
>>
>> 	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>> 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
>> 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
>>
>> It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
>> BIT(0).
>>
>> Sorry, I know it will be painful, but this change needs to be reverted.
>>
>>> 	- Reworked the resizer to allow cropping during streaming
>>> 	- Fixed a bug in NV12 output
>>>
>>> Changes in v3:
>>>
>>> 	- Mostly minor fixes suggested by Sakari
>>> 	- Fixed the sequencing of vb2 buffers to be synchronised across the two
>>> 	  capture devices.
>>>
>>> Changes in v2:
>>>
>>> 	- Clock handling
>>> 	- Fixed the warnings raised by the kernel test robot
>>>
>>>   drivers/media/platform/Kconfig                |   1 +
>>>   drivers/media/platform/Makefile               |   1 +
>>>   drivers/media/platform/arm/Kconfig            |   5 +
>>>   drivers/media/platform/arm/Makefile           |   2 +
>>>   drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
>>>   drivers/media/platform/arm/mali-c55/Makefile  |   9 +
>>>   .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
>>>   .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
>>>   .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
>>>   .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
>>>   .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
>>>   .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
>>>   .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
>>>   .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
>>>   14 files changed, 4452 insertions(+)
>>>   create mode 100644 drivers/media/platform/arm/Kconfig
>>>   create mode 100644 drivers/media/platform/arm/Makefile
>>>   create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
>>>   create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>> I've skipped review of capture.c and resizer.c as I already have plenty
>> of comments for the other files, and it's getting late. I'll try to
>> review the rest tomorrow.
>>
>>> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
>>> index 2d79bfc68c15..c929169766aa 100644
>>> --- a/drivers/media/platform/Kconfig
>>> +++ b/drivers/media/platform/Kconfig
>>> @@ -65,6 +65,7 @@ config VIDEO_MUX
>>>   source "drivers/media/platform/allegro-dvt/Kconfig"
>>>   source "drivers/media/platform/amlogic/Kconfig"
>>>   source "drivers/media/platform/amphion/Kconfig"
>>> +source "drivers/media/platform/arm/Kconfig"
>>>   source "drivers/media/platform/aspeed/Kconfig"
>>>   source "drivers/media/platform/atmel/Kconfig"
>>>   source "drivers/media/platform/broadcom/Kconfig"
>>> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
>>> index da17301f7439..9a647abd5218 100644
>>> --- a/drivers/media/platform/Makefile
>>> +++ b/drivers/media/platform/Makefile
>>> @@ -8,6 +8,7 @@
>>>   obj-y += allegro-dvt/
>>>   obj-y += amlogic/
>>>   obj-y += amphion/
>>> +obj-y += arm/
>>>   obj-y += aspeed/
>>>   obj-y += atmel/
>>>   obj-y += broadcom/
>>> diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
>>> new file mode 100644
>>> index 000000000000..4f0764c329c7
>>> --- /dev/null
>>> +++ b/drivers/media/platform/arm/Kconfig
>>> @@ -0,0 +1,5 @@
>>> +# SPDX-License-Identifier: GPL-2.0-only
>>> +
>>> +comment "ARM media platform drivers"
>>> +
>>> +source "drivers/media/platform/arm/mali-c55/Kconfig"
>>> diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
>>> new file mode 100644
>>> index 000000000000..8cc4918725ef
>>> --- /dev/null
>>> +++ b/drivers/media/platform/arm/Makefile
>>> @@ -0,0 +1,2 @@
>>> +# SPDX-License-Identifier: GPL-2.0-only
>>> +obj-y += mali-c55/
>>> diff --git a/drivers/media/platform/arm/mali-c55/Kconfig b/drivers/media/platform/arm/mali-c55/Kconfig
>>> new file mode 100644
>>> index 000000000000..602085e28b01
>>> --- /dev/null
>>> +++ b/drivers/media/platform/arm/mali-c55/Kconfig
>>> @@ -0,0 +1,18 @@
>>> +# SPDX-License-Identifier: GPL-2.0-only
>>> +config VIDEO_MALI_C55
>>> +	tristate "ARM Mali-C55 Image Signal Processor driver"
>>> +	depends on V4L_PLATFORM_DRIVERS
>>> +	depends on VIDEO_DEV && OF
>>> +	depends on ARCH_VEXPRESS || COMPILE_TEST
>>> +	select MEDIA_CONTROLLER
>>> +	select VIDEO_V4L2_SUBDEV_API
>>> +	select VIDEOBUF2_DMA_CONTIG
>>> +	select VIDEOBUF2_VMALLOC
>>> +	select V4L2_FWNODE
>>> +	select GENERIC_PHY_MIPI_DPHY
>> Alphabetical order ?
>>
>>> +	default n
>> That's the default, you don't have to specify ti.
>>
>>> +	help
>>> +	  Enable this to support Arm's Mali-C55 Image Signal Processor.
>>> +
>>> +	  To compile this driver as a module, choose M here: the module
>>> +	  will be called mali-c55.
>>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
>>> new file mode 100644
>>> index 000000000000..77dcb2fbf0f4
>>> --- /dev/null
>>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
>>> @@ -0,0 +1,9 @@
>>> +# SPDX-License-Identifier: GPL-2.0
>>> +
>>> +mali-c55-y := mali-c55-capture.o \
>>> +	      mali-c55-core.o \
>>> +	      mali-c55-isp.o \
>>> +	      mali-c55-tpg.o \
>>> +	      mali-c55-resizer.o
>> Alphabetical order here too.
>>
>>> +
>>> +obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>> new file mode 100644
>>> index 000000000000..1d539ac9c498
>>> --- /dev/null
>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>> @@ -0,0 +1,951 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +/*
>>> + * ARM Mali-C55 ISP Driver - Video capture devices
>>> + *
>>> + * Copyright (C) 2024 Ideas on Board Oy
>>> + */
>>> +
>>> +#include <linux/cleanup.h>
>>> +#include <linux/minmax.h>
>>> +#include <linux/pm_runtime.h>
>>> +#include <linux/string.h>
>>> +#include <linux/videodev2.h>
>>> +
>>> +#include <media/v4l2-dev.h>
>>> +#include <media/v4l2-event.h>
>>> +#include <media/v4l2-ioctl.h>
>>> +#include <media/v4l2-subdev.h>
>>> +#include <media/videobuf2-core.h>
>>> +#include <media/videobuf2-dma-contig.h>
>>> +
>>> +#include "mali-c55-common.h"
>>> +#include "mali-c55-registers.h"
>>> +
>>> +static const struct mali_c55_fmt mali_c55_fmts[] = {
>>> +	/*
>>> +	 * This table is missing some entries which need further work or
>>> +	 * investigation:
>>> +	 *
>>> +	 * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
>>> +	 * Base mode 5 is "Generic Data"
>>> +	 * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
>>> +	 * Base mode 9 seems to have no V4L2 equivalent
>>> +	 * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
>>> +	 * equivalent
>>> +	 */
>>> +	{
>>> +		.fourcc = V4L2_PIX_FMT_ARGB2101010,
>>> +		.mbus_codes = {
>>> +			MEDIA_BUS_FMT_RGB121212_1X36,
>>> +			MEDIA_BUS_FMT_RGB202020_1X60,
>>> +		},
>>> +		.is_raw = false,
>>> +		.registers = {
>>> +			.base_mode = MALI_C55_OUTPUT_A2R10G10B10,
>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>> +		}
>>> +	},
>>> +	{
>>> +		.fourcc = V4L2_PIX_FMT_RGB565,
>>> +		.mbus_codes = {
>>> +			MEDIA_BUS_FMT_RGB121212_1X36,
>>> +			MEDIA_BUS_FMT_RGB202020_1X60,
>>> +		},
>>> +		.is_raw = false,
>>> +		.registers = {
>>> +			.base_mode = MALI_C55_OUTPUT_RGB565,
>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>> +		}
>>> +	},
>>> +	{
>>> +		.fourcc = V4L2_PIX_FMT_BGR24,
>>> +		.mbus_codes = {
>>> +			MEDIA_BUS_FMT_RGB121212_1X36,
>>> +			MEDIA_BUS_FMT_RGB202020_1X60,
>>> +		},
>>> +		.is_raw = false,
>>> +		.registers = {
>>> +			.base_mode = MALI_C55_OUTPUT_RGB24,
>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>> +		}
>>> +	},
>>> +	{
>>> +		.fourcc = V4L2_PIX_FMT_YUYV,
>>> +		.mbus_codes = {
>>> +			MEDIA_BUS_FMT_YUV10_1X30,
>>> +		},
>>> +		.is_raw = false,
>>> +		.registers = {
>>> +			.base_mode = MALI_C55_OUTPUT_YUY2,
>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>> +		}
>>> +	},
>>> +	{
>>> +		.fourcc = V4L2_PIX_FMT_UYVY,
>>> +		.mbus_codes = {
>>> +			MEDIA_BUS_FMT_YUV10_1X30,
>>> +		},
>>> +		.is_raw = false,
>>> +		.registers = {
>>> +			.base_mode = MALI_C55_OUTPUT_UYVY,
>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>> +		}
>>> +	},
>>> +	{
>>> +		.fourcc = V4L2_PIX_FMT_Y210,
>>> +		.mbus_codes = {
>>> +			MEDIA_BUS_FMT_YUV10_1X30,
>>> +		},
>>> +		.is_raw = false,
>>> +		.registers = {
>>> +			.base_mode = MALI_C55_OUTPUT_Y210,
>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>> +		}
>>> +	},
>>> +	/*
>>> +	 * This is something of a hack, the ISP thinks it's running NV12M but
>>> +	 * by setting uv_plane = 0 we simply discard that planes and only output
>>> +	 * the Y-plane.
>>> +	 */
>>> +	{
>>> +		.fourcc = V4L2_PIX_FMT_GREY,
>>> +		.mbus_codes = {
>>> +			MEDIA_BUS_FMT_YUV10_1X30,
>>> +		},
>>> +		.is_raw = false,
>>> +		.registers = {
>>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>> +		}
>>> +	},
>>> +	{
>>> +		.fourcc = V4L2_PIX_FMT_NV12M,
>>> +		.mbus_codes = {
>>> +			MEDIA_BUS_FMT_YUV10_1X30,
>>> +		},
>>> +		.is_raw = false,
>>> +		.registers = {
>>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
>>> +		}
>>> +	},
>>> +	{
>>> +		.fourcc = V4L2_PIX_FMT_NV21M,
>>> +		.mbus_codes = {
>>> +			MEDIA_BUS_FMT_YUV10_1X30,
>>> +		},
>>> +		.is_raw = false,
>>> +		.registers = {
>>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
>>> +		}
>>> +	},
>>> +	/*
>>> +	 * RAW uncompressed formats are all packed in 16 bpp.
>>> +	 * TODO: Expand this list to encompass all possible RAW formats.
>>> +	 */
>>> +	{
>>> +		.fourcc = V4L2_PIX_FMT_SRGGB16,
>>> +		.mbus_codes = {
>>> +			MEDIA_BUS_FMT_SRGGB16_1X16,
>>> +		},
>>> +		.is_raw = true,
>>> +		.registers = {
>>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>> +		}
>>> +	},
>>> +	{
>>> +		.fourcc = V4L2_PIX_FMT_SBGGR16,
>>> +		.mbus_codes = {
>>> +			MEDIA_BUS_FMT_SBGGR16_1X16,
>>> +		},
>>> +		.is_raw = true,
>>> +		.registers = {
>>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>> +		}
>>> +	},
>>> +	{
>>> +		.fourcc = V4L2_PIX_FMT_SGBRG16,
>>> +		.mbus_codes = {
>>> +			MEDIA_BUS_FMT_SGBRG16_1X16,
>>> +		},
>>> +		.is_raw = true,
>>> +		.registers = {
>>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>> +		}
>>> +	},
>>> +	{
>>> +		.fourcc = V4L2_PIX_FMT_SGRBG16,
>>> +		.mbus_codes = {
>>> +			MEDIA_BUS_FMT_SGRBG16_1X16,
>>> +		},
>>> +		.is_raw = true,
>>> +		.registers = {
>>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>> +		}
>>> +	},
>>> +};
>>> +
>>> +static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
>>> +					       u32 code)
>>> +{
>>> +	unsigned int i;
>>> +
>>> +	for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
>>> +		if (fmt->mbus_codes[i] == code)
>>> +			return true;
>>> +	}
>>> +
>>> +	return false;
>>> +}
>>> +
>>> +bool mali_c55_format_is_raw(unsigned int mbus_code)
>>> +{
>>> +	unsigned int i;
>>> +
>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>> +		if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
>>> +			return mali_c55_fmts[i].is_raw;
>>> +	}
>>> +
>>> +	return false;
>>> +}
>>> +
>>> +static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
>>> +{
>>> +	unsigned int i;
>>> +
>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>> +		if (mali_c55_fmts[i].fourcc == pixelformat)
>>> +			return &mali_c55_fmts[i];
>>> +	}
>>> +
>>> +	/*
>>> +	 * If we find no matching pixelformat, we'll just default to the first
>>> +	 * one for now.
>>> +	 */
>>> +
>>> +	return &mali_c55_fmts[0];
>>> +}
>>> +
>>> +static const char * const capture_device_names[] = {
>>> +	"mali-c55 fr",
>>> +	"mali-c55 ds",
>>> +	"mali-c55 3a stats",
>>> +	"mali-c55 params",
> The last two entries are not used AFAICT, neither here, nor in
> subsequent patches.
>
>>> +};
>>> +
>>> +static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
>>> +{
>>> +	if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
>>> +		return capture_device_names[0];
>>> +
>>> +	if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
>>> +		return capture_device_names[1];
>>> +
>>> +	return "params/stat not supported yet";
>>> +}
> Use cap_dev->vdev.name instead of mali_c55_cap_dev_to_name(cap_dev) and
> drop this function.
>
>>> +
>>> +static int mali_c55_link_validate(struct media_link *link)
>>> +{
>>> +	struct video_device *vdev =
>>> +		media_entity_to_video_device(link->sink->entity);
>>> +	struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
>>> +	struct v4l2_subdev *sd =
>>> +		media_entity_to_v4l2_subdev(link->source->entity);
>>> +	const struct v4l2_pix_format_mplane *pix_mp;
>>> +	const struct mali_c55_fmt *cap_fmt;
>>> +	struct v4l2_subdev_format sd_fmt = {
>>> +		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
>>> +		.pad = link->source->index,
>>> +	};
>>> +	int ret;
>>> +
>>> +	ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
>>> +	if (ret)
>>> +		return ret;
>>> +
>>> +	pix_mp = &cap_dev->mode.pix_mp;
>>> +	cap_fmt = cap_dev->mode.capture_fmt;
>>> +
>>> +	if (sd_fmt.format.width != pix_mp->width ||
>>> +	    sd_fmt.format.height != pix_mp->height) {
>>> +		dev_dbg(cap_dev->mali_c55->dev,
>>> +			"link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
>>> +			link->source->entity->name, link->source->index,
>>> +			link->sink->entity->name, link->sink->index,
>>> +			sd_fmt.format.width, sd_fmt.format.height,
>>> +			pix_mp->width, pix_mp->height);
>>> +		return -EPIPE;
>>> +	}
>>> +
>>> +	if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
>>> +		dev_dbg(cap_dev->mali_c55->dev,
>>> +			"link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format %p4cc\n",
>>> +			link->source->entity->name, link->source->index,
>>> +			link->sink->entity->name, link->sink->index,
>>> +			sd_fmt.format.code, &pix_mp->pixelformat);
>>> +		return -EPIPE;
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static const struct media_entity_operations mali_c55_media_ops = {
>>> +	.link_validate = mali_c55_link_validate,
>>> +};
>>> +
>>> +static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
>>> +				    unsigned int *num_planes, unsigned int sizes[],
>>> +				    struct device *alloc_devs[])
>>> +{
>>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>> +	unsigned int i;
>>> +
>>> +	if (*num_planes) {
>>> +		if (*num_planes != cap_dev->mode.pix_mp.num_planes)
>>> +			return -EINVAL;
>>> +
>>> +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>> +			if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
>>> +				return -EINVAL;
>>> +	} else {
>>> +		*num_planes = cap_dev->mode.pix_mp.num_planes;
>>> +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>> +			sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static void mali_c55_buf_queue(struct vb2_buffer *vb)
>>> +{
>>> +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
>>> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>>> +	struct mali_c55_buffer *buf = container_of(vbuf,
>>> +						   struct mali_c55_buffer, vb);
>>> +	unsigned int i;
>>> +
>>> +	buf->plane_done[MALI_C55_PLANE_Y] = false;
>>> +
>>> +	/*
>>> +	 * If we're in a single-plane format we flag the other plane as done
>>> +	 * already so it's dequeued appropriately later
>>> +	 */
>>> +	buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
>>> +
>>> +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
>>> +		unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
>>> +
>>> +		vb2_set_plane_payload(vb, i, size);
>>> +	}
>>> +
>>> +	spin_lock(&cap_dev->buffers.lock);
>>> +	list_add_tail(&buf->queue, &cap_dev->buffers.queue);
>>> +	spin_unlock(&cap_dev->buffers.lock);
>>> +}
>>> +
>>> +static int mali_c55_buf_init(struct vb2_buffer *vb)
>>> +{
>>> +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
>>> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>>> +	struct mali_c55_buffer *buf = container_of(vbuf,
>>> +						   struct mali_c55_buffer, vb);
>>> +	unsigned int i;
>>> +
>>> +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>> +		buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
>>> +{
>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>> +
>>> +	guard(spinlock)(&cap_dev->buffers.lock);
>>> +
>>> +	cap_dev->buffers.curr = cap_dev->buffers.next;
>>> +	cap_dev->buffers.next = NULL;
>>> +
>>> +	if (!list_empty(&cap_dev->buffers.queue)) {
>>> +		struct v4l2_pix_format_mplane *pix_mp;
>>> +		const struct v4l2_format_info *info;
>>> +		u32 *addrs;
>>> +
>>> +		pix_mp = &cap_dev->mode.pix_mp;
>>> +		info = v4l2_format_info(pix_mp->pixelformat);
>>> +
>>> +		mali_c55_update_bits(mali_c55,
>>> +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
>>> +		if (cap_dev->mode.capture_fmt->registers.uv_plane)
>>> +			mali_c55_update_bits(mali_c55,
>>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
>>> +
>>> +		cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
>>> +							 struct mali_c55_buffer,
>>> +							 queue);
>>> +		list_del(&cap_dev->buffers.next->queue);
>>> +
>>> +		addrs = cap_dev->buffers.next->addrs;
>>> +		mali_c55_write(mali_c55,
>>> +			MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
>>> +			addrs[MALI_C55_PLANE_Y]);
>>> +		mali_c55_write(mali_c55,
>>> +			MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
>>> +			addrs[MALI_C55_PLANE_UV]);
>>> +		mali_c55_write(mali_c55,
>>> +			MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
>>> +			pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
>>> +		mali_c55_write(mali_c55,
>>> +			MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
>>> +			pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
>>> +			/ info->hdiv);
>>> +	} else {
>>> +		/*
>>> +		 * If we underflow then we can tell the ISP that we don't want
>>> +		 * to write out the next frame.
>>> +		 */
>>> +		mali_c55_update_bits(mali_c55,
>>> +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>> +		mali_c55_update_bits(mali_c55,
>>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>> +	}
>>> +}
>>> +
>>> +static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
>>> +				   unsigned int framecount)
>>> +{
>>> +	curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>>> +	curr_buf->vb.field = V4L2_FIELD_NONE;
> The could be set already when the buffer is queued.
>
>>> +	curr_buf->vb.sequence = framecount;
>>> +	vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
>>> +}
>>> +
>>> +/**
>>> + * mali_c55_set_plane_done - mark the plane as written and process the buffer if
>>> + *			     both planes are finished.
>>> + * @cap_dev:  pointer to the fr or ds pipe output
>>> + * @plane:    the plane to mark as completed
>>> + *
>>> + * The Mali C55 ISP has muliplanar outputs for some formats that come with two
>>> + * separate "buffer write completed" interrupts - we need to flag each plane's
>>> + * completion and check whether both planes are done - if so, complete the buf
>>> + * in vb2.
>>> + */
>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>>> +			     enum mali_c55_planes plane)
>>> +{
>>> +	struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
>>> +	struct mali_c55_buffer *curr_buf;
>>> +
>>> +	guard(spinlock)(&cap_dev->buffers.lock);
>>> +	curr_buf = cap_dev->buffers.curr;
>>> +
>>> +	/*
>>> +	 * This _should_ never happen. If no buffer was available from vb2 then
>>> +	 * we tell the ISP not to bother writing the next frame, which means the
>>> +	 * interrupts that call this function should never trigger. If it does
>>> +	 * happen then one of our assumptions is horribly wrong - complain
>>> +	 * loudly and do nothing.
>>> +	 */
>>> +	if (!curr_buf) {
>>> +		dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
>>> +			mali_c55_cap_dev_to_name(cap_dev), __func__);
>>> +		return;
>>> +	}
>>> +
>>> +	/* If the other plane is also done... */
>>> +	if (curr_buf->plane_done[~plane & 1]) {
>>> +		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
>>> +		cap_dev->buffers.curr = NULL;
>>> +		isp->frame_sequence++;
>>> +	} else {
>>> +		curr_buf->plane_done[plane] = true;
>>> +	}
>>> +}
>>> +
>>> +static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
>>> +{
>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>> +
>>> +	mali_c55_update_bits(mali_c55,
>>> +			     MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>> +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>> +	mali_c55_update_bits(mali_c55,
>>> +			     MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>> +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>> +}
>>> +
>>> +static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
>>> +{
>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>> +
>>> +	/*
>>> +	 * The Mali ISP can hold up to 5 buffer addresses and simply cycle
>>> +	 * through them, but it's not clear to me that the vb2 queue _guarantees_
>>> +	 * it will queue buffers to the driver in a fixed order, and ensuring
>>> +	 * we call vb2_buffer_done() for the right buffer seems to me to add
>>> +	 * pointless complexity given in multi-context mode we'd need to
>>> +	 * re-write those registers every frame anyway...so we tell the ISP to
>>> +	 * use a single register and update it for each frame.
>>> +	 */
> A single register sounds prone to error conditions. Is it at least
> shadowed in the hardware, or do you have to make sure you reprogram it
> during the vertical blanking only ?


It would have to be reprogrammed during the vertical blanking if we were running in a configuration 
with a single config space, otherwise you have the time it takes to process a frame plus vertical 
blanking. As I say, it'll have to work like this in multi-context mode anyway.


If we want to use the cycling...is it guaranteed that vb2 buffers will always be queued in order?

>
> I'll mostly skip buffer handling in this review, I need to first
> understand how the hardware operates to make an informed opinion.
>
>>> +	mali_c55_update_bits(mali_c55,
>>> +			MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
>>> +			MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
>>> +	mali_c55_update_bits(mali_c55,
>>> +			MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
>>> +			MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
>>> +
>>> +	/*
>>> +	 * We only queue a buffer in the streamon path if this is the first of
>>> +	 * the capture devices to start streaming. If the ISP is already running
>>> +	 * then we rely on the ISP_START interrupt to queue the first buffer for
>>> +	 * this capture device.
>>> +	 */
>>> +	if (mali_c55->pipe.start_count == 1)
>>> +		mali_c55_set_next_buffer(cap_dev);
> I think we'll have to revisit buffer handling to make sure it's 100%
> race-free.
>
>>> +}
>>> +
>>> +static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
>>> +					    enum vb2_buffer_state state)
>>> +{
>>> +	struct mali_c55_buffer *buf, *tmp;
>>> +
>>> +	guard(spinlock)(&cap_dev->buffers.lock);
>>> +
>>> +	if (cap_dev->buffers.curr) {
>>> +		vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
>>> +				state);
>>> +		cap_dev->buffers.curr = NULL;
>>> +	}
>>> +
>>> +	if (cap_dev->buffers.next) {
>>> +		vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
>>> +				state);
>>> +		cap_dev->buffers.next = NULL;
>>> +	}
>>> +
>>> +	list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
>>> +		list_del(&buf->queue);
>>> +		vb2_buffer_done(&buf->vb.vb2_buf, state);
>>> +	}
>>> +}
>>> +
>>> +static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
>>> +{
>>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>> +	struct mali_c55_resizer *rzr = cap_dev->rzr;
>>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>>> +	int ret;
>>> +
>>> +	guard(mutex)(&isp->lock);
> What's the reason for using the isp lock here and in
> mali_c55_vb2_stop_streaming() ? If you need a lock that covers all video
> nodes in order to synchronize start/stop, you may want to use the
> graph_mutex of the media device instead.


It's because I wanted to make sure that the ISP was in a known started/stopped state before possibly 
trying to start/stop it, which can be done from either of the two capture devices. This would go 
away if we were synchronising with the links anyway.

>
>>> +
>>> +	ret = pm_runtime_resume_and_get(mali_c55->dev);
>>> +	if (ret)
>>> +		return ret;
>>> +
>>> +	ret = video_device_pipeline_start(&cap_dev->vdev,
>>> +					  &cap_dev->mali_c55->pipe);
>>> +	if (ret) {
>>> +		dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
>>> +			mali_c55_cap_dev_to_name(cap_dev));
> Drop the message or make it dev_dbg() as it can be triggered by
> userspace.
>
>>> +		goto err_pm_put;
>>> +	}
>>> +
>>> +	mali_c55_cap_dev_stream_enable(cap_dev);
>>> +	mali_c55_rzr_start_stream(rzr);
>>> +
>>> +	/*
>>> +	 * We only start the ISP if we're the only capture device that's
>>> +	 * streaming. Otherwise, it'll already be active.
>>> +	 */
> I still think we should use link setup to indicate which video devices
> userspace plans to use, and then only start when they're all started.
> That includes stats and parameters buffers. We can continue this
> discussion in the context of the previous version of the patch series,
> or here, up to you.

Let's just continue here. I think I called it "clunky" before; from my perspective it's an 
unnecessary extra step - we can already signal to the driver that we don't want to use the video 
devices by not queuing buffers to them or starting the stream on them and although I understand that 
that means that one of the two image data capture devices will receive data before the other, I 
don't understand why that's considered to be a problem. Possibly that last part is the stickler; can 
you explain a bit why it's an issue for one capture queue to start earlier than the other?


>
>>> +	if (mali_c55->pipe.start_count == 1) {
>>> +		ret = mali_c55_isp_start_stream(isp);
>>> +		if (ret)
>>> +			goto err_disable_cap_dev;
>>> +	}
>>> +
>>> +	return 0;
>>> +
>>> +err_disable_cap_dev:
>>> +	mali_c55_cap_dev_stream_disable(cap_dev);
>>> +	video_device_pipeline_stop(&cap_dev->vdev);
>>> +err_pm_put:
>>> +	pm_runtime_put(mali_c55->dev);
>>> +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
>>> +{
>>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>> +	struct mali_c55_resizer *rzr = cap_dev->rzr;
>>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>>> +
>>> +	guard(mutex)(&isp->lock);
>>> +
>>> +	/*
>>> +	 * If one of the other capture nodes is streaming, we shouldn't
>>> +	 * disable the ISP here.
>>> +	 */
>>> +	if (mali_c55->pipe.start_count == 1)
>>> +		mali_c55_isp_stop_stream(&mali_c55->isp);
>>> +
>>> +	mali_c55_rzr_stop_stream(rzr);
>>> +	mali_c55_cap_dev_stream_disable(cap_dev);
>>> +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
>>> +	video_device_pipeline_stop(&cap_dev->vdev);
>>> +	pm_runtime_put(mali_c55->dev);
> I think runtime PM autosuspend would be very useful, as it will ensure
> that stop-reconfigure-start cycles get handled as efficiently as
> possible without powering the device down. It could be done on top as a
> separate patch.
Alright
>>> +}
>>> +
>>> +static const struct vb2_ops mali_c55_vb2_ops = {
>>> +	.queue_setup		= &mali_c55_vb2_queue_setup,
>>> +	.buf_queue		= &mali_c55_buf_queue,
>>> +	.buf_init		= &mali_c55_buf_init,
>>> +	.wait_prepare		= vb2_ops_wait_prepare,
>>> +	.wait_finish		= vb2_ops_wait_finish,
>>> +	.start_streaming	= &mali_c55_vb2_start_streaming,
>>> +	.stop_streaming		= &mali_c55_vb2_stop_streaming,
>>> +};
>>> +
>>> +static const struct v4l2_file_operations mali_c55_v4l2_fops = {
>>> +	.owner = THIS_MODULE,
>>> +	.unlocked_ioctl = video_ioctl2,
>>> +	.open = v4l2_fh_open,
>>> +	.release = vb2_fop_release,
>>> +	.poll = vb2_fop_poll,
>>> +	.mmap = vb2_fop_mmap,
>>> +};
>>> +
>>> +static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
>>> +{
>>> +	const struct mali_c55_fmt *capture_format;
>>> +	const struct v4l2_format_info *info;
>>> +	struct v4l2_plane_pix_format *plane;
>>> +	unsigned int i;
>>> +
>>> +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
>>> +	pix_mp->pixelformat = capture_format->fourcc;
>>> +
>>> +	pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
>>> +			      MALI_C55_MAX_WIDTH);
>>> +	pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
>>> +			       MALI_C55_MAX_HEIGHT);
> Ah, these clamps are right :-)
Hurrah!
>
>>> +
>>> +	pix_mp->field = V4L2_FIELD_NONE;
>>> +	pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
>>> +	pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>> +	pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
>>> +
>>> +	info = v4l2_format_info(pix_mp->pixelformat);
> This function may return NULL. That shouldn't be the case as long as it
> supports all formats that the C55 driver supports, so I suppose it's
> safe.
>
>>> +	pix_mp->num_planes = info->mem_planes;
>>> +	memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
>>> +
>>> +	pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;
> Does the hardware support configurable line strides ? If so we should
> support it.


You have to set the line stride in the DMA writer registers, which we do using this same 
value...might userspace have set bytesperline already then or something? Or is there some other 
place it could be configured?

>
>>> +	pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
>>> +				       * pix_mp->height;
> 	pix_mp->plane_fmt[0].sizeimage = pix_mp->plane_fmt[0].bytesperline
> 				       * pix_mp->height;
>
>>> +
>>> +	for (i = 1; i < info->comp_planes; i++) {
>>> +		plane = &pix_mp->plane_fmt[i];
>>> +
>>> +		plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
>>> +						   info->hdiv);
>>> +		plane->sizeimage = DIV_ROUND_UP(
>>> +					plane->bytesperline * pix_mp->height,
>>> +					info->vdiv);
>>> +	}
>>> +
>>> +	if (info->mem_planes == 1) {
>>> +		for (i = 1; i < info->comp_planes; i++) {
>>> +			plane = &pix_mp->plane_fmt[i];
>>> +			pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
>>> +		}
>>> +	}
> I'm wondering, could v4l2_fill_pixfmt_mp() help ? It doesn't support
> configurable strides though :-S Maybe the helper could be improved, if
> it's close enough to what you need ?

I'll take a look
>>> +}
>>> +
>>> +static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
>>> +					   struct v4l2_format *f)
>>> +{
>>> +	mali_c55_try_fmt(&f->fmt.pix_mp);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
>>> +				struct v4l2_pix_format_mplane *pix_mp)
>>> +{
>>> +	const struct mali_c55_fmt *capture_format;
>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>> +	const struct v4l2_format_info *info;
>>> +
>>> +	mali_c55_try_fmt(pix_mp);
>>> +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
>>> +	info = v4l2_format_info(pix_mp->pixelformat);
>>> +	if (WARN_ON(!info))
>>> +		return;
>>> +
>>> +	mali_c55_write(mali_c55,
>>> +		       MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>> +		       capture_format->registers.base_mode);
>>> +	mali_c55_write(mali_c55,
>>> +		       MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
>>> +		       MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
>>> +		       MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
> Could the register writes be moved to stream start time ?
>
>>> +
>>> +	if (info->mem_planes > 1) {
>>> +		mali_c55_write(mali_c55,
>>> +			       MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>> +			       capture_format->registers.base_mode);
>>> +		mali_c55_update_bits(mali_c55,
>>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>> +				MALI_C55_WRITER_SUBMODE_MASK,
>>> +				capture_format->registers.uv_plane);
>>> +
>>> +		mali_c55_write(mali_c55,
>>> +			MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
>>> +			MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
>>> +			MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
>>> +	}
>>> +
>>> +	if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
>>> +		/*
>>> +		 * TODO: Figure out the colour matrix coefficients and calculate
>>> +		 * and write them here.
>>> +		 */
> Ideally they should also be exposed directly to userspace as ISP
> parameters. I would probably go as far as saying that they should come
> directly from userspace, and not derived from the colorspace fields.


Yes I think I agree, I'll drop the todo from here.

>
>>> +
>>> +		mali_c55_write(mali_c55,
>>> +			       MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>> +			       MALI_C55_CS_CONV_MATRIX_MASK);
>>> +
>>> +		if (info->hdiv > 1)
>>> +			mali_c55_update_bits(mali_c55,
>>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>> +				MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
>>> +		if (info->vdiv > 1)
>>> +			mali_c55_update_bits(mali_c55,
>>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>> +				MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
>>> +		if (info->hdiv > 1 || info->vdiv > 1)
>>> +			mali_c55_update_bits(mali_c55,
>>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>> +				MALI_C55_CS_CONV_FILTER_MASK, 0x01);
>>> +	}
>>> +
>>> +	cap_dev->mode.pix_mp = *pix_mp;
>>> +	cap_dev->mode.capture_fmt = capture_format;
>>> +}
>>> +
>>> +static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
>>> +					 struct v4l2_format *f)
>>> +{
>>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>> +
>>> +	if (vb2_is_busy(&cap_dev->queue))
>>> +		return -EBUSY;
>>> +
>>> +	mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
>>> +					 struct v4l2_format *f)
>>> +{
>>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>> +
>>> +	f->fmt.pix_mp = cap_dev->mode.pix_mp;
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
>>> +					    struct v4l2_fmtdesc *f)
>>> +{
>>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>> +	unsigned int j = 0;
>>> +	unsigned int i;
>>> +
>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>> +		if (f->mbus_code &&
>>> +		    !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
>>> +						       f->mbus_code))
> Small indentation mistake.
>
>>> +			continue;
>>> +
>>> +		/* Downscale pipe can't output RAW formats */
>>> +		if (mali_c55_fmts[i].is_raw &&
>>> +		    cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
>>> +			continue;
>>> +
>>> +		if (j++ == f->index) {
>>> +			f->pixelformat = mali_c55_fmts[i].fourcc;
>>> +			return 0;
>>> +		}
>>> +	}
>>> +
>>> +	return -EINVAL;
>>> +}
>>> +
>>> +static int mali_c55_querycap(struct file *file, void *fh,
>>> +			     struct v4l2_capability *cap)
>>> +{
>>> +	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
>>> +	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
>>> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
>>> +	.vidioc_querybuf = vb2_ioctl_querybuf,
>>> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
>>> +	.vidioc_qbuf = vb2_ioctl_qbuf,
>>> +	.vidioc_expbuf = vb2_ioctl_expbuf,
>>> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
>>> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
>>> +	.vidioc_streamon = vb2_ioctl_streamon,
>>> +	.vidioc_streamoff = vb2_ioctl_streamoff,
>>> +	.vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
>>> +	.vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
>>> +	.vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
>>> +	.vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
>>> +	.vidioc_querycap = mali_c55_querycap,
>>> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
>>> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
>>> +};
>>> +
>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
>>> +{
>>> +	struct v4l2_pix_format_mplane pix_mp;
>>> +	struct mali_c55_cap_dev *cap_dev;
>>> +	struct video_device *vdev;
>>> +	struct vb2_queue *vb2q;
>>> +	unsigned int i;
>>> +	int ret;
>>> +
>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
> Moving the inner content to a separate mali_c55_register_capture_dev()
> function would increase readability I think, and remove usage of gotos.
> I would probably do the same for unregistration too, for consistency.
>
>>> +		cap_dev = &mali_c55->cap_devs[i];
>>> +		vdev = &cap_dev->vdev;
>>> +		vb2q = &cap_dev->queue;
>>> +
>>> +		/*
>>> +		 * The downscale output pipe is an optional block within the ISP
>>> +		 * so we need to check whether it's actually been fitted or not.
>>> +		 */
>>> +
>>> +		if (i == MALI_C55_CAP_DEV_DS &&
>>> +		    !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
>>> +			continue;
> Given that there's only two capture devices, and one is optional, when
> moving the inner code to a separate function you could unroll the loop.
> Up to you.
>
>>> +
>>> +		cap_dev->mali_c55 = mali_c55;
>>> +		mutex_init(&cap_dev->lock);
>>> +		INIT_LIST_HEAD(&cap_dev->buffers.queue);
>>> +
>>> +		switch (i) {
>>> +		case MALI_C55_CAP_DEV_FR:
>>> +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
>>> +			cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
>>> +			break;
>>> +		case MALI_C55_CAP_DEV_DS:
>>> +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
>>> +			cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
>>> +			break;
>>> +		default:
> That can't happen.
>
>>> +			mutex_destroy(&cap_dev->lock);
>>> +			ret = -EINVAL;
>>> +			goto err_destroy_mutex;
>>> +		}
>>> +
>>> +		cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
>>> +		ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
>>> +		if (ret) {
>>> +			mutex_destroy(&cap_dev->lock);
>>> +			goto err_destroy_mutex;
>>> +		}
>>> +
>>> +		vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>>> +		vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
>>> +		vb2q->drv_priv = cap_dev;
>>> +		vb2q->mem_ops = &vb2_dma_contig_memops;
>>> +		vb2q->ops = &mali_c55_vb2_ops;
>>> +		vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
>>> +		vb2q->min_queued_buffers = 1;
>>> +		vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
>>> +		vb2q->lock = &cap_dev->lock;
>>> +		vb2q->dev = mali_c55->dev;
>>> +
>>> +		ret = vb2_queue_init(vb2q);
>>> +		if (ret) {
>>> +			dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
>>> +				mali_c55_cap_dev_to_name(cap_dev));
>>> +			goto err_cleanup_media_entity;
>>> +		}
>>> +
>>> +		strscpy(cap_dev->vdev.name, capture_device_names[i],
>>> +			sizeof(cap_dev->vdev.name));
>>> +		vdev->release = video_device_release_empty;
>>> +		vdev->fops = &mali_c55_v4l2_fops;
>>> +		vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
>>> +		vdev->lock = &cap_dev->lock;
>>> +		vdev->v4l2_dev = &mali_c55->v4l2_dev;
>>> +		vdev->queue = &cap_dev->queue;
>>> +		vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>>> +				    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
>>> +		vdev->entity.ops = &mali_c55_media_ops;
>>> +		video_set_drvdata(vdev, cap_dev);
>>> +
>>> +		memset(&pix_mp, 0, sizeof(pix_mp));
>>> +		pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
>>> +		pix_mp.width = MALI_C55_DEFAULT_WIDTH;
>>> +		pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
>>> +		mali_c55_set_format(cap_dev, &pix_mp);
>>> +
>>> +		ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
>>> +		if (ret) {
>>> +			dev_err(mali_c55->dev,
>>> +				"%s failed to register video device\n",
>>> +				mali_c55_cap_dev_to_name(cap_dev));
>>> +			goto err_release_vb2q;
>>> +		}
>>> +	}
>>> +
>>> +	return 0;
>>> +
>>> +err_release_vb2q:
>>> +	vb2_queue_release(vb2q);
>>> +err_cleanup_media_entity:
>>> +	media_entity_cleanup(&cap_dev->vdev.entity);
>>> +err_destroy_mutex:
>>> +	mutex_destroy(&cap_dev->lock);
>>> +	mali_c55_unregister_capture_devs(mali_c55);
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
>>> +{
>>> +	struct mali_c55_cap_dev *cap_dev;
>>> +	unsigned int i;
>>> +
>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
>>> +		cap_dev = &mali_c55->cap_devs[i];
>>> +
>>> +		if (!video_is_registered(&cap_dev->vdev))
>>> +			continue;
>>> +
>>> +		vb2_video_unregister_device(&cap_dev->vdev);
>>> +		media_entity_cleanup(&cap_dev->vdev.entity);
>>> +		mutex_destroy(&cap_dev->lock);
>>> +	}
>>> +}
>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>> new file mode 100644
>>> index 000000000000..2d0c4d152beb
>>> --- /dev/null
>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>> @@ -0,0 +1,266 @@
>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>> +/*
>>> + * ARM Mali-C55 ISP Driver - Common definitions
>>> + *
>>> + * Copyright (C) 2024 Ideas on Board Oy
>>> + */
>>> +
>>> +#ifndef _MALI_C55_COMMON_H
>>> +#define _MALI_C55_COMMON_H
>>> +
>>> +#include <linux/clk.h>
>>> +#include <linux/io.h>
>>> +#include <linux/list.h>
>>> +#include <linux/mutex.h>
>>> +#include <linux/scatterlist.h>
>> I don't think this is needed. You're however missing spinlock.h.
>>
>>> +#include <linux/videodev2.h>
>>> +
>>> +#include <media/media-device.h>
>>> +#include <media/v4l2-async.h>
>>> +#include <media/v4l2-ctrls.h>
>>> +#include <media/v4l2-dev.h>
>>> +#include <media/v4l2-device.h>
>>> +#include <media/v4l2-subdev.h>
>>> +#include <media/videobuf2-core.h>
>>> +#include <media/videobuf2-v4l2.h>
>>> +
>>> +#define MALI_C55_DRIVER_NAME		"mali-c55"
>>> +
>>> +/* min and max values for the image sizes */
>>> +#define MALI_C55_MIN_WIDTH		640U
>>> +#define MALI_C55_MIN_HEIGHT		480U
>>> +#define MALI_C55_MAX_WIDTH		8192U
>>> +#define MALI_C55_MAX_HEIGHT		8192U
>>> +#define MALI_C55_DEFAULT_WIDTH		1920U
>>> +#define MALI_C55_DEFAULT_HEIGHT		1080U
>>> +
>>> +#define MALI_C55_DEFAULT_MEDIA_BUS_FMT	MEDIA_BUS_FMT_RGB121212_1X36
>>> +
>>> +struct mali_c55;
>>> +struct mali_c55_cap_dev;
>>> +struct platform_device;
>> You should also forward-declare
>>
>> struct device;
>> struct dma_chan;
>> struct resource;
>>
>>> +
>>> +static const char * const mali_c55_clk_names[] = {
>>> +	"aclk",
>>> +	"hclk",
>>> +};
>> This will end up duplicating the array in each compilation unit, not
>> great. Move it to mali-c55-core.c. You use it in this file just for its
>> size, replace that with a macro that defines the size, or allocate
>> mali_c55.clks dynamically with devm_kcalloc().
>>
>>> +
>>> +enum mali_c55_interrupts {
>>> +	MALI_C55_IRQ_ISP_START,
>>> +	MALI_C55_IRQ_ISP_DONE,
>>> +	MALI_C55_IRQ_MCM_ERROR,
>>> +	MALI_C55_IRQ_BROKEN_FRAME_ERROR,
>>> +	MALI_C55_IRQ_MET_AF_DONE,
>>> +	MALI_C55_IRQ_MET_AEXP_DONE,
>>> +	MALI_C55_IRQ_MET_AWB_DONE,
>>> +	MALI_C55_IRQ_AEXP_1024_DONE,
>>> +	MALI_C55_IRQ_IRIDIX_MET_DONE,
>>> +	MALI_C55_IRQ_LUT_INIT_DONE,
>>> +	MALI_C55_IRQ_FR_Y_DONE,
>>> +	MALI_C55_IRQ_FR_UV_DONE,
>>> +	MALI_C55_IRQ_DS_Y_DONE,
>>> +	MALI_C55_IRQ_DS_UV_DONE,
>>> +	MALI_C55_IRQ_LINEARIZATION_DONE,
>>> +	MALI_C55_IRQ_RAW_FRONTEND_DONE,
>>> +	MALI_C55_IRQ_NOISE_REDUCTION_DONE,
>>> +	MALI_C55_IRQ_IRIDIX_DONE,
>>> +	MALI_C55_IRQ_BAYER2RGB_DONE,
>>> +	MALI_C55_IRQ_WATCHDOG_TIMER,
>>> +	MALI_C55_IRQ_FRAME_COLLISION,
>>> +	MALI_C55_IRQ_UNUSED,
>>> +	MALI_C55_IRQ_DMA_ERROR,
>>> +	MALI_C55_IRQ_INPUT_STOPPED,
>>> +	MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
>>> +	MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
>>> +	MALI_C55_NUM_IRQ_BITS
>> Those are register bits, I think they belong to mali-c55-registers.h,
>> and should probably be macros instead of an enum.
>>
>>> +};
>>> +
>>> +enum mali_c55_isp_pads {
>>> +	MALI_C55_ISP_PAD_SINK_VIDEO,
>> As there's a single sink pad, maybe MALI_C55_ISP_PAD_SINK ? Ah, you're
>> probably preparing for ISP parameters support. It's fine.
>>
>>> +	MALI_C55_ISP_PAD_SOURCE,
>> Then maybe this should be named MALI_C55_ISP_PAD_SOURCE_VIDEO as I
>> assume there will be a stats source pad.
>>
>>> +	MALI_C55_ISP_PAD_SOURCE_BYPASS,
>>> +	MALI_C55_ISP_NUM_PADS,
>>> +};
>>> +
>>> +struct mali_c55_tpg {
>>> +	struct mali_c55 *mali_c55;
>>> +	struct v4l2_subdev sd;
>>> +	struct media_pad pad;
>>> +	struct mutex lock;
>>> +	struct mali_c55_tpg_ctrls {
>>> +		struct v4l2_ctrl_handler handler;
>>> +		struct v4l2_ctrl *test_pattern;
>> Set but never used. You can drop it.
>>
>>> +		struct v4l2_ctrl *hblank;
>> Set and used only once, in the same function. You can make it a local
>> variable.
>>
>>> +		struct v4l2_ctrl *vblank;
>>> +	} ctrls;
>>> +};
>> I wonder if this file should be split, with mali-c55-capture.h,
>> mali-c55-core.h, mali-c55-isp.h, ... I think it could increase
>> readability by clearly separating the different elements. Up to you.
>>
>>> +
>>> +struct mali_c55_isp {
>>> +	struct mali_c55 *mali_c55;
>>> +	struct v4l2_subdev sd;
>>> +	struct media_pad pads[MALI_C55_ISP_NUM_PADS];
>>> +	struct media_pad *remote_src;
>>> +	struct v4l2_async_notifier notifier;
>> I'm tempted to move the notifier to mali_c55, as it's related to
>> components external to the whole ISP, not to the ISP subdev itself.
>> Could you give it a try, to see if it could be done without any drawback
>> ?
>>
>>> +	struct mutex lock;
>> Locks require a comment to explain what they protect. Same below where
>> applicable (for both mutexes and spinlocks).
>>
>>> +	unsigned int frame_sequence;
>>> +};
>>> +
>>> +enum mali_c55_resizer_ids {
>>> +	MALI_C55_RZR_FR,
>>> +	MALI_C55_RZR_DS,
>>> +	MALI_C55_NUM_RZRS,
>> The usual abbreviation for "resizer" in drivers/media/ is "rsz", not
>> "rzr". I would have said we can leave it as-is as changing it would be a
>> bit annoying, but I then realized that "rzr" is not just unusual, it's
>> actually not used at all. Would you mind applying a sed globally ? :-)
>>
>>> +};
>>> +
>>> +enum mali_c55_rzr_pads {
>> Same enums/structs use abbreviations, some don't. Consistency would
>> help.
>>
>>> +	MALI_C55_RZR_SINK_PAD,
>>> +	MALI_C55_RZR_SOURCE_PAD,
>>> +	MALI_C55_RZR_SINK_BYPASS_PAD,
>>> +	MALI_C55_RZR_NUM_PADS
>>> +};
>>> +
>>> +struct mali_c55_resizer {
>>> +	struct mali_c55 *mali_c55;
>>> +	struct mali_c55_cap_dev *cap_dev;
>>> +	enum mali_c55_resizer_ids id;
>>> +	struct v4l2_subdev sd;
>>> +	struct media_pad pads[MALI_C55_RZR_NUM_PADS];
>>> +	unsigned int num_routes;
>>> +	bool streaming;
>>> +};
>>> +
>>> +enum mali_c55_cap_devs {
>>> +	MALI_C55_CAP_DEV_FR,
>>> +	MALI_C55_CAP_DEV_DS,
>>> +	MALI_C55_NUM_CAP_DEVS
>>> +};
>>> +
>>> +struct mali_c55_fmt {
>> mali_c55_format_info would be a better name I think, as this stores
>> format information, not formats.
>>
>>> +	u32 fourcc;
>>> +	unsigned int mbus_codes[2];
>> A comment to explain why we have two media bus codes would be useful.
>> You can document the whole structure if desired :-)
>>
>>> +	bool is_raw;
>>> +	struct mali_c55_fmt_registers {
>> Make it an anonymous structure, it's never used anywhere else.
>>
>>> +		unsigned int base_mode;
>>> +		unsigned int uv_plane;
>> If those are register field values, use u32 instead of unsigned int.
>>
>>> +	} registers;
>> It's funny, we tend to abbreviate different things, I would have used
>> "regs" here but written "format" in full in the structure name :-)
>>
>>> +};
>>> +
>>> +enum mali_c55_isp_bayer_order {
>>> +	MALI_C55_BAYER_ORDER_RGGB,
>>> +	MALI_C55_BAYER_ORDER_GRBG,
>>> +	MALI_C55_BAYER_ORDER_GBRG,
>>> +	MALI_C55_BAYER_ORDER_BGGR
>> These are registers values too, they belong to mali-c55-registers.h.
>>
>>> +};
>>> +
>>> +struct mali_c55_isp_fmt {
>> mali_c55_isp_format_info
>>
>>> +	u32 code;
>>> +	enum v4l2_pixel_encoding encoding;
>> Here you use v4l2_pixel_encoding, for mali_c55_fmt you use is_raw. Maybe
>> pick the same option for both structures ?
>>
>>> +	enum mali_c55_isp_bayer_order order;
>>> +};
>>> +
>>> +enum mali_c55_planes {
>>> +	MALI_C55_PLANE_Y,
>>> +	MALI_C55_PLANE_UV,
>>> +	MALI_C55_NUM_PLANES
>>> +};
>>> +
>>> +struct mali_c55_buffer {
>>> +	struct vb2_v4l2_buffer vb;
>>> +	bool plane_done[MALI_C55_NUM_PLANES];
>> I think tracking the pending state would simplify the logic in
>> mali_c55_set_plane_done(), which would become
>>
>> 	curr_buf->plane_pending[plane] = false;
>>
>> 	if (!curr_buf->plane_pending[0] && !curr_buf->plane_pending[1]) {
>> 		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
>> 		cap_dev->buffers.curr = NULL;
>> 		isp->frame_sequence++;
>> 	}
>>
>> Or a counter may be even easier (and would consume less memory).
>>
>>> +	struct list_head queue;
>>> +	u32 addrs[MALI_C55_NUM_PLANES];
>> This stores DMA addresses, use dma_addr_t.
>>
>>> +};
>>> +
>>> +struct mali_c55_cap_dev {
>>> +	struct mali_c55 *mali_c55;
>>> +	struct mali_c55_resizer *rzr;
>>> +	struct video_device vdev;
>>> +	struct media_pad pad;
>>> +	struct vb2_queue queue;
>>> +	struct mutex lock;
>>> +	unsigned int reg_offset;
>> Manual handling of the offset everywhere, with parametric macros for the
>> resizer register addresses, isn't very nice. Introduce resizer-specific
>> accessors and capture-specific accessors (mali_c55_rsz_write(), ...)
>> that take a mali_c55_resizer or mali_c55_cap_dev pointer, and handle the
>> offset there. The register macros should loose their offset parameter.
>>
>> You could also use a single set of accessors that would become
>> path/output-specific (mali_c55_path_write() ? mali_c55_output_write()
>> ?), that may make the code easier to read.
>>
>> You can also replace reg_offset with a void __iomem * base, which would
>> avoid the computation at runtime.
>>
>>> +
>>> +	struct mali_c55_mode {
>> Make the structure anonymous.
>>
>>> +		const struct mali_c55_fmt *capture_fmt;
>>> +		struct v4l2_pix_format_mplane pix_mp;
>>> +	} mode;
>> What's a "mode" ? I think I'd name this
>>
>> 	struct {
>> 		const struct mali_c55_fmt *info;
>> 		struct v4l2_pix_format_mplane format;
>> 	} format;
>>
>> Or you could just drop the structure and have
>>
>> 	const struct mali_c55_fmt *format_info;
>> 	struct v4l2_pix_format_mplane format;
>>
>> or something similar.
>>
>>> +
>>> +	struct {
>>> +		spinlock_t lock;
>>> +		struct list_head queue;
>>> +		struct mali_c55_buffer *curr;
>>> +		struct mali_c55_buffer *next;
>>> +	} buffers;
>>> +
>>> +	bool streaming;
>>> +};
>>> +
>>> +enum mali_c55_config_spaces {
>>> +	MALI_C55_CONFIG_PING,
>>> +	MALI_C55_CONFIG_PONG,
>>> +	MALI_C55_NUM_CONFIG_SPACES
>> The last enumerator is not used.
>>
>>> +};
>>> +
>>> +struct mali_c55_ctx {
>> mali_c55_context ?
>>
>>> +	struct mali_c55 *mali_c55;
>>> +	void *registers;
>> Please document this structure and explain that this field points to a
>> copy of the register space in system memory, I was about to write you're
>> missing __iomem :-)
>>
>>> +	phys_addr_t base;
>>> +	spinlock_t lock;
>>> +	struct list_head list;
>>> +};
>>> +
>>> +struct mali_c55 {
>>> +	struct device *dev;
>>> +	struct resource *res;
>> You could possibly drop this field by passing the physical address of
>> the register space from mali_c55_probe() to mali_c55_init_context() as a
>> function parameter.
>>
>>> +	void __iomem *base;
>>> +	struct dma_chan *channel;
>>> +	struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
>>> +
>>> +	u16 capabilities;
>>> +	struct media_device media_dev;
>>> +	struct v4l2_device v4l2_dev;
>>> +	struct media_pipeline pipe;
>>> +
>>> +	struct mali_c55_tpg tpg;
>>> +	struct mali_c55_isp isp;
>>> +	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
>>> +	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
>>> +
>>> +	struct list_head contexts;
>>> +	enum mali_c55_config_spaces next_config;
>>> +};
>>> +
>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
>>> +		  bool force_hardware);
>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
>>> +			  u32 mask, u32 val);
>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
>>> +			  enum mali_c55_config_spaces cfg_space);
>>> +
>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55);
>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55);
>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55);
>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>>> +			     enum mali_c55_planes plane);
>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
>>> +
>>> +bool mali_c55_format_is_raw(unsigned int mbus_code);
>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
>>> +
>>> +const struct mali_c55_isp_fmt *
>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
>>> +#define for_each_mali_isp_fmt(fmt)\
>> #define for_each_mali_isp_fmt(fmt) \
>>
>>> +	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
>> Looks like parentheses were on sale :-)
>>
>> 	for ((fmt) = NULL; (fmt) = mali_c55_isp_fmt_next(fmt); )
>>
>> This macro is used in two places only, in the mali-c55-isp.c file where
>> open-coding the loop without using mali_c55_isp_fmt_next() would be more
>> efficient, and in mali-c55-resizer.c where a function to return format
>> i'th would be more efficient. I think you can drop the macro and the
>> mali_c55_isp_fmt_next() function.
>>
>>> +
>>> +#endif /* _MALI_C55_COMMON_H */
>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>> new file mode 100644
>>> index 000000000000..50caf5ee7474
>>> --- /dev/null
>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>> @@ -0,0 +1,767 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +/*
>>> + * ARM Mali-C55 ISP Driver - Core driver code
>>> + *
>>> + * Copyright (C) 2024 Ideas on Board Oy
>>> + */
>>> +
>>> +#include <linux/bitops.h>
>>> +#include <linux/cleanup.h>
>>> +#include <linux/clk.h>
>>> +#include <linux/delay.h>
>>> +#include <linux/device.h>
>>> +#include <linux/dmaengine.h>
>>> +#include <linux/dma-mapping.h>
>>> +#include <linux/interrupt.h>
>>> +#include <linux/iopoll.h>
>>> +#include <linux/ioport.h>
>>> +#include <linux/mod_devicetable.h>
>>> +#include <linux/of.h>
>>> +#include <linux/of_reserved_mem.h>
>>> +#include <linux/platform_device.h>
>>> +#include <linux/pm_runtime.h>
>>> +#include <linux/scatterlist.h>
>> I don't think this is needed.
>>
>> Missing slab.h.
>>
>>> +#include <linux/string.h>
>>> +
>>> +#include <media/media-entity.h>
>>> +#include <media/v4l2-device.h>
>>> +#include <media/videobuf2-dma-contig.h>
>>> +
>>> +#include "mali-c55-common.h"
>>> +#include "mali-c55-registers.h"
>>> +
>>> +static const char * const mali_c55_interrupt_names[] = {
>>> +	[MALI_C55_IRQ_ISP_START] = "ISP start",
>>> +	[MALI_C55_IRQ_ISP_DONE] = "ISP done",
>>> +	[MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
>>> +	[MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
>>> +	[MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
>>> +	[MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
>>> +	[MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
>>> +	[MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
>>> +	[MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
>>> +	[MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
>>> +	[MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
>>> +	[MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
>>> +	[MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
>>> +	[MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
>>> +	[MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
>>> +	[MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
>>> +	[MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
>>> +	[MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
>>> +	[MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
>>> +	[MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
>>> +	[MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
>>> +	[MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
>>> +	[MALI_C55_IRQ_DMA_ERROR] = "DMA error",
>>> +	[MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
>>> +	[MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
>>> +	[MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
>>> +};
>>> +
>>> +static unsigned int config_space_addrs[] = {
>> const
>>
>>> +	[MALI_C55_CONFIG_PING] = 0x0AB6C,
>>> +	[MALI_C55_CONFIG_PONG] = 0x22B2C,
>> Lowercase hex constants.
>>
>> Don't the values belong to mali-c55-registers.h ?
>>
>>> +};
>>> +
>>> +/* System IO
>> /*
>>   * System IO
>>
>>> + *
>>> + * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
>>> + * and 'pong'), with the  expectation that the 'active' space will be left
>> s/the  /the /
>>
>>> + * untouched whilst a frame is being processed and the 'inactive' space
>>> + * configured ready to be passed during the blanking period before the next
>> s/to be passed/to be switched to/ ?
>>
>>> + * frame processing starts. These spaces should ideally be set via DMA transfer
>>> + * from a buffer rather than through individual register set operations. There
>>> + * is also a shared global register space which should be set normally. Of
>>> + * course, the ISP might be included in a system which lacks a suitable DMA
>>> + * engine, and the second configuration space might not be fitted at all, which
>>> + * means we need to support four scenarios:
>>> + *
>>> + * 1. Multi config space, with DMA engine.
>>> + * 2. Multi config space, no DMA engine.
>>> + * 3. Single config space, with DMA engine.
>>> + * 4. Single config space, no DMA engine.
>>> + *
>>> + * The first case is very easy, but the rest present annoying problems. The best
>>> + * way to solve them seems to be simply to replicate the concept of DMAing over
>>> + * the configuration buffer even if there's no DMA engine on the board, for
>>> + * which we rely on memcpy. To facilitate this any read/write call that is made
>>> + * to an address within those config spaces should infact be directed to a
>>> + * buffer that was allocated to hold them rather than the IO memory itself. The
>>> + * actual copy of that buffer to IO mem will happen on interrupt.
>>> + */
>>> +
>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
>>> +{
>>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>> +
>>> +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
>>> +		spin_lock(&ctx->lock);
>>> +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
>>> +		((u32 *)ctx->registers)[addr] = val;
>>> +		spin_unlock(&ctx->lock);
>>> +
>>> +		return;
>>> +	}
>> Ouch. This is likely the second comment you really won't like (after the
>> comment regarding mali_c55_update_bits() at the very top). I apologize
>> in advance.
>>
>> I really don't like this. Directing writes either to hardware registers
>> or to the shadow registers in the context makes the callers of the
>> read/write accessors very hard to read. The probe code, for instance,
>> mixes writes to hardware registers and writes to the context shadow
>> registers to initialize the value of some of the shadow registers.
>>
>> I'd like to split the read/write accessors into functions that access
>> the hardware registers (that's easy) and functions that access the
>> shadow registers. I think the latter should receive a mali_c55_ctx
>> pointer instead of a mali_c55 pointer to prepare for multi-context
>> support.
>>
>> You can add WARN_ON() guards to the two sets of functions, to ensure
>> that no register from the "other" space gets passed to the wrong
>> function by mistake.
>>
>>> +
>>> +	writel(val, mali_c55->base + addr);
>>> +}
>>> +
>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
>>> +		  bool force_hardware)
>> force_hardware is never set to true.
>>
>>> +{
>>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>> +	u32 val;
>>> +
>>> +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
>>> +		spin_lock(&ctx->lock);
>>> +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
>>> +		val = ((u32 *)ctx->registers)[addr];
>>> +		spin_unlock(&ctx->lock);
>>> +
>>> +		return val;
>>> +	}
>>> +
>>> +	return readl(mali_c55->base + addr);
>>> +}
>>> +
>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
>>> +			  u32 mask, u32 val)
>>> +{
>>> +	u32 orig, tmp;
>>> +
>>> +	orig = mali_c55_read(mali_c55, addr, false);
>>> +
>>> +	tmp = orig & ~mask;
>>> +	tmp |= (val << (ffs(mask) - 1)) & mask;
>>> +
>>> +	if (tmp != orig)
>>> +		mali_c55_write(mali_c55, addr, tmp);
>>> +}
>>> +
>>> +static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
>>> +			     dma_addr_t dst, enum dma_data_direction dir)
>>> +{
>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>>> +	struct dma_async_tx_descriptor *tx;
>>> +	enum dma_status status;
>>> +	dma_cookie_t cookie;
>>> +
>>> +	tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
>>> +				       MALI_C55_CONFIG_SPACE_SIZE, 0);
>>> +	if (!tx) {
>>> +		dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
>>> +		return -EIO;
>>> +	}
>>> +
>>> +	cookie = dmaengine_submit(tx);
>>> +	if (dma_submit_error(cookie)) {
>>> +		dev_err(mali_c55->dev, "error submitting dma transfer\n");
>>> +		return -EIO;
>>> +	}
>>> +
>>> +	status = dma_sync_wait(mali_c55->channel, cookie);
>> I've just realized this performs a busy-wait :-S See the comment in the
>> probe function about the threaded IRQ handler. I think we'll need to
>> rework all this. It could be done on top though.
>>
>>> +	if (status != DMA_COMPLETE) {
>>> +		dev_err(mali_c55->dev, "dma transfer failed\n");
>>> +		return -EIO;
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
>>> +			     enum mali_c55_config_spaces cfg_space)
>>> +{
>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>>> +	struct device *dma_dev = mali_c55->channel->device->dev;
>>> +	dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
>>> +	dma_addr_t dst;
>>> +	int ret;
>>> +
>>> +	guard(spinlock)(&ctx->lock);
>>> +
>>> +	dst = dma_map_single(dma_dev, ctx->registers,
>>> +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
>>> +	if (dma_mapping_error(dma_dev, dst)) {
>>> +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
>>> +		return -EIO;
>>> +	}
>>> +
>>> +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
>>> +	dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
>>> +			 DMA_FROM_DEVICE);
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
>>> +		       enum mali_c55_config_spaces cfg_space)
>>> +{
>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>>> +	struct device *dma_dev = mali_c55->channel->device->dev;
>>> +	dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
>>> +	dma_addr_t src;
>>> +	int ret;
>>> +
>>> +	guard(spinlock)(&ctx->lock);
>> The code below can take a large amount of time, holding a spinlock will
>> disable interrupts on the local CPU, that's not good :-(
>>
>>> +
>>> +	src = dma_map_single(dma_dev, ctx->registers,
>>> +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
>>> +	if (dma_mapping_error(dma_dev, src)) {
>>> +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
>>> +		return -EIO;
>>> +	}
>>> +
>>> +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
>>> +	dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
>>> +			 DMA_TO_DEVICE);
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +static int mali_c55_config_read(struct mali_c55_ctx *ctx,
>>> +				enum mali_c55_config_spaces cfg_space)
>>> +{
>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>>> +
>>> +	if (mali_c55->channel) {
>>> +		return mali_c55_dma_read(ctx, cfg_space);
>> As this function is used at probe time only, to initialize the context,
>> I think DMA is overkill.
>>
>>> +	} else {
>>> +		memcpy_fromio(ctx->registers,
>>> +			      mali_c55->base + config_space_addrs[cfg_space],
>>> +			      MALI_C55_CONFIG_SPACE_SIZE);
>>> +		return 0;
>>> +	}
>>> +}
>>> +
>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
>>> +			  enum mali_c55_config_spaces cfg_space)
>>> +{
>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
>>> +
>>> +	if (mali_c55->channel) {
>>> +		return mali_c55_dma_write(ctx, cfg_space);
>>> +	} else {
>>> +		memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
>>> +			    ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
>>> +		return 0;
>>> +	}
>> Could you measure the time it typically takes to write the registers
>> using DMA compared to using memcpy_toio() ?
>>
>>> +}
>>> +
>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
>>> +{
>>> +	return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);
>> I think it's too early to tell how multi-context support will look like.
>> I'm fine keeping mali_c55_get_active_context() as changing that would be
>> very intrusive (even if I think it will need to be changed), but the
>> list of contexts is neither the mechanism we'll use, nor something we
>> need now. Drop the list, embed the context in struct mali_c55, and
>> return the pointer to that single context from this function.
>>
>>> +}
>>> +
>>> +static void mali_c55_remove_links(struct mali_c55 *mali_c55)
>>> +{
>>> +	unsigned int i;
>>> +
>>> +	media_entity_remove_links(&mali_c55->tpg.sd.entity);
>>> +	media_entity_remove_links(&mali_c55->isp.sd.entity);
>>> +
>>> +	for (i = 0; i < MALI_C55_NUM_RZRS; i++)
>>> +		media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
>>> +
>>> +	for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
>>> +		media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
>>> +}
>>> +
>>> +static int mali_c55_create_links(struct mali_c55 *mali_c55)
>>> +{
>>> +	int ret;
>>> +
>>> +	/* Test pattern generator to ISP */
>>> +	ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
>>> +				    &mali_c55->isp.sd.entity,
>>> +				    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
>>> +	if (ret) {
>>> +		dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
>>> +		goto err_remove_links;
>>> +	}
>>> +
>>> +	/* Full resolution resizer pipe. */
>>> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>> +			MALI_C55_ISP_PAD_SOURCE,
>>> +			&mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
>>> +			MALI_C55_RZR_SINK_PAD,
>>> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>> +	if (ret) {
>>> +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
>>> +		goto err_remove_links;
>>> +	}
>>> +
>>> +	/* Full resolution bypass. */
>>> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>> +				    MALI_C55_ISP_PAD_SOURCE_BYPASS,
>>> +				    &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
>>> +				    MALI_C55_RZR_SINK_BYPASS_PAD,
>>> +				    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>> +	if (ret) {
>>> +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
>>> +		goto err_remove_links;
>>> +	}
>>> +
>>> +	/* Resizer pipe to video capture nodes. */
>>> +	ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
>>> +			MALI_C55_RZR_SOURCE_PAD,
>>> +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
>>> +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>> +	if (ret) {
>>> +		dev_err(mali_c55->dev,
>>> +			"failed to link FR resizer and video device\n");
>>> +		goto err_remove_links;
>>> +	}
>>> +
>>> +	/* The downscale pipe is an optional hardware block */
>>> +	if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
>>> +		ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>> +			MALI_C55_ISP_PAD_SOURCE,
>>> +			&mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
>>> +			MALI_C55_RZR_SINK_PAD,
>>> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>> +		if (ret) {
>>> +			dev_err(mali_c55->dev,
>>> +				"failed to link ISP and DS resizer\n");
>>> +			goto err_remove_links;
>>> +		}
>>> +
>>> +		ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
>>> +			MALI_C55_RZR_SOURCE_PAD,
>>> +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
>>> +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>> +		if (ret) {
>>> +			dev_err(mali_c55->dev,
>>> +				"failed to link DS resizer and video device\n");
>>> +			goto err_remove_links;
>>> +		}
>>> +	}
>>> +
>>> +	return 0;
>>> +
>>> +err_remove_links:
>>> +	mali_c55_remove_links(mali_c55);
>>> +	return ret;
>>> +}
>>> +
>>> +static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
>>> +{
>>> +	mali_c55_unregister_tpg(mali_c55);
>>> +	mali_c55_unregister_isp(mali_c55);
>>> +	mali_c55_unregister_resizers(mali_c55);
>>> +	mali_c55_unregister_capture_devs(mali_c55);
>>> +}
>>> +
>>> +static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>>> +{
>>> +	int ret;
>>> +
>>> +	ret = mali_c55_register_tpg(mali_c55);
>>> +	if (ret)
>>> +		return ret;
>>> +
>>> +	ret = mali_c55_register_isp(mali_c55);
>>> +	if (ret)
>>> +		goto err_unregister_entities;
>>> +
>>> +	ret = mali_c55_register_resizers(mali_c55);
>>> +	if (ret)
>>> +		goto err_unregister_entities;
>>> +
>>> +	ret = mali_c55_register_capture_devs(mali_c55);
>>> +	if (ret)
>>> +		goto err_unregister_entities;
>>> +
>>> +	ret = mali_c55_create_links(mali_c55);
>>> +	if (ret)
>>> +		goto err_unregister_entities;
>>> +
>>> +	return 0;
>>> +
>>> +err_unregister_entities:
>>> +	mali_c55_unregister_entities(mali_c55);
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
>>> +{
>>> +	u32 product, version, revision, capabilities;
>>> +
>>> +	product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
>>> +	version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
>>> +	revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
>>> +
>>> +	dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
>>> +		 product, version, revision);
>>> +
>>> +	capabilities = mali_c55_read(mali_c55,
>>> +				     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
>>> +				     false);
>>> +	mali_c55->capabilities = (capabilities & 0xffff);
>>> +
>>> +	/* TODO: Might as well start some debugfs */
>> If it's just to expose the version and capabilities, I think that's
>> overkill. It's not needed for debug purpose (you can get it from the
>> kernel log already). debugfs isn't meant to be accessible in production,
>> so an application that would need access to the information wouldn't be
>> able to use it.
>>
>>> +	dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);
>> Combine the two messages into one.
>>
>>> +	return version;
>>> +}
>>> +
>>> +static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
>>> +{
>>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>> +	u32 curr_config, next_config;
>>> +
>>> +	curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
>>> +	curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
>>> +		      >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
>>> +	next_config = curr_config ^ 1;
>>> +
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>> +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
>>> +	mali_c55_config_write(ctx, next_config ?
>>> +			      MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
>>> +}
>>> +
>>> +static irqreturn_t mali_c55_isr(int irq, void *context)
>>> +{
>>> +	struct device *dev = context;
>>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>> +	u32 interrupt_status;
>>> +	unsigned int i, j;
>>> +
>>> +	interrupt_status = mali_c55_read(mali_c55,
>>> +					 MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
>>> +					 false);
>>> +	if (!interrupt_status)
>>> +		return IRQ_NONE;
>>> +
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
>>> +		       interrupt_status);
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
>>> +
>>> +	for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
>>> +		if (!(interrupt_status & (1 << i)))
>>> +			continue;
>>> +
>>> +		switch (i) {
>>> +		case MALI_C55_IRQ_ISP_START:
>>> +			mali_c55_isp_queue_event_sof(mali_c55);
>>> +
>>> +			for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
>>> +				mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
>>> +
>>> +			mali_c55_swap_next_config(mali_c55);
>>> +
>>> +			break;
>>> +		case MALI_C55_IRQ_ISP_DONE:
>>> +			/*
>>> +			 * TODO: Where the ISP has no Pong config fitted, we'd
>>> +			 * have to do the mali_c55_swap_next_config() call here.
>>> +			 */
>>> +			break;
>>> +		case MALI_C55_IRQ_FR_Y_DONE:
>>> +			mali_c55_set_plane_done(
>>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
>>> +				MALI_C55_PLANE_Y);
>>> +			break;
>>> +		case MALI_C55_IRQ_FR_UV_DONE:
>>> +			mali_c55_set_plane_done(
>>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
>>> +				MALI_C55_PLANE_UV);
>>> +			break;
>>> +		case MALI_C55_IRQ_DS_Y_DONE:
>>> +			mali_c55_set_plane_done(
>>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
>>> +				MALI_C55_PLANE_Y);
>>> +			break;
>>> +		case MALI_C55_IRQ_DS_UV_DONE:
>>> +			mali_c55_set_plane_done(
>>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
>>> +				MALI_C55_PLANE_UV);
>>> +			break;
>>> +		default:
>>> +			/*
>>> +			 * Only the above interrupts are currently unmasked. If
>>> +			 * we receive anything else here then something weird
>>> +			 * has gone on.
>>> +			 */
>>> +			dev_err(dev, "masked interrupt %s triggered\n",
>>> +				mali_c55_interrupt_names[i]);
>>> +		}
>>> +	}
>>> +
>>> +	return IRQ_HANDLED;
>>> +}
>>> +
>>> +static int mali_c55_init_context(struct mali_c55 *mali_c55)
>>> +{
>>> +	struct mali_c55_ctx *ctx;
>>> +	int ret;
>>> +
>>> +	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
>>> +	if (!ctx) {
>>> +		dev_err(mali_c55->dev, "failed to allocate new context\n");
>> No need for an error message when memory allocation fails.
>>
>>> +		return -ENOMEM;
>>> +	}
>>> +
>>> +	ctx->base = mali_c55->res->start;
>>> +	ctx->mali_c55 = mali_c55;
>>> +
>>> +	ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
>>> +				 GFP_KERNEL | GFP_DMA);
>>> +	if (!ctx->registers) {
>>> +		ret = -ENOMEM;
>>> +		goto err_free_ctx;
>>> +	}
>>> +
>>> +	/*
>>> +	 * The allocated memory is empty, we need to load the default
>>> +	 * register settings. We just read Ping; it's identical to Pong.
>>> +	 */
>>> +	ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
>>> +	if (ret)
>>> +		goto err_free_registers;
>>> +
>>> +	list_add_tail(&ctx->list, &mali_c55->contexts);
>>> +
>>> +	/*
>>> +	 * Some features of the ISP need to be disabled by default and only
>>> +	 * enabled at the same time as they're configured by a parameters buffer
>>> +	 */
>>> +
>>> +	/* Bypass the sqrt and square compression and expansion modules */
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
>>> +			     MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
>>> +			     MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
>>> +
>>> +	/* Bypass the temper module */
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
>>> +		       MALI_C55_REG_BYPASS_2_TEMPER);
>>> +
>>> +	/* Bypass the colour noise reduction  */
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
>>> +		       MALI_C55_REG_BYPASS_4_CNR);
>>> +
>>> +	/* Disable the sinter module */
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
>>> +			     MALI_C55_SINTER_ENABLE_MASK, 0x00);
>>> +
>>> +	/* Disable the RGB Gamma module for each output */
>>> +	mali_c55_write(
>>> +		mali_c55,
>>> +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
>>> +		0x00);
>>> +	mali_c55_write(
>>> +		mali_c55,
>>> +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
>>> +		0x00);
>>> +
>>> +	/* Disable the colour correction matrix */
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
>>> +
>>> +	return 0;
>>> +
>>> +err_free_registers:
>>> +	kfree(ctx->registers);
>>> +err_free_ctx:
>>> +	kfree(ctx);
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +static int mali_c55_runtime_resume(struct device *dev)
>>> +{
>>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>> +	int ret;
>>> +
>>> +	ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
>>> +				      mali_c55->clks);
>>> +	if (ret)
>>> +		dev_err(mali_c55->dev, "failed to enable clocks\n");
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +static int mali_c55_runtime_suspend(struct device *dev)
>>> +{
>>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>> +
>>> +	clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
>>> +	return 0;
>>> +}
>>> +
>>> +static const struct dev_pm_ops mali_c55_pm_ops = {
>>> +	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
>>> +				pm_runtime_force_resume)
>>> +	SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
>>> +			   NULL)
>>> +};
>>> +
>>> +static int mali_c55_probe(struct platform_device *pdev)
>>> +{
>>> +	struct device *dev = &pdev->dev;
>>> +	struct mali_c55 *mali_c55;
>>> +	dma_cap_mask_t mask;
>>> +	u32 version;
>>> +	int ret;
>>> +	u32 val;
>>> +
>>> +	mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
>>> +	if (!mali_c55)
>>> +		return dev_err_probe(dev, -ENOMEM,
>>> +				     "failed to allocate memory\n");
>> 		return -ENOMEM;
>>
>> There's no need to print messages for memory allocation failures.
>>
>>> +
>>> +	mali_c55->dev = dev;
>>> +	platform_set_drvdata(pdev, mali_c55);
>>> +
>>> +	mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
>>> +								&mali_c55->res);
>>> +	if (IS_ERR(mali_c55->base))
>>> +		return dev_err_probe(dev, PTR_ERR(mali_c55->base),
>>> +				     "failed to map IO memory\n");
>>> +
>>> +	ret = platform_get_irq(pdev, 0);
>>> +	if (ret < 0)
>>> +		return dev_err_probe(dev, ret, "failed to get interrupt num\n");
>> s/ num// or s/num/number/
>>
>>> +
>>> +	ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
>>> +					mali_c55_isr, IRQF_ONESHOT,
>>> +					dev_driver_string(&pdev->dev),
>>> +					&pdev->dev);
>> Requested the IRQ should be done much lower, after you have initialized
>> everything, or an IRQ that would fire early would have really bad
>> consequences.
>>
>> A comment to explain why you need a threaded interrupt handler would be
>> good. I assume it is due only to the need to transfer the registers
>> using DMA. I wonder if we should then split the interrupt handler in
>> two, with a non-threaded part for the operations that can run quickly,
>> and a threaded part for the reprogramming.
>>
>> It may also be that we could just start the DMA transfer in the
>> non-threaded handler without waiting synchronously for it to complete.
>> That would be a bigger change, and would require checking race
>> conditions carefully. On the other hand, I'm a bit concerned about the
>> current implementation, have you tested what happens if the DMA transfer
>> takes too long to complete, and spans frame boundaries ? This part could
>> be addressed by a patch on top of this one.
>>
>>> +	if (ret)
>>> +		return dev_err_probe(dev, ret, "failed to request irq\n");
>>> +
>>> +	for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
>>> +		mali_c55->clks[i].id = mali_c55_clk_names[i];
>>> +
>>> +	ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
>>> +	if (ret)
>>> +		return dev_err_probe(dev, ret, "failed to acquire clocks\n");
>>> +
>>> +	pm_runtime_enable(&pdev->dev);
>>> +
>>> +	ret = pm_runtime_resume_and_get(&pdev->dev);
>>> +	if (ret)
>>> +		goto err_pm_runtime_disable;
>>> +
>>> +	of_reserved_mem_device_init(dev);
>> I'd move this, the vb2_dma_contig_set_max_seg_size() call and the
>> dma_cap_* calls before pm_runtime_enable() as they don't need the device
>> to be powered.
>>
>>> +	version = mali_c55_check_hwcfg(mali_c55);
>>> +	vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
>>> +
>>> +	/* Use "software only" context management. */
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>> +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
>>> +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
>> You handle that in mali_c55_isp_start(), does the register have to be
>> set here too ?
>>
>>> +
>>> +	dma_cap_zero(mask);
>>> +	dma_cap_set(DMA_MEMCPY, mask);
>>> +
>>> +	/*
>>> +	 * No error check, because we will just fallback on memcpy if there is
>>> +	 * no usable DMA channel on the system.
>>> +	 */
>>> +	mali_c55->channel = dma_request_channel(mask, NULL, NULL);
>>> +
>>> +	INIT_LIST_HEAD(&mali_c55->contexts);
>>> +	ret = mali_c55_init_context(mali_c55);
>>> +	if (ret)
>>> +		goto err_release_dma_channel;
>>> +
>> I'd move all the code from here ...
>>
>>> +	mali_c55->media_dev.dev = dev;
>>> +	strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
>>> +		sizeof(mali_c55->media_dev.model));
>>> +	mali_c55->media_dev.hw_revision = version;
>>> +
>>> +	media_device_init(&mali_c55->media_dev);
>>> +	ret = media_device_register(&mali_c55->media_dev);
>>> +	if (ret)
>>> +		goto err_cleanup_media_device;
>>> +
>>> +	mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
>>> +	ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
>>> +	if (ret) {
>>> +		dev_err(dev, "failed to register V4L2 device\n");
>>> +		goto err_unregister_media_device;
>>> +	};
>>> +
>>> +	ret = mali_c55_register_entities(mali_c55);
>>> +	if (ret) {
>>> +		dev_err(dev, "failed to register entities\n");
>>> +		goto err_unregister_v4l2_device;
>>> +	}
>> ... to here to a separate function, or maybe fold it all in
>> mali_c55_register_entities() (which should the be renamed). Same thing
>> for the cleanup code.
>>
>>> +
>>> +	/* Set safe stop to ensure we're in a non-streaming state */
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>>> +		       MALI_C55_INPUT_SAFE_STOP);
>>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>>> +
>>> +	/*
>>> +	 * We're ready to process interrupts. Clear any that are set and then
>>> +	 * unmask them for processing.
>>> +	 */
>>> +	mali_c55_write(mali_c55, 0x30, 0xffffffff);
>>> +	mali_c55_write(mali_c55, 0x34, 0xffffffff);
>>> +	mali_c55_write(mali_c55, 0x40, 0x01);
>>> +	mali_c55_write(mali_c55, 0x40, 0x00);
>> Please replace the register addresses with macros.
>>
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);
>> The value should use the interrupt bits macros.
>>
>>> +
>>> +	pm_runtime_put(&pdev->dev);
>> Once power gets cut, the registers your programmed above may be lost. I
>> think you should programe them in the runtime PM resume handler.
>>
>>> +
>>> +	return 0;
>>> +
>>> +err_unregister_v4l2_device:
>>> +	v4l2_device_unregister(&mali_c55->v4l2_dev);
>>> +err_unregister_media_device:
>>> +	media_device_unregister(&mali_c55->media_dev);
>>> +err_cleanup_media_device:
>>> +	media_device_cleanup(&mali_c55->media_dev);
>>> +err_release_dma_channel:
>>> +	dma_release_channel(mali_c55->channel);
>>> +err_pm_runtime_disable:
>>> +	pm_runtime_disable(&pdev->dev);
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +static void mali_c55_remove(struct platform_device *pdev)
>>> +{
>>> +	struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
>>> +	struct mali_c55_ctx *ctx, *tmp;
>>> +
>>> +	list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
>>> +		list_del(&ctx->list);
>>> +		kfree(ctx->registers);
>>> +		kfree(ctx);
>>> +	}
>>> +
>>> +	mali_c55_remove_links(mali_c55);
>>> +	mali_c55_unregister_entities(mali_c55);
>>> +	v4l2_device_put(&mali_c55->v4l2_dev);
>>> +	media_device_unregister(&mali_c55->media_dev);
>>> +	media_device_cleanup(&mali_c55->media_dev);
>>> +	dma_release_channel(mali_c55->channel);
>>> +}
>>> +
>>> +static const struct of_device_id mali_c55_of_match[] = {
>>> +	{ .compatible = "arm,mali-c55", },
>>> +	{},
>> 	{ /* Sentinel */ },
>>
>>> +};
>>> +MODULE_DEVICE_TABLE(of, mali_c55_of_match);
>>> +
>>> +static struct platform_driver mali_c55_driver = {
>>> +	.driver = {
>>> +		.name = "mali-c55",
>>> +		.of_match_table = of_match_ptr(mali_c55_of_match),
>> Drop of_match_ptr().
>>
>>> +		.pm = &mali_c55_pm_ops,
>>> +	},
>>> +	.probe = mali_c55_probe,
>>> +	.remove_new = mali_c55_remove,
>>> +};
>>> +
>>> +module_platform_driver(mali_c55_driver);
>> Blank line.
>>
>>> +MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
>>> +MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
>>> +MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
>>> +MODULE_LICENSE("GPL");
>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>> new file mode 100644
>>> index 000000000000..ea8b7b866e7a
>>> --- /dev/null
>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>> @@ -0,0 +1,611 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +/*
>>> + * ARM Mali-C55 ISP Driver - Image signal processor
>>> + *
>>> + * Copyright (C) 2024 Ideas on Board Oy
>>> + */
>>> +
>>> +#include <linux/delay.h>
>>> +#include <linux/iopoll.h>
>>> +#include <linux/property.h>
>>> +#include <linux/string.h>
>>> +
>>> +#include <media/media-entity.h>
>>> +#include <media/v4l2-common.h>
>>> +#include <media/v4l2-event.h>
>>> +#include <media/v4l2-mc.h>
>>> +#include <media/v4l2-subdev.h>
>>> +
>>> +#include "mali-c55-common.h"
>>> +#include "mali-c55-registers.h"
>>> +
>>> +static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
>>> +	{
>>> +		.code = MEDIA_BUS_FMT_SRGGB20_1X20,
>>> +		.order = MALI_C55_BAYER_ORDER_RGGB,
>>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>>> +	},
>>> +	{
>>> +		.code = MEDIA_BUS_FMT_SGRBG20_1X20,
>>> +		.order = MALI_C55_BAYER_ORDER_GRBG,
>>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>>> +	},
>>> +	{
>>> +		.code = MEDIA_BUS_FMT_SGBRG20_1X20,
>>> +		.order = MALI_C55_BAYER_ORDER_GBRG,
>>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>>> +	},
>>> +	{
>>> +		.code = MEDIA_BUS_FMT_SBGGR20_1X20,
>>> +		.order = MALI_C55_BAYER_ORDER_BGGR,
>>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
>>> +	},
>>> +	{
>>> +		.code = MEDIA_BUS_FMT_RGB202020_1X60,
>>> +		.order = 0, /* Not relevant for this format */
>>> +		.encoding = V4L2_PIXEL_ENC_RGB,
>>> +	}
>>> +	/*
>>> +	 * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
>>> +	 * also support YUV input from a sensor passed-through to the output. At
>>> +	 * present we have no mechanism to test that though so it may have to
>>> +	 * wait a while...
>>> +	 */
>>> +};
>>> +
>>> +const struct mali_c55_isp_fmt *
>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
>>> +{
>>> +	if (!fmt)
>>> +		fmt = &mali_c55_isp_fmts[0];
>>> +	else
>>> +		fmt++;
>>> +
>>> +	for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
>>> +		return fmt;
>> That's peculiar.
>>
>> 	if (!fmt)
>> 		fmt = &mali_c55_isp_fmts[0];
>> 	else if (fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)])
>> 		return ++fmt;
>> 	else
>> 		return NULL;
>>
>>> +
>>> +	return NULL;
>>> +}
>>> +
>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
>>> +{
>>> +	const struct mali_c55_isp_fmt *isp_fmt;
>>> +
>>> +	for_each_mali_isp_fmt(isp_fmt) {
>> I would open-code the loop instead of using the macro, like you do
>> below. It will be more efficient.
>>
>>> +		if (isp_fmt->code == mbus_code)
>>> +			return true;
>>> +	}
>>> +
>>> +	return false;
>>> +}
>>> +
>>> +static const struct mali_c55_isp_fmt *
>>> +mali_c55_isp_get_mbus_config_by_code(u32 code)
>>> +{
>>> +	unsigned int i;
>>> +
>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
>>> +		if (mali_c55_isp_fmts[i].code == code)
>>> +			return &mali_c55_isp_fmts[i];
>>> +	}
>>> +
>>> +	return NULL;
>>> +}
>>> +
>>> +static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
>>> +{
>>> +	u32 val;
>>> +
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);
>> 	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>> 		       MALI_C55_INPUT_SAFE_STOP);
>>
>>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>>> +}
>>> +
>>> +static int mali_c55_isp_start(struct mali_c55 *mali_c55)
>>> +{
>>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>> +	const struct mali_c55_isp_fmt *cfg;
>>> +	struct v4l2_mbus_framefmt *format;
>> const
>>
>>> +	struct v4l2_subdev_state *state;
>>> +	struct v4l2_rect *crop;
>> const
>>
>>> +	struct v4l2_subdev *sd;
>>> +	u32 val;
>>> +	int ret;
>>> +
>>> +	sd = &mali_c55->isp.sd;
>> Assign when declaring the variable.
>>
>>> +
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>> +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
>>> +
>>> +	/* Apply input windowing */
>>> +	state = v4l2_subdev_get_locked_active_state(sd);
>> Using .enable_streams() (see below) you'll get this for free.
>>
>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>> +	format = v4l2_subdev_state_get_format(state,
>>> +					      MALI_C55_ISP_PAD_SINK_VIDEO);
>>> +	cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
>>> +
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
>>> +		       MALI_C55_HC_START(crop->left));
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
>>> +		       MALI_C55_HC_SIZE(crop->width));
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
>>> +		       MALI_C55_VC_START(crop->top) |
>>> +		       MALI_C55_VC_SIZE(crop->height));
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
>>> +			     MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
>>> +			     MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
>>> +			     MALI_C55_BAYER_ORDER_MASK, cfg->order);
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
>>> +			     MALI_C55_INPUT_WIDTH_MASK,
>>> +			     MALI_C55_INPUT_WIDTH_20BIT);
>>> +
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>> +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
>>> +			     cfg->encoding == V4L2_PIXEL_ENC_RGB ?
>>> +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
>>> +
>>> +	ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
>>> +	if (ret) {
>>> +		dev_err(mali_c55->dev, "failed to DMA config\n");
>>> +		return ret;
>>> +	}
>>> +
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>>> +		       MALI_C55_INPUT_SAFE_START);
>>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>> Should you return an error in case of timeout ?
>>
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)
>> Why is this not handled wired to .s_stream() ? Or better,
>> .enable_streams() and .disable_streams().
>>
>>> +{
>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>>> +	struct v4l2_subdev *sd;
>>> +
>>> +	if (isp->remote_src) {
>>> +		sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
>>> +		v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
>>> +	}
>>> +	isp->remote_src = NULL;
>>> +
>>> +	mali_c55_isp_stop(mali_c55);
>>> +}
>>> +
>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
>>> +{
>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>>> +	struct media_pad *sink_pad;
>>> +	struct v4l2_subdev *sd;
>>> +	int ret;
>>> +
>>> +	sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
>>> +	isp->remote_src = media_pad_remote_pad_unique(sink_pad);
>>> +	if (IS_ERR(isp->remote_src)) {
>> If you set the MUST_CONNECT flag for the MALI_C55_ISP_PAD_SINK_VIDEO pad
>> I think you can drop this check.
>>
>>> +		dev_err(mali_c55->dev, "Failed to get source for ISP\n");
>>> +		return PTR_ERR(isp->remote_src);
>>> +	}
>>> +
>>> +	sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
>>> +
>>> +	isp->frame_sequence = 0;
>>> +	ret = mali_c55_isp_start(mali_c55);
>>> +	if (ret) {
>>> +		dev_err(mali_c55->dev, "Failed to start ISP\n");
>>> +		isp->remote_src = NULL;
>>> +		return ret;
>>> +	}
>>> +
>>> +	/*
>>> +	 * We only support a single input stream, so we can just enable the 1st
>>> +	 * entry in the streams mask.
>>> +	 */
>>> +	ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
>>> +	if (ret) {
>>> +		dev_err(mali_c55->dev, "Failed to start ISP source\n");
>>> +		mali_c55_isp_stop(mali_c55);
>>> +		return ret;
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>> +				       struct v4l2_subdev_state *state,
>>> +				       struct v4l2_subdev_mbus_code_enum *code)
>>> +{
>>> +	/*
>>> +	 * Only the internal RGB processed format is allowed on the regular
>>> +	 * processing source pad.
>>> +	 */
>>> +	if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
>>> +		if (code->index)
>>> +			return -EINVAL;
>>> +
>>> +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>> +		return 0;
>>> +	}
>>> +
>>> +	/* On the sink and bypass pads all the supported formats are allowed. */
>>> +	if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
>>> +		return -EINVAL;
>>> +
>>> +	code->code = mali_c55_isp_fmts[code->index].code;
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
>>> +					struct v4l2_subdev_state *state,
>>> +					struct v4l2_subdev_frame_size_enum *fse)
>>> +{
>>> +	const struct mali_c55_isp_fmt *cfg;
>>> +
>>> +	if (fse->index > 0)
>>> +		return -EINVAL;
>>> +
>>> +	/*
>>> +	 * Only the internal RGB processed format is allowed on the regular
>>> +	 * processing source pad.
>>> +	 *
>>> +	 * On the sink and bypass pads all the supported formats are allowed.
>>> +	 */
>>> +	if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
>>> +		if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
>>> +			return -EINVAL;
>>> +	} else {
>>> +		cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
>>> +		if (!cfg)
>>> +			return -EINVAL;
>>> +	}
>>> +
>>> +	fse->min_width = MALI_C55_MIN_WIDTH;
>>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
>>> +	fse->max_width = MALI_C55_MAX_WIDTH;
>>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
>>> +				struct v4l2_subdev_state *state,
>>> +				struct v4l2_subdev_format *format)
>>> +{
>>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>>> +	struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
>>> +	const struct mali_c55_isp_fmt *cfg;
>>> +	struct v4l2_rect *crop;
>>> +
>>> +	/*
>>> +	 * Disallow set_fmt on the source pads; format is fixed and the sizes
>>> +	 * are the result of applying the sink crop rectangle to the sink
>>> +	 * format.
>>> +	 */
>>> +	if (format->pad)
>> 	if (format->pad != MALI_C55_ISP_PAD_SINK_VIDEO)
>>
>>> +		return v4l2_subdev_get_fmt(sd, state, format);
>>> +
>>> +	cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
>>> +	if (!cfg)
>>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>> +	fmt->field = V4L2_FIELD_NONE;
>> Do you intentionally allow the colorspace fields to be overwritten to
>> any value ? rkisp1_isp_set_src_fmt() and rkisp1_isp_set_sink_fmt() may
>> show you how this could be handled.
>>
>>> +
>>> +	/*
>>> +	 * Clamp sizes in the accepted limits and clamp the crop rectangle in
>>> +	 * the new sizes.
>>> +	 */
>>> +	clamp_t(unsigned int, fmt->width,
>>> +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>> +	clamp_t(unsigned int, fmt->width,
>>> +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>> clamp_t() returns a value, which you ignore. Those are no-ops. You meant
>>
>> 	fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
>> 			     MALI_C55_MAX_WIDTH);
>> 	fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
>> 			      MALI_C55_MAX_HEIGHT);
>>
>> Same for every use of clamp_t() through the whole driver.
>>
>> Also, do you need clamp_t() ? I think all values are unsigned int, you
>> can use clamp().
>>
>> Are there any alignment constraints, such a multiples of two for bayer
>> formats ? Same in all the other locations where applicable.
>>
>>> +
>>> +	sink_fmt = v4l2_subdev_state_get_format(state,
>>> +						MALI_C55_ISP_PAD_SINK_VIDEO);
>>> +	*sink_fmt = *fmt;
>>> +
>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>> +	crop->left = 0;
>>> +	crop->top = 0;
>>> +	crop->width = fmt->width;
>>> +	crop->height = fmt->height;
>>> +
>>> +	/*
>>> +	 * Propagate format to source pads. On the 'regular' output pad use
>>> +	 * the internal RGB processed format, while on the bypass pad simply
>>> +	 * replicate the ISP sink format. The sizes on both pads are the same as
>>> +	 * the ISP sink crop rectangle.
>>> +	 */
>> Colorspace information will need to be propagated too.
>>
>>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
>>> +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>> +	src_fmt->width = crop->width;
>>> +	src_fmt->height = crop->height;
>>> +
>>> +	src_fmt = v4l2_subdev_state_get_format(state,
>>> +					       MALI_C55_ISP_PAD_SOURCE_BYPASS);
>>> +	src_fmt->code = fmt->code;
>>> +	src_fmt->width = crop->width;
>>> +	src_fmt->height = crop->height;
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
>>> +				      struct v4l2_subdev_state *state,
>>> +				      struct v4l2_subdev_selection *sel)
>>> +{
>>> +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
>> 	sel->pad != MALI_C55_ISP_PAD_SINK_VIDEO
>>
>>> +		return -EINVAL;
>>> +
>>> +	sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
>>> +				      struct v4l2_subdev_state *state,
>>> +				      struct v4l2_subdev_selection *sel)
>>> +{
>>> +	struct v4l2_mbus_framefmt *src_fmt;
>>> +	struct v4l2_mbus_framefmt *fmt;
>> const
>>
>>> +	struct v4l2_rect *crop;
>>> +
>>> +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
>> Ditto.
>>
>>> +		return -EINVAL;
>>> +
>>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>> +
>>> +	clamp_t(unsigned int, sel->r.left, 0, fmt->width);
>>> +	clamp_t(unsigned int, sel->r.top, 0, fmt->height);
>>> +	clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
>>> +		fmt->width - sel->r.left);
>>> +	clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
>>> +		fmt->height - sel->r.top);
>>> +
>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>> +	*crop = sel->r;
>>> +
>>> +	/* Propagate the crop rectangle sizes to the source pad format. */
>>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
>>> +	src_fmt->width = crop->width;
>>> +	src_fmt->height = crop->height;
>> Can you confirm that cropping doesn't affect the bypass path ? And maybe
>> add a comment to mention it.
>>
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
>>> +	.enum_mbus_code		= mali_c55_isp_enum_mbus_code,
>>> +	.enum_frame_size	= mali_c55_isp_enum_frame_size,
>>> +	.get_fmt		= v4l2_subdev_get_fmt,
>>> +	.set_fmt		= mali_c55_isp_set_fmt,
>>> +	.get_selection		= mali_c55_isp_get_selection,
>>> +	.set_selection		= mali_c55_isp_set_selection,
>>> +	.link_validate		= v4l2_subdev_link_validate_default,
>>> +};
>>> +
>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
>>> +{
>>> +	struct v4l2_event event = {
>>> +		.type = V4L2_EVENT_FRAME_SYNC,
>>> +	};
>>> +
>>> +	event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
>>> +	v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
>>> +}
>>> +
>>> +static int
>>> +mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
>>> +			     struct v4l2_event_subscription *sub)
>>> +{
>>> +	if (sub->type != V4L2_EVENT_FRAME_SYNC)
>>> +		return -EINVAL;
>>> +
>>> +	/* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
>>> +	if (sub->id != 0)
>>> +		return -EINVAL;
>>> +
>>> +	return v4l2_event_subscribe(fh, sub, 0, NULL);
>>> +}
>>> +
>>> +static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
>>> +	.subscribe_event = mali_c55_isp_subscribe_event,
>>> +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
>>> +};
>>> +
>>> +static const struct v4l2_subdev_ops mali_c55_isp_ops = {
>>> +	.pad	= &mali_c55_isp_pad_ops,
>>> +	.core	= &mali_c55_isp_core_ops,
>>> +};
>>> +
>>> +static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
>>> +				   struct v4l2_subdev_state *sd_state)
>> You name this variable state in every other subdev operation handler.
>>
>>> +{
>>> +	struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>> +	struct v4l2_rect *in_crop;
>>> +
>>> +	sink_fmt = v4l2_subdev_state_get_format(sd_state,
>>> +						MALI_C55_ISP_PAD_SINK_VIDEO);
>>> +	src_fmt = v4l2_subdev_state_get_format(sd_state,
>>> +					       MALI_C55_ISP_PAD_SOURCE);
>>> +	in_crop = v4l2_subdev_state_get_crop(sd_state,
>>> +					     MALI_C55_ISP_PAD_SINK_VIDEO);
>>> +
>>> +	sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
>>> +	sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>> +	sink_fmt->field = V4L2_FIELD_NONE;
>>> +	sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>> You should initialize the colorspace fields too. Same below.
>>
>>> +
>>> +	*v4l2_subdev_state_get_format(sd_state,
>>> +			      MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
>>> +
>>> +	src_fmt->width = MALI_C55_DEFAULT_WIDTH;
>>> +	src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>> +	src_fmt->field = V4L2_FIELD_NONE;
>>> +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>> +
>>> +	in_crop->top = 0;
>>> +	in_crop->left = 0;
>>> +	in_crop->width = MALI_C55_DEFAULT_WIDTH;
>>> +	in_crop->height = MALI_C55_DEFAULT_HEIGHT;
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
>>> +	.init_state = mali_c55_isp_init_state,
>>> +};
>>> +
>>> +static const struct media_entity_operations mali_c55_isp_media_ops = {
>>> +	.link_validate		= v4l2_subdev_link_validate,
>> 	.link_validate = v4l2_subdev_link_validate,
>>
>> to match mali_c55_isp_internal_ops.
>>
>>> +};
>>> +
>>> +static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
>>> +				       struct v4l2_subdev *subdev,
>>> +				       struct v4l2_async_connection *asc)
>>> +{
>>> +	struct mali_c55_isp *isp = container_of(notifier,
>>> +						struct mali_c55_isp, notifier);
>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>>> +	struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
>>> +	int ret;
>>> +
>>> +	/*
>>> +	 * By default we'll flag this link enabled and the TPG disabled, but
>>> +	 * no immutable flag because we need to be able to switch between the
>>> +	 * two.
>>> +	 */
>>> +	ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
>>> +					      MEDIA_LNK_FL_ENABLED);
>>> +	if (ret)
>>> +		dev_err(mali_c55->dev, "failed to create link for %s\n",
>>> +			subdev->name);
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
>>> +{
>>> +	struct mali_c55_isp *isp = container_of(notifier,
>>> +						struct mali_c55_isp, notifier);
>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>>> +
>>> +	return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
>>> +}
>>> +
>>> +static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
>>> +	.bound = mali_c55_isp_notifier_bound,
>>> +	.complete = mali_c55_isp_notifier_complete,
>>> +};
>>> +
>>> +static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
>>> +{
>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
>>> +	struct v4l2_async_connection *asc;
>>> +	struct fwnode_handle *ep;
>>> +	int ret;
>>> +
>>> +	v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
>>> +
>>> +	/*
>>> +	 * The ISP should have a single endpoint pointing to some flavour of
>>> +	 * CSI-2 receiver...but for now at least we do want everything to work
>>> +	 * normally even with no sensors connected, as we have the TPG. If we
>>> +	 * don't find a sensor just warn and return success.
>>> +	 */
>>> +	ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
>>> +					     0, 0, 0);
>>> +	if (!ep) {
>>> +		dev_warn(mali_c55->dev, "no local endpoint found\n");
>>> +		return 0;
>>> +	}
>>> +
>>> +	asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
>>> +					      struct v4l2_async_connection);
>>> +	if (IS_ERR(asc)) {
>>> +		dev_err(mali_c55->dev, "failed to add remote fwnode\n");
>>> +		ret = PTR_ERR(asc);
>>> +		goto err_put_ep;
>>> +	}
>>> +
>>> +	isp->notifier.ops = &mali_c55_isp_notifier_ops;
>>> +	ret = v4l2_async_nf_register(&isp->notifier);
>>> +	if (ret) {
>>> +		dev_err(mali_c55->dev, "failed to register notifier\n");
>>> +		goto err_cleanup_nf;
>>> +	}
>>> +
>>> +	fwnode_handle_put(ep);
>>> +
>>> +	return 0;
>>> +
>>> +err_cleanup_nf:
>>> +	v4l2_async_nf_cleanup(&isp->notifier);
>>> +err_put_ep:
>>> +	fwnode_handle_put(ep);
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55)
>>> +{
>>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>>> +	struct v4l2_subdev *sd = &isp->sd;
>>> +	int ret;
>>> +
>>> +	isp->mali_c55 = mali_c55;
>>> +
>>> +	v4l2_subdev_init(sd, &mali_c55_isp_ops);
>>> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
>>> +	sd->entity.ops = &mali_c55_isp_media_ops;
>>> +	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
>>> +	sd->internal_ops = &mali_c55_isp_internal_ops;
>>> +	strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
>>> +
>>> +	isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
>> The MUST_CONNECT flag would make sense here.
>>
>>> +	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>>> +	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
>>> +
>>> +	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
>>> +				     isp->pads);
>>> +	if (ret)
>>> +		return ret;
>>> +
>>> +	ret = v4l2_subdev_init_finalize(sd);
>>> +	if (ret)
>>> +		goto err_cleanup_media_entity;
>>> +
>>> +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>> +	if (ret)
>>> +		goto err_cleanup_subdev;
>>> +
>>> +	ret = mali_c55_isp_parse_endpoint(isp);
>>> +	if (ret)
>>> +		goto err_cleanup_subdev;
>> As noted elsewhere, I think this belongs to mali-c55-core.c.
>>
>>> +
>>> +	mutex_init(&isp->lock);
>> This lock is used in mali-c55-capture.c only, that seems weird.
>>
>>> +
>>> +	return 0;
>>> +
>>> +err_cleanup_subdev:
>>> +	v4l2_subdev_cleanup(sd);
>>> +err_cleanup_media_entity:
>>> +	media_entity_cleanup(&sd->entity);
>>> +	isp->mali_c55 = NULL;
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
>>> +{
>>> +	struct mali_c55_isp *isp = &mali_c55->isp;
>>> +
>>> +	if (!isp->mali_c55)
>>> +		return;
>>> +
>>> +	mutex_destroy(&isp->lock);
>>> +	v4l2_async_nf_unregister(&isp->notifier);
>>> +	v4l2_async_nf_cleanup(&isp->notifier);
>>> +	v4l2_device_unregister_subdev(&isp->sd);
>>> +	v4l2_subdev_cleanup(&isp->sd);
>>> +	media_entity_cleanup(&isp->sd.entity);
>>> +}
>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>> new file mode 100644
>>> index 000000000000..cb27abde2aa5
>>> --- /dev/null
>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>> @@ -0,0 +1,258 @@
>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>> +/*
>>> + * ARM Mali-C55 ISP Driver - Register definitions
>>> + *
>>> + * Copyright (C) 2024 Ideas on Board Oy
>>> + */
>>> +
>>> +#ifndef _MALI_C55_REGISTERS_H
>>> +#define _MALI_C55_REGISTERS_H
>>> +
>>> +#include <linux/bits.h>
>>> +
>>> +/* ISP Common 0x00000 - 0x000ff */
>>> +
>>> +#define MALI_C55_REG_API				0x00000
>>> +#define MALI_C55_REG_PRODUCT				0x00004
>>> +#define MALI_C55_REG_VERSION				0x00008
>>> +#define MALI_C55_REG_REVISION				0x0000c
>>> +#define MALI_C55_REG_PULSE_MODE				0x0003c
>>> +#define MALI_C55_REG_INPUT_MODE_REQUEST			0x0009c
>>> +#define MALI_C55_INPUT_SAFE_STOP			0x00
>>> +#define MALI_C55_INPUT_SAFE_START			0x01
>>> +#define MALI_C55_REG_MODE_STATUS			0x000a0
>>> +#define MALI_C55_REG_INTERRUPT_MASK_VECTOR		0x00030
>>> +#define MALI_C55_INTERRUPT_MASK_ALL			GENMASK(31, 0)
>>> +
>>> +#define MALI_C55_REG_GLOBAL_MONITOR			0x00050
>>> +
>>> +#define MALI_C55_REG_GEN_VIDEO				0x00080
>>> +#define MALI_C55_REG_GEN_VIDEO_ON_MASK			BIT(0)
>>> +#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK		BIT(1)
>>> +#define MALI_C55_REG_GEN_PREFETCH_MASK			GENMASK(31, 16)
>>> +
>>> +#define MALI_C55_REG_MCU_CONFIG				0x00020
>>> +#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK		BIT(0)
>> #define MALI_C55_REG_MCU_CONFIG_OVERRIDE		BIT(0)
>>
>> Same in other places where applicable.
>>
>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK		BIT(1)
>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PING		BIT(1)
>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG		0x00
>>> +#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK		BIT(8)
>>> +#define MALI_C55_REG_PING_PONG_READ			0x00024
>>> +#define MALI_C55_REG_PING_PONG_READ_MASK		BIT(2)
>>> +
>>> +#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR		0x00034
>>> +#define MALI_C55_REG_INTERRUPT_CLEAR			0x00040
>>> +#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR		0x00044
>>> +#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS		0x00068
>>> +#define MALI_C55_GPS_PONG_FITTED			BIT(0)
>>> +#define MALI_C55_GPS_WDR_FITTED				BIT(1)
>>> +#define MALI_C55_GPS_COMPRESSION_FITTED			BIT(2)
>>> +#define MALI_C55_GPS_TEMPER_FITTED			BIT(3)
>>> +#define MALI_C55_GPS_SINTER_LITE_FITTED			BIT(4)
>>> +#define MALI_C55_GPS_SINTER_FITTED			BIT(5)
>>> +#define MALI_C55_GPS_IRIDIX_LTM_FITTED			BIT(6)
>>> +#define MALI_C55_GPS_IRIDIX_GTM_FITTED			BIT(7)
>>> +#define MALI_C55_GPS_CNR_FITTED				BIT(8)
>>> +#define MALI_C55_GPS_FRSCALER_FITTED			BIT(9)
>>> +#define MALI_C55_GPS_DS_PIPE_FITTED			BIT(10)
>>> +
>>> +#define MALI_C55_REG_BLANKING				0x00084
>>> +#define MALI_C55_REG_HBLANK_MASK			GENMASK(15, 0)
>>> +#define MALI_C55_REG_VBLANK_MASK			GENMASK(31, 16)
>>> +
>>> +#define MALI_C55_REG_HC_START				0x00088
>>> +#define MALI_C55_HC_START(h)				(((h) & 0xffff) << 16)
>>> +#define MALI_C55_REG_HC_SIZE				0x0008c
>>> +#define MALI_C55_HC_SIZE(h)				((h) & 0xffff)
>>> +#define MALI_C55_REG_VC_START_SIZE			0x00094
>>> +#define MALI_C55_VC_START(v)				((v) & 0xffff)
>>> +#define MALI_C55_VC_SIZE(v)				(((v) & 0xffff) << 16)
>>> +
>>> +/* Ping/Pong Configuration Space */
>>> +#define MALI_C55_REG_BASE_ADDR				0x18e88
>>> +#define MALI_C55_REG_BYPASS_0				0x18eac
>>> +#define MALI_C55_REG_BYPASS_0_VIDEO_TEST		BIT(0)
>>> +#define MALI_C55_REG_BYPASS_0_INPUT_FMT			BIT(1)
>>> +#define MALI_C55_REG_BYPASS_0_DECOMPANDER		BIT(2)
>>> +#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR		BIT(3)
>>> +#define MALI_C55_REG_BYPASS_0_GAIN_WDR			BIT(4)
>>> +#define MALI_C55_REG_BYPASS_0_FRAME_STITCH		BIT(5)
>>> +#define MALI_C55_REG_BYPASS_1				0x18eb0
>>> +#define MALI_C55_REG_BYPASS_1_DIGI_GAIN			BIT(0)
>>> +#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS		BIT(1)
>>> +#define MALI_C55_REG_BYPASS_1_FE_SQRT			BIT(2)
>>> +#define MALI_C55_REG_BYPASS_1_RAW_FE			BIT(3)
>>> +#define MALI_C55_REG_BYPASS_2				0x18eb8
>>> +#define MALI_C55_REG_BYPASS_2_SINTER			BIT(0)
>>> +#define MALI_C55_REG_BYPASS_2_TEMPER			BIT(1)
>>> +#define MALI_C55_REG_BYPASS_3				0x18ebc
>>> +#define MALI_C55_REG_BYPASS_3_SQUARE_BE			BIT(0)
>>> +#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH	BIT(1)
>>> +#define MALI_C55_REG_BYPASS_3_MESH_SHADING		BIT(3)
>>> +#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE		BIT(4)
>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX			BIT(5)
>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN		BIT(6)
>>> +#define MALI_C55_REG_BYPASS_4				0x18ec0
>>> +#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB		BIT(1)
>>> +#define MALI_C55_REG_BYPASS_4_PF_CORRECTION		BIT(3)
>>> +#define MALI_C55_REG_BYPASS_4_CCM			BIT(4)
>>> +#define MALI_C55_REG_BYPASS_4_CNR			BIT(5)
>>> +#define MALI_C55_REG_FR_BYPASS				0x18ec4
>>> +#define MALI_C55_REG_DS_BYPASS				0x18ec8
>>> +#define MALI_C55_BYPASS_CROP				BIT(0)
>>> +#define MALI_C55_BYPASS_SCALER				BIT(1)
>>> +#define MALI_C55_BYPASS_GAMMA_RGB			BIT(2)
>>> +#define MALI_C55_BYPASS_SHARPEN				BIT(3)
>>> +#define MALI_C55_BYPASS_CS_CONV				BIT(4)
>>> +#define MALI_C55_REG_ISP_RAW_BYPASS			0x18ecc
>>> +#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK		BIT(0)
>>> +#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK		GENMASK(9, 8)
>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS		2
>>> +#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS		1
>>> +#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE		BIT(1)
>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS		BIT(0)
>>> +
>>> +#define MALI_C55_REG_ACTIVE_WIDTH_MASK			0xffff
>>> +#define MALI_C55_REG_ACTIVE_HEIGHT_MASK			0xffff0000
>>> +#define MALI_C55_REG_BAYER_ORDER			0x18e8c
>>> +#define MALI_C55_BAYER_ORDER_MASK			GENMASK(1, 0)
>>> +#define MALI_C55_REG_TPG_CH0				0x18ed8
>>> +#define MALI_C55_TEST_PATTERN_ON_OFF			BIT(0)
>>> +#define MALI_C55_TEST_PATTERN_RGB_MASK			BIT(1)
>>> +#define MALI_C55_REG_TPG_R_BACKGROUND			0x18ee0
>>> +#define MALI_C55_REG_TPG_G_BACKGROUND			0x18ee4
>>> +#define MALI_C55_REG_TPG_B_BACKGROUND			0x18ee8
>>> +#define MALI_C55_TPG_BACKGROUND_MAX			0xfffff
>>> +#define MALI_C55_REG_INPUT_WIDTH			0x18f98
>>> +#define MALI_C55_INPUT_WIDTH_MASK			GENMASK(18, 16)
>>> +#define MALI_C55_INPUT_WIDTH_8BIT			0
>>> +#define MALI_C55_INPUT_WIDTH_10BIT			1
>>> +#define MALI_C55_INPUT_WIDTH_12BIT			2
>>> +#define MALI_C55_INPUT_WIDTH_14BIT			3
>>> +#define MALI_C55_INPUT_WIDTH_16BIT			4
>>> +#define MALI_C55_INPUT_WIDTH_20BIT			5
>>> +#define MALI_C55_REG_SPACE_SIZE				0x4000
>>> +#define MALI_C55_REG_CONFIG_SPACES_OFFSET		0x0ab6c
>>> +#define MALI_C55_CONFIG_SPACE_SIZE			0x1231c
>>> +
>>> +#define MALI_C55_REG_SINTER_CONFIG			0x19348
>>> +#define MALI_C55_SINTER_VIEW_FILTER_MASK		GENMASK(1, 0)
>>> +#define MALI_C55_SINTER_SCALE_MODE_MASK			GENMASK(3, 2)
>>> +#define MALI_C55_SINTER_ENABLE_MASK			BIT(4)
>>> +#define MALI_C55_SINTER_FILTER_SELECT_MASK		BIT(5)
>>> +#define MALI_C55_SINTER_INT_SELECT_MASK			BIT(6)
>>> +#define MALI_C55_SINTER_RM_ENABLE_MASK			BIT(7)
>>> +
>>> +/* Colour Correction Matrix Configuration */
>>> +#define MALI_C55_REG_CCM_ENABLE				0x1b07c
>>> +#define MALI_C55_CCM_ENABLE_MASK			BIT(0)
>>> +#define MALI_C55_REG_CCM_COEF_R_R			0x1b080
>>> +#define MALI_C55_REG_CCM_COEF_R_G			0x1b084
>>> +#define MALI_C55_REG_CCM_COEF_R_B			0x1b088
>>> +#define MALI_C55_REG_CCM_COEF_G_R			0x1b090
>>> +#define MALI_C55_REG_CCM_COEF_G_G			0x1b094
>>> +#define MALI_C55_REG_CCM_COEF_G_B			0x1b098
>>> +#define MALI_C55_REG_CCM_COEF_B_R			0x1b0a0
>>> +#define MALI_C55_REG_CCM_COEF_B_G			0x1b0a4
>>> +#define MALI_C55_REG_CCM_COEF_B_B			0x1b0a8
>>> +#define MALI_C55_CCM_COEF_MASK				GENMASK(12, 0)
>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R			0x1b0b0
>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G			0x1b0b4
>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B			0x1b0b8
>>> +#define MALI_C55_CCM_ANTIFOG_GAIN_MASK			GENMASK(11, 0)
>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R		0x1b0c0
>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G		0x1b0c4
>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B		0x1b0c8
>>> +#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK		GENMASK(11, 0)
>>> +
>>> +/*
>>> + * The Mali-C55 ISP has up to two output pipes; known as full resolution and
>>> + * down scaled. The register space for these is laid out identically, but offset
>>> + * by 372 bytes.
>>> + */
>>> +#define MALI_C55_CAP_DEV_FR_REG_OFFSET		0x0
>>> +#define MALI_C55_CAP_DEV_DS_REG_OFFSET		0x174
>>> +
>>> +#define MALI_C55_REG_CS_CONV_CONFIG(offset)		(0x1c098 + (offset))
>>> +#define MALI_C55_CS_CONV_MATRIX_MASK			BIT(0)
>>> +#define MALI_C55_CS_CONV_FILTER_MASK			BIT(1)
>>> +#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK		BIT(2)
>>> +#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK		BIT(3)
>>> +#define MALI_C55_REG_Y_WRITER_MODE(offset)		(0x1c0ec + (offset))
>>> +#define MALI_C55_REG_UV_WRITER_MODE(offset)		(0x1c144 + (offset))
>>> +#define MALI_C55_WRITER_MODE_MASK			GENMASK(4, 0)
>>> +#define MALI_C55_WRITER_SUBMODE_MASK			GENMASK(7, 6)
>>> +#define MALI_C55_WRITER_FRAME_WRITE_MASK		BIT(9)
>>> +#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset)		(0x1c0f0 + (offset))
>>> +#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset)		(0x1c148 + (offset))
>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)		((w) << 0)
>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)		((h) << 16)
>>> +#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset)	(0x1c0f4 + (offset))
>>> +#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset)	(0x1c108 + (offset))
>>> +#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
>>> +#define MALI_C55_REG_Y_WRITER_BANKS_RESTART		BIT(3)
>>> +#define MALI_C55_REG_Y_WRITER_OFFSET(offset)		(0x1c10c + (offset))
>>> +#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset)	(0x1c14c + (offset))
>>> +#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset)	(0x1c160 + (offset))
>>> +#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
>>> +#define MALI_C55_REG_UV_WRITER_BANKS_RESTART		BIT(3)
>>> +#define MALI_C55_REG_UV_WRITER_OFFSET(offset)		(0x1c164 + (offset))
>>> +
>>> +#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
>>> +#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE		0x18edc
>>> +
>>> +#define MALI_C55_REG_CROP_EN(offset)			(0x1c028 + (offset))
>>> +#define MALI_C55_CROP_ENABLE				BIT(0)
>>> +#define MALI_C55_REG_CROP_X_START(offset)		(0x1c02c + (offset))
>>> +#define MALI_C55_REG_CROP_Y_START(offset)		(0x1c030 + (offset))
>>> +#define MALI_C55_REG_CROP_X_SIZE(offset)		(0x1c034 + (offset))
>>> +#define MALI_C55_REG_CROP_Y_SIZE(offset)		(0x1c038 + (offset))
>>> +#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset)		(0x1c040 + (offset))
>>> +#define MALI_C55_SCALER_TIMEOUT_EN			BIT(4)
>>> +#define MALI_C55_SCALER_TIMEOUT(t)			((t) << 16)
>>> +#define MALI_C55_REG_SCALER_IN_WIDTH(offset)		(0x1c044 + (offset))
>>> +#define MALI_C55_REG_SCALER_IN_HEIGHT(offset)		(0x1c048 + (offset))
>>> +#define MALI_C55_REG_SCALER_OUT_WIDTH(offset)		(0x1c04c + (offset))
>>> +#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset)		(0x1c050 + (offset))
>>> +#define MALI_C55_REG_SCALER_HFILT_TINC(offset)		(0x1c054 + (offset))
>>> +#define MALI_C55_REG_SCALER_HFILT_COEF(offset)		(0x1c058 + (offset))
>>> +#define MALI_C55_REG_SCALER_VFILT_TINC(offset)		(0x1c05c + (offset))
>>> +#define MALI_C55_REG_SCALER_VFILT_COEF(offset)		(0x1c060 + (offset))
>>> +
>>> +#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset)		(0x1c064 + (offset))
>>> +#define MALI_C55_GAMMA_ENABLE_MASK			BIT(0)
>>> +#define MALI_C55_REG_GAMMA_GAINS_1(offset)		(0x1c068 + (offset))
>>> +#define MALI_C55_GAMMA_GAIN_R_MASK			GENMASK(11, 0)
>>> +#define MALI_C55_GAMMA_GAIN_G_MASK			GENMASK(27, 16)
>>> +#define MALI_C55_REG_GAMMA_GAINS_2(offset)		(0x1c06c + (offset))
>>> +#define MALI_C55_GAMMA_GAIN_B_MASK			GENMASK(11, 0)
>>> +#define MALI_C55_REG_GAMMA_OFFSETS_1(offset)		(0x1c070 + (offset))
>>> +#define MALI_C55_GAMMA_OFFSET_R_MASK			GENMASK(11, 0)
>>> +#define MALI_C55_GAMMA_OFFSET_G_MASK			GENMASK(27, 16)
>>> +#define MALI_C55_REG_GAMMA_OFFSETS_2(offset)		(0x1c074 + (offset))
>>> +#define MALI_C55_GAMMA_OFFSET_B_MASK			GENMASK(11, 0)
>>> +
>>> +/* Output DMA Writer */
>>> +
>>> +#define MALI_C55_OUTPUT_DISABLED		0
>>> +#define MALI_C55_OUTPUT_RGB32			1
>>> +#define MALI_C55_OUTPUT_A2R10G10B10		2
>>> +#define MALI_C55_OUTPUT_RGB565			3
>>> +#define MALI_C55_OUTPUT_RGB24			4
>>> +#define MALI_C55_OUTPUT_GEN32			5
>>> +#define MALI_C55_OUTPUT_RAW16			6
>>> +#define MALI_C55_OUTPUT_AYUV			8
>>> +#define MALI_C55_OUTPUT_Y410			9
>>> +#define MALI_C55_OUTPUT_YUY2			10
>>> +#define MALI_C55_OUTPUT_UYVY			11
>>> +#define MALI_C55_OUTPUT_Y210			12
>>> +#define MALI_C55_OUTPUT_NV12_21			13
>>> +#define MALI_C55_OUTPUT_YUV_420_422		17
>>> +#define MALI_C55_OUTPUT_P210_P010		19
>>> +#define MALI_C55_OUTPUT_YUV422			20
>> I'd define this just below the MALI_C55_REG_UV_WRITER_MODE register
>> macro.
>>
>>> +
>>> +#define MALI_C55_OUTPUT_PLANE_ALT0		0
>>> +#define MALI_C55_OUTPUT_PLANE_ALT1		1
>>> +#define MALI_C55_OUTPUT_PLANE_ALT2		2
>> Same here ?
>>
>>> +
>>> +#endif /* _MALI_C55_REGISTERS_H */
>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>> new file mode 100644
>>> index 000000000000..8edae87f1e5f
>>> --- /dev/null
>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>> @@ -0,0 +1,382 @@
>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>> +/*
>>> + * ARM Mali-C55 ISP Driver - Resizer Coefficients
>>> + *
>>> + * Copyright (C) 2024 Ideas on Board Oy
>>> + */
>>> +
>>> +#ifndef _MALI_C55_RESIZER_COEFS_H
>>> +#define _MALI_C55_RESIZER_COEFS_H
>>> +
>>> +#include "mali-c55-common.h"
>>> +
>>> +#define MALI_C55_RESIZER_COEFS_NUM_BANKS	8
>>> +#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES	64
>> Do these belongs to mali-c55-registers.h ?
>>
>>> +
>>> +static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
>>> +	{	/* Bank 0 */
>>> +		0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
>>> +		0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
>>> +		0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
>>> +		0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
>>> +		0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
>>> +		0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
>>> +		0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
>>> +		0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
>>> +		0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
>>> +		0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
>>> +		0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
>>> +		0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
>>> +		0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
>>> +		0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
>>> +		0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
>>> +		0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
>>> +	},
>>> +	{	/* Bank 1 */
>>> +		0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
>>> +		0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
>>> +		0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
>>> +		0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
>>> +		0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
>>> +		0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
>>> +		0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
>>> +		0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
>>> +		0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
>>> +		0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
>>> +		0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
>>> +		0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
>>> +		0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
>>> +		0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
>>> +		0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
>>> +		0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
>>> +	},
>>> +	{	/* Bank 2 */
>>> +		0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
>>> +		0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
>>> +		0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
>>> +		0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
>>> +		0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
>>> +		0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
>>> +		0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
>>> +		0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
>>> +		0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
>>> +		0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
>>> +		0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
>>> +		0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
>>> +		0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
>>> +		0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
>>> +		0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
>>> +		0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
>>> +	},
>>> +	{	/* Bank 3 */
>>> +		0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
>>> +		0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
>>> +		0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
>>> +		0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
>>> +		0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
>>> +		0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
>>> +		0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
>>> +		0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
>>> +		0x20100000, 0x00000010, 0x1f110000, 0x00000010,
>>> +		0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
>>> +		0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
>>> +		0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
>>> +		0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
>>> +		0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
>>> +		0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
>>> +		0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
>>> +	},
>>> +	{	/* Bank 4 */
>>> +		0x17090000, 0x00000917, 0x18090000, 0x00000916,
>>> +		0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
>>> +		0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
>>> +		0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
>>> +		0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
>>> +		0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
>>> +		0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
>>> +		0x190f0300, 0x00000411, 0x18100300, 0x00000411,
>>> +		0x1a100300, 0x00000310, 0x18110400, 0x00000310,
>>> +		0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
>>> +		0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
>>> +		0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
>>> +		0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
>>> +		0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
>>> +		0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
>>> +		0x17160800, 0x0000010a, 0x18160900, 0x00000009,
>>> +	},
>>> +	{	/* Bank 5 */
>>> +		0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
>>> +		0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
>>> +		0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
>>> +		0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
>>> +		0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
>>> +		0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
>>> +		0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
>>> +		0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
>>> +		0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
>>> +		0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
>>> +		0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
>>> +		0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
>>> +		0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
>>> +		0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
>>> +		0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
>>> +		0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
>>> +	},
>>> +	{	/* Bank 6 */
>>> +		0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
>>> +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
>>> +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
>>> +		0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
>>> +		0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
>>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>> +		0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
>>> +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>> +		0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
>>> +		0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
>>> +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
>>> +		0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>> +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>> +	},
>>> +	{	/* Bank 7 */
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>> +	}
>>> +};
>>> +
>>> +static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
>>> +	{	/* Bank 0 */
>>> +		0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
>>> +		0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
>>> +		0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
>>> +		0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
>>> +		0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
>>> +		0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
>>> +		0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
>>> +		0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
>>> +		0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
>>> +		0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
>>> +		0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
>>> +		0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
>>> +		0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
>>> +		0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
>>> +		0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
>>> +		0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
>>> +	},
>>> +	{	/* Bank 1 */
>>> +		0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
>>> +		0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
>>> +		0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
>>> +		0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
>>> +		0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
>>> +		0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
>>> +		0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
>>> +		0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
>>> +		0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
>>> +		0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
>>> +		0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
>>> +		0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
>>> +		0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
>>> +		0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
>>> +		0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
>>> +		0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
>>> +	},
>>> +	{	/* Bank 2 */
>>> +		0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
>>> +		0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
>>> +		0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
>>> +		0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
>>> +		0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
>>> +		0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
>>> +		0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
>>> +		0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
>>> +		0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
>>> +		0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
>>> +		0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
>>> +		0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
>>> +		0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
>>> +		0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
>>> +		0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
>>> +		0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
>>> +	},
>>> +	{	/* Bank 3 */
>>> +		0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
>>> +		0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
>>> +		0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
>>> +		0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
>>> +		0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
>>> +		0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
>>> +		0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
>>> +		0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
>>> +		0x20100000, 0x00000010, 0x1f100000, 0x00000011,
>>> +		0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
>>> +		0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
>>> +		0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
>>> +		0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
>>> +		0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
>>> +		0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
>>> +		0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
>>> +	},
>>> +	{	/* Bank 4 */
>>> +		0x17170900, 0x00000009, 0x18160900, 0x00000009,
>>> +		0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
>>> +		0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
>>> +		0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
>>> +		0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
>>> +		0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
>>> +		0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
>>> +		0x19110400, 0x0000030f, 0x18110400, 0x00000310,
>>> +		0x1a100300, 0x00000310, 0x18100300, 0x00000411,
>>> +		0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
>>> +		0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
>>> +		0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
>>> +		0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
>>> +		0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
>>> +		0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
>>> +		0x170a0100, 0x00000816, 0x18090000, 0x00000916,
>>> +	},
>>> +	{	/* Bank 5 */
>>> +		0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
>>> +		0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
>>> +		0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
>>> +		0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
>>> +		0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
>>> +		0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
>>> +		0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
>>> +		0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
>>> +		0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
>>> +		0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
>>> +		0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
>>> +		0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
>>> +		0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
>>> +		0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
>>> +		0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
>>> +		0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
>>> +	},
>>> +	{	/* Bank 6 */
>>> +		0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
>>> +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>> +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
>>> +		0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
>>> +		0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
>>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>> +		0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>> +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>> +		0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
>>> +		0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
>>> +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
>>> +		0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
>>> +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
>>> +	},
>>> +	{	/* Bank 7 */
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>> +	}
>>> +};
>>> +
>>> +struct mali_c55_resizer_coef_bank {
>>> +	unsigned int bank;
>> This is always equal to the index of the entry in the
>> mali_c55_coefficient_banks array, you can drop it.
>>
>>> +	unsigned int top;
>>> +	unsigned int bottom;
>> The bottom value of bank N is always equal to the top value of bank N+1.
>> You can simplify this by storing a single value.
>>
>>> +};
>>> +
>>> +static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
>>> +	{
>>> +		.bank = 0,
>>> +		.top = 1000,
>>> +		.bottom = 770,
>>> +	},
>>> +	{
>>> +		.bank = 1,
>>> +		.top = 769,
>>> +		.bottom = 600,
>>> +	},
>>> +	{
>>> +		.bank = 2,
>>> +		.top = 599,
>>> +		.bottom = 460,
>>> +	},
>>> +	{
>>> +		.bank = 3,
>>> +		.top = 459,
>>> +		.bottom = 354,
>>> +	},
>>> +	{
>>> +		.bank = 4,
>>> +		.top = 353,
>>> +		.bottom = 273,
>>> +	},
>>> +	{
>>> +		.bank = 5,
>>> +		.top = 272,
>>> +		.bottom = 210,
>>> +	},
>>> +	{
>>> +		.bank = 6,
>>> +		.top = 209,
>>> +		.bottom = 162,
>>> +	},
>>> +	{
>>> +		.bank = 7,
>>> +		.top = 161,
>>> +		.bottom = 125,
>>> +	},
>>> +};
>>> +
>> A small comment would be nice, such as
>>
>> /* Select a bank of resizer coefficients, based on the scaling ratio. */
>>
>>> +static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,
>> This function is related to the resizers. Add "rsz" somewhere in the
>> function name, and pass a resizer pointer.
>>
>>> +						unsigned int crop,
>>> +						unsigned int scale)
>> I think those are the input and output sizes to the scaler. Rename them
>> to make it clearer.
>>
>>> +{
>>> +	unsigned int tmp;
>> tmp is almost always a bad variable name. Please use a more descriptive
>> name, size as rsz_ratio.
>>
>>> +	unsigned int i;
>>> +
>>> +	tmp = (scale * 1000U) / crop;
>>> +
>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
>>> +		if (tmp >= mali_c55_coefficient_banks[i].bottom &&
>>> +		    tmp <= mali_c55_coefficient_banks[i].top)
>>> +			return mali_c55_coefficient_banks[i].bank;
>>> +	}
>>> +
>>> +	/*
>>> +	 * We shouldn't ever get here, in theory. As we have no good choices
>>> +	 * simply warn the user and use the first bank of coefficients.
>>> +	 */
>>> +	dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
>>> +	return 0;
>>> +}
>> And everything else belongs to mali-c55-resizer.c. Drop this header
>> file.
>>
>>> +
>>> +#endif /* _MALI_C55_RESIZER_COEFS_H */
>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>> new file mode 100644
>>> index 000000000000..0a5a2969d3ce
>>> --- /dev/null
>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>> @@ -0,0 +1,779 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +/*
>>> + * ARM Mali-C55 ISP Driver - Image signal processor
>>> + *
>>> + * Copyright (C) 2024 Ideas on Board Oy
>>> + */
>>> +
>>> +#include <linux/math.h>
>>> +#include <linux/minmax.h>
>>> +
>>> +#include <media/media-entity.h>
>>> +#include <media/v4l2-subdev.h>
>>> +
>>> +#include "mali-c55-common.h"
>>> +#include "mali-c55-registers.h"
>>> +#include "mali-c55-resizer-coefs.h"
>>> +
>>> +/* Scaling factor in Q4.20 format. */
>>> +#define MALI_C55_RZR_SCALER_FACTOR	(1U << 20)
>>> +
>>> +static const u32 rzr_non_bypass_src_fmts[] = {
>>> +	MEDIA_BUS_FMT_RGB121212_1X36,
>>> +	MEDIA_BUS_FMT_YUV10_1X30
>>> +};
>>> +
>>> +static const char * const mali_c55_resizer_names[] = {
>>> +	[MALI_C55_RZR_FR] = "resizer fr",
>>> +	[MALI_C55_RZR_DS] = "resizer ds",
>>> +};
>>> +
>>> +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
>>> +				     struct v4l2_subdev_state *state)
>>> +{
>>> +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
>>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>>> +	struct v4l2_mbus_framefmt *fmt;
>>> +	struct v4l2_rect *crop;
> const
>
>>> +
>>> +	/* Verify if crop should be enabled. */
>>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
>>> +
>>> +	if (fmt->width == crop->width && fmt->height == crop->height)
>>> +		return MALI_C55_BYPASS_CROP;
>>> +
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
>>> +		       crop->left);
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
>>> +		       crop->top);
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
>>> +		       crop->width);
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
>>> +		       crop->height);
>>> +
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
>>> +		       MALI_C55_CROP_ENABLE);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
>>> +					struct v4l2_subdev_state *state)
>>> +{
>>> +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
>>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>>> +	struct v4l2_rect *crop, *scale;
> const
>
> Once "[PATCH v4 0/3] media: v4l2-subdev: Support const-awareness in
> state accessors" gets merged, the state argument to this function can be
> made const too. Same for other functions, as applicable.
>
>>> +	unsigned int h_bank, v_bank;
>>> +	u64 h_scale, v_scale;
>>> +
>>> +	/* Verify if scaling should be enabled. */
>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
>>> +	scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
>>> +
>>> +	if (crop->width == scale->width && crop->height == scale->height)
>>> +		return MALI_C55_BYPASS_SCALER;
>>> +
>>> +	/* Program the V/H scaling factor in Q4.20 format. */
>>> +	h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
>>> +	v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
>>> +
>>> +	do_div(h_scale, scale->width);
>>> +	do_div(v_scale, scale->height);
>>> +
>>> +	mali_c55_write(mali_c55,
>>> +		       MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
>>> +		       crop->width);
>>> +	mali_c55_write(mali_c55,
>>> +		       MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
>>> +		       crop->height);
>>> +
>>> +	mali_c55_write(mali_c55,
>>> +		       MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
>>> +		       scale->width);
>>> +	mali_c55_write(mali_c55,
>>> +		       MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
>>> +		       scale->height);
>>> +
>>> +	mali_c55_write(mali_c55,
>>> +		       MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
>>> +		       h_scale);
>>> +	mali_c55_write(mali_c55,
>>> +		       MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
>>> +		       v_scale);
>>> +
>>> +	h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
>>> +					     scale->width);
>>> +	mali_c55_write(mali_c55,
>>> +		       MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
>>> +		       h_bank);
>>> +
>>> +	v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
>>> +					     scale->height);
>>> +	mali_c55_write(mali_c55,
>>> +		       MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
>>> +		       v_bank);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
>>> +				 struct v4l2_subdev_state *state)
>>> +{
>>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>>> +	u32 bypass = 0;
>>> +
>>> +	/* Verify if cropping and scaling should be enabled. */
>>> +	bypass |= mali_c55_rzr_program_crop(rzr, state);
>>> +	bypass |= mali_c55_rzr_program_resizer(rzr, state);
>>> +
>>> +	mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
>>> +			     MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
>>> +			     MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
>>> +			     bypass);
>>> +}
>>> +
>>> +/*
>>> + * Inspect the routing table to know which of the two (mutually exclusive)
>>> + * routes is enabled and return the sink pad id of the active route.
>>> + */
>>> +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
>>> +{
>>> +	struct v4l2_subdev_krouting *routing = &state->routing;
>>> +	struct v4l2_subdev_route *route;
>>> +
>>> +	/* A single route is enabled at a time. */
>>> +	for_each_active_route(routing, route)
>>> +		return route->sink_pad;
>>> +
>>> +	return MALI_C55_RZR_SINK_PAD;
>>> +}
>>> +
>>> +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
>>> +{
>>> +	u32 corrected_code = 0;
>>> +
>>> +	/*
>>> +	 * The ISP takes input in a 20-bit format, but can only output 16-bit
>>> +	 * RAW bayer data (with the 4 least significant bits from the input
>>> +	 * being lost). Return the 16-bit version of the 20-bit input formats.
>>> +	 */
>>> +	switch (mbus_code) {
>>> +	case MEDIA_BUS_FMT_SBGGR20_1X20:
>>> +		corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
>>> +		break;
>>> +	case MEDIA_BUS_FMT_SGBRG20_1X20:
>>> +		corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
>>> +		break;
>>> +	case MEDIA_BUS_FMT_SGRBG20_1X20:
>>> +		corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
>>> +		break;
>>> +	case MEDIA_BUS_FMT_SRGGB20_1X20:
>>> +		corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
>>> +		break;
> Would it make sense to add the shifted code to mali_c55_isp_fmt ?
>
>>> +	}
>>> +
>>> +	return corrected_code;
>>> +}
>>> +
>>> +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
>>> +				      struct v4l2_subdev_state *state,
>>> +				      struct v4l2_subdev_krouting *routing)
> I think the last argument can be const.
>
>>> +{
>>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>> +						    sd);
> A to_mali_c55_resizer() static inline function would be useful. Same for
> other components, where applicable.
>
>>> +	unsigned int active_sink = UINT_MAX;
>>> +	struct v4l2_mbus_framefmt *src_fmt;
>>> +	struct v4l2_rect *crop, *compose;
>>> +	struct v4l2_subdev_route *route;
>>> +	unsigned int active_routes = 0;
>>> +	struct v4l2_mbus_framefmt *fmt;
>>> +	int ret;
>>> +
>>> +	ret = v4l2_subdev_routing_validate(sd, routing, 0);
>>> +	if (ret)
>>> +		return ret;
>>> +
>>> +	/* Only a single route can be enabled at a time. */
>>> +	for_each_active_route(routing, route) {
>>> +		if (++active_routes > 1) {
>>> +			dev_err(rzr->mali_c55->dev,
>>> +				"Only one route can be active");
> No kernel log message with a level higher than dev_dbg() from
> user-controlled paths please, here and where applicable. This is to
> avoid giving applications an easy way to flood the kernel log.
>
>>> +			return -EINVAL;
>>> +		}
>>> +
>>> +		active_sink = route->sink_pad;
>>> +	}
>>> +	if (active_sink == UINT_MAX) {
>>> +		dev_err(rzr->mali_c55->dev, "One route has to be active");
>>> +		return -EINVAL;
>>> +	}
> The recommended handling of invalid routing is to adjust the routing
> table, not to return errors.
>
>>> +
>>> +	ret = v4l2_subdev_set_routing(sd, state, routing);
>>> +	if (ret) {
>>> +		dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
>>> +		return ret;
>>> +	}
>>> +
>>> +	fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
>>> +	crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
>>> +	compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
>>> +
>>> +	fmt->width = MALI_C55_DEFAULT_WIDTH;
>>> +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>> +	fmt->colorspace = V4L2_COLORSPACE_SRGB;
> There are other colorspace-related fields.
>
>>> +	fmt->field = V4L2_FIELD_NONE;
> I wonder if we should really update the sink pad format, or just
> propagate it. If we update it, I think it should be set to defaults on
> both sink pads, not just the active sink pad.
>
>>> +
>>> +	if (active_sink == MALI_C55_RZR_SINK_PAD) {
>>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>> +
>>> +		crop->left = crop->top = 0;
> 		crop->left = 0;
> 		crop->top = 0;
>
>>> +		crop->width = MALI_C55_DEFAULT_WIDTH;
>>> +		crop->height = MALI_C55_DEFAULT_HEIGHT;
>>> +
>>> +		*compose = *crop;
>>> +	} else {
>>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>> +	}
>>> +
>>> +	/* Propagate the format to the source pad */
>>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
>>> +					       0);
>>> +	*src_fmt = *fmt;
>>> +
>>> +	/* In the event this is the bypass pad the mbus code needs correcting */
>>> +	if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
>>> +		src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
>>> +				       struct v4l2_subdev_state *state,
>>> +				       struct v4l2_subdev_mbus_code_enum *code)
>>> +{
>>> +	struct v4l2_mbus_framefmt *sink_fmt;
>>> +	const struct mali_c55_isp_fmt *fmt;
>>> +	unsigned int index = 0;
>>> +	u32 sink_pad;
>>> +
>>> +	switch (code->pad) {
>>> +	case MALI_C55_RZR_SINK_PAD:
>>> +		if (code->index)
>>> +			return -EINVAL;
>>> +
>>> +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>> +
>>> +		return 0;
>>> +	case MALI_C55_RZR_SOURCE_PAD:
>>> +		sink_pad = mali_c55_rzr_get_active_sink(state);
>>> +		sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
>>> +
>>> +		/*
>>> +		 * If the active route is from the Bypass sink pad, then the
>>> +		 * source pad is a simple passthrough of the sink format,
>>> +		 * downshifted to 16-bits.
>>> +		 */
>>> +
>>> +		if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>> +			if (code->index)
>>> +				return -EINVAL;
>>> +
>>> +			code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
>>> +			if (!code->code)
>>> +				return -EINVAL;
>>> +
>>> +			return 0;
>>> +		}
>>> +
>>> +		/*
>>> +		 * If the active route is from the non-bypass sink then we can
>>> +		 * select either RGB or conversion to YUV.
>>> +		 */
>>> +
>>> +		if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
>>> +			return -EINVAL;
>>> +
>>> +		code->code = rzr_non_bypass_src_fmts[code->index];
>>> +
>>> +		return 0;
>>> +	case MALI_C55_RZR_SINK_BYPASS_PAD:
>>> +		for_each_mali_isp_fmt(fmt) {
>>> +			if (index++ == code->index) {
>>> +				code->code = fmt->code;
>>> +				return 0;
>>> +			}
>>> +		}
>>> +
>>> +		break;
>>> +	}
>>> +
>>> +	return -EINVAL;
>>> +}
>>> +
>>> +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
>>> +					struct v4l2_subdev_state *state,
>>> +					struct v4l2_subdev_frame_size_enum *fse)
>>> +{
>>> +	if (fse->index)
>>> +		return -EINVAL;
>>> +
>>> +	fse->max_width = MALI_C55_MAX_WIDTH;
>>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
>>> +	fse->min_width = MALI_C55_MIN_WIDTH;
>>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
>>> +				     struct v4l2_subdev_state *state,
>>> +				     struct v4l2_subdev_format *format)
>>> +{
>>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>>> +	struct v4l2_rect *rect;
>>> +	unsigned int sink_pad;
>>> +
>>> +	/*
>>> +	 * Clamp to min/max and then reset crop and compose rectangles to the
>>> +	 * newly applied size.
>>> +	 */
>>> +	clamp_t(unsigned int, fmt->width,
>>> +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>> +	clamp_t(unsigned int, fmt->height,
>>> +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> Please check comments for other components related to the colorspace
> fields, to decide how to handle them here.
>
>>> +
>>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
>>> +	if (sink_pad == MALI_C55_RZR_SINK_PAD) {
> The selection here should depend on format->pad, not the active sink
> pad.
>
>>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>> +
>>> +		rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
>>> +		rect->left = 0;
>>> +		rect->top = 0;
>>> +		rect->width = fmt->width;
>>> +		rect->height = fmt->height;
>>> +
>>> +		rect = v4l2_subdev_state_get_compose(state,
>>> +						     MALI_C55_RZR_SINK_PAD);
>>> +		rect->left = 0;
>>> +		rect->top = 0;
>>> +		rect->width = fmt->width;
>>> +		rect->height = fmt->height;
>>> +	} else {
>>> +		/*
>>> +		 * Make sure the media bus code is one of the supported
>>> +		 * ISP input media bus codes.
>>> +		 */
>>> +		if (!mali_c55_isp_is_format_supported(fmt->code))
>>> +			fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
>>> +	}
>>> +
>>> +	*v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
>>> +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
> Propagation to the source pad, however, should depend on the active
> route. If format->pad is routed to the source pad, you should propagate,
> otherwise, you shouldn't.
>
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
>>> +				       struct v4l2_subdev_state *state,
>>> +				       struct v4l2_subdev_format *format)
>>> +{
>>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>> +						    sd);
>>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>>> +	struct v4l2_mbus_framefmt *sink_fmt;
>>> +	struct v4l2_rect *crop, *compose;
>>> +	unsigned int sink_pad;
>>> +	unsigned int i;
>>> +
>>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
>>> +	sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
>>> +	crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
>>> +	compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
>>> +
>>> +	/* FR Bypass pipe. */
>>> +
>>> +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>> +		/*
>>> +		 * Format on the source pad is the same as the one on the
>>> +		 * sink pad, downshifted to 16-bits.
>>> +		 */
>>> +		fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
>>> +		if (!fmt->code)
>>> +			return -EINVAL;
>>> +
>>> +		/* RAW bypass disables scaling and cropping. */
>>> +		crop->top = compose->top = 0;
>>> +		crop->left = compose->left = 0;
>>> +		fmt->width = crop->width = compose->width = sink_fmt->width;
>>> +		fmt->height = crop->height = compose->height = sink_fmt->height;
> I don't think this is right. This function sets the format on the source
> pad. Subdevs should propagate formats from the sink to the source, not
> the other way around.
>
> The only parameter that can be modified on the source pad (as far as I
> understand) is the media bus code. In the bypass path, I understand it's
> fixed, while in the other path, you can select between RGB and YUV. I
> think the following code is what you need to implement this function.
>
> static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> 				       struct v4l2_subdev_state *state,
> 				       struct v4l2_subdev_format *format)
> {
> 	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> 						    sd);
> 	struct v4l2_mbus_framefmt *fmt;
>
> 	fmt = v4l2_subdev_state_get_format(state, format->pad);
>
> 	/* In the non-bypass path the output format can be selected. */
> 	if (mali_c55_rzr_get_active_sink(state) == MALI_C55_RZR_SINK_PAD) {
> 		unsigned int i;
>
> 		fmt->code = format->format.code;
>
> 		for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> 			if (fmt->code == rzr_non_bypass_src_fmts[i])
> 				break;
> 		}
>
> 		if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts))
> 			fmt->code = rzr_non_bypass_src_fmts[0];
> 	}
>
> 	format->format = *fmt;
>
> 	return 0;
> }
>
>>> +
>>> +		*v4l2_subdev_state_get_format(state,
>>> +					      MALI_C55_RZR_SOURCE_PAD) = *fmt;
>>> +
>>> +		return 0;
>>> +	}
>>> +
>>> +	/* Regular processing pipe. */
>>> +
>>> +	for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
>>> +		if (fmt->code == rzr_non_bypass_src_fmts[i])
>>> +			break;
>>> +	}
>>> +
>>> +	if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
>>> +		dev_dbg(rzr->mali_c55->dev,
>>> +			"Unsupported mbus code 0x%x: using default\n",
>>> +			fmt->code);
> I think you can drop this message.
>
>>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>> +	}
>>> +
>>> +	/*
>>> +	 * The source pad format size comes directly from the sink pad
>>> +	 * compose rectangle.
>>> +	 */
>>> +	fmt->width = compose->width;
>>> +	fmt->height = compose->height;
>>> +
>>> +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
>>> +				struct v4l2_subdev_state *state,
>>> +				struct v4l2_subdev_format *format)
>>> +{
>>> +	/*
>>> +	 * On sink pads fmt is either fixed for the 'regular' processing
>>> +	 * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
>>> +	 * pad.
>>> +	 *
>>> +	 * On source pad sizes are the result of crop+compose on the sink
>>> +	 * pad sizes, while the format depends on the active route.
>>> +	 */
>>> +
>>> +	if (format->pad != MALI_C55_RZR_SOURCE_PAD)
>>> +		return mali_c55_rzr_set_sink_fmt(sd, state, format);
>>> +
>>> +	return mali_c55_rzr_set_source_fmt(sd, state, format);
> Nitpicking,
>
> 	if (format->pad == MALI_C55_RZR_SOURCE_PAD)
> 		return mali_c55_rzr_set_source_fmt(sd, state, format);
>
> 	return mali_c55_rzr_set_sink_fmt(sd, state, format);
>
> to match SOURCE_PAD and source_fmt.
>
>>> +}
>>> +
>>> +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
>>> +				      struct v4l2_subdev_state *state,
>>> +				      struct v4l2_subdev_selection *sel)
>>> +{
>>> +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
>>> +		return -EINVAL;
>>> +
>>> +	if (sel->target != V4L2_SEL_TGT_CROP &&
>>> +	    sel->target != V4L2_SEL_TGT_COMPOSE)
>>> +		return -EINVAL;
>>> +
>>> +	sel->r = sel->target == V4L2_SEL_TGT_CROP
>>> +	       ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
>>> +	       : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
>>> +				      struct v4l2_subdev_state *state,
>>> +				      struct v4l2_subdev_selection *sel)
>>> +{
>>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>> +						    sd);
>>> +	struct v4l2_mbus_framefmt *source_fmt;
>>> +	struct v4l2_mbus_framefmt *sink_fmt;
>>> +	struct v4l2_rect *crop, *compose;
>>> +
>>> +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
>>> +		return -EINVAL;
>>> +
>>> +	if (sel->target != V4L2_SEL_TGT_CROP &&
>>> +	    sel->target != V4L2_SEL_TGT_COMPOSE)
>>> +		return -EINVAL;
>>> +
>>> +	source_fmt = v4l2_subdev_state_get_format(state,
>>> +						  MALI_C55_RZR_SOURCE_PAD);
>>> +	sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
>>> +	compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
>>> +
>>> +	/* RAW bypass disables crop/scaling. */
>>> +	if (mali_c55_format_is_raw(source_fmt->code)) {
>>> +		crop->top = compose->top = 0;
>>> +		crop->left = compose->left = 0;
>>> +		crop->width = compose->width = sink_fmt->width;
>>> +		crop->height = compose->height = sink_fmt->height;
>>> +
>>> +		sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
>>> +
>>> +		return 0;
>>> +	}
>>> +
>>> +	/* During streaming, it is allowed to only change the crop rectangle. */
>>> +	if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
>>> +		return -EINVAL;
>>> +
>>> +	 /*
>>> +	  * Update the desired target and then clamp the crop rectangle to the
>>> +	  * sink format sizes and the compose size to the crop sizes.
>>> +	  */
>>> +	if (sel->target == V4L2_SEL_TGT_CROP)
>>> +		*crop = sel->r;
>>> +	else
>>> +		*compose = sel->r;
>>> +
>>> +	clamp_t(unsigned int, crop->left, 0,  sink_fmt->width);
>>> +	clamp_t(unsigned int, crop->top, 0,  sink_fmt->height);
>>> +	clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
>>> +		sink_fmt->width - crop->left);
>>> +	clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
>>> +		sink_fmt->height - crop->top);
>>> +
>>> +	if (rzr->streaming) {
>>> +		/*
>>> +		 * Apply at runtime a crop rectangle on the resizer's sink only
>>> +		 * if it doesn't require re-programming the scaler output sizes
>>> +		 * as it would require changing the output buffer sizes as well.
>>> +		 */
>>> +		if (sel->r.width < compose->width ||
>>> +		    sel->r.height < compose->height)
>>> +			return -EINVAL;
>>> +
>>> +		*crop = sel->r;
>>> +		mali_c55_rzr_program(rzr, state);
>>> +
>>> +		return 0;
>>> +	}
>>> +
>>> +	compose->left = 0;
>>> +	compose->top = 0;
>>> +	clamp_t(unsigned int, compose->left, 0,  sink_fmt->width);
>>> +	clamp_t(unsigned int, compose->top, 0,  sink_fmt->height);
>>> +	clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
>>> +	clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
>>> +
>>> +	sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
>>> +				    struct v4l2_subdev_state *state,
>>> +				    enum v4l2_subdev_format_whence which,
>>> +				    struct v4l2_subdev_krouting *routing)
>>> +{
>>> +	if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
>>> +	    media_entity_is_streaming(&sd->entity))
>>> +		return -EBUSY;
>>> +
>>> +	return __mali_c55_rzr_set_routing(sd, state, routing);
>>> +}
>>> +
>>> +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
>>> +	.enum_mbus_code		= mali_c55_rzr_enum_mbus_code,
>>> +	.enum_frame_size	= mali_c55_rzr_enum_frame_size,
>>> +	.get_fmt		= v4l2_subdev_get_fmt,
>>> +	.set_fmt		= mali_c55_rzr_set_fmt,
>>> +	.get_selection		= mali_c55_rzr_get_selection,
>>> +	.set_selection		= mali_c55_rzr_set_selection,
>>> +	.set_routing		= mali_c55_rzr_set_routing,
>>> +};
>>> +
>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
> Could this be handled through the .enable_streams() and
> .disable_streams() operations ? They ensure that the stream state stored
> internal is correct. That may not matter much today, but I think it will
> become increasingly important in the future for the V4L2 core.
>
>>> +{
>>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
>>> +	struct v4l2_subdev *sd = &rzr->sd;
>>> +	struct v4l2_subdev_state *state;
>>> +	unsigned int sink_pad;
>>> +
>>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>>> +
>>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
>>> +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>> +		/* Bypass FR pipe processing if the bypass route is active. */
>>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>> +				     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
>>> +				     MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
>>> +		goto unlock_state;
>>> +	}
>>> +
>>> +	/* Disable bypass and use regular processing. */
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>> +			     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
>>> +	mali_c55_rzr_program(rzr, state);
>>> +
>>> +unlock_state:
>>> +	rzr->streaming = true;
> And hopefully you'll be able to replace this with
> v4l2_subdev_is_streaming(), introduced in "[PATCH v6 00/11] media:
> subdev: Improve stream enable/disable machinery" (Sakari has sent a pull
> request for v6.11 yesterday).
>
>>> +	v4l2_subdev_unlock_state(state);
>>> +}
>>> +
>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
>>> +{
>>> +	struct v4l2_subdev *sd = &rzr->sd;
>>> +	struct v4l2_subdev_state *state;
>>> +
>>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>>> +	rzr->streaming = false;
>>> +	v4l2_subdev_unlock_state(state);
>>> +}
>>> +
>>> +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
>>> +	.pad	= &mali_c55_resizer_pad_ops,
>>> +};
>>> +
>>> +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
>>> +				   struct v4l2_subdev_state *state)
>>> +{
>>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>> +						    sd);
>>> +	struct v4l2_subdev_krouting routing = { };
>>> +	struct v4l2_subdev_route *routes;
>>> +	unsigned int i;
>>> +	int ret;
>>> +
>>> +	routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
>>> +	if (!routes)
>>> +		return -ENOMEM;
>>> +
>>> +	for (i = 0; i < rzr->num_routes; ++i) {
>>> +		struct v4l2_subdev_route *route = &routes[i];
>>> +
>>> +		route->sink_pad = i
>>> +				? MALI_C55_RZR_SINK_BYPASS_PAD
>>> +				: MALI_C55_RZR_SINK_PAD;
>>> +		route->source_pad = MALI_C55_RZR_SOURCE_PAD;
>>> +		if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
>>> +			route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
>>> +	}
>>> +
>>> +	routing.num_routes = rzr->num_routes;
>>> +	routing.routes = routes;
>>> +
>>> +	ret = __mali_c55_rzr_set_routing(sd, state, &routing);
>>> +	kfree(routes);
>>> +
>>> +	return ret;
> I think this could be simplified.
>
> 	struct v4l2_subdev_route routes[2] = {
> 		{
> 			.sink_pad = MALI_C55_RZR_SINK_PAD,
> 			.source_pad = MALI_C55_RZR_SOURCE_PAD,
> 			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> 		}, {
> 			.sink_pad = MALI_C55_RZR_SINK_BYPASS_PAD,
> 			.source_pad = MALI_C55_RZR_SOURCE_PAD,
> 		},
> 	};
> 	struct v4l2_subdev_krouting routing = {
> 		.num_routes = rzr->num_routes,
> 		.routes = routes,
> 	};
>
> 	return __mali_c55_rzr_set_routing(sd, state, &routing);
>
>>> +}
>>> +
>>> +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
>>> +	.init_state = mali_c55_rzr_init_state,
>>> +};
>>> +
>>> +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
>>> +						  unsigned int index)
>>> +{
>>> +	const unsigned int scaler_filt_coefmem_addrs[][2] = {
>>> +		[MALI_C55_RZR_FR] = {
>>> +			0x034A8, /* hfilt */
>>> +			0x044A8  /* vfilt */
>> Lowercase hex constants.
> And addresses belong to the mali-c55-registers.h file.
>
>>> +		},
>>> +		[MALI_C55_RZR_DS] = {
>>> +			0x014A8, /* hfilt */
>>> +			0x024A8  /* vfilt */
>>> +		},
>>> +	};
>>> +	unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
>>> +	unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
>>> +	unsigned int i, j;
>>> +
>>> +	for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
>>> +		for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
>>> +			mali_c55_write(mali_c55, haddr,
>>> +				mali_c55_scaler_h_filter_coefficients[i][j]);
>>> +			mali_c55_write(mali_c55, vaddr,
>>> +				mali_c55_scaler_v_filter_coefficients[i][j]);
>>> +
>>> +			haddr += sizeof(u32);
>>> +			vaddr += sizeof(u32);
>>> +		}
>>> +	}
> How about memcpy_toio() ? I suppose this function isn't
> performance sensitive, so maybe usage of mali_c55_write() is better from
> a consistency point of view.
>
>>> +}
>>> +
>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
>>> +{
>>> +	unsigned int i;
>>> +	int ret;
>>> +
>>> +	for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
> Moving the inner content to a separate mali_c55_register_resizer()
> function would increase readability I think, and remove usage of gotos.
> I would probably do the same for unregistration too, for consistency.
>
>>> +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
>>> +		struct v4l2_subdev *sd = &rzr->sd;
>>> +		unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
>>> +
>>> +		rzr->id = i;
>>> +		rzr->streaming = false;
>>> +
>>> +		if (rzr->id == MALI_C55_RZR_FR)
>>> +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
>>> +		else
>>> +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
>>> +
>>> +		mali_c55_resizer_program_coefficients(mali_c55, i);
> Should this be done at stream start, given that power may be cut off
> between streaming sessions ?
>
>>> +
>>> +		v4l2_subdev_init(sd, &mali_c55_resizer_ops);
>>> +		sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
>>> +			     | V4L2_SUBDEV_FL_STREAMS;
>>> +		sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
>>> +		sd->internal_ops = &mali_c55_resizer_internal_ops;
>>> +		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
> 		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s resizer %s",
>
> and drop the "resizer " prefix from mali_c55_resizer_names. You can also
> make mali_c55_resizer_names a local static const variable.
>
>>> +			 MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
>>> +
>>> +		rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
>>> +		rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
>>> +
>>> +		/* Only the FR pipe has a bypass pad. */
>>> +		if (rzr->id == MALI_C55_RZR_FR) {
>>> +			rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
>>> +							MEDIA_PAD_FL_SINK;
>>> +			rzr->num_routes = 2;
>>> +		} else {
>>> +			num_pads -= 1;
>>> +			rzr->num_routes = 1;
>>> +		}
>>> +
>>> +		ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
>>> +		if (ret)
>>> +			return ret;
>>> +
>>> +		ret = v4l2_subdev_init_finalize(sd);
>>> +		if (ret)
>>> +			goto err_cleanup;
>>> +
>>> +		ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>> +		if (ret)
>>> +			goto err_cleanup;
>>> +
>>> +		rzr->mali_c55 = mali_c55;
>>> +	}
>>> +
>>> +	return 0;
>>> +
>>> +err_cleanup:
>>> +	for (; i >= 0; --i) {
>>> +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
>>> +		struct v4l2_subdev *sd = &rzr->sd;
>>> +
>>> +		v4l2_subdev_cleanup(sd);
>>> +		media_entity_cleanup(&sd->entity);
>>> +	}
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
>>> +{
>>> +	unsigned int i;
>>> +
>>> +	for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
>>> +		struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
>>> +
>>> +		if (!resizer->mali_c55)
>>> +			continue;
>>> +
>>> +		v4l2_device_unregister_subdev(&resizer->sd);
>>> +		v4l2_subdev_cleanup(&resizer->sd);
>>> +		media_entity_cleanup(&resizer->sd.entity);
>>> +	}
>>> +}
>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>> new file mode 100644
>>> index 000000000000..c7e699741c6d
>>> --- /dev/null
>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>> @@ -0,0 +1,402 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +/*
>>> + * ARM Mali-C55 ISP Driver - Test pattern generator
>>> + *
>>> + * Copyright (C) 2024 Ideas on Board Oy
>>> + */
>>> +
>>> +#include <linux/minmax.h>
>>> +#include <linux/string.h>
>>> +
>>> +#include <media/media-entity.h>
>>> +#include <media/v4l2-ctrls.h>
>>> +#include <media/v4l2-subdev.h>
>>> +
>>> +#include "mali-c55-common.h"
>>> +#include "mali-c55-registers.h"
>>> +
>>> +#define MALI_C55_TPG_SRC_PAD		0
>>> +#define MALI_C55_TPG_FIXED_HBLANK	0x20
>>> +#define MALI_C55_TPG_MAX_VBLANK		0xFFFF
>> Lowercase hex constants.
>>
>>> +#define MALI_C55_TPG_PIXEL_RATE		100000000
>> This should be exposed to applications using the V4L2_CID_PIXEL_RATE
>> control (read-only).
>>
>>> +
>>> +static const char * const mali_c55_tpg_test_pattern_menu[] = {
>>> +	"Flat field",
>>> +	"Horizontal gradient",
>>> +	"Vertical gradient",
>>> +	"Vertical bars",
>>> +	"Arbitrary rectangle",
>>> +	"White frame on black field"
>>> +};
>>> +
>>> +static const u32 mali_c55_tpg_mbus_codes[] = {
>>> +	MEDIA_BUS_FMT_SRGGB20_1X20,
>>> +	MEDIA_BUS_FMT_RGB202020_1X60,
>>> +};
>>> +
>>> +static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
>>> +				       int *def_vblank, int *min_vblank)
>> unsigned int ?
>>
>>> +{
>>> +	unsigned int hts;
>>> +	int tgt_fps;
>>> +	int vblank;
>>> +
>>> +	hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
>>> +
>>> +	/*
>>> +	 * The ISP has minimum vertical blanking requirements that must be
>>> +	 * adhered to by the TPG. The minimum is a function of the Iridix blocks
>>> +	 * clocking requirements and the width of the image and horizontal
>>> +	 * blanking, but if we assume the worst case iVariance and sVariance
>>> +	 * values then it boils down to the below.
>>> +	 */
>>> +	*min_vblank = 15 + (120500 / hts);
>> I wonder if this should round up.
>>
>>> +
>>> +	/*
>>> +	 * We need to set a sensible default vblank for whatever format height
>>> +	 * we happen to be given from set_fmt(). This function just targets
>>> +	 * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
>>> +	 * If we can't get 5fps we'll take whatever the minimum vblank gives us.
>>> +	 */
>>> +	tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
>>> +
>>> +	if (tgt_fps < 5)
>>> +		vblank = *min_vblank;
>>> +	else
>>> +		vblank = MALI_C55_TPG_PIXEL_RATE / hts
>>> +		       / max(rounddown(tgt_fps, 15), 5);
>>> +
>>> +	*def_vblank = ALIGN_DOWN(vblank, 2) - format->height;
>> "vblank = vblank - height" doesn't seem right. The "else" branch stores
>> a vts in vblank, which doesn't seem right either. Maybe you meant
>> something like
>>
>> 	if (tgt_fps < 5)
>> 		def_vts = *min_vblank + format->height;
>> 	else
>> 		def_vts = MALI_C55_TPG_PIXEL_RATE / hts
>> 			/ max(rounddown(tgt_fps, 15), 5);
>>
>> 	*def_vblank = ALIGN_DOWN(def_vts, 2) - format->height;
>>
>>> +}
>>> +
>>> +static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
>>> +{
>>> +	struct mali_c55_tpg *tpg = container_of(ctrl->handler,
>>> +						struct mali_c55_tpg,
>>> +						ctrls.handler);
>>> +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
>>> +
>> Should you return here if the pipeline isn't streaming ?
>>
>>> +	switch (ctrl->id) {
>>> +	case V4L2_CID_TEST_PATTERN:
>>> +		mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
>>> +			       ctrl->val);
>>> +		break;
>>> +	case V4L2_CID_VBLANK:
>>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
>>> +				     MALI_C55_REG_VBLANK_MASK, ctrl->val);
>>> +		break;
>>> +	default:
>>> +		dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
>>> +		return -EINVAL;
>> Can this happen ?
>>
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
>>> +	.s_ctrl = &mali_c55_tpg_s_ctrl,
>>> +};
>>> +
>>> +static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
>>> +				   struct v4l2_subdev *sd)
>>> +{
>>> +	struct v4l2_subdev_state *state;
>>> +	struct v4l2_mbus_framefmt *fmt;
>>> +
>>> +	/*
>>> +	 * hblank needs setting, but is a read-only control and thus won't be
>>> +	 * called during __v4l2_ctrl_handler_setup(). Do it here instead.
>>> +	 */
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
>>> +			     MALI_C55_REG_HBLANK_MASK,
>>> +			     MALI_C55_TPG_FIXED_HBLANK);
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>> +			     MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
>>> +
>>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>> +
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>> +			     MALI_C55_TEST_PATTERN_RGB_MASK,
>>> +			     fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
>>> +					  0x01 : 0x0);
>>> +
>>> +	v4l2_subdev_unlock_state(state);
>>> +}
>>> +
>>> +static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
>>> +{
>>> +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
>>> +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
>>> +
>>> +	if (!enable) {
>>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>> +				MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
>>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>> +				MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
>>> +		return 0;
>>> +	}
>>> +
>>> +	/*
>>> +	 * One might reasonably expect the framesize to be set here
>>> +	 * given it's configurable in .set_fmt(), but it's done in the
>>> +	 * ISP subdevice's stream on func instead, as the same register
>> s/func/function/
>>
>>> +	 * is also used to indicate the size of the data coming from the
>>> +	 * sensor.
>>> +	 */
>>> +	mali_c55_tpg_configure(mali_c55, sd);
>> 	mali_c55_tpg_configure(tpg);
>>
>>> +	__v4l2_ctrl_handler_setup(sd->ctrl_handler);
>>> +
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>> +			     MALI_C55_TEST_PATTERN_ON_OFF,
>>> +			     MALI_C55_TEST_PATTERN_ON_OFF);
>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>> +			     MALI_C55_REG_GEN_VIDEO_ON_MASK,
>>> +			     MALI_C55_REG_GEN_VIDEO_ON_MASK);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
>>> +	.s_stream = &mali_c55_tpg_s_stream,
>> Can we use .enable_streams() and .disable_streams() ?
>>
>>> +};
>>> +
>>> +static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
>>> +				       struct v4l2_subdev_state *state,
>>> +				       struct v4l2_subdev_mbus_code_enum *code)
>>> +{
>>> +	if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>> +		return -EINVAL;
>>> +
>>> +	code->code = mali_c55_tpg_mbus_codes[code->index];
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
>>> +					struct v4l2_subdev_state *state,
>>> +					struct v4l2_subdev_frame_size_enum *fse)
>>> +{
>> You sohuld verify here that fse->code is a supported value and return
>> -EINVAL otherwise.
>>
>>> +	if (fse->index > 0 || fse->pad > sd->entity.num_pads)
>> Drop the pad check, it's done in the subdev core already.
>>
>>> +		return -EINVAL;
>>> +
>>> +	fse->min_width = MALI_C55_MIN_WIDTH;
>>> +	fse->max_width = MALI_C55_MAX_WIDTH;
>>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
>>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
>>> +				struct v4l2_subdev_state *state,
>>> +				struct v4l2_subdev_format *format)
>>> +{
>>> +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
>>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
>>> +	int vblank_def, vblank_min;
>>> +	unsigned int i;
>>> +
>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>>> +		if (fmt->code == mali_c55_tpg_mbus_codes[i])
>>> +			break;
>>> +	}
>>> +
>>> +	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>> +
>>> +	/*
>>> +	 * The TPG says that the test frame timing generation logic expects a
>>> +	 * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
>>> +	 * handle anything smaller than 128x128 it seems pointless to allow a
>>> +	 * smaller frame.
>>> +	 */
>>> +	clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
>>> +		MALI_C55_MAX_WIDTH);
>>> +	clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
>>> +		MALI_C55_MAX_HEIGHT);
>>> +
>>> +	*v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;
>> You're allowing userspace to set fmt->field, as well as all the
>> colorspace parameters, to random values. I would instead do something
>> like
>>
>> 	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>> 		if (format->format.code == mali_c55_tpg_mbus_codes[i])
>> 			break;
>> 	}
>>
>> 	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>> 		format->format.code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>
>> 	format->format.width = clamp(format->format.width,
>> 				     MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>> 	format->format.height = clamp(format->format.height,
>> 				      MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>
>> 	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>> 	fmt->code = format->format.code;
>> 	fmt->width = format->format.width;
>> 	fmt->height = format->format.height;
>>
>> 	format->format = *fmt;
>>
>> Alternatively (which I think I like better),
>>
>> 	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>
>> 	fmt->code = format->format.code;
>>
>> 	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>> 		if (fmt->code == mali_c55_tpg_mbus_codes[i])
>> 			break;
>> 	}
>>
>> 	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>> 		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>
>> 	fmt->width = clamp(format->format.width,
>> 			   MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>> 	fmt->height = clamp(format->format.height,
>> 			    MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>
>> 	format->format = *fmt;
>>
>>> +
>>> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY)
>>> +		return 0;
>>> +
>>> +	__mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
>>> +	__v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
>>> +				 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
>>> +	__v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);
>> Move those three calls to a separate function, it will be reused below.
>> I'd name is mali_c55_tpg_update_vblank(). You can fold
>> __mali_c55_tpg_calc_vblank() in it.
>>
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
>>> +	.enum_mbus_code		= mali_c55_tpg_enum_mbus_code,
>>> +	.enum_frame_size	= mali_c55_tpg_enum_frame_size,
>>> +	.get_fmt		= v4l2_subdev_get_fmt,
>>> +	.set_fmt		= mali_c55_tpg_set_fmt,
>>> +};
>>> +
>>> +static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
>>> +	.video	= &mali_c55_tpg_video_ops,
>>> +	.pad	= &mali_c55_tpg_pad_ops,
>>> +};
>>> +
>>> +static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
>>> +				   struct v4l2_subdev_state *sd_state)
>> You name this variable state in every other subdev operation handler.
>>
>>> +{
>>> +	struct v4l2_mbus_framefmt *fmt =
>>> +		v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
>>> +
>>> +	fmt->width = MALI_C55_DEFAULT_WIDTH;
>>> +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>> +	fmt->field = V4L2_FIELD_NONE;
>>> +	fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>> Initialize the colorspace fields too.
>>
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
>>> +	.init_state = mali_c55_tpg_init_state,
>>> +};
>>> +
>>> +static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
>>> +{
>>> +	struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
>>> +	struct v4l2_subdev *sd = &mali_c55->tpg.sd;
>>> +	struct v4l2_mbus_framefmt *format;
>>> +	struct v4l2_subdev_state *state;
>>> +	int vblank_def, vblank_min;
>>> +	int ret;
>>> +
>>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>>> +	format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>> +
>>> +	ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);
>> You have 3 controls.
>>
>>> +	if (ret)
>>> +		goto err_unlock;
>>> +
>>> +	ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
>>> +				&mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
>>> +				ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
>>> +				0, 3, mali_c55_tpg_test_pattern_menu);
>>> +
>>> +	/*
>>> +	 * We fix hblank at the minimum allowed value and control framerate
>>> +	 * solely through the vblank control.
>>> +	 */
>>> +	ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
>>> +				&mali_c55_tpg_ctrl_ops,
>>> +				V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
>>> +				MALI_C55_TPG_FIXED_HBLANK, 1,
>>> +				MALI_C55_TPG_FIXED_HBLANK);
>>> +	if (ctrls->hblank)
>>> +		ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
>>> +
>>> +	__mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);
>> Drop this and initialize the control with default values. You can then
>> update the value by calling mali_c55_tpg_update_vblank() in
>> mali_c55_register_tpg().
>>
>> The reason is to share the same mutex between the control handler and
>> the subdev active state without having to add a separate mutex in the
>> mali_c55_tpg structure. The simplest way to do so is to initialize the
>> controls first, set sd->state_lock to point to the control handler lock,
>> and call v4l2_subdev_init_finalize() as the last step. As a consequence,
>> you can't access the active state when initializing controls.
>>
>> You can alternatively keep the lock in mali_c55_tpg and set
>> sd->state_lock to point to it, but I think that's more complex.
>>
>>> +	ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
>>> +					  &mali_c55_tpg_ctrl_ops,
>>> +					  V4L2_CID_VBLANK, vblank_min,
>>> +					  MALI_C55_TPG_MAX_VBLANK, 1,
>>> +					  vblank_def);
>>> +
>>> +	if (ctrls->handler.error) {
>>> +		dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
>>> +		ret = ctrls->handler.error;
>>> +		goto err_free_handler;
>>> +	}
>>> +
>>> +	ctrls->handler.lock = &mali_c55->tpg.lock;
>> Drop this and drop the mutex. The control handler will use its internal
>> mutex.
>>
>>> +	mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
>>> +
>>> +	v4l2_subdev_unlock_state(state);
>>> +
>>> +	return 0;
>>> +
>>> +err_free_handler:
>>> +	v4l2_ctrl_handler_free(&ctrls->handler);
>>> +err_unlock:
>>> +	v4l2_subdev_unlock_state(state);
>>> +	return ret;
>>> +}
>>> +
>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55)
>>> +{
>>> +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
>>> +	struct v4l2_subdev *sd = &tpg->sd;
>>> +	struct media_pad *pad = &tpg->pad;
>>> +	int ret;
>>> +
>>> +	mutex_init(&tpg->lock);
>>> +
>>> +	v4l2_subdev_init(sd, &mali_c55_tpg_ops);
>>> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
>>> +	sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
>> Should we introduce a TPG function ?
>>
>>> +	sd->internal_ops = &mali_c55_tpg_internal_ops;
>>> +	strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
>>> +
>>> +	pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;
>> I don't think MEDIA_PAD_FL_MUST_CONNECT is right.
>>
>>> +	ret = media_entity_pads_init(&sd->entity, 1, pad);
>>> +	if (ret) {
>>> +		dev_err(mali_c55->dev,
>>> +			"Failed to initialize media entity pads\n");
>>> +		goto err_destroy_mutex;
>>> +	}
>>> +
>> 	sd->state_lock = sd->ctrl_handler->lock;
>>
>> to use the same lock for the controls and the active state. You need to
>> move this line and the v4l2_subdev_init_finalize() call after
>> mali_c55_tpg_init_controls() to get the control handler lock initialized
>> first.
>>
>>> +	ret = v4l2_subdev_init_finalize(sd);
>>> +	if (ret)
>>> +		goto err_cleanup_media_entity;
>>> +
>>> +	ret = mali_c55_tpg_init_controls(mali_c55);
>>> +	if (ret) {
>>> +		dev_err(mali_c55->dev,
>>> +			"Error initialising controls\n");
>>> +		goto err_cleanup_subdev;
>>> +	}
>>> +
>>> +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>> +	if (ret) {
>>> +		dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
>>> +		goto err_free_ctrl_handler;
>>> +	}
>>> +
>>> +	/*
>>> +	 * By default the colour settings lead to a very dim image that is
>>> +	 * nearly indistinguishable from black on some monitor settings. Ramp
>>> +	 * them up a bit so the image is brighter.
>>> +	 */
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
>>> +		       MALI_C55_TPG_BACKGROUND_MAX);
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
>>> +		       MALI_C55_TPG_BACKGROUND_MAX);
>>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
>>> +		       MALI_C55_TPG_BACKGROUND_MAX);
>>> +
>>> +	tpg->mali_c55 = mali_c55;
>>> +
>>> +	return 0;
>>> +
>>> +err_free_ctrl_handler:
>>> +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
>>> +err_cleanup_subdev:
>>> +	v4l2_subdev_cleanup(sd);
>>> +err_cleanup_media_entity:
>>> +	media_entity_cleanup(&sd->entity);
>>> +err_destroy_mutex:
>>> +	mutex_destroy(&tpg->lock);
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
>>> +{
>>> +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
>>> +
>>> +	if (!tpg->mali_c55)
>>> +		return;
>>> +
>>> +	v4l2_device_unregister_subdev(&tpg->sd);
>>> +	v4l2_subdev_cleanup(&tpg->sd);
>>> +	media_entity_cleanup(&tpg->sd.entity);
>>> +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
>> Free the control handler just after v4l2_device_unregister_subdev() to
>> match the order in mali_c55_register_tpg().
>>
>>> +	mutex_destroy(&tpg->lock);
>>> +}

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Dan Scally]

Hi Laurent

On 20/06/2024 15:33, Dan Scally wrote:
> Hi Laurent, thanks for the comments
>
> On 30/05/2024 22:43, Laurent Pinchart wrote:
>> And now the second part of the review, addressing mali-c55-capture.c and
>> mali-c55-resizer.c. I've reviewed the code from the bottom up, so some
>> messages may be repeated in an order that seems weird. Sorry about that.
>>
>> On Thu, May 30, 2024 at 03:15:10AM +0300, Laurent Pinchart wrote:
>>> On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
>>>> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
>>>> V4L2 and Media Controller compliant and creates subdevices to manage
>>>> the ISP itself, its internal test pattern generator as well as the
>>>> crop, scaler and output format functionality for each of its two
>>>> output devices.
>>>>
>>>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
>>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>>>> ---
>>>> Changes in v5:
>>>>
>>>>     - Reworked input formats - previously we allowed representing input data
>>>>       as any 8-16 bit format. Now we only allow input data to be represented
>>>>       by the new 20-bit bayer formats, which is corrected to the equivalent
>>>>       16-bit format in RAW bypass mode.
>>>>     - Stopped bypassing blocks that we haven't added supporting parameters
>>>>       for yet.
>>>>     - Addressed most of Sakari's comments from the list
>>>>
>>>> Changes not yet made in v5:
>>>>
>>>>     - The output pipelines can still be started and stopped independently of
>>>>       one another - I'd like to discuss that more.
>>>>     - the TPG subdev still uses .s_stream() - I need to rebase onto a tree
>>>>       with working .enable_streams() for a single-source-pad subdevice.
>>>>
>>>> Changes in v4:
>>>>
>>>>     - Reworked mali_c55_update_bits() to internally perform the bit-shift
>>> I really don't like that, it makes the code very confusing, even more so
>>> as it differs from regmap_update_bits().
>>>
>>> Look at this for instance:
>>>
>>>     mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>                  MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
>>>                  MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
>>>
>>> It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
>>> BIT(0).
>>>
>>> Sorry, I know it will be painful, but this change needs to be reverted.
>>>
>>>>     - Reworked the resizer to allow cropping during streaming
>>>>     - Fixed a bug in NV12 output
>>>>
>>>> Changes in v3:
>>>>
>>>>     - Mostly minor fixes suggested by Sakari
>>>>     - Fixed the sequencing of vb2 buffers to be synchronised across the two
>>>>       capture devices.
>>>>
>>>> Changes in v2:
>>>>
>>>>     - Clock handling
>>>>     - Fixed the warnings raised by the kernel test robot
>>>>
>>>>   drivers/media/platform/Kconfig                |   1 +
>>>>   drivers/media/platform/Makefile               |   1 +
>>>>   drivers/media/platform/arm/Kconfig            |   5 +
>>>>   drivers/media/platform/arm/Makefile           |   2 +
>>>>   drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
>>>>   drivers/media/platform/arm/mali-c55/Makefile  |   9 +
>>>>   .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
>>>>   .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
>>>>   .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
>>>>   .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
>>>>   .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
>>>>   .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
>>>>   .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
>>>>   .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
>>>>   14 files changed, 4452 insertions(+)
>>>>   create mode 100644 drivers/media/platform/arm/Kconfig
>>>>   create mode 100644 drivers/media/platform/arm/Makefile
>>>>   create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
>>>>   create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
>>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>> I've skipped review of capture.c and resizer.c as I already have plenty
>>> of comments for the other files, and it's getting late. I'll try to
>>> review the rest tomorrow.
>>>
>>>> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
>>>> index 2d79bfc68c15..c929169766aa 100644
>>>> --- a/drivers/media/platform/Kconfig
>>>> +++ b/drivers/media/platform/Kconfig
>>>> @@ -65,6 +65,7 @@ config VIDEO_MUX
>>>>   source "drivers/media/platform/allegro-dvt/Kconfig"
>>>>   source "drivers/media/platform/amlogic/Kconfig"
>>>>   source "drivers/media/platform/amphion/Kconfig"
>>>> +source "drivers/media/platform/arm/Kconfig"
>>>>   source "drivers/media/platform/aspeed/Kconfig"
>>>>   source "drivers/media/platform/atmel/Kconfig"
>>>>   source "drivers/media/platform/broadcom/Kconfig"
>>>> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
>>>> index da17301f7439..9a647abd5218 100644
>>>> --- a/drivers/media/platform/Makefile
>>>> +++ b/drivers/media/platform/Makefile
>>>> @@ -8,6 +8,7 @@
>>>>   obj-y += allegro-dvt/
>>>>   obj-y += amlogic/
>>>>   obj-y += amphion/
>>>> +obj-y += arm/
>>>>   obj-y += aspeed/
>>>>   obj-y += atmel/
>>>>   obj-y += broadcom/
>>>> diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
>>>> new file mode 100644
>>>> index 000000000000..4f0764c329c7
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/Kconfig
>>>> @@ -0,0 +1,5 @@
>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>> +
>>>> +comment "ARM media platform drivers"
>>>> +
>>>> +source "drivers/media/platform/arm/mali-c55/Kconfig"
>>>> diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
>>>> new file mode 100644
>>>> index 000000000000..8cc4918725ef
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/Makefile
>>>> @@ -0,0 +1,2 @@
>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>> +obj-y += mali-c55/
>>>> diff --git a/drivers/media/platform/arm/mali-c55/Kconfig 
>>>> b/drivers/media/platform/arm/mali-c55/Kconfig
>>>> new file mode 100644
>>>> index 000000000000..602085e28b01
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/Kconfig
>>>> @@ -0,0 +1,18 @@
>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>> +config VIDEO_MALI_C55
>>>> +    tristate "ARM Mali-C55 Image Signal Processor driver"
>>>> +    depends on V4L_PLATFORM_DRIVERS
>>>> +    depends on VIDEO_DEV && OF
>>>> +    depends on ARCH_VEXPRESS || COMPILE_TEST
>>>> +    select MEDIA_CONTROLLER
>>>> +    select VIDEO_V4L2_SUBDEV_API
>>>> +    select VIDEOBUF2_DMA_CONTIG
>>>> +    select VIDEOBUF2_VMALLOC
>>>> +    select V4L2_FWNODE
>>>> +    select GENERIC_PHY_MIPI_DPHY
>>> Alphabetical order ?
>>>
>>>> +    default n
>>> That's the default, you don't have to specify ti.
>>>
>>>> +    help
>>>> +      Enable this to support Arm's Mali-C55 Image Signal Processor.
>>>> +
>>>> +      To compile this driver as a module, choose M here: the module
>>>> +      will be called mali-c55.
>>>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile 
>>>> b/drivers/media/platform/arm/mali-c55/Makefile
>>>> new file mode 100644
>>>> index 000000000000..77dcb2fbf0f4
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
>>>> @@ -0,0 +1,9 @@
>>>> +# SPDX-License-Identifier: GPL-2.0
>>>> +
>>>> +mali-c55-y := mali-c55-capture.o \
>>>> +          mali-c55-core.o \
>>>> +          mali-c55-isp.o \
>>>> +          mali-c55-tpg.o \
>>>> +          mali-c55-resizer.o
>>> Alphabetical order here too.
>>>
>>>> +
>>>> +obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c 
>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>> new file mode 100644
>>>> index 000000000000..1d539ac9c498
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>> @@ -0,0 +1,951 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Video capture devices
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#include <linux/cleanup.h>
>>>> +#include <linux/minmax.h>
>>>> +#include <linux/pm_runtime.h>
>>>> +#include <linux/string.h>
>>>> +#include <linux/videodev2.h>
>>>> +
>>>> +#include <media/v4l2-dev.h>
>>>> +#include <media/v4l2-event.h>
>>>> +#include <media/v4l2-ioctl.h>
>>>> +#include <media/v4l2-subdev.h>
>>>> +#include <media/videobuf2-core.h>
>>>> +#include <media/videobuf2-dma-contig.h>
>>>> +
>>>> +#include "mali-c55-common.h"
>>>> +#include "mali-c55-registers.h"
>>>> +
>>>> +static const struct mali_c55_fmt mali_c55_fmts[] = {
>>>> +    /*
>>>> +     * This table is missing some entries which need further work or
>>>> +     * investigation:
>>>> +     *
>>>> +     * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
>>>> +     * Base mode 5 is "Generic Data"
>>>> +     * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
>>>> +     * Base mode 9 seems to have no V4L2 equivalent
>>>> +     * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
>>>> +     * equivalent
>>>> +     */
>>>> +    {
>>>> +        .fourcc = V4L2_PIX_FMT_ARGB2101010,
>>>> +        .mbus_codes = {
>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
>>>> +        },
>>>> +        .is_raw = false,
>>>> +        .registers = {
>>>> +            .base_mode = MALI_C55_OUTPUT_A2R10G10B10,
>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +        }
>>>> +    },
>>>> +    {
>>>> +        .fourcc = V4L2_PIX_FMT_RGB565,
>>>> +        .mbus_codes = {
>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
>>>> +        },
>>>> +        .is_raw = false,
>>>> +        .registers = {
>>>> +            .base_mode = MALI_C55_OUTPUT_RGB565,
>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +        }
>>>> +    },
>>>> +    {
>>>> +        .fourcc = V4L2_PIX_FMT_BGR24,
>>>> +        .mbus_codes = {
>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
>>>> +        },
>>>> +        .is_raw = false,
>>>> +        .registers = {
>>>> +            .base_mode = MALI_C55_OUTPUT_RGB24,
>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +        }
>>>> +    },
>>>> +    {
>>>> +        .fourcc = V4L2_PIX_FMT_YUYV,
>>>> +        .mbus_codes = {
>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>> +        },
>>>> +        .is_raw = false,
>>>> +        .registers = {
>>>> +            .base_mode = MALI_C55_OUTPUT_YUY2,
>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +        }
>>>> +    },
>>>> +    {
>>>> +        .fourcc = V4L2_PIX_FMT_UYVY,
>>>> +        .mbus_codes = {
>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>> +        },
>>>> +        .is_raw = false,
>>>> +        .registers = {
>>>> +            .base_mode = MALI_C55_OUTPUT_UYVY,
>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +        }
>>>> +    },
>>>> +    {
>>>> +        .fourcc = V4L2_PIX_FMT_Y210,
>>>> +        .mbus_codes = {
>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>> +        },
>>>> +        .is_raw = false,
>>>> +        .registers = {
>>>> +            .base_mode = MALI_C55_OUTPUT_Y210,
>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +        }
>>>> +    },
>>>> +    /*
>>>> +     * This is something of a hack, the ISP thinks it's running NV12M but
>>>> +     * by setting uv_plane = 0 we simply discard that planes and only output
>>>> +     * the Y-plane.
>>>> +     */
>>>> +    {
>>>> +        .fourcc = V4L2_PIX_FMT_GREY,
>>>> +        .mbus_codes = {
>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>> +        },
>>>> +        .is_raw = false,
>>>> +        .registers = {
>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +        }
>>>> +    },
>>>> +    {
>>>> +        .fourcc = V4L2_PIX_FMT_NV12M,
>>>> +        .mbus_codes = {
>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>> +        },
>>>> +        .is_raw = false,
>>>> +        .registers = {
>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
>>>> +        }
>>>> +    },
>>>> +    {
>>>> +        .fourcc = V4L2_PIX_FMT_NV21M,
>>>> +        .mbus_codes = {
>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>> +        },
>>>> +        .is_raw = false,
>>>> +        .registers = {
>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
>>>> +        }
>>>> +    },
>>>> +    /*
>>>> +     * RAW uncompressed formats are all packed in 16 bpp.
>>>> +     * TODO: Expand this list to encompass all possible RAW formats.
>>>> +     */
>>>> +    {
>>>> +        .fourcc = V4L2_PIX_FMT_SRGGB16,
>>>> +        .mbus_codes = {
>>>> +            MEDIA_BUS_FMT_SRGGB16_1X16,
>>>> +        },
>>>> +        .is_raw = true,
>>>> +        .registers = {
>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +        }
>>>> +    },
>>>> +    {
>>>> +        .fourcc = V4L2_PIX_FMT_SBGGR16,
>>>> +        .mbus_codes = {
>>>> +            MEDIA_BUS_FMT_SBGGR16_1X16,
>>>> +        },
>>>> +        .is_raw = true,
>>>> +        .registers = {
>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +        }
>>>> +    },
>>>> +    {
>>>> +        .fourcc = V4L2_PIX_FMT_SGBRG16,
>>>> +        .mbus_codes = {
>>>> +            MEDIA_BUS_FMT_SGBRG16_1X16,
>>>> +        },
>>>> +        .is_raw = true,
>>>> +        .registers = {
>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +        }
>>>> +    },
>>>> +    {
>>>> +        .fourcc = V4L2_PIX_FMT_SGRBG16,
>>>> +        .mbus_codes = {
>>>> +            MEDIA_BUS_FMT_SGRBG16_1X16,
>>>> +        },
>>>> +        .is_raw = true,
>>>> +        .registers = {
>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>> +        }
>>>> +    },
>>>> +};
>>>> +
>>>> +static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
>>>> +                           u32 code)
>>>> +{
>>>> +    unsigned int i;
>>>> +
>>>> +    for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
>>>> +        if (fmt->mbus_codes[i] == code)
>>>> +            return true;
>>>> +    }
>>>> +
>>>> +    return false;
>>>> +}
>>>> +
>>>> +bool mali_c55_format_is_raw(unsigned int mbus_code)
>>>> +{
>>>> +    unsigned int i;
>>>> +
>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>>> +        if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
>>>> +            return mali_c55_fmts[i].is_raw;
>>>> +    }
>>>> +
>>>> +    return false;
>>>> +}
>>>> +
>>>> +static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
>>>> +{
>>>> +    unsigned int i;
>>>> +
>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>>> +        if (mali_c55_fmts[i].fourcc == pixelformat)
>>>> +            return &mali_c55_fmts[i];
>>>> +    }
>>>> +
>>>> +    /*
>>>> +     * If we find no matching pixelformat, we'll just default to the first
>>>> +     * one for now.
>>>> +     */
>>>> +
>>>> +    return &mali_c55_fmts[0];
>>>> +}
>>>> +
>>>> +static const char * const capture_device_names[] = {
>>>> +    "mali-c55 fr",
>>>> +    "mali-c55 ds",
>>>> +    "mali-c55 3a stats",
>>>> +    "mali-c55 params",
>> The last two entries are not used AFAICT, neither here, nor in
>> subsequent patches.
>>
>>>> +};
>>>> +
>>>> +static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
>>>> +{
>>>> +    if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
>>>> +        return capture_device_names[0];
>>>> +
>>>> +    if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
>>>> +        return capture_device_names[1];
>>>> +
>>>> +    return "params/stat not supported yet";
>>>> +}
>> Use cap_dev->vdev.name instead of mali_c55_cap_dev_to_name(cap_dev) and
>> drop this function.
>>
>>>> +
>>>> +static int mali_c55_link_validate(struct media_link *link)
>>>> +{
>>>> +    struct video_device *vdev =
>>>> + media_entity_to_video_device(link->sink->entity);
>>>> +    struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
>>>> +    struct v4l2_subdev *sd =
>>>> + media_entity_to_v4l2_subdev(link->source->entity);
>>>> +    const struct v4l2_pix_format_mplane *pix_mp;
>>>> +    const struct mali_c55_fmt *cap_fmt;
>>>> +    struct v4l2_subdev_format sd_fmt = {
>>>> +        .which = V4L2_SUBDEV_FORMAT_ACTIVE,
>>>> +        .pad = link->source->index,
>>>> +    };
>>>> +    int ret;
>>>> +
>>>> +    ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
>>>> +    if (ret)
>>>> +        return ret;
>>>> +
>>>> +    pix_mp = &cap_dev->mode.pix_mp;
>>>> +    cap_fmt = cap_dev->mode.capture_fmt;
>>>> +
>>>> +    if (sd_fmt.format.width != pix_mp->width ||
>>>> +        sd_fmt.format.height != pix_mp->height) {
>>>> +        dev_dbg(cap_dev->mali_c55->dev,
>>>> +            "link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
>>>> +            link->source->entity->name, link->source->index,
>>>> +            link->sink->entity->name, link->sink->index,
>>>> +            sd_fmt.format.width, sd_fmt.format.height,
>>>> +            pix_mp->width, pix_mp->height);
>>>> +        return -EPIPE;
>>>> +    }
>>>> +
>>>> +    if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
>>>> +        dev_dbg(cap_dev->mali_c55->dev,
>>>> +            "link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format 
>>>> %p4cc\n",
>>>> +            link->source->entity->name, link->source->index,
>>>> +            link->sink->entity->name, link->sink->index,
>>>> +            sd_fmt.format.code, &pix_mp->pixelformat);
>>>> +        return -EPIPE;
>>>> +    }
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static const struct media_entity_operations mali_c55_media_ops = {
>>>> +    .link_validate = mali_c55_link_validate,
>>>> +};
>>>> +
>>>> +static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
>>>> +                    unsigned int *num_planes, unsigned int sizes[],
>>>> +                    struct device *alloc_devs[])
>>>> +{
>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>>> +    unsigned int i;
>>>> +
>>>> +    if (*num_planes) {
>>>> +        if (*num_planes != cap_dev->mode.pix_mp.num_planes)
>>>> +            return -EINVAL;
>>>> +
>>>> +        for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>>> +            if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
>>>> +                return -EINVAL;
>>>> +    } else {
>>>> +        *num_planes = cap_dev->mode.pix_mp.num_planes;
>>>> +        for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>>> +            sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
>>>> +    }
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static void mali_c55_buf_queue(struct vb2_buffer *vb)
>>>> +{
>>>> +    struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
>>>> +    struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>>>> +    struct mali_c55_buffer *buf = container_of(vbuf,
>>>> +                           struct mali_c55_buffer, vb);
>>>> +    unsigned int i;
>>>> +
>>>> +    buf->plane_done[MALI_C55_PLANE_Y] = false;
>>>> +
>>>> +    /*
>>>> +     * If we're in a single-plane format we flag the other plane as done
>>>> +     * already so it's dequeued appropriately later
>>>> +     */
>>>> +    buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
>>>> +
>>>> +    for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
>>>> +        unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
>>>> +
>>>> +        vb2_set_plane_payload(vb, i, size);
>>>> +    }
>>>> +
>>>> +    spin_lock(&cap_dev->buffers.lock);
>>>> +    list_add_tail(&buf->queue, &cap_dev->buffers.queue);
>>>> +    spin_unlock(&cap_dev->buffers.lock);
>>>> +}
>>>> +
>>>> +static int mali_c55_buf_init(struct vb2_buffer *vb)
>>>> +{
>>>> +    struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
>>>> +    struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>>>> +    struct mali_c55_buffer *buf = container_of(vbuf,
>>>> +                           struct mali_c55_buffer, vb);
>>>> +    unsigned int i;
>>>> +
>>>> +    for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>>> +        buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>> +
>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
>>>> +
>>>> +    cap_dev->buffers.curr = cap_dev->buffers.next;
>>>> +    cap_dev->buffers.next = NULL;
>>>> +
>>>> +    if (!list_empty(&cap_dev->buffers.queue)) {
>>>> +        struct v4l2_pix_format_mplane *pix_mp;
>>>> +        const struct v4l2_format_info *info;
>>>> +        u32 *addrs;
>>>> +
>>>> +        pix_mp = &cap_dev->mode.pix_mp;
>>>> +        info = v4l2_format_info(pix_mp->pixelformat);
>>>> +
>>>> +        mali_c55_update_bits(mali_c55,
>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
>>>> +        if (cap_dev->mode.capture_fmt->registers.uv_plane)
>>>> +            mali_c55_update_bits(mali_c55,
>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
>>>> +
>>>> +        cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
>>>> +                             struct mali_c55_buffer,
>>>> +                             queue);
>>>> +        list_del(&cap_dev->buffers.next->queue);
>>>> +
>>>> +        addrs = cap_dev->buffers.next->addrs;
>>>> +        mali_c55_write(mali_c55,
>>>> + MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
>>>> +            addrs[MALI_C55_PLANE_Y]);
>>>> +        mali_c55_write(mali_c55,
>>>> + MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
>>>> +            addrs[MALI_C55_PLANE_UV]);
>>>> +        mali_c55_write(mali_c55,
>>>> + MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
>>>> +            pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
>>>> +        mali_c55_write(mali_c55,
>>>> + MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
>>>> +            pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
>>>> +            / info->hdiv);
>>>> +    } else {
>>>> +        /*
>>>> +         * If we underflow then we can tell the ISP that we don't want
>>>> +         * to write out the next frame.
>>>> +         */
>>>> +        mali_c55_update_bits(mali_c55,
>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>> +        mali_c55_update_bits(mali_c55,
>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>> +    }
>>>> +}
>>>> +
>>>> +static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
>>>> +                   unsigned int framecount)
>>>> +{
>>>> +    curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>>>> +    curr_buf->vb.field = V4L2_FIELD_NONE;
>> The could be set already when the buffer is queued.
>>
>>>> +    curr_buf->vb.sequence = framecount;
>>>> +    vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
>>>> +}
>>>> +
>>>> +/**
>>>> + * mali_c55_set_plane_done - mark the plane as written and process the buffer if
>>>> + *                 both planes are finished.
>>>> + * @cap_dev:  pointer to the fr or ds pipe output
>>>> + * @plane:    the plane to mark as completed
>>>> + *
>>>> + * The Mali C55 ISP has muliplanar outputs for some formats that come with two
>>>> + * separate "buffer write completed" interrupts - we need to flag each plane's
>>>> + * completion and check whether both planes are done - if so, complete the buf
>>>> + * in vb2.
>>>> + */
>>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>>>> +                 enum mali_c55_planes plane)
>>>> +{
>>>> +    struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
>>>> +    struct mali_c55_buffer *curr_buf;
>>>> +
>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
>>>> +    curr_buf = cap_dev->buffers.curr;
>>>> +
>>>> +    /*
>>>> +     * This _should_ never happen. If no buffer was available from vb2 then
>>>> +     * we tell the ISP not to bother writing the next frame, which means the
>>>> +     * interrupts that call this function should never trigger. If it does
>>>> +     * happen then one of our assumptions is horribly wrong - complain
>>>> +     * loudly and do nothing.
>>>> +     */
>>>> +    if (!curr_buf) {
>>>> +        dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
>>>> +            mali_c55_cap_dev_to_name(cap_dev), __func__);
>>>> +        return;
>>>> +    }
>>>> +
>>>> +    /* If the other plane is also done... */
>>>> +    if (curr_buf->plane_done[~plane & 1]) {
>>>> +        mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
>>>> +        cap_dev->buffers.curr = NULL;
>>>> +        isp->frame_sequence++;
>>>> +    } else {
>>>> +        curr_buf->plane_done[plane] = true;
>>>> +    }
>>>> +}
>>>> +
>>>> +static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>> +
>>>> +    mali_c55_update_bits(mali_c55,
>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>> +                 MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>> +    mali_c55_update_bits(mali_c55,
>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>> +                 MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>> +}
>>>> +
>>>> +static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>> +
>>>> +    /*
>>>> +     * The Mali ISP can hold up to 5 buffer addresses and simply cycle
>>>> +     * through them, but it's not clear to me that the vb2 queue _guarantees_
>>>> +     * it will queue buffers to the driver in a fixed order, and ensuring
>>>> +     * we call vb2_buffer_done() for the right buffer seems to me to add
>>>> +     * pointless complexity given in multi-context mode we'd need to
>>>> +     * re-write those registers every frame anyway...so we tell the ISP to
>>>> +     * use a single register and update it for each frame.
>>>> +     */
>> A single register sounds prone to error conditions. Is it at least
>> shadowed in the hardware, or do you have to make sure you reprogram it
>> during the vertical blanking only ?
>
>
> It would have to be reprogrammed during the vertical blanking if we were running in a 
> configuration with a single config space, otherwise you have the time it takes to process a frame 
> plus vertical blanking. As I say, it'll have to work like this in multi-context mode anyway.
>
>
> If we want to use the cycling...is it guaranteed that vb2 buffers will always be queued in order?
>
>>
>> I'll mostly skip buffer handling in this review, I need to first
>> understand how the hardware operates to make an informed opinion.
>>
>>>> +    mali_c55_update_bits(mali_c55,
>>>> + MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
>>>> +            MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
>>>> +    mali_c55_update_bits(mali_c55,
>>>> + MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
>>>> +            MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
>>>> +
>>>> +    /*
>>>> +     * We only queue a buffer in the streamon path if this is the first of
>>>> +     * the capture devices to start streaming. If the ISP is already running
>>>> +     * then we rely on the ISP_START interrupt to queue the first buffer for
>>>> +     * this capture device.
>>>> +     */
>>>> +    if (mali_c55->pipe.start_count == 1)
>>>> +        mali_c55_set_next_buffer(cap_dev);
>> I think we'll have to revisit buffer handling to make sure it's 100%
>> race-free.
>>
>>>> +}
>>>> +
>>>> +static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
>>>> +                        enum vb2_buffer_state state)
>>>> +{
>>>> +    struct mali_c55_buffer *buf, *tmp;
>>>> +
>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
>>>> +
>>>> +    if (cap_dev->buffers.curr) {
>>>> + vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
>>>> +                state);
>>>> +        cap_dev->buffers.curr = NULL;
>>>> +    }
>>>> +
>>>> +    if (cap_dev->buffers.next) {
>>>> + vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
>>>> +                state);
>>>> +        cap_dev->buffers.next = NULL;
>>>> +    }
>>>> +
>>>> +    list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
>>>> +        list_del(&buf->queue);
>>>> +        vb2_buffer_done(&buf->vb.vb2_buf, state);
>>>> +    }
>>>> +}
>>>> +
>>>> +static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
>>>> +{
>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>> +    struct mali_c55_resizer *rzr = cap_dev->rzr;
>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
>>>> +    int ret;
>>>> +
>>>> +    guard(mutex)(&isp->lock);
>> What's the reason for using the isp lock here and in
>> mali_c55_vb2_stop_streaming() ? If you need a lock that covers all video
>> nodes in order to synchronize start/stop, you may want to use the
>> graph_mutex of the media device instead.
>
>
> It's because I wanted to make sure that the ISP was in a known started/stopped state before 
> possibly trying to start/stop it, which can be done from either of the two capture devices. This 
> would go away if we were synchronising with the links anyway.
>
>>
>>>> +
>>>> +    ret = pm_runtime_resume_and_get(mali_c55->dev);
>>>> +    if (ret)
>>>> +        return ret;
>>>> +
>>>> +    ret = video_device_pipeline_start(&cap_dev->vdev,
>>>> +                      &cap_dev->mali_c55->pipe);
>>>> +    if (ret) {
>>>> +        dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
>>>> +            mali_c55_cap_dev_to_name(cap_dev));
>> Drop the message or make it dev_dbg() as it can be triggered by
>> userspace.
>>
>>>> +        goto err_pm_put;
>>>> +    }
>>>> +
>>>> +    mali_c55_cap_dev_stream_enable(cap_dev);
>>>> +    mali_c55_rzr_start_stream(rzr);
>>>> +
>>>> +    /*
>>>> +     * We only start the ISP if we're the only capture device that's
>>>> +     * streaming. Otherwise, it'll already be active.
>>>> +     */
>> I still think we should use link setup to indicate which video devices
>> userspace plans to use, and then only start when they're all started.
>> That includes stats and parameters buffers. We can continue this
>> discussion in the context of the previous version of the patch series,
>> or here, up to you.
>
> Let's just continue here. I think I called it "clunky" before; from my perspective it's an 
> unnecessary extra step - we can already signal to the driver that we don't want to use the video 
> devices by not queuing buffers to them or starting the stream on them and although I understand 
> that that means that one of the two image data capture devices will receive data before the other, 
> I don't understand why that's considered to be a problem. Possibly that last part is the stickler; 
> can you explain a bit why it's an issue for one capture queue to start earlier than the other?
>
>
>>
>>>> +    if (mali_c55->pipe.start_count == 1) {
>>>> +        ret = mali_c55_isp_start_stream(isp);
>>>> +        if (ret)
>>>> +            goto err_disable_cap_dev;
>>>> +    }
>>>> +
>>>> +    return 0;
>>>> +
>>>> +err_disable_cap_dev:
>>>> +    mali_c55_cap_dev_stream_disable(cap_dev);
>>>> +    video_device_pipeline_stop(&cap_dev->vdev);
>>>> +err_pm_put:
>>>> +    pm_runtime_put(mali_c55->dev);
>>>> +    mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
>>>> +{
>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>> +    struct mali_c55_resizer *rzr = cap_dev->rzr;
>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
>>>> +
>>>> +    guard(mutex)(&isp->lock);
>>>> +
>>>> +    /*
>>>> +     * If one of the other capture nodes is streaming, we shouldn't
>>>> +     * disable the ISP here.
>>>> +     */
>>>> +    if (mali_c55->pipe.start_count == 1)
>>>> +        mali_c55_isp_stop_stream(&mali_c55->isp);
>>>> +
>>>> +    mali_c55_rzr_stop_stream(rzr);
>>>> +    mali_c55_cap_dev_stream_disable(cap_dev);
>>>> +    mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
>>>> +    video_device_pipeline_stop(&cap_dev->vdev);
>>>> +    pm_runtime_put(mali_c55->dev);
>> I think runtime PM autosuspend would be very useful, as it will ensure
>> that stop-reconfigure-start cycles get handled as efficiently as
>> possible without powering the device down. It could be done on top as a
>> separate patch.
> Alright
>>>> +}
>>>> +
>>>> +static const struct vb2_ops mali_c55_vb2_ops = {
>>>> +    .queue_setup        = &mali_c55_vb2_queue_setup,
>>>> +    .buf_queue        = &mali_c55_buf_queue,
>>>> +    .buf_init        = &mali_c55_buf_init,
>>>> +    .wait_prepare        = vb2_ops_wait_prepare,
>>>> +    .wait_finish        = vb2_ops_wait_finish,
>>>> +    .start_streaming    = &mali_c55_vb2_start_streaming,
>>>> +    .stop_streaming        = &mali_c55_vb2_stop_streaming,
>>>> +};
>>>> +
>>>> +static const struct v4l2_file_operations mali_c55_v4l2_fops = {
>>>> +    .owner = THIS_MODULE,
>>>> +    .unlocked_ioctl = video_ioctl2,
>>>> +    .open = v4l2_fh_open,
>>>> +    .release = vb2_fop_release,
>>>> +    .poll = vb2_fop_poll,
>>>> +    .mmap = vb2_fop_mmap,
>>>> +};
>>>> +
>>>> +static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
>>>> +{
>>>> +    const struct mali_c55_fmt *capture_format;
>>>> +    const struct v4l2_format_info *info;
>>>> +    struct v4l2_plane_pix_format *plane;
>>>> +    unsigned int i;
>>>> +
>>>> +    capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
>>>> +    pix_mp->pixelformat = capture_format->fourcc;
>>>> +
>>>> +    pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
>>>> +                  MALI_C55_MAX_WIDTH);
>>>> +    pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
>>>> +                   MALI_C55_MAX_HEIGHT);
>> Ah, these clamps are right :-)
> Hurrah!
>>
>>>> +
>>>> +    pix_mp->field = V4L2_FIELD_NONE;
>>>> +    pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>> +    pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>> +    pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
>>>> +
>>>> +    info = v4l2_format_info(pix_mp->pixelformat);
>> This function may return NULL. That shouldn't be the case as long as it
>> supports all formats that the C55 driver supports, so I suppose it's
>> safe.
>>
>>>> +    pix_mp->num_planes = info->mem_planes;
>>>> +    memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
>>>> +
>>>> +    pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;
>> Does the hardware support configurable line strides ? If so we should
>> support it.
>
>
> You have to set the line stride in the DMA writer registers, which we do using this same 
> value...might userspace have set bytesperline already then or something? Or is there some other 
> place it could be configured?
>
>>
>>>> +    pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
>>>> +                       * pix_mp->height;
>>     pix_mp->plane_fmt[0].sizeimage = pix_mp->plane_fmt[0].bytesperline
>>                        * pix_mp->height;
>>
>>>> +
>>>> +    for (i = 1; i < info->comp_planes; i++) {
>>>> +        plane = &pix_mp->plane_fmt[i];
>>>> +
>>>> +        plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
>>>> +                           info->hdiv);
>>>> +        plane->sizeimage = DIV_ROUND_UP(
>>>> +                    plane->bytesperline * pix_mp->height,
>>>> +                    info->vdiv);
>>>> +    }
>>>> +
>>>> +    if (info->mem_planes == 1) {
>>>> +        for (i = 1; i < info->comp_planes; i++) {
>>>> +            plane = &pix_mp->plane_fmt[i];
>>>> +            pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
>>>> +        }
>>>> +    }
>> I'm wondering, could v4l2_fill_pixfmt_mp() help ? It doesn't support
>> configurable strides though :-S Maybe the helper could be improved, if
>> it's close enough to what you need ?
>
> I'll take a look
>>>> +}
>>>> +
>>>> +static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>> +                       struct v4l2_format *f)
>>>> +{
>>>> +    mali_c55_try_fmt(&f->fmt.pix_mp);
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
>>>> +                struct v4l2_pix_format_mplane *pix_mp)
>>>> +{
>>>> +    const struct mali_c55_fmt *capture_format;
>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>> +    const struct v4l2_format_info *info;
>>>> +
>>>> +    mali_c55_try_fmt(pix_mp);
>>>> +    capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
>>>> +    info = v4l2_format_info(pix_mp->pixelformat);
>>>> +    if (WARN_ON(!info))
>>>> +        return;
>>>> +
>>>> +    mali_c55_write(mali_c55,
>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>> +               capture_format->registers.base_mode);
>>>> +    mali_c55_write(mali_c55,
>>>> + MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
>> Could the register writes be moved to stream start time ?
>>

Sorry missed this one. These are writes to the context's registers buffer, not to the hardware. Does 
it matter that they're not done at stream on time?

>>>> +
>>>> +    if (info->mem_planes > 1) {
>>>> +        mali_c55_write(mali_c55,
>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>> +                   capture_format->registers.base_mode);
>>>> +        mali_c55_update_bits(mali_c55,
>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>> +                MALI_C55_WRITER_SUBMODE_MASK,
>>>> +                capture_format->registers.uv_plane);
>>>> +
>>>> +        mali_c55_write(mali_c55,
>>>> + MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
>>>> +    }
>>>> +
>>>> +    if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
>>>> +        /*
>>>> +         * TODO: Figure out the colour matrix coefficients and calculate
>>>> +         * and write them here.
>>>> +         */
>> Ideally they should also be exposed directly to userspace as ISP
>> parameters. I would probably go as far as saying that they should come
>> directly from userspace, and not derived from the colorspace fields.
>
>
> Yes I think I agree, I'll drop the todo from here.
>
>>
>>>> +
>>>> +        mali_c55_write(mali_c55,
>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>> +                   MALI_C55_CS_CONV_MATRIX_MASK);
>>>> +
>>>> +        if (info->hdiv > 1)
>>>> +            mali_c55_update_bits(mali_c55,
>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>> +                MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
>>>> +        if (info->vdiv > 1)
>>>> +            mali_c55_update_bits(mali_c55,
>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>> +                MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
>>>> +        if (info->hdiv > 1 || info->vdiv > 1)
>>>> +            mali_c55_update_bits(mali_c55,
>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>> +                MALI_C55_CS_CONV_FILTER_MASK, 0x01);
>>>> +    }
>>>> +
>>>> +    cap_dev->mode.pix_mp = *pix_mp;
>>>> +    cap_dev->mode.capture_fmt = capture_format;
>>>> +}
>>>> +
>>>> +static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>> +                     struct v4l2_format *f)
>>>> +{
>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>>> +
>>>> +    if (vb2_is_busy(&cap_dev->queue))
>>>> +        return -EBUSY;
>>>> +
>>>> +    mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>> +                     struct v4l2_format *f)
>>>> +{
>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>>> +
>>>> +    f->fmt.pix_mp = cap_dev->mode.pix_mp;
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>> +                        struct v4l2_fmtdesc *f)
>>>> +{
>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>>> +    unsigned int j = 0;
>>>> +    unsigned int i;
>>>> +
>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>>> +        if (f->mbus_code &&
>>>> + !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
>>>> +                               f->mbus_code))
>> Small indentation mistake.
>>
>>>> +            continue;
>>>> +
>>>> +        /* Downscale pipe can't output RAW formats */
>>>> +        if (mali_c55_fmts[i].is_raw &&
>>>> +            cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
>>>> +            continue;
>>>> +
>>>> +        if (j++ == f->index) {
>>>> +            f->pixelformat = mali_c55_fmts[i].fourcc;
>>>> +            return 0;
>>>> +        }
>>>> +    }
>>>> +
>>>> +    return -EINVAL;
>>>> +}
>>>> +
>>>> +static int mali_c55_querycap(struct file *file, void *fh,
>>>> +                 struct v4l2_capability *cap)
>>>> +{
>>>> +    strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
>>>> +    strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
>>>> +    .vidioc_reqbufs = vb2_ioctl_reqbufs,
>>>> +    .vidioc_querybuf = vb2_ioctl_querybuf,
>>>> +    .vidioc_create_bufs = vb2_ioctl_create_bufs,
>>>> +    .vidioc_qbuf = vb2_ioctl_qbuf,
>>>> +    .vidioc_expbuf = vb2_ioctl_expbuf,
>>>> +    .vidioc_dqbuf = vb2_ioctl_dqbuf,
>>>> +    .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
>>>> +    .vidioc_streamon = vb2_ioctl_streamon,
>>>> +    .vidioc_streamoff = vb2_ioctl_streamoff,
>>>> +    .vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
>>>> +    .vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
>>>> +    .vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
>>>> +    .vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
>>>> +    .vidioc_querycap = mali_c55_querycap,
>>>> +    .vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
>>>> +    .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
>>>> +};
>>>> +
>>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    struct v4l2_pix_format_mplane pix_mp;
>>>> +    struct mali_c55_cap_dev *cap_dev;
>>>> +    struct video_device *vdev;
>>>> +    struct vb2_queue *vb2q;
>>>> +    unsigned int i;
>>>> +    int ret;
>>>> +
>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
>> Moving the inner content to a separate mali_c55_register_capture_dev()
>> function would increase readability I think, and remove usage of gotos.
>> I would probably do the same for unregistration too, for consistency.
>>
>>>> +        cap_dev = &mali_c55->cap_devs[i];
>>>> +        vdev = &cap_dev->vdev;
>>>> +        vb2q = &cap_dev->queue;
>>>> +
>>>> +        /*
>>>> +         * The downscale output pipe is an optional block within the ISP
>>>> +         * so we need to check whether it's actually been fitted or not.
>>>> +         */
>>>> +
>>>> +        if (i == MALI_C55_CAP_DEV_DS &&
>>>> +            !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
>>>> +            continue;
>> Given that there's only two capture devices, and one is optional, when
>> moving the inner code to a separate function you could unroll the loop.
>> Up to you.
>>
>>>> +
>>>> +        cap_dev->mali_c55 = mali_c55;
>>>> +        mutex_init(&cap_dev->lock);
>>>> +        INIT_LIST_HEAD(&cap_dev->buffers.queue);
>>>> +
>>>> +        switch (i) {
>>>> +        case MALI_C55_CAP_DEV_FR:
>>>> +            cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
>>>> +            cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
>>>> +            break;
>>>> +        case MALI_C55_CAP_DEV_DS:
>>>> +            cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
>>>> +            cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
>>>> +            break;
>>>> +        default:
>> That can't happen.
>>
>>>> + mutex_destroy(&cap_dev->lock);
>>>> +            ret = -EINVAL;
>>>> +            goto err_destroy_mutex;
>>>> +        }
>>>> +
>>>> +        cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
>>>> +        ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
>>>> +        if (ret) {
>>>> +            mutex_destroy(&cap_dev->lock);
>>>> +            goto err_destroy_mutex;
>>>> +        }
>>>> +
>>>> +        vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>>>> +        vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
>>>> +        vb2q->drv_priv = cap_dev;
>>>> +        vb2q->mem_ops = &vb2_dma_contig_memops;
>>>> +        vb2q->ops = &mali_c55_vb2_ops;
>>>> +        vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
>>>> +        vb2q->min_queued_buffers = 1;
>>>> +        vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
>>>> +        vb2q->lock = &cap_dev->lock;
>>>> +        vb2q->dev = mali_c55->dev;
>>>> +
>>>> +        ret = vb2_queue_init(vb2q);
>>>> +        if (ret) {
>>>> +            dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
>>>> +                mali_c55_cap_dev_to_name(cap_dev));
>>>> +            goto err_cleanup_media_entity;
>>>> +        }
>>>> +
>>>> +        strscpy(cap_dev->vdev.name, capture_device_names[i],
>>>> +            sizeof(cap_dev->vdev.name));
>>>> +        vdev->release = video_device_release_empty;
>>>> +        vdev->fops = &mali_c55_v4l2_fops;
>>>> +        vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
>>>> +        vdev->lock = &cap_dev->lock;
>>>> +        vdev->v4l2_dev = &mali_c55->v4l2_dev;
>>>> +        vdev->queue = &cap_dev->queue;
>>>> +        vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>>>> +                    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
>>>> +        vdev->entity.ops = &mali_c55_media_ops;
>>>> +        video_set_drvdata(vdev, cap_dev);
>>>> +
>>>> +        memset(&pix_mp, 0, sizeof(pix_mp));
>>>> +        pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
>>>> +        pix_mp.width = MALI_C55_DEFAULT_WIDTH;
>>>> +        pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
>>>> +        mali_c55_set_format(cap_dev, &pix_mp);
>>>> +
>>>> +        ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
>>>> +        if (ret) {
>>>> +            dev_err(mali_c55->dev,
>>>> +                "%s failed to register video device\n",
>>>> +                mali_c55_cap_dev_to_name(cap_dev));
>>>> +            goto err_release_vb2q;
>>>> +        }
>>>> +    }
>>>> +
>>>> +    return 0;
>>>> +
>>>> +err_release_vb2q:
>>>> +    vb2_queue_release(vb2q);
>>>> +err_cleanup_media_entity:
>>>> +    media_entity_cleanup(&cap_dev->vdev.entity);
>>>> +err_destroy_mutex:
>>>> +    mutex_destroy(&cap_dev->lock);
>>>> +    mali_c55_unregister_capture_devs(mali_c55);
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    struct mali_c55_cap_dev *cap_dev;
>>>> +    unsigned int i;
>>>> +
>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
>>>> +        cap_dev = &mali_c55->cap_devs[i];
>>>> +
>>>> +        if (!video_is_registered(&cap_dev->vdev))
>>>> +            continue;
>>>> +
>>>> +        vb2_video_unregister_device(&cap_dev->vdev);
>>>> +        media_entity_cleanup(&cap_dev->vdev.entity);
>>>> +        mutex_destroy(&cap_dev->lock);
>>>> +    }
>>>> +}
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h 
>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>> new file mode 100644
>>>> index 000000000000..2d0c4d152beb
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>> @@ -0,0 +1,266 @@
>>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Common definitions
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#ifndef _MALI_C55_COMMON_H
>>>> +#define _MALI_C55_COMMON_H
>>>> +
>>>> +#include <linux/clk.h>
>>>> +#include <linux/io.h>
>>>> +#include <linux/list.h>
>>>> +#include <linux/mutex.h>
>>>> +#include <linux/scatterlist.h>
>>> I don't think this is needed. You're however missing spinlock.h.
>>>
>>>> +#include <linux/videodev2.h>
>>>> +
>>>> +#include <media/media-device.h>
>>>> +#include <media/v4l2-async.h>
>>>> +#include <media/v4l2-ctrls.h>
>>>> +#include <media/v4l2-dev.h>
>>>> +#include <media/v4l2-device.h>
>>>> +#include <media/v4l2-subdev.h>
>>>> +#include <media/videobuf2-core.h>
>>>> +#include <media/videobuf2-v4l2.h>
>>>> +
>>>> +#define MALI_C55_DRIVER_NAME        "mali-c55"
>>>> +
>>>> +/* min and max values for the image sizes */
>>>> +#define MALI_C55_MIN_WIDTH        640U
>>>> +#define MALI_C55_MIN_HEIGHT        480U
>>>> +#define MALI_C55_MAX_WIDTH        8192U
>>>> +#define MALI_C55_MAX_HEIGHT        8192U
>>>> +#define MALI_C55_DEFAULT_WIDTH        1920U
>>>> +#define MALI_C55_DEFAULT_HEIGHT        1080U
>>>> +
>>>> +#define MALI_C55_DEFAULT_MEDIA_BUS_FMT MEDIA_BUS_FMT_RGB121212_1X36
>>>> +
>>>> +struct mali_c55;
>>>> +struct mali_c55_cap_dev;
>>>> +struct platform_device;
>>> You should also forward-declare
>>>
>>> struct device;
>>> struct dma_chan;
>>> struct resource;
>>>
>>>> +
>>>> +static const char * const mali_c55_clk_names[] = {
>>>> +    "aclk",
>>>> +    "hclk",
>>>> +};
>>> This will end up duplicating the array in each compilation unit, not
>>> great. Move it to mali-c55-core.c. You use it in this file just for its
>>> size, replace that with a macro that defines the size, or allocate
>>> mali_c55.clks dynamically with devm_kcalloc().
>>>
>>>> +
>>>> +enum mali_c55_interrupts {
>>>> +    MALI_C55_IRQ_ISP_START,
>>>> +    MALI_C55_IRQ_ISP_DONE,
>>>> +    MALI_C55_IRQ_MCM_ERROR,
>>>> +    MALI_C55_IRQ_BROKEN_FRAME_ERROR,
>>>> +    MALI_C55_IRQ_MET_AF_DONE,
>>>> +    MALI_C55_IRQ_MET_AEXP_DONE,
>>>> +    MALI_C55_IRQ_MET_AWB_DONE,
>>>> +    MALI_C55_IRQ_AEXP_1024_DONE,
>>>> +    MALI_C55_IRQ_IRIDIX_MET_DONE,
>>>> +    MALI_C55_IRQ_LUT_INIT_DONE,
>>>> +    MALI_C55_IRQ_FR_Y_DONE,
>>>> +    MALI_C55_IRQ_FR_UV_DONE,
>>>> +    MALI_C55_IRQ_DS_Y_DONE,
>>>> +    MALI_C55_IRQ_DS_UV_DONE,
>>>> +    MALI_C55_IRQ_LINEARIZATION_DONE,
>>>> +    MALI_C55_IRQ_RAW_FRONTEND_DONE,
>>>> +    MALI_C55_IRQ_NOISE_REDUCTION_DONE,
>>>> +    MALI_C55_IRQ_IRIDIX_DONE,
>>>> +    MALI_C55_IRQ_BAYER2RGB_DONE,
>>>> +    MALI_C55_IRQ_WATCHDOG_TIMER,
>>>> +    MALI_C55_IRQ_FRAME_COLLISION,
>>>> +    MALI_C55_IRQ_UNUSED,
>>>> +    MALI_C55_IRQ_DMA_ERROR,
>>>> +    MALI_C55_IRQ_INPUT_STOPPED,
>>>> +    MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
>>>> +    MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
>>>> +    MALI_C55_NUM_IRQ_BITS
>>> Those are register bits, I think they belong to mali-c55-registers.h,
>>> and should probably be macros instead of an enum.
>>>
>>>> +};
>>>> +
>>>> +enum mali_c55_isp_pads {
>>>> +    MALI_C55_ISP_PAD_SINK_VIDEO,
>>> As there's a single sink pad, maybe MALI_C55_ISP_PAD_SINK ? Ah, you're
>>> probably preparing for ISP parameters support. It's fine.
>>>
>>>> +    MALI_C55_ISP_PAD_SOURCE,
>>> Then maybe this should be named MALI_C55_ISP_PAD_SOURCE_VIDEO as I
>>> assume there will be a stats source pad.
>>>
>>>> +    MALI_C55_ISP_PAD_SOURCE_BYPASS,
>>>> +    MALI_C55_ISP_NUM_PADS,
>>>> +};
>>>> +
>>>> +struct mali_c55_tpg {
>>>> +    struct mali_c55 *mali_c55;
>>>> +    struct v4l2_subdev sd;
>>>> +    struct media_pad pad;
>>>> +    struct mutex lock;
>>>> +    struct mali_c55_tpg_ctrls {
>>>> +        struct v4l2_ctrl_handler handler;
>>>> +        struct v4l2_ctrl *test_pattern;
>>> Set but never used. You can drop it.
>>>
>>>> +        struct v4l2_ctrl *hblank;
>>> Set and used only once, in the same function. You can make it a local
>>> variable.
>>>
>>>> +        struct v4l2_ctrl *vblank;
>>>> +    } ctrls;
>>>> +};
>>> I wonder if this file should be split, with mali-c55-capture.h,
>>> mali-c55-core.h, mali-c55-isp.h, ... I think it could increase
>>> readability by clearly separating the different elements. Up to you.
>>>
>>>> +
>>>> +struct mali_c55_isp {
>>>> +    struct mali_c55 *mali_c55;
>>>> +    struct v4l2_subdev sd;
>>>> +    struct media_pad pads[MALI_C55_ISP_NUM_PADS];
>>>> +    struct media_pad *remote_src;
>>>> +    struct v4l2_async_notifier notifier;
>>> I'm tempted to move the notifier to mali_c55, as it's related to
>>> components external to the whole ISP, not to the ISP subdev itself.
>>> Could you give it a try, to see if it could be done without any drawback
>>> ?
>>>
>>>> +    struct mutex lock;
>>> Locks require a comment to explain what they protect. Same below where
>>> applicable (for both mutexes and spinlocks).
>>>
>>>> +    unsigned int frame_sequence;
>>>> +};
>>>> +
>>>> +enum mali_c55_resizer_ids {
>>>> +    MALI_C55_RZR_FR,
>>>> +    MALI_C55_RZR_DS,
>>>> +    MALI_C55_NUM_RZRS,
>>> The usual abbreviation for "resizer" in drivers/media/ is "rsz", not
>>> "rzr". I would have said we can leave it as-is as changing it would be a
>>> bit annoying, but I then realized that "rzr" is not just unusual, it's
>>> actually not used at all. Would you mind applying a sed globally ? :-)
>>>
>>>> +};
>>>> +
>>>> +enum mali_c55_rzr_pads {
>>> Same enums/structs use abbreviations, some don't. Consistency would
>>> help.
>>>
>>>> +    MALI_C55_RZR_SINK_PAD,
>>>> +    MALI_C55_RZR_SOURCE_PAD,
>>>> +    MALI_C55_RZR_SINK_BYPASS_PAD,
>>>> +    MALI_C55_RZR_NUM_PADS
>>>> +};
>>>> +
>>>> +struct mali_c55_resizer {
>>>> +    struct mali_c55 *mali_c55;
>>>> +    struct mali_c55_cap_dev *cap_dev;
>>>> +    enum mali_c55_resizer_ids id;
>>>> +    struct v4l2_subdev sd;
>>>> +    struct media_pad pads[MALI_C55_RZR_NUM_PADS];
>>>> +    unsigned int num_routes;
>>>> +    bool streaming;
>>>> +};
>>>> +
>>>> +enum mali_c55_cap_devs {
>>>> +    MALI_C55_CAP_DEV_FR,
>>>> +    MALI_C55_CAP_DEV_DS,
>>>> +    MALI_C55_NUM_CAP_DEVS
>>>> +};
>>>> +
>>>> +struct mali_c55_fmt {
>>> mali_c55_format_info would be a better name I think, as this stores
>>> format information, not formats.
>>>
>>>> +    u32 fourcc;
>>>> +    unsigned int mbus_codes[2];
>>> A comment to explain why we have two media bus codes would be useful.
>>> You can document the whole structure if desired :-)
>>>
>>>> +    bool is_raw;
>>>> +    struct mali_c55_fmt_registers {
>>> Make it an anonymous structure, it's never used anywhere else.
>>>
>>>> +        unsigned int base_mode;
>>>> +        unsigned int uv_plane;
>>> If those are register field values, use u32 instead of unsigned int.
>>>
>>>> +    } registers;
>>> It's funny, we tend to abbreviate different things, I would have used
>>> "regs" here but written "format" in full in the structure name :-)
>>>
>>>> +};
>>>> +
>>>> +enum mali_c55_isp_bayer_order {
>>>> +    MALI_C55_BAYER_ORDER_RGGB,
>>>> +    MALI_C55_BAYER_ORDER_GRBG,
>>>> +    MALI_C55_BAYER_ORDER_GBRG,
>>>> +    MALI_C55_BAYER_ORDER_BGGR
>>> These are registers values too, they belong to mali-c55-registers.h.
>>>
>>>> +};
>>>> +
>>>> +struct mali_c55_isp_fmt {
>>> mali_c55_isp_format_info
>>>
>>>> +    u32 code;
>>>> +    enum v4l2_pixel_encoding encoding;
>>> Here you use v4l2_pixel_encoding, for mali_c55_fmt you use is_raw. Maybe
>>> pick the same option for both structures ?
>>>
>>>> +    enum mali_c55_isp_bayer_order order;
>>>> +};
>>>> +
>>>> +enum mali_c55_planes {
>>>> +    MALI_C55_PLANE_Y,
>>>> +    MALI_C55_PLANE_UV,
>>>> +    MALI_C55_NUM_PLANES
>>>> +};
>>>> +
>>>> +struct mali_c55_buffer {
>>>> +    struct vb2_v4l2_buffer vb;
>>>> +    bool plane_done[MALI_C55_NUM_PLANES];
>>> I think tracking the pending state would simplify the logic in
>>> mali_c55_set_plane_done(), which would become
>>>
>>>     curr_buf->plane_pending[plane] = false;
>>>
>>>     if (!curr_buf->plane_pending[0] && !curr_buf->plane_pending[1]) {
>>>         mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
>>>         cap_dev->buffers.curr = NULL;
>>>         isp->frame_sequence++;
>>>     }
>>>
>>> Or a counter may be even easier (and would consume less memory).
>>>
>>>> +    struct list_head queue;
>>>> +    u32 addrs[MALI_C55_NUM_PLANES];
>>> This stores DMA addresses, use dma_addr_t.
>>>
>>>> +};
>>>> +
>>>> +struct mali_c55_cap_dev {
>>>> +    struct mali_c55 *mali_c55;
>>>> +    struct mali_c55_resizer *rzr;
>>>> +    struct video_device vdev;
>>>> +    struct media_pad pad;
>>>> +    struct vb2_queue queue;
>>>> +    struct mutex lock;
>>>> +    unsigned int reg_offset;
>>> Manual handling of the offset everywhere, with parametric macros for the
>>> resizer register addresses, isn't very nice. Introduce resizer-specific
>>> accessors and capture-specific accessors (mali_c55_rsz_write(), ...)
>>> that take a mali_c55_resizer or mali_c55_cap_dev pointer, and handle the
>>> offset there. The register macros should loose their offset parameter.
>>>
>>> You could also use a single set of accessors that would become
>>> path/output-specific (mali_c55_path_write() ? mali_c55_output_write()
>>> ?), that may make the code easier to read.
>>>
>>> You can also replace reg_offset with a void __iomem * base, which would
>>> avoid the computation at runtime.
>>>
>>>> +
>>>> +    struct mali_c55_mode {
>>> Make the structure anonymous.
>>>
>>>> +        const struct mali_c55_fmt *capture_fmt;
>>>> +        struct v4l2_pix_format_mplane pix_mp;
>>>> +    } mode;
>>> What's a "mode" ? I think I'd name this
>>>
>>>     struct {
>>>         const struct mali_c55_fmt *info;
>>>         struct v4l2_pix_format_mplane format;
>>>     } format;
>>>
>>> Or you could just drop the structure and have
>>>
>>>     const struct mali_c55_fmt *format_info;
>>>     struct v4l2_pix_format_mplane format;
>>>
>>> or something similar.
>>>
>>>> +
>>>> +    struct {
>>>> +        spinlock_t lock;
>>>> +        struct list_head queue;
>>>> +        struct mali_c55_buffer *curr;
>>>> +        struct mali_c55_buffer *next;
>>>> +    } buffers;
>>>> +
>>>> +    bool streaming;
>>>> +};
>>>> +
>>>> +enum mali_c55_config_spaces {
>>>> +    MALI_C55_CONFIG_PING,
>>>> +    MALI_C55_CONFIG_PONG,
>>>> +    MALI_C55_NUM_CONFIG_SPACES
>>> The last enumerator is not used.
>>>
>>>> +};
>>>> +
>>>> +struct mali_c55_ctx {
>>> mali_c55_context ?
>>>
>>>> +    struct mali_c55 *mali_c55;
>>>> +    void *registers;
>>> Please document this structure and explain that this field points to a
>>> copy of the register space in system memory, I was about to write you're
>>> missing __iomem :-)
>>>
>>>> +    phys_addr_t base;
>>>> +    spinlock_t lock;
>>>> +    struct list_head list;
>>>> +};
>>>> +
>>>> +struct mali_c55 {
>>>> +    struct device *dev;
>>>> +    struct resource *res;
>>> You could possibly drop this field by passing the physical address of
>>> the register space from mali_c55_probe() to mali_c55_init_context() as a
>>> function parameter.
>>>
>>>> +    void __iomem *base;
>>>> +    struct dma_chan *channel;
>>>> +    struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
>>>> +
>>>> +    u16 capabilities;
>>>> +    struct media_device media_dev;
>>>> +    struct v4l2_device v4l2_dev;
>>>> +    struct media_pipeline pipe;
>>>> +
>>>> +    struct mali_c55_tpg tpg;
>>>> +    struct mali_c55_isp isp;
>>>> +    struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
>>>> +    struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
>>>> +
>>>> +    struct list_head contexts;
>>>> +    enum mali_c55_config_spaces next_config;
>>>> +};
>>>> +
>>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
>>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
>>>> +          bool force_hardware);
>>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
>>>> +              u32 mask, u32 val);
>>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
>>>> +              enum mali_c55_config_spaces cfg_space);
>>>> +
>>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55);
>>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55);
>>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
>>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
>>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55);
>>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
>>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
>>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
>>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
>>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>>>> +                 enum mali_c55_planes plane);
>>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
>>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
>>>> +
>>>> +bool mali_c55_format_is_raw(unsigned int mbus_code);
>>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
>>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
>>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
>>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
>>>> +
>>>> +const struct mali_c55_isp_fmt *
>>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
>>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
>>>> +#define for_each_mali_isp_fmt(fmt)\
>>> #define for_each_mali_isp_fmt(fmt) \
>>>
>>>> +    for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
>>> Looks like parentheses were on sale :-)
>>>
>>>     for ((fmt) = NULL; (fmt) = mali_c55_isp_fmt_next(fmt); )
>>>
>>> This macro is used in two places only, in the mali-c55-isp.c file where
>>> open-coding the loop without using mali_c55_isp_fmt_next() would be more
>>> efficient, and in mali-c55-resizer.c where a function to return format
>>> i'th would be more efficient. I think you can drop the macro and the
>>> mali_c55_isp_fmt_next() function.
>>>
>>>> +
>>>> +#endif /* _MALI_C55_COMMON_H */
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c 
>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>> new file mode 100644
>>>> index 000000000000..50caf5ee7474
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>> @@ -0,0 +1,767 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Core driver code
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#include <linux/bitops.h>
>>>> +#include <linux/cleanup.h>
>>>> +#include <linux/clk.h>
>>>> +#include <linux/delay.h>
>>>> +#include <linux/device.h>
>>>> +#include <linux/dmaengine.h>
>>>> +#include <linux/dma-mapping.h>
>>>> +#include <linux/interrupt.h>
>>>> +#include <linux/iopoll.h>
>>>> +#include <linux/ioport.h>
>>>> +#include <linux/mod_devicetable.h>
>>>> +#include <linux/of.h>
>>>> +#include <linux/of_reserved_mem.h>
>>>> +#include <linux/platform_device.h>
>>>> +#include <linux/pm_runtime.h>
>>>> +#include <linux/scatterlist.h>
>>> I don't think this is needed.
>>>
>>> Missing slab.h.
>>>
>>>> +#include <linux/string.h>
>>>> +
>>>> +#include <media/media-entity.h>
>>>> +#include <media/v4l2-device.h>
>>>> +#include <media/videobuf2-dma-contig.h>
>>>> +
>>>> +#include "mali-c55-common.h"
>>>> +#include "mali-c55-registers.h"
>>>> +
>>>> +static const char * const mali_c55_interrupt_names[] = {
>>>> +    [MALI_C55_IRQ_ISP_START] = "ISP start",
>>>> +    [MALI_C55_IRQ_ISP_DONE] = "ISP done",
>>>> +    [MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
>>>> +    [MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
>>>> +    [MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
>>>> +    [MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
>>>> +    [MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
>>>> +    [MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
>>>> +    [MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
>>>> +    [MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
>>>> +    [MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
>>>> +    [MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
>>>> +    [MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
>>>> +    [MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
>>>> +    [MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
>>>> +    [MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
>>>> +    [MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
>>>> +    [MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
>>>> +    [MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
>>>> +    [MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
>>>> +    [MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
>>>> +    [MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
>>>> +    [MALI_C55_IRQ_DMA_ERROR] = "DMA error",
>>>> +    [MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
>>>> +    [MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
>>>> +    [MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
>>>> +};
>>>> +
>>>> +static unsigned int config_space_addrs[] = {
>>> const
>>>
>>>> +    [MALI_C55_CONFIG_PING] = 0x0AB6C,
>>>> +    [MALI_C55_CONFIG_PONG] = 0x22B2C,
>>> Lowercase hex constants.
>>>
>>> Don't the values belong to mali-c55-registers.h ?
>>>
>>>> +};
>>>> +
>>>> +/* System IO
>>> /*
>>>   * System IO
>>>
>>>> + *
>>>> + * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
>>>> + * and 'pong'), with the  expectation that the 'active' space will be left
>>> s/the  /the /
>>>
>>>> + * untouched whilst a frame is being processed and the 'inactive' space
>>>> + * configured ready to be passed during the blanking period before the next
>>> s/to be passed/to be switched to/ ?
>>>
>>>> + * frame processing starts. These spaces should ideally be set via DMA transfer
>>>> + * from a buffer rather than through individual register set operations. There
>>>> + * is also a shared global register space which should be set normally. Of
>>>> + * course, the ISP might be included in a system which lacks a suitable DMA
>>>> + * engine, and the second configuration space might not be fitted at all, which
>>>> + * means we need to support four scenarios:
>>>> + *
>>>> + * 1. Multi config space, with DMA engine.
>>>> + * 2. Multi config space, no DMA engine.
>>>> + * 3. Single config space, with DMA engine.
>>>> + * 4. Single config space, no DMA engine.
>>>> + *
>>>> + * The first case is very easy, but the rest present annoying problems. The best
>>>> + * way to solve them seems to be simply to replicate the concept of DMAing over
>>>> + * the configuration buffer even if there's no DMA engine on the board, for
>>>> + * which we rely on memcpy. To facilitate this any read/write call that is made
>>>> + * to an address within those config spaces should infact be directed to a
>>>> + * buffer that was allocated to hold them rather than the IO memory itself. The
>>>> + * actual copy of that buffer to IO mem will happen on interrupt.
>>>> + */
>>>> +
>>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
>>>> +{
>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>> +
>>>> +    if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
>>>> +        spin_lock(&ctx->lock);
>>>> +        addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
>>>> +        ((u32 *)ctx->registers)[addr] = val;
>>>> +        spin_unlock(&ctx->lock);
>>>> +
>>>> +        return;
>>>> +    }
>>> Ouch. This is likely the second comment you really won't like (after the
>>> comment regarding mali_c55_update_bits() at the very top). I apologize
>>> in advance.
>>>
>>> I really don't like this. Directing writes either to hardware registers
>>> or to the shadow registers in the context makes the callers of the
>>> read/write accessors very hard to read. The probe code, for instance,
>>> mixes writes to hardware registers and writes to the context shadow
>>> registers to initialize the value of some of the shadow registers.
>>>
>>> I'd like to split the read/write accessors into functions that access
>>> the hardware registers (that's easy) and functions that access the
>>> shadow registers. I think the latter should receive a mali_c55_ctx
>>> pointer instead of a mali_c55 pointer to prepare for multi-context
>>> support.
>>>
>>> You can add WARN_ON() guards to the two sets of functions, to ensure
>>> that no register from the "other" space gets passed to the wrong
>>> function by mistake.
>>>
>>>> +
>>>> +    writel(val, mali_c55->base + addr);
>>>> +}
>>>> +
>>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
>>>> +          bool force_hardware)
>>> force_hardware is never set to true.
>>>
>>>> +{
>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>> +    u32 val;
>>>> +
>>>> +    if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
>>>> +        spin_lock(&ctx->lock);
>>>> +        addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
>>>> +        val = ((u32 *)ctx->registers)[addr];
>>>> +        spin_unlock(&ctx->lock);
>>>> +
>>>> +        return val;
>>>> +    }
>>>> +
>>>> +    return readl(mali_c55->base + addr);
>>>> +}
>>>> +
>>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
>>>> +              u32 mask, u32 val)
>>>> +{
>>>> +    u32 orig, tmp;
>>>> +
>>>> +    orig = mali_c55_read(mali_c55, addr, false);
>>>> +
>>>> +    tmp = orig & ~mask;
>>>> +    tmp |= (val << (ffs(mask) - 1)) & mask;
>>>> +
>>>> +    if (tmp != orig)
>>>> +        mali_c55_write(mali_c55, addr, tmp);
>>>> +}
>>>> +
>>>> +static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
>>>> +                 dma_addr_t dst, enum dma_data_direction dir)
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>> +    struct dma_async_tx_descriptor *tx;
>>>> +    enum dma_status status;
>>>> +    dma_cookie_t cookie;
>>>> +
>>>> +    tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
>>>> +                       MALI_C55_CONFIG_SPACE_SIZE, 0);
>>>> +    if (!tx) {
>>>> +        dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
>>>> +        return -EIO;
>>>> +    }
>>>> +
>>>> +    cookie = dmaengine_submit(tx);
>>>> +    if (dma_submit_error(cookie)) {
>>>> +        dev_err(mali_c55->dev, "error submitting dma transfer\n");
>>>> +        return -EIO;
>>>> +    }
>>>> +
>>>> +    status = dma_sync_wait(mali_c55->channel, cookie);
>>> I've just realized this performs a busy-wait :-S See the comment in the
>>> probe function about the threaded IRQ handler. I think we'll need to
>>> rework all this. It could be done on top though.
>>>
>>>> +    if (status != DMA_COMPLETE) {
>>>> +        dev_err(mali_c55->dev, "dma transfer failed\n");
>>>> +        return -EIO;
>>>> +    }
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
>>>> +                 enum mali_c55_config_spaces cfg_space)
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>> +    struct device *dma_dev = mali_c55->channel->device->dev;
>>>> +    dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
>>>> +    dma_addr_t dst;
>>>> +    int ret;
>>>> +
>>>> +    guard(spinlock)(&ctx->lock);
>>>> +
>>>> +    dst = dma_map_single(dma_dev, ctx->registers,
>>>> +                 MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
>>>> +    if (dma_mapping_error(dma_dev, dst)) {
>>>> +        dev_err(mali_c55->dev, "failed to map DMA addr\n");
>>>> +        return -EIO;
>>>> +    }
>>>> +
>>>> +    ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
>>>> +    dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
>>>> +             DMA_FROM_DEVICE);
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
>>>> +               enum mali_c55_config_spaces cfg_space)
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>> +    struct device *dma_dev = mali_c55->channel->device->dev;
>>>> +    dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
>>>> +    dma_addr_t src;
>>>> +    int ret;
>>>> +
>>>> +    guard(spinlock)(&ctx->lock);
>>> The code below can take a large amount of time, holding a spinlock will
>>> disable interrupts on the local CPU, that's not good :-(
>>>
>>>> +
>>>> +    src = dma_map_single(dma_dev, ctx->registers,
>>>> +                 MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
>>>> +    if (dma_mapping_error(dma_dev, src)) {
>>>> +        dev_err(mali_c55->dev, "failed to map DMA addr\n");
>>>> +        return -EIO;
>>>> +    }
>>>> +
>>>> +    ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
>>>> +    dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
>>>> +             DMA_TO_DEVICE);
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +static int mali_c55_config_read(struct mali_c55_ctx *ctx,
>>>> +                enum mali_c55_config_spaces cfg_space)
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>> +
>>>> +    if (mali_c55->channel) {
>>>> +        return mali_c55_dma_read(ctx, cfg_space);
>>> As this function is used at probe time only, to initialize the context,
>>> I think DMA is overkill.
>>>
>>>> +    } else {
>>>> +        memcpy_fromio(ctx->registers,
>>>> +                  mali_c55->base + config_space_addrs[cfg_space],
>>>> +                  MALI_C55_CONFIG_SPACE_SIZE);
>>>> +        return 0;
>>>> +    }
>>>> +}
>>>> +
>>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
>>>> +              enum mali_c55_config_spaces cfg_space)
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>> +
>>>> +    if (mali_c55->channel) {
>>>> +        return mali_c55_dma_write(ctx, cfg_space);
>>>> +    } else {
>>>> +        memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
>>>> +                ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
>>>> +        return 0;
>>>> +    }
>>> Could you measure the time it typically takes to write the registers
>>> using DMA compared to using memcpy_toio() ?
>>>
>>>> +}
>>>> +
>>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);
>>> I think it's too early to tell how multi-context support will look like.
>>> I'm fine keeping mali_c55_get_active_context() as changing that would be
>>> very intrusive (even if I think it will need to be changed), but the
>>> list of contexts is neither the mechanism we'll use, nor something we
>>> need now. Drop the list, embed the context in struct mali_c55, and
>>> return the pointer to that single context from this function.
>>>
>>>> +}
>>>> +
>>>> +static void mali_c55_remove_links(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    unsigned int i;
>>>> +
>>>> + media_entity_remove_links(&mali_c55->tpg.sd.entity);
>>>> + media_entity_remove_links(&mali_c55->isp.sd.entity);
>>>> +
>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; i++)
>>>> + media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
>>>> +
>>>> +    for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
>>>> + media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
>>>> +}
>>>> +
>>>> +static int mali_c55_create_links(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    int ret;
>>>> +
>>>> +    /* Test pattern generator to ISP */
>>>> +    ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
>>>> +                    &mali_c55->isp.sd.entity,
>>>> +                    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
>>>> +    if (ret) {
>>>> +        dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
>>>> +        goto err_remove_links;
>>>> +    }
>>>> +
>>>> +    /* Full resolution resizer pipe. */
>>>> +    ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>>> +            MALI_C55_ISP_PAD_SOURCE,
>>>> + &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
>>>> +            MALI_C55_RZR_SINK_PAD,
>>>> +            MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>> +    if (ret) {
>>>> +        dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
>>>> +        goto err_remove_links;
>>>> +    }
>>>> +
>>>> +    /* Full resolution bypass. */
>>>> +    ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>>> +                    MALI_C55_ISP_PAD_SOURCE_BYPASS,
>>>> + &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
>>>> +                    MALI_C55_RZR_SINK_BYPASS_PAD,
>>>> +                    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>> +    if (ret) {
>>>> +        dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
>>>> +        goto err_remove_links;
>>>> +    }
>>>> +
>>>> +    /* Resizer pipe to video capture nodes. */
>>>> +    ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
>>>> +            MALI_C55_RZR_SOURCE_PAD,
>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
>>>> +            0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>> +    if (ret) {
>>>> +        dev_err(mali_c55->dev,
>>>> +            "failed to link FR resizer and video device\n");
>>>> +        goto err_remove_links;
>>>> +    }
>>>> +
>>>> +    /* The downscale pipe is an optional hardware block */
>>>> +    if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
>>>> +        ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>>> +            MALI_C55_ISP_PAD_SOURCE,
>>>> + &mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
>>>> +            MALI_C55_RZR_SINK_PAD,
>>>> +            MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>> +        if (ret) {
>>>> +            dev_err(mali_c55->dev,
>>>> +                "failed to link ISP and DS resizer\n");
>>>> +            goto err_remove_links;
>>>> +        }
>>>> +
>>>> +        ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
>>>> +            MALI_C55_RZR_SOURCE_PAD,
>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
>>>> +            0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>> +        if (ret) {
>>>> +            dev_err(mali_c55->dev,
>>>> +                "failed to link DS resizer and video device\n");
>>>> +            goto err_remove_links;
>>>> +        }
>>>> +    }
>>>> +
>>>> +    return 0;
>>>> +
>>>> +err_remove_links:
>>>> +    mali_c55_remove_links(mali_c55);
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    mali_c55_unregister_tpg(mali_c55);
>>>> +    mali_c55_unregister_isp(mali_c55);
>>>> +    mali_c55_unregister_resizers(mali_c55);
>>>> +    mali_c55_unregister_capture_devs(mali_c55);
>>>> +}
>>>> +
>>>> +static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    int ret;
>>>> +
>>>> +    ret = mali_c55_register_tpg(mali_c55);
>>>> +    if (ret)
>>>> +        return ret;
>>>> +
>>>> +    ret = mali_c55_register_isp(mali_c55);
>>>> +    if (ret)
>>>> +        goto err_unregister_entities;
>>>> +
>>>> +    ret = mali_c55_register_resizers(mali_c55);
>>>> +    if (ret)
>>>> +        goto err_unregister_entities;
>>>> +
>>>> +    ret = mali_c55_register_capture_devs(mali_c55);
>>>> +    if (ret)
>>>> +        goto err_unregister_entities;
>>>> +
>>>> +    ret = mali_c55_create_links(mali_c55);
>>>> +    if (ret)
>>>> +        goto err_unregister_entities;
>>>> +
>>>> +    return 0;
>>>> +
>>>> +err_unregister_entities:
>>>> +    mali_c55_unregister_entities(mali_c55);
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    u32 product, version, revision, capabilities;
>>>> +
>>>> +    product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
>>>> +    version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
>>>> +    revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
>>>> +
>>>> +    dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
>>>> +         product, version, revision);
>>>> +
>>>> +    capabilities = mali_c55_read(mali_c55,
>>>> +                     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
>>>> +                     false);
>>>> +    mali_c55->capabilities = (capabilities & 0xffff);
>>>> +
>>>> +    /* TODO: Might as well start some debugfs */
>>> If it's just to expose the version and capabilities, I think that's
>>> overkill. It's not needed for debug purpose (you can get it from the
>>> kernel log already). debugfs isn't meant to be accessible in production,
>>> so an application that would need access to the information wouldn't be
>>> able to use it.
>>>
>>>> +    dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);
>>> Combine the two messages into one.
>>>
>>>> +    return version;
>>>> +}
>>>> +
>>>> +static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>> +    u32 curr_config, next_config;
>>>> +
>>>> +    curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
>>>> +    curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
>>>> +              >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
>>>> +    next_config = curr_config ^ 1;
>>>> +
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>> +                 MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
>>>> +    mali_c55_config_write(ctx, next_config ?
>>>> +                  MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
>>>> +}
>>>> +
>>>> +static irqreturn_t mali_c55_isr(int irq, void *context)
>>>> +{
>>>> +    struct device *dev = context;
>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>>> +    u32 interrupt_status;
>>>> +    unsigned int i, j;
>>>> +
>>>> +    interrupt_status = mali_c55_read(mali_c55,
>>>> +                     MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
>>>> +                     false);
>>>> +    if (!interrupt_status)
>>>> +        return IRQ_NONE;
>>>> +
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
>>>> +               interrupt_status);
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
>>>> +
>>>> +    for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
>>>> +        if (!(interrupt_status & (1 << i)))
>>>> +            continue;
>>>> +
>>>> +        switch (i) {
>>>> +        case MALI_C55_IRQ_ISP_START:
>>>> +            mali_c55_isp_queue_event_sof(mali_c55);
>>>> +
>>>> +            for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
>>>> + mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
>>>> +
>>>> +            mali_c55_swap_next_config(mali_c55);
>>>> +
>>>> +            break;
>>>> +        case MALI_C55_IRQ_ISP_DONE:
>>>> +            /*
>>>> +             * TODO: Where the ISP has no Pong config fitted, we'd
>>>> +             * have to do the mali_c55_swap_next_config() call here.
>>>> +             */
>>>> +            break;
>>>> +        case MALI_C55_IRQ_FR_Y_DONE:
>>>> +            mali_c55_set_plane_done(
>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
>>>> +                MALI_C55_PLANE_Y);
>>>> +            break;
>>>> +        case MALI_C55_IRQ_FR_UV_DONE:
>>>> +            mali_c55_set_plane_done(
>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
>>>> +                MALI_C55_PLANE_UV);
>>>> +            break;
>>>> +        case MALI_C55_IRQ_DS_Y_DONE:
>>>> +            mali_c55_set_plane_done(
>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
>>>> +                MALI_C55_PLANE_Y);
>>>> +            break;
>>>> +        case MALI_C55_IRQ_DS_UV_DONE:
>>>> +            mali_c55_set_plane_done(
>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
>>>> +                MALI_C55_PLANE_UV);
>>>> +            break;
>>>> +        default:
>>>> +            /*
>>>> +             * Only the above interrupts are currently unmasked. If
>>>> +             * we receive anything else here then something weird
>>>> +             * has gone on.
>>>> +             */
>>>> +            dev_err(dev, "masked interrupt %s triggered\n",
>>>> +                mali_c55_interrupt_names[i]);
>>>> +        }
>>>> +    }
>>>> +
>>>> +    return IRQ_HANDLED;
>>>> +}
>>>> +
>>>> +static int mali_c55_init_context(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    struct mali_c55_ctx *ctx;
>>>> +    int ret;
>>>> +
>>>> +    ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
>>>> +    if (!ctx) {
>>>> +        dev_err(mali_c55->dev, "failed to allocate new context\n");
>>> No need for an error message when memory allocation fails.
>>>
>>>> +        return -ENOMEM;
>>>> +    }
>>>> +
>>>> +    ctx->base = mali_c55->res->start;
>>>> +    ctx->mali_c55 = mali_c55;
>>>> +
>>>> +    ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
>>>> +                 GFP_KERNEL | GFP_DMA);
>>>> +    if (!ctx->registers) {
>>>> +        ret = -ENOMEM;
>>>> +        goto err_free_ctx;
>>>> +    }
>>>> +
>>>> +    /*
>>>> +     * The allocated memory is empty, we need to load the default
>>>> +     * register settings. We just read Ping; it's identical to Pong.
>>>> +     */
>>>> +    ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
>>>> +    if (ret)
>>>> +        goto err_free_registers;
>>>> +
>>>> +    list_add_tail(&ctx->list, &mali_c55->contexts);
>>>> +
>>>> +    /*
>>>> +     * Some features of the ISP need to be disabled by default and only
>>>> +     * enabled at the same time as they're configured by a parameters buffer
>>>> +     */
>>>> +
>>>> +    /* Bypass the sqrt and square compression and expansion modules */
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
>>>> +                 MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
>>>> +                 MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
>>>> +
>>>> +    /* Bypass the temper module */
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
>>>> +               MALI_C55_REG_BYPASS_2_TEMPER);
>>>> +
>>>> +    /* Bypass the colour noise reduction  */
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
>>>> +               MALI_C55_REG_BYPASS_4_CNR);
>>>> +
>>>> +    /* Disable the sinter module */
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
>>>> +                 MALI_C55_SINTER_ENABLE_MASK, 0x00);
>>>> +
>>>> +    /* Disable the RGB Gamma module for each output */
>>>> +    mali_c55_write(
>>>> +        mali_c55,
>>>> + MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
>>>> +        0x00);
>>>> +    mali_c55_write(
>>>> +        mali_c55,
>>>> + MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
>>>> +        0x00);
>>>> +
>>>> +    /* Disable the colour correction matrix */
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
>>>> +
>>>> +    return 0;
>>>> +
>>>> +err_free_registers:
>>>> +    kfree(ctx->registers);
>>>> +err_free_ctx:
>>>> +    kfree(ctx);
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +static int mali_c55_runtime_resume(struct device *dev)
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>>> +    int ret;
>>>> +
>>>> +    ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
>>>> +                      mali_c55->clks);
>>>> +    if (ret)
>>>> +        dev_err(mali_c55->dev, "failed to enable clocks\n");
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +static int mali_c55_runtime_suspend(struct device *dev)
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>>> +
>>>> + clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static const struct dev_pm_ops mali_c55_pm_ops = {
>>>> +    SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
>>>> +                pm_runtime_force_resume)
>>>> +    SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
>>>> +               NULL)
>>>> +};
>>>> +
>>>> +static int mali_c55_probe(struct platform_device *pdev)
>>>> +{
>>>> +    struct device *dev = &pdev->dev;
>>>> +    struct mali_c55 *mali_c55;
>>>> +    dma_cap_mask_t mask;
>>>> +    u32 version;
>>>> +    int ret;
>>>> +    u32 val;
>>>> +
>>>> +    mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
>>>> +    if (!mali_c55)
>>>> +        return dev_err_probe(dev, -ENOMEM,
>>>> +                     "failed to allocate memory\n");
>>>         return -ENOMEM;
>>>
>>> There's no need to print messages for memory allocation failures.
>>>
>>>> +
>>>> +    mali_c55->dev = dev;
>>>> +    platform_set_drvdata(pdev, mali_c55);
>>>> +
>>>> +    mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
>>>> +                                &mali_c55->res);
>>>> +    if (IS_ERR(mali_c55->base))
>>>> +        return dev_err_probe(dev, PTR_ERR(mali_c55->base),
>>>> +                     "failed to map IO memory\n");
>>>> +
>>>> +    ret = platform_get_irq(pdev, 0);
>>>> +    if (ret < 0)
>>>> +        return dev_err_probe(dev, ret, "failed to get interrupt num\n");
>>> s/ num// or s/num/number/
>>>
>>>> +
>>>> +    ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
>>>> +                    mali_c55_isr, IRQF_ONESHOT,
>>>> +                    dev_driver_string(&pdev->dev),
>>>> +                    &pdev->dev);
>>> Requested the IRQ should be done much lower, after you have initialized
>>> everything, or an IRQ that would fire early would have really bad
>>> consequences.
>>>
>>> A comment to explain why you need a threaded interrupt handler would be
>>> good. I assume it is due only to the need to transfer the registers
>>> using DMA. I wonder if we should then split the interrupt handler in
>>> two, with a non-threaded part for the operations that can run quickly,
>>> and a threaded part for the reprogramming.
>>>
>>> It may also be that we could just start the DMA transfer in the
>>> non-threaded handler without waiting synchronously for it to complete.
>>> That would be a bigger change, and would require checking race
>>> conditions carefully. On the other hand, I'm a bit concerned about the
>>> current implementation, have you tested what happens if the DMA transfer
>>> takes too long to complete, and spans frame boundaries ? This part could
>>> be addressed by a patch on top of this one.
>>>
>>>> +    if (ret)
>>>> +        return dev_err_probe(dev, ret, "failed to request irq\n");
>>>> +
>>>> +    for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
>>>> +        mali_c55->clks[i].id = mali_c55_clk_names[i];
>>>> +
>>>> +    ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
>>>> +    if (ret)
>>>> +        return dev_err_probe(dev, ret, "failed to acquire clocks\n");
>>>> +
>>>> +    pm_runtime_enable(&pdev->dev);
>>>> +
>>>> +    ret = pm_runtime_resume_and_get(&pdev->dev);
>>>> +    if (ret)
>>>> +        goto err_pm_runtime_disable;
>>>> +
>>>> +    of_reserved_mem_device_init(dev);
>>> I'd move this, the vb2_dma_contig_set_max_seg_size() call and the
>>> dma_cap_* calls before pm_runtime_enable() as they don't need the device
>>> to be powered.
>>>
>>>> +    version = mali_c55_check_hwcfg(mali_c55);
>>>> +    vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
>>>> +
>>>> +    /* Use "software only" context management. */
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>> +                 MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
>>>> +                 MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
>>> You handle that in mali_c55_isp_start(), does the register have to be
>>> set here too ?
>>>
>>>> +
>>>> +    dma_cap_zero(mask);
>>>> +    dma_cap_set(DMA_MEMCPY, mask);
>>>> +
>>>> +    /*
>>>> +     * No error check, because we will just fallback on memcpy if there is
>>>> +     * no usable DMA channel on the system.
>>>> +     */
>>>> +    mali_c55->channel = dma_request_channel(mask, NULL, NULL);
>>>> +
>>>> +    INIT_LIST_HEAD(&mali_c55->contexts);
>>>> +    ret = mali_c55_init_context(mali_c55);
>>>> +    if (ret)
>>>> +        goto err_release_dma_channel;
>>>> +
>>> I'd move all the code from here ...
>>>
>>>> +    mali_c55->media_dev.dev = dev;
>>>> +    strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
>>>> +        sizeof(mali_c55->media_dev.model));
>>>> +    mali_c55->media_dev.hw_revision = version;
>>>> +
>>>> +    media_device_init(&mali_c55->media_dev);
>>>> +    ret = media_device_register(&mali_c55->media_dev);
>>>> +    if (ret)
>>>> +        goto err_cleanup_media_device;
>>>> +
>>>> +    mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
>>>> +    ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
>>>> +    if (ret) {
>>>> +        dev_err(dev, "failed to register V4L2 device\n");
>>>> +        goto err_unregister_media_device;
>>>> +    };
>>>> +
>>>> +    ret = mali_c55_register_entities(mali_c55);
>>>> +    if (ret) {
>>>> +        dev_err(dev, "failed to register entities\n");
>>>> +        goto err_unregister_v4l2_device;
>>>> +    }
>>> ... to here to a separate function, or maybe fold it all in
>>> mali_c55_register_entities() (which should the be renamed). Same thing
>>> for the cleanup code.
>>>
>>>> +
>>>> +    /* Set safe stop to ensure we're in a non-streaming state */
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>>>> +               MALI_C55_INPUT_SAFE_STOP);
>>>> +    readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>>>> +
>>>> +    /*
>>>> +     * We're ready to process interrupts. Clear any that are set and then
>>>> +     * unmask them for processing.
>>>> +     */
>>>> +    mali_c55_write(mali_c55, 0x30, 0xffffffff);
>>>> +    mali_c55_write(mali_c55, 0x34, 0xffffffff);
>>>> +    mali_c55_write(mali_c55, 0x40, 0x01);
>>>> +    mali_c55_write(mali_c55, 0x40, 0x00);
>>> Please replace the register addresses with macros.
>>>
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);
>>> The value should use the interrupt bits macros.
>>>
>>>> +
>>>> +    pm_runtime_put(&pdev->dev);
>>> Once power gets cut, the registers your programmed above may be lost. I
>>> think you should programe them in the runtime PM resume handler.
>>>
>>>> +
>>>> +    return 0;
>>>> +
>>>> +err_unregister_v4l2_device:
>>>> +    v4l2_device_unregister(&mali_c55->v4l2_dev);
>>>> +err_unregister_media_device:
>>>> +    media_device_unregister(&mali_c55->media_dev);
>>>> +err_cleanup_media_device:
>>>> +    media_device_cleanup(&mali_c55->media_dev);
>>>> +err_release_dma_channel:
>>>> +    dma_release_channel(mali_c55->channel);
>>>> +err_pm_runtime_disable:
>>>> +    pm_runtime_disable(&pdev->dev);
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +static void mali_c55_remove(struct platform_device *pdev)
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
>>>> +    struct mali_c55_ctx *ctx, *tmp;
>>>> +
>>>> +    list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
>>>> +        list_del(&ctx->list);
>>>> +        kfree(ctx->registers);
>>>> +        kfree(ctx);
>>>> +    }
>>>> +
>>>> +    mali_c55_remove_links(mali_c55);
>>>> +    mali_c55_unregister_entities(mali_c55);
>>>> +    v4l2_device_put(&mali_c55->v4l2_dev);
>>>> +    media_device_unregister(&mali_c55->media_dev);
>>>> +    media_device_cleanup(&mali_c55->media_dev);
>>>> +    dma_release_channel(mali_c55->channel);
>>>> +}
>>>> +
>>>> +static const struct of_device_id mali_c55_of_match[] = {
>>>> +    { .compatible = "arm,mali-c55", },
>>>> +    {},
>>>     { /* Sentinel */ },
>>>
>>>> +};
>>>> +MODULE_DEVICE_TABLE(of, mali_c55_of_match);
>>>> +
>>>> +static struct platform_driver mali_c55_driver = {
>>>> +    .driver = {
>>>> +        .name = "mali-c55",
>>>> +        .of_match_table = of_match_ptr(mali_c55_of_match),
>>> Drop of_match_ptr().
>>>
>>>> +        .pm = &mali_c55_pm_ops,
>>>> +    },
>>>> +    .probe = mali_c55_probe,
>>>> +    .remove_new = mali_c55_remove,
>>>> +};
>>>> +
>>>> +module_platform_driver(mali_c55_driver);
>>> Blank line.
>>>
>>>> +MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
>>>> +MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
>>>> +MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
>>>> +MODULE_LICENSE("GPL");
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c 
>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>> new file mode 100644
>>>> index 000000000000..ea8b7b866e7a
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>> @@ -0,0 +1,611 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Image signal processor
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#include <linux/delay.h>
>>>> +#include <linux/iopoll.h>
>>>> +#include <linux/property.h>
>>>> +#include <linux/string.h>
>>>> +
>>>> +#include <media/media-entity.h>
>>>> +#include <media/v4l2-common.h>
>>>> +#include <media/v4l2-event.h>
>>>> +#include <media/v4l2-mc.h>
>>>> +#include <media/v4l2-subdev.h>
>>>> +
>>>> +#include "mali-c55-common.h"
>>>> +#include "mali-c55-registers.h"
>>>> +
>>>> +static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
>>>> +    {
>>>> +        .code = MEDIA_BUS_FMT_SRGGB20_1X20,
>>>> +        .order = MALI_C55_BAYER_ORDER_RGGB,
>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
>>>> +    },
>>>> +    {
>>>> +        .code = MEDIA_BUS_FMT_SGRBG20_1X20,
>>>> +        .order = MALI_C55_BAYER_ORDER_GRBG,
>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
>>>> +    },
>>>> +    {
>>>> +        .code = MEDIA_BUS_FMT_SGBRG20_1X20,
>>>> +        .order = MALI_C55_BAYER_ORDER_GBRG,
>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
>>>> +    },
>>>> +    {
>>>> +        .code = MEDIA_BUS_FMT_SBGGR20_1X20,
>>>> +        .order = MALI_C55_BAYER_ORDER_BGGR,
>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
>>>> +    },
>>>> +    {
>>>> +        .code = MEDIA_BUS_FMT_RGB202020_1X60,
>>>> +        .order = 0, /* Not relevant for this format */
>>>> +        .encoding = V4L2_PIXEL_ENC_RGB,
>>>> +    }
>>>> +    /*
>>>> +     * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
>>>> +     * also support YUV input from a sensor passed-through to the output. At
>>>> +     * present we have no mechanism to test that though so it may have to
>>>> +     * wait a while...
>>>> +     */
>>>> +};
>>>> +
>>>> +const struct mali_c55_isp_fmt *
>>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
>>>> +{
>>>> +    if (!fmt)
>>>> +        fmt = &mali_c55_isp_fmts[0];
>>>> +    else
>>>> +        fmt++;
>>>> +
>>>> +    for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
>>>> +        return fmt;
>>> That's peculiar.
>>>
>>>     if (!fmt)
>>>         fmt = &mali_c55_isp_fmts[0];
>>>     else if (fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)])
>>>         return ++fmt;
>>>     else
>>>         return NULL;
>>>
>>>> +
>>>> +    return NULL;
>>>> +}
>>>> +
>>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
>>>> +{
>>>> +    const struct mali_c55_isp_fmt *isp_fmt;
>>>> +
>>>> +    for_each_mali_isp_fmt(isp_fmt) {
>>> I would open-code the loop instead of using the macro, like you do
>>> below. It will be more efficient.
>>>
>>>> +        if (isp_fmt->code == mbus_code)
>>>> +            return true;
>>>> +    }
>>>> +
>>>> +    return false;
>>>> +}
>>>> +
>>>> +static const struct mali_c55_isp_fmt *
>>>> +mali_c55_isp_get_mbus_config_by_code(u32 code)
>>>> +{
>>>> +    unsigned int i;
>>>> +
>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
>>>> +        if (mali_c55_isp_fmts[i].code == code)
>>>> +            return &mali_c55_isp_fmts[i];
>>>> +    }
>>>> +
>>>> +    return NULL;
>>>> +}
>>>> +
>>>> +static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    u32 val;
>>>> +
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);
>>>     mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>>>                MALI_C55_INPUT_SAFE_STOP);
>>>
>>>> + readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>>>> +}
>>>> +
>>>> +static int mali_c55_isp_start(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>> +    const struct mali_c55_isp_fmt *cfg;
>>>> +    struct v4l2_mbus_framefmt *format;
>>> const
>>>
>>>> +    struct v4l2_subdev_state *state;
>>>> +    struct v4l2_rect *crop;
>>> const
>>>
>>>> +    struct v4l2_subdev *sd;
>>>> +    u32 val;
>>>> +    int ret;
>>>> +
>>>> +    sd = &mali_c55->isp.sd;
>>> Assign when declaring the variable.
>>>
>>>> +
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>> +                 MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
>>>> +
>>>> +    /* Apply input windowing */
>>>> +    state = v4l2_subdev_get_locked_active_state(sd);
>>> Using .enable_streams() (see below) you'll get this for free.
>>>
>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +    format = v4l2_subdev_state_get_format(state,
>>>> +                          MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +    cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
>>>> +
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
>>>> +               MALI_C55_HC_START(crop->left));
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
>>>> +               MALI_C55_HC_SIZE(crop->width));
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
>>>> +               MALI_C55_VC_START(crop->top) |
>>>> +               MALI_C55_VC_SIZE(crop->height));
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
>>>> +                 MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
>>>> +                 MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
>>>> +                 MALI_C55_BAYER_ORDER_MASK, cfg->order);
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
>>>> +                 MALI_C55_INPUT_WIDTH_MASK,
>>>> +                 MALI_C55_INPUT_WIDTH_20BIT);
>>>> +
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>>> +                 MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
>>>> +                 cfg->encoding == V4L2_PIXEL_ENC_RGB ?
>>>> +                 MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
>>>> +
>>>> +    ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
>>>> +    if (ret) {
>>>> +        dev_err(mali_c55->dev, "failed to DMA config\n");
>>>> +        return ret;
>>>> +    }
>>>> +
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>>>> +               MALI_C55_INPUT_SAFE_START);
>>>> +    readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>>> Should you return an error in case of timeout ?
>>>
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)
>>> Why is this not handled wired to .s_stream() ? Or better,
>>> .enable_streams() and .disable_streams().
>>>
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>> +    struct v4l2_subdev *sd;
>>>> +
>>>> +    if (isp->remote_src) {
>>>> +        sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
>>>> +        v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
>>>> +    }
>>>> +    isp->remote_src = NULL;
>>>> +
>>>> +    mali_c55_isp_stop(mali_c55);
>>>> +}
>>>> +
>>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>> +    struct media_pad *sink_pad;
>>>> +    struct v4l2_subdev *sd;
>>>> +    int ret;
>>>> +
>>>> +    sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
>>>> +    isp->remote_src = media_pad_remote_pad_unique(sink_pad);
>>>> +    if (IS_ERR(isp->remote_src)) {
>>> If you set the MUST_CONNECT flag for the MALI_C55_ISP_PAD_SINK_VIDEO pad
>>> I think you can drop this check.
>>>
>>>> +        dev_err(mali_c55->dev, "Failed to get source for ISP\n");
>>>> +        return PTR_ERR(isp->remote_src);
>>>> +    }
>>>> +
>>>> +    sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
>>>> +
>>>> +    isp->frame_sequence = 0;
>>>> +    ret = mali_c55_isp_start(mali_c55);
>>>> +    if (ret) {
>>>> +        dev_err(mali_c55->dev, "Failed to start ISP\n");
>>>> +        isp->remote_src = NULL;
>>>> +        return ret;
>>>> +    }
>>>> +
>>>> +    /*
>>>> +     * We only support a single input stream, so we can just enable the 1st
>>>> +     * entry in the streams mask.
>>>> +     */
>>>> +    ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
>>>> +    if (ret) {
>>>> +        dev_err(mali_c55->dev, "Failed to start ISP source\n");
>>>> +        mali_c55_isp_stop(mali_c55);
>>>> +        return ret;
>>>> +    }
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>> +                       struct v4l2_subdev_state *state,
>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
>>>> +{
>>>> +    /*
>>>> +     * Only the internal RGB processed format is allowed on the regular
>>>> +     * processing source pad.
>>>> +     */
>>>> +    if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
>>>> +        if (code->index)
>>>> +            return -EINVAL;
>>>> +
>>>> +        code->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>> +        return 0;
>>>> +    }
>>>> +
>>>> +    /* On the sink and bypass pads all the supported formats are allowed. */
>>>> +    if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
>>>> +        return -EINVAL;
>>>> +
>>>> +    code->code = mali_c55_isp_fmts[code->index].code;
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
>>>> +                    struct v4l2_subdev_state *state,
>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
>>>> +{
>>>> +    const struct mali_c55_isp_fmt *cfg;
>>>> +
>>>> +    if (fse->index > 0)
>>>> +        return -EINVAL;
>>>> +
>>>> +    /*
>>>> +     * Only the internal RGB processed format is allowed on the regular
>>>> +     * processing source pad.
>>>> +     *
>>>> +     * On the sink and bypass pads all the supported formats are allowed.
>>>> +     */
>>>> +    if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
>>>> +        if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
>>>> +            return -EINVAL;
>>>> +    } else {
>>>> +        cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
>>>> +        if (!cfg)
>>>> +            return -EINVAL;
>>>> +    }
>>>> +
>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
>>>> +                struct v4l2_subdev_state *state,
>>>> +                struct v4l2_subdev_format *format)
>>>> +{
>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
>>>> +    struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
>>>> +    const struct mali_c55_isp_fmt *cfg;
>>>> +    struct v4l2_rect *crop;
>>>> +
>>>> +    /*
>>>> +     * Disallow set_fmt on the source pads; format is fixed and the sizes
>>>> +     * are the result of applying the sink crop rectangle to the sink
>>>> +     * format.
>>>> +     */
>>>> +    if (format->pad)
>>>     if (format->pad != MALI_C55_ISP_PAD_SINK_VIDEO)
>>>
>>>> +        return v4l2_subdev_get_fmt(sd, state, format);
>>>> +
>>>> +    cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
>>>> +    if (!cfg)
>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>> +    fmt->field = V4L2_FIELD_NONE;
>>> Do you intentionally allow the colorspace fields to be overwritten to
>>> any value ? rkisp1_isp_set_src_fmt() and rkisp1_isp_set_sink_fmt() may
>>> show you how this could be handled.
>>>
>>>> +
>>>> +    /*
>>>> +     * Clamp sizes in the accepted limits and clamp the crop rectangle in
>>>> +     * the new sizes.
>>>> +     */
>>>> +    clamp_t(unsigned int, fmt->width,
>>>> +        MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>>> +    clamp_t(unsigned int, fmt->width,
>>>> +        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>> clamp_t() returns a value, which you ignore. Those are no-ops. You meant
>>>
>>>     fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
>>>                  MALI_C55_MAX_WIDTH);
>>>     fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
>>>                   MALI_C55_MAX_HEIGHT);
>>>
>>> Same for every use of clamp_t() through the whole driver.
>>>
>>> Also, do you need clamp_t() ? I think all values are unsigned int, you
>>> can use clamp().
>>>
>>> Are there any alignment constraints, such a multiples of two for bayer
>>> formats ? Same in all the other locations where applicable.
>>>
>>>> +
>>>> +    sink_fmt = v4l2_subdev_state_get_format(state,
>>>> +                        MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +    *sink_fmt = *fmt;
>>>> +
>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +    crop->left = 0;
>>>> +    crop->top = 0;
>>>> +    crop->width = fmt->width;
>>>> +    crop->height = fmt->height;
>>>> +
>>>> +    /*
>>>> +     * Propagate format to source pads. On the 'regular' output pad use
>>>> +     * the internal RGB processed format, while on the bypass pad simply
>>>> +     * replicate the ISP sink format. The sizes on both pads are the same as
>>>> +     * the ISP sink crop rectangle.
>>>> +     */
>>> Colorspace information will need to be propagated too.
>>>
>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
>>>> +    src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>> +    src_fmt->width = crop->width;
>>>> +    src_fmt->height = crop->height;
>>>> +
>>>> +    src_fmt = v4l2_subdev_state_get_format(state,
>>>> +                           MALI_C55_ISP_PAD_SOURCE_BYPASS);
>>>> +    src_fmt->code = fmt->code;
>>>> +    src_fmt->width = crop->width;
>>>> +    src_fmt->height = crop->height;
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
>>>> +                      struct v4l2_subdev_state *state,
>>>> +                      struct v4l2_subdev_selection *sel)
>>>> +{
>>>> +    if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
>>>     sel->pad != MALI_C55_ISP_PAD_SINK_VIDEO
>>>
>>>> +        return -EINVAL;
>>>> +
>>>> +    sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
>>>> +                      struct v4l2_subdev_state *state,
>>>> +                      struct v4l2_subdev_selection *sel)
>>>> +{
>>>> +    struct v4l2_mbus_framefmt *src_fmt;
>>>> +    struct v4l2_mbus_framefmt *fmt;
>>> const
>>>
>>>> +    struct v4l2_rect *crop;
>>>> +
>>>> +    if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
>>> Ditto.
>>>
>>>> +        return -EINVAL;
>>>> +
>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +
>>>> +    clamp_t(unsigned int, sel->r.left, 0, fmt->width);
>>>> +    clamp_t(unsigned int, sel->r.top, 0, fmt->height);
>>>> +    clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
>>>> +        fmt->width - sel->r.left);
>>>> +    clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
>>>> +        fmt->height - sel->r.top);
>>>> +
>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +    *crop = sel->r;
>>>> +
>>>> +    /* Propagate the crop rectangle sizes to the source pad format. */
>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
>>>> +    src_fmt->width = crop->width;
>>>> +    src_fmt->height = crop->height;
>>> Can you confirm that cropping doesn't affect the bypass path ? And maybe
>>> add a comment to mention it.
>>>
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
>>>> +    .enum_mbus_code        = mali_c55_isp_enum_mbus_code,
>>>> +    .enum_frame_size    = mali_c55_isp_enum_frame_size,
>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
>>>> +    .set_fmt        = mali_c55_isp_set_fmt,
>>>> +    .get_selection        = mali_c55_isp_get_selection,
>>>> +    .set_selection        = mali_c55_isp_set_selection,
>>>> +    .link_validate        = v4l2_subdev_link_validate_default,
>>>> +};
>>>> +
>>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    struct v4l2_event event = {
>>>> +        .type = V4L2_EVENT_FRAME_SYNC,
>>>> +    };
>>>> +
>>>> +    event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
>>>> +    v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
>>>> +}
>>>> +
>>>> +static int
>>>> +mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
>>>> +                 struct v4l2_event_subscription *sub)
>>>> +{
>>>> +    if (sub->type != V4L2_EVENT_FRAME_SYNC)
>>>> +        return -EINVAL;
>>>> +
>>>> +    /* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
>>>> +    if (sub->id != 0)
>>>> +        return -EINVAL;
>>>> +
>>>> +    return v4l2_event_subscribe(fh, sub, 0, NULL);
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
>>>> +    .subscribe_event = mali_c55_isp_subscribe_event,
>>>> +    .unsubscribe_event = v4l2_event_subdev_unsubscribe,
>>>> +};
>>>> +
>>>> +static const struct v4l2_subdev_ops mali_c55_isp_ops = {
>>>> +    .pad    = &mali_c55_isp_pad_ops,
>>>> +    .core    = &mali_c55_isp_core_ops,
>>>> +};
>>>> +
>>>> +static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
>>>> +                   struct v4l2_subdev_state *sd_state)
>>> You name this variable state in every other subdev operation handler.
>>>
>>>> +{
>>>> +    struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>> +    struct v4l2_rect *in_crop;
>>>> +
>>>> +    sink_fmt = v4l2_subdev_state_get_format(sd_state,
>>>> +                        MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +    src_fmt = v4l2_subdev_state_get_format(sd_state,
>>>> +                           MALI_C55_ISP_PAD_SOURCE);
>>>> +    in_crop = v4l2_subdev_state_get_crop(sd_state,
>>>> +                         MALI_C55_ISP_PAD_SINK_VIDEO);
>>>> +
>>>> +    sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>> +    sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>> +    sink_fmt->field = V4L2_FIELD_NONE;
>>>> +    sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>> You should initialize the colorspace fields too. Same below.
>>>
>>>> +
>>>> +    *v4l2_subdev_state_get_format(sd_state,
>>>> +                  MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
>>>> +
>>>> +    src_fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>> +    src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>> +    src_fmt->field = V4L2_FIELD_NONE;
>>>> +    src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>> +
>>>> +    in_crop->top = 0;
>>>> +    in_crop->left = 0;
>>>> +    in_crop->width = MALI_C55_DEFAULT_WIDTH;
>>>> +    in_crop->height = MALI_C55_DEFAULT_HEIGHT;
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
>>>> +    .init_state = mali_c55_isp_init_state,
>>>> +};
>>>> +
>>>> +static const struct media_entity_operations mali_c55_isp_media_ops = {
>>>> +    .link_validate        = v4l2_subdev_link_validate,
>>>     .link_validate = v4l2_subdev_link_validate,
>>>
>>> to match mali_c55_isp_internal_ops.
>>>
>>>> +};
>>>> +
>>>> +static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
>>>> +                       struct v4l2_subdev *subdev,
>>>> +                       struct v4l2_async_connection *asc)
>>>> +{
>>>> +    struct mali_c55_isp *isp = container_of(notifier,
>>>> +                        struct mali_c55_isp, notifier);
>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>> +    struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
>>>> +    int ret;
>>>> +
>>>> +    /*
>>>> +     * By default we'll flag this link enabled and the TPG disabled, but
>>>> +     * no immutable flag because we need to be able to switch between the
>>>> +     * two.
>>>> +     */
>>>> +    ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
>>>> +                          MEDIA_LNK_FL_ENABLED);
>>>> +    if (ret)
>>>> +        dev_err(mali_c55->dev, "failed to create link for %s\n",
>>>> +            subdev->name);
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
>>>> +{
>>>> +    struct mali_c55_isp *isp = container_of(notifier,
>>>> +                        struct mali_c55_isp, notifier);
>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>> +
>>>> +    return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
>>>> +}
>>>> +
>>>> +static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
>>>> +    .bound = mali_c55_isp_notifier_bound,
>>>> +    .complete = mali_c55_isp_notifier_complete,
>>>> +};
>>>> +
>>>> +static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>> +    struct v4l2_async_connection *asc;
>>>> +    struct fwnode_handle *ep;
>>>> +    int ret;
>>>> +
>>>> +    v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
>>>> +
>>>> +    /*
>>>> +     * The ISP should have a single endpoint pointing to some flavour of
>>>> +     * CSI-2 receiver...but for now at least we do want everything to work
>>>> +     * normally even with no sensors connected, as we have the TPG. If we
>>>> +     * don't find a sensor just warn and return success.
>>>> +     */
>>>> +    ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
>>>> +                         0, 0, 0);
>>>> +    if (!ep) {
>>>> +        dev_warn(mali_c55->dev, "no local endpoint found\n");
>>>> +        return 0;
>>>> +    }
>>>> +
>>>> +    asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
>>>> +                          struct v4l2_async_connection);
>>>> +    if (IS_ERR(asc)) {
>>>> +        dev_err(mali_c55->dev, "failed to add remote fwnode\n");
>>>> +        ret = PTR_ERR(asc);
>>>> +        goto err_put_ep;
>>>> +    }
>>>> +
>>>> +    isp->notifier.ops = &mali_c55_isp_notifier_ops;
>>>> +    ret = v4l2_async_nf_register(&isp->notifier);
>>>> +    if (ret) {
>>>> +        dev_err(mali_c55->dev, "failed to register notifier\n");
>>>> +        goto err_cleanup_nf;
>>>> +    }
>>>> +
>>>> +    fwnode_handle_put(ep);
>>>> +
>>>> +    return 0;
>>>> +
>>>> +err_cleanup_nf:
>>>> +    v4l2_async_nf_cleanup(&isp->notifier);
>>>> +err_put_ep:
>>>> +    fwnode_handle_put(ep);
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
>>>> +    struct v4l2_subdev *sd = &isp->sd;
>>>> +    int ret;
>>>> +
>>>> +    isp->mali_c55 = mali_c55;
>>>> +
>>>> +    v4l2_subdev_init(sd, &mali_c55_isp_ops);
>>>> +    sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
>>>> +    sd->entity.ops = &mali_c55_isp_media_ops;
>>>> +    sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
>>>> +    sd->internal_ops = &mali_c55_isp_internal_ops;
>>>> +    strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
>>>> +
>>>> +    isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
>>> The MUST_CONNECT flag would make sense here.
>>>
>>>> + isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>>>> +    isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
>>>> +
>>>> +    ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
>>>> +                     isp->pads);
>>>> +    if (ret)
>>>> +        return ret;
>>>> +
>>>> +    ret = v4l2_subdev_init_finalize(sd);
>>>> +    if (ret)
>>>> +        goto err_cleanup_media_entity;
>>>> +
>>>> +    ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>>> +    if (ret)
>>>> +        goto err_cleanup_subdev;
>>>> +
>>>> +    ret = mali_c55_isp_parse_endpoint(isp);
>>>> +    if (ret)
>>>> +        goto err_cleanup_subdev;
>>> As noted elsewhere, I think this belongs to mali-c55-core.c.
>>>
>>>> +
>>>> +    mutex_init(&isp->lock);
>>> This lock is used in mali-c55-capture.c only, that seems weird.
>>>
>>>> +
>>>> +    return 0;
>>>> +
>>>> +err_cleanup_subdev:
>>>> +    v4l2_subdev_cleanup(sd);
>>>> +err_cleanup_media_entity:
>>>> +    media_entity_cleanup(&sd->entity);
>>>> +    isp->mali_c55 = NULL;
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
>>>> +
>>>> +    if (!isp->mali_c55)
>>>> +        return;
>>>> +
>>>> +    mutex_destroy(&isp->lock);
>>>> +    v4l2_async_nf_unregister(&isp->notifier);
>>>> +    v4l2_async_nf_cleanup(&isp->notifier);
>>>> +    v4l2_device_unregister_subdev(&isp->sd);
>>>> +    v4l2_subdev_cleanup(&isp->sd);
>>>> +    media_entity_cleanup(&isp->sd.entity);
>>>> +}
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h 
>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>> new file mode 100644
>>>> index 000000000000..cb27abde2aa5
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>> @@ -0,0 +1,258 @@
>>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Register definitions
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#ifndef _MALI_C55_REGISTERS_H
>>>> +#define _MALI_C55_REGISTERS_H
>>>> +
>>>> +#include <linux/bits.h>
>>>> +
>>>> +/* ISP Common 0x00000 - 0x000ff */
>>>> +
>>>> +#define MALI_C55_REG_API                0x00000
>>>> +#define MALI_C55_REG_PRODUCT                0x00004
>>>> +#define MALI_C55_REG_VERSION                0x00008
>>>> +#define MALI_C55_REG_REVISION                0x0000c
>>>> +#define MALI_C55_REG_PULSE_MODE                0x0003c
>>>> +#define MALI_C55_REG_INPUT_MODE_REQUEST            0x0009c
>>>> +#define MALI_C55_INPUT_SAFE_STOP            0x00
>>>> +#define MALI_C55_INPUT_SAFE_START            0x01
>>>> +#define MALI_C55_REG_MODE_STATUS            0x000a0
>>>> +#define MALI_C55_REG_INTERRUPT_MASK_VECTOR        0x00030
>>>> +#define MALI_C55_INTERRUPT_MASK_ALL            GENMASK(31, 0)
>>>> +
>>>> +#define MALI_C55_REG_GLOBAL_MONITOR            0x00050
>>>> +
>>>> +#define MALI_C55_REG_GEN_VIDEO                0x00080
>>>> +#define MALI_C55_REG_GEN_VIDEO_ON_MASK            BIT(0)
>>>> +#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK        BIT(1)
>>>> +#define MALI_C55_REG_GEN_PREFETCH_MASK GENMASK(31, 16)
>>>> +
>>>> +#define MALI_C55_REG_MCU_CONFIG                0x00020
>>>> +#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK        BIT(0)
>>> #define MALI_C55_REG_MCU_CONFIG_OVERRIDE        BIT(0)
>>>
>>> Same in other places where applicable.
>>>
>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK        BIT(1)
>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PING        BIT(1)
>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG        0x00
>>>> +#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK        BIT(8)
>>>> +#define MALI_C55_REG_PING_PONG_READ            0x00024
>>>> +#define MALI_C55_REG_PING_PONG_READ_MASK        BIT(2)
>>>> +
>>>> +#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR        0x00034
>>>> +#define MALI_C55_REG_INTERRUPT_CLEAR            0x00040
>>>> +#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR        0x00044
>>>> +#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS        0x00068
>>>> +#define MALI_C55_GPS_PONG_FITTED            BIT(0)
>>>> +#define MALI_C55_GPS_WDR_FITTED                BIT(1)
>>>> +#define MALI_C55_GPS_COMPRESSION_FITTED            BIT(2)
>>>> +#define MALI_C55_GPS_TEMPER_FITTED            BIT(3)
>>>> +#define MALI_C55_GPS_SINTER_LITE_FITTED            BIT(4)
>>>> +#define MALI_C55_GPS_SINTER_FITTED            BIT(5)
>>>> +#define MALI_C55_GPS_IRIDIX_LTM_FITTED            BIT(6)
>>>> +#define MALI_C55_GPS_IRIDIX_GTM_FITTED            BIT(7)
>>>> +#define MALI_C55_GPS_CNR_FITTED                BIT(8)
>>>> +#define MALI_C55_GPS_FRSCALER_FITTED            BIT(9)
>>>> +#define MALI_C55_GPS_DS_PIPE_FITTED            BIT(10)
>>>> +
>>>> +#define MALI_C55_REG_BLANKING                0x00084
>>>> +#define MALI_C55_REG_HBLANK_MASK            GENMASK(15, 0)
>>>> +#define MALI_C55_REG_VBLANK_MASK            GENMASK(31, 16)
>>>> +
>>>> +#define MALI_C55_REG_HC_START                0x00088
>>>> +#define MALI_C55_HC_START(h)                (((h) & 0xffff) << 16)
>>>> +#define MALI_C55_REG_HC_SIZE                0x0008c
>>>> +#define MALI_C55_HC_SIZE(h)                ((h) & 0xffff)
>>>> +#define MALI_C55_REG_VC_START_SIZE            0x00094
>>>> +#define MALI_C55_VC_START(v)                ((v) & 0xffff)
>>>> +#define MALI_C55_VC_SIZE(v)                (((v) & 0xffff) << 16)
>>>> +
>>>> +/* Ping/Pong Configuration Space */
>>>> +#define MALI_C55_REG_BASE_ADDR                0x18e88
>>>> +#define MALI_C55_REG_BYPASS_0                0x18eac
>>>> +#define MALI_C55_REG_BYPASS_0_VIDEO_TEST        BIT(0)
>>>> +#define MALI_C55_REG_BYPASS_0_INPUT_FMT            BIT(1)
>>>> +#define MALI_C55_REG_BYPASS_0_DECOMPANDER        BIT(2)
>>>> +#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR BIT(3)
>>>> +#define MALI_C55_REG_BYPASS_0_GAIN_WDR            BIT(4)
>>>> +#define MALI_C55_REG_BYPASS_0_FRAME_STITCH        BIT(5)
>>>> +#define MALI_C55_REG_BYPASS_1                0x18eb0
>>>> +#define MALI_C55_REG_BYPASS_1_DIGI_GAIN            BIT(0)
>>>> +#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS        BIT(1)
>>>> +#define MALI_C55_REG_BYPASS_1_FE_SQRT            BIT(2)
>>>> +#define MALI_C55_REG_BYPASS_1_RAW_FE            BIT(3)
>>>> +#define MALI_C55_REG_BYPASS_2                0x18eb8
>>>> +#define MALI_C55_REG_BYPASS_2_SINTER            BIT(0)
>>>> +#define MALI_C55_REG_BYPASS_2_TEMPER            BIT(1)
>>>> +#define MALI_C55_REG_BYPASS_3                0x18ebc
>>>> +#define MALI_C55_REG_BYPASS_3_SQUARE_BE            BIT(0)
>>>> +#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH BIT(1)
>>>> +#define MALI_C55_REG_BYPASS_3_MESH_SHADING        BIT(3)
>>>> +#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE        BIT(4)
>>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX            BIT(5)
>>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN        BIT(6)
>>>> +#define MALI_C55_REG_BYPASS_4                0x18ec0
>>>> +#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB        BIT(1)
>>>> +#define MALI_C55_REG_BYPASS_4_PF_CORRECTION        BIT(3)
>>>> +#define MALI_C55_REG_BYPASS_4_CCM            BIT(4)
>>>> +#define MALI_C55_REG_BYPASS_4_CNR            BIT(5)
>>>> +#define MALI_C55_REG_FR_BYPASS                0x18ec4
>>>> +#define MALI_C55_REG_DS_BYPASS                0x18ec8
>>>> +#define MALI_C55_BYPASS_CROP                BIT(0)
>>>> +#define MALI_C55_BYPASS_SCALER                BIT(1)
>>>> +#define MALI_C55_BYPASS_GAMMA_RGB            BIT(2)
>>>> +#define MALI_C55_BYPASS_SHARPEN                BIT(3)
>>>> +#define MALI_C55_BYPASS_CS_CONV                BIT(4)
>>>> +#define MALI_C55_REG_ISP_RAW_BYPASS            0x18ecc
>>>> +#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK        BIT(0)
>>>> +#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK GENMASK(9, 8)
>>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS        2
>>>> +#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS        1
>>>> +#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE BIT(1)
>>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS        BIT(0)
>>>> +
>>>> +#define MALI_C55_REG_ACTIVE_WIDTH_MASK            0xffff
>>>> +#define MALI_C55_REG_ACTIVE_HEIGHT_MASK 0xffff0000
>>>> +#define MALI_C55_REG_BAYER_ORDER            0x18e8c
>>>> +#define MALI_C55_BAYER_ORDER_MASK            GENMASK(1, 0)
>>>> +#define MALI_C55_REG_TPG_CH0                0x18ed8
>>>> +#define MALI_C55_TEST_PATTERN_ON_OFF            BIT(0)
>>>> +#define MALI_C55_TEST_PATTERN_RGB_MASK            BIT(1)
>>>> +#define MALI_C55_REG_TPG_R_BACKGROUND            0x18ee0
>>>> +#define MALI_C55_REG_TPG_G_BACKGROUND            0x18ee4
>>>> +#define MALI_C55_REG_TPG_B_BACKGROUND            0x18ee8
>>>> +#define MALI_C55_TPG_BACKGROUND_MAX            0xfffff
>>>> +#define MALI_C55_REG_INPUT_WIDTH            0x18f98
>>>> +#define MALI_C55_INPUT_WIDTH_MASK            GENMASK(18, 16)
>>>> +#define MALI_C55_INPUT_WIDTH_8BIT            0
>>>> +#define MALI_C55_INPUT_WIDTH_10BIT            1
>>>> +#define MALI_C55_INPUT_WIDTH_12BIT            2
>>>> +#define MALI_C55_INPUT_WIDTH_14BIT            3
>>>> +#define MALI_C55_INPUT_WIDTH_16BIT            4
>>>> +#define MALI_C55_INPUT_WIDTH_20BIT            5
>>>> +#define MALI_C55_REG_SPACE_SIZE                0x4000
>>>> +#define MALI_C55_REG_CONFIG_SPACES_OFFSET        0x0ab6c
>>>> +#define MALI_C55_CONFIG_SPACE_SIZE            0x1231c
>>>> +
>>>> +#define MALI_C55_REG_SINTER_CONFIG            0x19348
>>>> +#define MALI_C55_SINTER_VIEW_FILTER_MASK        GENMASK(1, 0)
>>>> +#define MALI_C55_SINTER_SCALE_MODE_MASK GENMASK(3, 2)
>>>> +#define MALI_C55_SINTER_ENABLE_MASK            BIT(4)
>>>> +#define MALI_C55_SINTER_FILTER_SELECT_MASK        BIT(5)
>>>> +#define MALI_C55_SINTER_INT_SELECT_MASK            BIT(6)
>>>> +#define MALI_C55_SINTER_RM_ENABLE_MASK            BIT(7)
>>>> +
>>>> +/* Colour Correction Matrix Configuration */
>>>> +#define MALI_C55_REG_CCM_ENABLE                0x1b07c
>>>> +#define MALI_C55_CCM_ENABLE_MASK            BIT(0)
>>>> +#define MALI_C55_REG_CCM_COEF_R_R            0x1b080
>>>> +#define MALI_C55_REG_CCM_COEF_R_G            0x1b084
>>>> +#define MALI_C55_REG_CCM_COEF_R_B            0x1b088
>>>> +#define MALI_C55_REG_CCM_COEF_G_R            0x1b090
>>>> +#define MALI_C55_REG_CCM_COEF_G_G            0x1b094
>>>> +#define MALI_C55_REG_CCM_COEF_G_B            0x1b098
>>>> +#define MALI_C55_REG_CCM_COEF_B_R            0x1b0a0
>>>> +#define MALI_C55_REG_CCM_COEF_B_G            0x1b0a4
>>>> +#define MALI_C55_REG_CCM_COEF_B_B            0x1b0a8
>>>> +#define MALI_C55_CCM_COEF_MASK                GENMASK(12, 0)
>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R            0x1b0b0
>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G            0x1b0b4
>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B            0x1b0b8
>>>> +#define MALI_C55_CCM_ANTIFOG_GAIN_MASK GENMASK(11, 0)
>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R        0x1b0c0
>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G        0x1b0c4
>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B        0x1b0c8
>>>> +#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK        GENMASK(11, 0)
>>>> +
>>>> +/*
>>>> + * The Mali-C55 ISP has up to two output pipes; known as full resolution and
>>>> + * down scaled. The register space for these is laid out identically, but offset
>>>> + * by 372 bytes.
>>>> + */
>>>> +#define MALI_C55_CAP_DEV_FR_REG_OFFSET        0x0
>>>> +#define MALI_C55_CAP_DEV_DS_REG_OFFSET        0x174
>>>> +
>>>> +#define MALI_C55_REG_CS_CONV_CONFIG(offset)        (0x1c098 + (offset))
>>>> +#define MALI_C55_CS_CONV_MATRIX_MASK            BIT(0)
>>>> +#define MALI_C55_CS_CONV_FILTER_MASK            BIT(1)
>>>> +#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK        BIT(2)
>>>> +#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK        BIT(3)
>>>> +#define MALI_C55_REG_Y_WRITER_MODE(offset)        (0x1c0ec + (offset))
>>>> +#define MALI_C55_REG_UV_WRITER_MODE(offset)        (0x1c144 + (offset))
>>>> +#define MALI_C55_WRITER_MODE_MASK            GENMASK(4, 0)
>>>> +#define MALI_C55_WRITER_SUBMODE_MASK            GENMASK(7, 6)
>>>> +#define MALI_C55_WRITER_FRAME_WRITE_MASK        BIT(9)
>>>> +#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset) (0x1c0f0 + (offset))
>>>> +#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset) (0x1c148 + (offset))
>>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)        ((w) << 0)
>>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)        ((h) << 16)
>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset) (0x1c0f4 + (offset))
>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset) (0x1c108 + (offset))
>>>> +#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK GENMASK(2, 0)
>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_RESTART        BIT(3)
>>>> +#define MALI_C55_REG_Y_WRITER_OFFSET(offset) (0x1c10c + (offset))
>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset) (0x1c14c + (offset))
>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset) (0x1c160 + (offset))
>>>> +#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK GENMASK(2, 0)
>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_RESTART        BIT(3)
>>>> +#define MALI_C55_REG_UV_WRITER_OFFSET(offset) (0x1c164 + (offset))
>>>> +
>>>> +#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
>>>> +#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE 0x18edc
>>>> +
>>>> +#define MALI_C55_REG_CROP_EN(offset)            (0x1c028 + (offset))
>>>> +#define MALI_C55_CROP_ENABLE                BIT(0)
>>>> +#define MALI_C55_REG_CROP_X_START(offset)        (0x1c02c + (offset))
>>>> +#define MALI_C55_REG_CROP_Y_START(offset)        (0x1c030 + (offset))
>>>> +#define MALI_C55_REG_CROP_X_SIZE(offset)        (0x1c034 + (offset))
>>>> +#define MALI_C55_REG_CROP_Y_SIZE(offset)        (0x1c038 + (offset))
>>>> +#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset) (0x1c040 + (offset))
>>>> +#define MALI_C55_SCALER_TIMEOUT_EN            BIT(4)
>>>> +#define MALI_C55_SCALER_TIMEOUT(t)            ((t) << 16)
>>>> +#define MALI_C55_REG_SCALER_IN_WIDTH(offset) (0x1c044 + (offset))
>>>> +#define MALI_C55_REG_SCALER_IN_HEIGHT(offset) (0x1c048 + (offset))
>>>> +#define MALI_C55_REG_SCALER_OUT_WIDTH(offset) (0x1c04c + (offset))
>>>> +#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset) (0x1c050 + (offset))
>>>> +#define MALI_C55_REG_SCALER_HFILT_TINC(offset) (0x1c054 + (offset))
>>>> +#define MALI_C55_REG_SCALER_HFILT_COEF(offset) (0x1c058 + (offset))
>>>> +#define MALI_C55_REG_SCALER_VFILT_TINC(offset) (0x1c05c + (offset))
>>>> +#define MALI_C55_REG_SCALER_VFILT_COEF(offset) (0x1c060 + (offset))
>>>> +
>>>> +#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset) (0x1c064 + (offset))
>>>> +#define MALI_C55_GAMMA_ENABLE_MASK            BIT(0)
>>>> +#define MALI_C55_REG_GAMMA_GAINS_1(offset)        (0x1c068 + (offset))
>>>> +#define MALI_C55_GAMMA_GAIN_R_MASK            GENMASK(11, 0)
>>>> +#define MALI_C55_GAMMA_GAIN_G_MASK            GENMASK(27, 16)
>>>> +#define MALI_C55_REG_GAMMA_GAINS_2(offset)        (0x1c06c + (offset))
>>>> +#define MALI_C55_GAMMA_GAIN_B_MASK            GENMASK(11, 0)
>>>> +#define MALI_C55_REG_GAMMA_OFFSETS_1(offset) (0x1c070 + (offset))
>>>> +#define MALI_C55_GAMMA_OFFSET_R_MASK            GENMASK(11, 0)
>>>> +#define MALI_C55_GAMMA_OFFSET_G_MASK            GENMASK(27, 16)
>>>> +#define MALI_C55_REG_GAMMA_OFFSETS_2(offset) (0x1c074 + (offset))
>>>> +#define MALI_C55_GAMMA_OFFSET_B_MASK            GENMASK(11, 0)
>>>> +
>>>> +/* Output DMA Writer */
>>>> +
>>>> +#define MALI_C55_OUTPUT_DISABLED        0
>>>> +#define MALI_C55_OUTPUT_RGB32            1
>>>> +#define MALI_C55_OUTPUT_A2R10G10B10        2
>>>> +#define MALI_C55_OUTPUT_RGB565            3
>>>> +#define MALI_C55_OUTPUT_RGB24            4
>>>> +#define MALI_C55_OUTPUT_GEN32            5
>>>> +#define MALI_C55_OUTPUT_RAW16            6
>>>> +#define MALI_C55_OUTPUT_AYUV            8
>>>> +#define MALI_C55_OUTPUT_Y410            9
>>>> +#define MALI_C55_OUTPUT_YUY2            10
>>>> +#define MALI_C55_OUTPUT_UYVY            11
>>>> +#define MALI_C55_OUTPUT_Y210            12
>>>> +#define MALI_C55_OUTPUT_NV12_21            13
>>>> +#define MALI_C55_OUTPUT_YUV_420_422        17
>>>> +#define MALI_C55_OUTPUT_P210_P010        19
>>>> +#define MALI_C55_OUTPUT_YUV422            20
>>> I'd define this just below the MALI_C55_REG_UV_WRITER_MODE register
>>> macro.
>>>
>>>> +
>>>> +#define MALI_C55_OUTPUT_PLANE_ALT0        0
>>>> +#define MALI_C55_OUTPUT_PLANE_ALT1        1
>>>> +#define MALI_C55_OUTPUT_PLANE_ALT2        2
>>> Same here ?
>>>
>>>> +
>>>> +#endif /* _MALI_C55_REGISTERS_H */
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h 
>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>> new file mode 100644
>>>> index 000000000000..8edae87f1e5f
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>> @@ -0,0 +1,382 @@
>>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Resizer Coefficients
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#ifndef _MALI_C55_RESIZER_COEFS_H
>>>> +#define _MALI_C55_RESIZER_COEFS_H
>>>> +
>>>> +#include "mali-c55-common.h"
>>>> +
>>>> +#define MALI_C55_RESIZER_COEFS_NUM_BANKS    8
>>>> +#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES    64
>>> Do these belongs to mali-c55-registers.h ?
>>>
>>>> +
>>>> +static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
>>>> +    {    /* Bank 0 */
>>>> +        0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
>>>> +        0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
>>>> +        0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
>>>> +        0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
>>>> +        0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
>>>> +        0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
>>>> +        0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
>>>> +        0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
>>>> +        0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
>>>> +        0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
>>>> +        0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
>>>> +        0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
>>>> +        0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
>>>> +        0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
>>>> +        0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
>>>> +        0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
>>>> +    },
>>>> +    {    /* Bank 1 */
>>>> +        0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
>>>> +        0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
>>>> +        0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
>>>> +        0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
>>>> +        0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
>>>> +        0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
>>>> +        0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
>>>> +        0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
>>>> +        0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
>>>> +        0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
>>>> +        0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
>>>> +        0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
>>>> +        0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
>>>> +        0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
>>>> +        0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
>>>> +        0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
>>>> +    },
>>>> +    {    /* Bank 2 */
>>>> +        0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
>>>> +        0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
>>>> +        0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
>>>> +        0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
>>>> +        0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
>>>> +        0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
>>>> +        0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
>>>> +        0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
>>>> +        0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
>>>> +        0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
>>>> +        0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
>>>> +        0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
>>>> +        0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
>>>> +        0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
>>>> +        0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
>>>> +        0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
>>>> +    },
>>>> +    {    /* Bank 3 */
>>>> +        0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
>>>> +        0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
>>>> +        0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
>>>> +        0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
>>>> +        0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
>>>> +        0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
>>>> +        0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
>>>> +        0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
>>>> +        0x20100000, 0x00000010, 0x1f110000, 0x00000010,
>>>> +        0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
>>>> +        0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
>>>> +        0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
>>>> +        0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
>>>> +        0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
>>>> +        0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
>>>> +        0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
>>>> +    },
>>>> +    {    /* Bank 4 */
>>>> +        0x17090000, 0x00000917, 0x18090000, 0x00000916,
>>>> +        0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
>>>> +        0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
>>>> +        0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
>>>> +        0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
>>>> +        0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
>>>> +        0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
>>>> +        0x190f0300, 0x00000411, 0x18100300, 0x00000411,
>>>> +        0x1a100300, 0x00000310, 0x18110400, 0x00000310,
>>>> +        0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
>>>> +        0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
>>>> +        0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
>>>> +        0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
>>>> +        0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
>>>> +        0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
>>>> +        0x17160800, 0x0000010a, 0x18160900, 0x00000009,
>>>> +    },
>>>> +    {    /* Bank 5 */
>>>> +        0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
>>>> +        0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
>>>> +        0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
>>>> +        0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
>>>> +        0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
>>>> +        0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
>>>> +        0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
>>>> +        0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
>>>> +        0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
>>>> +        0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
>>>> +        0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
>>>> +        0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
>>>> +        0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
>>>> +        0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
>>>> +        0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
>>>> +        0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
>>>> +    },
>>>> +    {    /* Bank 6 */
>>>> +        0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
>>>> +        0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
>>>> +        0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
>>>> +        0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
>>>> +        0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>> +        0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
>>>> +        0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>> +        0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
>>>> +        0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
>>>> +        0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
>>>> +        0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>>> +        0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>>> +    },
>>>> +    {    /* Bank 7 */
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>> +    }
>>>> +};
>>>> +
>>>> +static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
>>>> +    {    /* Bank 0 */
>>>> +        0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
>>>> +        0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
>>>> +        0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
>>>> +        0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
>>>> +        0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
>>>> +        0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
>>>> +        0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
>>>> +        0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
>>>> +        0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
>>>> +        0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
>>>> +        0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
>>>> +        0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
>>>> +        0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
>>>> +        0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
>>>> +        0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
>>>> +        0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
>>>> +    },
>>>> +    {    /* Bank 1 */
>>>> +        0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
>>>> +        0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
>>>> +        0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
>>>> +        0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
>>>> +        0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
>>>> +        0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
>>>> +        0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
>>>> +        0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
>>>> +        0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
>>>> +        0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
>>>> +        0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
>>>> +        0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
>>>> +        0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
>>>> +        0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
>>>> +        0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
>>>> +        0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
>>>> +    },
>>>> +    {    /* Bank 2 */
>>>> +        0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
>>>> +        0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
>>>> +        0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
>>>> +        0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
>>>> +        0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
>>>> +        0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
>>>> +        0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
>>>> +        0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
>>>> +        0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
>>>> +        0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
>>>> +        0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
>>>> +        0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
>>>> +        0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
>>>> +        0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
>>>> +        0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
>>>> +        0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
>>>> +    },
>>>> +    {    /* Bank 3 */
>>>> +        0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
>>>> +        0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
>>>> +        0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
>>>> +        0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
>>>> +        0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
>>>> +        0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
>>>> +        0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
>>>> +        0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
>>>> +        0x20100000, 0x00000010, 0x1f100000, 0x00000011,
>>>> +        0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
>>>> +        0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
>>>> +        0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
>>>> +        0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
>>>> +        0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
>>>> +        0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
>>>> +        0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
>>>> +    },
>>>> +    {    /* Bank 4 */
>>>> +        0x17170900, 0x00000009, 0x18160900, 0x00000009,
>>>> +        0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
>>>> +        0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
>>>> +        0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
>>>> +        0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
>>>> +        0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
>>>> +        0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
>>>> +        0x19110400, 0x0000030f, 0x18110400, 0x00000310,
>>>> +        0x1a100300, 0x00000310, 0x18100300, 0x00000411,
>>>> +        0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
>>>> +        0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
>>>> +        0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
>>>> +        0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
>>>> +        0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
>>>> +        0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
>>>> +        0x170a0100, 0x00000816, 0x18090000, 0x00000916,
>>>> +    },
>>>> +    {    /* Bank 5 */
>>>> +        0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
>>>> +        0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
>>>> +        0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
>>>> +        0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
>>>> +        0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
>>>> +        0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
>>>> +        0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
>>>> +        0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
>>>> +        0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
>>>> +        0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
>>>> +        0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
>>>> +        0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
>>>> +        0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
>>>> +        0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
>>>> +        0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
>>>> +        0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
>>>> +    },
>>>> +    {    /* Bank 6 */
>>>> +        0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
>>>> +        0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>>> +        0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
>>>> +        0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
>>>> +        0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>> +        0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>>> +        0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>> +        0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
>>>> +        0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
>>>> +        0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
>>>> +        0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
>>>> +        0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
>>>> +    },
>>>> +    {    /* Bank 7 */
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>> +    }
>>>> +};
>>>> +
>>>> +struct mali_c55_resizer_coef_bank {
>>>> +    unsigned int bank;
>>> This is always equal to the index of the entry in the
>>> mali_c55_coefficient_banks array, you can drop it.
>>>
>>>> +    unsigned int top;
>>>> +    unsigned int bottom;
>>> The bottom value of bank N is always equal to the top value of bank N+1.
>>> You can simplify this by storing a single value.
>>>
>>>> +};
>>>> +
>>>> +static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
>>>> +    {
>>>> +        .bank = 0,
>>>> +        .top = 1000,
>>>> +        .bottom = 770,
>>>> +    },
>>>> +    {
>>>> +        .bank = 1,
>>>> +        .top = 769,
>>>> +        .bottom = 600,
>>>> +    },
>>>> +    {
>>>> +        .bank = 2,
>>>> +        .top = 599,
>>>> +        .bottom = 460,
>>>> +    },
>>>> +    {
>>>> +        .bank = 3,
>>>> +        .top = 459,
>>>> +        .bottom = 354,
>>>> +    },
>>>> +    {
>>>> +        .bank = 4,
>>>> +        .top = 353,
>>>> +        .bottom = 273,
>>>> +    },
>>>> +    {
>>>> +        .bank = 5,
>>>> +        .top = 272,
>>>> +        .bottom = 210,
>>>> +    },
>>>> +    {
>>>> +        .bank = 6,
>>>> +        .top = 209,
>>>> +        .bottom = 162,
>>>> +    },
>>>> +    {
>>>> +        .bank = 7,
>>>> +        .top = 161,
>>>> +        .bottom = 125,
>>>> +    },
>>>> +};
>>>> +
>>> A small comment would be nice, such as
>>>
>>> /* Select a bank of resizer coefficients, based on the scaling ratio. */
>>>
>>>> +static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,
>>> This function is related to the resizers. Add "rsz" somewhere in the
>>> function name, and pass a resizer pointer.
>>>
>>>> +                        unsigned int crop,
>>>> +                        unsigned int scale)
>>> I think those are the input and output sizes to the scaler. Rename them
>>> to make it clearer.
>>>
>>>> +{
>>>> +    unsigned int tmp;
>>> tmp is almost always a bad variable name. Please use a more descriptive
>>> name, size as rsz_ratio.
>>>
>>>> +    unsigned int i;
>>>> +
>>>> +    tmp = (scale * 1000U) / crop;
>>>> +
>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
>>>> +        if (tmp >= mali_c55_coefficient_banks[i].bottom &&
>>>> +            tmp <= mali_c55_coefficient_banks[i].top)
>>>> +            return mali_c55_coefficient_banks[i].bank;
>>>> +    }
>>>> +
>>>> +    /*
>>>> +     * We shouldn't ever get here, in theory. As we have no good choices
>>>> +     * simply warn the user and use the first bank of coefficients.
>>>> +     */
>>>> +    dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
>>>> +    return 0;
>>>> +}
>>> And everything else belongs to mali-c55-resizer.c. Drop this header
>>> file.
>>>
>>>> +
>>>> +#endif /* _MALI_C55_RESIZER_COEFS_H */
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c 
>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>> new file mode 100644
>>>> index 000000000000..0a5a2969d3ce
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>> @@ -0,0 +1,779 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Image signal processor
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#include <linux/math.h>
>>>> +#include <linux/minmax.h>
>>>> +
>>>> +#include <media/media-entity.h>
>>>> +#include <media/v4l2-subdev.h>
>>>> +
>>>> +#include "mali-c55-common.h"
>>>> +#include "mali-c55-registers.h"
>>>> +#include "mali-c55-resizer-coefs.h"
>>>> +
>>>> +/* Scaling factor in Q4.20 format. */
>>>> +#define MALI_C55_RZR_SCALER_FACTOR    (1U << 20)
>>>> +
>>>> +static const u32 rzr_non_bypass_src_fmts[] = {
>>>> +    MEDIA_BUS_FMT_RGB121212_1X36,
>>>> +    MEDIA_BUS_FMT_YUV10_1X30
>>>> +};
>>>> +
>>>> +static const char * const mali_c55_resizer_names[] = {
>>>> +    [MALI_C55_RZR_FR] = "resizer fr",
>>>> +    [MALI_C55_RZR_DS] = "resizer ds",
>>>> +};
>>>> +
>>>> +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
>>>> +                     struct v4l2_subdev_state *state)
>>>> +{
>>>> +    unsigned int reg_offset = rzr->cap_dev->reg_offset;
>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>> +    struct v4l2_mbus_framefmt *fmt;
>>>> +    struct v4l2_rect *crop;
>> const
>>
>>>> +
>>>> +    /* Verify if crop should be enabled. */
>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
>>>> +
>>>> +    if (fmt->width == crop->width && fmt->height == crop->height)
>>>> +        return MALI_C55_BYPASS_CROP;
>>>> +
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
>>>> +               crop->left);
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
>>>> +               crop->top);
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
>>>> +               crop->width);
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
>>>> +               crop->height);
>>>> +
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
>>>> +               MALI_C55_CROP_ENABLE);
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
>>>> +                    struct v4l2_subdev_state *state)
>>>> +{
>>>> +    unsigned int reg_offset = rzr->cap_dev->reg_offset;
>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>> +    struct v4l2_rect *crop, *scale;
>> const
>>
>> Once "[PATCH v4 0/3] media: v4l2-subdev: Support const-awareness in
>> state accessors" gets merged, the state argument to this function can be
>> made const too. Same for other functions, as applicable.
>>
>>>> +    unsigned int h_bank, v_bank;
>>>> +    u64 h_scale, v_scale;
>>>> +
>>>> +    /* Verify if scaling should be enabled. */
>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
>>>> +    scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
>>>> +
>>>> +    if (crop->width == scale->width && crop->height == scale->height)
>>>> +        return MALI_C55_BYPASS_SCALER;
>>>> +
>>>> +    /* Program the V/H scaling factor in Q4.20 format. */
>>>> +    h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
>>>> +    v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
>>>> +
>>>> +    do_div(h_scale, scale->width);
>>>> +    do_div(v_scale, scale->height);
>>>> +
>>>> +    mali_c55_write(mali_c55,
>>>> +               MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
>>>> +               crop->width);
>>>> +    mali_c55_write(mali_c55,
>>>> +               MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
>>>> +               crop->height);
>>>> +
>>>> +    mali_c55_write(mali_c55,
>>>> +               MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
>>>> +               scale->width);
>>>> +    mali_c55_write(mali_c55,
>>>> +               MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
>>>> +               scale->height);
>>>> +
>>>> +    mali_c55_write(mali_c55,
>>>> +               MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
>>>> +               h_scale);
>>>> +    mali_c55_write(mali_c55,
>>>> +               MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
>>>> +               v_scale);
>>>> +
>>>> +    h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
>>>> +                         scale->width);
>>>> +    mali_c55_write(mali_c55,
>>>> +               MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
>>>> +               h_bank);
>>>> +
>>>> +    v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
>>>> +                         scale->height);
>>>> +    mali_c55_write(mali_c55,
>>>> +               MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
>>>> +               v_bank);
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
>>>> +                 struct v4l2_subdev_state *state)
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>> +    u32 bypass = 0;
>>>> +
>>>> +    /* Verify if cropping and scaling should be enabled. */
>>>> +    bypass |= mali_c55_rzr_program_crop(rzr, state);
>>>> +    bypass |= mali_c55_rzr_program_resizer(rzr, state);
>>>> +
>>>> +    mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
>>>> +                 MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
>>>> +                 MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
>>>> +                 bypass);
>>>> +}
>>>> +
>>>> +/*
>>>> + * Inspect the routing table to know which of the two (mutually exclusive)
>>>> + * routes is enabled and return the sink pad id of the active route.
>>>> + */
>>>> +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
>>>> +{
>>>> +    struct v4l2_subdev_krouting *routing = &state->routing;
>>>> +    struct v4l2_subdev_route *route;
>>>> +
>>>> +    /* A single route is enabled at a time. */
>>>> +    for_each_active_route(routing, route)
>>>> +        return route->sink_pad;
>>>> +
>>>> +    return MALI_C55_RZR_SINK_PAD;
>>>> +}
>>>> +
>>>> +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
>>>> +{
>>>> +    u32 corrected_code = 0;
>>>> +
>>>> +    /*
>>>> +     * The ISP takes input in a 20-bit format, but can only output 16-bit
>>>> +     * RAW bayer data (with the 4 least significant bits from the input
>>>> +     * being lost). Return the 16-bit version of the 20-bit input formats.
>>>> +     */
>>>> +    switch (mbus_code) {
>>>> +    case MEDIA_BUS_FMT_SBGGR20_1X20:
>>>> +        corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
>>>> +        break;
>>>> +    case MEDIA_BUS_FMT_SGBRG20_1X20:
>>>> +        corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
>>>> +        break;
>>>> +    case MEDIA_BUS_FMT_SGRBG20_1X20:
>>>> +        corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
>>>> +        break;
>>>> +    case MEDIA_BUS_FMT_SRGGB20_1X20:
>>>> +        corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
>>>> +        break;
>> Would it make sense to add the shifted code to mali_c55_isp_fmt ?
>>
>>>> +    }
>>>> +
>>>> +    return corrected_code;
>>>> +}
>>>> +
>>>> +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
>>>> +                      struct v4l2_subdev_state *state,
>>>> +                      struct v4l2_subdev_krouting *routing)
>> I think the last argument can be const.
>>
>>>> +{
>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>> +                            sd);
>> A to_mali_c55_resizer() static inline function would be useful. Same for
>> other components, where applicable.
>>
>>>> +    unsigned int active_sink = UINT_MAX;
>>>> +    struct v4l2_mbus_framefmt *src_fmt;
>>>> +    struct v4l2_rect *crop, *compose;
>>>> +    struct v4l2_subdev_route *route;
>>>> +    unsigned int active_routes = 0;
>>>> +    struct v4l2_mbus_framefmt *fmt;
>>>> +    int ret;
>>>> +
>>>> +    ret = v4l2_subdev_routing_validate(sd, routing, 0);
>>>> +    if (ret)
>>>> +        return ret;
>>>> +
>>>> +    /* Only a single route can be enabled at a time. */
>>>> +    for_each_active_route(routing, route) {
>>>> +        if (++active_routes > 1) {
>>>> +            dev_err(rzr->mali_c55->dev,
>>>> +                "Only one route can be active");
>> No kernel log message with a level higher than dev_dbg() from
>> user-controlled paths please, here and where applicable. This is to
>> avoid giving applications an easy way to flood the kernel log.
>>
>>>> +            return -EINVAL;
>>>> +        }
>>>> +
>>>> +        active_sink = route->sink_pad;
>>>> +    }
>>>> +    if (active_sink == UINT_MAX) {
>>>> +        dev_err(rzr->mali_c55->dev, "One route has to be active");
>>>> +        return -EINVAL;
>>>> +    }
>> The recommended handling of invalid routing is to adjust the routing
>> table, not to return errors.
>>
>>>> +
>>>> +    ret = v4l2_subdev_set_routing(sd, state, routing);
>>>> +    if (ret) {
>>>> +        dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
>>>> +        return ret;
>>>> +    }
>>>> +
>>>> +    fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
>>>> +    crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
>>>> +    compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
>>>> +
>>>> +    fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>> +    fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>> +    fmt->colorspace = V4L2_COLORSPACE_SRGB;
>> There are other colorspace-related fields.
>>
>>>> +    fmt->field = V4L2_FIELD_NONE;
>> I wonder if we should really update the sink pad format, or just
>> propagate it. If we update it, I think it should be set to defaults on
>> both sink pads, not just the active sink pad.
>>
>>>> +
>>>> +    if (active_sink == MALI_C55_RZR_SINK_PAD) {
>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>> +
>>>> +        crop->left = crop->top = 0;
>>         crop->left = 0;
>>         crop->top = 0;
>>
>>>> +        crop->width = MALI_C55_DEFAULT_WIDTH;
>>>> +        crop->height = MALI_C55_DEFAULT_HEIGHT;
>>>> +
>>>> +        *compose = *crop;
>>>> +    } else {
>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>> +    }
>>>> +
>>>> +    /* Propagate the format to the source pad */
>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
>>>> +                           0);
>>>> +    *src_fmt = *fmt;
>>>> +
>>>> +    /* In the event this is the bypass pad the mbus code needs correcting */
>>>> +    if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
>>>> +        src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
>>>> +                       struct v4l2_subdev_state *state,
>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
>>>> +{
>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
>>>> +    const struct mali_c55_isp_fmt *fmt;
>>>> +    unsigned int index = 0;
>>>> +    u32 sink_pad;
>>>> +
>>>> +    switch (code->pad) {
>>>> +    case MALI_C55_RZR_SINK_PAD:
>>>> +        if (code->index)
>>>> +            return -EINVAL;
>>>> +
>>>> +        code->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>> +
>>>> +        return 0;
>>>> +    case MALI_C55_RZR_SOURCE_PAD:
>>>> +        sink_pad = mali_c55_rzr_get_active_sink(state);
>>>> +        sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
>>>> +
>>>> +        /*
>>>> +         * If the active route is from the Bypass sink pad, then the
>>>> +         * source pad is a simple passthrough of the sink format,
>>>> +         * downshifted to 16-bits.
>>>> +         */
>>>> +
>>>> +        if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>>> +            if (code->index)
>>>> +                return -EINVAL;
>>>> +
>>>> +            code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
>>>> +            if (!code->code)
>>>> +                return -EINVAL;
>>>> +
>>>> +            return 0;
>>>> +        }
>>>> +
>>>> +        /*
>>>> +         * If the active route is from the non-bypass sink then we can
>>>> +         * select either RGB or conversion to YUV.
>>>> +         */
>>>> +
>>>> +        if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
>>>> +            return -EINVAL;
>>>> +
>>>> +        code->code = rzr_non_bypass_src_fmts[code->index];
>>>> +
>>>> +        return 0;
>>>> +    case MALI_C55_RZR_SINK_BYPASS_PAD:
>>>> +        for_each_mali_isp_fmt(fmt) {
>>>> +            if (index++ == code->index) {
>>>> +                code->code = fmt->code;
>>>> +                return 0;
>>>> +            }
>>>> +        }
>>>> +
>>>> +        break;
>>>> +    }
>>>> +
>>>> +    return -EINVAL;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
>>>> +                    struct v4l2_subdev_state *state,
>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
>>>> +{
>>>> +    if (fse->index)
>>>> +        return -EINVAL;
>>>> +
>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
>>>> +                     struct v4l2_subdev_state *state,
>>>> +                     struct v4l2_subdev_format *format)
>>>> +{
>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
>>>> +    struct v4l2_rect *rect;
>>>> +    unsigned int sink_pad;
>>>> +
>>>> +    /*
>>>> +     * Clamp to min/max and then reset crop and compose rectangles to the
>>>> +     * newly applied size.
>>>> +     */
>>>> +    clamp_t(unsigned int, fmt->width,
>>>> +        MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>>> +    clamp_t(unsigned int, fmt->height,
>>>> +        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>> Please check comments for other components related to the colorspace
>> fields, to decide how to handle them here.
>>
>>>> +
>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
>>>> +    if (sink_pad == MALI_C55_RZR_SINK_PAD) {
>> The selection here should depend on format->pad, not the active sink
>> pad.
>>
>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>> +
>>>> +        rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
>>>> +        rect->left = 0;
>>>> +        rect->top = 0;
>>>> +        rect->width = fmt->width;
>>>> +        rect->height = fmt->height;
>>>> +
>>>> +        rect = v4l2_subdev_state_get_compose(state,
>>>> +                             MALI_C55_RZR_SINK_PAD);
>>>> +        rect->left = 0;
>>>> +        rect->top = 0;
>>>> +        rect->width = fmt->width;
>>>> +        rect->height = fmt->height;
>>>> +    } else {
>>>> +        /*
>>>> +         * Make sure the media bus code is one of the supported
>>>> +         * ISP input media bus codes.
>>>> +         */
>>>> +        if (!mali_c55_isp_is_format_supported(fmt->code))
>>>> +            fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
>>>> +    }
>>>> +
>>>> +    *v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
>> Propagation to the source pad, however, should depend on the active
>> route. If format->pad is routed to the source pad, you should propagate,
>> otherwise, you shouldn't.
>>
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
>>>> +                       struct v4l2_subdev_state *state,
>>>> +                       struct v4l2_subdev_format *format)
>>>> +{
>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>> +                            sd);
>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
>>>> +    struct v4l2_rect *crop, *compose;
>>>> +    unsigned int sink_pad;
>>>> +    unsigned int i;
>>>> +
>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
>>>> +    sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
>>>> +    crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
>>>> +    compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
>>>> +
>>>> +    /* FR Bypass pipe. */
>>>> +
>>>> +    if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>>> +        /*
>>>> +         * Format on the source pad is the same as the one on the
>>>> +         * sink pad, downshifted to 16-bits.
>>>> +         */
>>>> +        fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
>>>> +        if (!fmt->code)
>>>> +            return -EINVAL;
>>>> +
>>>> +        /* RAW bypass disables scaling and cropping. */
>>>> +        crop->top = compose->top = 0;
>>>> +        crop->left = compose->left = 0;
>>>> +        fmt->width = crop->width = compose->width = sink_fmt->width;
>>>> +        fmt->height = crop->height = compose->height = sink_fmt->height;
>> I don't think this is right. This function sets the format on the source
>> pad. Subdevs should propagate formats from the sink to the source, not
>> the other way around.
>>
>> The only parameter that can be modified on the source pad (as far as I
>> understand) is the media bus code. In the bypass path, I understand it's
>> fixed, while in the other path, you can select between RGB and YUV. I
>> think the following code is what you need to implement this function.
>>
>> static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
>>                        struct v4l2_subdev_state *state,
>>                        struct v4l2_subdev_format *format)
>> {
>>     struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>                             sd);
>>     struct v4l2_mbus_framefmt *fmt;
>>
>>     fmt = v4l2_subdev_state_get_format(state, format->pad);
>>
>>     /* In the non-bypass path the output format can be selected. */
>>     if (mali_c55_rzr_get_active_sink(state) == MALI_C55_RZR_SINK_PAD) {
>>         unsigned int i;
>>
>>         fmt->code = format->format.code;
>>
>>         for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
>>             if (fmt->code == rzr_non_bypass_src_fmts[i])
>>                 break;
>>         }
>>
>>         if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts))
>>             fmt->code = rzr_non_bypass_src_fmts[0];
>>     }
>>
>>     format->format = *fmt;
>>
>>     return 0;
>> }
>>
>>>> +
>>>> +        *v4l2_subdev_state_get_format(state,
>>>> +                          MALI_C55_RZR_SOURCE_PAD) = *fmt;
>>>> +
>>>> +        return 0;
>>>> +    }
>>>> +
>>>> +    /* Regular processing pipe. */
>>>> +
>>>> +    for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
>>>> +        if (fmt->code == rzr_non_bypass_src_fmts[i])
>>>> +            break;
>>>> +    }
>>>> +
>>>> +    if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
>>>> +        dev_dbg(rzr->mali_c55->dev,
>>>> +            "Unsupported mbus code 0x%x: using default\n",
>>>> +            fmt->code);
>> I think you can drop this message.
>>
>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>> +    }
>>>> +
>>>> +    /*
>>>> +     * The source pad format size comes directly from the sink pad
>>>> +     * compose rectangle.
>>>> +     */
>>>> +    fmt->width = compose->width;
>>>> +    fmt->height = compose->height;
>>>> +
>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
>>>> +                struct v4l2_subdev_state *state,
>>>> +                struct v4l2_subdev_format *format)
>>>> +{
>>>> +    /*
>>>> +     * On sink pads fmt is either fixed for the 'regular' processing
>>>> +     * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
>>>> +     * pad.
>>>> +     *
>>>> +     * On source pad sizes are the result of crop+compose on the sink
>>>> +     * pad sizes, while the format depends on the active route.
>>>> +     */
>>>> +
>>>> +    if (format->pad != MALI_C55_RZR_SOURCE_PAD)
>>>> +        return mali_c55_rzr_set_sink_fmt(sd, state, format);
>>>> +
>>>> +    return mali_c55_rzr_set_source_fmt(sd, state, format);
>> Nitpicking,
>>
>>     if (format->pad == MALI_C55_RZR_SOURCE_PAD)
>>         return mali_c55_rzr_set_source_fmt(sd, state, format);
>>
>>     return mali_c55_rzr_set_sink_fmt(sd, state, format);
>>
>> to match SOURCE_PAD and source_fmt.
>>
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
>>>> +                      struct v4l2_subdev_state *state,
>>>> +                      struct v4l2_subdev_selection *sel)
>>>> +{
>>>> +    if (sel->pad != MALI_C55_RZR_SINK_PAD)
>>>> +        return -EINVAL;
>>>> +
>>>> +    if (sel->target != V4L2_SEL_TGT_CROP &&
>>>> +        sel->target != V4L2_SEL_TGT_COMPOSE)
>>>> +        return -EINVAL;
>>>> +
>>>> +    sel->r = sel->target == V4L2_SEL_TGT_CROP
>>>> +           ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
>>>> +           : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
>>>> +                      struct v4l2_subdev_state *state,
>>>> +                      struct v4l2_subdev_selection *sel)
>>>> +{
>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>> +                            sd);
>>>> +    struct v4l2_mbus_framefmt *source_fmt;
>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
>>>> +    struct v4l2_rect *crop, *compose;
>>>> +
>>>> +    if (sel->pad != MALI_C55_RZR_SINK_PAD)
>>>> +        return -EINVAL;
>>>> +
>>>> +    if (sel->target != V4L2_SEL_TGT_CROP &&
>>>> +        sel->target != V4L2_SEL_TGT_COMPOSE)
>>>> +        return -EINVAL;
>>>> +
>>>> +    source_fmt = v4l2_subdev_state_get_format(state,
>>>> +                          MALI_C55_RZR_SOURCE_PAD);
>>>> +    sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
>>>> +    compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
>>>> +
>>>> +    /* RAW bypass disables crop/scaling. */
>>>> +    if (mali_c55_format_is_raw(source_fmt->code)) {
>>>> +        crop->top = compose->top = 0;
>>>> +        crop->left = compose->left = 0;
>>>> +        crop->width = compose->width = sink_fmt->width;
>>>> +        crop->height = compose->height = sink_fmt->height;
>>>> +
>>>> +        sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
>>>> +
>>>> +        return 0;
>>>> +    }
>>>> +
>>>> +    /* During streaming, it is allowed to only change the crop rectangle. */
>>>> +    if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
>>>> +        return -EINVAL;
>>>> +
>>>> +     /*
>>>> +      * Update the desired target and then clamp the crop rectangle to the
>>>> +      * sink format sizes and the compose size to the crop sizes.
>>>> +      */
>>>> +    if (sel->target == V4L2_SEL_TGT_CROP)
>>>> +        *crop = sel->r;
>>>> +    else
>>>> +        *compose = sel->r;
>>>> +
>>>> +    clamp_t(unsigned int, crop->left, 0, sink_fmt->width);
>>>> +    clamp_t(unsigned int, crop->top, 0, sink_fmt->height);
>>>> +    clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
>>>> +        sink_fmt->width - crop->left);
>>>> +    clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
>>>> +        sink_fmt->height - crop->top);
>>>> +
>>>> +    if (rzr->streaming) {
>>>> +        /*
>>>> +         * Apply at runtime a crop rectangle on the resizer's sink only
>>>> +         * if it doesn't require re-programming the scaler output sizes
>>>> +         * as it would require changing the output buffer sizes as well.
>>>> +         */
>>>> +        if (sel->r.width < compose->width ||
>>>> +            sel->r.height < compose->height)
>>>> +            return -EINVAL;
>>>> +
>>>> +        *crop = sel->r;
>>>> +        mali_c55_rzr_program(rzr, state);
>>>> +
>>>> +        return 0;
>>>> +    }
>>>> +
>>>> +    compose->left = 0;
>>>> +    compose->top = 0;
>>>> +    clamp_t(unsigned int, compose->left, 0, sink_fmt->width);
>>>> +    clamp_t(unsigned int, compose->top, 0, sink_fmt->height);
>>>> +    clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
>>>> +    clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
>>>> +
>>>> +    sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
>>>> +                    struct v4l2_subdev_state *state,
>>>> +                    enum v4l2_subdev_format_whence which,
>>>> +                    struct v4l2_subdev_krouting *routing)
>>>> +{
>>>> +    if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
>>>> +        media_entity_is_streaming(&sd->entity))
>>>> +        return -EBUSY;
>>>> +
>>>> +    return __mali_c55_rzr_set_routing(sd, state, routing);
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
>>>> +    .enum_mbus_code        = mali_c55_rzr_enum_mbus_code,
>>>> +    .enum_frame_size    = mali_c55_rzr_enum_frame_size,
>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
>>>> +    .set_fmt        = mali_c55_rzr_set_fmt,
>>>> +    .get_selection        = mali_c55_rzr_get_selection,
>>>> +    .set_selection        = mali_c55_rzr_set_selection,
>>>> +    .set_routing        = mali_c55_rzr_set_routing,
>>>> +};
>>>> +
>>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
>> Could this be handled through the .enable_streams() and
>> .disable_streams() operations ? They ensure that the stream state stored
>> internal is correct. That may not matter much today, but I think it will
>> become increasingly important in the future for the V4L2 core.
>>
>>>> +{
>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>> +    struct v4l2_subdev *sd = &rzr->sd;
>>>> +    struct v4l2_subdev_state *state;
>>>> +    unsigned int sink_pad;
>>>> +
>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
>>>> +
>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
>>>> +    if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>>> +        /* Bypass FR pipe processing if the bypass route is active. */
>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>>> + MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
>>>> + MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
>>>> +        goto unlock_state;
>>>> +    }
>>>> +
>>>> +    /* Disable bypass and use regular processing. */
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>>> +                 MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
>>>> +    mali_c55_rzr_program(rzr, state);
>>>> +
>>>> +unlock_state:
>>>> +    rzr->streaming = true;
>> And hopefully you'll be able to replace this with
>> v4l2_subdev_is_streaming(), introduced in "[PATCH v6 00/11] media:
>> subdev: Improve stream enable/disable machinery" (Sakari has sent a pull
>> request for v6.11 yesterday).
>>
>>>> +    v4l2_subdev_unlock_state(state);
>>>> +}
>>>> +
>>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
>>>> +{
>>>> +    struct v4l2_subdev *sd = &rzr->sd;
>>>> +    struct v4l2_subdev_state *state;
>>>> +
>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
>>>> +    rzr->streaming = false;
>>>> +    v4l2_subdev_unlock_state(state);
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
>>>> +    .pad    = &mali_c55_resizer_pad_ops,
>>>> +};
>>>> +
>>>> +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
>>>> +                   struct v4l2_subdev_state *state)
>>>> +{
>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>> +                            sd);
>>>> +    struct v4l2_subdev_krouting routing = { };
>>>> +    struct v4l2_subdev_route *routes;
>>>> +    unsigned int i;
>>>> +    int ret;
>>>> +
>>>> +    routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
>>>> +    if (!routes)
>>>> +        return -ENOMEM;
>>>> +
>>>> +    for (i = 0; i < rzr->num_routes; ++i) {
>>>> +        struct v4l2_subdev_route *route = &routes[i];
>>>> +
>>>> +        route->sink_pad = i
>>>> +                ? MALI_C55_RZR_SINK_BYPASS_PAD
>>>> +                : MALI_C55_RZR_SINK_PAD;
>>>> +        route->source_pad = MALI_C55_RZR_SOURCE_PAD;
>>>> +        if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
>>>> +            route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
>>>> +    }
>>>> +
>>>> +    routing.num_routes = rzr->num_routes;
>>>> +    routing.routes = routes;
>>>> +
>>>> +    ret = __mali_c55_rzr_set_routing(sd, state, &routing);
>>>> +    kfree(routes);
>>>> +
>>>> +    return ret;
>> I think this could be simplified.
>>
>>     struct v4l2_subdev_route routes[2] = {
>>         {
>>             .sink_pad = MALI_C55_RZR_SINK_PAD,
>>             .source_pad = MALI_C55_RZR_SOURCE_PAD,
>>             .flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
>>         }, {
>>             .sink_pad = MALI_C55_RZR_SINK_BYPASS_PAD,
>>             .source_pad = MALI_C55_RZR_SOURCE_PAD,
>>         },
>>     };
>>     struct v4l2_subdev_krouting routing = {
>>         .num_routes = rzr->num_routes,
>>         .routes = routes,
>>     };
>>
>>     return __mali_c55_rzr_set_routing(sd, state, &routing);
>>
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
>>>> +    .init_state = mali_c55_rzr_init_state,
>>>> +};
>>>> +
>>>> +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
>>>> +                          unsigned int index)
>>>> +{
>>>> +    const unsigned int scaler_filt_coefmem_addrs[][2] = {
>>>> +        [MALI_C55_RZR_FR] = {
>>>> +            0x034A8, /* hfilt */
>>>> +            0x044A8  /* vfilt */
>>> Lowercase hex constants.
>> And addresses belong to the mali-c55-registers.h file.
>>
>>>> +        },
>>>> +        [MALI_C55_RZR_DS] = {
>>>> +            0x014A8, /* hfilt */
>>>> +            0x024A8  /* vfilt */
>>>> +        },
>>>> +    };
>>>> +    unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
>>>> +    unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
>>>> +    unsigned int i, j;
>>>> +
>>>> +    for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
>>>> +        for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
>>>> +            mali_c55_write(mali_c55, haddr,
>>>> + mali_c55_scaler_h_filter_coefficients[i][j]);
>>>> +            mali_c55_write(mali_c55, vaddr,
>>>> + mali_c55_scaler_v_filter_coefficients[i][j]);
>>>> +
>>>> +            haddr += sizeof(u32);
>>>> +            vaddr += sizeof(u32);
>>>> +        }
>>>> +    }
>> How about memcpy_toio() ? I suppose this function isn't
>> performance sensitive, so maybe usage of mali_c55_write() is better from
>> a consistency point of view.
>>
>>>> +}
>>>> +
>>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    unsigned int i;
>>>> +    int ret;
>>>> +
>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
>> Moving the inner content to a separate mali_c55_register_resizer()
>> function would increase readability I think, and remove usage of gotos.
>> I would probably do the same for unregistration too, for consistency.
>>
>>>> +        struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
>>>> +        struct v4l2_subdev *sd = &rzr->sd;
>>>> +        unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
>>>> +
>>>> +        rzr->id = i;
>>>> +        rzr->streaming = false;
>>>> +
>>>> +        if (rzr->id == MALI_C55_RZR_FR)
>>>> +            rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
>>>> +        else
>>>> +            rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
>>>> +
>>>> +        mali_c55_resizer_program_coefficients(mali_c55, i);
>> Should this be done at stream start, given that power may be cut off
>> between streaming sessions ?
>>
>>>> +
>>>> +        v4l2_subdev_init(sd, &mali_c55_resizer_ops);
>>>> +        sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
>>>> +                 | V4L2_SUBDEV_FL_STREAMS;
>>>> +        sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
>>>> +        sd->internal_ops = &mali_c55_resizer_internal_ops;
>>>> +        snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
>>         snprintf(sd->name, ARRAY_SIZE(sd->name), "%s resizer %s",
>>
>> and drop the "resizer " prefix from mali_c55_resizer_names. You can also
>> make mali_c55_resizer_names a local static const variable.
>>
>>>> +             MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
>>>> +
>>>> +        rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
>>>> +        rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
>>>> +
>>>> +        /* Only the FR pipe has a bypass pad. */
>>>> +        if (rzr->id == MALI_C55_RZR_FR) {
>>>> + rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
>>>> +                            MEDIA_PAD_FL_SINK;
>>>> +            rzr->num_routes = 2;
>>>> +        } else {
>>>> +            num_pads -= 1;
>>>> +            rzr->num_routes = 1;
>>>> +        }
>>>> +
>>>> +        ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
>>>> +        if (ret)
>>>> +            return ret;
>>>> +
>>>> +        ret = v4l2_subdev_init_finalize(sd);
>>>> +        if (ret)
>>>> +            goto err_cleanup;
>>>> +
>>>> +        ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>>> +        if (ret)
>>>> +            goto err_cleanup;
>>>> +
>>>> +        rzr->mali_c55 = mali_c55;
>>>> +    }
>>>> +
>>>> +    return 0;
>>>> +
>>>> +err_cleanup:
>>>> +    for (; i >= 0; --i) {
>>>> +        struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
>>>> +        struct v4l2_subdev *sd = &rzr->sd;
>>>> +
>>>> +        v4l2_subdev_cleanup(sd);
>>>> +        media_entity_cleanup(&sd->entity);
>>>> +    }
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    unsigned int i;
>>>> +
>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
>>>> +        struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
>>>> +
>>>> +        if (!resizer->mali_c55)
>>>> +            continue;
>>>> +
>>>> +        v4l2_device_unregister_subdev(&resizer->sd);
>>>> +        v4l2_subdev_cleanup(&resizer->sd);
>>>> +        media_entity_cleanup(&resizer->sd.entity);
>>>> +    }
>>>> +}
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c 
>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>>> new file mode 100644
>>>> index 000000000000..c7e699741c6d
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>>> @@ -0,0 +1,402 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - Test pattern generator
>>>> + *
>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#include <linux/minmax.h>
>>>> +#include <linux/string.h>
>>>> +
>>>> +#include <media/media-entity.h>
>>>> +#include <media/v4l2-ctrls.h>
>>>> +#include <media/v4l2-subdev.h>
>>>> +
>>>> +#include "mali-c55-common.h"
>>>> +#include "mali-c55-registers.h"
>>>> +
>>>> +#define MALI_C55_TPG_SRC_PAD        0
>>>> +#define MALI_C55_TPG_FIXED_HBLANK    0x20
>>>> +#define MALI_C55_TPG_MAX_VBLANK        0xFFFF
>>> Lowercase hex constants.
>>>
>>>> +#define MALI_C55_TPG_PIXEL_RATE        100000000
>>> This should be exposed to applications using the V4L2_CID_PIXEL_RATE
>>> control (read-only).
>>>
>>>> +
>>>> +static const char * const mali_c55_tpg_test_pattern_menu[] = {
>>>> +    "Flat field",
>>>> +    "Horizontal gradient",
>>>> +    "Vertical gradient",
>>>> +    "Vertical bars",
>>>> +    "Arbitrary rectangle",
>>>> +    "White frame on black field"
>>>> +};
>>>> +
>>>> +static const u32 mali_c55_tpg_mbus_codes[] = {
>>>> +    MEDIA_BUS_FMT_SRGGB20_1X20,
>>>> +    MEDIA_BUS_FMT_RGB202020_1X60,
>>>> +};
>>>> +
>>>> +static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
>>>> +                       int *def_vblank, int *min_vblank)
>>> unsigned int ?
>>>
>>>> +{
>>>> +    unsigned int hts;
>>>> +    int tgt_fps;
>>>> +    int vblank;
>>>> +
>>>> +    hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
>>>> +
>>>> +    /*
>>>> +     * The ISP has minimum vertical blanking requirements that must be
>>>> +     * adhered to by the TPG. The minimum is a function of the Iridix blocks
>>>> +     * clocking requirements and the width of the image and horizontal
>>>> +     * blanking, but if we assume the worst case iVariance and sVariance
>>>> +     * values then it boils down to the below.
>>>> +     */
>>>> +    *min_vblank = 15 + (120500 / hts);
>>> I wonder if this should round up.
>>>
>>>> +
>>>> +    /*
>>>> +     * We need to set a sensible default vblank for whatever format height
>>>> +     * we happen to be given from set_fmt(). This function just targets
>>>> +     * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
>>>> +     * If we can't get 5fps we'll take whatever the minimum vblank gives us.
>>>> +     */
>>>> +    tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
>>>> +
>>>> +    if (tgt_fps < 5)
>>>> +        vblank = *min_vblank;
>>>> +    else
>>>> +        vblank = MALI_C55_TPG_PIXEL_RATE / hts
>>>> +               / max(rounddown(tgt_fps, 15), 5);
>>>> +
>>>> +    *def_vblank = ALIGN_DOWN(vblank, 2) - format->height;
>>> "vblank = vblank - height" doesn't seem right. The "else" branch stores
>>> a vts in vblank, which doesn't seem right either. Maybe you meant
>>> something like
>>>
>>>     if (tgt_fps < 5)
>>>         def_vts = *min_vblank + format->height;
>>>     else
>>>         def_vts = MALI_C55_TPG_PIXEL_RATE / hts
>>>             / max(rounddown(tgt_fps, 15), 5);
>>>
>>>     *def_vblank = ALIGN_DOWN(def_vts, 2) - format->height;
>>>
>>>> +}
>>>> +
>>>> +static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
>>>> +{
>>>> +    struct mali_c55_tpg *tpg = container_of(ctrl->handler,
>>>> +                        struct mali_c55_tpg,
>>>> +                        ctrls.handler);
>>>> +    struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
>>>> +
>>> Should you return here if the pipeline isn't streaming ?
>>>
>>>> +    switch (ctrl->id) {
>>>> +    case V4L2_CID_TEST_PATTERN:
>>>> +        mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
>>>> +                   ctrl->val);
>>>> +        break;
>>>> +    case V4L2_CID_VBLANK:
>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
>>>> +                     MALI_C55_REG_VBLANK_MASK, ctrl->val);
>>>> +        break;
>>>> +    default:
>>>> +        dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
>>>> +        return -EINVAL;
>>> Can this happen ?
>>>
>>>> +    }
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
>>>> +    .s_ctrl = &mali_c55_tpg_s_ctrl,
>>>> +};
>>>> +
>>>> +static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
>>>> +                   struct v4l2_subdev *sd)
>>>> +{
>>>> +    struct v4l2_subdev_state *state;
>>>> +    struct v4l2_mbus_framefmt *fmt;
>>>> +
>>>> +    /*
>>>> +     * hblank needs setting, but is a read-only control and thus won't be
>>>> +     * called during __v4l2_ctrl_handler_setup(). Do it here instead.
>>>> +     */
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
>>>> +                 MALI_C55_REG_HBLANK_MASK,
>>>> +                 MALI_C55_TPG_FIXED_HBLANK);
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>>> +                 MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
>>>> +
>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>> +
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>>> +                 MALI_C55_TEST_PATTERN_RGB_MASK,
>>>> +                 fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
>>>> +                      0x01 : 0x0);
>>>> +
>>>> +    v4l2_subdev_unlock_state(state);
>>>> +}
>>>> +
>>>> +static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
>>>> +{
>>>> +    struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
>>>> +    struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
>>>> +
>>>> +    if (!enable) {
>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>>> +                MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>>> +                MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
>>>> +        return 0;
>>>> +    }
>>>> +
>>>> +    /*
>>>> +     * One might reasonably expect the framesize to be set here
>>>> +     * given it's configurable in .set_fmt(), but it's done in the
>>>> +     * ISP subdevice's stream on func instead, as the same register
>>> s/func/function/
>>>
>>>> +     * is also used to indicate the size of the data coming from the
>>>> +     * sensor.
>>>> +     */
>>>> +    mali_c55_tpg_configure(mali_c55, sd);
>>>     mali_c55_tpg_configure(tpg);
>>>
>>>> + __v4l2_ctrl_handler_setup(sd->ctrl_handler);
>>>> +
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>>> +                 MALI_C55_TEST_PATTERN_ON_OFF,
>>>> +                 MALI_C55_TEST_PATTERN_ON_OFF);
>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>>> +                 MALI_C55_REG_GEN_VIDEO_ON_MASK,
>>>> +                 MALI_C55_REG_GEN_VIDEO_ON_MASK);
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
>>>> +    .s_stream = &mali_c55_tpg_s_stream,
>>> Can we use .enable_streams() and .disable_streams() ?
>>>
>>>> +};
>>>> +
>>>> +static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
>>>> +                       struct v4l2_subdev_state *state,
>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
>>>> +{
>>>> +    if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>>> +        return -EINVAL;
>>>> +
>>>> +    code->code = mali_c55_tpg_mbus_codes[code->index];
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
>>>> +                    struct v4l2_subdev_state *state,
>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
>>>> +{
>>> You sohuld verify here that fse->code is a supported value and return
>>> -EINVAL otherwise.
>>>
>>>> +    if (fse->index > 0 || fse->pad > sd->entity.num_pads)
>>> Drop the pad check, it's done in the subdev core already.
>>>
>>>> +        return -EINVAL;
>>>> +
>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
>>>> +                struct v4l2_subdev_state *state,
>>>> +                struct v4l2_subdev_format *format)
>>>> +{
>>>> +    struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
>>>> +    int vblank_def, vblank_min;
>>>> +    unsigned int i;
>>>> +
>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>>>> +        if (fmt->code == mali_c55_tpg_mbus_codes[i])
>>>> +            break;
>>>> +    }
>>>> +
>>>> +    if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>> +
>>>> +    /*
>>>> +     * The TPG says that the test frame timing generation logic expects a
>>>> +     * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
>>>> +     * handle anything smaller than 128x128 it seems pointless to allow a
>>>> +     * smaller frame.
>>>> +     */
>>>> +    clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
>>>> +        MALI_C55_MAX_WIDTH);
>>>> +    clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
>>>> +        MALI_C55_MAX_HEIGHT);
>>>> +
>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;
>>> You're allowing userspace to set fmt->field, as well as all the
>>> colorspace parameters, to random values. I would instead do something
>>> like
>>>
>>>     for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>>>         if (format->format.code == mali_c55_tpg_mbus_codes[i])
>>>             break;
>>>     }
>>>
>>>     if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>>         format->format.code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>
>>>     format->format.width = clamp(format->format.width,
>>>                      MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>>     format->format.height = clamp(format->format.height,
>>>                       MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>>
>>>     fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>     fmt->code = format->format.code;
>>>     fmt->width = format->format.width;
>>>     fmt->height = format->format.height;
>>>
>>>     format->format = *fmt;
>>>
>>> Alternatively (which I think I like better),
>>>
>>>     fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>
>>>     fmt->code = format->format.code;
>>>
>>>     for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>>>         if (fmt->code == mali_c55_tpg_mbus_codes[i])
>>>             break;
>>>     }
>>>
>>>     if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>>         fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>
>>>     fmt->width = clamp(format->format.width,
>>>                MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>>     fmt->height = clamp(format->format.height,
>>>                 MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>>
>>>     format->format = *fmt;
>>>
>>>> +
>>>> +    if (format->which == V4L2_SUBDEV_FORMAT_TRY)
>>>> +        return 0;
>>>> +
>>>> +    __mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
>>>> +    __v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
>>>> +                 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
>>>> +    __v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);
>>> Move those three calls to a separate function, it will be reused below.
>>> I'd name is mali_c55_tpg_update_vblank(). You can fold
>>> __mali_c55_tpg_calc_vblank() in it.
>>>
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
>>>> +    .enum_mbus_code        = mali_c55_tpg_enum_mbus_code,
>>>> +    .enum_frame_size    = mali_c55_tpg_enum_frame_size,
>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
>>>> +    .set_fmt        = mali_c55_tpg_set_fmt,
>>>> +};
>>>> +
>>>> +static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
>>>> +    .video    = &mali_c55_tpg_video_ops,
>>>> +    .pad    = &mali_c55_tpg_pad_ops,
>>>> +};
>>>> +
>>>> +static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
>>>> +                   struct v4l2_subdev_state *sd_state)
>>> You name this variable state in every other subdev operation handler.
>>>
>>>> +{
>>>> +    struct v4l2_mbus_framefmt *fmt =
>>>> +        v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
>>>> +
>>>> +    fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>> +    fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>> +    fmt->field = V4L2_FIELD_NONE;
>>>> +    fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>> Initialize the colorspace fields too.
>>>
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
>>>> +    .init_state = mali_c55_tpg_init_state,
>>>> +};
>>>> +
>>>> +static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
>>>> +    struct v4l2_subdev *sd = &mali_c55->tpg.sd;
>>>> +    struct v4l2_mbus_framefmt *format;
>>>> +    struct v4l2_subdev_state *state;
>>>> +    int vblank_def, vblank_min;
>>>> +    int ret;
>>>> +
>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
>>>> +    format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>> +
>>>> +    ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);
>>> You have 3 controls.
>>>
>>>> +    if (ret)
>>>> +        goto err_unlock;
>>>> +
>>>> +    ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
>>>> +                &mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
>>>> +                ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
>>>> +                0, 3, mali_c55_tpg_test_pattern_menu);
>>>> +
>>>> +    /*
>>>> +     * We fix hblank at the minimum allowed value and control framerate
>>>> +     * solely through the vblank control.
>>>> +     */
>>>> +    ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
>>>> +                &mali_c55_tpg_ctrl_ops,
>>>> +                V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
>>>> +                MALI_C55_TPG_FIXED_HBLANK, 1,
>>>> +                MALI_C55_TPG_FIXED_HBLANK);
>>>> +    if (ctrls->hblank)
>>>> +        ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
>>>> +
>>>> +    __mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);
>>> Drop this and initialize the control with default values. You can then
>>> update the value by calling mali_c55_tpg_update_vblank() in
>>> mali_c55_register_tpg().
>>>
>>> The reason is to share the same mutex between the control handler and
>>> the subdev active state without having to add a separate mutex in the
>>> mali_c55_tpg structure. The simplest way to do so is to initialize the
>>> controls first, set sd->state_lock to point to the control handler lock,
>>> and call v4l2_subdev_init_finalize() as the last step. As a consequence,
>>> you can't access the active state when initializing controls.
>>>
>>> You can alternatively keep the lock in mali_c55_tpg and set
>>> sd->state_lock to point to it, but I think that's more complex.
>>>
>>>> +    ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
>>>> +                      &mali_c55_tpg_ctrl_ops,
>>>> +                      V4L2_CID_VBLANK, vblank_min,
>>>> +                      MALI_C55_TPG_MAX_VBLANK, 1,
>>>> +                      vblank_def);
>>>> +
>>>> +    if (ctrls->handler.error) {
>>>> +        dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
>>>> +        ret = ctrls->handler.error;
>>>> +        goto err_free_handler;
>>>> +    }
>>>> +
>>>> +    ctrls->handler.lock = &mali_c55->tpg.lock;
>>> Drop this and drop the mutex. The control handler will use its internal
>>> mutex.
>>>
>>>> +    mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
>>>> +
>>>> +    v4l2_subdev_unlock_state(state);
>>>> +
>>>> +    return 0;
>>>> +
>>>> +err_free_handler:
>>>> +    v4l2_ctrl_handler_free(&ctrls->handler);
>>>> +err_unlock:
>>>> +    v4l2_subdev_unlock_state(state);
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    struct mali_c55_tpg *tpg = &mali_c55->tpg;
>>>> +    struct v4l2_subdev *sd = &tpg->sd;
>>>> +    struct media_pad *pad = &tpg->pad;
>>>> +    int ret;
>>>> +
>>>> +    mutex_init(&tpg->lock);
>>>> +
>>>> +    v4l2_subdev_init(sd, &mali_c55_tpg_ops);
>>>> +    sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
>>>> +    sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
>>> Should we introduce a TPG function ?
>>>
>>>> +    sd->internal_ops = &mali_c55_tpg_internal_ops;
>>>> +    strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
>>>> +
>>>> +    pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;
>>> I don't think MEDIA_PAD_FL_MUST_CONNECT is right.
>>>
>>>> +    ret = media_entity_pads_init(&sd->entity, 1, pad);
>>>> +    if (ret) {
>>>> +        dev_err(mali_c55->dev,
>>>> +            "Failed to initialize media entity pads\n");
>>>> +        goto err_destroy_mutex;
>>>> +    }
>>>> +
>>>     sd->state_lock = sd->ctrl_handler->lock;
>>>
>>> to use the same lock for the controls and the active state. You need to
>>> move this line and the v4l2_subdev_init_finalize() call after
>>> mali_c55_tpg_init_controls() to get the control handler lock initialized
>>> first.
>>>
>>>> +    ret = v4l2_subdev_init_finalize(sd);
>>>> +    if (ret)
>>>> +        goto err_cleanup_media_entity;
>>>> +
>>>> +    ret = mali_c55_tpg_init_controls(mali_c55);
>>>> +    if (ret) {
>>>> +        dev_err(mali_c55->dev,
>>>> +            "Error initialising controls\n");
>>>> +        goto err_cleanup_subdev;
>>>> +    }
>>>> +
>>>> +    ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>>> +    if (ret) {
>>>> +        dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
>>>> +        goto err_free_ctrl_handler;
>>>> +    }
>>>> +
>>>> +    /*
>>>> +     * By default the colour settings lead to a very dim image that is
>>>> +     * nearly indistinguishable from black on some monitor settings. Ramp
>>>> +     * them up a bit so the image is brighter.
>>>> +     */
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
>>>> +
>>>> +    tpg->mali_c55 = mali_c55;
>>>> +
>>>> +    return 0;
>>>> +
>>>> +err_free_ctrl_handler:
>>>> +    v4l2_ctrl_handler_free(&tpg->ctrls.handler);
>>>> +err_cleanup_subdev:
>>>> +    v4l2_subdev_cleanup(sd);
>>>> +err_cleanup_media_entity:
>>>> +    media_entity_cleanup(&sd->entity);
>>>> +err_destroy_mutex:
>>>> +    mutex_destroy(&tpg->lock);
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
>>>> +{
>>>> +    struct mali_c55_tpg *tpg = &mali_c55->tpg;
>>>> +
>>>> +    if (!tpg->mali_c55)
>>>> +        return;
>>>> +
>>>> +    v4l2_device_unregister_subdev(&tpg->sd);
>>>> +    v4l2_subdev_cleanup(&tpg->sd);
>>>> +    media_entity_cleanup(&tpg->sd.entity);
>>>> +    v4l2_ctrl_handler_free(&tpg->ctrls.handler);
>>> Free the control handler just after v4l2_device_unregister_subdev() to
>>> match the order in mali_c55_register_tpg().
>>>
>>>> +    mutex_destroy(&tpg->lock);
>>>> +}

Re: [PATCH v5 10/16] media: platform: Add mali-c55 3a stats devnode

[Dan Scally]

Hi Laurent

On 16/06/2024 22:19, Laurent Pinchart wrote:
> Hi Dan,
>
> Thank you for the patch.
>
> On Wed, May 29, 2024 at 04:28:52PM +0100, Daniel Scally wrote:
>> Add a new code file to govern the 3a statistics capture node.
>>
>> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>> ---
>> Changes in v5:
>>
>> 	- New patch
>>
>>   drivers/media/platform/arm/mali-c55/Makefile  |   3 +-
>>   .../platform/arm/mali-c55/mali-c55-common.h   |  28 ++
>>   .../platform/arm/mali-c55/mali-c55-core.c     |  15 +
>>   .../platform/arm/mali-c55/mali-c55-isp.c      |   1 +
>>   .../arm/mali-c55/mali-c55-registers.h         |   3 +
>>   .../platform/arm/mali-c55/mali-c55-stats.c    | 350 ++++++++++++++++++
>>   6 files changed, 399 insertions(+), 1 deletion(-)
>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-stats.c
>>
>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
>> index 77dcb2fbf0f4..cd5a64bf0c62 100644
>> --- a/drivers/media/platform/arm/mali-c55/Makefile
>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
>> @@ -4,6 +4,7 @@ mali-c55-y := mali-c55-capture.o \
>>   	      mali-c55-core.o \
>>   	      mali-c55-isp.o \
>>   	      mali-c55-tpg.o \
>> -	      mali-c55-resizer.o
>> +	      mali-c55-resizer.o \
>> +	      mali-c55-stats.o
>>   
>>   obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>> index 2d0c4d152beb..44119e04009b 100644
>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>> @@ -79,6 +79,7 @@ enum mali_c55_isp_pads {
>>   	MALI_C55_ISP_PAD_SINK_VIDEO,
>>   	MALI_C55_ISP_PAD_SOURCE,
>>   	MALI_C55_ISP_PAD_SOURCE_BYPASS,
>> +	MALI_C55_ISP_PAD_SOURCE_3A,
> Functions and structures are named with a "stats" suffix, let's call
> this MALI_C55_ISP_PAD_SOURCE_STATS.
>
>>   	MALI_C55_ISP_NUM_PADS,
>>   };
>>   
>> @@ -194,6 +195,28 @@ struct mali_c55_cap_dev {
>>   	bool streaming;
>>   };
>>   
>> +struct mali_c55_stats_buf {
>> +	struct vb2_v4l2_buffer vb;
>> +	spinlock_t lock;
> All locks require a comment to document what they protect. Same below.
>
>> +	unsigned int segments_remaining;
>> +	struct list_head queue;
>> +	bool failed;
>> +};
>> +
>> +struct mali_c55_stats {
>> +	struct mali_c55 *mali_c55;
>> +	struct video_device vdev;
>> +	struct dma_chan *channel;
>> +	struct vb2_queue queue;
>> +	struct media_pad pad;
>> +	struct mutex lock;
>> +
>> +	struct {
>> +		spinlock_t lock;
>> +		struct list_head queue;
>> +	} buffers;
>> +};
>> +
>>   enum mali_c55_config_spaces {
>>   	MALI_C55_CONFIG_PING,
>>   	MALI_C55_CONFIG_PONG,
>> @@ -224,6 +247,7 @@ struct mali_c55 {
>>   	struct mali_c55_isp isp;
>>   	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
>>   	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
>> +	struct mali_c55_stats stats;
>>   
>>   	struct list_head contexts;
>>   	enum mali_c55_config_spaces next_config;
>> @@ -245,6 +269,8 @@ int mali_c55_register_resizers(struct mali_c55 *mali_c55);
>>   void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
>>   int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
>>   void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
>> +int mali_c55_register_stats(struct mali_c55 *mali_c55);
>> +void mali_c55_unregister_stats(struct mali_c55 *mali_c55);
>>   struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
>>   void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>>   			     enum mali_c55_planes plane);
>> @@ -262,5 +288,7 @@ mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
>>   bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
>>   #define for_each_mali_isp_fmt(fmt)\
>>   	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
>> +void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
>> +				enum mali_c55_config_spaces cfg_space);
>>   
>>   #endif /* _MALI_C55_COMMON_H */
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>> index 50caf5ee7474..9ea70010876c 100644
>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>> @@ -337,6 +337,16 @@ static int mali_c55_create_links(struct mali_c55 *mali_c55)
>>   		}
>>   	}
>>   
>> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>> +			MALI_C55_ISP_PAD_SOURCE_3A,
>> +			&mali_c55->stats.vdev.entity, 0,
>> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev,
>> +			"failed to link ISP and 3a stats node\n");
> s/3a stats/stats/
>
>> +		goto err_remove_links;
>> +	}
>> +
>>   	return 0;
>>   
>>   err_remove_links:
>> @@ -350,6 +360,7 @@ static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
>>   	mali_c55_unregister_isp(mali_c55);
>>   	mali_c55_unregister_resizers(mali_c55);
>>   	mali_c55_unregister_capture_devs(mali_c55);
>> +	mali_c55_unregister_stats(mali_c55);
>>   }
>>   
>>   static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>> @@ -372,6 +383,10 @@ static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>>   	if (ret)
>>   		goto err_unregister_entities;
>>   
>> +	ret = mali_c55_register_stats(mali_c55);
>> +	if (ret)
>> +		goto err_unregister_entities;
>> +
>>   	ret = mali_c55_create_links(mali_c55);
>>   	if (ret)
>>   		goto err_unregister_entities;
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>> index ea8b7b866e7a..94876fba3353 100644
>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>> @@ -564,6 +564,7 @@ int mali_c55_register_isp(struct mali_c55 *mali_c55)
>>   	isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
>>   	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>>   	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
>> +	isp->pads[MALI_C55_ISP_PAD_SOURCE_3A].flags = MEDIA_PAD_FL_SOURCE;
>>   
>>   	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
>>   				     isp->pads);
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>> index cb27abde2aa5..eb3719245ec3 100644
>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>> @@ -68,6 +68,9 @@
>>   #define MALI_C55_VC_START(v)				((v) & 0xffff)
>>   #define MALI_C55_VC_SIZE(v)				(((v) & 0xffff) << 16)
>>   
>> +#define MALI_C55_REG_1024BIN_HIST			0x054a8
>> +#define MALI_C55_1024BIN_HIST_SIZE			4096
>> +
>>   /* Ping/Pong Configuration Space */
>>   #define MALI_C55_REG_BASE_ADDR				0x18e88
>>   #define MALI_C55_REG_BYPASS_0				0x18eac
>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-stats.c b/drivers/media/platform/arm/mali-c55/mali-c55-stats.c
>> new file mode 100644
>> index 000000000000..aa40480ed814
>> --- /dev/null
>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-stats.c
>> @@ -0,0 +1,350 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ARM Mali-C55 ISP Driver - 3A Statistics capture device
>> + *
>> + * Copyright (C) 2023 Ideas on Board Oy
>> + */
>> +
>> +#include <linux/dmaengine.h>
>> +#include <linux/media/arm/mali-c55-config.h>
>> +#include <linux/spinlock.h>
> You're missing some headers here, for
>
> container_of()
> dev_err()
> list_*()
> mutex_init()
> strscpy()
> strscpy()
>
>> +
>> +#include <media/media-entity.h>
>> +#include <media/v4l2-dev.h>
>> +#include <media/v4l2-event.h>
>> +#include <media/v4l2-fh.h>
>> +#include <media/v4l2-ioctl.h>
>> +#include <media/videobuf2-core.h>
>> +#include <media/videobuf2-dma-contig.h>
>> +
>> +#include "mali-c55-common.h"
>> +#include "mali-c55-registers.h"
>> +
>> +static unsigned int metering_space_addrs[] = {
> const
>
>> +	[MALI_C55_CONFIG_PING] = 0x095AC,
>> +	[MALI_C55_CONFIG_PONG] = 0x2156C,
> Lower-case hex constants.
>
>> +};
>> +
>> +static int mali_c55_stats_enum_fmt_meta_cap(struct file *file, void *fh,
>> +					    struct v4l2_fmtdesc *f)
>> +{
>> +	if (f->index || f->type != V4L2_BUF_TYPE_META_CAPTURE)
>> +		return -EINVAL;
>> +
>> +	f->pixelformat = V4L2_META_FMT_MALI_C55_3A_STATS;
> The format could be called V4L2_META_FMT_MALI_C55_STATS. While most
> statistics are related to one of the 3A algorithms, I think it would be
> better to name this generically. It's name bikeshedding only of course.
>
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_stats_g_fmt_meta_cap(struct file *file, void *fh,
>> +					 struct v4l2_format *f)
>> +{
>> +	static const struct v4l2_meta_format mfmt = {
>> +		.dataformat = V4L2_META_FMT_MALI_C55_3A_STATS,
>> +		.buffersize = sizeof(struct mali_c55_stats_buffer)
>> +	};
>> +
>> +	if (f->type != V4L2_BUF_TYPE_META_CAPTURE)
>> +		return -EINVAL;
>> +
>> +	f->fmt.meta = mfmt;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mali_c55_stats_querycap(struct file *file,
>> +				   void *priv, struct v4l2_capability *cap)
>> +{
>> +	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
>> +	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_ioctl_ops mali_c55_stats_v4l2_ioctl_ops = {
>> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
>> +	.vidioc_querybuf = vb2_ioctl_querybuf,
>> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
>> +	.vidioc_qbuf = vb2_ioctl_qbuf,
>> +	.vidioc_expbuf = vb2_ioctl_expbuf,
>> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
>> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
>> +	.vidioc_streamon = vb2_ioctl_streamon,
>> +	.vidioc_streamoff = vb2_ioctl_streamoff,
>> +	.vidioc_enum_fmt_meta_cap = mali_c55_stats_enum_fmt_meta_cap,
>> +	.vidioc_g_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
>> +	.vidioc_s_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
>> +	.vidioc_try_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
>> +	.vidioc_querycap = mali_c55_stats_querycap,
>> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
>> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
>> +};
>> +
>> +static const struct v4l2_file_operations mali_c55_stats_v4l2_fops = {
>> +	.owner = THIS_MODULE,
>> +	.unlocked_ioctl = video_ioctl2,
>> +	.open = v4l2_fh_open,
>> +	.release = vb2_fop_release,
>> +	.poll = vb2_fop_poll,
>> +	.mmap = vb2_fop_mmap,
>> +};
>> +
>> +static int
>> +mali_c55_stats_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
>> +			   unsigned int *num_planes, unsigned int sizes[],
>> +			   struct device *alloc_devs[])
>> +{
>> +	struct mali_c55_stats *stats = vb2_get_drv_priv(q);
>> +
>> +	if (*num_planes && *num_planes > 1)
>> +		return -EINVAL;
>> +
>> +	if (sizes[0] && sizes[0] != sizeof(struct mali_c55_stats_buffer))
>> +		return -EINVAL;
>> +
>> +	*num_planes = 1;
>> +	sizes[0] = sizeof(struct mali_c55_stats_buffer);
>> +
>> +	if (stats->channel)
>> +		alloc_devs[0] = stats->channel->device->dev;
>> +
>> +	return 0;
>> +}
>> +
>> +static void mali_c55_stats_buf_queue(struct vb2_buffer *vb)
>> +{
>> +	struct mali_c55_stats *stats = vb2_get_drv_priv(vb->vb2_queue);
>> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>> +	struct mali_c55_stats_buf *buf = container_of(vbuf,
>> +						struct mali_c55_stats_buf, vb);
>> +
>> +	vb2_set_plane_payload(vb, 0, sizeof(struct mali_c55_stats_buffer));
>> +	buf->segments_remaining = 2;
>> +	buf->failed = false;
>> +
>> +	spin_lock(&stats->buffers.lock);
> Isn't the DMA completion handler run from IRQ context ? If so you'll need to use spin_lock_irq() here and in the other function that are not
> called with interrupts disabled.


They're run in the bottom half of the interrupt handler; I'm under the impression that that means 
the interrupts aren't disabled, and it's safe to do...is that mistaken?

>
>> +	list_add_tail(&buf->queue, &stats->buffers.queue);
>> +	spin_unlock(&stats->buffers.lock);
>> +}
>> +
>> +static void mali_c55_stats_stop_streaming(struct vb2_queue *q)
>> +{
>> +	struct mali_c55_stats *stats = vb2_get_drv_priv(q);
>> +	struct mali_c55_stats_buf *buf, *tmp;
>> +
>> +	spin_lock(&stats->buffers.lock);
>> +
>> +	list_for_each_entry_safe(buf, tmp, &stats->buffers.queue, queue) {
>> +		list_del(&buf->queue);
>> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
>> +	}
>> +
>> +	spin_unlock(&stats->buffers.lock);
>> +}
>> +
>> +static const struct vb2_ops mali_c55_stats_vb2_ops = {
>> +	.queue_setup = mali_c55_stats_queue_setup,
>> +	.buf_queue = mali_c55_stats_buf_queue,
>> +	.wait_prepare = vb2_ops_wait_prepare,
>> +	.wait_finish = vb2_ops_wait_finish,
>> +	.stop_streaming = mali_c55_stats_stop_streaming,
>> +};
>> +
>> +static void
>> +mali_c55_stats_metering_complete(void *param,
>> +				 const struct dmaengine_result *result)
>> +{
>> +	struct mali_c55_stats_buf *buf = param;
>> +
>> +	spin_lock(&buf->lock);
> I wonder if this is needed. Can the DMA engine call the completion
> handlers of two sequential DMA transfers in parallel ?


The DMA engine that's on the system we have can't...I wasn't sure whether that was generically true.

>
>> +
>> +	if (buf->failed)
>> +		goto out_unlock;
>> +
>> +	buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>> +
>> +	if (result->result != DMA_TRANS_NOERROR) {
>> +		buf->failed = true;
>> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
> This will possibly return the buffer to userspace after the first DMA
> transfer. Userspace could then requeue the buffer to the kernel before
> the completion of the second DMA transfer. That will cause trouble. I
> think you should instead do something like
>
> 	spin_lock(&buf->lock);
>
> 	if (result->result != DMA_TRANS_NOERROR)
> 		buf->failed = true;
>
> 	if (!--buf->segments_remaining) {
> 		buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> 		vb2_buffer_done(&buf->vb.vb2_buf, buf->failed ?
> 				VB2_BUF_STATE_ERROR : VB2_BUF_STATE_DONE);
> 	}
>
> 	spin_unlock(&buf->lock);
>
> The
>
> 	buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>
> line could also be moved to mali_c55_stats_fill_buffer(), which would
> make sure the timestamp is filled in case of DMA submission failures.


Okedokey

>
>> +		goto out_unlock;
>> +	}
>> +
>> +	if (!--buf->segments_remaining)
>> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
>> +
>> +out_unlock:
>> +	spin_unlock(&buf->lock);
>> +}
>> +
>> +static int mali_c55_stats_dma_xfer(struct mali_c55_stats *stats, dma_addr_t src,
>> +				   dma_addr_t dst,
>> +				   struct mali_c55_stats_buf *buf,
>> +				   size_t length,
>> +				   void (*callback)(void *, const struct dmaengine_result *result))
> The same callback is used for both invocations of this function, you can
> drop the parameter and hardcode it below.
Yeah, not even sure now why I had a parameter.
>> +{
>> +	struct dma_async_tx_descriptor *tx;
>> +	dma_cookie_t cookie;
>> +
>> +	tx = dmaengine_prep_dma_memcpy(stats->channel, dst, src, length, 0);
>> +	if (!tx) {
>> +		dev_err(stats->mali_c55->dev, "failed to prep stats DMA\n");
>> +		return -EIO;
>> +	}
>> +
>> +	tx->callback_result = callback;
>> +	tx->callback_param = buf;
>> +
>> +	cookie = dmaengine_submit(tx);
>> +	if (dma_submit_error(cookie)) {
>> +		dev_err(stats->mali_c55->dev, "failed to submit stats DMA\n");
>> +		return -EIO;
>> +	}
>> +
>> +	dma_async_issue_pending(stats->channel);
>> +	return 0;
>> +}
>> +
>> +void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
>> +				enum mali_c55_config_spaces cfg_space)
>> +{
>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>> +	struct mali_c55_stats *stats = &mali_c55->stats;
>> +	struct mali_c55_stats_buf *buf = NULL;
>> +	dma_addr_t src, dst;
>> +	int ret;
>> +
>> +	spin_lock(&stats->buffers.lock);
>> +	if (!list_empty(&stats->buffers.queue)) {
>> +		buf = list_first_entry(&stats->buffers.queue,
>> +				       struct mali_c55_stats_buf, queue);
>> +		list_del(&buf->queue);
>> +	}
>> +	spin_unlock(&stats->buffers.lock);
>> +
>> +	if (!buf)
>> +		return;
>> +
>> +	buf->vb.sequence = mali_c55->isp.frame_sequence;
>> +
>> +	/*
>> +	 * There are infact two noncontiguous sections of the ISP's
> s/infact/in fact/
>
>> +	 * memory space that hold statistics for 3a algorithms to use. A
> s/use. A/use: a/
>
>> +	 * section in each config space and a global section holding
>> +	 * histograms which is double buffered and so holds data for the
>> +	 * last frame. We need to read both.
>> +	 */
>> +	src = ctx->base + MALI_C55_REG_1024BIN_HIST;
>> +	dst = vb2_dma_contig_plane_dma_addr(&buf->vb.vb2_buf, 0);
>> +
>> +	ret = mali_c55_stats_dma_xfer(stats, src, dst, buf,
>> +				      MALI_C55_1024BIN_HIST_SIZE,
>> +				      mali_c55_stats_metering_complete);
>> +	if (ret)
>> +		goto err_fail_buffer;
>> +
>> +	src = ctx->base + metering_space_addrs[cfg_space];
>> +	dst += MALI_C55_1024BIN_HIST_SIZE;
>> +
>> +	ret = mali_c55_stats_dma_xfer(
>> +		stats, src, dst, buf,
>> +		sizeof(struct mali_c55_stats_buffer) - MALI_C55_1024BIN_HIST_SIZE,
>> +		mali_c55_stats_metering_complete);
>> +	if (ret) {
>> +		dmaengine_terminate_sync(stats->channel);
>> +		goto err_fail_buffer;
>> +	}
> I think you will need to terminate DMA transfers at stream off time.
>
>> +
>> +	return;
>> +
>> +err_fail_buffer:
>> +	vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
>> +}
>> +
>> +void mali_c55_unregister_stats(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_stats *stats = &mali_c55->stats;
>> +
>> +	if (!video_is_registered(&stats->vdev))
>> +		return;
>> +
>> +	vb2_video_unregister_device(&stats->vdev);
>> +	media_entity_cleanup(&stats->vdev.entity);
>> +	dma_release_channel(stats->channel);
>> +	mutex_destroy(&stats->lock);
>> +}
>> +
>> +int mali_c55_register_stats(struct mali_c55 *mali_c55)
>> +{
>> +	struct mali_c55_stats *stats = &mali_c55->stats;
>> +	struct video_device *vdev = &stats->vdev;
>> +	struct vb2_queue *vb2q = &stats->queue;
>> +	dma_cap_mask_t mask;
>> +	int ret;
>> +
>> +	mutex_init(&stats->lock);
>> +	INIT_LIST_HEAD(&stats->buffers.queue);
>> +
>> +	dma_cap_zero(mask);
>> +	dma_cap_set(DMA_MEMCPY, mask);
>> +
>> +	stats->channel = dma_request_channel(mask, 0, NULL);
> Do we need a CPU fallback in case no DMA is available ?


Yes, actually.

>
> I'm still very curious to know how long it takes to perform the DMA
> transfer, compared to copying the data with the CPU, and especially
> compared to the frame duration.


On my list of things to test and report :)

>
>> +	if (!stats->channel) {
>> +		ret = -ENODEV;
>> +		goto err_destroy_mutex;
>> +	}
>> +
>> +	stats->pad.flags = MEDIA_PAD_FL_SINK;
>> +	ret = media_entity_pads_init(&stats->vdev.entity, 1, &stats->pad);
>> +	if (ret)
>> +		goto err_release_dma_channel;
>> +
>> +	vb2q->type = V4L2_BUF_TYPE_META_CAPTURE;
>> +	vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
>> +	vb2q->drv_priv = stats;
>> +	vb2q->mem_ops = &vb2_dma_contig_memops;
>> +	vb2q->ops = &mali_c55_stats_vb2_ops;
>> +	vb2q->buf_struct_size = sizeof(struct mali_c55_stats_buf);
>> +	vb2q->min_queued_buffers = 1;
>> +	vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
>> +	vb2q->lock = &stats->lock;
>> +	vb2q->dev = mali_c55->dev;
> That's not the right device. The device that performs the DMA operation
> is the DMA engine, and that's what you need to pass to vb2. Otherwise
> the DMA address returned by vb2_dma_contig_plane_dma_addr() will be
> mapped to the ISP device, not the DMA engine. In practice, if neither
> are behind an IOMMU, things will likely work, but when that's not the
> case, run into problems.
>
>> +
>> +	ret = vb2_queue_init(vb2q);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev, "stats vb2 queue init failed\n");
>> +		goto err_cleanup_entity;
>> +	}
>> +
>> +	strscpy(stats->vdev.name, "mali-c55 3a stats", sizeof(stats->vdev.name));
> s/3a //
>
>> +	vdev->release = video_device_release_empty;
> That's never right. You should refcount the data structures to ensure
> proper lifetime management.
>
>> +	vdev->fops = &mali_c55_stats_v4l2_fops;
>> +	vdev->ioctl_ops = &mali_c55_stats_v4l2_ioctl_ops;
>> +	vdev->lock = &stats->lock;
>> +	vdev->v4l2_dev = &mali_c55->v4l2_dev;
>> +	vdev->queue = &stats->queue;
>> +	vdev->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING;
>> +	vdev->vfl_dir = VFL_DIR_RX;
>> +	video_set_drvdata(vdev, stats);
>> +
>> +	ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
>> +	if (ret) {
>> +		dev_err(mali_c55->dev,
>> +			"failed to register stats video device\n");
>> +		goto err_release_vb2q;
>> +	}
>> +
>> +	stats->mali_c55 = mali_c55;
>> +
>> +	return 0;
>> +
>> +err_release_vb2q:
>> +	vb2_queue_release(vb2q);
>> +err_cleanup_entity:
>> +	media_entity_cleanup(&stats->vdev.entity);
>> +err_release_dma_channel:
>> +	dma_release_channel(stats->channel);
>> +err_destroy_mutex:
>> +	mutex_destroy(&stats->lock);
>> +
>> +	return ret;
>> +}

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Laurent Pinchart]

Hi Dan,

On Thu, Jun 20, 2024 at 03:49:23PM +0100, Daniel Scally wrote:
> On 20/06/2024 15:33, Dan Scally wrote:
> > On 30/05/2024 22:43, Laurent Pinchart wrote:
> >> And now the second part of the review, addressing mali-c55-capture.c and
> >> mali-c55-resizer.c. I've reviewed the code from the bottom up, so some
> >> messages may be repeated in an order that seems weird. Sorry about that.
> >>
> >> On Thu, May 30, 2024 at 03:15:10AM +0300, Laurent Pinchart wrote:
> >>> On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
> >>>> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
> >>>> V4L2 and Media Controller compliant and creates subdevices to manage
> >>>> the ISP itself, its internal test pattern generator as well as the
> >>>> crop, scaler and output format functionality for each of its two
> >>>> output devices.
> >>>>
> >>>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
> >>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> >>>> ---
> >>>> Changes in v5:
> >>>>
> >>>>     - Reworked input formats - previously we allowed representing input data
> >>>>       as any 8-16 bit format. Now we only allow input data to be represented
> >>>>       by the new 20-bit bayer formats, which is corrected to the equivalent
> >>>>       16-bit format in RAW bypass mode.
> >>>>     - Stopped bypassing blocks that we haven't added supporting parameters
> >>>>       for yet.
> >>>>     - Addressed most of Sakari's comments from the list
> >>>>
> >>>> Changes not yet made in v5:
> >>>>
> >>>>     - The output pipelines can still be started and stopped independently of
> >>>>       one another - I'd like to discuss that more.
> >>>>     - the TPG subdev still uses .s_stream() - I need to rebase onto a tree
> >>>>       with working .enable_streams() for a single-source-pad subdevice.
> >>>>
> >>>> Changes in v4:
> >>>>
> >>>>     - Reworked mali_c55_update_bits() to internally perform the bit-shift
> >>>
> >>> I really don't like that, it makes the code very confusing, even more so
> >>> as it differs from regmap_update_bits().
> >>>
> >>> Look at this for instance:
> >>>
> >>>     mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>                  MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> >>>                  MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> >>>
> >>> It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
> >>> BIT(0).
> >>>
> >>> Sorry, I know it will be painful, but this change needs to be reverted.
> >>>
> >>>>     - Reworked the resizer to allow cropping during streaming
> >>>>     - Fixed a bug in NV12 output
> >>>>
> >>>> Changes in v3:
> >>>>
> >>>>     - Mostly minor fixes suggested by Sakari
> >>>>     - Fixed the sequencing of vb2 buffers to be synchronised across the two
> >>>>       capture devices.
> >>>>
> >>>> Changes in v2:
> >>>>
> >>>>     - Clock handling
> >>>>     - Fixed the warnings raised by the kernel test robot
> >>>>
> >>>>   drivers/media/platform/Kconfig                |   1 +
> >>>>   drivers/media/platform/Makefile               |   1 +
> >>>>   drivers/media/platform/arm/Kconfig            |   5 +
> >>>>   drivers/media/platform/arm/Makefile           |   2 +
> >>>>   drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
> >>>>   drivers/media/platform/arm/mali-c55/Makefile  |   9 +
> >>>>   .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
> >>>>   .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
> >>>>   .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
> >>>>   .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
> >>>>   .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
> >>>>   .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
> >>>>   .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
> >>>>   .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
> >>>>   14 files changed, 4452 insertions(+)
> >>>>   create mode 100644 drivers/media/platform/arm/Kconfig
> >>>>   create mode 100644 drivers/media/platform/arm/Makefile
> >>>>   create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
> >>>>   create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
> >>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>>>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >>>
> >>> I've skipped review of capture.c and resizer.c as I already have plenty
> >>> of comments for the other files, and it's getting late. I'll try to
> >>> review the rest tomorrow.
> >>>
> >>>> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> >>>> index 2d79bfc68c15..c929169766aa 100644
> >>>> --- a/drivers/media/platform/Kconfig
> >>>> +++ b/drivers/media/platform/Kconfig
> >>>> @@ -65,6 +65,7 @@ config VIDEO_MUX
> >>>>   source "drivers/media/platform/allegro-dvt/Kconfig"
> >>>>   source "drivers/media/platform/amlogic/Kconfig"
> >>>>   source "drivers/media/platform/amphion/Kconfig"
> >>>> +source "drivers/media/platform/arm/Kconfig"
> >>>>   source "drivers/media/platform/aspeed/Kconfig"
> >>>>   source "drivers/media/platform/atmel/Kconfig"
> >>>>   source "drivers/media/platform/broadcom/Kconfig"
> >>>> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> >>>> index da17301f7439..9a647abd5218 100644
> >>>> --- a/drivers/media/platform/Makefile
> >>>> +++ b/drivers/media/platform/Makefile
> >>>> @@ -8,6 +8,7 @@
> >>>>   obj-y += allegro-dvt/
> >>>>   obj-y += amlogic/
> >>>>   obj-y += amphion/
> >>>> +obj-y += arm/
> >>>>   obj-y += aspeed/
> >>>>   obj-y += atmel/
> >>>>   obj-y += broadcom/
> >>>> diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
> >>>> new file mode 100644
> >>>> index 000000000000..4f0764c329c7
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/Kconfig
> >>>> @@ -0,0 +1,5 @@
> >>>> +# SPDX-License-Identifier: GPL-2.0-only
> >>>> +
> >>>> +comment "ARM media platform drivers"
> >>>> +
> >>>> +source "drivers/media/platform/arm/mali-c55/Kconfig"
> >>>> diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
> >>>> new file mode 100644
> >>>> index 000000000000..8cc4918725ef
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/Makefile
> >>>> @@ -0,0 +1,2 @@
> >>>> +# SPDX-License-Identifier: GPL-2.0-only
> >>>> +obj-y += mali-c55/
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/Kconfig 
> >>>> b/drivers/media/platform/arm/mali-c55/Kconfig
> >>>> new file mode 100644
> >>>> index 000000000000..602085e28b01
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/Kconfig
> >>>> @@ -0,0 +1,18 @@
> >>>> +# SPDX-License-Identifier: GPL-2.0-only
> >>>> +config VIDEO_MALI_C55
> >>>> +    tristate "ARM Mali-C55 Image Signal Processor driver"
> >>>> +    depends on V4L_PLATFORM_DRIVERS
> >>>> +    depends on VIDEO_DEV && OF
> >>>> +    depends on ARCH_VEXPRESS || COMPILE_TEST
> >>>> +    select MEDIA_CONTROLLER
> >>>> +    select VIDEO_V4L2_SUBDEV_API
> >>>> +    select VIDEOBUF2_DMA_CONTIG
> >>>> +    select VIDEOBUF2_VMALLOC
> >>>> +    select V4L2_FWNODE
> >>>> +    select GENERIC_PHY_MIPI_DPHY
> >>>
> >>> Alphabetical order ?
> >>>
> >>>> +    default n
> >>>
> >>> That's the default, you don't have to specify ti.
> >>>
> >>>> +    help
> >>>> +      Enable this to support Arm's Mali-C55 Image Signal Processor.
> >>>> +
> >>>> +      To compile this driver as a module, choose M here: the module
> >>>> +      will be called mali-c55.
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile 
> >>>> b/drivers/media/platform/arm/mali-c55/Makefile
> >>>> new file mode 100644
> >>>> index 000000000000..77dcb2fbf0f4
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
> >>>> @@ -0,0 +1,9 @@
> >>>> +# SPDX-License-Identifier: GPL-2.0
> >>>> +
> >>>> +mali-c55-y := mali-c55-capture.o \
> >>>> +          mali-c55-core.o \
> >>>> +          mali-c55-isp.o \
> >>>> +          mali-c55-tpg.o \
> >>>> +          mali-c55-resizer.o
> >>>
> >>> Alphabetical order here too.
> >>>
> >>>> +
> >>>> +obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c 
> >>>> b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>>> new file mode 100644
> >>>> index 000000000000..1d539ac9c498
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>>> @@ -0,0 +1,951 @@
> >>>> +// SPDX-License-Identifier: GPL-2.0
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Video capture devices
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#include <linux/cleanup.h>
> >>>> +#include <linux/minmax.h>
> >>>> +#include <linux/pm_runtime.h>
> >>>> +#include <linux/string.h>
> >>>> +#include <linux/videodev2.h>
> >>>> +
> >>>> +#include <media/v4l2-dev.h>
> >>>> +#include <media/v4l2-event.h>
> >>>> +#include <media/v4l2-ioctl.h>
> >>>> +#include <media/v4l2-subdev.h>
> >>>> +#include <media/videobuf2-core.h>
> >>>> +#include <media/videobuf2-dma-contig.h>
> >>>> +
> >>>> +#include "mali-c55-common.h"
> >>>> +#include "mali-c55-registers.h"
> >>>> +
> >>>> +static const struct mali_c55_fmt mali_c55_fmts[] = {
> >>>> +    /*
> >>>> +     * This table is missing some entries which need further work or
> >>>> +     * investigation:
> >>>> +     *
> >>>> +     * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
> >>>> +     * Base mode 5 is "Generic Data"
> >>>> +     * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
> >>>> +     * Base mode 9 seems to have no V4L2 equivalent
> >>>> +     * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
> >>>> +     * equivalent
> >>>> +     */
> >>>> +    {
> >>>> +        .fourcc = V4L2_PIX_FMT_ARGB2101010,
> >>>> +        .mbus_codes = {
> >>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
> >>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
> >>>> +        },
> >>>> +        .is_raw = false,
> >>>> +        .registers = {
> >>>> +            .base_mode = MALI_C55_OUTPUT_A2R10G10B10,
> >>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +        }
> >>>> +    },
> >>>> +    {
> >>>> +        .fourcc = V4L2_PIX_FMT_RGB565,
> >>>> +        .mbus_codes = {
> >>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
> >>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
> >>>> +        },
> >>>> +        .is_raw = false,
> >>>> +        .registers = {
> >>>> +            .base_mode = MALI_C55_OUTPUT_RGB565,
> >>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +        }
> >>>> +    },
> >>>> +    {
> >>>> +        .fourcc = V4L2_PIX_FMT_BGR24,
> >>>> +        .mbus_codes = {
> >>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
> >>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
> >>>> +        },
> >>>> +        .is_raw = false,
> >>>> +        .registers = {
> >>>> +            .base_mode = MALI_C55_OUTPUT_RGB24,
> >>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +        }
> >>>> +    },
> >>>> +    {
> >>>> +        .fourcc = V4L2_PIX_FMT_YUYV,
> >>>> +        .mbus_codes = {
> >>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>> +        },
> >>>> +        .is_raw = false,
> >>>> +        .registers = {
> >>>> +            .base_mode = MALI_C55_OUTPUT_YUY2,
> >>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +        }
> >>>> +    },
> >>>> +    {
> >>>> +        .fourcc = V4L2_PIX_FMT_UYVY,
> >>>> +        .mbus_codes = {
> >>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>> +        },
> >>>> +        .is_raw = false,
> >>>> +        .registers = {
> >>>> +            .base_mode = MALI_C55_OUTPUT_UYVY,
> >>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +        }
> >>>> +    },
> >>>> +    {
> >>>> +        .fourcc = V4L2_PIX_FMT_Y210,
> >>>> +        .mbus_codes = {
> >>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>> +        },
> >>>> +        .is_raw = false,
> >>>> +        .registers = {
> >>>> +            .base_mode = MALI_C55_OUTPUT_Y210,
> >>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +        }
> >>>> +    },
> >>>> +    /*
> >>>> +     * This is something of a hack, the ISP thinks it's running NV12M but
> >>>> +     * by setting uv_plane = 0 we simply discard that planes and only output
> >>>> +     * the Y-plane.
> >>>> +     */
> >>>> +    {
> >>>> +        .fourcc = V4L2_PIX_FMT_GREY,
> >>>> +        .mbus_codes = {
> >>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>> +        },
> >>>> +        .is_raw = false,
> >>>> +        .registers = {
> >>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
> >>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +        }
> >>>> +    },
> >>>> +    {
> >>>> +        .fourcc = V4L2_PIX_FMT_NV12M,
> >>>> +        .mbus_codes = {
> >>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>> +        },
> >>>> +        .is_raw = false,
> >>>> +        .registers = {
> >>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
> >>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
> >>>> +        }
> >>>> +    },
> >>>> +    {
> >>>> +        .fourcc = V4L2_PIX_FMT_NV21M,
> >>>> +        .mbus_codes = {
> >>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>> +        },
> >>>> +        .is_raw = false,
> >>>> +        .registers = {
> >>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
> >>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
> >>>> +        }
> >>>> +    },
> >>>> +    /*
> >>>> +     * RAW uncompressed formats are all packed in 16 bpp.
> >>>> +     * TODO: Expand this list to encompass all possible RAW formats.
> >>>> +     */
> >>>> +    {
> >>>> +        .fourcc = V4L2_PIX_FMT_SRGGB16,
> >>>> +        .mbus_codes = {
> >>>> +            MEDIA_BUS_FMT_SRGGB16_1X16,
> >>>> +        },
> >>>> +        .is_raw = true,
> >>>> +        .registers = {
> >>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
> >>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +        }
> >>>> +    },
> >>>> +    {
> >>>> +        .fourcc = V4L2_PIX_FMT_SBGGR16,
> >>>> +        .mbus_codes = {
> >>>> +            MEDIA_BUS_FMT_SBGGR16_1X16,
> >>>> +        },
> >>>> +        .is_raw = true,
> >>>> +        .registers = {
> >>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
> >>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +        }
> >>>> +    },
> >>>> +    {
> >>>> +        .fourcc = V4L2_PIX_FMT_SGBRG16,
> >>>> +        .mbus_codes = {
> >>>> +            MEDIA_BUS_FMT_SGBRG16_1X16,
> >>>> +        },
> >>>> +        .is_raw = true,
> >>>> +        .registers = {
> >>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
> >>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +        }
> >>>> +    },
> >>>> +    {
> >>>> +        .fourcc = V4L2_PIX_FMT_SGRBG16,
> >>>> +        .mbus_codes = {
> >>>> +            MEDIA_BUS_FMT_SGRBG16_1X16,
> >>>> +        },
> >>>> +        .is_raw = true,
> >>>> +        .registers = {
> >>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
> >>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +        }
> >>>> +    },
> >>>> +};
> >>>> +
> >>>> +static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
> >>>> +                           u32 code)
> >>>> +{
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
> >>>> +        if (fmt->mbus_codes[i] == code)
> >>>> +            return true;
> >>>> +    }
> >>>> +
> >>>> +    return false;
> >>>> +}
> >>>> +
> >>>> +bool mali_c55_format_is_raw(unsigned int mbus_code)
> >>>> +{
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >>>> +        if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
> >>>> +            return mali_c55_fmts[i].is_raw;
> >>>> +    }
> >>>> +
> >>>> +    return false;
> >>>> +}
> >>>> +
> >>>> +static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
> >>>> +{
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >>>> +        if (mali_c55_fmts[i].fourcc == pixelformat)
> >>>> +            return &mali_c55_fmts[i];
> >>>> +    }
> >>>> +
> >>>> +    /*
> >>>> +     * If we find no matching pixelformat, we'll just default to the first
> >>>> +     * one for now.
> >>>> +     */
> >>>> +
> >>>> +    return &mali_c55_fmts[0];
> >>>> +}
> >>>> +
> >>>> +static const char * const capture_device_names[] = {
> >>>> +    "mali-c55 fr",
> >>>> +    "mali-c55 ds",
> >>>> +    "mali-c55 3a stats",
> >>>> +    "mali-c55 params",
> >>
> >> The last two entries are not used AFAICT, neither here, nor in
> >> subsequent patches.
> >>
> >>>> +};
> >>>> +
> >>>> +static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
> >>>> +{
> >>>> +    if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
> >>>> +        return capture_device_names[0];
> >>>> +
> >>>> +    if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
> >>>> +        return capture_device_names[1];
> >>>> +
> >>>> +    return "params/stat not supported yet";
> >>>> +}
> >>
> >> Use cap_dev->vdev.name instead of mali_c55_cap_dev_to_name(cap_dev) and
> >> drop this function.
> >>
> >>>> +
> >>>> +static int mali_c55_link_validate(struct media_link *link)
> >>>> +{
> >>>> +    struct video_device *vdev =
> >>>> + media_entity_to_video_device(link->sink->entity);
> >>>> +    struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
> >>>> +    struct v4l2_subdev *sd =
> >>>> + media_entity_to_v4l2_subdev(link->source->entity);
> >>>> +    const struct v4l2_pix_format_mplane *pix_mp;
> >>>> +    const struct mali_c55_fmt *cap_fmt;
> >>>> +    struct v4l2_subdev_format sd_fmt = {
> >>>> +        .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> >>>> +        .pad = link->source->index,
> >>>> +    };
> >>>> +    int ret;
> >>>> +
> >>>> +    ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
> >>>> +    if (ret)
> >>>> +        return ret;
> >>>> +
> >>>> +    pix_mp = &cap_dev->mode.pix_mp;
> >>>> +    cap_fmt = cap_dev->mode.capture_fmt;
> >>>> +
> >>>> +    if (sd_fmt.format.width != pix_mp->width ||
> >>>> +        sd_fmt.format.height != pix_mp->height) {
> >>>> +        dev_dbg(cap_dev->mali_c55->dev,
> >>>> +            "link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
> >>>> +            link->source->entity->name, link->source->index,
> >>>> +            link->sink->entity->name, link->sink->index,
> >>>> +            sd_fmt.format.width, sd_fmt.format.height,
> >>>> +            pix_mp->width, pix_mp->height);
> >>>> +        return -EPIPE;
> >>>> +    }
> >>>> +
> >>>> +    if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
> >>>> +        dev_dbg(cap_dev->mali_c55->dev,
> >>>> +            "link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format 
> >>>> %p4cc\n",
> >>>> +            link->source->entity->name, link->source->index,
> >>>> +            link->sink->entity->name, link->sink->index,
> >>>> +            sd_fmt.format.code, &pix_mp->pixelformat);
> >>>> +        return -EPIPE;
> >>>> +    }
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct media_entity_operations mali_c55_media_ops = {
> >>>> +    .link_validate = mali_c55_link_validate,
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
> >>>> +                    unsigned int *num_planes, unsigned int sizes[],
> >>>> +                    struct device *alloc_devs[])
> >>>> +{
> >>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    if (*num_planes) {
> >>>> +        if (*num_planes != cap_dev->mode.pix_mp.num_planes)
> >>>> +            return -EINVAL;
> >>>> +
> >>>> +        for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >>>> +            if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
> >>>> +                return -EINVAL;
> >>>> +    } else {
> >>>> +        *num_planes = cap_dev->mode.pix_mp.num_planes;
> >>>> +        for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >>>> +            sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
> >>>> +    }
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_buf_queue(struct vb2_buffer *vb)
> >>>> +{
> >>>> +    struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
> >>>> +    struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> >>>> +    struct mali_c55_buffer *buf = container_of(vbuf,
> >>>> +                           struct mali_c55_buffer, vb);
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    buf->plane_done[MALI_C55_PLANE_Y] = false;
> >>>> +
> >>>> +    /*
> >>>> +     * If we're in a single-plane format we flag the other plane as done
> >>>> +     * already so it's dequeued appropriately later
> >>>> +     */
> >>>> +    buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
> >>>> +
> >>>> +    for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
> >>>> +        unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
> >>>> +
> >>>> +        vb2_set_plane_payload(vb, i, size);
> >>>> +    }
> >>>> +
> >>>> +    spin_lock(&cap_dev->buffers.lock);
> >>>> +    list_add_tail(&buf->queue, &cap_dev->buffers.queue);
> >>>> +    spin_unlock(&cap_dev->buffers.lock);
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_buf_init(struct vb2_buffer *vb)
> >>>> +{
> >>>> +    struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
> >>>> +    struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> >>>> +    struct mali_c55_buffer *buf = container_of(vbuf,
> >>>> +                           struct mali_c55_buffer, vb);
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >>>> +        buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>> +
> >>>> +    guard(spinlock)(&cap_dev->buffers.lock);
> >>>> +
> >>>> +    cap_dev->buffers.curr = cap_dev->buffers.next;
> >>>> +    cap_dev->buffers.next = NULL;
> >>>> +
> >>>> +    if (!list_empty(&cap_dev->buffers.queue)) {
> >>>> +        struct v4l2_pix_format_mplane *pix_mp;
> >>>> +        const struct v4l2_format_info *info;
> >>>> +        u32 *addrs;
> >>>> +
> >>>> +        pix_mp = &cap_dev->mode.pix_mp;
> >>>> +        info = v4l2_format_info(pix_mp->pixelformat);
> >>>> +
> >>>> +        mali_c55_update_bits(mali_c55,
> >>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
> >>>> +        if (cap_dev->mode.capture_fmt->registers.uv_plane)
> >>>> +            mali_c55_update_bits(mali_c55,
> >>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
> >>>> +
> >>>> +        cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
> >>>> +                             struct mali_c55_buffer,
> >>>> +                             queue);
> >>>> +        list_del(&cap_dev->buffers.next->queue);
> >>>> +
> >>>> +        addrs = cap_dev->buffers.next->addrs;
> >>>> +        mali_c55_write(mali_c55,
> >>>> + MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
> >>>> +            addrs[MALI_C55_PLANE_Y]);
> >>>> +        mali_c55_write(mali_c55,
> >>>> + MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
> >>>> +            addrs[MALI_C55_PLANE_UV]);
> >>>> +        mali_c55_write(mali_c55,
> >>>> + MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
> >>>> +            pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
> >>>> +        mali_c55_write(mali_c55,
> >>>> + MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
> >>>> +            pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
> >>>> +            / info->hdiv);
> >>>> +    } else {
> >>>> +        /*
> >>>> +         * If we underflow then we can tell the ISP that we don't want
> >>>> +         * to write out the next frame.
> >>>> +         */
> >>>> +        mali_c55_update_bits(mali_c55,
> >>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>> +        mali_c55_update_bits(mali_c55,
> >>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>> +    }
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
> >>>> +                   unsigned int framecount)
> >>>> +{
> >>>> +    curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> >>>> +    curr_buf->vb.field = V4L2_FIELD_NONE;
> >>
> >> The could be set already when the buffer is queued.
> >>
> >>>> +    curr_buf->vb.sequence = framecount;
> >>>> +    vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
> >>>> +}
> >>>> +
> >>>> +/**
> >>>> + * mali_c55_set_plane_done - mark the plane as written and process the buffer if
> >>>> + *                 both planes are finished.
> >>>> + * @cap_dev:  pointer to the fr or ds pipe output
> >>>> + * @plane:    the plane to mark as completed
> >>>> + *
> >>>> + * The Mali C55 ISP has muliplanar outputs for some formats that come with two
> >>>> + * separate "buffer write completed" interrupts - we need to flag each plane's
> >>>> + * completion and check whether both planes are done - if so, complete the buf
> >>>> + * in vb2.
> >>>> + */
> >>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> >>>> +                 enum mali_c55_planes plane)
> >>>> +{
> >>>> +    struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
> >>>> +    struct mali_c55_buffer *curr_buf;
> >>>> +
> >>>> +    guard(spinlock)(&cap_dev->buffers.lock);
> >>>> +    curr_buf = cap_dev->buffers.curr;
> >>>> +
> >>>> +    /*
> >>>> +     * This _should_ never happen. If no buffer was available from vb2 then
> >>>> +     * we tell the ISP not to bother writing the next frame, which means the
> >>>> +     * interrupts that call this function should never trigger. If it does
> >>>> +     * happen then one of our assumptions is horribly wrong - complain
> >>>> +     * loudly and do nothing.
> >>>> +     */
> >>>> +    if (!curr_buf) {
> >>>> +        dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
> >>>> +            mali_c55_cap_dev_to_name(cap_dev), __func__);
> >>>> +        return;
> >>>> +    }
> >>>> +
> >>>> +    /* If the other plane is also done... */
> >>>> +    if (curr_buf->plane_done[~plane & 1]) {
> >>>> +        mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> >>>> +        cap_dev->buffers.curr = NULL;
> >>>> +        isp->frame_sequence++;
> >>>> +    } else {
> >>>> +        curr_buf->plane_done[plane] = true;
> >>>> +    }
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>> +
> >>>> +    mali_c55_update_bits(mali_c55,
> >>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>> +                 MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>> +    mali_c55_update_bits(mali_c55,
> >>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>> +                 MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>> +
> >>>> +    /*
> >>>> +     * The Mali ISP can hold up to 5 buffer addresses and simply cycle
> >>>> +     * through them, but it's not clear to me that the vb2 queue _guarantees_
> >>>> +     * it will queue buffers to the driver in a fixed order, and ensuring
> >>>> +     * we call vb2_buffer_done() for the right buffer seems to me to add
> >>>> +     * pointless complexity given in multi-context mode we'd need to
> >>>> +     * re-write those registers every frame anyway...so we tell the ISP to
> >>>> +     * use a single register and update it for each frame.
> >>>> +     */
> >>
> >> A single register sounds prone to error conditions. Is it at least
> >> shadowed in the hardware, or do you have to make sure you reprogram it
> >> during the vertical blanking only ?
> >
> > It would have to be reprogrammed during the vertical blanking if we were running in a 
> > configuration with a single config space, otherwise you have the time it takes to process a frame 
> > plus vertical blanking. As I say, it'll have to work like this in multi-context mode anyway.
> >
> > If we want to use the cycling...is it guaranteed that vb2 buffers will always be queued in order?

In which order ?

> >> I'll mostly skip buffer handling in this review, I need to first
> >> understand how the hardware operates to make an informed opinion.
> >>
> >>>> +    mali_c55_update_bits(mali_c55,
> >>>> + MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
> >>>> +            MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
> >>>> +    mali_c55_update_bits(mali_c55,
> >>>> + MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
> >>>> +            MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
> >>>> +
> >>>> +    /*
> >>>> +     * We only queue a buffer in the streamon path if this is the first of
> >>>> +     * the capture devices to start streaming. If the ISP is already running
> >>>> +     * then we rely on the ISP_START interrupt to queue the first buffer for
> >>>> +     * this capture device.
> >>>> +     */
> >>>> +    if (mali_c55->pipe.start_count == 1)
> >>>> +        mali_c55_set_next_buffer(cap_dev);
> >>
> >> I think we'll have to revisit buffer handling to make sure it's 100%
> >> race-free.
> >>
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
> >>>> +                        enum vb2_buffer_state state)
> >>>> +{
> >>>> +    struct mali_c55_buffer *buf, *tmp;
> >>>> +
> >>>> +    guard(spinlock)(&cap_dev->buffers.lock);
> >>>> +
> >>>> +    if (cap_dev->buffers.curr) {
> >>>> + vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
> >>>> +                state);
> >>>> +        cap_dev->buffers.curr = NULL;
> >>>> +    }
> >>>> +
> >>>> +    if (cap_dev->buffers.next) {
> >>>> + vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
> >>>> +                state);
> >>>> +        cap_dev->buffers.next = NULL;
> >>>> +    }
> >>>> +
> >>>> +    list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
> >>>> +        list_del(&buf->queue);
> >>>> +        vb2_buffer_done(&buf->vb.vb2_buf, state);
> >>>> +    }
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
> >>>> +{
> >>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>> +    struct mali_c55_resizer *rzr = cap_dev->rzr;
> >>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
> >>>> +    int ret;
> >>>> +
> >>>> +    guard(mutex)(&isp->lock);
> >>
> >> What's the reason for using the isp lock here and in
> >> mali_c55_vb2_stop_streaming() ? If you need a lock that covers all video
> >> nodes in order to synchronize start/stop, you may want to use the
> >> graph_mutex of the media device instead.
> >
> > It's because I wanted to make sure that the ISP was in a known started/stopped state before 
> > possibly trying to start/stop it, which can be done from either of the two capture devices. This 
> > would go away if we were synchronising with the links anyway.

OK.

> >>>> +
> >>>> +    ret = pm_runtime_resume_and_get(mali_c55->dev);
> >>>> +    if (ret)
> >>>> +        return ret;
> >>>> +
> >>>> +    ret = video_device_pipeline_start(&cap_dev->vdev,
> >>>> +                      &cap_dev->mali_c55->pipe);
> >>>> +    if (ret) {
> >>>> +        dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
> >>>> +            mali_c55_cap_dev_to_name(cap_dev));
> >>
> >> Drop the message or make it dev_dbg() as it can be triggered by
> >> userspace.
> >>
> >>>> +        goto err_pm_put;
> >>>> +    }
> >>>> +
> >>>> +    mali_c55_cap_dev_stream_enable(cap_dev);
> >>>> +    mali_c55_rzr_start_stream(rzr);
> >>>> +
> >>>> +    /*
> >>>> +     * We only start the ISP if we're the only capture device that's
> >>>> +     * streaming. Otherwise, it'll already be active.
> >>>> +     */
> >>
> >> I still think we should use link setup to indicate which video devices
> >> userspace plans to use, and then only start when they're all started.
> >> That includes stats and parameters buffers. We can continue this
> >> discussion in the context of the previous version of the patch series,
> >> or here, up to you.
> >
> > Let's just continue here. I think I called it "clunky" before; from my perspective it's an 
> > unnecessary extra step - we can already signal to the driver that we don't want to use the video 
> > devices by not queuing buffers to them or starting the stream on them and although I understand 

By not starting streaming, perhaps, but by not queuing buffers, no. The
reason is that there's no synchronization between buffer queues. If you
queue

Frame	FR	DS
--------------------
1	x	x
2	x
3	x	x
4	x	x

it will not be distinguishable by the driver from

Frame	FR	DS
--------------------
1	x	x
2	x	x
3	x	x
4	x

> > that that means that one of the two image data capture devices will receive data before the other, 
> > I don't understand why that's considered to be a problem. Possibly that last part is the stickler; 
> > can you explain a bit why it's an issue for one capture queue to start earlier than the other?

Because from a userspace point of view, if you want to capture frames
from both pipelines, you will expect to receive a buffer from each
pipeline for every frame. If that's not guaranteed at stream start, you
will then need to implement synchronization code that will drop buffers
on one pipeline until you get the first buffer on the other pipeline
(assuming you can synchronize them by sequence number). That will be
more work, and can introduce latency.

> >>>> +    if (mali_c55->pipe.start_count == 1) {
> >>>> +        ret = mali_c55_isp_start_stream(isp);
> >>>> +        if (ret)
> >>>> +            goto err_disable_cap_dev;
> >>>> +    }
> >>>> +
> >>>> +    return 0;
> >>>> +
> >>>> +err_disable_cap_dev:
> >>>> +    mali_c55_cap_dev_stream_disable(cap_dev);
> >>>> +    video_device_pipeline_stop(&cap_dev->vdev);
> >>>> +err_pm_put:
> >>>> +    pm_runtime_put(mali_c55->dev);
> >>>> +    mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
> >>>> +
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
> >>>> +{
> >>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>> +    struct mali_c55_resizer *rzr = cap_dev->rzr;
> >>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
> >>>> +
> >>>> +    guard(mutex)(&isp->lock);
> >>>> +
> >>>> +    /*
> >>>> +     * If one of the other capture nodes is streaming, we shouldn't
> >>>> +     * disable the ISP here.
> >>>> +     */
> >>>> +    if (mali_c55->pipe.start_count == 1)
> >>>> +        mali_c55_isp_stop_stream(&mali_c55->isp);
> >>>> +
> >>>> +    mali_c55_rzr_stop_stream(rzr);
> >>>> +    mali_c55_cap_dev_stream_disable(cap_dev);
> >>>> +    mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
> >>>> +    video_device_pipeline_stop(&cap_dev->vdev);
> >>>> +    pm_runtime_put(mali_c55->dev);
> >>
> >> I think runtime PM autosuspend would be very useful, as it will ensure
> >> that stop-reconfigure-start cycles get handled as efficiently as
> >> possible without powering the device down. It could be done on top as a
> >> separate patch.
> >
> > Alright
> >
> >>>> +}
> >>>> +
> >>>> +static const struct vb2_ops mali_c55_vb2_ops = {
> >>>> +    .queue_setup        = &mali_c55_vb2_queue_setup,
> >>>> +    .buf_queue        = &mali_c55_buf_queue,
> >>>> +    .buf_init        = &mali_c55_buf_init,
> >>>> +    .wait_prepare        = vb2_ops_wait_prepare,
> >>>> +    .wait_finish        = vb2_ops_wait_finish,
> >>>> +    .start_streaming    = &mali_c55_vb2_start_streaming,
> >>>> +    .stop_streaming        = &mali_c55_vb2_stop_streaming,
> >>>> +};
> >>>> +
> >>>> +static const struct v4l2_file_operations mali_c55_v4l2_fops = {
> >>>> +    .owner = THIS_MODULE,
> >>>> +    .unlocked_ioctl = video_ioctl2,
> >>>> +    .open = v4l2_fh_open,
> >>>> +    .release = vb2_fop_release,
> >>>> +    .poll = vb2_fop_poll,
> >>>> +    .mmap = vb2_fop_mmap,
> >>>> +};
> >>>> +
> >>>> +static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
> >>>> +{
> >>>> +    const struct mali_c55_fmt *capture_format;
> >>>> +    const struct v4l2_format_info *info;
> >>>> +    struct v4l2_plane_pix_format *plane;
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
> >>>> +    pix_mp->pixelformat = capture_format->fourcc;
> >>>> +
> >>>> +    pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
> >>>> +                  MALI_C55_MAX_WIDTH);
> >>>> +    pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
> >>>> +                   MALI_C55_MAX_HEIGHT);
> >>
> >> Ah, these clamps are right :-)
> >
> > Hurrah!
> >
> >>>> +
> >>>> +    pix_mp->field = V4L2_FIELD_NONE;
> >>>> +    pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>> +    pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>> +    pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
> >>>> +
> >>>> +    info = v4l2_format_info(pix_mp->pixelformat);
> >>
> >> This function may return NULL. That shouldn't be the case as long as it
> >> supports all formats that the C55 driver supports, so I suppose it's
> >> safe.
> >>
> >>>> +    pix_mp->num_planes = info->mem_planes;
> >>>> +    memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
> >>>> +
> >>>> +    pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;
> >>
> >> Does the hardware support configurable line strides ? If so we should
> >> support it.
> >
> > You have to set the line stride in the DMA writer registers, which we do using this same 
> > value...might userspace have set bytesperline already then or something? Or is there some other 
> > place it could be configured?

Userspace can request a specific stride by setting bytesperline, yes. If
that's set, you should honour it (and of course adjust it to a
reasonable [min, max] range as well as align it based on hardware
constraints).

> >>>> +    pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
> >>>> +                       * pix_mp->height;
> >>
> >>     pix_mp->plane_fmt[0].sizeimage = pix_mp->plane_fmt[0].bytesperline
> >>                        * pix_mp->height;
> >>
> >>>> +
> >>>> +    for (i = 1; i < info->comp_planes; i++) {
> >>>> +        plane = &pix_mp->plane_fmt[i];
> >>>> +
> >>>> +        plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
> >>>> +                           info->hdiv);
> >>>> +        plane->sizeimage = DIV_ROUND_UP(
> >>>> +                    plane->bytesperline * pix_mp->height,
> >>>> +                    info->vdiv);
> >>>> +    }
> >>>> +
> >>>> +    if (info->mem_planes == 1) {
> >>>> +        for (i = 1; i < info->comp_planes; i++) {
> >>>> +            plane = &pix_mp->plane_fmt[i];
> >>>> +            pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
> >>>> +        }
> >>>> +    }
> >>
> >> I'm wondering, could v4l2_fill_pixfmt_mp() help ? It doesn't support
> >> configurable strides though :-S Maybe the helper could be improved, if
> >> it's close enough to what you need ?
> >
> > I'll take a look
> >
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>> +                       struct v4l2_format *f)
> >>>> +{
> >>>> +    mali_c55_try_fmt(&f->fmt.pix_mp);
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
> >>>> +                struct v4l2_pix_format_mplane *pix_mp)
> >>>> +{
> >>>> +    const struct mali_c55_fmt *capture_format;
> >>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>> +    const struct v4l2_format_info *info;
> >>>> +
> >>>> +    mali_c55_try_fmt(pix_mp);
> >>>> +    capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
> >>>> +    info = v4l2_format_info(pix_mp->pixelformat);
> >>>> +    if (WARN_ON(!info))
> >>>> +        return;
> >>>> +
> >>>> +    mali_c55_write(mali_c55,
> >>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>> +               capture_format->registers.base_mode);
> >>>> +    mali_c55_write(mali_c55,
> >>>> + MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
> >>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
> >>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
> >>
> >> Could the register writes be moved to stream start time ?
> 
> Sorry missed this one. These are writes to the context's registers
> buffer, not to the hardware. Does it matter that they're not done at
> stream on time?

Writing them here means you'll have to call pm_runtime_resume_and_get()
here. If power is then cut off, registers may or may not lose their
contents, so you would need to write them at stream on time anyway. I
think it's best to move all the hardware configuration at stream on
time.

> >>>> +
> >>>> +    if (info->mem_planes > 1) {
> >>>> +        mali_c55_write(mali_c55,
> >>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>> +                   capture_format->registers.base_mode);
> >>>> +        mali_c55_update_bits(mali_c55,
> >>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>> +                MALI_C55_WRITER_SUBMODE_MASK,
> >>>> +                capture_format->registers.uv_plane);
> >>>> +
> >>>> +        mali_c55_write(mali_c55,
> >>>> + MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
> >>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
> >>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
> >>>> +    }
> >>>> +
> >>>> +    if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
> >>>> +        /*
> >>>> +         * TODO: Figure out the colour matrix coefficients and calculate
> >>>> +         * and write them here.
> >>>> +         */
> >>
> >> Ideally they should also be exposed directly to userspace as ISP
> >> parameters. I would probably go as far as saying that they should come
> >> directly from userspace, and not derived from the colorspace fields.
> >
> > Yes I think I agree, I'll drop the todo from here.
> >
> >>>> +
> >>>> +        mali_c55_write(mali_c55,
> >>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>> +                   MALI_C55_CS_CONV_MATRIX_MASK);
> >>>> +
> >>>> +        if (info->hdiv > 1)
> >>>> +            mali_c55_update_bits(mali_c55,
> >>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>> +                MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
> >>>> +        if (info->vdiv > 1)
> >>>> +            mali_c55_update_bits(mali_c55,
> >>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>> +                MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
> >>>> +        if (info->hdiv > 1 || info->vdiv > 1)
> >>>> +            mali_c55_update_bits(mali_c55,
> >>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>> +                MALI_C55_CS_CONV_FILTER_MASK, 0x01);
> >>>> +    }
> >>>> +
> >>>> +    cap_dev->mode.pix_mp = *pix_mp;
> >>>> +    cap_dev->mode.capture_fmt = capture_format;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>> +                     struct v4l2_format *f)
> >>>> +{
> >>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >>>> +
> >>>> +    if (vb2_is_busy(&cap_dev->queue))
> >>>> +        return -EBUSY;
> >>>> +
> >>>> +    mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>> +                     struct v4l2_format *f)
> >>>> +{
> >>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >>>> +
> >>>> +    f->fmt.pix_mp = cap_dev->mode.pix_mp;
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>> +                        struct v4l2_fmtdesc *f)
> >>>> +{
> >>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >>>> +    unsigned int j = 0;
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >>>> +        if (f->mbus_code &&
> >>>> + !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
> >>>> +                               f->mbus_code))
> >>
> >> Small indentation mistake.
> >>
> >>>> +            continue;
> >>>> +
> >>>> +        /* Downscale pipe can't output RAW formats */
> >>>> +        if (mali_c55_fmts[i].is_raw &&
> >>>> +            cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
> >>>> +            continue;
> >>>> +
> >>>> +        if (j++ == f->index) {
> >>>> +            f->pixelformat = mali_c55_fmts[i].fourcc;
> >>>> +            return 0;
> >>>> +        }
> >>>> +    }
> >>>> +
> >>>> +    return -EINVAL;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_querycap(struct file *file, void *fh,
> >>>> +                 struct v4l2_capability *cap)
> >>>> +{
> >>>> +    strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
> >>>> +    strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
> >>>> +    .vidioc_reqbufs = vb2_ioctl_reqbufs,
> >>>> +    .vidioc_querybuf = vb2_ioctl_querybuf,
> >>>> +    .vidioc_create_bufs = vb2_ioctl_create_bufs,
> >>>> +    .vidioc_qbuf = vb2_ioctl_qbuf,
> >>>> +    .vidioc_expbuf = vb2_ioctl_expbuf,
> >>>> +    .vidioc_dqbuf = vb2_ioctl_dqbuf,
> >>>> +    .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> >>>> +    .vidioc_streamon = vb2_ioctl_streamon,
> >>>> +    .vidioc_streamoff = vb2_ioctl_streamoff,
> >>>> +    .vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
> >>>> +    .vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
> >>>> +    .vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
> >>>> +    .vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
> >>>> +    .vidioc_querycap = mali_c55_querycap,
> >>>> +    .vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
> >>>> +    .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
> >>>> +};
> >>>> +
> >>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    struct v4l2_pix_format_mplane pix_mp;
> >>>> +    struct mali_c55_cap_dev *cap_dev;
> >>>> +    struct video_device *vdev;
> >>>> +    struct vb2_queue *vb2q;
> >>>> +    unsigned int i;
> >>>> +    int ret;
> >>>> +
> >>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
> >>
> >> Moving the inner content to a separate mali_c55_register_capture_dev()
> >> function would increase readability I think, and remove usage of gotos.
> >> I would probably do the same for unregistration too, for consistency.
> >>
> >>>> +        cap_dev = &mali_c55->cap_devs[i];
> >>>> +        vdev = &cap_dev->vdev;
> >>>> +        vb2q = &cap_dev->queue;
> >>>> +
> >>>> +        /*
> >>>> +         * The downscale output pipe is an optional block within the ISP
> >>>> +         * so we need to check whether it's actually been fitted or not.
> >>>> +         */
> >>>> +
> >>>> +        if (i == MALI_C55_CAP_DEV_DS &&
> >>>> +            !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
> >>>> +            continue;
> >>
> >> Given that there's only two capture devices, and one is optional, when
> >> moving the inner code to a separate function you could unroll the loop.
> >> Up to you.
> >>
> >>>> +
> >>>> +        cap_dev->mali_c55 = mali_c55;
> >>>> +        mutex_init(&cap_dev->lock);
> >>>> +        INIT_LIST_HEAD(&cap_dev->buffers.queue);
> >>>> +
> >>>> +        switch (i) {
> >>>> +        case MALI_C55_CAP_DEV_FR:
> >>>> +            cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
> >>>> +            cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
> >>>> +            break;
> >>>> +        case MALI_C55_CAP_DEV_DS:
> >>>> +            cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
> >>>> +            cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
> >>>> +            break;
> >>>> +        default:
> >>
> >> That can't happen.
> >>
> >>>> + mutex_destroy(&cap_dev->lock);
> >>>> +            ret = -EINVAL;
> >>>> +            goto err_destroy_mutex;
> >>>> +        }
> >>>> +
> >>>> +        cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
> >>>> +        ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
> >>>> +        if (ret) {
> >>>> +            mutex_destroy(&cap_dev->lock);
> >>>> +            goto err_destroy_mutex;
> >>>> +        }
> >>>> +
> >>>> +        vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> >>>> +        vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
> >>>> +        vb2q->drv_priv = cap_dev;
> >>>> +        vb2q->mem_ops = &vb2_dma_contig_memops;
> >>>> +        vb2q->ops = &mali_c55_vb2_ops;
> >>>> +        vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
> >>>> +        vb2q->min_queued_buffers = 1;
> >>>> +        vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> >>>> +        vb2q->lock = &cap_dev->lock;
> >>>> +        vb2q->dev = mali_c55->dev;
> >>>> +
> >>>> +        ret = vb2_queue_init(vb2q);
> >>>> +        if (ret) {
> >>>> +            dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
> >>>> +                mali_c55_cap_dev_to_name(cap_dev));
> >>>> +            goto err_cleanup_media_entity;
> >>>> +        }
> >>>> +
> >>>> +        strscpy(cap_dev->vdev.name, capture_device_names[i],
> >>>> +            sizeof(cap_dev->vdev.name));
> >>>> +        vdev->release = video_device_release_empty;
> >>>> +        vdev->fops = &mali_c55_v4l2_fops;
> >>>> +        vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
> >>>> +        vdev->lock = &cap_dev->lock;
> >>>> +        vdev->v4l2_dev = &mali_c55->v4l2_dev;
> >>>> +        vdev->queue = &cap_dev->queue;
> >>>> +        vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> >>>> +                    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
> >>>> +        vdev->entity.ops = &mali_c55_media_ops;
> >>>> +        video_set_drvdata(vdev, cap_dev);
> >>>> +
> >>>> +        memset(&pix_mp, 0, sizeof(pix_mp));
> >>>> +        pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
> >>>> +        pix_mp.width = MALI_C55_DEFAULT_WIDTH;
> >>>> +        pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
> >>>> +        mali_c55_set_format(cap_dev, &pix_mp);
> >>>> +
> >>>> +        ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
> >>>> +        if (ret) {
> >>>> +            dev_err(mali_c55->dev,
> >>>> +                "%s failed to register video device\n",
> >>>> +                mali_c55_cap_dev_to_name(cap_dev));
> >>>> +            goto err_release_vb2q;
> >>>> +        }
> >>>> +    }
> >>>> +
> >>>> +    return 0;
> >>>> +
> >>>> +err_release_vb2q:
> >>>> +    vb2_queue_release(vb2q);
> >>>> +err_cleanup_media_entity:
> >>>> +    media_entity_cleanup(&cap_dev->vdev.entity);
> >>>> +err_destroy_mutex:
> >>>> +    mutex_destroy(&cap_dev->lock);
> >>>> +    mali_c55_unregister_capture_devs(mali_c55);
> >>>> +
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    struct mali_c55_cap_dev *cap_dev;
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
> >>>> +        cap_dev = &mali_c55->cap_devs[i];
> >>>> +
> >>>> +        if (!video_is_registered(&cap_dev->vdev))
> >>>> +            continue;
> >>>> +
> >>>> +        vb2_video_unregister_device(&cap_dev->vdev);
> >>>> +        media_entity_cleanup(&cap_dev->vdev.entity);
> >>>> +        mutex_destroy(&cap_dev->lock);
> >>>> +    }
> >>>> +}
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h 
> >>>> b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>> new file mode 100644
> >>>> index 000000000000..2d0c4d152beb
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>> @@ -0,0 +1,266 @@
> >>>> +/* SPDX-License-Identifier: GPL-2.0 */
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Common definitions
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#ifndef _MALI_C55_COMMON_H
> >>>> +#define _MALI_C55_COMMON_H
> >>>> +
> >>>> +#include <linux/clk.h>
> >>>> +#include <linux/io.h>
> >>>> +#include <linux/list.h>
> >>>> +#include <linux/mutex.h>
> >>>> +#include <linux/scatterlist.h>
> >>>
> >>> I don't think this is needed. You're however missing spinlock.h.
> >>>
> >>>> +#include <linux/videodev2.h>
> >>>> +
> >>>> +#include <media/media-device.h>
> >>>> +#include <media/v4l2-async.h>
> >>>> +#include <media/v4l2-ctrls.h>
> >>>> +#include <media/v4l2-dev.h>
> >>>> +#include <media/v4l2-device.h>
> >>>> +#include <media/v4l2-subdev.h>
> >>>> +#include <media/videobuf2-core.h>
> >>>> +#include <media/videobuf2-v4l2.h>
> >>>> +
> >>>> +#define MALI_C55_DRIVER_NAME        "mali-c55"
> >>>> +
> >>>> +/* min and max values for the image sizes */
> >>>> +#define MALI_C55_MIN_WIDTH        640U
> >>>> +#define MALI_C55_MIN_HEIGHT        480U
> >>>> +#define MALI_C55_MAX_WIDTH        8192U
> >>>> +#define MALI_C55_MAX_HEIGHT        8192U
> >>>> +#define MALI_C55_DEFAULT_WIDTH        1920U
> >>>> +#define MALI_C55_DEFAULT_HEIGHT        1080U
> >>>> +
> >>>> +#define MALI_C55_DEFAULT_MEDIA_BUS_FMT MEDIA_BUS_FMT_RGB121212_1X36
> >>>> +
> >>>> +struct mali_c55;
> >>>> +struct mali_c55_cap_dev;
> >>>> +struct platform_device;
> >>>
> >>> You should also forward-declare
> >>>
> >>> struct device;
> >>> struct dma_chan;
> >>> struct resource;
> >>>
> >>>> +
> >>>> +static const char * const mali_c55_clk_names[] = {
> >>>> +    "aclk",
> >>>> +    "hclk",
> >>>> +};
> >>>
> >>> This will end up duplicating the array in each compilation unit, not
> >>> great. Move it to mali-c55-core.c. You use it in this file just for its
> >>> size, replace that with a macro that defines the size, or allocate
> >>> mali_c55.clks dynamically with devm_kcalloc().
> >>>
> >>>> +
> >>>> +enum mali_c55_interrupts {
> >>>> +    MALI_C55_IRQ_ISP_START,
> >>>> +    MALI_C55_IRQ_ISP_DONE,
> >>>> +    MALI_C55_IRQ_MCM_ERROR,
> >>>> +    MALI_C55_IRQ_BROKEN_FRAME_ERROR,
> >>>> +    MALI_C55_IRQ_MET_AF_DONE,
> >>>> +    MALI_C55_IRQ_MET_AEXP_DONE,
> >>>> +    MALI_C55_IRQ_MET_AWB_DONE,
> >>>> +    MALI_C55_IRQ_AEXP_1024_DONE,
> >>>> +    MALI_C55_IRQ_IRIDIX_MET_DONE,
> >>>> +    MALI_C55_IRQ_LUT_INIT_DONE,
> >>>> +    MALI_C55_IRQ_FR_Y_DONE,
> >>>> +    MALI_C55_IRQ_FR_UV_DONE,
> >>>> +    MALI_C55_IRQ_DS_Y_DONE,
> >>>> +    MALI_C55_IRQ_DS_UV_DONE,
> >>>> +    MALI_C55_IRQ_LINEARIZATION_DONE,
> >>>> +    MALI_C55_IRQ_RAW_FRONTEND_DONE,
> >>>> +    MALI_C55_IRQ_NOISE_REDUCTION_DONE,
> >>>> +    MALI_C55_IRQ_IRIDIX_DONE,
> >>>> +    MALI_C55_IRQ_BAYER2RGB_DONE,
> >>>> +    MALI_C55_IRQ_WATCHDOG_TIMER,
> >>>> +    MALI_C55_IRQ_FRAME_COLLISION,
> >>>> +    MALI_C55_IRQ_UNUSED,
> >>>> +    MALI_C55_IRQ_DMA_ERROR,
> >>>> +    MALI_C55_IRQ_INPUT_STOPPED,
> >>>> +    MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
> >>>> +    MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
> >>>> +    MALI_C55_NUM_IRQ_BITS
> >>>
> >>> Those are register bits, I think they belong to mali-c55-registers.h,
> >>> and should probably be macros instead of an enum.
> >>>
> >>>> +};
> >>>> +
> >>>> +enum mali_c55_isp_pads {
> >>>> +    MALI_C55_ISP_PAD_SINK_VIDEO,
> >>>
> >>> As there's a single sink pad, maybe MALI_C55_ISP_PAD_SINK ? Ah, you're
> >>> probably preparing for ISP parameters support. It's fine.
> >>>
> >>>> +    MALI_C55_ISP_PAD_SOURCE,
> >>>
> >>> Then maybe this should be named MALI_C55_ISP_PAD_SOURCE_VIDEO as I
> >>> assume there will be a stats source pad.
> >>>
> >>>> +    MALI_C55_ISP_PAD_SOURCE_BYPASS,
> >>>> +    MALI_C55_ISP_NUM_PADS,
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_tpg {
> >>>> +    struct mali_c55 *mali_c55;
> >>>> +    struct v4l2_subdev sd;
> >>>> +    struct media_pad pad;
> >>>> +    struct mutex lock;
> >>>> +    struct mali_c55_tpg_ctrls {
> >>>> +        struct v4l2_ctrl_handler handler;
> >>>> +        struct v4l2_ctrl *test_pattern;
> >>>
> >>> Set but never used. You can drop it.
> >>>
> >>>> +        struct v4l2_ctrl *hblank;
> >>>
> >>> Set and used only once, in the same function. You can make it a local
> >>> variable.
> >>>
> >>>> +        struct v4l2_ctrl *vblank;
> >>>> +    } ctrls;
> >>>> +};
> >>>
> >>> I wonder if this file should be split, with mali-c55-capture.h,
> >>> mali-c55-core.h, mali-c55-isp.h, ... I think it could increase
> >>> readability by clearly separating the different elements. Up to you.
> >>>
> >>>> +
> >>>> +struct mali_c55_isp {
> >>>> +    struct mali_c55 *mali_c55;
> >>>> +    struct v4l2_subdev sd;
> >>>> +    struct media_pad pads[MALI_C55_ISP_NUM_PADS];
> >>>> +    struct media_pad *remote_src;
> >>>> +    struct v4l2_async_notifier notifier;
> >>>
> >>> I'm tempted to move the notifier to mali_c55, as it's related to
> >>> components external to the whole ISP, not to the ISP subdev itself.
> >>> Could you give it a try, to see if it could be done without any drawback
> >>> ?
> >>>
> >>>> +    struct mutex lock;
> >>>
> >>> Locks require a comment to explain what they protect. Same below where
> >>> applicable (for both mutexes and spinlocks).
> >>>
> >>>> +    unsigned int frame_sequence;
> >>>> +};
> >>>> +
> >>>> +enum mali_c55_resizer_ids {
> >>>> +    MALI_C55_RZR_FR,
> >>>> +    MALI_C55_RZR_DS,
> >>>> +    MALI_C55_NUM_RZRS,
> >>>
> >>> The usual abbreviation for "resizer" in drivers/media/ is "rsz", not
> >>> "rzr". I would have said we can leave it as-is as changing it would be a
> >>> bit annoying, but I then realized that "rzr" is not just unusual, it's
> >>> actually not used at all. Would you mind applying a sed globally ? :-)
> >>>
> >>>> +};
> >>>> +
> >>>> +enum mali_c55_rzr_pads {
> >>>
> >>> Same enums/structs use abbreviations, some don't. Consistency would
> >>> help.
> >>>
> >>>> +    MALI_C55_RZR_SINK_PAD,
> >>>> +    MALI_C55_RZR_SOURCE_PAD,
> >>>> +    MALI_C55_RZR_SINK_BYPASS_PAD,
> >>>> +    MALI_C55_RZR_NUM_PADS
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_resizer {
> >>>> +    struct mali_c55 *mali_c55;
> >>>> +    struct mali_c55_cap_dev *cap_dev;
> >>>> +    enum mali_c55_resizer_ids id;
> >>>> +    struct v4l2_subdev sd;
> >>>> +    struct media_pad pads[MALI_C55_RZR_NUM_PADS];
> >>>> +    unsigned int num_routes;
> >>>> +    bool streaming;
> >>>> +};
> >>>> +
> >>>> +enum mali_c55_cap_devs {
> >>>> +    MALI_C55_CAP_DEV_FR,
> >>>> +    MALI_C55_CAP_DEV_DS,
> >>>> +    MALI_C55_NUM_CAP_DEVS
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_fmt {
> >>>
> >>> mali_c55_format_info would be a better name I think, as this stores
> >>> format information, not formats.
> >>>
> >>>> +    u32 fourcc;
> >>>> +    unsigned int mbus_codes[2];
> >>>
> >>> A comment to explain why we have two media bus codes would be useful.
> >>> You can document the whole structure if desired :-)
> >>>
> >>>> +    bool is_raw;
> >>>> +    struct mali_c55_fmt_registers {
> >>>
> >>> Make it an anonymous structure, it's never used anywhere else.
> >>>
> >>>> +        unsigned int base_mode;
> >>>> +        unsigned int uv_plane;
> >>>
> >>> If those are register field values, use u32 instead of unsigned int.
> >>>
> >>>> +    } registers;
> >>>
> >>> It's funny, we tend to abbreviate different things, I would have used
> >>> "regs" here but written "format" in full in the structure name :-)
> >>>
> >>>> +};
> >>>> +
> >>>> +enum mali_c55_isp_bayer_order {
> >>>> +    MALI_C55_BAYER_ORDER_RGGB,
> >>>> +    MALI_C55_BAYER_ORDER_GRBG,
> >>>> +    MALI_C55_BAYER_ORDER_GBRG,
> >>>> +    MALI_C55_BAYER_ORDER_BGGR
> >>>
> >>> These are registers values too, they belong to mali-c55-registers.h.
> >>>
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_isp_fmt {
> >>>
> >>> mali_c55_isp_format_info
> >>>
> >>>> +    u32 code;
> >>>> +    enum v4l2_pixel_encoding encoding;
> >>>
> >>> Here you use v4l2_pixel_encoding, for mali_c55_fmt you use is_raw. Maybe
> >>> pick the same option for both structures ?
> >>>
> >>>> +    enum mali_c55_isp_bayer_order order;
> >>>> +};
> >>>> +
> >>>> +enum mali_c55_planes {
> >>>> +    MALI_C55_PLANE_Y,
> >>>> +    MALI_C55_PLANE_UV,
> >>>> +    MALI_C55_NUM_PLANES
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_buffer {
> >>>> +    struct vb2_v4l2_buffer vb;
> >>>> +    bool plane_done[MALI_C55_NUM_PLANES];
> >>>
> >>> I think tracking the pending state would simplify the logic in
> >>> mali_c55_set_plane_done(), which would become
> >>>
> >>>     curr_buf->plane_pending[plane] = false;
> >>>
> >>>     if (!curr_buf->plane_pending[0] && !curr_buf->plane_pending[1]) {
> >>>         mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> >>>         cap_dev->buffers.curr = NULL;
> >>>         isp->frame_sequence++;
> >>>     }
> >>>
> >>> Or a counter may be even easier (and would consume less memory).
> >>>
> >>>> +    struct list_head queue;
> >>>> +    u32 addrs[MALI_C55_NUM_PLANES];
> >>>
> >>> This stores DMA addresses, use dma_addr_t.
> >>>
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_cap_dev {
> >>>> +    struct mali_c55 *mali_c55;
> >>>> +    struct mali_c55_resizer *rzr;
> >>>> +    struct video_device vdev;
> >>>> +    struct media_pad pad;
> >>>> +    struct vb2_queue queue;
> >>>> +    struct mutex lock;
> >>>> +    unsigned int reg_offset;
> >>>
> >>> Manual handling of the offset everywhere, with parametric macros for the
> >>> resizer register addresses, isn't very nice. Introduce resizer-specific
> >>> accessors and capture-specific accessors (mali_c55_rsz_write(), ...)
> >>> that take a mali_c55_resizer or mali_c55_cap_dev pointer, and handle the
> >>> offset there. The register macros should loose their offset parameter.
> >>>
> >>> You could also use a single set of accessors that would become
> >>> path/output-specific (mali_c55_path_write() ? mali_c55_output_write()
> >>> ?), that may make the code easier to read.
> >>>
> >>> You can also replace reg_offset with a void __iomem * base, which would
> >>> avoid the computation at runtime.
> >>>
> >>>> +
> >>>> +    struct mali_c55_mode {
> >>>
> >>> Make the structure anonymous.
> >>>
> >>>> +        const struct mali_c55_fmt *capture_fmt;
> >>>> +        struct v4l2_pix_format_mplane pix_mp;
> >>>> +    } mode;
> >>>
> >>> What's a "mode" ? I think I'd name this
> >>>
> >>>     struct {
> >>>         const struct mali_c55_fmt *info;
> >>>         struct v4l2_pix_format_mplane format;
> >>>     } format;
> >>>
> >>> Or you could just drop the structure and have
> >>>
> >>>     const struct mali_c55_fmt *format_info;
> >>>     struct v4l2_pix_format_mplane format;
> >>>
> >>> or something similar.
> >>>
> >>>> +
> >>>> +    struct {
> >>>> +        spinlock_t lock;
> >>>> +        struct list_head queue;
> >>>> +        struct mali_c55_buffer *curr;
> >>>> +        struct mali_c55_buffer *next;
> >>>> +    } buffers;
> >>>> +
> >>>> +    bool streaming;
> >>>> +};
> >>>> +
> >>>> +enum mali_c55_config_spaces {
> >>>> +    MALI_C55_CONFIG_PING,
> >>>> +    MALI_C55_CONFIG_PONG,
> >>>> +    MALI_C55_NUM_CONFIG_SPACES
> >>>
> >>> The last enumerator is not used.
> >>>
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_ctx {
> >>>
> >>> mali_c55_context ?
> >>>
> >>>> +    struct mali_c55 *mali_c55;
> >>>> +    void *registers;
> >>>
> >>> Please document this structure and explain that this field points to a
> >>> copy of the register space in system memory, I was about to write you're
> >>> missing __iomem :-)
> >>>
> >>>> +    phys_addr_t base;
> >>>> +    spinlock_t lock;
> >>>> +    struct list_head list;
> >>>> +};
> >>>> +
> >>>> +struct mali_c55 {
> >>>> +    struct device *dev;
> >>>> +    struct resource *res;
> >>>
> >>> You could possibly drop this field by passing the physical address of
> >>> the register space from mali_c55_probe() to mali_c55_init_context() as a
> >>> function parameter.
> >>>
> >>>> +    void __iomem *base;
> >>>> +    struct dma_chan *channel;
> >>>> +    struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
> >>>> +
> >>>> +    u16 capabilities;
> >>>> +    struct media_device media_dev;
> >>>> +    struct v4l2_device v4l2_dev;
> >>>> +    struct media_pipeline pipe;
> >>>> +
> >>>> +    struct mali_c55_tpg tpg;
> >>>> +    struct mali_c55_isp isp;
> >>>> +    struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
> >>>> +    struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
> >>>> +
> >>>> +    struct list_head contexts;
> >>>> +    enum mali_c55_config_spaces next_config;
> >>>> +};
> >>>> +
> >>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
> >>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
> >>>> +          bool force_hardware);
> >>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
> >>>> +              u32 mask, u32 val);
> >>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
> >>>> +              enum mali_c55_config_spaces cfg_space);
> >>>> +
> >>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55);
> >>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55);
> >>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
> >>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
> >>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55);
> >>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
> >>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
> >>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
> >>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
> >>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> >>>> +                 enum mali_c55_planes plane);
> >>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
> >>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
> >>>> +
> >>>> +bool mali_c55_format_is_raw(unsigned int mbus_code);
> >>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
> >>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
> >>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
> >>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
> >>>> +
> >>>> +const struct mali_c55_isp_fmt *
> >>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
> >>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
> >>>> +#define for_each_mali_isp_fmt(fmt)\
> >>>
> >>> #define for_each_mali_isp_fmt(fmt) \
> >>>
> >>>> +    for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
> >>>
> >>> Looks like parentheses were on sale :-)
> >>>
> >>>     for ((fmt) = NULL; (fmt) = mali_c55_isp_fmt_next(fmt); )
> >>>
> >>> This macro is used in two places only, in the mali-c55-isp.c file where
> >>> open-coding the loop without using mali_c55_isp_fmt_next() would be more
> >>> efficient, and in mali-c55-resizer.c where a function to return format
> >>> i'th would be more efficient. I think you can drop the macro and the
> >>> mali_c55_isp_fmt_next() function.
> >>>
> >>>> +
> >>>> +#endif /* _MALI_C55_COMMON_H */
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c 
> >>>> b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>> new file mode 100644
> >>>> index 000000000000..50caf5ee7474
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>> @@ -0,0 +1,767 @@
> >>>> +// SPDX-License-Identifier: GPL-2.0
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Core driver code
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#include <linux/bitops.h>
> >>>> +#include <linux/cleanup.h>
> >>>> +#include <linux/clk.h>
> >>>> +#include <linux/delay.h>
> >>>> +#include <linux/device.h>
> >>>> +#include <linux/dmaengine.h>
> >>>> +#include <linux/dma-mapping.h>
> >>>> +#include <linux/interrupt.h>
> >>>> +#include <linux/iopoll.h>
> >>>> +#include <linux/ioport.h>
> >>>> +#include <linux/mod_devicetable.h>
> >>>> +#include <linux/of.h>
> >>>> +#include <linux/of_reserved_mem.h>
> >>>> +#include <linux/platform_device.h>
> >>>> +#include <linux/pm_runtime.h>
> >>>> +#include <linux/scatterlist.h>
> >>>
> >>> I don't think this is needed.
> >>>
> >>> Missing slab.h.
> >>>
> >>>> +#include <linux/string.h>
> >>>> +
> >>>> +#include <media/media-entity.h>
> >>>> +#include <media/v4l2-device.h>
> >>>> +#include <media/videobuf2-dma-contig.h>
> >>>> +
> >>>> +#include "mali-c55-common.h"
> >>>> +#include "mali-c55-registers.h"
> >>>> +
> >>>> +static const char * const mali_c55_interrupt_names[] = {
> >>>> +    [MALI_C55_IRQ_ISP_START] = "ISP start",
> >>>> +    [MALI_C55_IRQ_ISP_DONE] = "ISP done",
> >>>> +    [MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
> >>>> +    [MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
> >>>> +    [MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
> >>>> +    [MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
> >>>> +    [MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
> >>>> +    [MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
> >>>> +    [MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
> >>>> +    [MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
> >>>> +    [MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
> >>>> +    [MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
> >>>> +    [MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
> >>>> +    [MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
> >>>> +    [MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
> >>>> +    [MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
> >>>> +    [MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
> >>>> +    [MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
> >>>> +    [MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
> >>>> +    [MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
> >>>> +    [MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
> >>>> +    [MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
> >>>> +    [MALI_C55_IRQ_DMA_ERROR] = "DMA error",
> >>>> +    [MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
> >>>> +    [MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
> >>>> +    [MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
> >>>> +};
> >>>> +
> >>>> +static unsigned int config_space_addrs[] = {
> >>>
> >>> const
> >>>
> >>>> +    [MALI_C55_CONFIG_PING] = 0x0AB6C,
> >>>> +    [MALI_C55_CONFIG_PONG] = 0x22B2C,
> >>>
> >>> Lowercase hex constants.
> >>>
> >>> Don't the values belong to mali-c55-registers.h ?
> >>>
> >>>> +};
> >>>> +
> >>>> +/* System IO
> >>>
> >>> /*
> >>>   * System IO
> >>>
> >>>> + *
> >>>> + * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
> >>>> + * and 'pong'), with the  expectation that the 'active' space will be left
> >>>
> >>> s/the  /the /
> >>>
> >>>> + * untouched whilst a frame is being processed and the 'inactive' space
> >>>> + * configured ready to be passed during the blanking period before the next
> >>>
> >>> s/to be passed/to be switched to/ ?
> >>>
> >>>> + * frame processing starts. These spaces should ideally be set via DMA transfer
> >>>> + * from a buffer rather than through individual register set operations. There
> >>>> + * is also a shared global register space which should be set normally. Of
> >>>> + * course, the ISP might be included in a system which lacks a suitable DMA
> >>>> + * engine, and the second configuration space might not be fitted at all, which
> >>>> + * means we need to support four scenarios:
> >>>> + *
> >>>> + * 1. Multi config space, with DMA engine.
> >>>> + * 2. Multi config space, no DMA engine.
> >>>> + * 3. Single config space, with DMA engine.
> >>>> + * 4. Single config space, no DMA engine.
> >>>> + *
> >>>> + * The first case is very easy, but the rest present annoying problems. The best
> >>>> + * way to solve them seems to be simply to replicate the concept of DMAing over
> >>>> + * the configuration buffer even if there's no DMA engine on the board, for
> >>>> + * which we rely on memcpy. To facilitate this any read/write call that is made
> >>>> + * to an address within those config spaces should infact be directed to a
> >>>> + * buffer that was allocated to hold them rather than the IO memory itself. The
> >>>> + * actual copy of that buffer to IO mem will happen on interrupt.
> >>>> + */
> >>>> +
> >>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
> >>>> +{
> >>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>> +
> >>>> +    if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
> >>>> +        spin_lock(&ctx->lock);
> >>>> +        addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
> >>>> +        ((u32 *)ctx->registers)[addr] = val;
> >>>> +        spin_unlock(&ctx->lock);
> >>>> +
> >>>> +        return;
> >>>> +    }
> >>>
> >>> Ouch. This is likely the second comment you really won't like (after the
> >>> comment regarding mali_c55_update_bits() at the very top). I apologize
> >>> in advance.
> >>>
> >>> I really don't like this. Directing writes either to hardware registers
> >>> or to the shadow registers in the context makes the callers of the
> >>> read/write accessors very hard to read. The probe code, for instance,
> >>> mixes writes to hardware registers and writes to the context shadow
> >>> registers to initialize the value of some of the shadow registers.
> >>>
> >>> I'd like to split the read/write accessors into functions that access
> >>> the hardware registers (that's easy) and functions that access the
> >>> shadow registers. I think the latter should receive a mali_c55_ctx
> >>> pointer instead of a mali_c55 pointer to prepare for multi-context
> >>> support.
> >>>
> >>> You can add WARN_ON() guards to the two sets of functions, to ensure
> >>> that no register from the "other" space gets passed to the wrong
> >>> function by mistake.
> >>>
> >>>> +
> >>>> +    writel(val, mali_c55->base + addr);
> >>>> +}
> >>>> +
> >>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
> >>>> +          bool force_hardware)
> >>>
> >>> force_hardware is never set to true.
> >>>
> >>>> +{
> >>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>> +    u32 val;
> >>>> +
> >>>> +    if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
> >>>> +        spin_lock(&ctx->lock);
> >>>> +        addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
> >>>> +        val = ((u32 *)ctx->registers)[addr];
> >>>> +        spin_unlock(&ctx->lock);
> >>>> +
> >>>> +        return val;
> >>>> +    }
> >>>> +
> >>>> +    return readl(mali_c55->base + addr);
> >>>> +}
> >>>> +
> >>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
> >>>> +              u32 mask, u32 val)
> >>>> +{
> >>>> +    u32 orig, tmp;
> >>>> +
> >>>> +    orig = mali_c55_read(mali_c55, addr, false);
> >>>> +
> >>>> +    tmp = orig & ~mask;
> >>>> +    tmp |= (val << (ffs(mask) - 1)) & mask;
> >>>> +
> >>>> +    if (tmp != orig)
> >>>> +        mali_c55_write(mali_c55, addr, tmp);
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
> >>>> +                 dma_addr_t dst, enum dma_data_direction dir)
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>> +    struct dma_async_tx_descriptor *tx;
> >>>> +    enum dma_status status;
> >>>> +    dma_cookie_t cookie;
> >>>> +
> >>>> +    tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
> >>>> +                       MALI_C55_CONFIG_SPACE_SIZE, 0);
> >>>> +    if (!tx) {
> >>>> +        dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
> >>>> +        return -EIO;
> >>>> +    }
> >>>> +
> >>>> +    cookie = dmaengine_submit(tx);
> >>>> +    if (dma_submit_error(cookie)) {
> >>>> +        dev_err(mali_c55->dev, "error submitting dma transfer\n");
> >>>> +        return -EIO;
> >>>> +    }
> >>>> +
> >>>> +    status = dma_sync_wait(mali_c55->channel, cookie);
> >>>
> >>> I've just realized this performs a busy-wait :-S See the comment in the
> >>> probe function about the threaded IRQ handler. I think we'll need to
> >>> rework all this. It could be done on top though.
> >>>
> >>>> +    if (status != DMA_COMPLETE) {
> >>>> +        dev_err(mali_c55->dev, "dma transfer failed\n");
> >>>> +        return -EIO;
> >>>> +    }
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
> >>>> +                 enum mali_c55_config_spaces cfg_space)
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>> +    struct device *dma_dev = mali_c55->channel->device->dev;
> >>>> +    dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
> >>>> +    dma_addr_t dst;
> >>>> +    int ret;
> >>>> +
> >>>> +    guard(spinlock)(&ctx->lock);
> >>>> +
> >>>> +    dst = dma_map_single(dma_dev, ctx->registers,
> >>>> +                 MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
> >>>> +    if (dma_mapping_error(dma_dev, dst)) {
> >>>> +        dev_err(mali_c55->dev, "failed to map DMA addr\n");
> >>>> +        return -EIO;
> >>>> +    }
> >>>> +
> >>>> +    ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
> >>>> +    dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
> >>>> +             DMA_FROM_DEVICE);
> >>>> +
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
> >>>> +               enum mali_c55_config_spaces cfg_space)
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>> +    struct device *dma_dev = mali_c55->channel->device->dev;
> >>>> +    dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
> >>>> +    dma_addr_t src;
> >>>> +    int ret;
> >>>> +
> >>>> +    guard(spinlock)(&ctx->lock);
> >>>
> >>> The code below can take a large amount of time, holding a spinlock will
> >>> disable interrupts on the local CPU, that's not good :-(
> >>>
> >>>> +
> >>>> +    src = dma_map_single(dma_dev, ctx->registers,
> >>>> +                 MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
> >>>> +    if (dma_mapping_error(dma_dev, src)) {
> >>>> +        dev_err(mali_c55->dev, "failed to map DMA addr\n");
> >>>> +        return -EIO;
> >>>> +    }
> >>>> +
> >>>> +    ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
> >>>> +    dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
> >>>> +             DMA_TO_DEVICE);
> >>>> +
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_config_read(struct mali_c55_ctx *ctx,
> >>>> +                enum mali_c55_config_spaces cfg_space)
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>> +
> >>>> +    if (mali_c55->channel) {
> >>>> +        return mali_c55_dma_read(ctx, cfg_space);
> >>>
> >>> As this function is used at probe time only, to initialize the context,
> >>> I think DMA is overkill.
> >>>
> >>>> +    } else {
> >>>> +        memcpy_fromio(ctx->registers,
> >>>> +                  mali_c55->base + config_space_addrs[cfg_space],
> >>>> +                  MALI_C55_CONFIG_SPACE_SIZE);
> >>>> +        return 0;
> >>>> +    }
> >>>> +}
> >>>> +
> >>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
> >>>> +              enum mali_c55_config_spaces cfg_space)
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>> +
> >>>> +    if (mali_c55->channel) {
> >>>> +        return mali_c55_dma_write(ctx, cfg_space);
> >>>> +    } else {
> >>>> +        memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
> >>>> +                ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
> >>>> +        return 0;
> >>>> +    }
> >>>
> >>> Could you measure the time it typically takes to write the registers
> >>> using DMA compared to using memcpy_toio() ?
> >>>
> >>>> +}
> >>>> +
> >>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);
> >>>
> >>> I think it's too early to tell how multi-context support will look like.
> >>> I'm fine keeping mali_c55_get_active_context() as changing that would be
> >>> very intrusive (even if I think it will need to be changed), but the
> >>> list of contexts is neither the mechanism we'll use, nor something we
> >>> need now. Drop the list, embed the context in struct mali_c55, and
> >>> return the pointer to that single context from this function.
> >>>
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_remove_links(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    unsigned int i;
> >>>> +
> >>>> + media_entity_remove_links(&mali_c55->tpg.sd.entity);
> >>>> + media_entity_remove_links(&mali_c55->isp.sd.entity);
> >>>> +
> >>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; i++)
> >>>> + media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
> >>>> +
> >>>> +    for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
> >>>> + media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_create_links(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    int ret;
> >>>> +
> >>>> +    /* Test pattern generator to ISP */
> >>>> +    ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
> >>>> +                    &mali_c55->isp.sd.entity,
> >>>> +                    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
> >>>> +    if (ret) {
> >>>> +        dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
> >>>> +        goto err_remove_links;
> >>>> +    }
> >>>> +
> >>>> +    /* Full resolution resizer pipe. */
> >>>> +    ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >>>> +            MALI_C55_ISP_PAD_SOURCE,
> >>>> + &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
> >>>> +            MALI_C55_RZR_SINK_PAD,
> >>>> +            MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>> +    if (ret) {
> >>>> +        dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
> >>>> +        goto err_remove_links;
> >>>> +    }
> >>>> +
> >>>> +    /* Full resolution bypass. */
> >>>> +    ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >>>> +                    MALI_C55_ISP_PAD_SOURCE_BYPASS,
> >>>> + &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
> >>>> +                    MALI_C55_RZR_SINK_BYPASS_PAD,
> >>>> +                    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>> +    if (ret) {
> >>>> +        dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
> >>>> +        goto err_remove_links;
> >>>> +    }
> >>>> +
> >>>> +    /* Resizer pipe to video capture nodes. */
> >>>> +    ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
> >>>> +            MALI_C55_RZR_SOURCE_PAD,
> >>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
> >>>> +            0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>> +    if (ret) {
> >>>> +        dev_err(mali_c55->dev,
> >>>> +            "failed to link FR resizer and video device\n");
> >>>> +        goto err_remove_links;
> >>>> +    }
> >>>> +
> >>>> +    /* The downscale pipe is an optional hardware block */
> >>>> +    if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
> >>>> +        ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >>>> +            MALI_C55_ISP_PAD_SOURCE,
> >>>> + &mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
> >>>> +            MALI_C55_RZR_SINK_PAD,
> >>>> +            MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>> +        if (ret) {
> >>>> +            dev_err(mali_c55->dev,
> >>>> +                "failed to link ISP and DS resizer\n");
> >>>> +            goto err_remove_links;
> >>>> +        }
> >>>> +
> >>>> +        ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
> >>>> +            MALI_C55_RZR_SOURCE_PAD,
> >>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
> >>>> +            0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>> +        if (ret) {
> >>>> +            dev_err(mali_c55->dev,
> >>>> +                "failed to link DS resizer and video device\n");
> >>>> +            goto err_remove_links;
> >>>> +        }
> >>>> +    }
> >>>> +
> >>>> +    return 0;
> >>>> +
> >>>> +err_remove_links:
> >>>> +    mali_c55_remove_links(mali_c55);
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    mali_c55_unregister_tpg(mali_c55);
> >>>> +    mali_c55_unregister_isp(mali_c55);
> >>>> +    mali_c55_unregister_resizers(mali_c55);
> >>>> +    mali_c55_unregister_capture_devs(mali_c55);
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_register_entities(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    int ret;
> >>>> +
> >>>> +    ret = mali_c55_register_tpg(mali_c55);
> >>>> +    if (ret)
> >>>> +        return ret;
> >>>> +
> >>>> +    ret = mali_c55_register_isp(mali_c55);
> >>>> +    if (ret)
> >>>> +        goto err_unregister_entities;
> >>>> +
> >>>> +    ret = mali_c55_register_resizers(mali_c55);
> >>>> +    if (ret)
> >>>> +        goto err_unregister_entities;
> >>>> +
> >>>> +    ret = mali_c55_register_capture_devs(mali_c55);
> >>>> +    if (ret)
> >>>> +        goto err_unregister_entities;
> >>>> +
> >>>> +    ret = mali_c55_create_links(mali_c55);
> >>>> +    if (ret)
> >>>> +        goto err_unregister_entities;
> >>>> +
> >>>> +    return 0;
> >>>> +
> >>>> +err_unregister_entities:
> >>>> +    mali_c55_unregister_entities(mali_c55);
> >>>> +
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    u32 product, version, revision, capabilities;
> >>>> +
> >>>> +    product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
> >>>> +    version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
> >>>> +    revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
> >>>> +
> >>>> +    dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
> >>>> +         product, version, revision);
> >>>> +
> >>>> +    capabilities = mali_c55_read(mali_c55,
> >>>> +                     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
> >>>> +                     false);
> >>>> +    mali_c55->capabilities = (capabilities & 0xffff);
> >>>> +
> >>>> +    /* TODO: Might as well start some debugfs */
> >>>
> >>> If it's just to expose the version and capabilities, I think that's
> >>> overkill. It's not needed for debug purpose (you can get it from the
> >>> kernel log already). debugfs isn't meant to be accessible in production,
> >>> so an application that would need access to the information wouldn't be
> >>> able to use it.
> >>>
> >>>> +    dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);
> >>>
> >>> Combine the two messages into one.
> >>>
> >>>> +    return version;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>> +    u32 curr_config, next_config;
> >>>> +
> >>>> +    curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
> >>>> +    curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
> >>>> +              >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
> >>>> +    next_config = curr_config ^ 1;
> >>>> +
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>> +                 MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
> >>>> +    mali_c55_config_write(ctx, next_config ?
> >>>> +                  MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
> >>>> +}
> >>>> +
> >>>> +static irqreturn_t mali_c55_isr(int irq, void *context)
> >>>> +{
> >>>> +    struct device *dev = context;
> >>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >>>> +    u32 interrupt_status;
> >>>> +    unsigned int i, j;
> >>>> +
> >>>> +    interrupt_status = mali_c55_read(mali_c55,
> >>>> +                     MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
> >>>> +                     false);
> >>>> +    if (!interrupt_status)
> >>>> +        return IRQ_NONE;
> >>>> +
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
> >>>> +               interrupt_status);
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
> >>>> +
> >>>> +    for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
> >>>> +        if (!(interrupt_status & (1 << i)))
> >>>> +            continue;
> >>>> +
> >>>> +        switch (i) {
> >>>> +        case MALI_C55_IRQ_ISP_START:
> >>>> +            mali_c55_isp_queue_event_sof(mali_c55);
> >>>> +
> >>>> +            for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
> >>>> + mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
> >>>> +
> >>>> +            mali_c55_swap_next_config(mali_c55);
> >>>> +
> >>>> +            break;
> >>>> +        case MALI_C55_IRQ_ISP_DONE:
> >>>> +            /*
> >>>> +             * TODO: Where the ISP has no Pong config fitted, we'd
> >>>> +             * have to do the mali_c55_swap_next_config() call here.
> >>>> +             */
> >>>> +            break;
> >>>> +        case MALI_C55_IRQ_FR_Y_DONE:
> >>>> +            mali_c55_set_plane_done(
> >>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
> >>>> +                MALI_C55_PLANE_Y);
> >>>> +            break;
> >>>> +        case MALI_C55_IRQ_FR_UV_DONE:
> >>>> +            mali_c55_set_plane_done(
> >>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
> >>>> +                MALI_C55_PLANE_UV);
> >>>> +            break;
> >>>> +        case MALI_C55_IRQ_DS_Y_DONE:
> >>>> +            mali_c55_set_plane_done(
> >>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
> >>>> +                MALI_C55_PLANE_Y);
> >>>> +            break;
> >>>> +        case MALI_C55_IRQ_DS_UV_DONE:
> >>>> +            mali_c55_set_plane_done(
> >>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
> >>>> +                MALI_C55_PLANE_UV);
> >>>> +            break;
> >>>> +        default:
> >>>> +            /*
> >>>> +             * Only the above interrupts are currently unmasked. If
> >>>> +             * we receive anything else here then something weird
> >>>> +             * has gone on.
> >>>> +             */
> >>>> +            dev_err(dev, "masked interrupt %s triggered\n",
> >>>> +                mali_c55_interrupt_names[i]);
> >>>> +        }
> >>>> +    }
> >>>> +
> >>>> +    return IRQ_HANDLED;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_init_context(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    struct mali_c55_ctx *ctx;
> >>>> +    int ret;
> >>>> +
> >>>> +    ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
> >>>> +    if (!ctx) {
> >>>> +        dev_err(mali_c55->dev, "failed to allocate new context\n");
> >>>
> >>> No need for an error message when memory allocation fails.
> >>>
> >>>> +        return -ENOMEM;
> >>>> +    }
> >>>> +
> >>>> +    ctx->base = mali_c55->res->start;
> >>>> +    ctx->mali_c55 = mali_c55;
> >>>> +
> >>>> +    ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
> >>>> +                 GFP_KERNEL | GFP_DMA);
> >>>> +    if (!ctx->registers) {
> >>>> +        ret = -ENOMEM;
> >>>> +        goto err_free_ctx;
> >>>> +    }
> >>>> +
> >>>> +    /*
> >>>> +     * The allocated memory is empty, we need to load the default
> >>>> +     * register settings. We just read Ping; it's identical to Pong.
> >>>> +     */
> >>>> +    ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
> >>>> +    if (ret)
> >>>> +        goto err_free_registers;
> >>>> +
> >>>> +    list_add_tail(&ctx->list, &mali_c55->contexts);
> >>>> +
> >>>> +    /*
> >>>> +     * Some features of the ISP need to be disabled by default and only
> >>>> +     * enabled at the same time as they're configured by a parameters buffer
> >>>> +     */
> >>>> +
> >>>> +    /* Bypass the sqrt and square compression and expansion modules */
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
> >>>> +                 MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
> >>>> +                 MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
> >>>> +
> >>>> +    /* Bypass the temper module */
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
> >>>> +               MALI_C55_REG_BYPASS_2_TEMPER);
> >>>> +
> >>>> +    /* Bypass the colour noise reduction  */
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
> >>>> +               MALI_C55_REG_BYPASS_4_CNR);
> >>>> +
> >>>> +    /* Disable the sinter module */
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
> >>>> +                 MALI_C55_SINTER_ENABLE_MASK, 0x00);
> >>>> +
> >>>> +    /* Disable the RGB Gamma module for each output */
> >>>> +    mali_c55_write(
> >>>> +        mali_c55,
> >>>> + MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
> >>>> +        0x00);
> >>>> +    mali_c55_write(
> >>>> +        mali_c55,
> >>>> + MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
> >>>> +        0x00);
> >>>> +
> >>>> +    /* Disable the colour correction matrix */
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
> >>>> +
> >>>> +    return 0;
> >>>> +
> >>>> +err_free_registers:
> >>>> +    kfree(ctx->registers);
> >>>> +err_free_ctx:
> >>>> +    kfree(ctx);
> >>>> +
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_runtime_resume(struct device *dev)
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >>>> +    int ret;
> >>>> +
> >>>> +    ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
> >>>> +                      mali_c55->clks);
> >>>> +    if (ret)
> >>>> +        dev_err(mali_c55->dev, "failed to enable clocks\n");
> >>>> +
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_runtime_suspend(struct device *dev)
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >>>> +
> >>>> + clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct dev_pm_ops mali_c55_pm_ops = {
> >>>> +    SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> >>>> +                pm_runtime_force_resume)
> >>>> +    SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
> >>>> +               NULL)
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_probe(struct platform_device *pdev)
> >>>> +{
> >>>> +    struct device *dev = &pdev->dev;
> >>>> +    struct mali_c55 *mali_c55;
> >>>> +    dma_cap_mask_t mask;
> >>>> +    u32 version;
> >>>> +    int ret;
> >>>> +    u32 val;
> >>>> +
> >>>> +    mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
> >>>> +    if (!mali_c55)
> >>>> +        return dev_err_probe(dev, -ENOMEM,
> >>>> +                     "failed to allocate memory\n");
> >>>
> >>>         return -ENOMEM;
> >>>
> >>> There's no need to print messages for memory allocation failures.
> >>>
> >>>> +
> >>>> +    mali_c55->dev = dev;
> >>>> +    platform_set_drvdata(pdev, mali_c55);
> >>>> +
> >>>> +    mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
> >>>> +                                &mali_c55->res);
> >>>> +    if (IS_ERR(mali_c55->base))
> >>>> +        return dev_err_probe(dev, PTR_ERR(mali_c55->base),
> >>>> +                     "failed to map IO memory\n");
> >>>> +
> >>>> +    ret = platform_get_irq(pdev, 0);
> >>>> +    if (ret < 0)
> >>>> +        return dev_err_probe(dev, ret, "failed to get interrupt num\n");
> >>>
> >>> s/ num// or s/num/number/
> >>>
> >>>> +
> >>>> +    ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
> >>>> +                    mali_c55_isr, IRQF_ONESHOT,
> >>>> +                    dev_driver_string(&pdev->dev),
> >>>> +                    &pdev->dev);
> >>>
> >>> Requested the IRQ should be done much lower, after you have initialized
> >>> everything, or an IRQ that would fire early would have really bad
> >>> consequences.
> >>>
> >>> A comment to explain why you need a threaded interrupt handler would be
> >>> good. I assume it is due only to the need to transfer the registers
> >>> using DMA. I wonder if we should then split the interrupt handler in
> >>> two, with a non-threaded part for the operations that can run quickly,
> >>> and a threaded part for the reprogramming.
> >>>
> >>> It may also be that we could just start the DMA transfer in the
> >>> non-threaded handler without waiting synchronously for it to complete.
> >>> That would be a bigger change, and would require checking race
> >>> conditions carefully. On the other hand, I'm a bit concerned about the
> >>> current implementation, have you tested what happens if the DMA transfer
> >>> takes too long to complete, and spans frame boundaries ? This part could
> >>> be addressed by a patch on top of this one.
> >>>
> >>>> +    if (ret)
> >>>> +        return dev_err_probe(dev, ret, "failed to request irq\n");
> >>>> +
> >>>> +    for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
> >>>> +        mali_c55->clks[i].id = mali_c55_clk_names[i];
> >>>> +
> >>>> +    ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
> >>>> +    if (ret)
> >>>> +        return dev_err_probe(dev, ret, "failed to acquire clocks\n");
> >>>> +
> >>>> +    pm_runtime_enable(&pdev->dev);
> >>>> +
> >>>> +    ret = pm_runtime_resume_and_get(&pdev->dev);
> >>>> +    if (ret)
> >>>> +        goto err_pm_runtime_disable;
> >>>> +
> >>>> +    of_reserved_mem_device_init(dev);
> >>>
> >>> I'd move this, the vb2_dma_contig_set_max_seg_size() call and the
> >>> dma_cap_* calls before pm_runtime_enable() as they don't need the device
> >>> to be powered.
> >>>
> >>>> +    version = mali_c55_check_hwcfg(mali_c55);
> >>>> +    vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
> >>>> +
> >>>> +    /* Use "software only" context management. */
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>> +                 MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> >>>> +                 MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> >>>
> >>> You handle that in mali_c55_isp_start(), does the register have to be
> >>> set here too ?
> >>>
> >>>> +
> >>>> +    dma_cap_zero(mask);
> >>>> +    dma_cap_set(DMA_MEMCPY, mask);
> >>>> +
> >>>> +    /*
> >>>> +     * No error check, because we will just fallback on memcpy if there is
> >>>> +     * no usable DMA channel on the system.
> >>>> +     */
> >>>> +    mali_c55->channel = dma_request_channel(mask, NULL, NULL);
> >>>> +
> >>>> +    INIT_LIST_HEAD(&mali_c55->contexts);
> >>>> +    ret = mali_c55_init_context(mali_c55);
> >>>> +    if (ret)
> >>>> +        goto err_release_dma_channel;
> >>>> +
> >>>
> >>> I'd move all the code from here ...
> >>>
> >>>> +    mali_c55->media_dev.dev = dev;
> >>>> +    strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
> >>>> +        sizeof(mali_c55->media_dev.model));
> >>>> +    mali_c55->media_dev.hw_revision = version;
> >>>> +
> >>>> +    media_device_init(&mali_c55->media_dev);
> >>>> +    ret = media_device_register(&mali_c55->media_dev);
> >>>> +    if (ret)
> >>>> +        goto err_cleanup_media_device;
> >>>> +
> >>>> +    mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
> >>>> +    ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
> >>>> +    if (ret) {
> >>>> +        dev_err(dev, "failed to register V4L2 device\n");
> >>>> +        goto err_unregister_media_device;
> >>>> +    };
> >>>> +
> >>>> +    ret = mali_c55_register_entities(mali_c55);
> >>>> +    if (ret) {
> >>>> +        dev_err(dev, "failed to register entities\n");
> >>>> +        goto err_unregister_v4l2_device;
> >>>> +    }
> >>>
> >>> ... to here to a separate function, or maybe fold it all in
> >>> mali_c55_register_entities() (which should the be renamed). Same thing
> >>> for the cleanup code.
> >>>
> >>>> +
> >>>> +    /* Set safe stop to ensure we're in a non-streaming state */
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> >>>> +               MALI_C55_INPUT_SAFE_STOP);
> >>>> +    readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >>>> +
> >>>> +    /*
> >>>> +     * We're ready to process interrupts. Clear any that are set and then
> >>>> +     * unmask them for processing.
> >>>> +     */
> >>>> +    mali_c55_write(mali_c55, 0x30, 0xffffffff);
> >>>> +    mali_c55_write(mali_c55, 0x34, 0xffffffff);
> >>>> +    mali_c55_write(mali_c55, 0x40, 0x01);
> >>>> +    mali_c55_write(mali_c55, 0x40, 0x00);
> >>>
> >>> Please replace the register addresses with macros.
> >>>
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);
> >>>
> >>> The value should use the interrupt bits macros.
> >>>
> >>>> +
> >>>> +    pm_runtime_put(&pdev->dev);
> >>>
> >>> Once power gets cut, the registers your programmed above may be lost. I
> >>> think you should programe them in the runtime PM resume handler.
> >>>
> >>>> +
> >>>> +    return 0;
> >>>> +
> >>>> +err_unregister_v4l2_device:
> >>>> +    v4l2_device_unregister(&mali_c55->v4l2_dev);
> >>>> +err_unregister_media_device:
> >>>> +    media_device_unregister(&mali_c55->media_dev);
> >>>> +err_cleanup_media_device:
> >>>> +    media_device_cleanup(&mali_c55->media_dev);
> >>>> +err_release_dma_channel:
> >>>> +    dma_release_channel(mali_c55->channel);
> >>>> +err_pm_runtime_disable:
> >>>> +    pm_runtime_disable(&pdev->dev);
> >>>> +
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_remove(struct platform_device *pdev)
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
> >>>> +    struct mali_c55_ctx *ctx, *tmp;
> >>>> +
> >>>> +    list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
> >>>> +        list_del(&ctx->list);
> >>>> +        kfree(ctx->registers);
> >>>> +        kfree(ctx);
> >>>> +    }
> >>>> +
> >>>> +    mali_c55_remove_links(mali_c55);
> >>>> +    mali_c55_unregister_entities(mali_c55);
> >>>> +    v4l2_device_put(&mali_c55->v4l2_dev);
> >>>> +    media_device_unregister(&mali_c55->media_dev);
> >>>> +    media_device_cleanup(&mali_c55->media_dev);
> >>>> +    dma_release_channel(mali_c55->channel);
> >>>> +}
> >>>> +
> >>>> +static const struct of_device_id mali_c55_of_match[] = {
> >>>> +    { .compatible = "arm,mali-c55", },
> >>>> +    {},
> >>>
> >>>     { /* Sentinel */ },
> >>>
> >>>> +};
> >>>> +MODULE_DEVICE_TABLE(of, mali_c55_of_match);
> >>>> +
> >>>> +static struct platform_driver mali_c55_driver = {
> >>>> +    .driver = {
> >>>> +        .name = "mali-c55",
> >>>> +        .of_match_table = of_match_ptr(mali_c55_of_match),
> >>>
> >>> Drop of_match_ptr().
> >>>
> >>>> +        .pm = &mali_c55_pm_ops,
> >>>> +    },
> >>>> +    .probe = mali_c55_probe,
> >>>> +    .remove_new = mali_c55_remove,
> >>>> +};
> >>>> +
> >>>> +module_platform_driver(mali_c55_driver);
> >>>
> >>> Blank line.
> >>>
> >>>> +MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
> >>>> +MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
> >>>> +MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
> >>>> +MODULE_LICENSE("GPL");
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c 
> >>>> b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>> new file mode 100644
> >>>> index 000000000000..ea8b7b866e7a
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>> @@ -0,0 +1,611 @@
> >>>> +// SPDX-License-Identifier: GPL-2.0
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Image signal processor
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#include <linux/delay.h>
> >>>> +#include <linux/iopoll.h>
> >>>> +#include <linux/property.h>
> >>>> +#include <linux/string.h>
> >>>> +
> >>>> +#include <media/media-entity.h>
> >>>> +#include <media/v4l2-common.h>
> >>>> +#include <media/v4l2-event.h>
> >>>> +#include <media/v4l2-mc.h>
> >>>> +#include <media/v4l2-subdev.h>
> >>>> +
> >>>> +#include "mali-c55-common.h"
> >>>> +#include "mali-c55-registers.h"
> >>>> +
> >>>> +static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
> >>>> +    {
> >>>> +        .code = MEDIA_BUS_FMT_SRGGB20_1X20,
> >>>> +        .order = MALI_C55_BAYER_ORDER_RGGB,
> >>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
> >>>> +    },
> >>>> +    {
> >>>> +        .code = MEDIA_BUS_FMT_SGRBG20_1X20,
> >>>> +        .order = MALI_C55_BAYER_ORDER_GRBG,
> >>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
> >>>> +    },
> >>>> +    {
> >>>> +        .code = MEDIA_BUS_FMT_SGBRG20_1X20,
> >>>> +        .order = MALI_C55_BAYER_ORDER_GBRG,
> >>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
> >>>> +    },
> >>>> +    {
> >>>> +        .code = MEDIA_BUS_FMT_SBGGR20_1X20,
> >>>> +        .order = MALI_C55_BAYER_ORDER_BGGR,
> >>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
> >>>> +    },
> >>>> +    {
> >>>> +        .code = MEDIA_BUS_FMT_RGB202020_1X60,
> >>>> +        .order = 0, /* Not relevant for this format */
> >>>> +        .encoding = V4L2_PIXEL_ENC_RGB,
> >>>> +    }
> >>>> +    /*
> >>>> +     * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
> >>>> +     * also support YUV input from a sensor passed-through to the output. At
> >>>> +     * present we have no mechanism to test that though so it may have to
> >>>> +     * wait a while...
> >>>> +     */
> >>>> +};
> >>>> +
> >>>> +const struct mali_c55_isp_fmt *
> >>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
> >>>> +{
> >>>> +    if (!fmt)
> >>>> +        fmt = &mali_c55_isp_fmts[0];
> >>>> +    else
> >>>> +        fmt++;
> >>>> +
> >>>> +    for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
> >>>> +        return fmt;
> >>>
> >>> That's peculiar.
> >>>
> >>>     if (!fmt)
> >>>         fmt = &mali_c55_isp_fmts[0];
> >>>     else if (fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)])
> >>>         return ++fmt;
> >>>     else
> >>>         return NULL;
> >>>
> >>>> +
> >>>> +    return NULL;
> >>>> +}
> >>>> +
> >>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
> >>>> +{
> >>>> +    const struct mali_c55_isp_fmt *isp_fmt;
> >>>> +
> >>>> +    for_each_mali_isp_fmt(isp_fmt) {
> >>>
> >>> I would open-code the loop instead of using the macro, like you do
> >>> below. It will be more efficient.
> >>>
> >>>> +        if (isp_fmt->code == mbus_code)
> >>>> +            return true;
> >>>> +    }
> >>>> +
> >>>> +    return false;
> >>>> +}
> >>>> +
> >>>> +static const struct mali_c55_isp_fmt *
> >>>> +mali_c55_isp_get_mbus_config_by_code(u32 code)
> >>>> +{
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
> >>>> +        if (mali_c55_isp_fmts[i].code == code)
> >>>> +            return &mali_c55_isp_fmts[i];
> >>>> +    }
> >>>> +
> >>>> +    return NULL;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    u32 val;
> >>>> +
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);
> >>>
> >>>     mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> >>>                MALI_C55_INPUT_SAFE_STOP);
> >>>
> >>>> + readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_isp_start(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>> +    const struct mali_c55_isp_fmt *cfg;
> >>>> +    struct v4l2_mbus_framefmt *format;
> >>>
> >>> const
> >>>
> >>>> +    struct v4l2_subdev_state *state;
> >>>> +    struct v4l2_rect *crop;
> >>>
> >>> const
> >>>
> >>>> +    struct v4l2_subdev *sd;
> >>>> +    u32 val;
> >>>> +    int ret;
> >>>> +
> >>>> +    sd = &mali_c55->isp.sd;
> >>>
> >>> Assign when declaring the variable.
> >>>
> >>>> +
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>> +                 MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
> >>>> +
> >>>> +    /* Apply input windowing */
> >>>> +    state = v4l2_subdev_get_locked_active_state(sd);
> >>>
> >>> Using .enable_streams() (see below) you'll get this for free.
> >>>
> >>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +    format = v4l2_subdev_state_get_format(state,
> >>>> +                          MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +    cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
> >>>> +
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
> >>>> +               MALI_C55_HC_START(crop->left));
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
> >>>> +               MALI_C55_HC_SIZE(crop->width));
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
> >>>> +               MALI_C55_VC_START(crop->top) |
> >>>> +               MALI_C55_VC_SIZE(crop->height));
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
> >>>> +                 MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
> >>>> +                 MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
> >>>> +                 MALI_C55_BAYER_ORDER_MASK, cfg->order);
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
> >>>> +                 MALI_C55_INPUT_WIDTH_MASK,
> >>>> +                 MALI_C55_INPUT_WIDTH_20BIT);
> >>>> +
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >>>> +                 MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
> >>>> +                 cfg->encoding == V4L2_PIXEL_ENC_RGB ?
> >>>> +                 MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
> >>>> +
> >>>> +    ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
> >>>> +    if (ret) {
> >>>> +        dev_err(mali_c55->dev, "failed to DMA config\n");
> >>>> +        return ret;
> >>>> +    }
> >>>> +
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> >>>> +               MALI_C55_INPUT_SAFE_START);
> >>>> +    readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >>>
> >>> Should you return an error in case of timeout ?
> >>>
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)
> >>>
> >>> Why is this not handled wired to .s_stream() ? Or better,
> >>> .enable_streams() and .disable_streams().
> >>>
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>> +    struct v4l2_subdev *sd;
> >>>> +
> >>>> +    if (isp->remote_src) {
> >>>> +        sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
> >>>> +        v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
> >>>> +    }
> >>>> +    isp->remote_src = NULL;
> >>>> +
> >>>> +    mali_c55_isp_stop(mali_c55);
> >>>> +}
> >>>> +
> >>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>> +    struct media_pad *sink_pad;
> >>>> +    struct v4l2_subdev *sd;
> >>>> +    int ret;
> >>>> +
> >>>> +    sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
> >>>> +    isp->remote_src = media_pad_remote_pad_unique(sink_pad);
> >>>> +    if (IS_ERR(isp->remote_src)) {
> >>>
> >>> If you set the MUST_CONNECT flag for the MALI_C55_ISP_PAD_SINK_VIDEO pad
> >>> I think you can drop this check.
> >>>
> >>>> +        dev_err(mali_c55->dev, "Failed to get source for ISP\n");
> >>>> +        return PTR_ERR(isp->remote_src);
> >>>> +    }
> >>>> +
> >>>> +    sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
> >>>> +
> >>>> +    isp->frame_sequence = 0;
> >>>> +    ret = mali_c55_isp_start(mali_c55);
> >>>> +    if (ret) {
> >>>> +        dev_err(mali_c55->dev, "Failed to start ISP\n");
> >>>> +        isp->remote_src = NULL;
> >>>> +        return ret;
> >>>> +    }
> >>>> +
> >>>> +    /*
> >>>> +     * We only support a single input stream, so we can just enable the 1st
> >>>> +     * entry in the streams mask.
> >>>> +     */
> >>>> +    ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
> >>>> +    if (ret) {
> >>>> +        dev_err(mali_c55->dev, "Failed to start ISP source\n");
> >>>> +        mali_c55_isp_stop(mali_c55);
> >>>> +        return ret;
> >>>> +    }
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >>>> +                       struct v4l2_subdev_state *state,
> >>>> +                       struct v4l2_subdev_mbus_code_enum *code)
> >>>> +{
> >>>> +    /*
> >>>> +     * Only the internal RGB processed format is allowed on the regular
> >>>> +     * processing source pad.
> >>>> +     */
> >>>> +    if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
> >>>> +        if (code->index)
> >>>> +            return -EINVAL;
> >>>> +
> >>>> +        code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>> +        return 0;
> >>>> +    }
> >>>> +
> >>>> +    /* On the sink and bypass pads all the supported formats are allowed. */
> >>>> +    if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
> >>>> +        return -EINVAL;
> >>>> +
> >>>> +    code->code = mali_c55_isp_fmts[code->index].code;
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
> >>>> +                    struct v4l2_subdev_state *state,
> >>>> +                    struct v4l2_subdev_frame_size_enum *fse)
> >>>> +{
> >>>> +    const struct mali_c55_isp_fmt *cfg;
> >>>> +
> >>>> +    if (fse->index > 0)
> >>>> +        return -EINVAL;
> >>>> +
> >>>> +    /*
> >>>> +     * Only the internal RGB processed format is allowed on the regular
> >>>> +     * processing source pad.
> >>>> +     *
> >>>> +     * On the sink and bypass pads all the supported formats are allowed.
> >>>> +     */
> >>>> +    if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
> >>>> +        if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
> >>>> +            return -EINVAL;
> >>>> +    } else {
> >>>> +        cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
> >>>> +        if (!cfg)
> >>>> +            return -EINVAL;
> >>>> +    }
> >>>> +
> >>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
> >>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
> >>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
> >>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
> >>>> +                struct v4l2_subdev_state *state,
> >>>> +                struct v4l2_subdev_format *format)
> >>>> +{
> >>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>> +    struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
> >>>> +    const struct mali_c55_isp_fmt *cfg;
> >>>> +    struct v4l2_rect *crop;
> >>>> +
> >>>> +    /*
> >>>> +     * Disallow set_fmt on the source pads; format is fixed and the sizes
> >>>> +     * are the result of applying the sink crop rectangle to the sink
> >>>> +     * format.
> >>>> +     */
> >>>> +    if (format->pad)
> >>>
> >>>     if (format->pad != MALI_C55_ISP_PAD_SINK_VIDEO)
> >>>
> >>>> +        return v4l2_subdev_get_fmt(sd, state, format);
> >>>> +
> >>>> +    cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
> >>>> +    if (!cfg)
> >>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>> +    fmt->field = V4L2_FIELD_NONE;
> >>>
> >>> Do you intentionally allow the colorspace fields to be overwritten to
> >>> any value ? rkisp1_isp_set_src_fmt() and rkisp1_isp_set_sink_fmt() may
> >>> show you how this could be handled.
> >>>
> >>>> +
> >>>> +    /*
> >>>> +     * Clamp sizes in the accepted limits and clamp the crop rectangle in
> >>>> +     * the new sizes.
> >>>> +     */
> >>>> +    clamp_t(unsigned int, fmt->width,
> >>>> +        MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>>> +    clamp_t(unsigned int, fmt->width,
> >>>> +        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>>
> >>> clamp_t() returns a value, which you ignore. Those are no-ops. You meant
> >>>
> >>>     fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> >>>                  MALI_C55_MAX_WIDTH);
> >>>     fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> >>>                   MALI_C55_MAX_HEIGHT);
> >>>
> >>> Same for every use of clamp_t() through the whole driver.
> >>>
> >>> Also, do you need clamp_t() ? I think all values are unsigned int, you
> >>> can use clamp().
> >>>
> >>> Are there any alignment constraints, such a multiples of two for bayer
> >>> formats ? Same in all the other locations where applicable.
> >>>
> >>>> +
> >>>> +    sink_fmt = v4l2_subdev_state_get_format(state,
> >>>> +                        MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +    *sink_fmt = *fmt;
> >>>> +
> >>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +    crop->left = 0;
> >>>> +    crop->top = 0;
> >>>> +    crop->width = fmt->width;
> >>>> +    crop->height = fmt->height;
> >>>> +
> >>>> +    /*
> >>>> +     * Propagate format to source pads. On the 'regular' output pad use
> >>>> +     * the internal RGB processed format, while on the bypass pad simply
> >>>> +     * replicate the ISP sink format. The sizes on both pads are the same as
> >>>> +     * the ISP sink crop rectangle.
> >>>> +     */
> >>>
> >>> Colorspace information will need to be propagated too.
> >>>
> >>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
> >>>> +    src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>> +    src_fmt->width = crop->width;
> >>>> +    src_fmt->height = crop->height;
> >>>> +
> >>>> +    src_fmt = v4l2_subdev_state_get_format(state,
> >>>> +                           MALI_C55_ISP_PAD_SOURCE_BYPASS);
> >>>> +    src_fmt->code = fmt->code;
> >>>> +    src_fmt->width = crop->width;
> >>>> +    src_fmt->height = crop->height;
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
> >>>> +                      struct v4l2_subdev_state *state,
> >>>> +                      struct v4l2_subdev_selection *sel)
> >>>> +{
> >>>> +    if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> >>>
> >>>     sel->pad != MALI_C55_ISP_PAD_SINK_VIDEO
> >>>
> >>>> +        return -EINVAL;
> >>>> +
> >>>> +    sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
> >>>> +                      struct v4l2_subdev_state *state,
> >>>> +                      struct v4l2_subdev_selection *sel)
> >>>> +{
> >>>> +    struct v4l2_mbus_framefmt *src_fmt;
> >>>> +    struct v4l2_mbus_framefmt *fmt;
> >>>
> >>> const
> >>>
> >>>> +    struct v4l2_rect *crop;
> >>>> +
> >>>> +    if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> >>>
> >>> Ditto.
> >>>
> >>>> +        return -EINVAL;
> >>>> +
> >>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +
> >>>> +    clamp_t(unsigned int, sel->r.left, 0, fmt->width);
> >>>> +    clamp_t(unsigned int, sel->r.top, 0, fmt->height);
> >>>> +    clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
> >>>> +        fmt->width - sel->r.left);
> >>>> +    clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
> >>>> +        fmt->height - sel->r.top);
> >>>> +
> >>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +    *crop = sel->r;
> >>>> +
> >>>> +    /* Propagate the crop rectangle sizes to the source pad format. */
> >>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
> >>>> +    src_fmt->width = crop->width;
> >>>> +    src_fmt->height = crop->height;
> >>>
> >>> Can you confirm that cropping doesn't affect the bypass path ? And maybe
> >>> add a comment to mention it.
> >>>
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
> >>>> +    .enum_mbus_code        = mali_c55_isp_enum_mbus_code,
> >>>> +    .enum_frame_size    = mali_c55_isp_enum_frame_size,
> >>>> +    .get_fmt        = v4l2_subdev_get_fmt,
> >>>> +    .set_fmt        = mali_c55_isp_set_fmt,
> >>>> +    .get_selection        = mali_c55_isp_get_selection,
> >>>> +    .set_selection        = mali_c55_isp_set_selection,
> >>>> +    .link_validate        = v4l2_subdev_link_validate_default,
> >>>> +};
> >>>> +
> >>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    struct v4l2_event event = {
> >>>> +        .type = V4L2_EVENT_FRAME_SYNC,
> >>>> +    };
> >>>> +
> >>>> +    event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
> >>>> +    v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
> >>>> +}
> >>>> +
> >>>> +static int
> >>>> +mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
> >>>> +                 struct v4l2_event_subscription *sub)
> >>>> +{
> >>>> +    if (sub->type != V4L2_EVENT_FRAME_SYNC)
> >>>> +        return -EINVAL;
> >>>> +
> >>>> +    /* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
> >>>> +    if (sub->id != 0)
> >>>> +        return -EINVAL;
> >>>> +
> >>>> +    return v4l2_event_subscribe(fh, sub, 0, NULL);
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
> >>>> +    .subscribe_event = mali_c55_isp_subscribe_event,
> >>>> +    .unsubscribe_event = v4l2_event_subdev_unsubscribe,
> >>>> +};
> >>>> +
> >>>> +static const struct v4l2_subdev_ops mali_c55_isp_ops = {
> >>>> +    .pad    = &mali_c55_isp_pad_ops,
> >>>> +    .core    = &mali_c55_isp_core_ops,
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
> >>>> +                   struct v4l2_subdev_state *sd_state)
> >>>
> >>> You name this variable state in every other subdev operation handler.
> >>>
> >>>> +{
> >>>> +    struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >>>> +    struct v4l2_rect *in_crop;
> >>>> +
> >>>> +    sink_fmt = v4l2_subdev_state_get_format(sd_state,
> >>>> +                        MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +    src_fmt = v4l2_subdev_state_get_format(sd_state,
> >>>> +                           MALI_C55_ISP_PAD_SOURCE);
> >>>> +    in_crop = v4l2_subdev_state_get_crop(sd_state,
> >>>> +                         MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +
> >>>> +    sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>> +    sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>> +    sink_fmt->field = V4L2_FIELD_NONE;
> >>>> +    sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>
> >>> You should initialize the colorspace fields too. Same below.
> >>>
> >>>> +
> >>>> +    *v4l2_subdev_state_get_format(sd_state,
> >>>> +                  MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
> >>>> +
> >>>> +    src_fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>> +    src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>> +    src_fmt->field = V4L2_FIELD_NONE;
> >>>> +    src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>> +
> >>>> +    in_crop->top = 0;
> >>>> +    in_crop->left = 0;
> >>>> +    in_crop->width = MALI_C55_DEFAULT_WIDTH;
> >>>> +    in_crop->height = MALI_C55_DEFAULT_HEIGHT;
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
> >>>> +    .init_state = mali_c55_isp_init_state,
> >>>> +};
> >>>> +
> >>>> +static const struct media_entity_operations mali_c55_isp_media_ops = {
> >>>> +    .link_validate        = v4l2_subdev_link_validate,
> >>>
> >>>     .link_validate = v4l2_subdev_link_validate,
> >>>
> >>> to match mali_c55_isp_internal_ops.
> >>>
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
> >>>> +                       struct v4l2_subdev *subdev,
> >>>> +                       struct v4l2_async_connection *asc)
> >>>> +{
> >>>> +    struct mali_c55_isp *isp = container_of(notifier,
> >>>> +                        struct mali_c55_isp, notifier);
> >>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>> +    struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
> >>>> +    int ret;
> >>>> +
> >>>> +    /*
> >>>> +     * By default we'll flag this link enabled and the TPG disabled, but
> >>>> +     * no immutable flag because we need to be able to switch between the
> >>>> +     * two.
> >>>> +     */
> >>>> +    ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
> >>>> +                          MEDIA_LNK_FL_ENABLED);
> >>>> +    if (ret)
> >>>> +        dev_err(mali_c55->dev, "failed to create link for %s\n",
> >>>> +            subdev->name);
> >>>> +
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
> >>>> +{
> >>>> +    struct mali_c55_isp *isp = container_of(notifier,
> >>>> +                        struct mali_c55_isp, notifier);
> >>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>> +
> >>>> +    return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
> >>>> +    .bound = mali_c55_isp_notifier_bound,
> >>>> +    .complete = mali_c55_isp_notifier_complete,
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>> +    struct v4l2_async_connection *asc;
> >>>> +    struct fwnode_handle *ep;
> >>>> +    int ret;
> >>>> +
> >>>> +    v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
> >>>> +
> >>>> +    /*
> >>>> +     * The ISP should have a single endpoint pointing to some flavour of
> >>>> +     * CSI-2 receiver...but for now at least we do want everything to work
> >>>> +     * normally even with no sensors connected, as we have the TPG. If we
> >>>> +     * don't find a sensor just warn and return success.
> >>>> +     */
> >>>> +    ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
> >>>> +                         0, 0, 0);
> >>>> +    if (!ep) {
> >>>> +        dev_warn(mali_c55->dev, "no local endpoint found\n");
> >>>> +        return 0;
> >>>> +    }
> >>>> +
> >>>> +    asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
> >>>> +                          struct v4l2_async_connection);
> >>>> +    if (IS_ERR(asc)) {
> >>>> +        dev_err(mali_c55->dev, "failed to add remote fwnode\n");
> >>>> +        ret = PTR_ERR(asc);
> >>>> +        goto err_put_ep;
> >>>> +    }
> >>>> +
> >>>> +    isp->notifier.ops = &mali_c55_isp_notifier_ops;
> >>>> +    ret = v4l2_async_nf_register(&isp->notifier);
> >>>> +    if (ret) {
> >>>> +        dev_err(mali_c55->dev, "failed to register notifier\n");
> >>>> +        goto err_cleanup_nf;
> >>>> +    }
> >>>> +
> >>>> +    fwnode_handle_put(ep);
> >>>> +
> >>>> +    return 0;
> >>>> +
> >>>> +err_cleanup_nf:
> >>>> +    v4l2_async_nf_cleanup(&isp->notifier);
> >>>> +err_put_ep:
> >>>> +    fwnode_handle_put(ep);
> >>>> +
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
> >>>> +    struct v4l2_subdev *sd = &isp->sd;
> >>>> +    int ret;
> >>>> +
> >>>> +    isp->mali_c55 = mali_c55;
> >>>> +
> >>>> +    v4l2_subdev_init(sd, &mali_c55_isp_ops);
> >>>> +    sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> >>>> +    sd->entity.ops = &mali_c55_isp_media_ops;
> >>>> +    sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
> >>>> +    sd->internal_ops = &mali_c55_isp_internal_ops;
> >>>> +    strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
> >>>> +
> >>>> +    isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
> >>>
> >>> The MUST_CONNECT flag would make sense here.
> >>>
> >>>> + isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> >>>> +    isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
> >>>> +
> >>>> +    ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
> >>>> +                     isp->pads);
> >>>> +    if (ret)
> >>>> +        return ret;
> >>>> +
> >>>> +    ret = v4l2_subdev_init_finalize(sd);
> >>>> +    if (ret)
> >>>> +        goto err_cleanup_media_entity;
> >>>> +
> >>>> +    ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >>>> +    if (ret)
> >>>> +        goto err_cleanup_subdev;
> >>>> +
> >>>> +    ret = mali_c55_isp_parse_endpoint(isp);
> >>>> +    if (ret)
> >>>> +        goto err_cleanup_subdev;
> >>>
> >>> As noted elsewhere, I think this belongs to mali-c55-core.c.
> >>>
> >>>> +
> >>>> +    mutex_init(&isp->lock);
> >>>
> >>> This lock is used in mali-c55-capture.c only, that seems weird.
> >>>
> >>>> +
> >>>> +    return 0;
> >>>> +
> >>>> +err_cleanup_subdev:
> >>>> +    v4l2_subdev_cleanup(sd);
> >>>> +err_cleanup_media_entity:
> >>>> +    media_entity_cleanup(&sd->entity);
> >>>> +    isp->mali_c55 = NULL;
> >>>> +
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
> >>>> +
> >>>> +    if (!isp->mali_c55)
> >>>> +        return;
> >>>> +
> >>>> +    mutex_destroy(&isp->lock);
> >>>> +    v4l2_async_nf_unregister(&isp->notifier);
> >>>> +    v4l2_async_nf_cleanup(&isp->notifier);
> >>>> +    v4l2_device_unregister_subdev(&isp->sd);
> >>>> +    v4l2_subdev_cleanup(&isp->sd);
> >>>> +    media_entity_cleanup(&isp->sd.entity);
> >>>> +}
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h 
> >>>> b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>> new file mode 100644
> >>>> index 000000000000..cb27abde2aa5
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>> @@ -0,0 +1,258 @@
> >>>> +/* SPDX-License-Identifier: GPL-2.0 */
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Register definitions
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#ifndef _MALI_C55_REGISTERS_H
> >>>> +#define _MALI_C55_REGISTERS_H
> >>>> +
> >>>> +#include <linux/bits.h>
> >>>> +
> >>>> +/* ISP Common 0x00000 - 0x000ff */
> >>>> +
> >>>> +#define MALI_C55_REG_API                0x00000
> >>>> +#define MALI_C55_REG_PRODUCT                0x00004
> >>>> +#define MALI_C55_REG_VERSION                0x00008
> >>>> +#define MALI_C55_REG_REVISION                0x0000c
> >>>> +#define MALI_C55_REG_PULSE_MODE                0x0003c
> >>>> +#define MALI_C55_REG_INPUT_MODE_REQUEST            0x0009c
> >>>> +#define MALI_C55_INPUT_SAFE_STOP            0x00
> >>>> +#define MALI_C55_INPUT_SAFE_START            0x01
> >>>> +#define MALI_C55_REG_MODE_STATUS            0x000a0
> >>>> +#define MALI_C55_REG_INTERRUPT_MASK_VECTOR        0x00030
> >>>> +#define MALI_C55_INTERRUPT_MASK_ALL            GENMASK(31, 0)
> >>>> +
> >>>> +#define MALI_C55_REG_GLOBAL_MONITOR            0x00050
> >>>> +
> >>>> +#define MALI_C55_REG_GEN_VIDEO                0x00080
> >>>> +#define MALI_C55_REG_GEN_VIDEO_ON_MASK            BIT(0)
> >>>> +#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK        BIT(1)
> >>>> +#define MALI_C55_REG_GEN_PREFETCH_MASK GENMASK(31, 16)
> >>>> +
> >>>> +#define MALI_C55_REG_MCU_CONFIG                0x00020
> >>>> +#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK        BIT(0)
> >>>
> >>> #define MALI_C55_REG_MCU_CONFIG_OVERRIDE        BIT(0)
> >>>
> >>> Same in other places where applicable.
> >>>
> >>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK        BIT(1)
> >>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PING        BIT(1)
> >>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG        0x00
> >>>> +#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK        BIT(8)
> >>>> +#define MALI_C55_REG_PING_PONG_READ            0x00024
> >>>> +#define MALI_C55_REG_PING_PONG_READ_MASK        BIT(2)
> >>>> +
> >>>> +#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR        0x00034
> >>>> +#define MALI_C55_REG_INTERRUPT_CLEAR            0x00040
> >>>> +#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR        0x00044
> >>>> +#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS        0x00068
> >>>> +#define MALI_C55_GPS_PONG_FITTED            BIT(0)
> >>>> +#define MALI_C55_GPS_WDR_FITTED                BIT(1)
> >>>> +#define MALI_C55_GPS_COMPRESSION_FITTED            BIT(2)
> >>>> +#define MALI_C55_GPS_TEMPER_FITTED            BIT(3)
> >>>> +#define MALI_C55_GPS_SINTER_LITE_FITTED            BIT(4)
> >>>> +#define MALI_C55_GPS_SINTER_FITTED            BIT(5)
> >>>> +#define MALI_C55_GPS_IRIDIX_LTM_FITTED            BIT(6)
> >>>> +#define MALI_C55_GPS_IRIDIX_GTM_FITTED            BIT(7)
> >>>> +#define MALI_C55_GPS_CNR_FITTED                BIT(8)
> >>>> +#define MALI_C55_GPS_FRSCALER_FITTED            BIT(9)
> >>>> +#define MALI_C55_GPS_DS_PIPE_FITTED            BIT(10)
> >>>> +
> >>>> +#define MALI_C55_REG_BLANKING                0x00084
> >>>> +#define MALI_C55_REG_HBLANK_MASK            GENMASK(15, 0)
> >>>> +#define MALI_C55_REG_VBLANK_MASK            GENMASK(31, 16)
> >>>> +
> >>>> +#define MALI_C55_REG_HC_START                0x00088
> >>>> +#define MALI_C55_HC_START(h)                (((h) & 0xffff) << 16)
> >>>> +#define MALI_C55_REG_HC_SIZE                0x0008c
> >>>> +#define MALI_C55_HC_SIZE(h)                ((h) & 0xffff)
> >>>> +#define MALI_C55_REG_VC_START_SIZE            0x00094
> >>>> +#define MALI_C55_VC_START(v)                ((v) & 0xffff)
> >>>> +#define MALI_C55_VC_SIZE(v)                (((v) & 0xffff) << 16)
> >>>> +
> >>>> +/* Ping/Pong Configuration Space */
> >>>> +#define MALI_C55_REG_BASE_ADDR                0x18e88
> >>>> +#define MALI_C55_REG_BYPASS_0                0x18eac
> >>>> +#define MALI_C55_REG_BYPASS_0_VIDEO_TEST        BIT(0)
> >>>> +#define MALI_C55_REG_BYPASS_0_INPUT_FMT            BIT(1)
> >>>> +#define MALI_C55_REG_BYPASS_0_DECOMPANDER        BIT(2)
> >>>> +#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR BIT(3)
> >>>> +#define MALI_C55_REG_BYPASS_0_GAIN_WDR            BIT(4)
> >>>> +#define MALI_C55_REG_BYPASS_0_FRAME_STITCH        BIT(5)
> >>>> +#define MALI_C55_REG_BYPASS_1                0x18eb0
> >>>> +#define MALI_C55_REG_BYPASS_1_DIGI_GAIN            BIT(0)
> >>>> +#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS        BIT(1)
> >>>> +#define MALI_C55_REG_BYPASS_1_FE_SQRT            BIT(2)
> >>>> +#define MALI_C55_REG_BYPASS_1_RAW_FE            BIT(3)
> >>>> +#define MALI_C55_REG_BYPASS_2                0x18eb8
> >>>> +#define MALI_C55_REG_BYPASS_2_SINTER            BIT(0)
> >>>> +#define MALI_C55_REG_BYPASS_2_TEMPER            BIT(1)
> >>>> +#define MALI_C55_REG_BYPASS_3                0x18ebc
> >>>> +#define MALI_C55_REG_BYPASS_3_SQUARE_BE            BIT(0)
> >>>> +#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH BIT(1)
> >>>> +#define MALI_C55_REG_BYPASS_3_MESH_SHADING        BIT(3)
> >>>> +#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE        BIT(4)
> >>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX            BIT(5)
> >>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN        BIT(6)
> >>>> +#define MALI_C55_REG_BYPASS_4                0x18ec0
> >>>> +#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB        BIT(1)
> >>>> +#define MALI_C55_REG_BYPASS_4_PF_CORRECTION        BIT(3)
> >>>> +#define MALI_C55_REG_BYPASS_4_CCM            BIT(4)
> >>>> +#define MALI_C55_REG_BYPASS_4_CNR            BIT(5)
> >>>> +#define MALI_C55_REG_FR_BYPASS                0x18ec4
> >>>> +#define MALI_C55_REG_DS_BYPASS                0x18ec8
> >>>> +#define MALI_C55_BYPASS_CROP                BIT(0)
> >>>> +#define MALI_C55_BYPASS_SCALER                BIT(1)
> >>>> +#define MALI_C55_BYPASS_GAMMA_RGB            BIT(2)
> >>>> +#define MALI_C55_BYPASS_SHARPEN                BIT(3)
> >>>> +#define MALI_C55_BYPASS_CS_CONV                BIT(4)
> >>>> +#define MALI_C55_REG_ISP_RAW_BYPASS            0x18ecc
> >>>> +#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK        BIT(0)
> >>>> +#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK GENMASK(9, 8)
> >>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS        2
> >>>> +#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS        1
> >>>> +#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE BIT(1)
> >>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS        BIT(0)
> >>>> +
> >>>> +#define MALI_C55_REG_ACTIVE_WIDTH_MASK            0xffff
> >>>> +#define MALI_C55_REG_ACTIVE_HEIGHT_MASK 0xffff0000
> >>>> +#define MALI_C55_REG_BAYER_ORDER            0x18e8c
> >>>> +#define MALI_C55_BAYER_ORDER_MASK            GENMASK(1, 0)
> >>>> +#define MALI_C55_REG_TPG_CH0                0x18ed8
> >>>> +#define MALI_C55_TEST_PATTERN_ON_OFF            BIT(0)
> >>>> +#define MALI_C55_TEST_PATTERN_RGB_MASK            BIT(1)
> >>>> +#define MALI_C55_REG_TPG_R_BACKGROUND            0x18ee0
> >>>> +#define MALI_C55_REG_TPG_G_BACKGROUND            0x18ee4
> >>>> +#define MALI_C55_REG_TPG_B_BACKGROUND            0x18ee8
> >>>> +#define MALI_C55_TPG_BACKGROUND_MAX            0xfffff
> >>>> +#define MALI_C55_REG_INPUT_WIDTH            0x18f98
> >>>> +#define MALI_C55_INPUT_WIDTH_MASK            GENMASK(18, 16)
> >>>> +#define MALI_C55_INPUT_WIDTH_8BIT            0
> >>>> +#define MALI_C55_INPUT_WIDTH_10BIT            1
> >>>> +#define MALI_C55_INPUT_WIDTH_12BIT            2
> >>>> +#define MALI_C55_INPUT_WIDTH_14BIT            3
> >>>> +#define MALI_C55_INPUT_WIDTH_16BIT            4
> >>>> +#define MALI_C55_INPUT_WIDTH_20BIT            5
> >>>> +#define MALI_C55_REG_SPACE_SIZE                0x4000
> >>>> +#define MALI_C55_REG_CONFIG_SPACES_OFFSET        0x0ab6c
> >>>> +#define MALI_C55_CONFIG_SPACE_SIZE            0x1231c
> >>>> +
> >>>> +#define MALI_C55_REG_SINTER_CONFIG            0x19348
> >>>> +#define MALI_C55_SINTER_VIEW_FILTER_MASK        GENMASK(1, 0)
> >>>> +#define MALI_C55_SINTER_SCALE_MODE_MASK GENMASK(3, 2)
> >>>> +#define MALI_C55_SINTER_ENABLE_MASK            BIT(4)
> >>>> +#define MALI_C55_SINTER_FILTER_SELECT_MASK        BIT(5)
> >>>> +#define MALI_C55_SINTER_INT_SELECT_MASK            BIT(6)
> >>>> +#define MALI_C55_SINTER_RM_ENABLE_MASK            BIT(7)
> >>>> +
> >>>> +/* Colour Correction Matrix Configuration */
> >>>> +#define MALI_C55_REG_CCM_ENABLE                0x1b07c
> >>>> +#define MALI_C55_CCM_ENABLE_MASK            BIT(0)
> >>>> +#define MALI_C55_REG_CCM_COEF_R_R            0x1b080
> >>>> +#define MALI_C55_REG_CCM_COEF_R_G            0x1b084
> >>>> +#define MALI_C55_REG_CCM_COEF_R_B            0x1b088
> >>>> +#define MALI_C55_REG_CCM_COEF_G_R            0x1b090
> >>>> +#define MALI_C55_REG_CCM_COEF_G_G            0x1b094
> >>>> +#define MALI_C55_REG_CCM_COEF_G_B            0x1b098
> >>>> +#define MALI_C55_REG_CCM_COEF_B_R            0x1b0a0
> >>>> +#define MALI_C55_REG_CCM_COEF_B_G            0x1b0a4
> >>>> +#define MALI_C55_REG_CCM_COEF_B_B            0x1b0a8
> >>>> +#define MALI_C55_CCM_COEF_MASK                GENMASK(12, 0)
> >>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R            0x1b0b0
> >>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G            0x1b0b4
> >>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B            0x1b0b8
> >>>> +#define MALI_C55_CCM_ANTIFOG_GAIN_MASK GENMASK(11, 0)
> >>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R        0x1b0c0
> >>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G        0x1b0c4
> >>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B        0x1b0c8
> >>>> +#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK        GENMASK(11, 0)
> >>>> +
> >>>> +/*
> >>>> + * The Mali-C55 ISP has up to two output pipes; known as full resolution and
> >>>> + * down scaled. The register space for these is laid out identically, but offset
> >>>> + * by 372 bytes.
> >>>> + */
> >>>> +#define MALI_C55_CAP_DEV_FR_REG_OFFSET        0x0
> >>>> +#define MALI_C55_CAP_DEV_DS_REG_OFFSET        0x174
> >>>> +
> >>>> +#define MALI_C55_REG_CS_CONV_CONFIG(offset)        (0x1c098 + (offset))
> >>>> +#define MALI_C55_CS_CONV_MATRIX_MASK            BIT(0)
> >>>> +#define MALI_C55_CS_CONV_FILTER_MASK            BIT(1)
> >>>> +#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK        BIT(2)
> >>>> +#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK        BIT(3)
> >>>> +#define MALI_C55_REG_Y_WRITER_MODE(offset)        (0x1c0ec + (offset))
> >>>> +#define MALI_C55_REG_UV_WRITER_MODE(offset)        (0x1c144 + (offset))
> >>>> +#define MALI_C55_WRITER_MODE_MASK            GENMASK(4, 0)
> >>>> +#define MALI_C55_WRITER_SUBMODE_MASK            GENMASK(7, 6)
> >>>> +#define MALI_C55_WRITER_FRAME_WRITE_MASK        BIT(9)
> >>>> +#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset) (0x1c0f0 + (offset))
> >>>> +#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset) (0x1c148 + (offset))
> >>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)        ((w) << 0)
> >>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)        ((h) << 16)
> >>>> +#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset) (0x1c0f4 + (offset))
> >>>> +#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset) (0x1c108 + (offset))
> >>>> +#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK GENMASK(2, 0)
> >>>> +#define MALI_C55_REG_Y_WRITER_BANKS_RESTART        BIT(3)
> >>>> +#define MALI_C55_REG_Y_WRITER_OFFSET(offset) (0x1c10c + (offset))
> >>>> +#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset) (0x1c14c + (offset))
> >>>> +#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset) (0x1c160 + (offset))
> >>>> +#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK GENMASK(2, 0)
> >>>> +#define MALI_C55_REG_UV_WRITER_BANKS_RESTART        BIT(3)
> >>>> +#define MALI_C55_REG_UV_WRITER_OFFSET(offset) (0x1c164 + (offset))
> >>>> +
> >>>> +#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
> >>>> +#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE 0x18edc
> >>>> +
> >>>> +#define MALI_C55_REG_CROP_EN(offset)            (0x1c028 + (offset))
> >>>> +#define MALI_C55_CROP_ENABLE                BIT(0)
> >>>> +#define MALI_C55_REG_CROP_X_START(offset)        (0x1c02c + (offset))
> >>>> +#define MALI_C55_REG_CROP_Y_START(offset)        (0x1c030 + (offset))
> >>>> +#define MALI_C55_REG_CROP_X_SIZE(offset)        (0x1c034 + (offset))
> >>>> +#define MALI_C55_REG_CROP_Y_SIZE(offset)        (0x1c038 + (offset))
> >>>> +#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset) (0x1c040 + (offset))
> >>>> +#define MALI_C55_SCALER_TIMEOUT_EN            BIT(4)
> >>>> +#define MALI_C55_SCALER_TIMEOUT(t)            ((t) << 16)
> >>>> +#define MALI_C55_REG_SCALER_IN_WIDTH(offset) (0x1c044 + (offset))
> >>>> +#define MALI_C55_REG_SCALER_IN_HEIGHT(offset) (0x1c048 + (offset))
> >>>> +#define MALI_C55_REG_SCALER_OUT_WIDTH(offset) (0x1c04c + (offset))
> >>>> +#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset) (0x1c050 + (offset))
> >>>> +#define MALI_C55_REG_SCALER_HFILT_TINC(offset) (0x1c054 + (offset))
> >>>> +#define MALI_C55_REG_SCALER_HFILT_COEF(offset) (0x1c058 + (offset))
> >>>> +#define MALI_C55_REG_SCALER_VFILT_TINC(offset) (0x1c05c + (offset))
> >>>> +#define MALI_C55_REG_SCALER_VFILT_COEF(offset) (0x1c060 + (offset))
> >>>> +
> >>>> +#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset) (0x1c064 + (offset))
> >>>> +#define MALI_C55_GAMMA_ENABLE_MASK            BIT(0)
> >>>> +#define MALI_C55_REG_GAMMA_GAINS_1(offset)        (0x1c068 + (offset))
> >>>> +#define MALI_C55_GAMMA_GAIN_R_MASK            GENMASK(11, 0)
> >>>> +#define MALI_C55_GAMMA_GAIN_G_MASK            GENMASK(27, 16)
> >>>> +#define MALI_C55_REG_GAMMA_GAINS_2(offset)        (0x1c06c + (offset))
> >>>> +#define MALI_C55_GAMMA_GAIN_B_MASK            GENMASK(11, 0)
> >>>> +#define MALI_C55_REG_GAMMA_OFFSETS_1(offset) (0x1c070 + (offset))
> >>>> +#define MALI_C55_GAMMA_OFFSET_R_MASK            GENMASK(11, 0)
> >>>> +#define MALI_C55_GAMMA_OFFSET_G_MASK            GENMASK(27, 16)
> >>>> +#define MALI_C55_REG_GAMMA_OFFSETS_2(offset) (0x1c074 + (offset))
> >>>> +#define MALI_C55_GAMMA_OFFSET_B_MASK            GENMASK(11, 0)
> >>>> +
> >>>> +/* Output DMA Writer */
> >>>> +
> >>>> +#define MALI_C55_OUTPUT_DISABLED        0
> >>>> +#define MALI_C55_OUTPUT_RGB32            1
> >>>> +#define MALI_C55_OUTPUT_A2R10G10B10        2
> >>>> +#define MALI_C55_OUTPUT_RGB565            3
> >>>> +#define MALI_C55_OUTPUT_RGB24            4
> >>>> +#define MALI_C55_OUTPUT_GEN32            5
> >>>> +#define MALI_C55_OUTPUT_RAW16            6
> >>>> +#define MALI_C55_OUTPUT_AYUV            8
> >>>> +#define MALI_C55_OUTPUT_Y410            9
> >>>> +#define MALI_C55_OUTPUT_YUY2            10
> >>>> +#define MALI_C55_OUTPUT_UYVY            11
> >>>> +#define MALI_C55_OUTPUT_Y210            12
> >>>> +#define MALI_C55_OUTPUT_NV12_21            13
> >>>> +#define MALI_C55_OUTPUT_YUV_420_422        17
> >>>> +#define MALI_C55_OUTPUT_P210_P010        19
> >>>> +#define MALI_C55_OUTPUT_YUV422            20
> >>>
> >>> I'd define this just below the MALI_C55_REG_UV_WRITER_MODE register
> >>> macro.
> >>>
> >>>> +
> >>>> +#define MALI_C55_OUTPUT_PLANE_ALT0        0
> >>>> +#define MALI_C55_OUTPUT_PLANE_ALT1        1
> >>>> +#define MALI_C55_OUTPUT_PLANE_ALT2        2
> >>>
> >>> Same here ?
> >>>
> >>>> +
> >>>> +#endif /* _MALI_C55_REGISTERS_H */
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h 
> >>>> b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>>> new file mode 100644
> >>>> index 000000000000..8edae87f1e5f
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>>> @@ -0,0 +1,382 @@
> >>>> +/* SPDX-License-Identifier: GPL-2.0 */
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Resizer Coefficients
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#ifndef _MALI_C55_RESIZER_COEFS_H
> >>>> +#define _MALI_C55_RESIZER_COEFS_H
> >>>> +
> >>>> +#include "mali-c55-common.h"
> >>>> +
> >>>> +#define MALI_C55_RESIZER_COEFS_NUM_BANKS    8
> >>>> +#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES    64
> >>>
> >>> Do these belongs to mali-c55-registers.h ?
> >>>
> >>>> +
> >>>> +static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
> >>>> +    {    /* Bank 0 */
> >>>> +        0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
> >>>> +        0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
> >>>> +        0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
> >>>> +        0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
> >>>> +        0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
> >>>> +        0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
> >>>> +        0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
> >>>> +        0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
> >>>> +        0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
> >>>> +        0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
> >>>> +        0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
> >>>> +        0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
> >>>> +        0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
> >>>> +        0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
> >>>> +        0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
> >>>> +        0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
> >>>> +    },
> >>>> +    {    /* Bank 1 */
> >>>> +        0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
> >>>> +        0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
> >>>> +        0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
> >>>> +        0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
> >>>> +        0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
> >>>> +        0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
> >>>> +        0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
> >>>> +        0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
> >>>> +        0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
> >>>> +        0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
> >>>> +        0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
> >>>> +        0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
> >>>> +        0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
> >>>> +        0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
> >>>> +        0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
> >>>> +        0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
> >>>> +    },
> >>>> +    {    /* Bank 2 */
> >>>> +        0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
> >>>> +        0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
> >>>> +        0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
> >>>> +        0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
> >>>> +        0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
> >>>> +        0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
> >>>> +        0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
> >>>> +        0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
> >>>> +        0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
> >>>> +        0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
> >>>> +        0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
> >>>> +        0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
> >>>> +        0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
> >>>> +        0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
> >>>> +        0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
> >>>> +        0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
> >>>> +    },
> >>>> +    {    /* Bank 3 */
> >>>> +        0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
> >>>> +        0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
> >>>> +        0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
> >>>> +        0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
> >>>> +        0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
> >>>> +        0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
> >>>> +        0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
> >>>> +        0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
> >>>> +        0x20100000, 0x00000010, 0x1f110000, 0x00000010,
> >>>> +        0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
> >>>> +        0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
> >>>> +        0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
> >>>> +        0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
> >>>> +        0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
> >>>> +        0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
> >>>> +        0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
> >>>> +    },
> >>>> +    {    /* Bank 4 */
> >>>> +        0x17090000, 0x00000917, 0x18090000, 0x00000916,
> >>>> +        0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
> >>>> +        0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
> >>>> +        0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
> >>>> +        0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
> >>>> +        0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
> >>>> +        0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
> >>>> +        0x190f0300, 0x00000411, 0x18100300, 0x00000411,
> >>>> +        0x1a100300, 0x00000310, 0x18110400, 0x00000310,
> >>>> +        0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
> >>>> +        0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
> >>>> +        0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
> >>>> +        0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
> >>>> +        0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
> >>>> +        0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
> >>>> +        0x17160800, 0x0000010a, 0x18160900, 0x00000009,
> >>>> +    },
> >>>> +    {    /* Bank 5 */
> >>>> +        0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
> >>>> +        0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
> >>>> +        0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
> >>>> +        0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
> >>>> +        0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
> >>>> +        0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
> >>>> +        0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
> >>>> +        0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
> >>>> +        0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
> >>>> +        0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
> >>>> +        0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
> >>>> +        0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
> >>>> +        0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
> >>>> +        0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
> >>>> +        0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
> >>>> +        0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
> >>>> +    },
> >>>> +    {    /* Bank 6 */
> >>>> +        0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
> >>>> +        0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
> >>>> +        0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
> >>>> +        0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
> >>>> +        0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
> >>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>> +        0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
> >>>> +        0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>> +        0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
> >>>> +        0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
> >>>> +        0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
> >>>> +        0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >>>> +        0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >>>> +    },
> >>>> +    {    /* Bank 7 */
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +    }
> >>>> +};
> >>>> +
> >>>> +static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
> >>>> +    {    /* Bank 0 */
> >>>> +        0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
> >>>> +        0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
> >>>> +        0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
> >>>> +        0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
> >>>> +        0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
> >>>> +        0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
> >>>> +        0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
> >>>> +        0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
> >>>> +        0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
> >>>> +        0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
> >>>> +        0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
> >>>> +        0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
> >>>> +        0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
> >>>> +        0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
> >>>> +        0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
> >>>> +        0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
> >>>> +    },
> >>>> +    {    /* Bank 1 */
> >>>> +        0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
> >>>> +        0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
> >>>> +        0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
> >>>> +        0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
> >>>> +        0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
> >>>> +        0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
> >>>> +        0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
> >>>> +        0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
> >>>> +        0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
> >>>> +        0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
> >>>> +        0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
> >>>> +        0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
> >>>> +        0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
> >>>> +        0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
> >>>> +        0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
> >>>> +        0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
> >>>> +    },
> >>>> +    {    /* Bank 2 */
> >>>> +        0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
> >>>> +        0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
> >>>> +        0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
> >>>> +        0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
> >>>> +        0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
> >>>> +        0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
> >>>> +        0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
> >>>> +        0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
> >>>> +        0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
> >>>> +        0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
> >>>> +        0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
> >>>> +        0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
> >>>> +        0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
> >>>> +        0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
> >>>> +        0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
> >>>> +        0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
> >>>> +    },
> >>>> +    {    /* Bank 3 */
> >>>> +        0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
> >>>> +        0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
> >>>> +        0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
> >>>> +        0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
> >>>> +        0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
> >>>> +        0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
> >>>> +        0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
> >>>> +        0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
> >>>> +        0x20100000, 0x00000010, 0x1f100000, 0x00000011,
> >>>> +        0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
> >>>> +        0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
> >>>> +        0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
> >>>> +        0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
> >>>> +        0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
> >>>> +        0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
> >>>> +        0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
> >>>> +    },
> >>>> +    {    /* Bank 4 */
> >>>> +        0x17170900, 0x00000009, 0x18160900, 0x00000009,
> >>>> +        0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
> >>>> +        0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
> >>>> +        0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
> >>>> +        0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
> >>>> +        0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
> >>>> +        0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
> >>>> +        0x19110400, 0x0000030f, 0x18110400, 0x00000310,
> >>>> +        0x1a100300, 0x00000310, 0x18100300, 0x00000411,
> >>>> +        0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
> >>>> +        0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
> >>>> +        0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
> >>>> +        0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
> >>>> +        0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
> >>>> +        0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
> >>>> +        0x170a0100, 0x00000816, 0x18090000, 0x00000916,
> >>>> +    },
> >>>> +    {    /* Bank 5 */
> >>>> +        0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
> >>>> +        0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
> >>>> +        0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
> >>>> +        0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
> >>>> +        0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
> >>>> +        0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
> >>>> +        0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
> >>>> +        0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
> >>>> +        0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
> >>>> +        0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
> >>>> +        0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
> >>>> +        0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
> >>>> +        0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
> >>>> +        0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
> >>>> +        0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
> >>>> +        0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
> >>>> +    },
> >>>> +    {    /* Bank 6 */
> >>>> +        0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
> >>>> +        0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >>>> +        0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
> >>>> +        0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
> >>>> +        0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
> >>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>> +        0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >>>> +        0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>> +        0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
> >>>> +        0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
> >>>> +        0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
> >>>> +        0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
> >>>> +        0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
> >>>> +    },
> >>>> +    {    /* Bank 7 */
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +    }
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_resizer_coef_bank {
> >>>> +    unsigned int bank;
> >>>
> >>> This is always equal to the index of the entry in the
> >>> mali_c55_coefficient_banks array, you can drop it.
> >>>
> >>>> +    unsigned int top;
> >>>> +    unsigned int bottom;
> >>>
> >>> The bottom value of bank N is always equal to the top value of bank N+1.
> >>> You can simplify this by storing a single value.
> >>>
> >>>> +};
> >>>> +
> >>>> +static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
> >>>> +    {
> >>>> +        .bank = 0,
> >>>> +        .top = 1000,
> >>>> +        .bottom = 770,
> >>>> +    },
> >>>> +    {
> >>>> +        .bank = 1,
> >>>> +        .top = 769,
> >>>> +        .bottom = 600,
> >>>> +    },
> >>>> +    {
> >>>> +        .bank = 2,
> >>>> +        .top = 599,
> >>>> +        .bottom = 460,
> >>>> +    },
> >>>> +    {
> >>>> +        .bank = 3,
> >>>> +        .top = 459,
> >>>> +        .bottom = 354,
> >>>> +    },
> >>>> +    {
> >>>> +        .bank = 4,
> >>>> +        .top = 353,
> >>>> +        .bottom = 273,
> >>>> +    },
> >>>> +    {
> >>>> +        .bank = 5,
> >>>> +        .top = 272,
> >>>> +        .bottom = 210,
> >>>> +    },
> >>>> +    {
> >>>> +        .bank = 6,
> >>>> +        .top = 209,
> >>>> +        .bottom = 162,
> >>>> +    },
> >>>> +    {
> >>>> +        .bank = 7,
> >>>> +        .top = 161,
> >>>> +        .bottom = 125,
> >>>> +    },
> >>>> +};
> >>>> +
> >>>
> >>> A small comment would be nice, such as
> >>>
> >>> /* Select a bank of resizer coefficients, based on the scaling ratio. */
> >>>
> >>>> +static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,
> >>>
> >>> This function is related to the resizers. Add "rsz" somewhere in the
> >>> function name, and pass a resizer pointer.
> >>>
> >>>> +                        unsigned int crop,
> >>>> +                        unsigned int scale)
> >>>
> >>> I think those are the input and output sizes to the scaler. Rename them
> >>> to make it clearer.
> >>>
> >>>> +{
> >>>> +    unsigned int tmp;
> >>>
> >>> tmp is almost always a bad variable name. Please use a more descriptive
> >>> name, size as rsz_ratio.
> >>>
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    tmp = (scale * 1000U) / crop;
> >>>> +
> >>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
> >>>> +        if (tmp >= mali_c55_coefficient_banks[i].bottom &&
> >>>> +            tmp <= mali_c55_coefficient_banks[i].top)
> >>>> +            return mali_c55_coefficient_banks[i].bank;
> >>>> +    }
> >>>> +
> >>>> +    /*
> >>>> +     * We shouldn't ever get here, in theory. As we have no good choices
> >>>> +     * simply warn the user and use the first bank of coefficients.
> >>>> +     */
> >>>> +    dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
> >>>> +    return 0;
> >>>> +}
> >>>
> >>> And everything else belongs to mali-c55-resizer.c. Drop this header
> >>> file.
> >>>
> >>>> +
> >>>> +#endif /* _MALI_C55_RESIZER_COEFS_H */
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c 
> >>>> b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>>> new file mode 100644
> >>>> index 000000000000..0a5a2969d3ce
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>>> @@ -0,0 +1,779 @@
> >>>> +// SPDX-License-Identifier: GPL-2.0
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Image signal processor
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#include <linux/math.h>
> >>>> +#include <linux/minmax.h>
> >>>> +
> >>>> +#include <media/media-entity.h>
> >>>> +#include <media/v4l2-subdev.h>
> >>>> +
> >>>> +#include "mali-c55-common.h"
> >>>> +#include "mali-c55-registers.h"
> >>>> +#include "mali-c55-resizer-coefs.h"
> >>>> +
> >>>> +/* Scaling factor in Q4.20 format. */
> >>>> +#define MALI_C55_RZR_SCALER_FACTOR    (1U << 20)
> >>>> +
> >>>> +static const u32 rzr_non_bypass_src_fmts[] = {
> >>>> +    MEDIA_BUS_FMT_RGB121212_1X36,
> >>>> +    MEDIA_BUS_FMT_YUV10_1X30
> >>>> +};
> >>>> +
> >>>> +static const char * const mali_c55_resizer_names[] = {
> >>>> +    [MALI_C55_RZR_FR] = "resizer fr",
> >>>> +    [MALI_C55_RZR_DS] = "resizer ds",
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
> >>>> +                     struct v4l2_subdev_state *state)
> >>>> +{
> >>>> +    unsigned int reg_offset = rzr->cap_dev->reg_offset;
> >>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>> +    struct v4l2_mbus_framefmt *fmt;
> >>>> +    struct v4l2_rect *crop;
> >>
> >> const
> >>
> >>>> +
> >>>> +    /* Verify if crop should be enabled. */
> >>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>> +
> >>>> +    if (fmt->width == crop->width && fmt->height == crop->height)
> >>>> +        return MALI_C55_BYPASS_CROP;
> >>>> +
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
> >>>> +               crop->left);
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
> >>>> +               crop->top);
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
> >>>> +               crop->width);
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
> >>>> +               crop->height);
> >>>> +
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
> >>>> +               MALI_C55_CROP_ENABLE);
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
> >>>> +                    struct v4l2_subdev_state *state)
> >>>> +{
> >>>> +    unsigned int reg_offset = rzr->cap_dev->reg_offset;
> >>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>> +    struct v4l2_rect *crop, *scale;
> >>
> >> const
> >>
> >> Once "[PATCH v4 0/3] media: v4l2-subdev: Support const-awareness in
> >> state accessors" gets merged, the state argument to this function can be
> >> made const too. Same for other functions, as applicable.
> >>
> >>>> +    unsigned int h_bank, v_bank;
> >>>> +    u64 h_scale, v_scale;
> >>>> +
> >>>> +    /* Verify if scaling should be enabled. */
> >>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>> +    scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>> +
> >>>> +    if (crop->width == scale->width && crop->height == scale->height)
> >>>> +        return MALI_C55_BYPASS_SCALER;
> >>>> +
> >>>> +    /* Program the V/H scaling factor in Q4.20 format. */
> >>>> +    h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
> >>>> +    v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
> >>>> +
> >>>> +    do_div(h_scale, scale->width);
> >>>> +    do_div(v_scale, scale->height);
> >>>> +
> >>>> +    mali_c55_write(mali_c55,
> >>>> +               MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
> >>>> +               crop->width);
> >>>> +    mali_c55_write(mali_c55,
> >>>> +               MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
> >>>> +               crop->height);
> >>>> +
> >>>> +    mali_c55_write(mali_c55,
> >>>> +               MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
> >>>> +               scale->width);
> >>>> +    mali_c55_write(mali_c55,
> >>>> +               MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
> >>>> +               scale->height);
> >>>> +
> >>>> +    mali_c55_write(mali_c55,
> >>>> +               MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
> >>>> +               h_scale);
> >>>> +    mali_c55_write(mali_c55,
> >>>> +               MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
> >>>> +               v_scale);
> >>>> +
> >>>> +    h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
> >>>> +                         scale->width);
> >>>> +    mali_c55_write(mali_c55,
> >>>> +               MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
> >>>> +               h_bank);
> >>>> +
> >>>> +    v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
> >>>> +                         scale->height);
> >>>> +    mali_c55_write(mali_c55,
> >>>> +               MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
> >>>> +               v_bank);
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
> >>>> +                 struct v4l2_subdev_state *state)
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>> +    u32 bypass = 0;
> >>>> +
> >>>> +    /* Verify if cropping and scaling should be enabled. */
> >>>> +    bypass |= mali_c55_rzr_program_crop(rzr, state);
> >>>> +    bypass |= mali_c55_rzr_program_resizer(rzr, state);
> >>>> +
> >>>> +    mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
> >>>> +                 MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
> >>>> +                 MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
> >>>> +                 bypass);
> >>>> +}
> >>>> +
> >>>> +/*
> >>>> + * Inspect the routing table to know which of the two (mutually exclusive)
> >>>> + * routes is enabled and return the sink pad id of the active route.
> >>>> + */
> >>>> +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
> >>>> +{
> >>>> +    struct v4l2_subdev_krouting *routing = &state->routing;
> >>>> +    struct v4l2_subdev_route *route;
> >>>> +
> >>>> +    /* A single route is enabled at a time. */
> >>>> +    for_each_active_route(routing, route)
> >>>> +        return route->sink_pad;
> >>>> +
> >>>> +    return MALI_C55_RZR_SINK_PAD;
> >>>> +}
> >>>> +
> >>>> +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
> >>>> +{
> >>>> +    u32 corrected_code = 0;
> >>>> +
> >>>> +    /*
> >>>> +     * The ISP takes input in a 20-bit format, but can only output 16-bit
> >>>> +     * RAW bayer data (with the 4 least significant bits from the input
> >>>> +     * being lost). Return the 16-bit version of the 20-bit input formats.
> >>>> +     */
> >>>> +    switch (mbus_code) {
> >>>> +    case MEDIA_BUS_FMT_SBGGR20_1X20:
> >>>> +        corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
> >>>> +        break;
> >>>> +    case MEDIA_BUS_FMT_SGBRG20_1X20:
> >>>> +        corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
> >>>> +        break;
> >>>> +    case MEDIA_BUS_FMT_SGRBG20_1X20:
> >>>> +        corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
> >>>> +        break;
> >>>> +    case MEDIA_BUS_FMT_SRGGB20_1X20:
> >>>> +        corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
> >>>> +        break;
> >>
> >> Would it make sense to add the shifted code to mali_c55_isp_fmt ?
> >>
> >>>> +    }
> >>>> +
> >>>> +    return corrected_code;
> >>>> +}
> >>>> +
> >>>> +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> >>>> +                      struct v4l2_subdev_state *state,
> >>>> +                      struct v4l2_subdev_krouting *routing)
> >>
> >> I think the last argument can be const.
> >>
> >>>> +{
> >>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>> +                            sd);
> >>
> >> A to_mali_c55_resizer() static inline function would be useful. Same for
> >> other components, where applicable.
> >>
> >>>> +    unsigned int active_sink = UINT_MAX;
> >>>> +    struct v4l2_mbus_framefmt *src_fmt;
> >>>> +    struct v4l2_rect *crop, *compose;
> >>>> +    struct v4l2_subdev_route *route;
> >>>> +    unsigned int active_routes = 0;
> >>>> +    struct v4l2_mbus_framefmt *fmt;
> >>>> +    int ret;
> >>>> +
> >>>> +    ret = v4l2_subdev_routing_validate(sd, routing, 0);
> >>>> +    if (ret)
> >>>> +        return ret;
> >>>> +
> >>>> +    /* Only a single route can be enabled at a time. */
> >>>> +    for_each_active_route(routing, route) {
> >>>> +        if (++active_routes > 1) {
> >>>> +            dev_err(rzr->mali_c55->dev,
> >>>> +                "Only one route can be active");
> >>
> >> No kernel log message with a level higher than dev_dbg() from
> >> user-controlled paths please, here and where applicable. This is to
> >> avoid giving applications an easy way to flood the kernel log.
> >>
> >>>> +            return -EINVAL;
> >>>> +        }
> >>>> +
> >>>> +        active_sink = route->sink_pad;
> >>>> +    }
> >>>> +    if (active_sink == UINT_MAX) {
> >>>> +        dev_err(rzr->mali_c55->dev, "One route has to be active");
> >>>> +        return -EINVAL;
> >>>> +    }
> >>
> >> The recommended handling of invalid routing is to adjust the routing
> >> table, not to return errors.
> >>
> >>>> +
> >>>> +    ret = v4l2_subdev_set_routing(sd, state, routing);
> >>>> +    if (ret) {
> >>>> +        dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
> >>>> +        return ret;
> >>>> +    }
> >>>> +
> >>>> +    fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
> >>>> +    crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
> >>>> +    compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
> >>>> +
> >>>> +    fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>> +    fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>> +    fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>
> >> There are other colorspace-related fields.
> >>
> >>>> +    fmt->field = V4L2_FIELD_NONE;
> >>
> >> I wonder if we should really update the sink pad format, or just
> >> propagate it. If we update it, I think it should be set to defaults on
> >> both sink pads, not just the active sink pad.
> >>
> >>>> +
> >>>> +    if (active_sink == MALI_C55_RZR_SINK_PAD) {
> >>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>> +
> >>>> +        crop->left = crop->top = 0;
> >>
> >>         crop->left = 0;
> >>         crop->top = 0;
> >>
> >>>> +        crop->width = MALI_C55_DEFAULT_WIDTH;
> >>>> +        crop->height = MALI_C55_DEFAULT_HEIGHT;
> >>>> +
> >>>> +        *compose = *crop;
> >>>> +    } else {
> >>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>> +    }
> >>>> +
> >>>> +    /* Propagate the format to the source pad */
> >>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
> >>>> +                           0);
> >>>> +    *src_fmt = *fmt;
> >>>> +
> >>>> +    /* In the event this is the bypass pad the mbus code needs correcting */
> >>>> +    if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
> >>>> +        src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
> >>>> +                       struct v4l2_subdev_state *state,
> >>>> +                       struct v4l2_subdev_mbus_code_enum *code)
> >>>> +{
> >>>> +    struct v4l2_mbus_framefmt *sink_fmt;
> >>>> +    const struct mali_c55_isp_fmt *fmt;
> >>>> +    unsigned int index = 0;
> >>>> +    u32 sink_pad;
> >>>> +
> >>>> +    switch (code->pad) {
> >>>> +    case MALI_C55_RZR_SINK_PAD:
> >>>> +        if (code->index)
> >>>> +            return -EINVAL;
> >>>> +
> >>>> +        code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>> +
> >>>> +        return 0;
> >>>> +    case MALI_C55_RZR_SOURCE_PAD:
> >>>> +        sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>> +        sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> >>>> +
> >>>> +        /*
> >>>> +         * If the active route is from the Bypass sink pad, then the
> >>>> +         * source pad is a simple passthrough of the sink format,
> >>>> +         * downshifted to 16-bits.
> >>>> +         */
> >>>> +
> >>>> +        if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >>>> +            if (code->index)
> >>>> +                return -EINVAL;
> >>>> +
> >>>> +            code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> >>>> +            if (!code->code)
> >>>> +                return -EINVAL;
> >>>> +
> >>>> +            return 0;
> >>>> +        }
> >>>> +
> >>>> +        /*
> >>>> +         * If the active route is from the non-bypass sink then we can
> >>>> +         * select either RGB or conversion to YUV.
> >>>> +         */
> >>>> +
> >>>> +        if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
> >>>> +            return -EINVAL;
> >>>> +
> >>>> +        code->code = rzr_non_bypass_src_fmts[code->index];
> >>>> +
> >>>> +        return 0;
> >>>> +    case MALI_C55_RZR_SINK_BYPASS_PAD:
> >>>> +        for_each_mali_isp_fmt(fmt) {
> >>>> +            if (index++ == code->index) {
> >>>> +                code->code = fmt->code;
> >>>> +                return 0;
> >>>> +            }
> >>>> +        }
> >>>> +
> >>>> +        break;
> >>>> +    }
> >>>> +
> >>>> +    return -EINVAL;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
> >>>> +                    struct v4l2_subdev_state *state,
> >>>> +                    struct v4l2_subdev_frame_size_enum *fse)
> >>>> +{
> >>>> +    if (fse->index)
> >>>> +        return -EINVAL;
> >>>> +
> >>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
> >>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
> >>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
> >>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
> >>>> +                     struct v4l2_subdev_state *state,
> >>>> +                     struct v4l2_subdev_format *format)
> >>>> +{
> >>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>> +    struct v4l2_rect *rect;
> >>>> +    unsigned int sink_pad;
> >>>> +
> >>>> +    /*
> >>>> +     * Clamp to min/max and then reset crop and compose rectangles to the
> >>>> +     * newly applied size.
> >>>> +     */
> >>>> +    clamp_t(unsigned int, fmt->width,
> >>>> +        MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>>> +    clamp_t(unsigned int, fmt->height,
> >>>> +        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>
> >> Please check comments for other components related to the colorspace
> >> fields, to decide how to handle them here.
> >>
> >>>> +
> >>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>> +    if (sink_pad == MALI_C55_RZR_SINK_PAD) {
> >>
> >> The selection here should depend on format->pad, not the active sink
> >> pad.
> >>
> >>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>> +
> >>>> +        rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> >>>> +        rect->left = 0;
> >>>> +        rect->top = 0;
> >>>> +        rect->width = fmt->width;
> >>>> +        rect->height = fmt->height;
> >>>> +
> >>>> +        rect = v4l2_subdev_state_get_compose(state,
> >>>> +                             MALI_C55_RZR_SINK_PAD);
> >>>> +        rect->left = 0;
> >>>> +        rect->top = 0;
> >>>> +        rect->width = fmt->width;
> >>>> +        rect->height = fmt->height;
> >>>> +    } else {
> >>>> +        /*
> >>>> +         * Make sure the media bus code is one of the supported
> >>>> +         * ISP input media bus codes.
> >>>> +         */
> >>>> +        if (!mali_c55_isp_is_format_supported(fmt->code))
> >>>> +            fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
> >>>> +    }
> >>>> +
> >>>> +    *v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
> >>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
> >>
> >> Propagation to the source pad, however, should depend on the active
> >> route. If format->pad is routed to the source pad, you should propagate,
> >> otherwise, you shouldn't.
> >>
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> >>>> +                       struct v4l2_subdev_state *state,
> >>>> +                       struct v4l2_subdev_format *format)
> >>>> +{
> >>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>> +                            sd);
> >>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>> +    struct v4l2_mbus_framefmt *sink_fmt;
> >>>> +    struct v4l2_rect *crop, *compose;
> >>>> +    unsigned int sink_pad;
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>> +    sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> >>>> +    crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
> >>>> +    compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
> >>>> +
> >>>> +    /* FR Bypass pipe. */
> >>>> +
> >>>> +    if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >>>> +        /*
> >>>> +         * Format on the source pad is the same as the one on the
> >>>> +         * sink pad, downshifted to 16-bits.
> >>>> +         */
> >>>> +        fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> >>>> +        if (!fmt->code)
> >>>> +            return -EINVAL;
> >>>> +
> >>>> +        /* RAW bypass disables scaling and cropping. */
> >>>> +        crop->top = compose->top = 0;
> >>>> +        crop->left = compose->left = 0;
> >>>> +        fmt->width = crop->width = compose->width = sink_fmt->width;
> >>>> +        fmt->height = crop->height = compose->height = sink_fmt->height;
> >>
> >> I don't think this is right. This function sets the format on the source
> >> pad. Subdevs should propagate formats from the sink to the source, not
> >> the other way around.
> >>
> >> The only parameter that can be modified on the source pad (as far as I
> >> understand) is the media bus code. In the bypass path, I understand it's
> >> fixed, while in the other path, you can select between RGB and YUV. I
> >> think the following code is what you need to implement this function.
> >>
> >> static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> >>                        struct v4l2_subdev_state *state,
> >>                        struct v4l2_subdev_format *format)
> >> {
> >>     struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>                             sd);
> >>     struct v4l2_mbus_framefmt *fmt;
> >>
> >>     fmt = v4l2_subdev_state_get_format(state, format->pad);
> >>
> >>     /* In the non-bypass path the output format can be selected. */
> >>     if (mali_c55_rzr_get_active_sink(state) == MALI_C55_RZR_SINK_PAD) {
> >>         unsigned int i;
> >>
> >>         fmt->code = format->format.code;
> >>
> >>         for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> >>             if (fmt->code == rzr_non_bypass_src_fmts[i])
> >>                 break;
> >>         }
> >>
> >>         if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts))
> >>             fmt->code = rzr_non_bypass_src_fmts[0];
> >>     }
> >>
> >>     format->format = *fmt;
> >>
> >>     return 0;
> >> }
> >>
> >>>> +
> >>>> +        *v4l2_subdev_state_get_format(state,
> >>>> +                          MALI_C55_RZR_SOURCE_PAD) = *fmt;
> >>>> +
> >>>> +        return 0;
> >>>> +    }
> >>>> +
> >>>> +    /* Regular processing pipe. */
> >>>> +
> >>>> +    for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> >>>> +        if (fmt->code == rzr_non_bypass_src_fmts[i])
> >>>> +            break;
> >>>> +    }
> >>>> +
> >>>> +    if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
> >>>> +        dev_dbg(rzr->mali_c55->dev,
> >>>> +            "Unsupported mbus code 0x%x: using default\n",
> >>>> +            fmt->code);
> >>
> >> I think you can drop this message.
> >>
> >>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>> +    }
> >>>> +
> >>>> +    /*
> >>>> +     * The source pad format size comes directly from the sink pad
> >>>> +     * compose rectangle.
> >>>> +     */
> >>>> +    fmt->width = compose->width;
> >>>> +    fmt->height = compose->height;
> >>>> +
> >>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
> >>>> +                struct v4l2_subdev_state *state,
> >>>> +                struct v4l2_subdev_format *format)
> >>>> +{
> >>>> +    /*
> >>>> +     * On sink pads fmt is either fixed for the 'regular' processing
> >>>> +     * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
> >>>> +     * pad.
> >>>> +     *
> >>>> +     * On source pad sizes are the result of crop+compose on the sink
> >>>> +     * pad sizes, while the format depends on the active route.
> >>>> +     */
> >>>> +
> >>>> +    if (format->pad != MALI_C55_RZR_SOURCE_PAD)
> >>>> +        return mali_c55_rzr_set_sink_fmt(sd, state, format);
> >>>> +
> >>>> +    return mali_c55_rzr_set_source_fmt(sd, state, format);
> >>
> >> Nitpicking,
> >>
> >>     if (format->pad == MALI_C55_RZR_SOURCE_PAD)
> >>         return mali_c55_rzr_set_source_fmt(sd, state, format);
> >>
> >>     return mali_c55_rzr_set_sink_fmt(sd, state, format);
> >>
> >> to match SOURCE_PAD and source_fmt.
> >>
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
> >>>> +                      struct v4l2_subdev_state *state,
> >>>> +                      struct v4l2_subdev_selection *sel)
> >>>> +{
> >>>> +    if (sel->pad != MALI_C55_RZR_SINK_PAD)
> >>>> +        return -EINVAL;
> >>>> +
> >>>> +    if (sel->target != V4L2_SEL_TGT_CROP &&
> >>>> +        sel->target != V4L2_SEL_TGT_COMPOSE)
> >>>> +        return -EINVAL;
> >>>> +
> >>>> +    sel->r = sel->target == V4L2_SEL_TGT_CROP
> >>>> +           ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
> >>>> +           : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
> >>>> +                      struct v4l2_subdev_state *state,
> >>>> +                      struct v4l2_subdev_selection *sel)
> >>>> +{
> >>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>> +                            sd);
> >>>> +    struct v4l2_mbus_framefmt *source_fmt;
> >>>> +    struct v4l2_mbus_framefmt *sink_fmt;
> >>>> +    struct v4l2_rect *crop, *compose;
> >>>> +
> >>>> +    if (sel->pad != MALI_C55_RZR_SINK_PAD)
> >>>> +        return -EINVAL;
> >>>> +
> >>>> +    if (sel->target != V4L2_SEL_TGT_CROP &&
> >>>> +        sel->target != V4L2_SEL_TGT_COMPOSE)
> >>>> +        return -EINVAL;
> >>>> +
> >>>> +    source_fmt = v4l2_subdev_state_get_format(state,
> >>>> +                          MALI_C55_RZR_SOURCE_PAD);
> >>>> +    sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
> >>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> >>>> +    compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> >>>> +
> >>>> +    /* RAW bypass disables crop/scaling. */
> >>>> +    if (mali_c55_format_is_raw(source_fmt->code)) {
> >>>> +        crop->top = compose->top = 0;
> >>>> +        crop->left = compose->left = 0;
> >>>> +        crop->width = compose->width = sink_fmt->width;
> >>>> +        crop->height = compose->height = sink_fmt->height;
> >>>> +
> >>>> +        sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> >>>> +
> >>>> +        return 0;
> >>>> +    }
> >>>> +
> >>>> +    /* During streaming, it is allowed to only change the crop rectangle. */
> >>>> +    if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
> >>>> +        return -EINVAL;
> >>>> +
> >>>> +     /*
> >>>> +      * Update the desired target and then clamp the crop rectangle to the
> >>>> +      * sink format sizes and the compose size to the crop sizes.
> >>>> +      */
> >>>> +    if (sel->target == V4L2_SEL_TGT_CROP)
> >>>> +        *crop = sel->r;
> >>>> +    else
> >>>> +        *compose = sel->r;
> >>>> +
> >>>> +    clamp_t(unsigned int, crop->left, 0, sink_fmt->width);
> >>>> +    clamp_t(unsigned int, crop->top, 0, sink_fmt->height);
> >>>> +    clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
> >>>> +        sink_fmt->width - crop->left);
> >>>> +    clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
> >>>> +        sink_fmt->height - crop->top);
> >>>> +
> >>>> +    if (rzr->streaming) {
> >>>> +        /*
> >>>> +         * Apply at runtime a crop rectangle on the resizer's sink only
> >>>> +         * if it doesn't require re-programming the scaler output sizes
> >>>> +         * as it would require changing the output buffer sizes as well.
> >>>> +         */
> >>>> +        if (sel->r.width < compose->width ||
> >>>> +            sel->r.height < compose->height)
> >>>> +            return -EINVAL;
> >>>> +
> >>>> +        *crop = sel->r;
> >>>> +        mali_c55_rzr_program(rzr, state);
> >>>> +
> >>>> +        return 0;
> >>>> +    }
> >>>> +
> >>>> +    compose->left = 0;
> >>>> +    compose->top = 0;
> >>>> +    clamp_t(unsigned int, compose->left, 0, sink_fmt->width);
> >>>> +    clamp_t(unsigned int, compose->top, 0, sink_fmt->height);
> >>>> +    clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
> >>>> +    clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
> >>>> +
> >>>> +    sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> >>>> +                    struct v4l2_subdev_state *state,
> >>>> +                    enum v4l2_subdev_format_whence which,
> >>>> +                    struct v4l2_subdev_krouting *routing)
> >>>> +{
> >>>> +    if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
> >>>> +        media_entity_is_streaming(&sd->entity))
> >>>> +        return -EBUSY;
> >>>> +
> >>>> +    return __mali_c55_rzr_set_routing(sd, state, routing);
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
> >>>> +    .enum_mbus_code        = mali_c55_rzr_enum_mbus_code,
> >>>> +    .enum_frame_size    = mali_c55_rzr_enum_frame_size,
> >>>> +    .get_fmt        = v4l2_subdev_get_fmt,
> >>>> +    .set_fmt        = mali_c55_rzr_set_fmt,
> >>>> +    .get_selection        = mali_c55_rzr_get_selection,
> >>>> +    .set_selection        = mali_c55_rzr_set_selection,
> >>>> +    .set_routing        = mali_c55_rzr_set_routing,
> >>>> +};
> >>>> +
> >>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
> >>
> >> Could this be handled through the .enable_streams() and
> >> .disable_streams() operations ? They ensure that the stream state stored
> >> internal is correct. That may not matter much today, but I think it will
> >> become increasingly important in the future for the V4L2 core.
> >>
> >>>> +{
> >>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>> +    struct v4l2_subdev *sd = &rzr->sd;
> >>>> +    struct v4l2_subdev_state *state;
> >>>> +    unsigned int sink_pad;
> >>>> +
> >>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>> +
> >>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>> +    if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >>>> +        /* Bypass FR pipe processing if the bypass route is active. */
> >>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >>>> + MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
> >>>> + MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
> >>>> +        goto unlock_state;
> >>>> +    }
> >>>> +
> >>>> +    /* Disable bypass and use regular processing. */
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >>>> +                 MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
> >>>> +    mali_c55_rzr_program(rzr, state);
> >>>> +
> >>>> +unlock_state:
> >>>> +    rzr->streaming = true;
> >>
> >> And hopefully you'll be able to replace this with
> >> v4l2_subdev_is_streaming(), introduced in "[PATCH v6 00/11] media:
> >> subdev: Improve stream enable/disable machinery" (Sakari has sent a pull
> >> request for v6.11 yesterday).
> >>
> >>>> +    v4l2_subdev_unlock_state(state);
> >>>> +}
> >>>> +
> >>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
> >>>> +{
> >>>> +    struct v4l2_subdev *sd = &rzr->sd;
> >>>> +    struct v4l2_subdev_state *state;
> >>>> +
> >>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>> +    rzr->streaming = false;
> >>>> +    v4l2_subdev_unlock_state(state);
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
> >>>> +    .pad    = &mali_c55_resizer_pad_ops,
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
> >>>> +                   struct v4l2_subdev_state *state)
> >>>> +{
> >>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>> +                            sd);
> >>>> +    struct v4l2_subdev_krouting routing = { };
> >>>> +    struct v4l2_subdev_route *routes;
> >>>> +    unsigned int i;
> >>>> +    int ret;
> >>>> +
> >>>> +    routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
> >>>> +    if (!routes)
> >>>> +        return -ENOMEM;
> >>>> +
> >>>> +    for (i = 0; i < rzr->num_routes; ++i) {
> >>>> +        struct v4l2_subdev_route *route = &routes[i];
> >>>> +
> >>>> +        route->sink_pad = i
> >>>> +                ? MALI_C55_RZR_SINK_BYPASS_PAD
> >>>> +                : MALI_C55_RZR_SINK_PAD;
> >>>> +        route->source_pad = MALI_C55_RZR_SOURCE_PAD;
> >>>> +        if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
> >>>> +            route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
> >>>> +    }
> >>>> +
> >>>> +    routing.num_routes = rzr->num_routes;
> >>>> +    routing.routes = routes;
> >>>> +
> >>>> +    ret = __mali_c55_rzr_set_routing(sd, state, &routing);
> >>>> +    kfree(routes);
> >>>> +
> >>>> +    return ret;
> >>
> >> I think this could be simplified.
> >>
> >>     struct v4l2_subdev_route routes[2] = {
> >>         {
> >>             .sink_pad = MALI_C55_RZR_SINK_PAD,
> >>             .source_pad = MALI_C55_RZR_SOURCE_PAD,
> >>             .flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> >>         }, {
> >>             .sink_pad = MALI_C55_RZR_SINK_BYPASS_PAD,
> >>             .source_pad = MALI_C55_RZR_SOURCE_PAD,
> >>         },
> >>     };
> >>     struct v4l2_subdev_krouting routing = {
> >>         .num_routes = rzr->num_routes,
> >>         .routes = routes,
> >>     };
> >>
> >>     return __mali_c55_rzr_set_routing(sd, state, &routing);
> >>
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
> >>>> +    .init_state = mali_c55_rzr_init_state,
> >>>> +};
> >>>> +
> >>>> +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
> >>>> +                          unsigned int index)
> >>>> +{
> >>>> +    const unsigned int scaler_filt_coefmem_addrs[][2] = {
> >>>> +        [MALI_C55_RZR_FR] = {
> >>>> +            0x034A8, /* hfilt */
> >>>> +            0x044A8  /* vfilt */
> >>>
> >>> Lowercase hex constants.
> >>
> >> And addresses belong to the mali-c55-registers.h file.
> >>
> >>>> +        },
> >>>> +        [MALI_C55_RZR_DS] = {
> >>>> +            0x014A8, /* hfilt */
> >>>> +            0x024A8  /* vfilt */
> >>>> +        },
> >>>> +    };
> >>>> +    unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
> >>>> +    unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
> >>>> +    unsigned int i, j;
> >>>> +
> >>>> +    for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
> >>>> +        for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
> >>>> +            mali_c55_write(mali_c55, haddr,
> >>>> + mali_c55_scaler_h_filter_coefficients[i][j]);
> >>>> +            mali_c55_write(mali_c55, vaddr,
> >>>> + mali_c55_scaler_v_filter_coefficients[i][j]);
> >>>> +
> >>>> +            haddr += sizeof(u32);
> >>>> +            vaddr += sizeof(u32);
> >>>> +        }
> >>>> +    }
> >>
> >> How about memcpy_toio() ? I suppose this function isn't
> >> performance sensitive, so maybe usage of mali_c55_write() is better from
> >> a consistency point of view.
> >>
> >>>> +}
> >>>> +
> >>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    unsigned int i;
> >>>> +    int ret;
> >>>> +
> >>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
> >>
> >> Moving the inner content to a separate mali_c55_register_resizer()
> >> function would increase readability I think, and remove usage of gotos.
> >> I would probably do the same for unregistration too, for consistency.
> >>
> >>>> +        struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> >>>> +        struct v4l2_subdev *sd = &rzr->sd;
> >>>> +        unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
> >>>> +
> >>>> +        rzr->id = i;
> >>>> +        rzr->streaming = false;
> >>>> +
> >>>> +        if (rzr->id == MALI_C55_RZR_FR)
> >>>> +            rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
> >>>> +        else
> >>>> +            rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
> >>>> +
> >>>> +        mali_c55_resizer_program_coefficients(mali_c55, i);
> >>
> >> Should this be done at stream start, given that power may be cut off
> >> between streaming sessions ?
> >>
> >>>> +
> >>>> +        v4l2_subdev_init(sd, &mali_c55_resizer_ops);
> >>>> +        sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
> >>>> +                 | V4L2_SUBDEV_FL_STREAMS;
> >>>> +        sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
> >>>> +        sd->internal_ops = &mali_c55_resizer_internal_ops;
> >>>> +        snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
> >>
> >>         snprintf(sd->name, ARRAY_SIZE(sd->name), "%s resizer %s",
> >>
> >> and drop the "resizer " prefix from mali_c55_resizer_names. You can also
> >> make mali_c55_resizer_names a local static const variable.
> >>
> >>>> +             MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
> >>>> +
> >>>> +        rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
> >>>> +        rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
> >>>> +
> >>>> +        /* Only the FR pipe has a bypass pad. */
> >>>> +        if (rzr->id == MALI_C55_RZR_FR) {
> >>>> + rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
> >>>> +                            MEDIA_PAD_FL_SINK;
> >>>> +            rzr->num_routes = 2;
> >>>> +        } else {
> >>>> +            num_pads -= 1;
> >>>> +            rzr->num_routes = 1;
> >>>> +        }
> >>>> +
> >>>> +        ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
> >>>> +        if (ret)
> >>>> +            return ret;
> >>>> +
> >>>> +        ret = v4l2_subdev_init_finalize(sd);
> >>>> +        if (ret)
> >>>> +            goto err_cleanup;
> >>>> +
> >>>> +        ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >>>> +        if (ret)
> >>>> +            goto err_cleanup;
> >>>> +
> >>>> +        rzr->mali_c55 = mali_c55;
> >>>> +    }
> >>>> +
> >>>> +    return 0;
> >>>> +
> >>>> +err_cleanup:
> >>>> +    for (; i >= 0; --i) {
> >>>> +        struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> >>>> +        struct v4l2_subdev *sd = &rzr->sd;
> >>>> +
> >>>> +        v4l2_subdev_cleanup(sd);
> >>>> +        media_entity_cleanup(&sd->entity);
> >>>> +    }
> >>>> +
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
> >>>> +        struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
> >>>> +
> >>>> +        if (!resizer->mali_c55)
> >>>> +            continue;
> >>>> +
> >>>> +        v4l2_device_unregister_subdev(&resizer->sd);
> >>>> +        v4l2_subdev_cleanup(&resizer->sd);
> >>>> +        media_entity_cleanup(&resizer->sd.entity);
> >>>> +    }
> >>>> +}
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c 
> >>>> b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >>>> new file mode 100644
> >>>> index 000000000000..c7e699741c6d
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >>>> @@ -0,0 +1,402 @@
> >>>> +// SPDX-License-Identifier: GPL-2.0
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Test pattern generator
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#include <linux/minmax.h>
> >>>> +#include <linux/string.h>
> >>>> +
> >>>> +#include <media/media-entity.h>
> >>>> +#include <media/v4l2-ctrls.h>
> >>>> +#include <media/v4l2-subdev.h>
> >>>> +
> >>>> +#include "mali-c55-common.h"
> >>>> +#include "mali-c55-registers.h"
> >>>> +
> >>>> +#define MALI_C55_TPG_SRC_PAD        0
> >>>> +#define MALI_C55_TPG_FIXED_HBLANK    0x20
> >>>> +#define MALI_C55_TPG_MAX_VBLANK        0xFFFF
> >>>
> >>> Lowercase hex constants.
> >>>
> >>>> +#define MALI_C55_TPG_PIXEL_RATE        100000000
> >>>
> >>> This should be exposed to applications using the V4L2_CID_PIXEL_RATE
> >>> control (read-only).
> >>>
> >>>> +
> >>>> +static const char * const mali_c55_tpg_test_pattern_menu[] = {
> >>>> +    "Flat field",
> >>>> +    "Horizontal gradient",
> >>>> +    "Vertical gradient",
> >>>> +    "Vertical bars",
> >>>> +    "Arbitrary rectangle",
> >>>> +    "White frame on black field"
> >>>> +};
> >>>> +
> >>>> +static const u32 mali_c55_tpg_mbus_codes[] = {
> >>>> +    MEDIA_BUS_FMT_SRGGB20_1X20,
> >>>> +    MEDIA_BUS_FMT_RGB202020_1X60,
> >>>> +};
> >>>> +
> >>>> +static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
> >>>> +                       int *def_vblank, int *min_vblank)
> >>>
> >>> unsigned int ?
> >>>
> >>>> +{
> >>>> +    unsigned int hts;
> >>>> +    int tgt_fps;
> >>>> +    int vblank;
> >>>> +
> >>>> +    hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
> >>>> +
> >>>> +    /*
> >>>> +     * The ISP has minimum vertical blanking requirements that must be
> >>>> +     * adhered to by the TPG. The minimum is a function of the Iridix blocks
> >>>> +     * clocking requirements and the width of the image and horizontal
> >>>> +     * blanking, but if we assume the worst case iVariance and sVariance
> >>>> +     * values then it boils down to the below.
> >>>> +     */
> >>>> +    *min_vblank = 15 + (120500 / hts);
> >>>
> >>> I wonder if this should round up.
> >>>
> >>>> +
> >>>> +    /*
> >>>> +     * We need to set a sensible default vblank for whatever format height
> >>>> +     * we happen to be given from set_fmt(). This function just targets
> >>>> +     * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
> >>>> +     * If we can't get 5fps we'll take whatever the minimum vblank gives us.
> >>>> +     */
> >>>> +    tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
> >>>> +
> >>>> +    if (tgt_fps < 5)
> >>>> +        vblank = *min_vblank;
> >>>> +    else
> >>>> +        vblank = MALI_C55_TPG_PIXEL_RATE / hts
> >>>> +               / max(rounddown(tgt_fps, 15), 5);
> >>>> +
> >>>> +    *def_vblank = ALIGN_DOWN(vblank, 2) - format->height;
> >>>
> >>> "vblank = vblank - height" doesn't seem right. The "else" branch stores
> >>> a vts in vblank, which doesn't seem right either. Maybe you meant
> >>> something like
> >>>
> >>>     if (tgt_fps < 5)
> >>>         def_vts = *min_vblank + format->height;
> >>>     else
> >>>         def_vts = MALI_C55_TPG_PIXEL_RATE / hts
> >>>             / max(rounddown(tgt_fps, 15), 5);
> >>>
> >>>     *def_vblank = ALIGN_DOWN(def_vts, 2) - format->height;
> >>>
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
> >>>> +{
> >>>> +    struct mali_c55_tpg *tpg = container_of(ctrl->handler,
> >>>> +                        struct mali_c55_tpg,
> >>>> +                        ctrls.handler);
> >>>> +    struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
> >>>> +
> >>>
> >>> Should you return here if the pipeline isn't streaming ?
> >>>
> >>>> +    switch (ctrl->id) {
> >>>> +    case V4L2_CID_TEST_PATTERN:
> >>>> +        mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
> >>>> +                   ctrl->val);
> >>>> +        break;
> >>>> +    case V4L2_CID_VBLANK:
> >>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
> >>>> +                     MALI_C55_REG_VBLANK_MASK, ctrl->val);
> >>>> +        break;
> >>>> +    default:
> >>>> +        dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
> >>>> +        return -EINVAL;
> >>>
> >>> Can this happen ?
> >>>
> >>>> +    }
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
> >>>> +    .s_ctrl = &mali_c55_tpg_s_ctrl,
> >>>> +};
> >>>> +
> >>>> +static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
> >>>> +                   struct v4l2_subdev *sd)
> >>>> +{
> >>>> +    struct v4l2_subdev_state *state;
> >>>> +    struct v4l2_mbus_framefmt *fmt;
> >>>> +
> >>>> +    /*
> >>>> +     * hblank needs setting, but is a read-only control and thus won't be
> >>>> +     * called during __v4l2_ctrl_handler_setup(). Do it here instead.
> >>>> +     */
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
> >>>> +                 MALI_C55_REG_HBLANK_MASK,
> >>>> +                 MALI_C55_TPG_FIXED_HBLANK);
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >>>> +                 MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
> >>>> +
> >>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>> +
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >>>> +                 MALI_C55_TEST_PATTERN_RGB_MASK,
> >>>> +                 fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
> >>>> +                      0x01 : 0x0);
> >>>> +
> >>>> +    v4l2_subdev_unlock_state(state);
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
> >>>> +{
> >>>> +    struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
> >>>> +    struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
> >>>> +
> >>>> +    if (!enable) {
> >>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >>>> +                MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
> >>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >>>> +                MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
> >>>> +        return 0;
> >>>> +    }
> >>>> +
> >>>> +    /*
> >>>> +     * One might reasonably expect the framesize to be set here
> >>>> +     * given it's configurable in .set_fmt(), but it's done in the
> >>>> +     * ISP subdevice's stream on func instead, as the same register
> >>>
> >>> s/func/function/
> >>>
> >>>> +     * is also used to indicate the size of the data coming from the
> >>>> +     * sensor.
> >>>> +     */
> >>>> +    mali_c55_tpg_configure(mali_c55, sd);
> >>>
> >>>     mali_c55_tpg_configure(tpg);
> >>>
> >>>> + __v4l2_ctrl_handler_setup(sd->ctrl_handler);
> >>>> +
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >>>> +                 MALI_C55_TEST_PATTERN_ON_OFF,
> >>>> +                 MALI_C55_TEST_PATTERN_ON_OFF);
> >>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >>>> +                 MALI_C55_REG_GEN_VIDEO_ON_MASK,
> >>>> +                 MALI_C55_REG_GEN_VIDEO_ON_MASK);
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
> >>>> +    .s_stream = &mali_c55_tpg_s_stream,
> >>>
> >>> Can we use .enable_streams() and .disable_streams() ?
> >>>
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
> >>>> +                       struct v4l2_subdev_state *state,
> >>>> +                       struct v4l2_subdev_mbus_code_enum *code)
> >>>> +{
> >>>> +    if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>>> +        return -EINVAL;
> >>>> +
> >>>> +    code->code = mali_c55_tpg_mbus_codes[code->index];
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
> >>>> +                    struct v4l2_subdev_state *state,
> >>>> +                    struct v4l2_subdev_frame_size_enum *fse)
> >>>> +{
> >>>
> >>> You sohuld verify here that fse->code is a supported value and return
> >>> -EINVAL otherwise.
> >>>
> >>>> +    if (fse->index > 0 || fse->pad > sd->entity.num_pads)
> >>>
> >>> Drop the pad check, it's done in the subdev core already.
> >>>
> >>>> +        return -EINVAL;
> >>>> +
> >>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
> >>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
> >>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
> >>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
> >>>> +                struct v4l2_subdev_state *state,
> >>>> +                struct v4l2_subdev_format *format)
> >>>> +{
> >>>> +    struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
> >>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>> +    int vblank_def, vblank_min;
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> >>>> +        if (fmt->code == mali_c55_tpg_mbus_codes[i])
> >>>> +            break;
> >>>> +    }
> >>>> +
> >>>> +    if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>> +
> >>>> +    /*
> >>>> +     * The TPG says that the test frame timing generation logic expects a
> >>>> +     * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
> >>>> +     * handle anything smaller than 128x128 it seems pointless to allow a
> >>>> +     * smaller frame.
> >>>> +     */
> >>>> +    clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> >>>> +        MALI_C55_MAX_WIDTH);
> >>>> +    clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> >>>> +        MALI_C55_MAX_HEIGHT);
> >>>> +
> >>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;
> >>>
> >>> You're allowing userspace to set fmt->field, as well as all the
> >>> colorspace parameters, to random values. I would instead do something
> >>> like
> >>>
> >>>     for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> >>>         if (format->format.code == mali_c55_tpg_mbus_codes[i])
> >>>             break;
> >>>     }
> >>>
> >>>     if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>>         format->format.code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>
> >>>     format->format.width = clamp(format->format.width,
> >>>                      MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>>     format->format.height = clamp(format->format.height,
> >>>                       MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>>
> >>>     fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>     fmt->code = format->format.code;
> >>>     fmt->width = format->format.width;
> >>>     fmt->height = format->format.height;
> >>>
> >>>     format->format = *fmt;
> >>>
> >>> Alternatively (which I think I like better),
> >>>
> >>>     fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>
> >>>     fmt->code = format->format.code;
> >>>
> >>>     for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> >>>         if (fmt->code == mali_c55_tpg_mbus_codes[i])
> >>>             break;
> >>>     }
> >>>
> >>>     if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>>         fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>
> >>>     fmt->width = clamp(format->format.width,
> >>>                MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>>     fmt->height = clamp(format->format.height,
> >>>                 MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>>
> >>>     format->format = *fmt;
> >>>
> >>>> +
> >>>> +    if (format->which == V4L2_SUBDEV_FORMAT_TRY)
> >>>> +        return 0;
> >>>> +
> >>>> +    __mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
> >>>> +    __v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
> >>>> +                 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
> >>>> +    __v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);
> >>>
> >>> Move those three calls to a separate function, it will be reused below.
> >>> I'd name is mali_c55_tpg_update_vblank(). You can fold
> >>> __mali_c55_tpg_calc_vblank() in it.
> >>>
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
> >>>> +    .enum_mbus_code        = mali_c55_tpg_enum_mbus_code,
> >>>> +    .enum_frame_size    = mali_c55_tpg_enum_frame_size,
> >>>> +    .get_fmt        = v4l2_subdev_get_fmt,
> >>>> +    .set_fmt        = mali_c55_tpg_set_fmt,
> >>>> +};
> >>>> +
> >>>> +static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
> >>>> +    .video    = &mali_c55_tpg_video_ops,
> >>>> +    .pad    = &mali_c55_tpg_pad_ops,
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
> >>>> +                   struct v4l2_subdev_state *sd_state)
> >>>
> >>> You name this variable state in every other subdev operation handler.
> >>>
> >>>> +{
> >>>> +    struct v4l2_mbus_framefmt *fmt =
> >>>> +        v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
> >>>> +
> >>>> +    fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>> +    fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>> +    fmt->field = V4L2_FIELD_NONE;
> >>>> +    fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>
> >>> Initialize the colorspace fields too.
> >>>
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
> >>>> +    .init_state = mali_c55_tpg_init_state,
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
> >>>> +    struct v4l2_subdev *sd = &mali_c55->tpg.sd;
> >>>> +    struct v4l2_mbus_framefmt *format;
> >>>> +    struct v4l2_subdev_state *state;
> >>>> +    int vblank_def, vblank_min;
> >>>> +    int ret;
> >>>> +
> >>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>> +    format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>> +
> >>>> +    ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);
> >>>
> >>> You have 3 controls.
> >>>
> >>>> +    if (ret)
> >>>> +        goto err_unlock;
> >>>> +
> >>>> +    ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
> >>>> +                &mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
> >>>> +                ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
> >>>> +                0, 3, mali_c55_tpg_test_pattern_menu);
> >>>> +
> >>>> +    /*
> >>>> +     * We fix hblank at the minimum allowed value and control framerate
> >>>> +     * solely through the vblank control.
> >>>> +     */
> >>>> +    ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
> >>>> +                &mali_c55_tpg_ctrl_ops,
> >>>> +                V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
> >>>> +                MALI_C55_TPG_FIXED_HBLANK, 1,
> >>>> +                MALI_C55_TPG_FIXED_HBLANK);
> >>>> +    if (ctrls->hblank)
> >>>> +        ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
> >>>> +
> >>>> +    __mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);
> >>>
> >>> Drop this and initialize the control with default values. You can then
> >>> update the value by calling mali_c55_tpg_update_vblank() in
> >>> mali_c55_register_tpg().
> >>>
> >>> The reason is to share the same mutex between the control handler and
> >>> the subdev active state without having to add a separate mutex in the
> >>> mali_c55_tpg structure. The simplest way to do so is to initialize the
> >>> controls first, set sd->state_lock to point to the control handler lock,
> >>> and call v4l2_subdev_init_finalize() as the last step. As a consequence,
> >>> you can't access the active state when initializing controls.
> >>>
> >>> You can alternatively keep the lock in mali_c55_tpg and set
> >>> sd->state_lock to point to it, but I think that's more complex.
> >>>
> >>>> +    ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
> >>>> +                      &mali_c55_tpg_ctrl_ops,
> >>>> +                      V4L2_CID_VBLANK, vblank_min,
> >>>> +                      MALI_C55_TPG_MAX_VBLANK, 1,
> >>>> +                      vblank_def);
> >>>> +
> >>>> +    if (ctrls->handler.error) {
> >>>> +        dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
> >>>> +        ret = ctrls->handler.error;
> >>>> +        goto err_free_handler;
> >>>> +    }
> >>>> +
> >>>> +    ctrls->handler.lock = &mali_c55->tpg.lock;
> >>>
> >>> Drop this and drop the mutex. The control handler will use its internal
> >>> mutex.
> >>>
> >>>> +    mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
> >>>> +
> >>>> +    v4l2_subdev_unlock_state(state);
> >>>> +
> >>>> +    return 0;
> >>>> +
> >>>> +err_free_handler:
> >>>> +    v4l2_ctrl_handler_free(&ctrls->handler);
> >>>> +err_unlock:
> >>>> +    v4l2_subdev_unlock_state(state);
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    struct mali_c55_tpg *tpg = &mali_c55->tpg;
> >>>> +    struct v4l2_subdev *sd = &tpg->sd;
> >>>> +    struct media_pad *pad = &tpg->pad;
> >>>> +    int ret;
> >>>> +
> >>>> +    mutex_init(&tpg->lock);
> >>>> +
> >>>> +    v4l2_subdev_init(sd, &mali_c55_tpg_ops);
> >>>> +    sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> >>>> +    sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
> >>>
> >>> Should we introduce a TPG function ?
> >>>
> >>>> +    sd->internal_ops = &mali_c55_tpg_internal_ops;
> >>>> +    strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
> >>>> +
> >>>> +    pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;
> >>>
> >>> I don't think MEDIA_PAD_FL_MUST_CONNECT is right.
> >>>
> >>>> +    ret = media_entity_pads_init(&sd->entity, 1, pad);
> >>>> +    if (ret) {
> >>>> +        dev_err(mali_c55->dev,
> >>>> +            "Failed to initialize media entity pads\n");
> >>>> +        goto err_destroy_mutex;
> >>>> +    }
> >>>> +
> >>>
> >>>     sd->state_lock = sd->ctrl_handler->lock;
> >>>
> >>> to use the same lock for the controls and the active state. You need to
> >>> move this line and the v4l2_subdev_init_finalize() call after
> >>> mali_c55_tpg_init_controls() to get the control handler lock initialized
> >>> first.
> >>>
> >>>> +    ret = v4l2_subdev_init_finalize(sd);
> >>>> +    if (ret)
> >>>> +        goto err_cleanup_media_entity;
> >>>> +
> >>>> +    ret = mali_c55_tpg_init_controls(mali_c55);
> >>>> +    if (ret) {
> >>>> +        dev_err(mali_c55->dev,
> >>>> +            "Error initialising controls\n");
> >>>> +        goto err_cleanup_subdev;
> >>>> +    }
> >>>> +
> >>>> +    ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >>>> +    if (ret) {
> >>>> +        dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
> >>>> +        goto err_free_ctrl_handler;
> >>>> +    }
> >>>> +
> >>>> +    /*
> >>>> +     * By default the colour settings lead to a very dim image that is
> >>>> +     * nearly indistinguishable from black on some monitor settings. Ramp
> >>>> +     * them up a bit so the image is brighter.
> >>>> +     */
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
> >>>> +               MALI_C55_TPG_BACKGROUND_MAX);
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
> >>>> +               MALI_C55_TPG_BACKGROUND_MAX);
> >>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
> >>>> +               MALI_C55_TPG_BACKGROUND_MAX);
> >>>> +
> >>>> +    tpg->mali_c55 = mali_c55;
> >>>> +
> >>>> +    return 0;
> >>>> +
> >>>> +err_free_ctrl_handler:
> >>>> +    v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> >>>> +err_cleanup_subdev:
> >>>> +    v4l2_subdev_cleanup(sd);
> >>>> +err_cleanup_media_entity:
> >>>> +    media_entity_cleanup(&sd->entity);
> >>>> +err_destroy_mutex:
> >>>> +    mutex_destroy(&tpg->lock);
> >>>> +
> >>>> +    return ret;
> >>>> +}
> >>>> +
> >>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +    struct mali_c55_tpg *tpg = &mali_c55->tpg;
> >>>> +
> >>>> +    if (!tpg->mali_c55)
> >>>> +        return;
> >>>> +
> >>>> +    v4l2_device_unregister_subdev(&tpg->sd);
> >>>> +    v4l2_subdev_cleanup(&tpg->sd);
> >>>> +    media_entity_cleanup(&tpg->sd.entity);
> >>>> +    v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> >>>
> >>> Free the control handler just after v4l2_device_unregister_subdev() to
> >>> match the order in mali_c55_register_tpg().
> >>>
> >>>> +    mutex_destroy(&tpg->lock);
> >>>> +}

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Dan Scally]

Hi Laurent

On 20/06/2024 16:23, Laurent Pinchart wrote:
> Hi Dan,
>
> On Thu, Jun 20, 2024 at 03:49:23PM +0100, Daniel Scally wrote:
>> On 20/06/2024 15:33, Dan Scally wrote:
>>> On 30/05/2024 22:43, Laurent Pinchart wrote:
>>>> And now the second part of the review, addressing mali-c55-capture.c and
>>>> mali-c55-resizer.c. I've reviewed the code from the bottom up, so some
>>>> messages may be repeated in an order that seems weird. Sorry about that.
>>>>
>>>> On Thu, May 30, 2024 at 03:15:10AM +0300, Laurent Pinchart wrote:
>>>>> On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
>>>>>> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
>>>>>> V4L2 and Media Controller compliant and creates subdevices to manage
>>>>>> the ISP itself, its internal test pattern generator as well as the
>>>>>> crop, scaler and output format functionality for each of its two
>>>>>> output devices.
>>>>>>
>>>>>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
>>>>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>>>>>> ---
>>>>>> Changes in v5:
>>>>>>
>>>>>>      - Reworked input formats - previously we allowed representing input data
>>>>>>        as any 8-16 bit format. Now we only allow input data to be represented
>>>>>>        by the new 20-bit bayer formats, which is corrected to the equivalent
>>>>>>        16-bit format in RAW bypass mode.
>>>>>>      - Stopped bypassing blocks that we haven't added supporting parameters
>>>>>>        for yet.
>>>>>>      - Addressed most of Sakari's comments from the list
>>>>>>
>>>>>> Changes not yet made in v5:
>>>>>>
>>>>>>      - The output pipelines can still be started and stopped independently of
>>>>>>        one another - I'd like to discuss that more.
>>>>>>      - the TPG subdev still uses .s_stream() - I need to rebase onto a tree
>>>>>>        with working .enable_streams() for a single-source-pad subdevice.
>>>>>>
>>>>>> Changes in v4:
>>>>>>
>>>>>>      - Reworked mali_c55_update_bits() to internally perform the bit-shift
>>>>> I really don't like that, it makes the code very confusing, even more so
>>>>> as it differs from regmap_update_bits().
>>>>>
>>>>> Look at this for instance:
>>>>>
>>>>>      mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>>>                   MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
>>>>>                   MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
>>>>>
>>>>> It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
>>>>> BIT(0).
>>>>>
>>>>> Sorry, I know it will be painful, but this change needs to be reverted.
>>>>>
>>>>>>      - Reworked the resizer to allow cropping during streaming
>>>>>>      - Fixed a bug in NV12 output
>>>>>>
>>>>>> Changes in v3:
>>>>>>
>>>>>>      - Mostly minor fixes suggested by Sakari
>>>>>>      - Fixed the sequencing of vb2 buffers to be synchronised across the two
>>>>>>        capture devices.
>>>>>>
>>>>>> Changes in v2:
>>>>>>
>>>>>>      - Clock handling
>>>>>>      - Fixed the warnings raised by the kernel test robot
>>>>>>
>>>>>>    drivers/media/platform/Kconfig                |   1 +
>>>>>>    drivers/media/platform/Makefile               |   1 +
>>>>>>    drivers/media/platform/arm/Kconfig            |   5 +
>>>>>>    drivers/media/platform/arm/Makefile           |   2 +
>>>>>>    drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
>>>>>>    drivers/media/platform/arm/mali-c55/Makefile  |   9 +
>>>>>>    .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
>>>>>>    .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
>>>>>>    .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
>>>>>>    .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
>>>>>>    .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
>>>>>>    .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
>>>>>>    .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
>>>>>>    .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
>>>>>>    14 files changed, 4452 insertions(+)
>>>>>>    create mode 100644 drivers/media/platform/arm/Kconfig
>>>>>>    create mode 100644 drivers/media/platform/arm/Makefile
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>>>> I've skipped review of capture.c and resizer.c as I already have plenty
>>>>> of comments for the other files, and it's getting late. I'll try to
>>>>> review the rest tomorrow.
>>>>>
>>>>>> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
>>>>>> index 2d79bfc68c15..c929169766aa 100644
>>>>>> --- a/drivers/media/platform/Kconfig
>>>>>> +++ b/drivers/media/platform/Kconfig
>>>>>> @@ -65,6 +65,7 @@ config VIDEO_MUX
>>>>>>    source "drivers/media/platform/allegro-dvt/Kconfig"
>>>>>>    source "drivers/media/platform/amlogic/Kconfig"
>>>>>>    source "drivers/media/platform/amphion/Kconfig"
>>>>>> +source "drivers/media/platform/arm/Kconfig"
>>>>>>    source "drivers/media/platform/aspeed/Kconfig"
>>>>>>    source "drivers/media/platform/atmel/Kconfig"
>>>>>>    source "drivers/media/platform/broadcom/Kconfig"
>>>>>> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
>>>>>> index da17301f7439..9a647abd5218 100644
>>>>>> --- a/drivers/media/platform/Makefile
>>>>>> +++ b/drivers/media/platform/Makefile
>>>>>> @@ -8,6 +8,7 @@
>>>>>>    obj-y += allegro-dvt/
>>>>>>    obj-y += amlogic/
>>>>>>    obj-y += amphion/
>>>>>> +obj-y += arm/
>>>>>>    obj-y += aspeed/
>>>>>>    obj-y += atmel/
>>>>>>    obj-y += broadcom/
>>>>>> diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
>>>>>> new file mode 100644
>>>>>> index 000000000000..4f0764c329c7
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/Kconfig
>>>>>> @@ -0,0 +1,5 @@
>>>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>>>> +
>>>>>> +comment "ARM media platform drivers"
>>>>>> +
>>>>>> +source "drivers/media/platform/arm/mali-c55/Kconfig"
>>>>>> diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
>>>>>> new file mode 100644
>>>>>> index 000000000000..8cc4918725ef
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/Makefile
>>>>>> @@ -0,0 +1,2 @@
>>>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>>>> +obj-y += mali-c55/
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/Kconfig
>>>>>> b/drivers/media/platform/arm/mali-c55/Kconfig
>>>>>> new file mode 100644
>>>>>> index 000000000000..602085e28b01
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/Kconfig
>>>>>> @@ -0,0 +1,18 @@
>>>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>>>> +config VIDEO_MALI_C55
>>>>>> +    tristate "ARM Mali-C55 Image Signal Processor driver"
>>>>>> +    depends on V4L_PLATFORM_DRIVERS
>>>>>> +    depends on VIDEO_DEV && OF
>>>>>> +    depends on ARCH_VEXPRESS || COMPILE_TEST
>>>>>> +    select MEDIA_CONTROLLER
>>>>>> +    select VIDEO_V4L2_SUBDEV_API
>>>>>> +    select VIDEOBUF2_DMA_CONTIG
>>>>>> +    select VIDEOBUF2_VMALLOC
>>>>>> +    select V4L2_FWNODE
>>>>>> +    select GENERIC_PHY_MIPI_DPHY
>>>>> Alphabetical order ?
>>>>>
>>>>>> +    default n
>>>>> That's the default, you don't have to specify ti.
>>>>>
>>>>>> +    help
>>>>>> +      Enable this to support Arm's Mali-C55 Image Signal Processor.
>>>>>> +
>>>>>> +      To compile this driver as a module, choose M here: the module
>>>>>> +      will be called mali-c55.
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile
>>>>>> b/drivers/media/platform/arm/mali-c55/Makefile
>>>>>> new file mode 100644
>>>>>> index 000000000000..77dcb2fbf0f4
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
>>>>>> @@ -0,0 +1,9 @@
>>>>>> +# SPDX-License-Identifier: GPL-2.0
>>>>>> +
>>>>>> +mali-c55-y := mali-c55-capture.o \
>>>>>> +          mali-c55-core.o \
>>>>>> +          mali-c55-isp.o \
>>>>>> +          mali-c55-tpg.o \
>>>>>> +          mali-c55-resizer.o
>>>>> Alphabetical order here too.
>>>>>
>>>>>> +
>>>>>> +obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>>>> new file mode 100644
>>>>>> index 000000000000..1d539ac9c498
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>>>> @@ -0,0 +1,951 @@
>>>>>> +// SPDX-License-Identifier: GPL-2.0
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Video capture devices
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#include <linux/cleanup.h>
>>>>>> +#include <linux/minmax.h>
>>>>>> +#include <linux/pm_runtime.h>
>>>>>> +#include <linux/string.h>
>>>>>> +#include <linux/videodev2.h>
>>>>>> +
>>>>>> +#include <media/v4l2-dev.h>
>>>>>> +#include <media/v4l2-event.h>
>>>>>> +#include <media/v4l2-ioctl.h>
>>>>>> +#include <media/v4l2-subdev.h>
>>>>>> +#include <media/videobuf2-core.h>
>>>>>> +#include <media/videobuf2-dma-contig.h>
>>>>>> +
>>>>>> +#include "mali-c55-common.h"
>>>>>> +#include "mali-c55-registers.h"
>>>>>> +
>>>>>> +static const struct mali_c55_fmt mali_c55_fmts[] = {
>>>>>> +    /*
>>>>>> +     * This table is missing some entries which need further work or
>>>>>> +     * investigation:
>>>>>> +     *
>>>>>> +     * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
>>>>>> +     * Base mode 5 is "Generic Data"
>>>>>> +     * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
>>>>>> +     * Base mode 9 seems to have no V4L2 equivalent
>>>>>> +     * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
>>>>>> +     * equivalent
>>>>>> +     */
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_ARGB2101010,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
>>>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_A2R10G10B10,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_RGB565,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
>>>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_RGB565,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_BGR24,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
>>>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_RGB24,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_YUYV,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_YUY2,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_UYVY,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_UYVY,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_Y210,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_Y210,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    /*
>>>>>> +     * This is something of a hack, the ISP thinks it's running NV12M but
>>>>>> +     * by setting uv_plane = 0 we simply discard that planes and only output
>>>>>> +     * the Y-plane.
>>>>>> +     */
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_GREY,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_NV12M,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_NV21M,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
>>>>>> +        }
>>>>>> +    },
>>>>>> +    /*
>>>>>> +     * RAW uncompressed formats are all packed in 16 bpp.
>>>>>> +     * TODO: Expand this list to encompass all possible RAW formats.
>>>>>> +     */
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_SRGGB16,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_SRGGB16_1X16,
>>>>>> +        },
>>>>>> +        .is_raw = true,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_SBGGR16,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_SBGGR16_1X16,
>>>>>> +        },
>>>>>> +        .is_raw = true,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_SGBRG16,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_SGBRG16_1X16,
>>>>>> +        },
>>>>>> +        .is_raw = true,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_SGRBG16,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_SGRBG16_1X16,
>>>>>> +        },
>>>>>> +        .is_raw = true,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +};
>>>>>> +
>>>>>> +static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
>>>>>> +                           u32 code)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
>>>>>> +        if (fmt->mbus_codes[i] == code)
>>>>>> +            return true;
>>>>>> +    }
>>>>>> +
>>>>>> +    return false;
>>>>>> +}
>>>>>> +
>>>>>> +bool mali_c55_format_is_raw(unsigned int mbus_code)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>>>>> +        if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
>>>>>> +            return mali_c55_fmts[i].is_raw;
>>>>>> +    }
>>>>>> +
>>>>>> +    return false;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>>>>> +        if (mali_c55_fmts[i].fourcc == pixelformat)
>>>>>> +            return &mali_c55_fmts[i];
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * If we find no matching pixelformat, we'll just default to the first
>>>>>> +     * one for now.
>>>>>> +     */
>>>>>> +
>>>>>> +    return &mali_c55_fmts[0];
>>>>>> +}
>>>>>> +
>>>>>> +static const char * const capture_device_names[] = {
>>>>>> +    "mali-c55 fr",
>>>>>> +    "mali-c55 ds",
>>>>>> +    "mali-c55 3a stats",
>>>>>> +    "mali-c55 params",
>>>> The last two entries are not used AFAICT, neither here, nor in
>>>> subsequent patches.
>>>>
>>>>>> +};
>>>>>> +
>>>>>> +static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
>>>>>> +{
>>>>>> +    if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
>>>>>> +        return capture_device_names[0];
>>>>>> +
>>>>>> +    if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
>>>>>> +        return capture_device_names[1];
>>>>>> +
>>>>>> +    return "params/stat not supported yet";
>>>>>> +}
>>>> Use cap_dev->vdev.name instead of mali_c55_cap_dev_to_name(cap_dev) and
>>>> drop this function.
>>>>
>>>>>> +
>>>>>> +static int mali_c55_link_validate(struct media_link *link)
>>>>>> +{
>>>>>> +    struct video_device *vdev =
>>>>>> + media_entity_to_video_device(link->sink->entity);
>>>>>> +    struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
>>>>>> +    struct v4l2_subdev *sd =
>>>>>> + media_entity_to_v4l2_subdev(link->source->entity);
>>>>>> +    const struct v4l2_pix_format_mplane *pix_mp;
>>>>>> +    const struct mali_c55_fmt *cap_fmt;
>>>>>> +    struct v4l2_subdev_format sd_fmt = {
>>>>>> +        .which = V4L2_SUBDEV_FORMAT_ACTIVE,
>>>>>> +        .pad = link->source->index,
>>>>>> +    };
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
>>>>>> +    if (ret)
>>>>>> +        return ret;
>>>>>> +
>>>>>> +    pix_mp = &cap_dev->mode.pix_mp;
>>>>>> +    cap_fmt = cap_dev->mode.capture_fmt;
>>>>>> +
>>>>>> +    if (sd_fmt.format.width != pix_mp->width ||
>>>>>> +        sd_fmt.format.height != pix_mp->height) {
>>>>>> +        dev_dbg(cap_dev->mali_c55->dev,
>>>>>> +            "link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
>>>>>> +            link->source->entity->name, link->source->index,
>>>>>> +            link->sink->entity->name, link->sink->index,
>>>>>> +            sd_fmt.format.width, sd_fmt.format.height,
>>>>>> +            pix_mp->width, pix_mp->height);
>>>>>> +        return -EPIPE;
>>>>>> +    }
>>>>>> +
>>>>>> +    if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
>>>>>> +        dev_dbg(cap_dev->mali_c55->dev,
>>>>>> +            "link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format
>>>>>> %p4cc\n",
>>>>>> +            link->source->entity->name, link->source->index,
>>>>>> +            link->sink->entity->name, link->sink->index,
>>>>>> +            sd_fmt.format.code, &pix_mp->pixelformat);
>>>>>> +        return -EPIPE;
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct media_entity_operations mali_c55_media_ops = {
>>>>>> +    .link_validate = mali_c55_link_validate,
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
>>>>>> +                    unsigned int *num_planes, unsigned int sizes[],
>>>>>> +                    struct device *alloc_devs[])
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    if (*num_planes) {
>>>>>> +        if (*num_planes != cap_dev->mode.pix_mp.num_planes)
>>>>>> +            return -EINVAL;
>>>>>> +
>>>>>> +        for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>>>>> +            if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
>>>>>> +                return -EINVAL;
>>>>>> +    } else {
>>>>>> +        *num_planes = cap_dev->mode.pix_mp.num_planes;
>>>>>> +        for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>>>>> +            sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_buf_queue(struct vb2_buffer *vb)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
>>>>>> +    struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>>>>>> +    struct mali_c55_buffer *buf = container_of(vbuf,
>>>>>> +                           struct mali_c55_buffer, vb);
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    buf->plane_done[MALI_C55_PLANE_Y] = false;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * If we're in a single-plane format we flag the other plane as done
>>>>>> +     * already so it's dequeued appropriately later
>>>>>> +     */
>>>>>> +    buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
>>>>>> +
>>>>>> +    for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
>>>>>> +        unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
>>>>>> +
>>>>>> +        vb2_set_plane_payload(vb, i, size);
>>>>>> +    }
>>>>>> +
>>>>>> +    spin_lock(&cap_dev->buffers.lock);
>>>>>> +    list_add_tail(&buf->queue, &cap_dev->buffers.queue);
>>>>>> +    spin_unlock(&cap_dev->buffers.lock);
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_buf_init(struct vb2_buffer *vb)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
>>>>>> +    struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>>>>>> +    struct mali_c55_buffer *buf = container_of(vbuf,
>>>>>> +                           struct mali_c55_buffer, vb);
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>>>>> +        buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>>>> +
>>>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
>>>>>> +
>>>>>> +    cap_dev->buffers.curr = cap_dev->buffers.next;
>>>>>> +    cap_dev->buffers.next = NULL;
>>>>>> +
>>>>>> +    if (!list_empty(&cap_dev->buffers.queue)) {
>>>>>> +        struct v4l2_pix_format_mplane *pix_mp;
>>>>>> +        const struct v4l2_format_info *info;
>>>>>> +        u32 *addrs;
>>>>>> +
>>>>>> +        pix_mp = &cap_dev->mode.pix_mp;
>>>>>> +        info = v4l2_format_info(pix_mp->pixelformat);
>>>>>> +
>>>>>> +        mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
>>>>>> +        if (cap_dev->mode.capture_fmt->registers.uv_plane)
>>>>>> +            mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
>>>>>> +
>>>>>> +        cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
>>>>>> +                             struct mali_c55_buffer,
>>>>>> +                             queue);
>>>>>> +        list_del(&cap_dev->buffers.next->queue);
>>>>>> +
>>>>>> +        addrs = cap_dev->buffers.next->addrs;
>>>>>> +        mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
>>>>>> +            addrs[MALI_C55_PLANE_Y]);
>>>>>> +        mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
>>>>>> +            addrs[MALI_C55_PLANE_UV]);
>>>>>> +        mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
>>>>>> +            pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
>>>>>> +        mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
>>>>>> +            pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
>>>>>> +            / info->hdiv);
>>>>>> +    } else {
>>>>>> +        /*
>>>>>> +         * If we underflow then we can tell the ISP that we don't want
>>>>>> +         * to write out the next frame.
>>>>>> +         */
>>>>>> +        mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>>>> +        mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>>>> +    }
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
>>>>>> +                   unsigned int framecount)
>>>>>> +{
>>>>>> +    curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>>>>>> +    curr_buf->vb.field = V4L2_FIELD_NONE;
>>>> The could be set already when the buffer is queued.
>>>>
>>>>>> +    curr_buf->vb.sequence = framecount;
>>>>>> +    vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
>>>>>> +}
>>>>>> +
>>>>>> +/**
>>>>>> + * mali_c55_set_plane_done - mark the plane as written and process the buffer if
>>>>>> + *                 both planes are finished.
>>>>>> + * @cap_dev:  pointer to the fr or ds pipe output
>>>>>> + * @plane:    the plane to mark as completed
>>>>>> + *
>>>>>> + * The Mali C55 ISP has muliplanar outputs for some formats that come with two
>>>>>> + * separate "buffer write completed" interrupts - we need to flag each plane's
>>>>>> + * completion and check whether both planes are done - if so, complete the buf
>>>>>> + * in vb2.
>>>>>> + */
>>>>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>>>>>> +                 enum mali_c55_planes plane)
>>>>>> +{
>>>>>> +    struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
>>>>>> +    struct mali_c55_buffer *curr_buf;
>>>>>> +
>>>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
>>>>>> +    curr_buf = cap_dev->buffers.curr;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * This _should_ never happen. If no buffer was available from vb2 then
>>>>>> +     * we tell the ISP not to bother writing the next frame, which means the
>>>>>> +     * interrupts that call this function should never trigger. If it does
>>>>>> +     * happen then one of our assumptions is horribly wrong - complain
>>>>>> +     * loudly and do nothing.
>>>>>> +     */
>>>>>> +    if (!curr_buf) {
>>>>>> +        dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
>>>>>> +            mali_c55_cap_dev_to_name(cap_dev), __func__);
>>>>>> +        return;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* If the other plane is also done... */
>>>>>> +    if (curr_buf->plane_done[~plane & 1]) {
>>>>>> +        mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
>>>>>> +        cap_dev->buffers.curr = NULL;
>>>>>> +        isp->frame_sequence++;
>>>>>> +    } else {
>>>>>> +        curr_buf->plane_done[plane] = true;
>>>>>> +    }
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>>>> +
>>>>>> +    mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                 MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>>>> +    mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                 MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The Mali ISP can hold up to 5 buffer addresses and simply cycle
>>>>>> +     * through them, but it's not clear to me that the vb2 queue _guarantees_
>>>>>> +     * it will queue buffers to the driver in a fixed order, and ensuring
>>>>>> +     * we call vb2_buffer_done() for the right buffer seems to me to add
>>>>>> +     * pointless complexity given in multi-context mode we'd need to
>>>>>> +     * re-write those registers every frame anyway...so we tell the ISP to
>>>>>> +     * use a single register and update it for each frame.
>>>>>> +     */
>>>> A single register sounds prone to error conditions. Is it at least
>>>> shadowed in the hardware, or do you have to make sure you reprogram it
>>>> during the vertical blanking only ?
>>> It would have to be reprogrammed during the vertical blanking if we were running in a
>>> configuration with a single config space, otherwise you have the time it takes to process a frame
>>> plus vertical blanking. As I say, it'll have to work like this in multi-context mode anyway.
>>>
>>> If we want to use the cycling...is it guaranteed that vb2 buffers will always be queued in order?
> In which order ?
>
>>>> I'll mostly skip buffer handling in this review, I need to first
>>>> understand how the hardware operates to make an informed opinion.
>>>>
>>>>>> +    mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
>>>>>> +            MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
>>>>>> +    mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
>>>>>> +            MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * We only queue a buffer in the streamon path if this is the first of
>>>>>> +     * the capture devices to start streaming. If the ISP is already running
>>>>>> +     * then we rely on the ISP_START interrupt to queue the first buffer for
>>>>>> +     * this capture device.
>>>>>> +     */
>>>>>> +    if (mali_c55->pipe.start_count == 1)
>>>>>> +        mali_c55_set_next_buffer(cap_dev);
>>>> I think we'll have to revisit buffer handling to make sure it's 100%
>>>> race-free.
>>>>
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
>>>>>> +                        enum vb2_buffer_state state)
>>>>>> +{
>>>>>> +    struct mali_c55_buffer *buf, *tmp;
>>>>>> +
>>>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
>>>>>> +
>>>>>> +    if (cap_dev->buffers.curr) {
>>>>>> + vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
>>>>>> +                state);
>>>>>> +        cap_dev->buffers.curr = NULL;
>>>>>> +    }
>>>>>> +
>>>>>> +    if (cap_dev->buffers.next) {
>>>>>> + vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
>>>>>> +                state);
>>>>>> +        cap_dev->buffers.next = NULL;
>>>>>> +    }
>>>>>> +
>>>>>> +    list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
>>>>>> +        list_del(&buf->queue);
>>>>>> +        vb2_buffer_done(&buf->vb.vb2_buf, state);
>>>>>> +    }
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>>>> +    struct mali_c55_resizer *rzr = cap_dev->rzr;
>>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    guard(mutex)(&isp->lock);
>>>> What's the reason for using the isp lock here and in
>>>> mali_c55_vb2_stop_streaming() ? If you need a lock that covers all video
>>>> nodes in order to synchronize start/stop, you may want to use the
>>>> graph_mutex of the media device instead.
>>> It's because I wanted to make sure that the ISP was in a known started/stopped state before
>>> possibly trying to start/stop it, which can be done from either of the two capture devices. This
>>> would go away if we were synchronising with the links anyway.
> OK.
>
>>>>>> +
>>>>>> +    ret = pm_runtime_resume_and_get(mali_c55->dev);
>>>>>> +    if (ret)
>>>>>> +        return ret;
>>>>>> +
>>>>>> +    ret = video_device_pipeline_start(&cap_dev->vdev,
>>>>>> +                      &cap_dev->mali_c55->pipe);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
>>>>>> +            mali_c55_cap_dev_to_name(cap_dev));
>>>> Drop the message or make it dev_dbg() as it can be triggered by
>>>> userspace.
>>>>
>>>>>> +        goto err_pm_put;
>>>>>> +    }
>>>>>> +
>>>>>> +    mali_c55_cap_dev_stream_enable(cap_dev);
>>>>>> +    mali_c55_rzr_start_stream(rzr);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * We only start the ISP if we're the only capture device that's
>>>>>> +     * streaming. Otherwise, it'll already be active.
>>>>>> +     */
>>>> I still think we should use link setup to indicate which video devices
>>>> userspace plans to use, and then only start when they're all started.
>>>> That includes stats and parameters buffers. We can continue this
>>>> discussion in the context of the previous version of the patch series,
>>>> or here, up to you.
>>> Let's just continue here. I think I called it "clunky" before; from my perspective it's an
>>> unnecessary extra step - we can already signal to the driver that we don't want to use the video
>>> devices by not queuing buffers to them or starting the stream on them and although I understand
> By not starting streaming, perhaps, but by not queuing buffers, no. The
> reason is that there's no synchronization between buffer queues. If you
> queue
>
> Frame	FR	DS
> --------------------
> 1	x	x
> 2	x
> 3	x	x
> 4	x	x
>
> it will not be distinguishable by the driver from
>
> Frame	FR	DS
> --------------------
> 1	x	x
> 2	x	x
> 3	x	x
> 4	x
>
>>> that that means that one of the two image data capture devices will receive data before the other,
>>> I don't understand why that's considered to be a problem. Possibly that last part is the stickler;
>>> can you explain a bit why it's an issue for one capture queue to start earlier than the other?
> Because from a userspace point of view, if you want to capture frames
> from both pipelines, you will expect to receive a buffer from each
> pipeline for every frame. If that's not guaranteed at stream start, you
> will then need to implement synchronization code that will drop buffers
> on one pipeline until you get the first buffer on the other pipeline
> (assuming you can synchronize them by sequence number). That will be
> more work, and can introduce latency.
>
>>>>>> +    if (mali_c55->pipe.start_count == 1) {
>>>>>> +        ret = mali_c55_isp_start_stream(isp);
>>>>>> +        if (ret)
>>>>>> +            goto err_disable_cap_dev;
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_disable_cap_dev:
>>>>>> +    mali_c55_cap_dev_stream_disable(cap_dev);
>>>>>> +    video_device_pipeline_stop(&cap_dev->vdev);
>>>>>> +err_pm_put:
>>>>>> +    pm_runtime_put(mali_c55->dev);
>>>>>> +    mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>>>> +    struct mali_c55_resizer *rzr = cap_dev->rzr;
>>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
>>>>>> +
>>>>>> +    guard(mutex)(&isp->lock);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * If one of the other capture nodes is streaming, we shouldn't
>>>>>> +     * disable the ISP here.
>>>>>> +     */
>>>>>> +    if (mali_c55->pipe.start_count == 1)
>>>>>> +        mali_c55_isp_stop_stream(&mali_c55->isp);
>>>>>> +
>>>>>> +    mali_c55_rzr_stop_stream(rzr);
>>>>>> +    mali_c55_cap_dev_stream_disable(cap_dev);
>>>>>> +    mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
>>>>>> +    video_device_pipeline_stop(&cap_dev->vdev);
>>>>>> +    pm_runtime_put(mali_c55->dev);
>>>> I think runtime PM autosuspend would be very useful, as it will ensure
>>>> that stop-reconfigure-start cycles get handled as efficiently as
>>>> possible without powering the device down. It could be done on top as a
>>>> separate patch.
>>> Alright
>>>
>>>>>> +}
>>>>>> +
>>>>>> +static const struct vb2_ops mali_c55_vb2_ops = {
>>>>>> +    .queue_setup        = &mali_c55_vb2_queue_setup,
>>>>>> +    .buf_queue        = &mali_c55_buf_queue,
>>>>>> +    .buf_init        = &mali_c55_buf_init,
>>>>>> +    .wait_prepare        = vb2_ops_wait_prepare,
>>>>>> +    .wait_finish        = vb2_ops_wait_finish,
>>>>>> +    .start_streaming    = &mali_c55_vb2_start_streaming,
>>>>>> +    .stop_streaming        = &mali_c55_vb2_stop_streaming,
>>>>>> +};
>>>>>> +
>>>>>> +static const struct v4l2_file_operations mali_c55_v4l2_fops = {
>>>>>> +    .owner = THIS_MODULE,
>>>>>> +    .unlocked_ioctl = video_ioctl2,
>>>>>> +    .open = v4l2_fh_open,
>>>>>> +    .release = vb2_fop_release,
>>>>>> +    .poll = vb2_fop_poll,
>>>>>> +    .mmap = vb2_fop_mmap,
>>>>>> +};
>>>>>> +
>>>>>> +static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
>>>>>> +{
>>>>>> +    const struct mali_c55_fmt *capture_format;
>>>>>> +    const struct v4l2_format_info *info;
>>>>>> +    struct v4l2_plane_pix_format *plane;
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
>>>>>> +    pix_mp->pixelformat = capture_format->fourcc;
>>>>>> +
>>>>>> +    pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
>>>>>> +                  MALI_C55_MAX_WIDTH);
>>>>>> +    pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
>>>>>> +                   MALI_C55_MAX_HEIGHT);
>>>> Ah, these clamps are right :-)
>>> Hurrah!
>>>
>>>>>> +
>>>>>> +    pix_mp->field = V4L2_FIELD_NONE;
>>>>>> +    pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>>> +    pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>>> +    pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
>>>>>> +
>>>>>> +    info = v4l2_format_info(pix_mp->pixelformat);
>>>> This function may return NULL. That shouldn't be the case as long as it
>>>> supports all formats that the C55 driver supports, so I suppose it's
>>>> safe.
>>>>
>>>>>> +    pix_mp->num_planes = info->mem_planes;
>>>>>> +    memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
>>>>>> +
>>>>>> +    pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;
>>>> Does the hardware support configurable line strides ? If so we should
>>>> support it.
>>> You have to set the line stride in the DMA writer registers, which we do using this same
>>> value...might userspace have set bytesperline already then or something? Or is there some other
>>> place it could be configured?
> Userspace can request a specific stride by setting bytesperline, yes. If
> that's set, you should honour it (and of course adjust it to a
> reasonable [min, max] range as well as align it based on hardware
> constraints).
>
>>>>>> +    pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
>>>>>> +                       * pix_mp->height;
>>>>      pix_mp->plane_fmt[0].sizeimage = pix_mp->plane_fmt[0].bytesperline
>>>>                         * pix_mp->height;
>>>>
>>>>>> +
>>>>>> +    for (i = 1; i < info->comp_planes; i++) {
>>>>>> +        plane = &pix_mp->plane_fmt[i];
>>>>>> +
>>>>>> +        plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
>>>>>> +                           info->hdiv);
>>>>>> +        plane->sizeimage = DIV_ROUND_UP(
>>>>>> +                    plane->bytesperline * pix_mp->height,
>>>>>> +                    info->vdiv);
>>>>>> +    }
>>>>>> +
>>>>>> +    if (info->mem_planes == 1) {
>>>>>> +        for (i = 1; i < info->comp_planes; i++) {
>>>>>> +            plane = &pix_mp->plane_fmt[i];
>>>>>> +            pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
>>>>>> +        }
>>>>>> +    }
>>>> I'm wondering, could v4l2_fill_pixfmt_mp() help ? It doesn't support
>>>> configurable strides though :-S Maybe the helper could be improved, if
>>>> it's close enough to what you need ?
>>> I'll take a look
>>>
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>>>> +                       struct v4l2_format *f)
>>>>>> +{
>>>>>> +    mali_c55_try_fmt(&f->fmt.pix_mp);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
>>>>>> +                struct v4l2_pix_format_mplane *pix_mp)
>>>>>> +{
>>>>>> +    const struct mali_c55_fmt *capture_format;
>>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>>>> +    const struct v4l2_format_info *info;
>>>>>> +
>>>>>> +    mali_c55_try_fmt(pix_mp);
>>>>>> +    capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
>>>>>> +    info = v4l2_format_info(pix_mp->pixelformat);
>>>>>> +    if (WARN_ON(!info))
>>>>>> +        return;
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +               capture_format->registers.base_mode);
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
>>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
>>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
>>>> Could the register writes be moved to stream start time ?
>> Sorry missed this one. These are writes to the context's registers
>> buffer, not to the hardware. Does it matter that they're not done at
>> stream on time?
> Writing them here means you'll have to call pm_runtime_resume_and_get()
> here. If power is then cut off, registers may or may not lose their
> contents, so you would need to write them at stream on time anyway. I
> think it's best to move all the hardware configuration at stream on
> time.


They won't be lost, because they're not written to the hardware here, only to the registers buffer 
we allocate in mali_c55_init_context(). They're automatically written to the hardware at stream on 
time when the config is DMAd over, so unless I'm missing something this is safe as an 
operation...though possibly the confusion makes them worth moving anyway.

>>>>>> +
>>>>>> +    if (info->mem_planes > 1) {
>>>>>> +        mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                   capture_format->registers.base_mode);
>>>>>> +        mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                MALI_C55_WRITER_SUBMODE_MASK,
>>>>>> +                capture_format->registers.uv_plane);
>>>>>> +
>>>>>> +        mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
>>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
>>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
>>>>>> +    }
>>>>>> +
>>>>>> +    if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
>>>>>> +        /*
>>>>>> +         * TODO: Figure out the colour matrix coefficients and calculate
>>>>>> +         * and write them here.
>>>>>> +         */
>>>> Ideally they should also be exposed directly to userspace as ISP
>>>> parameters. I would probably go as far as saying that they should come
>>>> directly from userspace, and not derived from the colorspace fields.
>>> Yes I think I agree, I'll drop the todo from here.
>>>
>>>>>> +
>>>>>> +        mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>>>> +                   MALI_C55_CS_CONV_MATRIX_MASK);
>>>>>> +
>>>>>> +        if (info->hdiv > 1)
>>>>>> +            mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>>>> +                MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
>>>>>> +        if (info->vdiv > 1)
>>>>>> +            mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>>>> +                MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
>>>>>> +        if (info->hdiv > 1 || info->vdiv > 1)
>>>>>> +            mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>>>> +                MALI_C55_CS_CONV_FILTER_MASK, 0x01);
>>>>>> +    }
>>>>>> +
>>>>>> +    cap_dev->mode.pix_mp = *pix_mp;
>>>>>> +    cap_dev->mode.capture_fmt = capture_format;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>>>> +                     struct v4l2_format *f)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>>>>> +
>>>>>> +    if (vb2_is_busy(&cap_dev->queue))
>>>>>> +        return -EBUSY;
>>>>>> +
>>>>>> +    mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>>>> +                     struct v4l2_format *f)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>>>>> +
>>>>>> +    f->fmt.pix_mp = cap_dev->mode.pix_mp;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>>>> +                        struct v4l2_fmtdesc *f)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>>>>> +    unsigned int j = 0;
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>>>>> +        if (f->mbus_code &&
>>>>>> + !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
>>>>>> +                               f->mbus_code))
>>>> Small indentation mistake.
>>>>
>>>>>> +            continue;
>>>>>> +
>>>>>> +        /* Downscale pipe can't output RAW formats */
>>>>>> +        if (mali_c55_fmts[i].is_raw &&
>>>>>> +            cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
>>>>>> +            continue;
>>>>>> +
>>>>>> +        if (j++ == f->index) {
>>>>>> +            f->pixelformat = mali_c55_fmts[i].fourcc;
>>>>>> +            return 0;
>>>>>> +        }
>>>>>> +    }
>>>>>> +
>>>>>> +    return -EINVAL;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_querycap(struct file *file, void *fh,
>>>>>> +                 struct v4l2_capability *cap)
>>>>>> +{
>>>>>> +    strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
>>>>>> +    strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
>>>>>> +    .vidioc_reqbufs = vb2_ioctl_reqbufs,
>>>>>> +    .vidioc_querybuf = vb2_ioctl_querybuf,
>>>>>> +    .vidioc_create_bufs = vb2_ioctl_create_bufs,
>>>>>> +    .vidioc_qbuf = vb2_ioctl_qbuf,
>>>>>> +    .vidioc_expbuf = vb2_ioctl_expbuf,
>>>>>> +    .vidioc_dqbuf = vb2_ioctl_dqbuf,
>>>>>> +    .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
>>>>>> +    .vidioc_streamon = vb2_ioctl_streamon,
>>>>>> +    .vidioc_streamoff = vb2_ioctl_streamoff,
>>>>>> +    .vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
>>>>>> +    .vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
>>>>>> +    .vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
>>>>>> +    .vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
>>>>>> +    .vidioc_querycap = mali_c55_querycap,
>>>>>> +    .vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
>>>>>> +    .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
>>>>>> +};
>>>>>> +
>>>>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct v4l2_pix_format_mplane pix_mp;
>>>>>> +    struct mali_c55_cap_dev *cap_dev;
>>>>>> +    struct video_device *vdev;
>>>>>> +    struct vb2_queue *vb2q;
>>>>>> +    unsigned int i;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
>>>> Moving the inner content to a separate mali_c55_register_capture_dev()
>>>> function would increase readability I think, and remove usage of gotos.
>>>> I would probably do the same for unregistration too, for consistency.
>>>>
>>>>>> +        cap_dev = &mali_c55->cap_devs[i];
>>>>>> +        vdev = &cap_dev->vdev;
>>>>>> +        vb2q = &cap_dev->queue;
>>>>>> +
>>>>>> +        /*
>>>>>> +         * The downscale output pipe is an optional block within the ISP
>>>>>> +         * so we need to check whether it's actually been fitted or not.
>>>>>> +         */
>>>>>> +
>>>>>> +        if (i == MALI_C55_CAP_DEV_DS &&
>>>>>> +            !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
>>>>>> +            continue;
>>>> Given that there's only two capture devices, and one is optional, when
>>>> moving the inner code to a separate function you could unroll the loop.
>>>> Up to you.
>>>>
>>>>>> +
>>>>>> +        cap_dev->mali_c55 = mali_c55;
>>>>>> +        mutex_init(&cap_dev->lock);
>>>>>> +        INIT_LIST_HEAD(&cap_dev->buffers.queue);
>>>>>> +
>>>>>> +        switch (i) {
>>>>>> +        case MALI_C55_CAP_DEV_FR:
>>>>>> +            cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
>>>>>> +            cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
>>>>>> +            break;
>>>>>> +        case MALI_C55_CAP_DEV_DS:
>>>>>> +            cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
>>>>>> +            cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
>>>>>> +            break;
>>>>>> +        default:
>>>> That can't happen.
>>>>
>>>>>> + mutex_destroy(&cap_dev->lock);
>>>>>> +            ret = -EINVAL;
>>>>>> +            goto err_destroy_mutex;
>>>>>> +        }
>>>>>> +
>>>>>> +        cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
>>>>>> +        ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
>>>>>> +        if (ret) {
>>>>>> +            mutex_destroy(&cap_dev->lock);
>>>>>> +            goto err_destroy_mutex;
>>>>>> +        }
>>>>>> +
>>>>>> +        vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>>>>>> +        vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
>>>>>> +        vb2q->drv_priv = cap_dev;
>>>>>> +        vb2q->mem_ops = &vb2_dma_contig_memops;
>>>>>> +        vb2q->ops = &mali_c55_vb2_ops;
>>>>>> +        vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
>>>>>> +        vb2q->min_queued_buffers = 1;
>>>>>> +        vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
>>>>>> +        vb2q->lock = &cap_dev->lock;
>>>>>> +        vb2q->dev = mali_c55->dev;
>>>>>> +
>>>>>> +        ret = vb2_queue_init(vb2q);
>>>>>> +        if (ret) {
>>>>>> +            dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
>>>>>> +                mali_c55_cap_dev_to_name(cap_dev));
>>>>>> +            goto err_cleanup_media_entity;
>>>>>> +        }
>>>>>> +
>>>>>> +        strscpy(cap_dev->vdev.name, capture_device_names[i],
>>>>>> +            sizeof(cap_dev->vdev.name));
>>>>>> +        vdev->release = video_device_release_empty;
>>>>>> +        vdev->fops = &mali_c55_v4l2_fops;
>>>>>> +        vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
>>>>>> +        vdev->lock = &cap_dev->lock;
>>>>>> +        vdev->v4l2_dev = &mali_c55->v4l2_dev;
>>>>>> +        vdev->queue = &cap_dev->queue;
>>>>>> +        vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>>>>>> +                    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
>>>>>> +        vdev->entity.ops = &mali_c55_media_ops;
>>>>>> +        video_set_drvdata(vdev, cap_dev);
>>>>>> +
>>>>>> +        memset(&pix_mp, 0, sizeof(pix_mp));
>>>>>> +        pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
>>>>>> +        pix_mp.width = MALI_C55_DEFAULT_WIDTH;
>>>>>> +        pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
>>>>>> +        mali_c55_set_format(cap_dev, &pix_mp);
>>>>>> +
>>>>>> +        ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
>>>>>> +        if (ret) {
>>>>>> +            dev_err(mali_c55->dev,
>>>>>> +                "%s failed to register video device\n",
>>>>>> +                mali_c55_cap_dev_to_name(cap_dev));
>>>>>> +            goto err_release_vb2q;
>>>>>> +        }
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_release_vb2q:
>>>>>> +    vb2_queue_release(vb2q);
>>>>>> +err_cleanup_media_entity:
>>>>>> +    media_entity_cleanup(&cap_dev->vdev.entity);
>>>>>> +err_destroy_mutex:
>>>>>> +    mutex_destroy(&cap_dev->lock);
>>>>>> +    mali_c55_unregister_capture_devs(mali_c55);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev;
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
>>>>>> +        cap_dev = &mali_c55->cap_devs[i];
>>>>>> +
>>>>>> +        if (!video_is_registered(&cap_dev->vdev))
>>>>>> +            continue;
>>>>>> +
>>>>>> +        vb2_video_unregister_device(&cap_dev->vdev);
>>>>>> +        media_entity_cleanup(&cap_dev->vdev.entity);
>>>>>> +        mutex_destroy(&cap_dev->lock);
>>>>>> +    }
>>>>>> +}
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>>>> new file mode 100644
>>>>>> index 000000000000..2d0c4d152beb
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>>>> @@ -0,0 +1,266 @@
>>>>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Common definitions
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#ifndef _MALI_C55_COMMON_H
>>>>>> +#define _MALI_C55_COMMON_H
>>>>>> +
>>>>>> +#include <linux/clk.h>
>>>>>> +#include <linux/io.h>
>>>>>> +#include <linux/list.h>
>>>>>> +#include <linux/mutex.h>
>>>>>> +#include <linux/scatterlist.h>
>>>>> I don't think this is needed. You're however missing spinlock.h.
>>>>>
>>>>>> +#include <linux/videodev2.h>
>>>>>> +
>>>>>> +#include <media/media-device.h>
>>>>>> +#include <media/v4l2-async.h>
>>>>>> +#include <media/v4l2-ctrls.h>
>>>>>> +#include <media/v4l2-dev.h>
>>>>>> +#include <media/v4l2-device.h>
>>>>>> +#include <media/v4l2-subdev.h>
>>>>>> +#include <media/videobuf2-core.h>
>>>>>> +#include <media/videobuf2-v4l2.h>
>>>>>> +
>>>>>> +#define MALI_C55_DRIVER_NAME        "mali-c55"
>>>>>> +
>>>>>> +/* min and max values for the image sizes */
>>>>>> +#define MALI_C55_MIN_WIDTH        640U
>>>>>> +#define MALI_C55_MIN_HEIGHT        480U
>>>>>> +#define MALI_C55_MAX_WIDTH        8192U
>>>>>> +#define MALI_C55_MAX_HEIGHT        8192U
>>>>>> +#define MALI_C55_DEFAULT_WIDTH        1920U
>>>>>> +#define MALI_C55_DEFAULT_HEIGHT        1080U
>>>>>> +
>>>>>> +#define MALI_C55_DEFAULT_MEDIA_BUS_FMT MEDIA_BUS_FMT_RGB121212_1X36
>>>>>> +
>>>>>> +struct mali_c55;
>>>>>> +struct mali_c55_cap_dev;
>>>>>> +struct platform_device;
>>>>> You should also forward-declare
>>>>>
>>>>> struct device;
>>>>> struct dma_chan;
>>>>> struct resource;
>>>>>
>>>>>> +
>>>>>> +static const char * const mali_c55_clk_names[] = {
>>>>>> +    "aclk",
>>>>>> +    "hclk",
>>>>>> +};
>>>>> This will end up duplicating the array in each compilation unit, not
>>>>> great. Move it to mali-c55-core.c. You use it in this file just for its
>>>>> size, replace that with a macro that defines the size, or allocate
>>>>> mali_c55.clks dynamically with devm_kcalloc().
>>>>>
>>>>>> +
>>>>>> +enum mali_c55_interrupts {
>>>>>> +    MALI_C55_IRQ_ISP_START,
>>>>>> +    MALI_C55_IRQ_ISP_DONE,
>>>>>> +    MALI_C55_IRQ_MCM_ERROR,
>>>>>> +    MALI_C55_IRQ_BROKEN_FRAME_ERROR,
>>>>>> +    MALI_C55_IRQ_MET_AF_DONE,
>>>>>> +    MALI_C55_IRQ_MET_AEXP_DONE,
>>>>>> +    MALI_C55_IRQ_MET_AWB_DONE,
>>>>>> +    MALI_C55_IRQ_AEXP_1024_DONE,
>>>>>> +    MALI_C55_IRQ_IRIDIX_MET_DONE,
>>>>>> +    MALI_C55_IRQ_LUT_INIT_DONE,
>>>>>> +    MALI_C55_IRQ_FR_Y_DONE,
>>>>>> +    MALI_C55_IRQ_FR_UV_DONE,
>>>>>> +    MALI_C55_IRQ_DS_Y_DONE,
>>>>>> +    MALI_C55_IRQ_DS_UV_DONE,
>>>>>> +    MALI_C55_IRQ_LINEARIZATION_DONE,
>>>>>> +    MALI_C55_IRQ_RAW_FRONTEND_DONE,
>>>>>> +    MALI_C55_IRQ_NOISE_REDUCTION_DONE,
>>>>>> +    MALI_C55_IRQ_IRIDIX_DONE,
>>>>>> +    MALI_C55_IRQ_BAYER2RGB_DONE,
>>>>>> +    MALI_C55_IRQ_WATCHDOG_TIMER,
>>>>>> +    MALI_C55_IRQ_FRAME_COLLISION,
>>>>>> +    MALI_C55_IRQ_UNUSED,
>>>>>> +    MALI_C55_IRQ_DMA_ERROR,
>>>>>> +    MALI_C55_IRQ_INPUT_STOPPED,
>>>>>> +    MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
>>>>>> +    MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
>>>>>> +    MALI_C55_NUM_IRQ_BITS
>>>>> Those are register bits, I think they belong to mali-c55-registers.h,
>>>>> and should probably be macros instead of an enum.
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +enum mali_c55_isp_pads {
>>>>>> +    MALI_C55_ISP_PAD_SINK_VIDEO,
>>>>> As there's a single sink pad, maybe MALI_C55_ISP_PAD_SINK ? Ah, you're
>>>>> probably preparing for ISP parameters support. It's fine.
>>>>>
>>>>>> +    MALI_C55_ISP_PAD_SOURCE,
>>>>> Then maybe this should be named MALI_C55_ISP_PAD_SOURCE_VIDEO as I
>>>>> assume there will be a stats source pad.
>>>>>
>>>>>> +    MALI_C55_ISP_PAD_SOURCE_BYPASS,
>>>>>> +    MALI_C55_ISP_NUM_PADS,
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_tpg {
>>>>>> +    struct mali_c55 *mali_c55;
>>>>>> +    struct v4l2_subdev sd;
>>>>>> +    struct media_pad pad;
>>>>>> +    struct mutex lock;
>>>>>> +    struct mali_c55_tpg_ctrls {
>>>>>> +        struct v4l2_ctrl_handler handler;
>>>>>> +        struct v4l2_ctrl *test_pattern;
>>>>> Set but never used. You can drop it.
>>>>>
>>>>>> +        struct v4l2_ctrl *hblank;
>>>>> Set and used only once, in the same function. You can make it a local
>>>>> variable.
>>>>>
>>>>>> +        struct v4l2_ctrl *vblank;
>>>>>> +    } ctrls;
>>>>>> +};
>>>>> I wonder if this file should be split, with mali-c55-capture.h,
>>>>> mali-c55-core.h, mali-c55-isp.h, ... I think it could increase
>>>>> readability by clearly separating the different elements. Up to you.
>>>>>
>>>>>> +
>>>>>> +struct mali_c55_isp {
>>>>>> +    struct mali_c55 *mali_c55;
>>>>>> +    struct v4l2_subdev sd;
>>>>>> +    struct media_pad pads[MALI_C55_ISP_NUM_PADS];
>>>>>> +    struct media_pad *remote_src;
>>>>>> +    struct v4l2_async_notifier notifier;
>>>>> I'm tempted to move the notifier to mali_c55, as it's related to
>>>>> components external to the whole ISP, not to the ISP subdev itself.
>>>>> Could you give it a try, to see if it could be done without any drawback
>>>>> ?
>>>>>
>>>>>> +    struct mutex lock;
>>>>> Locks require a comment to explain what they protect. Same below where
>>>>> applicable (for both mutexes and spinlocks).
>>>>>
>>>>>> +    unsigned int frame_sequence;
>>>>>> +};
>>>>>> +
>>>>>> +enum mali_c55_resizer_ids {
>>>>>> +    MALI_C55_RZR_FR,
>>>>>> +    MALI_C55_RZR_DS,
>>>>>> +    MALI_C55_NUM_RZRS,
>>>>> The usual abbreviation for "resizer" in drivers/media/ is "rsz", not
>>>>> "rzr". I would have said we can leave it as-is as changing it would be a
>>>>> bit annoying, but I then realized that "rzr" is not just unusual, it's
>>>>> actually not used at all. Would you mind applying a sed globally ? :-)
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +enum mali_c55_rzr_pads {
>>>>> Same enums/structs use abbreviations, some don't. Consistency would
>>>>> help.
>>>>>
>>>>>> +    MALI_C55_RZR_SINK_PAD,
>>>>>> +    MALI_C55_RZR_SOURCE_PAD,
>>>>>> +    MALI_C55_RZR_SINK_BYPASS_PAD,
>>>>>> +    MALI_C55_RZR_NUM_PADS
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_resizer {
>>>>>> +    struct mali_c55 *mali_c55;
>>>>>> +    struct mali_c55_cap_dev *cap_dev;
>>>>>> +    enum mali_c55_resizer_ids id;
>>>>>> +    struct v4l2_subdev sd;
>>>>>> +    struct media_pad pads[MALI_C55_RZR_NUM_PADS];
>>>>>> +    unsigned int num_routes;
>>>>>> +    bool streaming;
>>>>>> +};
>>>>>> +
>>>>>> +enum mali_c55_cap_devs {
>>>>>> +    MALI_C55_CAP_DEV_FR,
>>>>>> +    MALI_C55_CAP_DEV_DS,
>>>>>> +    MALI_C55_NUM_CAP_DEVS
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_fmt {
>>>>> mali_c55_format_info would be a better name I think, as this stores
>>>>> format information, not formats.
>>>>>
>>>>>> +    u32 fourcc;
>>>>>> +    unsigned int mbus_codes[2];
>>>>> A comment to explain why we have two media bus codes would be useful.
>>>>> You can document the whole structure if desired :-)
>>>>>
>>>>>> +    bool is_raw;
>>>>>> +    struct mali_c55_fmt_registers {
>>>>> Make it an anonymous structure, it's never used anywhere else.
>>>>>
>>>>>> +        unsigned int base_mode;
>>>>>> +        unsigned int uv_plane;
>>>>> If those are register field values, use u32 instead of unsigned int.
>>>>>
>>>>>> +    } registers;
>>>>> It's funny, we tend to abbreviate different things, I would have used
>>>>> "regs" here but written "format" in full in the structure name :-)
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +enum mali_c55_isp_bayer_order {
>>>>>> +    MALI_C55_BAYER_ORDER_RGGB,
>>>>>> +    MALI_C55_BAYER_ORDER_GRBG,
>>>>>> +    MALI_C55_BAYER_ORDER_GBRG,
>>>>>> +    MALI_C55_BAYER_ORDER_BGGR
>>>>> These are registers values too, they belong to mali-c55-registers.h.
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_isp_fmt {
>>>>> mali_c55_isp_format_info
>>>>>
>>>>>> +    u32 code;
>>>>>> +    enum v4l2_pixel_encoding encoding;
>>>>> Here you use v4l2_pixel_encoding, for mali_c55_fmt you use is_raw. Maybe
>>>>> pick the same option for both structures ?
>>>>>
>>>>>> +    enum mali_c55_isp_bayer_order order;
>>>>>> +};
>>>>>> +
>>>>>> +enum mali_c55_planes {
>>>>>> +    MALI_C55_PLANE_Y,
>>>>>> +    MALI_C55_PLANE_UV,
>>>>>> +    MALI_C55_NUM_PLANES
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_buffer {
>>>>>> +    struct vb2_v4l2_buffer vb;
>>>>>> +    bool plane_done[MALI_C55_NUM_PLANES];
>>>>> I think tracking the pending state would simplify the logic in
>>>>> mali_c55_set_plane_done(), which would become
>>>>>
>>>>>      curr_buf->plane_pending[plane] = false;
>>>>>
>>>>>      if (!curr_buf->plane_pending[0] && !curr_buf->plane_pending[1]) {
>>>>>          mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
>>>>>          cap_dev->buffers.curr = NULL;
>>>>>          isp->frame_sequence++;
>>>>>      }
>>>>>
>>>>> Or a counter may be even easier (and would consume less memory).
>>>>>
>>>>>> +    struct list_head queue;
>>>>>> +    u32 addrs[MALI_C55_NUM_PLANES];
>>>>> This stores DMA addresses, use dma_addr_t.
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_cap_dev {
>>>>>> +    struct mali_c55 *mali_c55;
>>>>>> +    struct mali_c55_resizer *rzr;
>>>>>> +    struct video_device vdev;
>>>>>> +    struct media_pad pad;
>>>>>> +    struct vb2_queue queue;
>>>>>> +    struct mutex lock;
>>>>>> +    unsigned int reg_offset;
>>>>> Manual handling of the offset everywhere, with parametric macros for the
>>>>> resizer register addresses, isn't very nice. Introduce resizer-specific
>>>>> accessors and capture-specific accessors (mali_c55_rsz_write(), ...)
>>>>> that take a mali_c55_resizer or mali_c55_cap_dev pointer, and handle the
>>>>> offset there. The register macros should loose their offset parameter.
>>>>>
>>>>> You could also use a single set of accessors that would become
>>>>> path/output-specific (mali_c55_path_write() ? mali_c55_output_write()
>>>>> ?), that may make the code easier to read.
>>>>>
>>>>> You can also replace reg_offset with a void __iomem * base, which would
>>>>> avoid the computation at runtime.
>>>>>
>>>>>> +
>>>>>> +    struct mali_c55_mode {
>>>>> Make the structure anonymous.
>>>>>
>>>>>> +        const struct mali_c55_fmt *capture_fmt;
>>>>>> +        struct v4l2_pix_format_mplane pix_mp;
>>>>>> +    } mode;
>>>>> What's a "mode" ? I think I'd name this
>>>>>
>>>>>      struct {
>>>>>          const struct mali_c55_fmt *info;
>>>>>          struct v4l2_pix_format_mplane format;
>>>>>      } format;
>>>>>
>>>>> Or you could just drop the structure and have
>>>>>
>>>>>      const struct mali_c55_fmt *format_info;
>>>>>      struct v4l2_pix_format_mplane format;
>>>>>
>>>>> or something similar.
>>>>>
>>>>>> +
>>>>>> +    struct {
>>>>>> +        spinlock_t lock;
>>>>>> +        struct list_head queue;
>>>>>> +        struct mali_c55_buffer *curr;
>>>>>> +        struct mali_c55_buffer *next;
>>>>>> +    } buffers;
>>>>>> +
>>>>>> +    bool streaming;
>>>>>> +};
>>>>>> +
>>>>>> +enum mali_c55_config_spaces {
>>>>>> +    MALI_C55_CONFIG_PING,
>>>>>> +    MALI_C55_CONFIG_PONG,
>>>>>> +    MALI_C55_NUM_CONFIG_SPACES
>>>>> The last enumerator is not used.
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_ctx {
>>>>> mali_c55_context ?
>>>>>
>>>>>> +    struct mali_c55 *mali_c55;
>>>>>> +    void *registers;
>>>>> Please document this structure and explain that this field points to a
>>>>> copy of the register space in system memory, I was about to write you're
>>>>> missing __iomem :-)
>>>>>
>>>>>> +    phys_addr_t base;
>>>>>> +    spinlock_t lock;
>>>>>> +    struct list_head list;
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55 {
>>>>>> +    struct device *dev;
>>>>>> +    struct resource *res;
>>>>> You could possibly drop this field by passing the physical address of
>>>>> the register space from mali_c55_probe() to mali_c55_init_context() as a
>>>>> function parameter.
>>>>>
>>>>>> +    void __iomem *base;
>>>>>> +    struct dma_chan *channel;
>>>>>> +    struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
>>>>>> +
>>>>>> +    u16 capabilities;
>>>>>> +    struct media_device media_dev;
>>>>>> +    struct v4l2_device v4l2_dev;
>>>>>> +    struct media_pipeline pipe;
>>>>>> +
>>>>>> +    struct mali_c55_tpg tpg;
>>>>>> +    struct mali_c55_isp isp;
>>>>>> +    struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
>>>>>> +    struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
>>>>>> +
>>>>>> +    struct list_head contexts;
>>>>>> +    enum mali_c55_config_spaces next_config;
>>>>>> +};
>>>>>> +
>>>>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
>>>>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
>>>>>> +          bool force_hardware);
>>>>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
>>>>>> +              u32 mask, u32 val);
>>>>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
>>>>>> +              enum mali_c55_config_spaces cfg_space);
>>>>>> +
>>>>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55);
>>>>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55);
>>>>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
>>>>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
>>>>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55);
>>>>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
>>>>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
>>>>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
>>>>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
>>>>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>>>>>> +                 enum mali_c55_planes plane);
>>>>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
>>>>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
>>>>>> +
>>>>>> +bool mali_c55_format_is_raw(unsigned int mbus_code);
>>>>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
>>>>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
>>>>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
>>>>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
>>>>>> +
>>>>>> +const struct mali_c55_isp_fmt *
>>>>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
>>>>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
>>>>>> +#define for_each_mali_isp_fmt(fmt)\
>>>>> #define for_each_mali_isp_fmt(fmt) \
>>>>>
>>>>>> +    for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
>>>>> Looks like parentheses were on sale :-)
>>>>>
>>>>>      for ((fmt) = NULL; (fmt) = mali_c55_isp_fmt_next(fmt); )
>>>>>
>>>>> This macro is used in two places only, in the mali-c55-isp.c file where
>>>>> open-coding the loop without using mali_c55_isp_fmt_next() would be more
>>>>> efficient, and in mali-c55-resizer.c where a function to return format
>>>>> i'th would be more efficient. I think you can drop the macro and the
>>>>> mali_c55_isp_fmt_next() function.
>>>>>
>>>>>> +
>>>>>> +#endif /* _MALI_C55_COMMON_H */
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>>>> new file mode 100644
>>>>>> index 000000000000..50caf5ee7474
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>>>> @@ -0,0 +1,767 @@
>>>>>> +// SPDX-License-Identifier: GPL-2.0
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Core driver code
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#include <linux/bitops.h>
>>>>>> +#include <linux/cleanup.h>
>>>>>> +#include <linux/clk.h>
>>>>>> +#include <linux/delay.h>
>>>>>> +#include <linux/device.h>
>>>>>> +#include <linux/dmaengine.h>
>>>>>> +#include <linux/dma-mapping.h>
>>>>>> +#include <linux/interrupt.h>
>>>>>> +#include <linux/iopoll.h>
>>>>>> +#include <linux/ioport.h>
>>>>>> +#include <linux/mod_devicetable.h>
>>>>>> +#include <linux/of.h>
>>>>>> +#include <linux/of_reserved_mem.h>
>>>>>> +#include <linux/platform_device.h>
>>>>>> +#include <linux/pm_runtime.h>
>>>>>> +#include <linux/scatterlist.h>
>>>>> I don't think this is needed.
>>>>>
>>>>> Missing slab.h.
>>>>>
>>>>>> +#include <linux/string.h>
>>>>>> +
>>>>>> +#include <media/media-entity.h>
>>>>>> +#include <media/v4l2-device.h>
>>>>>> +#include <media/videobuf2-dma-contig.h>
>>>>>> +
>>>>>> +#include "mali-c55-common.h"
>>>>>> +#include "mali-c55-registers.h"
>>>>>> +
>>>>>> +static const char * const mali_c55_interrupt_names[] = {
>>>>>> +    [MALI_C55_IRQ_ISP_START] = "ISP start",
>>>>>> +    [MALI_C55_IRQ_ISP_DONE] = "ISP done",
>>>>>> +    [MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
>>>>>> +    [MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
>>>>>> +    [MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
>>>>>> +    [MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
>>>>>> +    [MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
>>>>>> +    [MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
>>>>>> +    [MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
>>>>>> +    [MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
>>>>>> +    [MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
>>>>>> +    [MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
>>>>>> +    [MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
>>>>>> +    [MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
>>>>>> +    [MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
>>>>>> +    [MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
>>>>>> +    [MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
>>>>>> +    [MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
>>>>>> +    [MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
>>>>>> +    [MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
>>>>>> +    [MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
>>>>>> +    [MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
>>>>>> +    [MALI_C55_IRQ_DMA_ERROR] = "DMA error",
>>>>>> +    [MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
>>>>>> +    [MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
>>>>>> +    [MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
>>>>>> +};
>>>>>> +
>>>>>> +static unsigned int config_space_addrs[] = {
>>>>> const
>>>>>
>>>>>> +    [MALI_C55_CONFIG_PING] = 0x0AB6C,
>>>>>> +    [MALI_C55_CONFIG_PONG] = 0x22B2C,
>>>>> Lowercase hex constants.
>>>>>
>>>>> Don't the values belong to mali-c55-registers.h ?
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +/* System IO
>>>>> /*
>>>>>    * System IO
>>>>>
>>>>>> + *
>>>>>> + * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
>>>>>> + * and 'pong'), with the  expectation that the 'active' space will be left
>>>>> s/the  /the /
>>>>>
>>>>>> + * untouched whilst a frame is being processed and the 'inactive' space
>>>>>> + * configured ready to be passed during the blanking period before the next
>>>>> s/to be passed/to be switched to/ ?
>>>>>
>>>>>> + * frame processing starts. These spaces should ideally be set via DMA transfer
>>>>>> + * from a buffer rather than through individual register set operations. There
>>>>>> + * is also a shared global register space which should be set normally. Of
>>>>>> + * course, the ISP might be included in a system which lacks a suitable DMA
>>>>>> + * engine, and the second configuration space might not be fitted at all, which
>>>>>> + * means we need to support four scenarios:
>>>>>> + *
>>>>>> + * 1. Multi config space, with DMA engine.
>>>>>> + * 2. Multi config space, no DMA engine.
>>>>>> + * 3. Single config space, with DMA engine.
>>>>>> + * 4. Single config space, no DMA engine.
>>>>>> + *
>>>>>> + * The first case is very easy, but the rest present annoying problems. The best
>>>>>> + * way to solve them seems to be simply to replicate the concept of DMAing over
>>>>>> + * the configuration buffer even if there's no DMA engine on the board, for
>>>>>> + * which we rely on memcpy. To facilitate this any read/write call that is made
>>>>>> + * to an address within those config spaces should infact be directed to a
>>>>>> + * buffer that was allocated to hold them rather than the IO memory itself. The
>>>>>> + * actual copy of that buffer to IO mem will happen on interrupt.
>>>>>> + */
>>>>>> +
>>>>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
>>>>>> +{
>>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>>>> +
>>>>>> +    if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
>>>>>> +        spin_lock(&ctx->lock);
>>>>>> +        addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
>>>>>> +        ((u32 *)ctx->registers)[addr] = val;
>>>>>> +        spin_unlock(&ctx->lock);
>>>>>> +
>>>>>> +        return;
>>>>>> +    }
>>>>> Ouch. This is likely the second comment you really won't like (after the
>>>>> comment regarding mali_c55_update_bits() at the very top). I apologize
>>>>> in advance.
>>>>>
>>>>> I really don't like this. Directing writes either to hardware registers
>>>>> or to the shadow registers in the context makes the callers of the
>>>>> read/write accessors very hard to read. The probe code, for instance,
>>>>> mixes writes to hardware registers and writes to the context shadow
>>>>> registers to initialize the value of some of the shadow registers.
>>>>>
>>>>> I'd like to split the read/write accessors into functions that access
>>>>> the hardware registers (that's easy) and functions that access the
>>>>> shadow registers. I think the latter should receive a mali_c55_ctx
>>>>> pointer instead of a mali_c55 pointer to prepare for multi-context
>>>>> support.
>>>>>
>>>>> You can add WARN_ON() guards to the two sets of functions, to ensure
>>>>> that no register from the "other" space gets passed to the wrong
>>>>> function by mistake.
>>>>>
>>>>>> +
>>>>>> +    writel(val, mali_c55->base + addr);
>>>>>> +}
>>>>>> +
>>>>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
>>>>>> +          bool force_hardware)
>>>>> force_hardware is never set to true.
>>>>>
>>>>>> +{
>>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>>>> +    u32 val;
>>>>>> +
>>>>>> +    if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
>>>>>> +        spin_lock(&ctx->lock);
>>>>>> +        addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
>>>>>> +        val = ((u32 *)ctx->registers)[addr];
>>>>>> +        spin_unlock(&ctx->lock);
>>>>>> +
>>>>>> +        return val;
>>>>>> +    }
>>>>>> +
>>>>>> +    return readl(mali_c55->base + addr);
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
>>>>>> +              u32 mask, u32 val)
>>>>>> +{
>>>>>> +    u32 orig, tmp;
>>>>>> +
>>>>>> +    orig = mali_c55_read(mali_c55, addr, false);
>>>>>> +
>>>>>> +    tmp = orig & ~mask;
>>>>>> +    tmp |= (val << (ffs(mask) - 1)) & mask;
>>>>>> +
>>>>>> +    if (tmp != orig)
>>>>>> +        mali_c55_write(mali_c55, addr, tmp);
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
>>>>>> +                 dma_addr_t dst, enum dma_data_direction dir)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>>>> +    struct dma_async_tx_descriptor *tx;
>>>>>> +    enum dma_status status;
>>>>>> +    dma_cookie_t cookie;
>>>>>> +
>>>>>> +    tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
>>>>>> +                       MALI_C55_CONFIG_SPACE_SIZE, 0);
>>>>>> +    if (!tx) {
>>>>>> +        dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
>>>>>> +        return -EIO;
>>>>>> +    }
>>>>>> +
>>>>>> +    cookie = dmaengine_submit(tx);
>>>>>> +    if (dma_submit_error(cookie)) {
>>>>>> +        dev_err(mali_c55->dev, "error submitting dma transfer\n");
>>>>>> +        return -EIO;
>>>>>> +    }
>>>>>> +
>>>>>> +    status = dma_sync_wait(mali_c55->channel, cookie);
>>>>> I've just realized this performs a busy-wait :-S See the comment in the
>>>>> probe function about the threaded IRQ handler. I think we'll need to
>>>>> rework all this. It could be done on top though.
>>>>>
>>>>>> +    if (status != DMA_COMPLETE) {
>>>>>> +        dev_err(mali_c55->dev, "dma transfer failed\n");
>>>>>> +        return -EIO;
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
>>>>>> +                 enum mali_c55_config_spaces cfg_space)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>>>> +    struct device *dma_dev = mali_c55->channel->device->dev;
>>>>>> +    dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
>>>>>> +    dma_addr_t dst;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    guard(spinlock)(&ctx->lock);
>>>>>> +
>>>>>> +    dst = dma_map_single(dma_dev, ctx->registers,
>>>>>> +                 MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
>>>>>> +    if (dma_mapping_error(dma_dev, dst)) {
>>>>>> +        dev_err(mali_c55->dev, "failed to map DMA addr\n");
>>>>>> +        return -EIO;
>>>>>> +    }
>>>>>> +
>>>>>> +    ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
>>>>>> +    dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
>>>>>> +             DMA_FROM_DEVICE);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
>>>>>> +               enum mali_c55_config_spaces cfg_space)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>>>> +    struct device *dma_dev = mali_c55->channel->device->dev;
>>>>>> +    dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
>>>>>> +    dma_addr_t src;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    guard(spinlock)(&ctx->lock);
>>>>> The code below can take a large amount of time, holding a spinlock will
>>>>> disable interrupts on the local CPU, that's not good :-(
>>>>>
>>>>>> +
>>>>>> +    src = dma_map_single(dma_dev, ctx->registers,
>>>>>> +                 MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
>>>>>> +    if (dma_mapping_error(dma_dev, src)) {
>>>>>> +        dev_err(mali_c55->dev, "failed to map DMA addr\n");
>>>>>> +        return -EIO;
>>>>>> +    }
>>>>>> +
>>>>>> +    ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
>>>>>> +    dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
>>>>>> +             DMA_TO_DEVICE);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_config_read(struct mali_c55_ctx *ctx,
>>>>>> +                enum mali_c55_config_spaces cfg_space)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>>>> +
>>>>>> +    if (mali_c55->channel) {
>>>>>> +        return mali_c55_dma_read(ctx, cfg_space);
>>>>> As this function is used at probe time only, to initialize the context,
>>>>> I think DMA is overkill.
>>>>>
>>>>>> +    } else {
>>>>>> +        memcpy_fromio(ctx->registers,
>>>>>> +                  mali_c55->base + config_space_addrs[cfg_space],
>>>>>> +                  MALI_C55_CONFIG_SPACE_SIZE);
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +}
>>>>>> +
>>>>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
>>>>>> +              enum mali_c55_config_spaces cfg_space)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>>>> +
>>>>>> +    if (mali_c55->channel) {
>>>>>> +        return mali_c55_dma_write(ctx, cfg_space);
>>>>>> +    } else {
>>>>>> +        memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
>>>>>> +                ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
>>>>>> +        return 0;
>>>>>> +    }
>>>>> Could you measure the time it typically takes to write the registers
>>>>> using DMA compared to using memcpy_toio() ?
>>>>>
>>>>>> +}
>>>>>> +
>>>>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);
>>>>> I think it's too early to tell how multi-context support will look like.
>>>>> I'm fine keeping mali_c55_get_active_context() as changing that would be
>>>>> very intrusive (even if I think it will need to be changed), but the
>>>>> list of contexts is neither the mechanism we'll use, nor something we
>>>>> need now. Drop the list, embed the context in struct mali_c55, and
>>>>> return the pointer to that single context from this function.
>>>>>
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_remove_links(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> + media_entity_remove_links(&mali_c55->tpg.sd.entity);
>>>>>> + media_entity_remove_links(&mali_c55->isp.sd.entity);
>>>>>> +
>>>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; i++)
>>>>>> + media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
>>>>>> +
>>>>>> +    for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
>>>>>> + media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_create_links(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    /* Test pattern generator to ISP */
>>>>>> +    ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
>>>>>> +                    &mali_c55->isp.sd.entity,
>>>>>> +                    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
>>>>>> +        goto err_remove_links;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* Full resolution resizer pipe. */
>>>>>> +    ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>>>>> +            MALI_C55_ISP_PAD_SOURCE,
>>>>>> + &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
>>>>>> +            MALI_C55_RZR_SINK_PAD,
>>>>>> +            MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
>>>>>> +        goto err_remove_links;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* Full resolution bypass. */
>>>>>> +    ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>>>>> +                    MALI_C55_ISP_PAD_SOURCE_BYPASS,
>>>>>> + &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
>>>>>> +                    MALI_C55_RZR_SINK_BYPASS_PAD,
>>>>>> +                    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
>>>>>> +        goto err_remove_links;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* Resizer pipe to video capture nodes. */
>>>>>> +    ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
>>>>>> +            MALI_C55_RZR_SOURCE_PAD,
>>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
>>>>>> +            0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev,
>>>>>> +            "failed to link FR resizer and video device\n");
>>>>>> +        goto err_remove_links;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* The downscale pipe is an optional hardware block */
>>>>>> +    if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
>>>>>> +        ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>>>>> +            MALI_C55_ISP_PAD_SOURCE,
>>>>>> + &mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
>>>>>> +            MALI_C55_RZR_SINK_PAD,
>>>>>> +            MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>>>> +        if (ret) {
>>>>>> +            dev_err(mali_c55->dev,
>>>>>> +                "failed to link ISP and DS resizer\n");
>>>>>> +            goto err_remove_links;
>>>>>> +        }
>>>>>> +
>>>>>> +        ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
>>>>>> +            MALI_C55_RZR_SOURCE_PAD,
>>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
>>>>>> +            0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>>>> +        if (ret) {
>>>>>> +            dev_err(mali_c55->dev,
>>>>>> +                "failed to link DS resizer and video device\n");
>>>>>> +            goto err_remove_links;
>>>>>> +        }
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_remove_links:
>>>>>> +    mali_c55_remove_links(mali_c55);
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    mali_c55_unregister_tpg(mali_c55);
>>>>>> +    mali_c55_unregister_isp(mali_c55);
>>>>>> +    mali_c55_unregister_resizers(mali_c55);
>>>>>> +    mali_c55_unregister_capture_devs(mali_c55);
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    ret = mali_c55_register_tpg(mali_c55);
>>>>>> +    if (ret)
>>>>>> +        return ret;
>>>>>> +
>>>>>> +    ret = mali_c55_register_isp(mali_c55);
>>>>>> +    if (ret)
>>>>>> +        goto err_unregister_entities;
>>>>>> +
>>>>>> +    ret = mali_c55_register_resizers(mali_c55);
>>>>>> +    if (ret)
>>>>>> +        goto err_unregister_entities;
>>>>>> +
>>>>>> +    ret = mali_c55_register_capture_devs(mali_c55);
>>>>>> +    if (ret)
>>>>>> +        goto err_unregister_entities;
>>>>>> +
>>>>>> +    ret = mali_c55_create_links(mali_c55);
>>>>>> +    if (ret)
>>>>>> +        goto err_unregister_entities;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_unregister_entities:
>>>>>> +    mali_c55_unregister_entities(mali_c55);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    u32 product, version, revision, capabilities;
>>>>>> +
>>>>>> +    product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
>>>>>> +    version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
>>>>>> +    revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
>>>>>> +
>>>>>> +    dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
>>>>>> +         product, version, revision);
>>>>>> +
>>>>>> +    capabilities = mali_c55_read(mali_c55,
>>>>>> +                     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
>>>>>> +                     false);
>>>>>> +    mali_c55->capabilities = (capabilities & 0xffff);
>>>>>> +
>>>>>> +    /* TODO: Might as well start some debugfs */
>>>>> If it's just to expose the version and capabilities, I think that's
>>>>> overkill. It's not needed for debug purpose (you can get it from the
>>>>> kernel log already). debugfs isn't meant to be accessible in production,
>>>>> so an application that would need access to the information wouldn't be
>>>>> able to use it.
>>>>>
>>>>>> +    dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);
>>>>> Combine the two messages into one.
>>>>>
>>>>>> +    return version;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>>>> +    u32 curr_config, next_config;
>>>>>> +
>>>>>> +    curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
>>>>>> +    curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
>>>>>> +              >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
>>>>>> +    next_config = curr_config ^ 1;
>>>>>> +
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>>>> +                 MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
>>>>>> +    mali_c55_config_write(ctx, next_config ?
>>>>>> +                  MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
>>>>>> +}
>>>>>> +
>>>>>> +static irqreturn_t mali_c55_isr(int irq, void *context)
>>>>>> +{
>>>>>> +    struct device *dev = context;
>>>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>>>>> +    u32 interrupt_status;
>>>>>> +    unsigned int i, j;
>>>>>> +
>>>>>> +    interrupt_status = mali_c55_read(mali_c55,
>>>>>> +                     MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
>>>>>> +                     false);
>>>>>> +    if (!interrupt_status)
>>>>>> +        return IRQ_NONE;
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
>>>>>> +               interrupt_status);
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
>>>>>> +
>>>>>> +    for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
>>>>>> +        if (!(interrupt_status & (1 << i)))
>>>>>> +            continue;
>>>>>> +
>>>>>> +        switch (i) {
>>>>>> +        case MALI_C55_IRQ_ISP_START:
>>>>>> +            mali_c55_isp_queue_event_sof(mali_c55);
>>>>>> +
>>>>>> +            for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
>>>>>> + mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
>>>>>> +
>>>>>> +            mali_c55_swap_next_config(mali_c55);
>>>>>> +
>>>>>> +            break;
>>>>>> +        case MALI_C55_IRQ_ISP_DONE:
>>>>>> +            /*
>>>>>> +             * TODO: Where the ISP has no Pong config fitted, we'd
>>>>>> +             * have to do the mali_c55_swap_next_config() call here.
>>>>>> +             */
>>>>>> +            break;
>>>>>> +        case MALI_C55_IRQ_FR_Y_DONE:
>>>>>> +            mali_c55_set_plane_done(
>>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
>>>>>> +                MALI_C55_PLANE_Y);
>>>>>> +            break;
>>>>>> +        case MALI_C55_IRQ_FR_UV_DONE:
>>>>>> +            mali_c55_set_plane_done(
>>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
>>>>>> +                MALI_C55_PLANE_UV);
>>>>>> +            break;
>>>>>> +        case MALI_C55_IRQ_DS_Y_DONE:
>>>>>> +            mali_c55_set_plane_done(
>>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
>>>>>> +                MALI_C55_PLANE_Y);
>>>>>> +            break;
>>>>>> +        case MALI_C55_IRQ_DS_UV_DONE:
>>>>>> +            mali_c55_set_plane_done(
>>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
>>>>>> +                MALI_C55_PLANE_UV);
>>>>>> +            break;
>>>>>> +        default:
>>>>>> +            /*
>>>>>> +             * Only the above interrupts are currently unmasked. If
>>>>>> +             * we receive anything else here then something weird
>>>>>> +             * has gone on.
>>>>>> +             */
>>>>>> +            dev_err(dev, "masked interrupt %s triggered\n",
>>>>>> +                mali_c55_interrupt_names[i]);
>>>>>> +        }
>>>>>> +    }
>>>>>> +
>>>>>> +    return IRQ_HANDLED;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_init_context(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_ctx *ctx;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
>>>>>> +    if (!ctx) {
>>>>>> +        dev_err(mali_c55->dev, "failed to allocate new context\n");
>>>>> No need for an error message when memory allocation fails.
>>>>>
>>>>>> +        return -ENOMEM;
>>>>>> +    }
>>>>>> +
>>>>>> +    ctx->base = mali_c55->res->start;
>>>>>> +    ctx->mali_c55 = mali_c55;
>>>>>> +
>>>>>> +    ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
>>>>>> +                 GFP_KERNEL | GFP_DMA);
>>>>>> +    if (!ctx->registers) {
>>>>>> +        ret = -ENOMEM;
>>>>>> +        goto err_free_ctx;
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The allocated memory is empty, we need to load the default
>>>>>> +     * register settings. We just read Ping; it's identical to Pong.
>>>>>> +     */
>>>>>> +    ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
>>>>>> +    if (ret)
>>>>>> +        goto err_free_registers;
>>>>>> +
>>>>>> +    list_add_tail(&ctx->list, &mali_c55->contexts);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Some features of the ISP need to be disabled by default and only
>>>>>> +     * enabled at the same time as they're configured by a parameters buffer
>>>>>> +     */
>>>>>> +
>>>>>> +    /* Bypass the sqrt and square compression and expansion modules */
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
>>>>>> +                 MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
>>>>>> +                 MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
>>>>>> +
>>>>>> +    /* Bypass the temper module */
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
>>>>>> +               MALI_C55_REG_BYPASS_2_TEMPER);
>>>>>> +
>>>>>> +    /* Bypass the colour noise reduction  */
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
>>>>>> +               MALI_C55_REG_BYPASS_4_CNR);
>>>>>> +
>>>>>> +    /* Disable the sinter module */
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
>>>>>> +                 MALI_C55_SINTER_ENABLE_MASK, 0x00);
>>>>>> +
>>>>>> +    /* Disable the RGB Gamma module for each output */
>>>>>> +    mali_c55_write(
>>>>>> +        mali_c55,
>>>>>> + MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
>>>>>> +        0x00);
>>>>>> +    mali_c55_write(
>>>>>> +        mali_c55,
>>>>>> + MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
>>>>>> +        0x00);
>>>>>> +
>>>>>> +    /* Disable the colour correction matrix */
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_free_registers:
>>>>>> +    kfree(ctx->registers);
>>>>>> +err_free_ctx:
>>>>>> +    kfree(ctx);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_runtime_resume(struct device *dev)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
>>>>>> +                      mali_c55->clks);
>>>>>> +    if (ret)
>>>>>> +        dev_err(mali_c55->dev, "failed to enable clocks\n");
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_runtime_suspend(struct device *dev)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>>>>> +
>>>>>> + clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct dev_pm_ops mali_c55_pm_ops = {
>>>>>> +    SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
>>>>>> +                pm_runtime_force_resume)
>>>>>> +    SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
>>>>>> +               NULL)
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_probe(struct platform_device *pdev)
>>>>>> +{
>>>>>> +    struct device *dev = &pdev->dev;
>>>>>> +    struct mali_c55 *mali_c55;
>>>>>> +    dma_cap_mask_t mask;
>>>>>> +    u32 version;
>>>>>> +    int ret;
>>>>>> +    u32 val;
>>>>>> +
>>>>>> +    mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
>>>>>> +    if (!mali_c55)
>>>>>> +        return dev_err_probe(dev, -ENOMEM,
>>>>>> +                     "failed to allocate memory\n");
>>>>>          return -ENOMEM;
>>>>>
>>>>> There's no need to print messages for memory allocation failures.
>>>>>
>>>>>> +
>>>>>> +    mali_c55->dev = dev;
>>>>>> +    platform_set_drvdata(pdev, mali_c55);
>>>>>> +
>>>>>> +    mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
>>>>>> +                                &mali_c55->res);
>>>>>> +    if (IS_ERR(mali_c55->base))
>>>>>> +        return dev_err_probe(dev, PTR_ERR(mali_c55->base),
>>>>>> +                     "failed to map IO memory\n");
>>>>>> +
>>>>>> +    ret = platform_get_irq(pdev, 0);
>>>>>> +    if (ret < 0)
>>>>>> +        return dev_err_probe(dev, ret, "failed to get interrupt num\n");
>>>>> s/ num// or s/num/number/
>>>>>
>>>>>> +
>>>>>> +    ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
>>>>>> +                    mali_c55_isr, IRQF_ONESHOT,
>>>>>> +                    dev_driver_string(&pdev->dev),
>>>>>> +                    &pdev->dev);
>>>>> Requested the IRQ should be done much lower, after you have initialized
>>>>> everything, or an IRQ that would fire early would have really bad
>>>>> consequences.
>>>>>
>>>>> A comment to explain why you need a threaded interrupt handler would be
>>>>> good. I assume it is due only to the need to transfer the registers
>>>>> using DMA. I wonder if we should then split the interrupt handler in
>>>>> two, with a non-threaded part for the operations that can run quickly,
>>>>> and a threaded part for the reprogramming.
>>>>>
>>>>> It may also be that we could just start the DMA transfer in the
>>>>> non-threaded handler without waiting synchronously for it to complete.
>>>>> That would be a bigger change, and would require checking race
>>>>> conditions carefully. On the other hand, I'm a bit concerned about the
>>>>> current implementation, have you tested what happens if the DMA transfer
>>>>> takes too long to complete, and spans frame boundaries ? This part could
>>>>> be addressed by a patch on top of this one.
>>>>>
>>>>>> +    if (ret)
>>>>>> +        return dev_err_probe(dev, ret, "failed to request irq\n");
>>>>>> +
>>>>>> +    for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
>>>>>> +        mali_c55->clks[i].id = mali_c55_clk_names[i];
>>>>>> +
>>>>>> +    ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
>>>>>> +    if (ret)
>>>>>> +        return dev_err_probe(dev, ret, "failed to acquire clocks\n");
>>>>>> +
>>>>>> +    pm_runtime_enable(&pdev->dev);
>>>>>> +
>>>>>> +    ret = pm_runtime_resume_and_get(&pdev->dev);
>>>>>> +    if (ret)
>>>>>> +        goto err_pm_runtime_disable;
>>>>>> +
>>>>>> +    of_reserved_mem_device_init(dev);
>>>>> I'd move this, the vb2_dma_contig_set_max_seg_size() call and the
>>>>> dma_cap_* calls before pm_runtime_enable() as they don't need the device
>>>>> to be powered.
>>>>>
>>>>>> +    version = mali_c55_check_hwcfg(mali_c55);
>>>>>> +    vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
>>>>>> +
>>>>>> +    /* Use "software only" context management. */
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>>>> +                 MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
>>>>>> +                 MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
>>>>> You handle that in mali_c55_isp_start(), does the register have to be
>>>>> set here too ?
>>>>>
>>>>>> +
>>>>>> +    dma_cap_zero(mask);
>>>>>> +    dma_cap_set(DMA_MEMCPY, mask);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * No error check, because we will just fallback on memcpy if there is
>>>>>> +     * no usable DMA channel on the system.
>>>>>> +     */
>>>>>> +    mali_c55->channel = dma_request_channel(mask, NULL, NULL);
>>>>>> +
>>>>>> +    INIT_LIST_HEAD(&mali_c55->contexts);
>>>>>> +    ret = mali_c55_init_context(mali_c55);
>>>>>> +    if (ret)
>>>>>> +        goto err_release_dma_channel;
>>>>>> +
>>>>> I'd move all the code from here ...
>>>>>
>>>>>> +    mali_c55->media_dev.dev = dev;
>>>>>> +    strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
>>>>>> +        sizeof(mali_c55->media_dev.model));
>>>>>> +    mali_c55->media_dev.hw_revision = version;
>>>>>> +
>>>>>> +    media_device_init(&mali_c55->media_dev);
>>>>>> +    ret = media_device_register(&mali_c55->media_dev);
>>>>>> +    if (ret)
>>>>>> +        goto err_cleanup_media_device;
>>>>>> +
>>>>>> +    mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
>>>>>> +    ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(dev, "failed to register V4L2 device\n");
>>>>>> +        goto err_unregister_media_device;
>>>>>> +    };
>>>>>> +
>>>>>> +    ret = mali_c55_register_entities(mali_c55);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(dev, "failed to register entities\n");
>>>>>> +        goto err_unregister_v4l2_device;
>>>>>> +    }
>>>>> ... to here to a separate function, or maybe fold it all in
>>>>> mali_c55_register_entities() (which should the be renamed). Same thing
>>>>> for the cleanup code.
>>>>>
>>>>>> +
>>>>>> +    /* Set safe stop to ensure we're in a non-streaming state */
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>>>>>> +               MALI_C55_INPUT_SAFE_STOP);
>>>>>> +    readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * We're ready to process interrupts. Clear any that are set and then
>>>>>> +     * unmask them for processing.
>>>>>> +     */
>>>>>> +    mali_c55_write(mali_c55, 0x30, 0xffffffff);
>>>>>> +    mali_c55_write(mali_c55, 0x34, 0xffffffff);
>>>>>> +    mali_c55_write(mali_c55, 0x40, 0x01);
>>>>>> +    mali_c55_write(mali_c55, 0x40, 0x00);
>>>>> Please replace the register addresses with macros.
>>>>>
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);
>>>>> The value should use the interrupt bits macros.
>>>>>
>>>>>> +
>>>>>> +    pm_runtime_put(&pdev->dev);
>>>>> Once power gets cut, the registers your programmed above may be lost. I
>>>>> think you should programe them in the runtime PM resume handler.
>>>>>
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_unregister_v4l2_device:
>>>>>> +    v4l2_device_unregister(&mali_c55->v4l2_dev);
>>>>>> +err_unregister_media_device:
>>>>>> +    media_device_unregister(&mali_c55->media_dev);
>>>>>> +err_cleanup_media_device:
>>>>>> +    media_device_cleanup(&mali_c55->media_dev);
>>>>>> +err_release_dma_channel:
>>>>>> +    dma_release_channel(mali_c55->channel);
>>>>>> +err_pm_runtime_disable:
>>>>>> +    pm_runtime_disable(&pdev->dev);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_remove(struct platform_device *pdev)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
>>>>>> +    struct mali_c55_ctx *ctx, *tmp;
>>>>>> +
>>>>>> +    list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
>>>>>> +        list_del(&ctx->list);
>>>>>> +        kfree(ctx->registers);
>>>>>> +        kfree(ctx);
>>>>>> +    }
>>>>>> +
>>>>>> +    mali_c55_remove_links(mali_c55);
>>>>>> +    mali_c55_unregister_entities(mali_c55);
>>>>>> +    v4l2_device_put(&mali_c55->v4l2_dev);
>>>>>> +    media_device_unregister(&mali_c55->media_dev);
>>>>>> +    media_device_cleanup(&mali_c55->media_dev);
>>>>>> +    dma_release_channel(mali_c55->channel);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct of_device_id mali_c55_of_match[] = {
>>>>>> +    { .compatible = "arm,mali-c55", },
>>>>>> +    {},
>>>>>      { /* Sentinel */ },
>>>>>
>>>>>> +};
>>>>>> +MODULE_DEVICE_TABLE(of, mali_c55_of_match);
>>>>>> +
>>>>>> +static struct platform_driver mali_c55_driver = {
>>>>>> +    .driver = {
>>>>>> +        .name = "mali-c55",
>>>>>> +        .of_match_table = of_match_ptr(mali_c55_of_match),
>>>>> Drop of_match_ptr().
>>>>>
>>>>>> +        .pm = &mali_c55_pm_ops,
>>>>>> +    },
>>>>>> +    .probe = mali_c55_probe,
>>>>>> +    .remove_new = mali_c55_remove,
>>>>>> +};
>>>>>> +
>>>>>> +module_platform_driver(mali_c55_driver);
>>>>> Blank line.
>>>>>
>>>>>> +MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
>>>>>> +MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
>>>>>> +MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
>>>>>> +MODULE_LICENSE("GPL");
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>>>> new file mode 100644
>>>>>> index 000000000000..ea8b7b866e7a
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>>>> @@ -0,0 +1,611 @@
>>>>>> +// SPDX-License-Identifier: GPL-2.0
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Image signal processor
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#include <linux/delay.h>
>>>>>> +#include <linux/iopoll.h>
>>>>>> +#include <linux/property.h>
>>>>>> +#include <linux/string.h>
>>>>>> +
>>>>>> +#include <media/media-entity.h>
>>>>>> +#include <media/v4l2-common.h>
>>>>>> +#include <media/v4l2-event.h>
>>>>>> +#include <media/v4l2-mc.h>
>>>>>> +#include <media/v4l2-subdev.h>
>>>>>> +
>>>>>> +#include "mali-c55-common.h"
>>>>>> +#include "mali-c55-registers.h"
>>>>>> +
>>>>>> +static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
>>>>>> +    {
>>>>>> +        .code = MEDIA_BUS_FMT_SRGGB20_1X20,
>>>>>> +        .order = MALI_C55_BAYER_ORDER_RGGB,
>>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .code = MEDIA_BUS_FMT_SGRBG20_1X20,
>>>>>> +        .order = MALI_C55_BAYER_ORDER_GRBG,
>>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .code = MEDIA_BUS_FMT_SGBRG20_1X20,
>>>>>> +        .order = MALI_C55_BAYER_ORDER_GBRG,
>>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .code = MEDIA_BUS_FMT_SBGGR20_1X20,
>>>>>> +        .order = MALI_C55_BAYER_ORDER_BGGR,
>>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .code = MEDIA_BUS_FMT_RGB202020_1X60,
>>>>>> +        .order = 0, /* Not relevant for this format */
>>>>>> +        .encoding = V4L2_PIXEL_ENC_RGB,
>>>>>> +    }
>>>>>> +    /*
>>>>>> +     * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
>>>>>> +     * also support YUV input from a sensor passed-through to the output. At
>>>>>> +     * present we have no mechanism to test that though so it may have to
>>>>>> +     * wait a while...
>>>>>> +     */
>>>>>> +};
>>>>>> +
>>>>>> +const struct mali_c55_isp_fmt *
>>>>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
>>>>>> +{
>>>>>> +    if (!fmt)
>>>>>> +        fmt = &mali_c55_isp_fmts[0];
>>>>>> +    else
>>>>>> +        fmt++;
>>>>>> +
>>>>>> +    for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
>>>>>> +        return fmt;
>>>>> That's peculiar.
>>>>>
>>>>>      if (!fmt)
>>>>>          fmt = &mali_c55_isp_fmts[0];
>>>>>      else if (fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)])
>>>>>          return ++fmt;
>>>>>      else
>>>>>          return NULL;
>>>>>
>>>>>> +
>>>>>> +    return NULL;
>>>>>> +}
>>>>>> +
>>>>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
>>>>>> +{
>>>>>> +    const struct mali_c55_isp_fmt *isp_fmt;
>>>>>> +
>>>>>> +    for_each_mali_isp_fmt(isp_fmt) {
>>>>> I would open-code the loop instead of using the macro, like you do
>>>>> below. It will be more efficient.
>>>>>
>>>>>> +        if (isp_fmt->code == mbus_code)
>>>>>> +            return true;
>>>>>> +    }
>>>>>> +
>>>>>> +    return false;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct mali_c55_isp_fmt *
>>>>>> +mali_c55_isp_get_mbus_config_by_code(u32 code)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
>>>>>> +        if (mali_c55_isp_fmts[i].code == code)
>>>>>> +            return &mali_c55_isp_fmts[i];
>>>>>> +    }
>>>>>> +
>>>>>> +    return NULL;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    u32 val;
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);
>>>>>      mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>>>>>                 MALI_C55_INPUT_SAFE_STOP);
>>>>>
>>>>>> + readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_isp_start(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>>>> +    const struct mali_c55_isp_fmt *cfg;
>>>>>> +    struct v4l2_mbus_framefmt *format;
>>>>> const
>>>>>
>>>>>> +    struct v4l2_subdev_state *state;
>>>>>> +    struct v4l2_rect *crop;
>>>>> const
>>>>>
>>>>>> +    struct v4l2_subdev *sd;
>>>>>> +    u32 val;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    sd = &mali_c55->isp.sd;
>>>>> Assign when declaring the variable.
>>>>>
>>>>>> +
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>>>> +                 MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
>>>>>> +
>>>>>> +    /* Apply input windowing */
>>>>>> +    state = v4l2_subdev_get_locked_active_state(sd);
>>>>> Using .enable_streams() (see below) you'll get this for free.
>>>>>
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +    format = v4l2_subdev_state_get_format(state,
>>>>>> +                          MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +    cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
>>>>>> +               MALI_C55_HC_START(crop->left));
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
>>>>>> +               MALI_C55_HC_SIZE(crop->width));
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
>>>>>> +               MALI_C55_VC_START(crop->top) |
>>>>>> +               MALI_C55_VC_SIZE(crop->height));
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
>>>>>> +                 MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
>>>>>> +                 MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
>>>>>> +                 MALI_C55_BAYER_ORDER_MASK, cfg->order);
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
>>>>>> +                 MALI_C55_INPUT_WIDTH_MASK,
>>>>>> +                 MALI_C55_INPUT_WIDTH_20BIT);
>>>>>> +
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>>>>> +                 MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
>>>>>> +                 cfg->encoding == V4L2_PIXEL_ENC_RGB ?
>>>>>> +                 MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
>>>>>> +
>>>>>> +    ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "failed to DMA config\n");
>>>>>> +        return ret;
>>>>>> +    }
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>>>>>> +               MALI_C55_INPUT_SAFE_START);
>>>>>> +    readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>>>>> Should you return an error in case of timeout ?
>>>>>
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)
>>>>> Why is this not handled wired to .s_stream() ? Or better,
>>>>> .enable_streams() and .disable_streams().
>>>>>
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>>>> +    struct v4l2_subdev *sd;
>>>>>> +
>>>>>> +    if (isp->remote_src) {
>>>>>> +        sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
>>>>>> +        v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
>>>>>> +    }
>>>>>> +    isp->remote_src = NULL;
>>>>>> +
>>>>>> +    mali_c55_isp_stop(mali_c55);
>>>>>> +}
>>>>>> +
>>>>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>>>> +    struct media_pad *sink_pad;
>>>>>> +    struct v4l2_subdev *sd;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
>>>>>> +    isp->remote_src = media_pad_remote_pad_unique(sink_pad);
>>>>>> +    if (IS_ERR(isp->remote_src)) {
>>>>> If you set the MUST_CONNECT flag for the MALI_C55_ISP_PAD_SINK_VIDEO pad
>>>>> I think you can drop this check.
>>>>>
>>>>>> +        dev_err(mali_c55->dev, "Failed to get source for ISP\n");
>>>>>> +        return PTR_ERR(isp->remote_src);
>>>>>> +    }
>>>>>> +
>>>>>> +    sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
>>>>>> +
>>>>>> +    isp->frame_sequence = 0;
>>>>>> +    ret = mali_c55_isp_start(mali_c55);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "Failed to start ISP\n");
>>>>>> +        isp->remote_src = NULL;
>>>>>> +        return ret;
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * We only support a single input stream, so we can just enable the 1st
>>>>>> +     * entry in the streams mask.
>>>>>> +     */
>>>>>> +    ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "Failed to start ISP source\n");
>>>>>> +        mali_c55_isp_stop(mali_c55);
>>>>>> +        return ret;
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>> +                       struct v4l2_subdev_state *state,
>>>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
>>>>>> +{
>>>>>> +    /*
>>>>>> +     * Only the internal RGB processed format is allowed on the regular
>>>>>> +     * processing source pad.
>>>>>> +     */
>>>>>> +    if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
>>>>>> +        if (code->index)
>>>>>> +            return -EINVAL;
>>>>>> +
>>>>>> +        code->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* On the sink and bypass pads all the supported formats are allowed. */
>>>>>> +    if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    code->code = mali_c55_isp_fmts[code->index].code;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
>>>>>> +                    struct v4l2_subdev_state *state,
>>>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
>>>>>> +{
>>>>>> +    const struct mali_c55_isp_fmt *cfg;
>>>>>> +
>>>>>> +    if (fse->index > 0)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Only the internal RGB processed format is allowed on the regular
>>>>>> +     * processing source pad.
>>>>>> +     *
>>>>>> +     * On the sink and bypass pads all the supported formats are allowed.
>>>>>> +     */
>>>>>> +    if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
>>>>>> +        if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
>>>>>> +            return -EINVAL;
>>>>>> +    } else {
>>>>>> +        cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
>>>>>> +        if (!cfg)
>>>>>> +            return -EINVAL;
>>>>>> +    }
>>>>>> +
>>>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
>>>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
>>>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
>>>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
>>>>>> +                struct v4l2_subdev_state *state,
>>>>>> +                struct v4l2_subdev_format *format)
>>>>>> +{
>>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
>>>>>> +    struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
>>>>>> +    const struct mali_c55_isp_fmt *cfg;
>>>>>> +    struct v4l2_rect *crop;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Disallow set_fmt on the source pads; format is fixed and the sizes
>>>>>> +     * are the result of applying the sink crop rectangle to the sink
>>>>>> +     * format.
>>>>>> +     */
>>>>>> +    if (format->pad)
>>>>>      if (format->pad != MALI_C55_ISP_PAD_SINK_VIDEO)
>>>>>
>>>>>> +        return v4l2_subdev_get_fmt(sd, state, format);
>>>>>> +
>>>>>> +    cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
>>>>>> +    if (!cfg)
>>>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>>>> +    fmt->field = V4L2_FIELD_NONE;
>>>>> Do you intentionally allow the colorspace fields to be overwritten to
>>>>> any value ? rkisp1_isp_set_src_fmt() and rkisp1_isp_set_sink_fmt() may
>>>>> show you how this could be handled.
>>>>>
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Clamp sizes in the accepted limits and clamp the crop rectangle in
>>>>>> +     * the new sizes.
>>>>>> +     */
>>>>>> +    clamp_t(unsigned int, fmt->width,
>>>>>> +        MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>>>>> +    clamp_t(unsigned int, fmt->width,
>>>>>> +        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>>>> clamp_t() returns a value, which you ignore. Those are no-ops. You meant
>>>>>
>>>>>      fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
>>>>>                   MALI_C55_MAX_WIDTH);
>>>>>      fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
>>>>>                    MALI_C55_MAX_HEIGHT);
>>>>>
>>>>> Same for every use of clamp_t() through the whole driver.
>>>>>
>>>>> Also, do you need clamp_t() ? I think all values are unsigned int, you
>>>>> can use clamp().
>>>>>
>>>>> Are there any alignment constraints, such a multiples of two for bayer
>>>>> formats ? Same in all the other locations where applicable.
>>>>>
>>>>>> +
>>>>>> +    sink_fmt = v4l2_subdev_state_get_format(state,
>>>>>> +                        MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +    *sink_fmt = *fmt;
>>>>>> +
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +    crop->left = 0;
>>>>>> +    crop->top = 0;
>>>>>> +    crop->width = fmt->width;
>>>>>> +    crop->height = fmt->height;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Propagate format to source pads. On the 'regular' output pad use
>>>>>> +     * the internal RGB processed format, while on the bypass pad simply
>>>>>> +     * replicate the ISP sink format. The sizes on both pads are the same as
>>>>>> +     * the ISP sink crop rectangle.
>>>>>> +     */
>>>>> Colorspace information will need to be propagated too.
>>>>>
>>>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
>>>>>> +    src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>>>> +    src_fmt->width = crop->width;
>>>>>> +    src_fmt->height = crop->height;
>>>>>> +
>>>>>> +    src_fmt = v4l2_subdev_state_get_format(state,
>>>>>> +                           MALI_C55_ISP_PAD_SOURCE_BYPASS);
>>>>>> +    src_fmt->code = fmt->code;
>>>>>> +    src_fmt->width = crop->width;
>>>>>> +    src_fmt->height = crop->height;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
>>>>>> +                      struct v4l2_subdev_state *state,
>>>>>> +                      struct v4l2_subdev_selection *sel)
>>>>>> +{
>>>>>> +    if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
>>>>>      sel->pad != MALI_C55_ISP_PAD_SINK_VIDEO
>>>>>
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
>>>>>> +                      struct v4l2_subdev_state *state,
>>>>>> +                      struct v4l2_subdev_selection *sel)
>>>>>> +{
>>>>>> +    struct v4l2_mbus_framefmt *src_fmt;
>>>>>> +    struct v4l2_mbus_framefmt *fmt;
>>>>> const
>>>>>
>>>>>> +    struct v4l2_rect *crop;
>>>>>> +
>>>>>> +    if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
>>>>> Ditto.
>>>>>
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +
>>>>>> +    clamp_t(unsigned int, sel->r.left, 0, fmt->width);
>>>>>> +    clamp_t(unsigned int, sel->r.top, 0, fmt->height);
>>>>>> +    clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
>>>>>> +        fmt->width - sel->r.left);
>>>>>> +    clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
>>>>>> +        fmt->height - sel->r.top);
>>>>>> +
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +    *crop = sel->r;
>>>>>> +
>>>>>> +    /* Propagate the crop rectangle sizes to the source pad format. */
>>>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
>>>>>> +    src_fmt->width = crop->width;
>>>>>> +    src_fmt->height = crop->height;
>>>>> Can you confirm that cropping doesn't affect the bypass path ? And maybe
>>>>> add a comment to mention it.
>>>>>
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
>>>>>> +    .enum_mbus_code        = mali_c55_isp_enum_mbus_code,
>>>>>> +    .enum_frame_size    = mali_c55_isp_enum_frame_size,
>>>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
>>>>>> +    .set_fmt        = mali_c55_isp_set_fmt,
>>>>>> +    .get_selection        = mali_c55_isp_get_selection,
>>>>>> +    .set_selection        = mali_c55_isp_set_selection,
>>>>>> +    .link_validate        = v4l2_subdev_link_validate_default,
>>>>>> +};
>>>>>> +
>>>>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct v4l2_event event = {
>>>>>> +        .type = V4L2_EVENT_FRAME_SYNC,
>>>>>> +    };
>>>>>> +
>>>>>> +    event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
>>>>>> +    v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
>>>>>> +}
>>>>>> +
>>>>>> +static int
>>>>>> +mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
>>>>>> +                 struct v4l2_event_subscription *sub)
>>>>>> +{
>>>>>> +    if (sub->type != V4L2_EVENT_FRAME_SYNC)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    /* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
>>>>>> +    if (sub->id != 0)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    return v4l2_event_subscribe(fh, sub, 0, NULL);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
>>>>>> +    .subscribe_event = mali_c55_isp_subscribe_event,
>>>>>> +    .unsubscribe_event = v4l2_event_subdev_unsubscribe,
>>>>>> +};
>>>>>> +
>>>>>> +static const struct v4l2_subdev_ops mali_c55_isp_ops = {
>>>>>> +    .pad    = &mali_c55_isp_pad_ops,
>>>>>> +    .core    = &mali_c55_isp_core_ops,
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
>>>>>> +                   struct v4l2_subdev_state *sd_state)
>>>>> You name this variable state in every other subdev operation handler.
>>>>>
>>>>>> +{
>>>>>> +    struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>> +    struct v4l2_rect *in_crop;
>>>>>> +
>>>>>> +    sink_fmt = v4l2_subdev_state_get_format(sd_state,
>>>>>> +                        MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +    src_fmt = v4l2_subdev_state_get_format(sd_state,
>>>>>> +                           MALI_C55_ISP_PAD_SOURCE);
>>>>>> +    in_crop = v4l2_subdev_state_get_crop(sd_state,
>>>>>> +                         MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +
>>>>>> +    sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>>>> +    sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>>>> +    sink_fmt->field = V4L2_FIELD_NONE;
>>>>>> +    sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>>> You should initialize the colorspace fields too. Same below.
>>>>>
>>>>>> +
>>>>>> +    *v4l2_subdev_state_get_format(sd_state,
>>>>>> +                  MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
>>>>>> +
>>>>>> +    src_fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>>>> +    src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>>>> +    src_fmt->field = V4L2_FIELD_NONE;
>>>>>> +    src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>>>> +
>>>>>> +    in_crop->top = 0;
>>>>>> +    in_crop->left = 0;
>>>>>> +    in_crop->width = MALI_C55_DEFAULT_WIDTH;
>>>>>> +    in_crop->height = MALI_C55_DEFAULT_HEIGHT;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
>>>>>> +    .init_state = mali_c55_isp_init_state,
>>>>>> +};
>>>>>> +
>>>>>> +static const struct media_entity_operations mali_c55_isp_media_ops = {
>>>>>> +    .link_validate        = v4l2_subdev_link_validate,
>>>>>      .link_validate = v4l2_subdev_link_validate,
>>>>>
>>>>> to match mali_c55_isp_internal_ops.
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
>>>>>> +                       struct v4l2_subdev *subdev,
>>>>>> +                       struct v4l2_async_connection *asc)
>>>>>> +{
>>>>>> +    struct mali_c55_isp *isp = container_of(notifier,
>>>>>> +                        struct mali_c55_isp, notifier);
>>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>>>> +    struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * By default we'll flag this link enabled and the TPG disabled, but
>>>>>> +     * no immutable flag because we need to be able to switch between the
>>>>>> +     * two.
>>>>>> +     */
>>>>>> +    ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
>>>>>> +                          MEDIA_LNK_FL_ENABLED);
>>>>>> +    if (ret)
>>>>>> +        dev_err(mali_c55->dev, "failed to create link for %s\n",
>>>>>> +            subdev->name);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
>>>>>> +{
>>>>>> +    struct mali_c55_isp *isp = container_of(notifier,
>>>>>> +                        struct mali_c55_isp, notifier);
>>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>>>> +
>>>>>> +    return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
>>>>>> +    .bound = mali_c55_isp_notifier_bound,
>>>>>> +    .complete = mali_c55_isp_notifier_complete,
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>>>> +    struct v4l2_async_connection *asc;
>>>>>> +    struct fwnode_handle *ep;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The ISP should have a single endpoint pointing to some flavour of
>>>>>> +     * CSI-2 receiver...but for now at least we do want everything to work
>>>>>> +     * normally even with no sensors connected, as we have the TPG. If we
>>>>>> +     * don't find a sensor just warn and return success.
>>>>>> +     */
>>>>>> +    ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
>>>>>> +                         0, 0, 0);
>>>>>> +    if (!ep) {
>>>>>> +        dev_warn(mali_c55->dev, "no local endpoint found\n");
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
>>>>>> +                          struct v4l2_async_connection);
>>>>>> +    if (IS_ERR(asc)) {
>>>>>> +        dev_err(mali_c55->dev, "failed to add remote fwnode\n");
>>>>>> +        ret = PTR_ERR(asc);
>>>>>> +        goto err_put_ep;
>>>>>> +    }
>>>>>> +
>>>>>> +    isp->notifier.ops = &mali_c55_isp_notifier_ops;
>>>>>> +    ret = v4l2_async_nf_register(&isp->notifier);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "failed to register notifier\n");
>>>>>> +        goto err_cleanup_nf;
>>>>>> +    }
>>>>>> +
>>>>>> +    fwnode_handle_put(ep);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_cleanup_nf:
>>>>>> +    v4l2_async_nf_cleanup(&isp->notifier);
>>>>>> +err_put_ep:
>>>>>> +    fwnode_handle_put(ep);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
>>>>>> +    struct v4l2_subdev *sd = &isp->sd;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    isp->mali_c55 = mali_c55;
>>>>>> +
>>>>>> +    v4l2_subdev_init(sd, &mali_c55_isp_ops);
>>>>>> +    sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
>>>>>> +    sd->entity.ops = &mali_c55_isp_media_ops;
>>>>>> +    sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
>>>>>> +    sd->internal_ops = &mali_c55_isp_internal_ops;
>>>>>> +    strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
>>>>>> +
>>>>>> +    isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
>>>>> The MUST_CONNECT flag would make sense here.
>>>>>
>>>>>> + isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>>>>>> +    isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
>>>>>> +
>>>>>> +    ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
>>>>>> +                     isp->pads);
>>>>>> +    if (ret)
>>>>>> +        return ret;
>>>>>> +
>>>>>> +    ret = v4l2_subdev_init_finalize(sd);
>>>>>> +    if (ret)
>>>>>> +        goto err_cleanup_media_entity;
>>>>>> +
>>>>>> +    ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>>>>> +    if (ret)
>>>>>> +        goto err_cleanup_subdev;
>>>>>> +
>>>>>> +    ret = mali_c55_isp_parse_endpoint(isp);
>>>>>> +    if (ret)
>>>>>> +        goto err_cleanup_subdev;
>>>>> As noted elsewhere, I think this belongs to mali-c55-core.c.
>>>>>
>>>>>> +
>>>>>> +    mutex_init(&isp->lock);
>>>>> This lock is used in mali-c55-capture.c only, that seems weird.
>>>>>
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_cleanup_subdev:
>>>>>> +    v4l2_subdev_cleanup(sd);
>>>>>> +err_cleanup_media_entity:
>>>>>> +    media_entity_cleanup(&sd->entity);
>>>>>> +    isp->mali_c55 = NULL;
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
>>>>>> +
>>>>>> +    if (!isp->mali_c55)
>>>>>> +        return;
>>>>>> +
>>>>>> +    mutex_destroy(&isp->lock);
>>>>>> +    v4l2_async_nf_unregister(&isp->notifier);
>>>>>> +    v4l2_async_nf_cleanup(&isp->notifier);
>>>>>> +    v4l2_device_unregister_subdev(&isp->sd);
>>>>>> +    v4l2_subdev_cleanup(&isp->sd);
>>>>>> +    media_entity_cleanup(&isp->sd.entity);
>>>>>> +}
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>>>> new file mode 100644
>>>>>> index 000000000000..cb27abde2aa5
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>>>> @@ -0,0 +1,258 @@
>>>>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Register definitions
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#ifndef _MALI_C55_REGISTERS_H
>>>>>> +#define _MALI_C55_REGISTERS_H
>>>>>> +
>>>>>> +#include <linux/bits.h>
>>>>>> +
>>>>>> +/* ISP Common 0x00000 - 0x000ff */
>>>>>> +
>>>>>> +#define MALI_C55_REG_API                0x00000
>>>>>> +#define MALI_C55_REG_PRODUCT                0x00004
>>>>>> +#define MALI_C55_REG_VERSION                0x00008
>>>>>> +#define MALI_C55_REG_REVISION                0x0000c
>>>>>> +#define MALI_C55_REG_PULSE_MODE                0x0003c
>>>>>> +#define MALI_C55_REG_INPUT_MODE_REQUEST            0x0009c
>>>>>> +#define MALI_C55_INPUT_SAFE_STOP            0x00
>>>>>> +#define MALI_C55_INPUT_SAFE_START            0x01
>>>>>> +#define MALI_C55_REG_MODE_STATUS            0x000a0
>>>>>> +#define MALI_C55_REG_INTERRUPT_MASK_VECTOR        0x00030
>>>>>> +#define MALI_C55_INTERRUPT_MASK_ALL            GENMASK(31, 0)
>>>>>> +
>>>>>> +#define MALI_C55_REG_GLOBAL_MONITOR            0x00050
>>>>>> +
>>>>>> +#define MALI_C55_REG_GEN_VIDEO                0x00080
>>>>>> +#define MALI_C55_REG_GEN_VIDEO_ON_MASK            BIT(0)
>>>>>> +#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK        BIT(1)
>>>>>> +#define MALI_C55_REG_GEN_PREFETCH_MASK GENMASK(31, 16)
>>>>>> +
>>>>>> +#define MALI_C55_REG_MCU_CONFIG                0x00020
>>>>>> +#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK        BIT(0)
>>>>> #define MALI_C55_REG_MCU_CONFIG_OVERRIDE        BIT(0)
>>>>>
>>>>> Same in other places where applicable.
>>>>>
>>>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK        BIT(1)
>>>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PING        BIT(1)
>>>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG        0x00
>>>>>> +#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK        BIT(8)
>>>>>> +#define MALI_C55_REG_PING_PONG_READ            0x00024
>>>>>> +#define MALI_C55_REG_PING_PONG_READ_MASK        BIT(2)
>>>>>> +
>>>>>> +#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR        0x00034
>>>>>> +#define MALI_C55_REG_INTERRUPT_CLEAR            0x00040
>>>>>> +#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR        0x00044
>>>>>> +#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS        0x00068
>>>>>> +#define MALI_C55_GPS_PONG_FITTED            BIT(0)
>>>>>> +#define MALI_C55_GPS_WDR_FITTED                BIT(1)
>>>>>> +#define MALI_C55_GPS_COMPRESSION_FITTED            BIT(2)
>>>>>> +#define MALI_C55_GPS_TEMPER_FITTED            BIT(3)
>>>>>> +#define MALI_C55_GPS_SINTER_LITE_FITTED            BIT(4)
>>>>>> +#define MALI_C55_GPS_SINTER_FITTED            BIT(5)
>>>>>> +#define MALI_C55_GPS_IRIDIX_LTM_FITTED            BIT(6)
>>>>>> +#define MALI_C55_GPS_IRIDIX_GTM_FITTED            BIT(7)
>>>>>> +#define MALI_C55_GPS_CNR_FITTED                BIT(8)
>>>>>> +#define MALI_C55_GPS_FRSCALER_FITTED            BIT(9)
>>>>>> +#define MALI_C55_GPS_DS_PIPE_FITTED            BIT(10)
>>>>>> +
>>>>>> +#define MALI_C55_REG_BLANKING                0x00084
>>>>>> +#define MALI_C55_REG_HBLANK_MASK            GENMASK(15, 0)
>>>>>> +#define MALI_C55_REG_VBLANK_MASK            GENMASK(31, 16)
>>>>>> +
>>>>>> +#define MALI_C55_REG_HC_START                0x00088
>>>>>> +#define MALI_C55_HC_START(h)                (((h) & 0xffff) << 16)
>>>>>> +#define MALI_C55_REG_HC_SIZE                0x0008c
>>>>>> +#define MALI_C55_HC_SIZE(h)                ((h) & 0xffff)
>>>>>> +#define MALI_C55_REG_VC_START_SIZE            0x00094
>>>>>> +#define MALI_C55_VC_START(v)                ((v) & 0xffff)
>>>>>> +#define MALI_C55_VC_SIZE(v)                (((v) & 0xffff) << 16)
>>>>>> +
>>>>>> +/* Ping/Pong Configuration Space */
>>>>>> +#define MALI_C55_REG_BASE_ADDR                0x18e88
>>>>>> +#define MALI_C55_REG_BYPASS_0                0x18eac
>>>>>> +#define MALI_C55_REG_BYPASS_0_VIDEO_TEST        BIT(0)
>>>>>> +#define MALI_C55_REG_BYPASS_0_INPUT_FMT            BIT(1)
>>>>>> +#define MALI_C55_REG_BYPASS_0_DECOMPANDER        BIT(2)
>>>>>> +#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR BIT(3)
>>>>>> +#define MALI_C55_REG_BYPASS_0_GAIN_WDR            BIT(4)
>>>>>> +#define MALI_C55_REG_BYPASS_0_FRAME_STITCH        BIT(5)
>>>>>> +#define MALI_C55_REG_BYPASS_1                0x18eb0
>>>>>> +#define MALI_C55_REG_BYPASS_1_DIGI_GAIN            BIT(0)
>>>>>> +#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS        BIT(1)
>>>>>> +#define MALI_C55_REG_BYPASS_1_FE_SQRT            BIT(2)
>>>>>> +#define MALI_C55_REG_BYPASS_1_RAW_FE            BIT(3)
>>>>>> +#define MALI_C55_REG_BYPASS_2                0x18eb8
>>>>>> +#define MALI_C55_REG_BYPASS_2_SINTER            BIT(0)
>>>>>> +#define MALI_C55_REG_BYPASS_2_TEMPER            BIT(1)
>>>>>> +#define MALI_C55_REG_BYPASS_3                0x18ebc
>>>>>> +#define MALI_C55_REG_BYPASS_3_SQUARE_BE            BIT(0)
>>>>>> +#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH BIT(1)
>>>>>> +#define MALI_C55_REG_BYPASS_3_MESH_SHADING        BIT(3)
>>>>>> +#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE        BIT(4)
>>>>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX            BIT(5)
>>>>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN        BIT(6)
>>>>>> +#define MALI_C55_REG_BYPASS_4                0x18ec0
>>>>>> +#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB        BIT(1)
>>>>>> +#define MALI_C55_REG_BYPASS_4_PF_CORRECTION        BIT(3)
>>>>>> +#define MALI_C55_REG_BYPASS_4_CCM            BIT(4)
>>>>>> +#define MALI_C55_REG_BYPASS_4_CNR            BIT(5)
>>>>>> +#define MALI_C55_REG_FR_BYPASS                0x18ec4
>>>>>> +#define MALI_C55_REG_DS_BYPASS                0x18ec8
>>>>>> +#define MALI_C55_BYPASS_CROP                BIT(0)
>>>>>> +#define MALI_C55_BYPASS_SCALER                BIT(1)
>>>>>> +#define MALI_C55_BYPASS_GAMMA_RGB            BIT(2)
>>>>>> +#define MALI_C55_BYPASS_SHARPEN                BIT(3)
>>>>>> +#define MALI_C55_BYPASS_CS_CONV                BIT(4)
>>>>>> +#define MALI_C55_REG_ISP_RAW_BYPASS            0x18ecc
>>>>>> +#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK        BIT(0)
>>>>>> +#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK GENMASK(9, 8)
>>>>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS        2
>>>>>> +#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS        1
>>>>>> +#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE BIT(1)
>>>>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS        BIT(0)
>>>>>> +
>>>>>> +#define MALI_C55_REG_ACTIVE_WIDTH_MASK            0xffff
>>>>>> +#define MALI_C55_REG_ACTIVE_HEIGHT_MASK 0xffff0000
>>>>>> +#define MALI_C55_REG_BAYER_ORDER            0x18e8c
>>>>>> +#define MALI_C55_BAYER_ORDER_MASK            GENMASK(1, 0)
>>>>>> +#define MALI_C55_REG_TPG_CH0                0x18ed8
>>>>>> +#define MALI_C55_TEST_PATTERN_ON_OFF            BIT(0)
>>>>>> +#define MALI_C55_TEST_PATTERN_RGB_MASK            BIT(1)
>>>>>> +#define MALI_C55_REG_TPG_R_BACKGROUND            0x18ee0
>>>>>> +#define MALI_C55_REG_TPG_G_BACKGROUND            0x18ee4
>>>>>> +#define MALI_C55_REG_TPG_B_BACKGROUND            0x18ee8
>>>>>> +#define MALI_C55_TPG_BACKGROUND_MAX            0xfffff
>>>>>> +#define MALI_C55_REG_INPUT_WIDTH            0x18f98
>>>>>> +#define MALI_C55_INPUT_WIDTH_MASK            GENMASK(18, 16)
>>>>>> +#define MALI_C55_INPUT_WIDTH_8BIT            0
>>>>>> +#define MALI_C55_INPUT_WIDTH_10BIT            1
>>>>>> +#define MALI_C55_INPUT_WIDTH_12BIT            2
>>>>>> +#define MALI_C55_INPUT_WIDTH_14BIT            3
>>>>>> +#define MALI_C55_INPUT_WIDTH_16BIT            4
>>>>>> +#define MALI_C55_INPUT_WIDTH_20BIT            5
>>>>>> +#define MALI_C55_REG_SPACE_SIZE                0x4000
>>>>>> +#define MALI_C55_REG_CONFIG_SPACES_OFFSET        0x0ab6c
>>>>>> +#define MALI_C55_CONFIG_SPACE_SIZE            0x1231c
>>>>>> +
>>>>>> +#define MALI_C55_REG_SINTER_CONFIG            0x19348
>>>>>> +#define MALI_C55_SINTER_VIEW_FILTER_MASK        GENMASK(1, 0)
>>>>>> +#define MALI_C55_SINTER_SCALE_MODE_MASK GENMASK(3, 2)
>>>>>> +#define MALI_C55_SINTER_ENABLE_MASK            BIT(4)
>>>>>> +#define MALI_C55_SINTER_FILTER_SELECT_MASK        BIT(5)
>>>>>> +#define MALI_C55_SINTER_INT_SELECT_MASK            BIT(6)
>>>>>> +#define MALI_C55_SINTER_RM_ENABLE_MASK            BIT(7)
>>>>>> +
>>>>>> +/* Colour Correction Matrix Configuration */
>>>>>> +#define MALI_C55_REG_CCM_ENABLE                0x1b07c
>>>>>> +#define MALI_C55_CCM_ENABLE_MASK            BIT(0)
>>>>>> +#define MALI_C55_REG_CCM_COEF_R_R            0x1b080
>>>>>> +#define MALI_C55_REG_CCM_COEF_R_G            0x1b084
>>>>>> +#define MALI_C55_REG_CCM_COEF_R_B            0x1b088
>>>>>> +#define MALI_C55_REG_CCM_COEF_G_R            0x1b090
>>>>>> +#define MALI_C55_REG_CCM_COEF_G_G            0x1b094
>>>>>> +#define MALI_C55_REG_CCM_COEF_G_B            0x1b098
>>>>>> +#define MALI_C55_REG_CCM_COEF_B_R            0x1b0a0
>>>>>> +#define MALI_C55_REG_CCM_COEF_B_G            0x1b0a4
>>>>>> +#define MALI_C55_REG_CCM_COEF_B_B            0x1b0a8
>>>>>> +#define MALI_C55_CCM_COEF_MASK                GENMASK(12, 0)
>>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R            0x1b0b0
>>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G            0x1b0b4
>>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B            0x1b0b8
>>>>>> +#define MALI_C55_CCM_ANTIFOG_GAIN_MASK GENMASK(11, 0)
>>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R        0x1b0c0
>>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G        0x1b0c4
>>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B        0x1b0c8
>>>>>> +#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK        GENMASK(11, 0)
>>>>>> +
>>>>>> +/*
>>>>>> + * The Mali-C55 ISP has up to two output pipes; known as full resolution and
>>>>>> + * down scaled. The register space for these is laid out identically, but offset
>>>>>> + * by 372 bytes.
>>>>>> + */
>>>>>> +#define MALI_C55_CAP_DEV_FR_REG_OFFSET        0x0
>>>>>> +#define MALI_C55_CAP_DEV_DS_REG_OFFSET        0x174
>>>>>> +
>>>>>> +#define MALI_C55_REG_CS_CONV_CONFIG(offset)        (0x1c098 + (offset))
>>>>>> +#define MALI_C55_CS_CONV_MATRIX_MASK            BIT(0)
>>>>>> +#define MALI_C55_CS_CONV_FILTER_MASK            BIT(1)
>>>>>> +#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK        BIT(2)
>>>>>> +#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK        BIT(3)
>>>>>> +#define MALI_C55_REG_Y_WRITER_MODE(offset)        (0x1c0ec + (offset))
>>>>>> +#define MALI_C55_REG_UV_WRITER_MODE(offset)        (0x1c144 + (offset))
>>>>>> +#define MALI_C55_WRITER_MODE_MASK            GENMASK(4, 0)
>>>>>> +#define MALI_C55_WRITER_SUBMODE_MASK            GENMASK(7, 6)
>>>>>> +#define MALI_C55_WRITER_FRAME_WRITE_MASK        BIT(9)
>>>>>> +#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset) (0x1c0f0 + (offset))
>>>>>> +#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset) (0x1c148 + (offset))
>>>>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)        ((w) << 0)
>>>>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)        ((h) << 16)
>>>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset) (0x1c0f4 + (offset))
>>>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset) (0x1c108 + (offset))
>>>>>> +#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK GENMASK(2, 0)
>>>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_RESTART        BIT(3)
>>>>>> +#define MALI_C55_REG_Y_WRITER_OFFSET(offset) (0x1c10c + (offset))
>>>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset) (0x1c14c + (offset))
>>>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset) (0x1c160 + (offset))
>>>>>> +#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK GENMASK(2, 0)
>>>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_RESTART        BIT(3)
>>>>>> +#define MALI_C55_REG_UV_WRITER_OFFSET(offset) (0x1c164 + (offset))
>>>>>> +
>>>>>> +#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
>>>>>> +#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE 0x18edc
>>>>>> +
>>>>>> +#define MALI_C55_REG_CROP_EN(offset)            (0x1c028 + (offset))
>>>>>> +#define MALI_C55_CROP_ENABLE                BIT(0)
>>>>>> +#define MALI_C55_REG_CROP_X_START(offset)        (0x1c02c + (offset))
>>>>>> +#define MALI_C55_REG_CROP_Y_START(offset)        (0x1c030 + (offset))
>>>>>> +#define MALI_C55_REG_CROP_X_SIZE(offset)        (0x1c034 + (offset))
>>>>>> +#define MALI_C55_REG_CROP_Y_SIZE(offset)        (0x1c038 + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset) (0x1c040 + (offset))
>>>>>> +#define MALI_C55_SCALER_TIMEOUT_EN            BIT(4)
>>>>>> +#define MALI_C55_SCALER_TIMEOUT(t)            ((t) << 16)
>>>>>> +#define MALI_C55_REG_SCALER_IN_WIDTH(offset) (0x1c044 + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_IN_HEIGHT(offset) (0x1c048 + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_OUT_WIDTH(offset) (0x1c04c + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset) (0x1c050 + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_HFILT_TINC(offset) (0x1c054 + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_HFILT_COEF(offset) (0x1c058 + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_VFILT_TINC(offset) (0x1c05c + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_VFILT_COEF(offset) (0x1c060 + (offset))
>>>>>> +
>>>>>> +#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset) (0x1c064 + (offset))
>>>>>> +#define MALI_C55_GAMMA_ENABLE_MASK            BIT(0)
>>>>>> +#define MALI_C55_REG_GAMMA_GAINS_1(offset)        (0x1c068 + (offset))
>>>>>> +#define MALI_C55_GAMMA_GAIN_R_MASK            GENMASK(11, 0)
>>>>>> +#define MALI_C55_GAMMA_GAIN_G_MASK            GENMASK(27, 16)
>>>>>> +#define MALI_C55_REG_GAMMA_GAINS_2(offset)        (0x1c06c + (offset))
>>>>>> +#define MALI_C55_GAMMA_GAIN_B_MASK            GENMASK(11, 0)
>>>>>> +#define MALI_C55_REG_GAMMA_OFFSETS_1(offset) (0x1c070 + (offset))
>>>>>> +#define MALI_C55_GAMMA_OFFSET_R_MASK            GENMASK(11, 0)
>>>>>> +#define MALI_C55_GAMMA_OFFSET_G_MASK            GENMASK(27, 16)
>>>>>> +#define MALI_C55_REG_GAMMA_OFFSETS_2(offset) (0x1c074 + (offset))
>>>>>> +#define MALI_C55_GAMMA_OFFSET_B_MASK            GENMASK(11, 0)
>>>>>> +
>>>>>> +/* Output DMA Writer */
>>>>>> +
>>>>>> +#define MALI_C55_OUTPUT_DISABLED        0
>>>>>> +#define MALI_C55_OUTPUT_RGB32            1
>>>>>> +#define MALI_C55_OUTPUT_A2R10G10B10        2
>>>>>> +#define MALI_C55_OUTPUT_RGB565            3
>>>>>> +#define MALI_C55_OUTPUT_RGB24            4
>>>>>> +#define MALI_C55_OUTPUT_GEN32            5
>>>>>> +#define MALI_C55_OUTPUT_RAW16            6
>>>>>> +#define MALI_C55_OUTPUT_AYUV            8
>>>>>> +#define MALI_C55_OUTPUT_Y410            9
>>>>>> +#define MALI_C55_OUTPUT_YUY2            10
>>>>>> +#define MALI_C55_OUTPUT_UYVY            11
>>>>>> +#define MALI_C55_OUTPUT_Y210            12
>>>>>> +#define MALI_C55_OUTPUT_NV12_21            13
>>>>>> +#define MALI_C55_OUTPUT_YUV_420_422        17
>>>>>> +#define MALI_C55_OUTPUT_P210_P010        19
>>>>>> +#define MALI_C55_OUTPUT_YUV422            20
>>>>> I'd define this just below the MALI_C55_REG_UV_WRITER_MODE register
>>>>> macro.
>>>>>
>>>>>> +
>>>>>> +#define MALI_C55_OUTPUT_PLANE_ALT0        0
>>>>>> +#define MALI_C55_OUTPUT_PLANE_ALT1        1
>>>>>> +#define MALI_C55_OUTPUT_PLANE_ALT2        2
>>>>> Same here ?
>>>>>
>>>>>> +
>>>>>> +#endif /* _MALI_C55_REGISTERS_H */
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>>>> new file mode 100644
>>>>>> index 000000000000..8edae87f1e5f
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>>>> @@ -0,0 +1,382 @@
>>>>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Resizer Coefficients
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#ifndef _MALI_C55_RESIZER_COEFS_H
>>>>>> +#define _MALI_C55_RESIZER_COEFS_H
>>>>>> +
>>>>>> +#include "mali-c55-common.h"
>>>>>> +
>>>>>> +#define MALI_C55_RESIZER_COEFS_NUM_BANKS    8
>>>>>> +#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES    64
>>>>> Do these belongs to mali-c55-registers.h ?
>>>>>
>>>>>> +
>>>>>> +static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
>>>>>> +    {    /* Bank 0 */
>>>>>> +        0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
>>>>>> +        0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
>>>>>> +        0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
>>>>>> +        0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
>>>>>> +        0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
>>>>>> +        0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
>>>>>> +        0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
>>>>>> +        0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
>>>>>> +        0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
>>>>>> +        0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
>>>>>> +        0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
>>>>>> +        0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
>>>>>> +        0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
>>>>>> +        0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
>>>>>> +        0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
>>>>>> +        0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
>>>>>> +    },
>>>>>> +    {    /* Bank 1 */
>>>>>> +        0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
>>>>>> +        0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
>>>>>> +        0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
>>>>>> +        0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
>>>>>> +        0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
>>>>>> +        0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
>>>>>> +        0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
>>>>>> +        0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
>>>>>> +        0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
>>>>>> +        0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
>>>>>> +        0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
>>>>>> +        0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
>>>>>> +        0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
>>>>>> +        0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
>>>>>> +        0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
>>>>>> +        0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
>>>>>> +    },
>>>>>> +    {    /* Bank 2 */
>>>>>> +        0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
>>>>>> +        0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
>>>>>> +        0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
>>>>>> +        0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
>>>>>> +        0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
>>>>>> +        0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
>>>>>> +        0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
>>>>>> +        0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
>>>>>> +        0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
>>>>>> +        0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
>>>>>> +        0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
>>>>>> +        0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
>>>>>> +        0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
>>>>>> +        0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
>>>>>> +        0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
>>>>>> +        0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
>>>>>> +    },
>>>>>> +    {    /* Bank 3 */
>>>>>> +        0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
>>>>>> +        0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
>>>>>> +        0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
>>>>>> +        0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
>>>>>> +        0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
>>>>>> +        0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
>>>>>> +        0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
>>>>>> +        0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
>>>>>> +        0x20100000, 0x00000010, 0x1f110000, 0x00000010,
>>>>>> +        0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
>>>>>> +        0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
>>>>>> +        0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
>>>>>> +        0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
>>>>>> +        0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
>>>>>> +        0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
>>>>>> +        0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
>>>>>> +    },
>>>>>> +    {    /* Bank 4 */
>>>>>> +        0x17090000, 0x00000917, 0x18090000, 0x00000916,
>>>>>> +        0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
>>>>>> +        0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
>>>>>> +        0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
>>>>>> +        0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
>>>>>> +        0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
>>>>>> +        0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
>>>>>> +        0x190f0300, 0x00000411, 0x18100300, 0x00000411,
>>>>>> +        0x1a100300, 0x00000310, 0x18110400, 0x00000310,
>>>>>> +        0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
>>>>>> +        0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
>>>>>> +        0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
>>>>>> +        0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
>>>>>> +        0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
>>>>>> +        0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
>>>>>> +        0x17160800, 0x0000010a, 0x18160900, 0x00000009,
>>>>>> +    },
>>>>>> +    {    /* Bank 5 */
>>>>>> +        0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
>>>>>> +        0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
>>>>>> +        0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
>>>>>> +        0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
>>>>>> +        0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
>>>>>> +        0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
>>>>>> +        0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
>>>>>> +        0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
>>>>>> +        0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
>>>>>> +        0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
>>>>>> +        0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
>>>>>> +        0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
>>>>>> +        0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
>>>>>> +        0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
>>>>>> +        0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
>>>>>> +        0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
>>>>>> +    },
>>>>>> +    {    /* Bank 6 */
>>>>>> +        0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
>>>>>> +        0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
>>>>>> +        0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
>>>>>> +        0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
>>>>>> +        0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
>>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>>>> +        0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
>>>>>> +        0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>>>> +        0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
>>>>>> +        0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
>>>>>> +        0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
>>>>>> +        0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>>>>> +        0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>>>>> +    },
>>>>>> +    {    /* Bank 7 */
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +    }
>>>>>> +};
>>>>>> +
>>>>>> +static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
>>>>>> +    {    /* Bank 0 */
>>>>>> +        0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
>>>>>> +        0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
>>>>>> +        0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
>>>>>> +        0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
>>>>>> +        0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
>>>>>> +        0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
>>>>>> +        0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
>>>>>> +        0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
>>>>>> +        0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
>>>>>> +        0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
>>>>>> +        0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
>>>>>> +        0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
>>>>>> +        0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
>>>>>> +        0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
>>>>>> +        0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
>>>>>> +        0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
>>>>>> +    },
>>>>>> +    {    /* Bank 1 */
>>>>>> +        0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
>>>>>> +        0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
>>>>>> +        0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
>>>>>> +        0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
>>>>>> +        0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
>>>>>> +        0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
>>>>>> +        0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
>>>>>> +        0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
>>>>>> +        0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
>>>>>> +        0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
>>>>>> +        0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
>>>>>> +        0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
>>>>>> +        0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
>>>>>> +        0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
>>>>>> +        0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
>>>>>> +        0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
>>>>>> +    },
>>>>>> +    {    /* Bank 2 */
>>>>>> +        0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
>>>>>> +        0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
>>>>>> +        0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
>>>>>> +        0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
>>>>>> +        0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
>>>>>> +        0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
>>>>>> +        0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
>>>>>> +        0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
>>>>>> +        0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
>>>>>> +        0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
>>>>>> +        0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
>>>>>> +        0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
>>>>>> +        0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
>>>>>> +        0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
>>>>>> +        0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
>>>>>> +        0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
>>>>>> +    },
>>>>>> +    {    /* Bank 3 */
>>>>>> +        0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
>>>>>> +        0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
>>>>>> +        0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
>>>>>> +        0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
>>>>>> +        0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
>>>>>> +        0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
>>>>>> +        0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
>>>>>> +        0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
>>>>>> +        0x20100000, 0x00000010, 0x1f100000, 0x00000011,
>>>>>> +        0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
>>>>>> +        0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
>>>>>> +        0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
>>>>>> +        0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
>>>>>> +        0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
>>>>>> +        0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
>>>>>> +        0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
>>>>>> +    },
>>>>>> +    {    /* Bank 4 */
>>>>>> +        0x17170900, 0x00000009, 0x18160900, 0x00000009,
>>>>>> +        0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
>>>>>> +        0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
>>>>>> +        0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
>>>>>> +        0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
>>>>>> +        0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
>>>>>> +        0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
>>>>>> +        0x19110400, 0x0000030f, 0x18110400, 0x00000310,
>>>>>> +        0x1a100300, 0x00000310, 0x18100300, 0x00000411,
>>>>>> +        0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
>>>>>> +        0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
>>>>>> +        0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
>>>>>> +        0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
>>>>>> +        0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
>>>>>> +        0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
>>>>>> +        0x170a0100, 0x00000816, 0x18090000, 0x00000916,
>>>>>> +    },
>>>>>> +    {    /* Bank 5 */
>>>>>> +        0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
>>>>>> +        0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
>>>>>> +        0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
>>>>>> +        0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
>>>>>> +        0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
>>>>>> +        0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
>>>>>> +        0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
>>>>>> +        0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
>>>>>> +        0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
>>>>>> +        0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
>>>>>> +        0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
>>>>>> +        0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
>>>>>> +        0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
>>>>>> +        0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
>>>>>> +        0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
>>>>>> +        0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
>>>>>> +    },
>>>>>> +    {    /* Bank 6 */
>>>>>> +        0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
>>>>>> +        0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>>>>> +        0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
>>>>>> +        0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
>>>>>> +        0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
>>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>>>> +        0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>>>>> +        0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>>>> +        0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
>>>>>> +        0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
>>>>>> +        0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
>>>>>> +        0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
>>>>>> +        0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
>>>>>> +    },
>>>>>> +    {    /* Bank 7 */
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +    }
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_resizer_coef_bank {
>>>>>> +    unsigned int bank;
>>>>> This is always equal to the index of the entry in the
>>>>> mali_c55_coefficient_banks array, you can drop it.
>>>>>
>>>>>> +    unsigned int top;
>>>>>> +    unsigned int bottom;
>>>>> The bottom value of bank N is always equal to the top value of bank N+1.
>>>>> You can simplify this by storing a single value.
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
>>>>>> +    {
>>>>>> +        .bank = 0,
>>>>>> +        .top = 1000,
>>>>>> +        .bottom = 770,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .bank = 1,
>>>>>> +        .top = 769,
>>>>>> +        .bottom = 600,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .bank = 2,
>>>>>> +        .top = 599,
>>>>>> +        .bottom = 460,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .bank = 3,
>>>>>> +        .top = 459,
>>>>>> +        .bottom = 354,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .bank = 4,
>>>>>> +        .top = 353,
>>>>>> +        .bottom = 273,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .bank = 5,
>>>>>> +        .top = 272,
>>>>>> +        .bottom = 210,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .bank = 6,
>>>>>> +        .top = 209,
>>>>>> +        .bottom = 162,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .bank = 7,
>>>>>> +        .top = 161,
>>>>>> +        .bottom = 125,
>>>>>> +    },
>>>>>> +};
>>>>>> +
>>>>> A small comment would be nice, such as
>>>>>
>>>>> /* Select a bank of resizer coefficients, based on the scaling ratio. */
>>>>>
>>>>>> +static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,
>>>>> This function is related to the resizers. Add "rsz" somewhere in the
>>>>> function name, and pass a resizer pointer.
>>>>>
>>>>>> +                        unsigned int crop,
>>>>>> +                        unsigned int scale)
>>>>> I think those are the input and output sizes to the scaler. Rename them
>>>>> to make it clearer.
>>>>>
>>>>>> +{
>>>>>> +    unsigned int tmp;
>>>>> tmp is almost always a bad variable name. Please use a more descriptive
>>>>> name, size as rsz_ratio.
>>>>>
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    tmp = (scale * 1000U) / crop;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
>>>>>> +        if (tmp >= mali_c55_coefficient_banks[i].bottom &&
>>>>>> +            tmp <= mali_c55_coefficient_banks[i].top)
>>>>>> +            return mali_c55_coefficient_banks[i].bank;
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * We shouldn't ever get here, in theory. As we have no good choices
>>>>>> +     * simply warn the user and use the first bank of coefficients.
>>>>>> +     */
>>>>>> +    dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
>>>>>> +    return 0;
>>>>>> +}
>>>>> And everything else belongs to mali-c55-resizer.c. Drop this header
>>>>> file.
>>>>>
>>>>>> +
>>>>>> +#endif /* _MALI_C55_RESIZER_COEFS_H */
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>>>> new file mode 100644
>>>>>> index 000000000000..0a5a2969d3ce
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>>>> @@ -0,0 +1,779 @@
>>>>>> +// SPDX-License-Identifier: GPL-2.0
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Image signal processor
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#include <linux/math.h>
>>>>>> +#include <linux/minmax.h>
>>>>>> +
>>>>>> +#include <media/media-entity.h>
>>>>>> +#include <media/v4l2-subdev.h>
>>>>>> +
>>>>>> +#include "mali-c55-common.h"
>>>>>> +#include "mali-c55-registers.h"
>>>>>> +#include "mali-c55-resizer-coefs.h"
>>>>>> +
>>>>>> +/* Scaling factor in Q4.20 format. */
>>>>>> +#define MALI_C55_RZR_SCALER_FACTOR    (1U << 20)
>>>>>> +
>>>>>> +static const u32 rzr_non_bypass_src_fmts[] = {
>>>>>> +    MEDIA_BUS_FMT_RGB121212_1X36,
>>>>>> +    MEDIA_BUS_FMT_YUV10_1X30
>>>>>> +};
>>>>>> +
>>>>>> +static const char * const mali_c55_resizer_names[] = {
>>>>>> +    [MALI_C55_RZR_FR] = "resizer fr",
>>>>>> +    [MALI_C55_RZR_DS] = "resizer ds",
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
>>>>>> +                     struct v4l2_subdev_state *state)
>>>>>> +{
>>>>>> +    unsigned int reg_offset = rzr->cap_dev->reg_offset;
>>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>>>> +    struct v4l2_mbus_framefmt *fmt;
>>>>>> +    struct v4l2_rect *crop;
>>>> const
>>>>
>>>>>> +
>>>>>> +    /* Verify if crop should be enabled. */
>>>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
>>>>>> +
>>>>>> +    if (fmt->width == crop->width && fmt->height == crop->height)
>>>>>> +        return MALI_C55_BYPASS_CROP;
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
>>>>>> +               crop->left);
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
>>>>>> +               crop->top);
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
>>>>>> +               crop->width);
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
>>>>>> +               crop->height);
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
>>>>>> +               MALI_C55_CROP_ENABLE);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
>>>>>> +                    struct v4l2_subdev_state *state)
>>>>>> +{
>>>>>> +    unsigned int reg_offset = rzr->cap_dev->reg_offset;
>>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>>>> +    struct v4l2_rect *crop, *scale;
>>>> const
>>>>
>>>> Once "[PATCH v4 0/3] media: v4l2-subdev: Support const-awareness in
>>>> state accessors" gets merged, the state argument to this function can be
>>>> made const too. Same for other functions, as applicable.
>>>>
>>>>>> +    unsigned int h_bank, v_bank;
>>>>>> +    u64 h_scale, v_scale;
>>>>>> +
>>>>>> +    /* Verify if scaling should be enabled. */
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
>>>>>> +    scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
>>>>>> +
>>>>>> +    if (crop->width == scale->width && crop->height == scale->height)
>>>>>> +        return MALI_C55_BYPASS_SCALER;
>>>>>> +
>>>>>> +    /* Program the V/H scaling factor in Q4.20 format. */
>>>>>> +    h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
>>>>>> +    v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
>>>>>> +
>>>>>> +    do_div(h_scale, scale->width);
>>>>>> +    do_div(v_scale, scale->height);
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
>>>>>> +               crop->width);
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
>>>>>> +               crop->height);
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
>>>>>> +               scale->width);
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
>>>>>> +               scale->height);
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
>>>>>> +               h_scale);
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
>>>>>> +               v_scale);
>>>>>> +
>>>>>> +    h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
>>>>>> +                         scale->width);
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
>>>>>> +               h_bank);
>>>>>> +
>>>>>> +    v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
>>>>>> +                         scale->height);
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
>>>>>> +               v_bank);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
>>>>>> +                 struct v4l2_subdev_state *state)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>>>> +    u32 bypass = 0;
>>>>>> +
>>>>>> +    /* Verify if cropping and scaling should be enabled. */
>>>>>> +    bypass |= mali_c55_rzr_program_crop(rzr, state);
>>>>>> +    bypass |= mali_c55_rzr_program_resizer(rzr, state);
>>>>>> +
>>>>>> +    mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
>>>>>> +                 MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
>>>>>> +                 MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
>>>>>> +                 bypass);
>>>>>> +}
>>>>>> +
>>>>>> +/*
>>>>>> + * Inspect the routing table to know which of the two (mutually exclusive)
>>>>>> + * routes is enabled and return the sink pad id of the active route.
>>>>>> + */
>>>>>> +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
>>>>>> +{
>>>>>> +    struct v4l2_subdev_krouting *routing = &state->routing;
>>>>>> +    struct v4l2_subdev_route *route;
>>>>>> +
>>>>>> +    /* A single route is enabled at a time. */
>>>>>> +    for_each_active_route(routing, route)
>>>>>> +        return route->sink_pad;
>>>>>> +
>>>>>> +    return MALI_C55_RZR_SINK_PAD;
>>>>>> +}
>>>>>> +
>>>>>> +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
>>>>>> +{
>>>>>> +    u32 corrected_code = 0;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The ISP takes input in a 20-bit format, but can only output 16-bit
>>>>>> +     * RAW bayer data (with the 4 least significant bits from the input
>>>>>> +     * being lost). Return the 16-bit version of the 20-bit input formats.
>>>>>> +     */
>>>>>> +    switch (mbus_code) {
>>>>>> +    case MEDIA_BUS_FMT_SBGGR20_1X20:
>>>>>> +        corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
>>>>>> +        break;
>>>>>> +    case MEDIA_BUS_FMT_SGBRG20_1X20:
>>>>>> +        corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
>>>>>> +        break;
>>>>>> +    case MEDIA_BUS_FMT_SGRBG20_1X20:
>>>>>> +        corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
>>>>>> +        break;
>>>>>> +    case MEDIA_BUS_FMT_SRGGB20_1X20:
>>>>>> +        corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
>>>>>> +        break;
>>>> Would it make sense to add the shifted code to mali_c55_isp_fmt ?
>>>>
>>>>>> +    }
>>>>>> +
>>>>>> +    return corrected_code;
>>>>>> +}
>>>>>> +
>>>>>> +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
>>>>>> +                      struct v4l2_subdev_state *state,
>>>>>> +                      struct v4l2_subdev_krouting *routing)
>>>> I think the last argument can be const.
>>>>
>>>>>> +{
>>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>>>> +                            sd);
>>>> A to_mali_c55_resizer() static inline function would be useful. Same for
>>>> other components, where applicable.
>>>>
>>>>>> +    unsigned int active_sink = UINT_MAX;
>>>>>> +    struct v4l2_mbus_framefmt *src_fmt;
>>>>>> +    struct v4l2_rect *crop, *compose;
>>>>>> +    struct v4l2_subdev_route *route;
>>>>>> +    unsigned int active_routes = 0;
>>>>>> +    struct v4l2_mbus_framefmt *fmt;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    ret = v4l2_subdev_routing_validate(sd, routing, 0);
>>>>>> +    if (ret)
>>>>>> +        return ret;
>>>>>> +
>>>>>> +    /* Only a single route can be enabled at a time. */
>>>>>> +    for_each_active_route(routing, route) {
>>>>>> +        if (++active_routes > 1) {
>>>>>> +            dev_err(rzr->mali_c55->dev,
>>>>>> +                "Only one route can be active");
>>>> No kernel log message with a level higher than dev_dbg() from
>>>> user-controlled paths please, here and where applicable. This is to
>>>> avoid giving applications an easy way to flood the kernel log.
>>>>
>>>>>> +            return -EINVAL;
>>>>>> +        }
>>>>>> +
>>>>>> +        active_sink = route->sink_pad;
>>>>>> +    }
>>>>>> +    if (active_sink == UINT_MAX) {
>>>>>> +        dev_err(rzr->mali_c55->dev, "One route has to be active");
>>>>>> +        return -EINVAL;
>>>>>> +    }
>>>> The recommended handling of invalid routing is to adjust the routing
>>>> table, not to return errors.
>>>>
>>>>>> +
>>>>>> +    ret = v4l2_subdev_set_routing(sd, state, routing);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
>>>>>> +        return ret;
>>>>>> +    }
>>>>>> +
>>>>>> +    fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
>>>>>> +    compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
>>>>>> +
>>>>>> +    fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>>>> +    fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>>>> +    fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>> There are other colorspace-related fields.
>>>>
>>>>>> +    fmt->field = V4L2_FIELD_NONE;
>>>> I wonder if we should really update the sink pad format, or just
>>>> propagate it. If we update it, I think it should be set to defaults on
>>>> both sink pads, not just the active sink pad.
>>>>
>>>>>> +
>>>>>> +    if (active_sink == MALI_C55_RZR_SINK_PAD) {
>>>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>>>> +
>>>>>> +        crop->left = crop->top = 0;
>>>>          crop->left = 0;
>>>>          crop->top = 0;
>>>>
>>>>>> +        crop->width = MALI_C55_DEFAULT_WIDTH;
>>>>>> +        crop->height = MALI_C55_DEFAULT_HEIGHT;
>>>>>> +
>>>>>> +        *compose = *crop;
>>>>>> +    } else {
>>>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* Propagate the format to the source pad */
>>>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
>>>>>> +                           0);
>>>>>> +    *src_fmt = *fmt;
>>>>>> +
>>>>>> +    /* In the event this is the bypass pad the mbus code needs correcting */
>>>>>> +    if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
>>>>>> +        src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>> +                       struct v4l2_subdev_state *state,
>>>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
>>>>>> +{
>>>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
>>>>>> +    const struct mali_c55_isp_fmt *fmt;
>>>>>> +    unsigned int index = 0;
>>>>>> +    u32 sink_pad;
>>>>>> +
>>>>>> +    switch (code->pad) {
>>>>>> +    case MALI_C55_RZR_SINK_PAD:
>>>>>> +        if (code->index)
>>>>>> +            return -EINVAL;
>>>>>> +
>>>>>> +        code->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    case MALI_C55_RZR_SOURCE_PAD:
>>>>>> +        sink_pad = mali_c55_rzr_get_active_sink(state);
>>>>>> +        sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
>>>>>> +
>>>>>> +        /*
>>>>>> +         * If the active route is from the Bypass sink pad, then the
>>>>>> +         * source pad is a simple passthrough of the sink format,
>>>>>> +         * downshifted to 16-bits.
>>>>>> +         */
>>>>>> +
>>>>>> +        if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>>>>> +            if (code->index)
>>>>>> +                return -EINVAL;
>>>>>> +
>>>>>> +            code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
>>>>>> +            if (!code->code)
>>>>>> +                return -EINVAL;
>>>>>> +
>>>>>> +            return 0;
>>>>>> +        }
>>>>>> +
>>>>>> +        /*
>>>>>> +         * If the active route is from the non-bypass sink then we can
>>>>>> +         * select either RGB or conversion to YUV.
>>>>>> +         */
>>>>>> +
>>>>>> +        if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
>>>>>> +            return -EINVAL;
>>>>>> +
>>>>>> +        code->code = rzr_non_bypass_src_fmts[code->index];
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    case MALI_C55_RZR_SINK_BYPASS_PAD:
>>>>>> +        for_each_mali_isp_fmt(fmt) {
>>>>>> +            if (index++ == code->index) {
>>>>>> +                code->code = fmt->code;
>>>>>> +                return 0;
>>>>>> +            }
>>>>>> +        }
>>>>>> +
>>>>>> +        break;
>>>>>> +    }
>>>>>> +
>>>>>> +    return -EINVAL;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
>>>>>> +                    struct v4l2_subdev_state *state,
>>>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
>>>>>> +{
>>>>>> +    if (fse->index)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
>>>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
>>>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
>>>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
>>>>>> +                     struct v4l2_subdev_state *state,
>>>>>> +                     struct v4l2_subdev_format *format)
>>>>>> +{
>>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
>>>>>> +    struct v4l2_rect *rect;
>>>>>> +    unsigned int sink_pad;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Clamp to min/max and then reset crop and compose rectangles to the
>>>>>> +     * newly applied size.
>>>>>> +     */
>>>>>> +    clamp_t(unsigned int, fmt->width,
>>>>>> +        MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>>>>> +    clamp_t(unsigned int, fmt->height,
>>>>>> +        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>>> Please check comments for other components related to the colorspace
>>>> fields, to decide how to handle them here.
>>>>
>>>>>> +
>>>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
>>>>>> +    if (sink_pad == MALI_C55_RZR_SINK_PAD) {
>>>> The selection here should depend on format->pad, not the active sink
>>>> pad.
>>>>
>>>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>>>> +
>>>>>> +        rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
>>>>>> +        rect->left = 0;
>>>>>> +        rect->top = 0;
>>>>>> +        rect->width = fmt->width;
>>>>>> +        rect->height = fmt->height;
>>>>>> +
>>>>>> +        rect = v4l2_subdev_state_get_compose(state,
>>>>>> +                             MALI_C55_RZR_SINK_PAD);
>>>>>> +        rect->left = 0;
>>>>>> +        rect->top = 0;
>>>>>> +        rect->width = fmt->width;
>>>>>> +        rect->height = fmt->height;
>>>>>> +    } else {
>>>>>> +        /*
>>>>>> +         * Make sure the media bus code is one of the supported
>>>>>> +         * ISP input media bus codes.
>>>>>> +         */
>>>>>> +        if (!mali_c55_isp_is_format_supported(fmt->code))
>>>>>> +            fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
>>>>>> +    }
>>>>>> +
>>>>>> +    *v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
>>>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
>>>> Propagation to the source pad, however, should depend on the active
>>>> route. If format->pad is routed to the source pad, you should propagate,
>>>> otherwise, you shouldn't.
>>>>
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
>>>>>> +                       struct v4l2_subdev_state *state,
>>>>>> +                       struct v4l2_subdev_format *format)
>>>>>> +{
>>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>>>> +                            sd);
>>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
>>>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
>>>>>> +    struct v4l2_rect *crop, *compose;
>>>>>> +    unsigned int sink_pad;
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
>>>>>> +    sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
>>>>>> +    compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
>>>>>> +
>>>>>> +    /* FR Bypass pipe. */
>>>>>> +
>>>>>> +    if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>>>>> +        /*
>>>>>> +         * Format on the source pad is the same as the one on the
>>>>>> +         * sink pad, downshifted to 16-bits.
>>>>>> +         */
>>>>>> +        fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
>>>>>> +        if (!fmt->code)
>>>>>> +            return -EINVAL;
>>>>>> +
>>>>>> +        /* RAW bypass disables scaling and cropping. */
>>>>>> +        crop->top = compose->top = 0;
>>>>>> +        crop->left = compose->left = 0;
>>>>>> +        fmt->width = crop->width = compose->width = sink_fmt->width;
>>>>>> +        fmt->height = crop->height = compose->height = sink_fmt->height;
>>>> I don't think this is right. This function sets the format on the source
>>>> pad. Subdevs should propagate formats from the sink to the source, not
>>>> the other way around.
>>>>
>>>> The only parameter that can be modified on the source pad (as far as I
>>>> understand) is the media bus code. In the bypass path, I understand it's
>>>> fixed, while in the other path, you can select between RGB and YUV. I
>>>> think the following code is what you need to implement this function.
>>>>
>>>> static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
>>>>                         struct v4l2_subdev_state *state,
>>>>                         struct v4l2_subdev_format *format)
>>>> {
>>>>      struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>>                              sd);
>>>>      struct v4l2_mbus_framefmt *fmt;
>>>>
>>>>      fmt = v4l2_subdev_state_get_format(state, format->pad);
>>>>
>>>>      /* In the non-bypass path the output format can be selected. */
>>>>      if (mali_c55_rzr_get_active_sink(state) == MALI_C55_RZR_SINK_PAD) {
>>>>          unsigned int i;
>>>>
>>>>          fmt->code = format->format.code;
>>>>
>>>>          for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
>>>>              if (fmt->code == rzr_non_bypass_src_fmts[i])
>>>>                  break;
>>>>          }
>>>>
>>>>          if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts))
>>>>              fmt->code = rzr_non_bypass_src_fmts[0];
>>>>      }
>>>>
>>>>      format->format = *fmt;
>>>>
>>>>      return 0;
>>>> }
>>>>
>>>>>> +
>>>>>> +        *v4l2_subdev_state_get_format(state,
>>>>>> +                          MALI_C55_RZR_SOURCE_PAD) = *fmt;
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* Regular processing pipe. */
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
>>>>>> +        if (fmt->code == rzr_non_bypass_src_fmts[i])
>>>>>> +            break;
>>>>>> +    }
>>>>>> +
>>>>>> +    if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
>>>>>> +        dev_dbg(rzr->mali_c55->dev,
>>>>>> +            "Unsupported mbus code 0x%x: using default\n",
>>>>>> +            fmt->code);
>>>> I think you can drop this message.
>>>>
>>>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The source pad format size comes directly from the sink pad
>>>>>> +     * compose rectangle.
>>>>>> +     */
>>>>>> +    fmt->width = compose->width;
>>>>>> +    fmt->height = compose->height;
>>>>>> +
>>>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
>>>>>> +                struct v4l2_subdev_state *state,
>>>>>> +                struct v4l2_subdev_format *format)
>>>>>> +{
>>>>>> +    /*
>>>>>> +     * On sink pads fmt is either fixed for the 'regular' processing
>>>>>> +     * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
>>>>>> +     * pad.
>>>>>> +     *
>>>>>> +     * On source pad sizes are the result of crop+compose on the sink
>>>>>> +     * pad sizes, while the format depends on the active route.
>>>>>> +     */
>>>>>> +
>>>>>> +    if (format->pad != MALI_C55_RZR_SOURCE_PAD)
>>>>>> +        return mali_c55_rzr_set_sink_fmt(sd, state, format);
>>>>>> +
>>>>>> +    return mali_c55_rzr_set_source_fmt(sd, state, format);
>>>> Nitpicking,
>>>>
>>>>      if (format->pad == MALI_C55_RZR_SOURCE_PAD)
>>>>          return mali_c55_rzr_set_source_fmt(sd, state, format);
>>>>
>>>>      return mali_c55_rzr_set_sink_fmt(sd, state, format);
>>>>
>>>> to match SOURCE_PAD and source_fmt.
>>>>
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
>>>>>> +                      struct v4l2_subdev_state *state,
>>>>>> +                      struct v4l2_subdev_selection *sel)
>>>>>> +{
>>>>>> +    if (sel->pad != MALI_C55_RZR_SINK_PAD)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    if (sel->target != V4L2_SEL_TGT_CROP &&
>>>>>> +        sel->target != V4L2_SEL_TGT_COMPOSE)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    sel->r = sel->target == V4L2_SEL_TGT_CROP
>>>>>> +           ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
>>>>>> +           : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
>>>>>> +                      struct v4l2_subdev_state *state,
>>>>>> +                      struct v4l2_subdev_selection *sel)
>>>>>> +{
>>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>>>> +                            sd);
>>>>>> +    struct v4l2_mbus_framefmt *source_fmt;
>>>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
>>>>>> +    struct v4l2_rect *crop, *compose;
>>>>>> +
>>>>>> +    if (sel->pad != MALI_C55_RZR_SINK_PAD)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    if (sel->target != V4L2_SEL_TGT_CROP &&
>>>>>> +        sel->target != V4L2_SEL_TGT_COMPOSE)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    source_fmt = v4l2_subdev_state_get_format(state,
>>>>>> +                          MALI_C55_RZR_SOURCE_PAD);
>>>>>> +    sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
>>>>>> +    compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
>>>>>> +
>>>>>> +    /* RAW bypass disables crop/scaling. */
>>>>>> +    if (mali_c55_format_is_raw(source_fmt->code)) {
>>>>>> +        crop->top = compose->top = 0;
>>>>>> +        crop->left = compose->left = 0;
>>>>>> +        crop->width = compose->width = sink_fmt->width;
>>>>>> +        crop->height = compose->height = sink_fmt->height;
>>>>>> +
>>>>>> +        sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* During streaming, it is allowed to only change the crop rectangle. */
>>>>>> +    if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +     /*
>>>>>> +      * Update the desired target and then clamp the crop rectangle to the
>>>>>> +      * sink format sizes and the compose size to the crop sizes.
>>>>>> +      */
>>>>>> +    if (sel->target == V4L2_SEL_TGT_CROP)
>>>>>> +        *crop = sel->r;
>>>>>> +    else
>>>>>> +        *compose = sel->r;
>>>>>> +
>>>>>> +    clamp_t(unsigned int, crop->left, 0, sink_fmt->width);
>>>>>> +    clamp_t(unsigned int, crop->top, 0, sink_fmt->height);
>>>>>> +    clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
>>>>>> +        sink_fmt->width - crop->left);
>>>>>> +    clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
>>>>>> +        sink_fmt->height - crop->top);
>>>>>> +
>>>>>> +    if (rzr->streaming) {
>>>>>> +        /*
>>>>>> +         * Apply at runtime a crop rectangle on the resizer's sink only
>>>>>> +         * if it doesn't require re-programming the scaler output sizes
>>>>>> +         * as it would require changing the output buffer sizes as well.
>>>>>> +         */
>>>>>> +        if (sel->r.width < compose->width ||
>>>>>> +            sel->r.height < compose->height)
>>>>>> +            return -EINVAL;
>>>>>> +
>>>>>> +        *crop = sel->r;
>>>>>> +        mali_c55_rzr_program(rzr, state);
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    compose->left = 0;
>>>>>> +    compose->top = 0;
>>>>>> +    clamp_t(unsigned int, compose->left, 0, sink_fmt->width);
>>>>>> +    clamp_t(unsigned int, compose->top, 0, sink_fmt->height);
>>>>>> +    clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
>>>>>> +    clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
>>>>>> +
>>>>>> +    sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
>>>>>> +                    struct v4l2_subdev_state *state,
>>>>>> +                    enum v4l2_subdev_format_whence which,
>>>>>> +                    struct v4l2_subdev_krouting *routing)
>>>>>> +{
>>>>>> +    if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
>>>>>> +        media_entity_is_streaming(&sd->entity))
>>>>>> +        return -EBUSY;
>>>>>> +
>>>>>> +    return __mali_c55_rzr_set_routing(sd, state, routing);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
>>>>>> +    .enum_mbus_code        = mali_c55_rzr_enum_mbus_code,
>>>>>> +    .enum_frame_size    = mali_c55_rzr_enum_frame_size,
>>>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
>>>>>> +    .set_fmt        = mali_c55_rzr_set_fmt,
>>>>>> +    .get_selection        = mali_c55_rzr_get_selection,
>>>>>> +    .set_selection        = mali_c55_rzr_set_selection,
>>>>>> +    .set_routing        = mali_c55_rzr_set_routing,
>>>>>> +};
>>>>>> +
>>>>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
>>>> Could this be handled through the .enable_streams() and
>>>> .disable_streams() operations ? They ensure that the stream state stored
>>>> internal is correct. That may not matter much today, but I think it will
>>>> become increasingly important in the future for the V4L2 core.
>>>>
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>>>> +    struct v4l2_subdev *sd = &rzr->sd;
>>>>>> +    struct v4l2_subdev_state *state;
>>>>>> +    unsigned int sink_pad;
>>>>>> +
>>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
>>>>>> +
>>>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
>>>>>> +    if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>>>>> +        /* Bypass FR pipe processing if the bypass route is active. */
>>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>>>>> + MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
>>>>>> + MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
>>>>>> +        goto unlock_state;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* Disable bypass and use regular processing. */
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>>>>> +                 MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
>>>>>> +    mali_c55_rzr_program(rzr, state);
>>>>>> +
>>>>>> +unlock_state:
>>>>>> +    rzr->streaming = true;
>>>> And hopefully you'll be able to replace this with
>>>> v4l2_subdev_is_streaming(), introduced in "[PATCH v6 00/11] media:
>>>> subdev: Improve stream enable/disable machinery" (Sakari has sent a pull
>>>> request for v6.11 yesterday).
>>>>
>>>>>> +    v4l2_subdev_unlock_state(state);
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
>>>>>> +{
>>>>>> +    struct v4l2_subdev *sd = &rzr->sd;
>>>>>> +    struct v4l2_subdev_state *state;
>>>>>> +
>>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
>>>>>> +    rzr->streaming = false;
>>>>>> +    v4l2_subdev_unlock_state(state);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
>>>>>> +    .pad    = &mali_c55_resizer_pad_ops,
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
>>>>>> +                   struct v4l2_subdev_state *state)
>>>>>> +{
>>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>>>> +                            sd);
>>>>>> +    struct v4l2_subdev_krouting routing = { };
>>>>>> +    struct v4l2_subdev_route *routes;
>>>>>> +    unsigned int i;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
>>>>>> +    if (!routes)
>>>>>> +        return -ENOMEM;
>>>>>> +
>>>>>> +    for (i = 0; i < rzr->num_routes; ++i) {
>>>>>> +        struct v4l2_subdev_route *route = &routes[i];
>>>>>> +
>>>>>> +        route->sink_pad = i
>>>>>> +                ? MALI_C55_RZR_SINK_BYPASS_PAD
>>>>>> +                : MALI_C55_RZR_SINK_PAD;
>>>>>> +        route->source_pad = MALI_C55_RZR_SOURCE_PAD;
>>>>>> +        if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
>>>>>> +            route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
>>>>>> +    }
>>>>>> +
>>>>>> +    routing.num_routes = rzr->num_routes;
>>>>>> +    routing.routes = routes;
>>>>>> +
>>>>>> +    ret = __mali_c55_rzr_set_routing(sd, state, &routing);
>>>>>> +    kfree(routes);
>>>>>> +
>>>>>> +    return ret;
>>>> I think this could be simplified.
>>>>
>>>>      struct v4l2_subdev_route routes[2] = {
>>>>          {
>>>>              .sink_pad = MALI_C55_RZR_SINK_PAD,
>>>>              .source_pad = MALI_C55_RZR_SOURCE_PAD,
>>>>              .flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
>>>>          }, {
>>>>              .sink_pad = MALI_C55_RZR_SINK_BYPASS_PAD,
>>>>              .source_pad = MALI_C55_RZR_SOURCE_PAD,
>>>>          },
>>>>      };
>>>>      struct v4l2_subdev_krouting routing = {
>>>>          .num_routes = rzr->num_routes,
>>>>          .routes = routes,
>>>>      };
>>>>
>>>>      return __mali_c55_rzr_set_routing(sd, state, &routing);
>>>>
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
>>>>>> +    .init_state = mali_c55_rzr_init_state,
>>>>>> +};
>>>>>> +
>>>>>> +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
>>>>>> +                          unsigned int index)
>>>>>> +{
>>>>>> +    const unsigned int scaler_filt_coefmem_addrs[][2] = {
>>>>>> +        [MALI_C55_RZR_FR] = {
>>>>>> +            0x034A8, /* hfilt */
>>>>>> +            0x044A8  /* vfilt */
>>>>> Lowercase hex constants.
>>>> And addresses belong to the mali-c55-registers.h file.
>>>>
>>>>>> +        },
>>>>>> +        [MALI_C55_RZR_DS] = {
>>>>>> +            0x014A8, /* hfilt */
>>>>>> +            0x024A8  /* vfilt */
>>>>>> +        },
>>>>>> +    };
>>>>>> +    unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
>>>>>> +    unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
>>>>>> +    unsigned int i, j;
>>>>>> +
>>>>>> +    for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
>>>>>> +        for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
>>>>>> +            mali_c55_write(mali_c55, haddr,
>>>>>> + mali_c55_scaler_h_filter_coefficients[i][j]);
>>>>>> +            mali_c55_write(mali_c55, vaddr,
>>>>>> + mali_c55_scaler_v_filter_coefficients[i][j]);
>>>>>> +
>>>>>> +            haddr += sizeof(u32);
>>>>>> +            vaddr += sizeof(u32);
>>>>>> +        }
>>>>>> +    }
>>>> How about memcpy_toio() ? I suppose this function isn't
>>>> performance sensitive, so maybe usage of mali_c55_write() is better from
>>>> a consistency point of view.
>>>>
>>>>>> +}
>>>>>> +
>>>>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
>>>> Moving the inner content to a separate mali_c55_register_resizer()
>>>> function would increase readability I think, and remove usage of gotos.
>>>> I would probably do the same for unregistration too, for consistency.
>>>>
>>>>>> +        struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
>>>>>> +        struct v4l2_subdev *sd = &rzr->sd;
>>>>>> +        unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
>>>>>> +
>>>>>> +        rzr->id = i;
>>>>>> +        rzr->streaming = false;
>>>>>> +
>>>>>> +        if (rzr->id == MALI_C55_RZR_FR)
>>>>>> +            rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
>>>>>> +        else
>>>>>> +            rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
>>>>>> +
>>>>>> +        mali_c55_resizer_program_coefficients(mali_c55, i);
>>>> Should this be done at stream start, given that power may be cut off
>>>> between streaming sessions ?
>>>>
>>>>>> +
>>>>>> +        v4l2_subdev_init(sd, &mali_c55_resizer_ops);
>>>>>> +        sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
>>>>>> +                 | V4L2_SUBDEV_FL_STREAMS;
>>>>>> +        sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
>>>>>> +        sd->internal_ops = &mali_c55_resizer_internal_ops;
>>>>>> +        snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
>>>>          snprintf(sd->name, ARRAY_SIZE(sd->name), "%s resizer %s",
>>>>
>>>> and drop the "resizer " prefix from mali_c55_resizer_names. You can also
>>>> make mali_c55_resizer_names a local static const variable.
>>>>
>>>>>> +             MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
>>>>>> +
>>>>>> +        rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
>>>>>> +        rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
>>>>>> +
>>>>>> +        /* Only the FR pipe has a bypass pad. */
>>>>>> +        if (rzr->id == MALI_C55_RZR_FR) {
>>>>>> + rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
>>>>>> +                            MEDIA_PAD_FL_SINK;
>>>>>> +            rzr->num_routes = 2;
>>>>>> +        } else {
>>>>>> +            num_pads -= 1;
>>>>>> +            rzr->num_routes = 1;
>>>>>> +        }
>>>>>> +
>>>>>> +        ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
>>>>>> +        if (ret)
>>>>>> +            return ret;
>>>>>> +
>>>>>> +        ret = v4l2_subdev_init_finalize(sd);
>>>>>> +        if (ret)
>>>>>> +            goto err_cleanup;
>>>>>> +
>>>>>> +        ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>>>>> +        if (ret)
>>>>>> +            goto err_cleanup;
>>>>>> +
>>>>>> +        rzr->mali_c55 = mali_c55;
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_cleanup:
>>>>>> +    for (; i >= 0; --i) {
>>>>>> +        struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
>>>>>> +        struct v4l2_subdev *sd = &rzr->sd;
>>>>>> +
>>>>>> +        v4l2_subdev_cleanup(sd);
>>>>>> +        media_entity_cleanup(&sd->entity);
>>>>>> +    }
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
>>>>>> +        struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
>>>>>> +
>>>>>> +        if (!resizer->mali_c55)
>>>>>> +            continue;
>>>>>> +
>>>>>> +        v4l2_device_unregister_subdev(&resizer->sd);
>>>>>> +        v4l2_subdev_cleanup(&resizer->sd);
>>>>>> +        media_entity_cleanup(&resizer->sd.entity);
>>>>>> +    }
>>>>>> +}
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>>>>> new file mode 100644
>>>>>> index 000000000000..c7e699741c6d
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>>>>> @@ -0,0 +1,402 @@
>>>>>> +// SPDX-License-Identifier: GPL-2.0
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Test pattern generator
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#include <linux/minmax.h>
>>>>>> +#include <linux/string.h>
>>>>>> +
>>>>>> +#include <media/media-entity.h>
>>>>>> +#include <media/v4l2-ctrls.h>
>>>>>> +#include <media/v4l2-subdev.h>
>>>>>> +
>>>>>> +#include "mali-c55-common.h"
>>>>>> +#include "mali-c55-registers.h"
>>>>>> +
>>>>>> +#define MALI_C55_TPG_SRC_PAD        0
>>>>>> +#define MALI_C55_TPG_FIXED_HBLANK    0x20
>>>>>> +#define MALI_C55_TPG_MAX_VBLANK        0xFFFF
>>>>> Lowercase hex constants.
>>>>>
>>>>>> +#define MALI_C55_TPG_PIXEL_RATE        100000000
>>>>> This should be exposed to applications using the V4L2_CID_PIXEL_RATE
>>>>> control (read-only).
>>>>>
>>>>>> +
>>>>>> +static const char * const mali_c55_tpg_test_pattern_menu[] = {
>>>>>> +    "Flat field",
>>>>>> +    "Horizontal gradient",
>>>>>> +    "Vertical gradient",
>>>>>> +    "Vertical bars",
>>>>>> +    "Arbitrary rectangle",
>>>>>> +    "White frame on black field"
>>>>>> +};
>>>>>> +
>>>>>> +static const u32 mali_c55_tpg_mbus_codes[] = {
>>>>>> +    MEDIA_BUS_FMT_SRGGB20_1X20,
>>>>>> +    MEDIA_BUS_FMT_RGB202020_1X60,
>>>>>> +};
>>>>>> +
>>>>>> +static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
>>>>>> +                       int *def_vblank, int *min_vblank)
>>>>> unsigned int ?
>>>>>
>>>>>> +{
>>>>>> +    unsigned int hts;
>>>>>> +    int tgt_fps;
>>>>>> +    int vblank;
>>>>>> +
>>>>>> +    hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The ISP has minimum vertical blanking requirements that must be
>>>>>> +     * adhered to by the TPG. The minimum is a function of the Iridix blocks
>>>>>> +     * clocking requirements and the width of the image and horizontal
>>>>>> +     * blanking, but if we assume the worst case iVariance and sVariance
>>>>>> +     * values then it boils down to the below.
>>>>>> +     */
>>>>>> +    *min_vblank = 15 + (120500 / hts);
>>>>> I wonder if this should round up.
>>>>>
>>>>>> +
>>>>>> +    /*
>>>>>> +     * We need to set a sensible default vblank for whatever format height
>>>>>> +     * we happen to be given from set_fmt(). This function just targets
>>>>>> +     * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
>>>>>> +     * If we can't get 5fps we'll take whatever the minimum vblank gives us.
>>>>>> +     */
>>>>>> +    tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
>>>>>> +
>>>>>> +    if (tgt_fps < 5)
>>>>>> +        vblank = *min_vblank;
>>>>>> +    else
>>>>>> +        vblank = MALI_C55_TPG_PIXEL_RATE / hts
>>>>>> +               / max(rounddown(tgt_fps, 15), 5);
>>>>>> +
>>>>>> +    *def_vblank = ALIGN_DOWN(vblank, 2) - format->height;
>>>>> "vblank = vblank - height" doesn't seem right. The "else" branch stores
>>>>> a vts in vblank, which doesn't seem right either. Maybe you meant
>>>>> something like
>>>>>
>>>>>      if (tgt_fps < 5)
>>>>>          def_vts = *min_vblank + format->height;
>>>>>      else
>>>>>          def_vts = MALI_C55_TPG_PIXEL_RATE / hts
>>>>>              / max(rounddown(tgt_fps, 15), 5);
>>>>>
>>>>>      *def_vblank = ALIGN_DOWN(def_vts, 2) - format->height;
>>>>>
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
>>>>>> +{
>>>>>> +    struct mali_c55_tpg *tpg = container_of(ctrl->handler,
>>>>>> +                        struct mali_c55_tpg,
>>>>>> +                        ctrls.handler);
>>>>>> +    struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
>>>>>> +
>>>>> Should you return here if the pipeline isn't streaming ?
>>>>>
>>>>>> +    switch (ctrl->id) {
>>>>>> +    case V4L2_CID_TEST_PATTERN:
>>>>>> +        mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
>>>>>> +                   ctrl->val);
>>>>>> +        break;
>>>>>> +    case V4L2_CID_VBLANK:
>>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
>>>>>> +                     MALI_C55_REG_VBLANK_MASK, ctrl->val);
>>>>>> +        break;
>>>>>> +    default:
>>>>>> +        dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
>>>>>> +        return -EINVAL;
>>>>> Can this happen ?
>>>>>
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
>>>>>> +    .s_ctrl = &mali_c55_tpg_s_ctrl,
>>>>>> +};
>>>>>> +
>>>>>> +static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
>>>>>> +                   struct v4l2_subdev *sd)
>>>>>> +{
>>>>>> +    struct v4l2_subdev_state *state;
>>>>>> +    struct v4l2_mbus_framefmt *fmt;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * hblank needs setting, but is a read-only control and thus won't be
>>>>>> +     * called during __v4l2_ctrl_handler_setup(). Do it here instead.
>>>>>> +     */
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
>>>>>> +                 MALI_C55_REG_HBLANK_MASK,
>>>>>> +                 MALI_C55_TPG_FIXED_HBLANK);
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>>>>> +                 MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
>>>>>> +
>>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
>>>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>>>> +
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>>>>> +                 MALI_C55_TEST_PATTERN_RGB_MASK,
>>>>>> +                 fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
>>>>>> +                      0x01 : 0x0);
>>>>>> +
>>>>>> +    v4l2_subdev_unlock_state(state);
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
>>>>>> +{
>>>>>> +    struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
>>>>>> +    struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
>>>>>> +
>>>>>> +    if (!enable) {
>>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>>>>> +                MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
>>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>>>>> +                MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * One might reasonably expect the framesize to be set here
>>>>>> +     * given it's configurable in .set_fmt(), but it's done in the
>>>>>> +     * ISP subdevice's stream on func instead, as the same register
>>>>> s/func/function/
>>>>>
>>>>>> +     * is also used to indicate the size of the data coming from the
>>>>>> +     * sensor.
>>>>>> +     */
>>>>>> +    mali_c55_tpg_configure(mali_c55, sd);
>>>>>      mali_c55_tpg_configure(tpg);
>>>>>
>>>>>> + __v4l2_ctrl_handler_setup(sd->ctrl_handler);
>>>>>> +
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>>>>> +                 MALI_C55_TEST_PATTERN_ON_OFF,
>>>>>> +                 MALI_C55_TEST_PATTERN_ON_OFF);
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>>>>> +                 MALI_C55_REG_GEN_VIDEO_ON_MASK,
>>>>>> +                 MALI_C55_REG_GEN_VIDEO_ON_MASK);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
>>>>>> +    .s_stream = &mali_c55_tpg_s_stream,
>>>>> Can we use .enable_streams() and .disable_streams() ?
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>> +                       struct v4l2_subdev_state *state,
>>>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
>>>>>> +{
>>>>>> +    if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    code->code = mali_c55_tpg_mbus_codes[code->index];
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
>>>>>> +                    struct v4l2_subdev_state *state,
>>>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
>>>>>> +{
>>>>> You sohuld verify here that fse->code is a supported value and return
>>>>> -EINVAL otherwise.
>>>>>
>>>>>> +    if (fse->index > 0 || fse->pad > sd->entity.num_pads)
>>>>> Drop the pad check, it's done in the subdev core already.
>>>>>
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
>>>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
>>>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
>>>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
>>>>>> +                struct v4l2_subdev_state *state,
>>>>>> +                struct v4l2_subdev_format *format)
>>>>>> +{
>>>>>> +    struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
>>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
>>>>>> +    int vblank_def, vblank_min;
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>>>>>> +        if (fmt->code == mali_c55_tpg_mbus_codes[i])
>>>>>> +            break;
>>>>>> +    }
>>>>>> +
>>>>>> +    if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The TPG says that the test frame timing generation logic expects a
>>>>>> +     * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
>>>>>> +     * handle anything smaller than 128x128 it seems pointless to allow a
>>>>>> +     * smaller frame.
>>>>>> +     */
>>>>>> +    clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
>>>>>> +        MALI_C55_MAX_WIDTH);
>>>>>> +    clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
>>>>>> +        MALI_C55_MAX_HEIGHT);
>>>>>> +
>>>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;
>>>>> You're allowing userspace to set fmt->field, as well as all the
>>>>> colorspace parameters, to random values. I would instead do something
>>>>> like
>>>>>
>>>>>      for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>>>>>          if (format->format.code == mali_c55_tpg_mbus_codes[i])
>>>>>              break;
>>>>>      }
>>>>>
>>>>>      if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>>>>          format->format.code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>>>
>>>>>      format->format.width = clamp(format->format.width,
>>>>>                       MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>>>>      format->format.height = clamp(format->format.height,
>>>>>                        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>>>>
>>>>>      fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>>>      fmt->code = format->format.code;
>>>>>      fmt->width = format->format.width;
>>>>>      fmt->height = format->format.height;
>>>>>
>>>>>      format->format = *fmt;
>>>>>
>>>>> Alternatively (which I think I like better),
>>>>>
>>>>>      fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>>>
>>>>>      fmt->code = format->format.code;
>>>>>
>>>>>      for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>>>>>          if (fmt->code == mali_c55_tpg_mbus_codes[i])
>>>>>              break;
>>>>>      }
>>>>>
>>>>>      if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>>>>          fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>>>
>>>>>      fmt->width = clamp(format->format.width,
>>>>>                 MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>>>>      fmt->height = clamp(format->format.height,
>>>>>                  MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>>>>
>>>>>      format->format = *fmt;
>>>>>
>>>>>> +
>>>>>> +    if (format->which == V4L2_SUBDEV_FORMAT_TRY)
>>>>>> +        return 0;
>>>>>> +
>>>>>> +    __mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
>>>>>> +    __v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
>>>>>> +                 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
>>>>>> +    __v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);
>>>>> Move those three calls to a separate function, it will be reused below.
>>>>> I'd name is mali_c55_tpg_update_vblank(). You can fold
>>>>> __mali_c55_tpg_calc_vblank() in it.
>>>>>
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
>>>>>> +    .enum_mbus_code        = mali_c55_tpg_enum_mbus_code,
>>>>>> +    .enum_frame_size    = mali_c55_tpg_enum_frame_size,
>>>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
>>>>>> +    .set_fmt        = mali_c55_tpg_set_fmt,
>>>>>> +};
>>>>>> +
>>>>>> +static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
>>>>>> +    .video    = &mali_c55_tpg_video_ops,
>>>>>> +    .pad    = &mali_c55_tpg_pad_ops,
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
>>>>>> +                   struct v4l2_subdev_state *sd_state)
>>>>> You name this variable state in every other subdev operation handler.
>>>>>
>>>>>> +{
>>>>>> +    struct v4l2_mbus_framefmt *fmt =
>>>>>> +        v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
>>>>>> +
>>>>>> +    fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>>>> +    fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>>>> +    fmt->field = V4L2_FIELD_NONE;
>>>>>> +    fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>>> Initialize the colorspace fields too.
>>>>>
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
>>>>>> +    .init_state = mali_c55_tpg_init_state,
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
>>>>>> +    struct v4l2_subdev *sd = &mali_c55->tpg.sd;
>>>>>> +    struct v4l2_mbus_framefmt *format;
>>>>>> +    struct v4l2_subdev_state *state;
>>>>>> +    int vblank_def, vblank_min;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
>>>>>> +    format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>>>> +
>>>>>> +    ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);
>>>>> You have 3 controls.
>>>>>
>>>>>> +    if (ret)
>>>>>> +        goto err_unlock;
>>>>>> +
>>>>>> +    ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
>>>>>> +                &mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
>>>>>> +                ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
>>>>>> +                0, 3, mali_c55_tpg_test_pattern_menu);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * We fix hblank at the minimum allowed value and control framerate
>>>>>> +     * solely through the vblank control.
>>>>>> +     */
>>>>>> +    ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
>>>>>> +                &mali_c55_tpg_ctrl_ops,
>>>>>> +                V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
>>>>>> +                MALI_C55_TPG_FIXED_HBLANK, 1,
>>>>>> +                MALI_C55_TPG_FIXED_HBLANK);
>>>>>> +    if (ctrls->hblank)
>>>>>> +        ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
>>>>>> +
>>>>>> +    __mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);
>>>>> Drop this and initialize the control with default values. You can then
>>>>> update the value by calling mali_c55_tpg_update_vblank() in
>>>>> mali_c55_register_tpg().
>>>>>
>>>>> The reason is to share the same mutex between the control handler and
>>>>> the subdev active state without having to add a separate mutex in the
>>>>> mali_c55_tpg structure. The simplest way to do so is to initialize the
>>>>> controls first, set sd->state_lock to point to the control handler lock,
>>>>> and call v4l2_subdev_init_finalize() as the last step. As a consequence,
>>>>> you can't access the active state when initializing controls.
>>>>>
>>>>> You can alternatively keep the lock in mali_c55_tpg and set
>>>>> sd->state_lock to point to it, but I think that's more complex.
>>>>>
>>>>>> +    ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
>>>>>> +                      &mali_c55_tpg_ctrl_ops,
>>>>>> +                      V4L2_CID_VBLANK, vblank_min,
>>>>>> +                      MALI_C55_TPG_MAX_VBLANK, 1,
>>>>>> +                      vblank_def);
>>>>>> +
>>>>>> +    if (ctrls->handler.error) {
>>>>>> +        dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
>>>>>> +        ret = ctrls->handler.error;
>>>>>> +        goto err_free_handler;
>>>>>> +    }
>>>>>> +
>>>>>> +    ctrls->handler.lock = &mali_c55->tpg.lock;
>>>>> Drop this and drop the mutex. The control handler will use its internal
>>>>> mutex.
>>>>>
>>>>>> +    mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
>>>>>> +
>>>>>> +    v4l2_subdev_unlock_state(state);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_free_handler:
>>>>>> +    v4l2_ctrl_handler_free(&ctrls->handler);
>>>>>> +err_unlock:
>>>>>> +    v4l2_subdev_unlock_state(state);
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_tpg *tpg = &mali_c55->tpg;
>>>>>> +    struct v4l2_subdev *sd = &tpg->sd;
>>>>>> +    struct media_pad *pad = &tpg->pad;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    mutex_init(&tpg->lock);
>>>>>> +
>>>>>> +    v4l2_subdev_init(sd, &mali_c55_tpg_ops);
>>>>>> +    sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
>>>>>> +    sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
>>>>> Should we introduce a TPG function ?
>>>>>
>>>>>> +    sd->internal_ops = &mali_c55_tpg_internal_ops;
>>>>>> +    strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
>>>>>> +
>>>>>> +    pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;
>>>>> I don't think MEDIA_PAD_FL_MUST_CONNECT is right.
>>>>>
>>>>>> +    ret = media_entity_pads_init(&sd->entity, 1, pad);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev,
>>>>>> +            "Failed to initialize media entity pads\n");
>>>>>> +        goto err_destroy_mutex;
>>>>>> +    }
>>>>>> +
>>>>>      sd->state_lock = sd->ctrl_handler->lock;
>>>>>
>>>>> to use the same lock for the controls and the active state. You need to
>>>>> move this line and the v4l2_subdev_init_finalize() call after
>>>>> mali_c55_tpg_init_controls() to get the control handler lock initialized
>>>>> first.
>>>>>
>>>>>> +    ret = v4l2_subdev_init_finalize(sd);
>>>>>> +    if (ret)
>>>>>> +        goto err_cleanup_media_entity;
>>>>>> +
>>>>>> +    ret = mali_c55_tpg_init_controls(mali_c55);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev,
>>>>>> +            "Error initialising controls\n");
>>>>>> +        goto err_cleanup_subdev;
>>>>>> +    }
>>>>>> +
>>>>>> +    ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
>>>>>> +        goto err_free_ctrl_handler;
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * By default the colour settings lead to a very dim image that is
>>>>>> +     * nearly indistinguishable from black on some monitor settings. Ramp
>>>>>> +     * them up a bit so the image is brighter.
>>>>>> +     */
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
>>>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
>>>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
>>>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
>>>>>> +
>>>>>> +    tpg->mali_c55 = mali_c55;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_free_ctrl_handler:
>>>>>> +    v4l2_ctrl_handler_free(&tpg->ctrls.handler);
>>>>>> +err_cleanup_subdev:
>>>>>> +    v4l2_subdev_cleanup(sd);
>>>>>> +err_cleanup_media_entity:
>>>>>> +    media_entity_cleanup(&sd->entity);
>>>>>> +err_destroy_mutex:
>>>>>> +    mutex_destroy(&tpg->lock);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_tpg *tpg = &mali_c55->tpg;
>>>>>> +
>>>>>> +    if (!tpg->mali_c55)
>>>>>> +        return;
>>>>>> +
>>>>>> +    v4l2_device_unregister_subdev(&tpg->sd);
>>>>>> +    v4l2_subdev_cleanup(&tpg->sd);
>>>>>> +    media_entity_cleanup(&tpg->sd.entity);
>>>>>> +    v4l2_ctrl_handler_free(&tpg->ctrls.handler);
>>>>> Free the control handler just after v4l2_device_unregister_subdev() to
>>>>> match the order in mali_c55_register_tpg().
>>>>>
>>>>>> +    mutex_destroy(&tpg->lock);
>>>>>> +}

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Dan Scally]

Hi Laurent - sorry, responding in multiple emails again...

On 20/06/2024 16:23, Laurent Pinchart wrote:
> Hi Dan,
>
> On Thu, Jun 20, 2024 at 03:49:23PM +0100, Daniel Scally wrote:
>> On 20/06/2024 15:33, Dan Scally wrote:
>>> On 30/05/2024 22:43, Laurent Pinchart wrote:
>>>> And now the second part of the review, addressing mali-c55-capture.c and
>>>> mali-c55-resizer.c. I've reviewed the code from the bottom up, so some
>>>> messages may be repeated in an order that seems weird. Sorry about that.
>>>>
>>>> On Thu, May 30, 2024 at 03:15:10AM +0300, Laurent Pinchart wrote:
>>>>> On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
>>>>>> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
>>>>>> V4L2 and Media Controller compliant and creates subdevices to manage
>>>>>> the ISP itself, its internal test pattern generator as well as the
>>>>>> crop, scaler and output format functionality for each of its two
>>>>>> output devices.
>>>>>>
>>>>>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
>>>>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>>>>>> ---
>>>>>> Changes in v5:
>>>>>>
>>>>>>      - Reworked input formats - previously we allowed representing input data
>>>>>>        as any 8-16 bit format. Now we only allow input data to be represented
>>>>>>        by the new 20-bit bayer formats, which is corrected to the equivalent
>>>>>>        16-bit format in RAW bypass mode.
>>>>>>      - Stopped bypassing blocks that we haven't added supporting parameters
>>>>>>        for yet.
>>>>>>      - Addressed most of Sakari's comments from the list
>>>>>>
>>>>>> Changes not yet made in v5:
>>>>>>
>>>>>>      - The output pipelines can still be started and stopped independently of
>>>>>>        one another - I'd like to discuss that more.
>>>>>>      - the TPG subdev still uses .s_stream() - I need to rebase onto a tree
>>>>>>        with working .enable_streams() for a single-source-pad subdevice.
>>>>>>
>>>>>> Changes in v4:
>>>>>>
>>>>>>      - Reworked mali_c55_update_bits() to internally perform the bit-shift
>>>>> I really don't like that, it makes the code very confusing, even more so
>>>>> as it differs from regmap_update_bits().
>>>>>
>>>>> Look at this for instance:
>>>>>
>>>>>      mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>>>                   MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
>>>>>                   MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
>>>>>
>>>>> It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
>>>>> BIT(0).
>>>>>
>>>>> Sorry, I know it will be painful, but this change needs to be reverted.
>>>>>
>>>>>>      - Reworked the resizer to allow cropping during streaming
>>>>>>      - Fixed a bug in NV12 output
>>>>>>
>>>>>> Changes in v3:
>>>>>>
>>>>>>      - Mostly minor fixes suggested by Sakari
>>>>>>      - Fixed the sequencing of vb2 buffers to be synchronised across the two
>>>>>>        capture devices.
>>>>>>
>>>>>> Changes in v2:
>>>>>>
>>>>>>      - Clock handling
>>>>>>      - Fixed the warnings raised by the kernel test robot
>>>>>>
>>>>>>    drivers/media/platform/Kconfig                |   1 +
>>>>>>    drivers/media/platform/Makefile               |   1 +
>>>>>>    drivers/media/platform/arm/Kconfig            |   5 +
>>>>>>    drivers/media/platform/arm/Makefile           |   2 +
>>>>>>    drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
>>>>>>    drivers/media/platform/arm/mali-c55/Makefile  |   9 +
>>>>>>    .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
>>>>>>    .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
>>>>>>    .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
>>>>>>    .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
>>>>>>    .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
>>>>>>    .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
>>>>>>    .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
>>>>>>    .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
>>>>>>    14 files changed, 4452 insertions(+)
>>>>>>    create mode 100644 drivers/media/platform/arm/Kconfig
>>>>>>    create mode 100644 drivers/media/platform/arm/Makefile
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>>>> I've skipped review of capture.c and resizer.c as I already have plenty
>>>>> of comments for the other files, and it's getting late. I'll try to
>>>>> review the rest tomorrow.
>>>>>
>>>>>> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
>>>>>> index 2d79bfc68c15..c929169766aa 100644
>>>>>> --- a/drivers/media/platform/Kconfig
>>>>>> +++ b/drivers/media/platform/Kconfig
>>>>>> @@ -65,6 +65,7 @@ config VIDEO_MUX
>>>>>>    source "drivers/media/platform/allegro-dvt/Kconfig"
>>>>>>    source "drivers/media/platform/amlogic/Kconfig"
>>>>>>    source "drivers/media/platform/amphion/Kconfig"
>>>>>> +source "drivers/media/platform/arm/Kconfig"
>>>>>>    source "drivers/media/platform/aspeed/Kconfig"
>>>>>>    source "drivers/media/platform/atmel/Kconfig"
>>>>>>    source "drivers/media/platform/broadcom/Kconfig"
>>>>>> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
>>>>>> index da17301f7439..9a647abd5218 100644
>>>>>> --- a/drivers/media/platform/Makefile
>>>>>> +++ b/drivers/media/platform/Makefile
>>>>>> @@ -8,6 +8,7 @@
>>>>>>    obj-y += allegro-dvt/
>>>>>>    obj-y += amlogic/
>>>>>>    obj-y += amphion/
>>>>>> +obj-y += arm/
>>>>>>    obj-y += aspeed/
>>>>>>    obj-y += atmel/
>>>>>>    obj-y += broadcom/
>>>>>> diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
>>>>>> new file mode 100644
>>>>>> index 000000000000..4f0764c329c7
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/Kconfig
>>>>>> @@ -0,0 +1,5 @@
>>>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>>>> +
>>>>>> +comment "ARM media platform drivers"
>>>>>> +
>>>>>> +source "drivers/media/platform/arm/mali-c55/Kconfig"
>>>>>> diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
>>>>>> new file mode 100644
>>>>>> index 000000000000..8cc4918725ef
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/Makefile
>>>>>> @@ -0,0 +1,2 @@
>>>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>>>> +obj-y += mali-c55/
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/Kconfig
>>>>>> b/drivers/media/platform/arm/mali-c55/Kconfig
>>>>>> new file mode 100644
>>>>>> index 000000000000..602085e28b01
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/Kconfig
>>>>>> @@ -0,0 +1,18 @@
>>>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>>>> +config VIDEO_MALI_C55
>>>>>> +    tristate "ARM Mali-C55 Image Signal Processor driver"
>>>>>> +    depends on V4L_PLATFORM_DRIVERS
>>>>>> +    depends on VIDEO_DEV && OF
>>>>>> +    depends on ARCH_VEXPRESS || COMPILE_TEST
>>>>>> +    select MEDIA_CONTROLLER
>>>>>> +    select VIDEO_V4L2_SUBDEV_API
>>>>>> +    select VIDEOBUF2_DMA_CONTIG
>>>>>> +    select VIDEOBUF2_VMALLOC
>>>>>> +    select V4L2_FWNODE
>>>>>> +    select GENERIC_PHY_MIPI_DPHY
>>>>> Alphabetical order ?
>>>>>
>>>>>> +    default n
>>>>> That's the default, you don't have to specify ti.
>>>>>
>>>>>> +    help
>>>>>> +      Enable this to support Arm's Mali-C55 Image Signal Processor.
>>>>>> +
>>>>>> +      To compile this driver as a module, choose M here: the module
>>>>>> +      will be called mali-c55.
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile
>>>>>> b/drivers/media/platform/arm/mali-c55/Makefile
>>>>>> new file mode 100644
>>>>>> index 000000000000..77dcb2fbf0f4
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
>>>>>> @@ -0,0 +1,9 @@
>>>>>> +# SPDX-License-Identifier: GPL-2.0
>>>>>> +
>>>>>> +mali-c55-y := mali-c55-capture.o \
>>>>>> +          mali-c55-core.o \
>>>>>> +          mali-c55-isp.o \
>>>>>> +          mali-c55-tpg.o \
>>>>>> +          mali-c55-resizer.o
>>>>> Alphabetical order here too.
>>>>>
>>>>>> +
>>>>>> +obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>>>> new file mode 100644
>>>>>> index 000000000000..1d539ac9c498
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
>>>>>> @@ -0,0 +1,951 @@
>>>>>> +// SPDX-License-Identifier: GPL-2.0
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Video capture devices
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#include <linux/cleanup.h>
>>>>>> +#include <linux/minmax.h>
>>>>>> +#include <linux/pm_runtime.h>
>>>>>> +#include <linux/string.h>
>>>>>> +#include <linux/videodev2.h>
>>>>>> +
>>>>>> +#include <media/v4l2-dev.h>
>>>>>> +#include <media/v4l2-event.h>
>>>>>> +#include <media/v4l2-ioctl.h>
>>>>>> +#include <media/v4l2-subdev.h>
>>>>>> +#include <media/videobuf2-core.h>
>>>>>> +#include <media/videobuf2-dma-contig.h>
>>>>>> +
>>>>>> +#include "mali-c55-common.h"
>>>>>> +#include "mali-c55-registers.h"
>>>>>> +
>>>>>> +static const struct mali_c55_fmt mali_c55_fmts[] = {
>>>>>> +    /*
>>>>>> +     * This table is missing some entries which need further work or
>>>>>> +     * investigation:
>>>>>> +     *
>>>>>> +     * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
>>>>>> +     * Base mode 5 is "Generic Data"
>>>>>> +     * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
>>>>>> +     * Base mode 9 seems to have no V4L2 equivalent
>>>>>> +     * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
>>>>>> +     * equivalent
>>>>>> +     */
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_ARGB2101010,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
>>>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_A2R10G10B10,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_RGB565,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
>>>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_RGB565,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_BGR24,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
>>>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_RGB24,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_YUYV,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_YUY2,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_UYVY,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_UYVY,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_Y210,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_Y210,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    /*
>>>>>> +     * This is something of a hack, the ISP thinks it's running NV12M but
>>>>>> +     * by setting uv_plane = 0 we simply discard that planes and only output
>>>>>> +     * the Y-plane.
>>>>>> +     */
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_GREY,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_NV12M,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_NV21M,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
>>>>>> +        },
>>>>>> +        .is_raw = false,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
>>>>>> +        }
>>>>>> +    },
>>>>>> +    /*
>>>>>> +     * RAW uncompressed formats are all packed in 16 bpp.
>>>>>> +     * TODO: Expand this list to encompass all possible RAW formats.
>>>>>> +     */
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_SRGGB16,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_SRGGB16_1X16,
>>>>>> +        },
>>>>>> +        .is_raw = true,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_SBGGR16,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_SBGGR16_1X16,
>>>>>> +        },
>>>>>> +        .is_raw = true,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_SGBRG16,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_SGBRG16_1X16,
>>>>>> +        },
>>>>>> +        .is_raw = true,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .fourcc = V4L2_PIX_FMT_SGRBG16,
>>>>>> +        .mbus_codes = {
>>>>>> +            MEDIA_BUS_FMT_SGRBG16_1X16,
>>>>>> +        },
>>>>>> +        .is_raw = true,
>>>>>> +        .registers = {
>>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
>>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
>>>>>> +        }
>>>>>> +    },
>>>>>> +};
>>>>>> +
>>>>>> +static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
>>>>>> +                           u32 code)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
>>>>>> +        if (fmt->mbus_codes[i] == code)
>>>>>> +            return true;
>>>>>> +    }
>>>>>> +
>>>>>> +    return false;
>>>>>> +}
>>>>>> +
>>>>>> +bool mali_c55_format_is_raw(unsigned int mbus_code)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>>>>> +        if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
>>>>>> +            return mali_c55_fmts[i].is_raw;
>>>>>> +    }
>>>>>> +
>>>>>> +    return false;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>>>>> +        if (mali_c55_fmts[i].fourcc == pixelformat)
>>>>>> +            return &mali_c55_fmts[i];
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * If we find no matching pixelformat, we'll just default to the first
>>>>>> +     * one for now.
>>>>>> +     */
>>>>>> +
>>>>>> +    return &mali_c55_fmts[0];
>>>>>> +}
>>>>>> +
>>>>>> +static const char * const capture_device_names[] = {
>>>>>> +    "mali-c55 fr",
>>>>>> +    "mali-c55 ds",
>>>>>> +    "mali-c55 3a stats",
>>>>>> +    "mali-c55 params",
>>>> The last two entries are not used AFAICT, neither here, nor in
>>>> subsequent patches.
>>>>
>>>>>> +};
>>>>>> +
>>>>>> +static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
>>>>>> +{
>>>>>> +    if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
>>>>>> +        return capture_device_names[0];
>>>>>> +
>>>>>> +    if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
>>>>>> +        return capture_device_names[1];
>>>>>> +
>>>>>> +    return "params/stat not supported yet";
>>>>>> +}
>>>> Use cap_dev->vdev.name instead of mali_c55_cap_dev_to_name(cap_dev) and
>>>> drop this function.
>>>>
>>>>>> +
>>>>>> +static int mali_c55_link_validate(struct media_link *link)
>>>>>> +{
>>>>>> +    struct video_device *vdev =
>>>>>> + media_entity_to_video_device(link->sink->entity);
>>>>>> +    struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
>>>>>> +    struct v4l2_subdev *sd =
>>>>>> + media_entity_to_v4l2_subdev(link->source->entity);
>>>>>> +    const struct v4l2_pix_format_mplane *pix_mp;
>>>>>> +    const struct mali_c55_fmt *cap_fmt;
>>>>>> +    struct v4l2_subdev_format sd_fmt = {
>>>>>> +        .which = V4L2_SUBDEV_FORMAT_ACTIVE,
>>>>>> +        .pad = link->source->index,
>>>>>> +    };
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
>>>>>> +    if (ret)
>>>>>> +        return ret;
>>>>>> +
>>>>>> +    pix_mp = &cap_dev->mode.pix_mp;
>>>>>> +    cap_fmt = cap_dev->mode.capture_fmt;
>>>>>> +
>>>>>> +    if (sd_fmt.format.width != pix_mp->width ||
>>>>>> +        sd_fmt.format.height != pix_mp->height) {
>>>>>> +        dev_dbg(cap_dev->mali_c55->dev,
>>>>>> +            "link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
>>>>>> +            link->source->entity->name, link->source->index,
>>>>>> +            link->sink->entity->name, link->sink->index,
>>>>>> +            sd_fmt.format.width, sd_fmt.format.height,
>>>>>> +            pix_mp->width, pix_mp->height);
>>>>>> +        return -EPIPE;
>>>>>> +    }
>>>>>> +
>>>>>> +    if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
>>>>>> +        dev_dbg(cap_dev->mali_c55->dev,
>>>>>> +            "link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format
>>>>>> %p4cc\n",
>>>>>> +            link->source->entity->name, link->source->index,
>>>>>> +            link->sink->entity->name, link->sink->index,
>>>>>> +            sd_fmt.format.code, &pix_mp->pixelformat);
>>>>>> +        return -EPIPE;
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct media_entity_operations mali_c55_media_ops = {
>>>>>> +    .link_validate = mali_c55_link_validate,
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
>>>>>> +                    unsigned int *num_planes, unsigned int sizes[],
>>>>>> +                    struct device *alloc_devs[])
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    if (*num_planes) {
>>>>>> +        if (*num_planes != cap_dev->mode.pix_mp.num_planes)
>>>>>> +            return -EINVAL;
>>>>>> +
>>>>>> +        for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>>>>> +            if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
>>>>>> +                return -EINVAL;
>>>>>> +    } else {
>>>>>> +        *num_planes = cap_dev->mode.pix_mp.num_planes;
>>>>>> +        for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>>>>> +            sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_buf_queue(struct vb2_buffer *vb)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
>>>>>> +    struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>>>>>> +    struct mali_c55_buffer *buf = container_of(vbuf,
>>>>>> +                           struct mali_c55_buffer, vb);
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    buf->plane_done[MALI_C55_PLANE_Y] = false;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * If we're in a single-plane format we flag the other plane as done
>>>>>> +     * already so it's dequeued appropriately later
>>>>>> +     */
>>>>>> +    buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
>>>>>> +
>>>>>> +    for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
>>>>>> +        unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
>>>>>> +
>>>>>> +        vb2_set_plane_payload(vb, i, size);
>>>>>> +    }
>>>>>> +
>>>>>> +    spin_lock(&cap_dev->buffers.lock);
>>>>>> +    list_add_tail(&buf->queue, &cap_dev->buffers.queue);
>>>>>> +    spin_unlock(&cap_dev->buffers.lock);
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_buf_init(struct vb2_buffer *vb)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
>>>>>> +    struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>>>>>> +    struct mali_c55_buffer *buf = container_of(vbuf,
>>>>>> +                           struct mali_c55_buffer, vb);
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
>>>>>> +        buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>>>> +
>>>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
>>>>>> +
>>>>>> +    cap_dev->buffers.curr = cap_dev->buffers.next;
>>>>>> +    cap_dev->buffers.next = NULL;
>>>>>> +
>>>>>> +    if (!list_empty(&cap_dev->buffers.queue)) {
>>>>>> +        struct v4l2_pix_format_mplane *pix_mp;
>>>>>> +        const struct v4l2_format_info *info;
>>>>>> +        u32 *addrs;
>>>>>> +
>>>>>> +        pix_mp = &cap_dev->mode.pix_mp;
>>>>>> +        info = v4l2_format_info(pix_mp->pixelformat);
>>>>>> +
>>>>>> +        mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
>>>>>> +        if (cap_dev->mode.capture_fmt->registers.uv_plane)
>>>>>> +            mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
>>>>>> +
>>>>>> +        cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
>>>>>> +                             struct mali_c55_buffer,
>>>>>> +                             queue);
>>>>>> +        list_del(&cap_dev->buffers.next->queue);
>>>>>> +
>>>>>> +        addrs = cap_dev->buffers.next->addrs;
>>>>>> +        mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
>>>>>> +            addrs[MALI_C55_PLANE_Y]);
>>>>>> +        mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
>>>>>> +            addrs[MALI_C55_PLANE_UV]);
>>>>>> +        mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
>>>>>> +            pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
>>>>>> +        mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
>>>>>> +            pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
>>>>>> +            / info->hdiv);
>>>>>> +    } else {
>>>>>> +        /*
>>>>>> +         * If we underflow then we can tell the ISP that we don't want
>>>>>> +         * to write out the next frame.
>>>>>> +         */
>>>>>> +        mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>>>> +        mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>>>> +    }
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
>>>>>> +                   unsigned int framecount)
>>>>>> +{
>>>>>> +    curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>>>>>> +    curr_buf->vb.field = V4L2_FIELD_NONE;
>>>> The could be set already when the buffer is queued.
>>>>
>>>>>> +    curr_buf->vb.sequence = framecount;
>>>>>> +    vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
>>>>>> +}
>>>>>> +
>>>>>> +/**
>>>>>> + * mali_c55_set_plane_done - mark the plane as written and process the buffer if
>>>>>> + *                 both planes are finished.
>>>>>> + * @cap_dev:  pointer to the fr or ds pipe output
>>>>>> + * @plane:    the plane to mark as completed
>>>>>> + *
>>>>>> + * The Mali C55 ISP has muliplanar outputs for some formats that come with two
>>>>>> + * separate "buffer write completed" interrupts - we need to flag each plane's
>>>>>> + * completion and check whether both planes are done - if so, complete the buf
>>>>>> + * in vb2.
>>>>>> + */
>>>>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>>>>>> +                 enum mali_c55_planes plane)
>>>>>> +{
>>>>>> +    struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
>>>>>> +    struct mali_c55_buffer *curr_buf;
>>>>>> +
>>>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
>>>>>> +    curr_buf = cap_dev->buffers.curr;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * This _should_ never happen. If no buffer was available from vb2 then
>>>>>> +     * we tell the ISP not to bother writing the next frame, which means the
>>>>>> +     * interrupts that call this function should never trigger. If it does
>>>>>> +     * happen then one of our assumptions is horribly wrong - complain
>>>>>> +     * loudly and do nothing.
>>>>>> +     */
>>>>>> +    if (!curr_buf) {
>>>>>> +        dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
>>>>>> +            mali_c55_cap_dev_to_name(cap_dev), __func__);
>>>>>> +        return;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* If the other plane is also done... */
>>>>>> +    if (curr_buf->plane_done[~plane & 1]) {
>>>>>> +        mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
>>>>>> +        cap_dev->buffers.curr = NULL;
>>>>>> +        isp->frame_sequence++;
>>>>>> +    } else {
>>>>>> +        curr_buf->plane_done[plane] = true;
>>>>>> +    }
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>>>> +
>>>>>> +    mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                 MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>>>> +    mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                 MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The Mali ISP can hold up to 5 buffer addresses and simply cycle
>>>>>> +     * through them, but it's not clear to me that the vb2 queue _guarantees_
>>>>>> +     * it will queue buffers to the driver in a fixed order, and ensuring
>>>>>> +     * we call vb2_buffer_done() for the right buffer seems to me to add
>>>>>> +     * pointless complexity given in multi-context mode we'd need to
>>>>>> +     * re-write those registers every frame anyway...so we tell the ISP to
>>>>>> +     * use a single register and update it for each frame.
>>>>>> +     */
>>>> A single register sounds prone to error conditions. Is it at least
>>>> shadowed in the hardware, or do you have to make sure you reprogram it
>>>> during the vertical blanking only ?
>>> It would have to be reprogrammed during the vertical blanking if we were running in a
>>> configuration with a single config space, otherwise you have the time it takes to process a frame
>>> plus vertical blanking. As I say, it'll have to work like this in multi-context mode anyway.
>>>
>>> If we want to use the cycling...is it guaranteed that vb2 buffers will always be queued in order?
> In which order ?


vb2_buffer.index order...the ISP cycles through writing to the addresses (called "banks") you give 
it automatically, so if we're initially queued 4 buffers and write their addresses to banks 0-3, the 
ISP would expect to just keep writing to those addresses cyclically in that order. The problem is I 
don't think that there's necessarily a guarantee that when a buffer that was returned to userspace 
and processed is re-queued to the driver that it is queued in the order it was allocated, right? So 
we'd need to keep track of which bank was currently being written to and when a buffer is queued, 
populate its address to bankN+1 or N+2...or wrap around to 0.


Or perhaps we could just use two banks instead of one and alternate them, and that would at least 
provide the shadowing functionality in the event that the Pong config wasn't fitted?

>
>>>> I'll mostly skip buffer handling in this review, I need to first
>>>> understand how the hardware operates to make an informed opinion.
>>>>
>>>>>> +    mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
>>>>>> +            MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
>>>>>> +    mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
>>>>>> +            MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * We only queue a buffer in the streamon path if this is the first of
>>>>>> +     * the capture devices to start streaming. If the ISP is already running
>>>>>> +     * then we rely on the ISP_START interrupt to queue the first buffer for
>>>>>> +     * this capture device.
>>>>>> +     */
>>>>>> +    if (mali_c55->pipe.start_count == 1)
>>>>>> +        mali_c55_set_next_buffer(cap_dev);
>>>> I think we'll have to revisit buffer handling to make sure it's 100%
>>>> race-free.
>>>>
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
>>>>>> +                        enum vb2_buffer_state state)
>>>>>> +{
>>>>>> +    struct mali_c55_buffer *buf, *tmp;
>>>>>> +
>>>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
>>>>>> +
>>>>>> +    if (cap_dev->buffers.curr) {
>>>>>> + vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
>>>>>> +                state);
>>>>>> +        cap_dev->buffers.curr = NULL;
>>>>>> +    }
>>>>>> +
>>>>>> +    if (cap_dev->buffers.next) {
>>>>>> + vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
>>>>>> +                state);
>>>>>> +        cap_dev->buffers.next = NULL;
>>>>>> +    }
>>>>>> +
>>>>>> +    list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
>>>>>> +        list_del(&buf->queue);
>>>>>> +        vb2_buffer_done(&buf->vb.vb2_buf, state);
>>>>>> +    }
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>>>> +    struct mali_c55_resizer *rzr = cap_dev->rzr;
>>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    guard(mutex)(&isp->lock);
>>>> What's the reason for using the isp lock here and in
>>>> mali_c55_vb2_stop_streaming() ? If you need a lock that covers all video
>>>> nodes in order to synchronize start/stop, you may want to use the
>>>> graph_mutex of the media device instead.
>>> It's because I wanted to make sure that the ISP was in a known started/stopped state before
>>> possibly trying to start/stop it, which can be done from either of the two capture devices. This
>>> would go away if we were synchronising with the links anyway.
> OK.
>
>>>>>> +
>>>>>> +    ret = pm_runtime_resume_and_get(mali_c55->dev);
>>>>>> +    if (ret)
>>>>>> +        return ret;
>>>>>> +
>>>>>> +    ret = video_device_pipeline_start(&cap_dev->vdev,
>>>>>> +                      &cap_dev->mali_c55->pipe);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
>>>>>> +            mali_c55_cap_dev_to_name(cap_dev));
>>>> Drop the message or make it dev_dbg() as it can be triggered by
>>>> userspace.
>>>>
>>>>>> +        goto err_pm_put;
>>>>>> +    }
>>>>>> +
>>>>>> +    mali_c55_cap_dev_stream_enable(cap_dev);
>>>>>> +    mali_c55_rzr_start_stream(rzr);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * We only start the ISP if we're the only capture device that's
>>>>>> +     * streaming. Otherwise, it'll already be active.
>>>>>> +     */
>>>> I still think we should use link setup to indicate which video devices
>>>> userspace plans to use, and then only start when they're all started.
>>>> That includes stats and parameters buffers. We can continue this
>>>> discussion in the context of the previous version of the patch series,
>>>> or here, up to you.
>>> Let's just continue here. I think I called it "clunky" before; from my perspective it's an
>>> unnecessary extra step - we can already signal to the driver that we don't want to use the video
>>> devices by not queuing buffers to them or starting the stream on them and although I understand
> By not starting streaming, perhaps, but by not queuing buffers, no. The
> reason is that there's no synchronization between buffer queues. If you
> queue
>
> Frame	FR	DS
> --------------------
> 1	x	x
> 2	x
> 3	x	x
> 4	x	x
>
> it will not be distinguishable by the driver from
>
> Frame	FR	DS
> --------------------
> 1	x	x
> 2	x	x
> 3	x	x
> 4	x
>
>>> that that means that one of the two image data capture devices will receive data before the other,
>>> I don't understand why that's considered to be a problem. Possibly that last part is the stickler;
>>> can you explain a bit why it's an issue for one capture queue to start earlier than the other?
> Because from a userspace point of view, if you want to capture frames
> from both pipelines, you will expect to receive a buffer from each
> pipeline for every frame. If that's not guaranteed at stream start, you
> will then need to implement synchronization code that will drop buffers
> on one pipeline until you get the first buffer on the other pipeline
> (assuming you can synchronize them by sequence number). That will be
> more work, and can introduce latency.


Alright, that makes sense. I'll move to this method then...probably drawing on the IPU6 isys code 
since Sakari mentioned it worked that way already.

>
>>>>>> +    if (mali_c55->pipe.start_count == 1) {
>>>>>> +        ret = mali_c55_isp_start_stream(isp);
>>>>>> +        if (ret)
>>>>>> +            goto err_disable_cap_dev;
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_disable_cap_dev:
>>>>>> +    mali_c55_cap_dev_stream_disable(cap_dev);
>>>>>> +    video_device_pipeline_stop(&cap_dev->vdev);
>>>>>> +err_pm_put:
>>>>>> +    pm_runtime_put(mali_c55->dev);
>>>>>> +    mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
>>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>>>> +    struct mali_c55_resizer *rzr = cap_dev->rzr;
>>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
>>>>>> +
>>>>>> +    guard(mutex)(&isp->lock);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * If one of the other capture nodes is streaming, we shouldn't
>>>>>> +     * disable the ISP here.
>>>>>> +     */
>>>>>> +    if (mali_c55->pipe.start_count == 1)
>>>>>> +        mali_c55_isp_stop_stream(&mali_c55->isp);
>>>>>> +
>>>>>> +    mali_c55_rzr_stop_stream(rzr);
>>>>>> +    mali_c55_cap_dev_stream_disable(cap_dev);
>>>>>> +    mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
>>>>>> +    video_device_pipeline_stop(&cap_dev->vdev);
>>>>>> +    pm_runtime_put(mali_c55->dev);
>>>> I think runtime PM autosuspend would be very useful, as it will ensure
>>>> that stop-reconfigure-start cycles get handled as efficiently as
>>>> possible without powering the device down. It could be done on top as a
>>>> separate patch.
>>> Alright
>>>
>>>>>> +}
>>>>>> +
>>>>>> +static const struct vb2_ops mali_c55_vb2_ops = {
>>>>>> +    .queue_setup        = &mali_c55_vb2_queue_setup,
>>>>>> +    .buf_queue        = &mali_c55_buf_queue,
>>>>>> +    .buf_init        = &mali_c55_buf_init,
>>>>>> +    .wait_prepare        = vb2_ops_wait_prepare,
>>>>>> +    .wait_finish        = vb2_ops_wait_finish,
>>>>>> +    .start_streaming    = &mali_c55_vb2_start_streaming,
>>>>>> +    .stop_streaming        = &mali_c55_vb2_stop_streaming,
>>>>>> +};
>>>>>> +
>>>>>> +static const struct v4l2_file_operations mali_c55_v4l2_fops = {
>>>>>> +    .owner = THIS_MODULE,
>>>>>> +    .unlocked_ioctl = video_ioctl2,
>>>>>> +    .open = v4l2_fh_open,
>>>>>> +    .release = vb2_fop_release,
>>>>>> +    .poll = vb2_fop_poll,
>>>>>> +    .mmap = vb2_fop_mmap,
>>>>>> +};
>>>>>> +
>>>>>> +static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
>>>>>> +{
>>>>>> +    const struct mali_c55_fmt *capture_format;
>>>>>> +    const struct v4l2_format_info *info;
>>>>>> +    struct v4l2_plane_pix_format *plane;
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
>>>>>> +    pix_mp->pixelformat = capture_format->fourcc;
>>>>>> +
>>>>>> +    pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
>>>>>> +                  MALI_C55_MAX_WIDTH);
>>>>>> +    pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
>>>>>> +                   MALI_C55_MAX_HEIGHT);
>>>> Ah, these clamps are right :-)
>>> Hurrah!
>>>
>>>>>> +
>>>>>> +    pix_mp->field = V4L2_FIELD_NONE;
>>>>>> +    pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>>> +    pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>>> +    pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
>>>>>> +
>>>>>> +    info = v4l2_format_info(pix_mp->pixelformat);
>>>> This function may return NULL. That shouldn't be the case as long as it
>>>> supports all formats that the C55 driver supports, so I suppose it's
>>>> safe.
>>>>
>>>>>> +    pix_mp->num_planes = info->mem_planes;
>>>>>> +    memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
>>>>>> +
>>>>>> +    pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;
>>>> Does the hardware support configurable line strides ? If so we should
>>>> support it.
>>> You have to set the line stride in the DMA writer registers, which we do using this same
>>> value...might userspace have set bytesperline already then or something? Or is there some other
>>> place it could be configured?
> Userspace can request a specific stride by setting bytesperline, yes. If
> that's set, you should honour it (and of course adjust it to a
> reasonable [min, max] range as well as align it based on hardware
> constraints).


That seems pretty weird...the bytes per pixel is a fixed value dependent on the format, if the 
stride is changed from (bpp * width) then the width of the image won't match what was requested, so 
we couldn't honour both things at the same time. How is that supposed to work? Or am I 
misunderstanding something?

>
>>>>>> +    pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
>>>>>> +                       * pix_mp->height;
>>>>      pix_mp->plane_fmt[0].sizeimage = pix_mp->plane_fmt[0].bytesperline
>>>>                         * pix_mp->height;
>>>>
>>>>>> +
>>>>>> +    for (i = 1; i < info->comp_planes; i++) {
>>>>>> +        plane = &pix_mp->plane_fmt[i];
>>>>>> +
>>>>>> +        plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
>>>>>> +                           info->hdiv);
>>>>>> +        plane->sizeimage = DIV_ROUND_UP(
>>>>>> +                    plane->bytesperline * pix_mp->height,
>>>>>> +                    info->vdiv);
>>>>>> +    }
>>>>>> +
>>>>>> +    if (info->mem_planes == 1) {
>>>>>> +        for (i = 1; i < info->comp_planes; i++) {
>>>>>> +            plane = &pix_mp->plane_fmt[i];
>>>>>> +            pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
>>>>>> +        }
>>>>>> +    }
>>>> I'm wondering, could v4l2_fill_pixfmt_mp() help ? It doesn't support
>>>> configurable strides though :-S Maybe the helper could be improved, if
>>>> it's close enough to what you need ?
>>> I'll take a look
>>>
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>>>> +                       struct v4l2_format *f)
>>>>>> +{
>>>>>> +    mali_c55_try_fmt(&f->fmt.pix_mp);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
>>>>>> +                struct v4l2_pix_format_mplane *pix_mp)
>>>>>> +{
>>>>>> +    const struct mali_c55_fmt *capture_format;
>>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
>>>>>> +    const struct v4l2_format_info *info;
>>>>>> +
>>>>>> +    mali_c55_try_fmt(pix_mp);
>>>>>> +    capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
>>>>>> +    info = v4l2_format_info(pix_mp->pixelformat);
>>>>>> +    if (WARN_ON(!info))
>>>>>> +        return;
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +               capture_format->registers.base_mode);
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
>>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
>>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
>>>> Could the register writes be moved to stream start time ?
>> Sorry missed this one. These are writes to the context's registers
>> buffer, not to the hardware. Does it matter that they're not done at
>> stream on time?
> Writing them here means you'll have to call pm_runtime_resume_and_get()
> here. If power is then cut off, registers may or may not lose their
> contents, so you would need to write them at stream on time anyway. I
> think it's best to move all the hardware configuration at stream on
> time.
>
>>>>>> +
>>>>>> +    if (info->mem_planes > 1) {
>>>>>> +        mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                   capture_format->registers.base_mode);
>>>>>> +        mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
>>>>>> +                MALI_C55_WRITER_SUBMODE_MASK,
>>>>>> +                capture_format->registers.uv_plane);
>>>>>> +
>>>>>> +        mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
>>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
>>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
>>>>>> +    }
>>>>>> +
>>>>>> +    if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
>>>>>> +        /*
>>>>>> +         * TODO: Figure out the colour matrix coefficients and calculate
>>>>>> +         * and write them here.
>>>>>> +         */
>>>> Ideally they should also be exposed directly to userspace as ISP
>>>> parameters. I would probably go as far as saying that they should come
>>>> directly from userspace, and not derived from the colorspace fields.
>>> Yes I think I agree, I'll drop the todo from here.
>>>
>>>>>> +
>>>>>> +        mali_c55_write(mali_c55,
>>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>>>> +                   MALI_C55_CS_CONV_MATRIX_MASK);
>>>>>> +
>>>>>> +        if (info->hdiv > 1)
>>>>>> +            mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>>>> +                MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
>>>>>> +        if (info->vdiv > 1)
>>>>>> +            mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>>>> +                MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
>>>>>> +        if (info->hdiv > 1 || info->vdiv > 1)
>>>>>> +            mali_c55_update_bits(mali_c55,
>>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
>>>>>> +                MALI_C55_CS_CONV_FILTER_MASK, 0x01);
>>>>>> +    }
>>>>>> +
>>>>>> +    cap_dev->mode.pix_mp = *pix_mp;
>>>>>> +    cap_dev->mode.capture_fmt = capture_format;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>>>> +                     struct v4l2_format *f)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>>>>> +
>>>>>> +    if (vb2_is_busy(&cap_dev->queue))
>>>>>> +        return -EBUSY;
>>>>>> +
>>>>>> +    mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>>>> +                     struct v4l2_format *f)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>>>>> +
>>>>>> +    f->fmt.pix_mp = cap_dev->mode.pix_mp;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
>>>>>> +                        struct v4l2_fmtdesc *f)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
>>>>>> +    unsigned int j = 0;
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
>>>>>> +        if (f->mbus_code &&
>>>>>> + !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
>>>>>> +                               f->mbus_code))
>>>> Small indentation mistake.
>>>>
>>>>>> +            continue;
>>>>>> +
>>>>>> +        /* Downscale pipe can't output RAW formats */
>>>>>> +        if (mali_c55_fmts[i].is_raw &&
>>>>>> +            cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
>>>>>> +            continue;
>>>>>> +
>>>>>> +        if (j++ == f->index) {
>>>>>> +            f->pixelformat = mali_c55_fmts[i].fourcc;
>>>>>> +            return 0;
>>>>>> +        }
>>>>>> +    }
>>>>>> +
>>>>>> +    return -EINVAL;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_querycap(struct file *file, void *fh,
>>>>>> +                 struct v4l2_capability *cap)
>>>>>> +{
>>>>>> +    strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
>>>>>> +    strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
>>>>>> +    .vidioc_reqbufs = vb2_ioctl_reqbufs,
>>>>>> +    .vidioc_querybuf = vb2_ioctl_querybuf,
>>>>>> +    .vidioc_create_bufs = vb2_ioctl_create_bufs,
>>>>>> +    .vidioc_qbuf = vb2_ioctl_qbuf,
>>>>>> +    .vidioc_expbuf = vb2_ioctl_expbuf,
>>>>>> +    .vidioc_dqbuf = vb2_ioctl_dqbuf,
>>>>>> +    .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
>>>>>> +    .vidioc_streamon = vb2_ioctl_streamon,
>>>>>> +    .vidioc_streamoff = vb2_ioctl_streamoff,
>>>>>> +    .vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
>>>>>> +    .vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
>>>>>> +    .vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
>>>>>> +    .vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
>>>>>> +    .vidioc_querycap = mali_c55_querycap,
>>>>>> +    .vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
>>>>>> +    .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
>>>>>> +};
>>>>>> +
>>>>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct v4l2_pix_format_mplane pix_mp;
>>>>>> +    struct mali_c55_cap_dev *cap_dev;
>>>>>> +    struct video_device *vdev;
>>>>>> +    struct vb2_queue *vb2q;
>>>>>> +    unsigned int i;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
>>>> Moving the inner content to a separate mali_c55_register_capture_dev()
>>>> function would increase readability I think, and remove usage of gotos.
>>>> I would probably do the same for unregistration too, for consistency.
>>>>
>>>>>> +        cap_dev = &mali_c55->cap_devs[i];
>>>>>> +        vdev = &cap_dev->vdev;
>>>>>> +        vb2q = &cap_dev->queue;
>>>>>> +
>>>>>> +        /*
>>>>>> +         * The downscale output pipe is an optional block within the ISP
>>>>>> +         * so we need to check whether it's actually been fitted or not.
>>>>>> +         */
>>>>>> +
>>>>>> +        if (i == MALI_C55_CAP_DEV_DS &&
>>>>>> +            !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
>>>>>> +            continue;
>>>> Given that there's only two capture devices, and one is optional, when
>>>> moving the inner code to a separate function you could unroll the loop.
>>>> Up to you.
>>>>
>>>>>> +
>>>>>> +        cap_dev->mali_c55 = mali_c55;
>>>>>> +        mutex_init(&cap_dev->lock);
>>>>>> +        INIT_LIST_HEAD(&cap_dev->buffers.queue);
>>>>>> +
>>>>>> +        switch (i) {
>>>>>> +        case MALI_C55_CAP_DEV_FR:
>>>>>> +            cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
>>>>>> +            cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
>>>>>> +            break;
>>>>>> +        case MALI_C55_CAP_DEV_DS:
>>>>>> +            cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
>>>>>> +            cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
>>>>>> +            break;
>>>>>> +        default:
>>>> That can't happen.
>>>>
>>>>>> + mutex_destroy(&cap_dev->lock);
>>>>>> +            ret = -EINVAL;
>>>>>> +            goto err_destroy_mutex;
>>>>>> +        }
>>>>>> +
>>>>>> +        cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
>>>>>> +        ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
>>>>>> +        if (ret) {
>>>>>> +            mutex_destroy(&cap_dev->lock);
>>>>>> +            goto err_destroy_mutex;
>>>>>> +        }
>>>>>> +
>>>>>> +        vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>>>>>> +        vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
>>>>>> +        vb2q->drv_priv = cap_dev;
>>>>>> +        vb2q->mem_ops = &vb2_dma_contig_memops;
>>>>>> +        vb2q->ops = &mali_c55_vb2_ops;
>>>>>> +        vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
>>>>>> +        vb2q->min_queued_buffers = 1;
>>>>>> +        vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
>>>>>> +        vb2q->lock = &cap_dev->lock;
>>>>>> +        vb2q->dev = mali_c55->dev;
>>>>>> +
>>>>>> +        ret = vb2_queue_init(vb2q);
>>>>>> +        if (ret) {
>>>>>> +            dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
>>>>>> +                mali_c55_cap_dev_to_name(cap_dev));
>>>>>> +            goto err_cleanup_media_entity;
>>>>>> +        }
>>>>>> +
>>>>>> +        strscpy(cap_dev->vdev.name, capture_device_names[i],
>>>>>> +            sizeof(cap_dev->vdev.name));
>>>>>> +        vdev->release = video_device_release_empty;
>>>>>> +        vdev->fops = &mali_c55_v4l2_fops;
>>>>>> +        vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
>>>>>> +        vdev->lock = &cap_dev->lock;
>>>>>> +        vdev->v4l2_dev = &mali_c55->v4l2_dev;
>>>>>> +        vdev->queue = &cap_dev->queue;
>>>>>> +        vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>>>>>> +                    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
>>>>>> +        vdev->entity.ops = &mali_c55_media_ops;
>>>>>> +        video_set_drvdata(vdev, cap_dev);
>>>>>> +
>>>>>> +        memset(&pix_mp, 0, sizeof(pix_mp));
>>>>>> +        pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
>>>>>> +        pix_mp.width = MALI_C55_DEFAULT_WIDTH;
>>>>>> +        pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
>>>>>> +        mali_c55_set_format(cap_dev, &pix_mp);
>>>>>> +
>>>>>> +        ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
>>>>>> +        if (ret) {
>>>>>> +            dev_err(mali_c55->dev,
>>>>>> +                "%s failed to register video device\n",
>>>>>> +                mali_c55_cap_dev_to_name(cap_dev));
>>>>>> +            goto err_release_vb2q;
>>>>>> +        }
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_release_vb2q:
>>>>>> +    vb2_queue_release(vb2q);
>>>>>> +err_cleanup_media_entity:
>>>>>> +    media_entity_cleanup(&cap_dev->vdev.entity);
>>>>>> +err_destroy_mutex:
>>>>>> +    mutex_destroy(&cap_dev->lock);
>>>>>> +    mali_c55_unregister_capture_devs(mali_c55);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_cap_dev *cap_dev;
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
>>>>>> +        cap_dev = &mali_c55->cap_devs[i];
>>>>>> +
>>>>>> +        if (!video_is_registered(&cap_dev->vdev))
>>>>>> +            continue;
>>>>>> +
>>>>>> +        vb2_video_unregister_device(&cap_dev->vdev);
>>>>>> +        media_entity_cleanup(&cap_dev->vdev.entity);
>>>>>> +        mutex_destroy(&cap_dev->lock);
>>>>>> +    }
>>>>>> +}
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>>>> new file mode 100644
>>>>>> index 000000000000..2d0c4d152beb
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>>>> @@ -0,0 +1,266 @@
>>>>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Common definitions
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#ifndef _MALI_C55_COMMON_H
>>>>>> +#define _MALI_C55_COMMON_H
>>>>>> +
>>>>>> +#include <linux/clk.h>
>>>>>> +#include <linux/io.h>
>>>>>> +#include <linux/list.h>
>>>>>> +#include <linux/mutex.h>
>>>>>> +#include <linux/scatterlist.h>
>>>>> I don't think this is needed. You're however missing spinlock.h.
>>>>>
>>>>>> +#include <linux/videodev2.h>
>>>>>> +
>>>>>> +#include <media/media-device.h>
>>>>>> +#include <media/v4l2-async.h>
>>>>>> +#include <media/v4l2-ctrls.h>
>>>>>> +#include <media/v4l2-dev.h>
>>>>>> +#include <media/v4l2-device.h>
>>>>>> +#include <media/v4l2-subdev.h>
>>>>>> +#include <media/videobuf2-core.h>
>>>>>> +#include <media/videobuf2-v4l2.h>
>>>>>> +
>>>>>> +#define MALI_C55_DRIVER_NAME        "mali-c55"
>>>>>> +
>>>>>> +/* min and max values for the image sizes */
>>>>>> +#define MALI_C55_MIN_WIDTH        640U
>>>>>> +#define MALI_C55_MIN_HEIGHT        480U
>>>>>> +#define MALI_C55_MAX_WIDTH        8192U
>>>>>> +#define MALI_C55_MAX_HEIGHT        8192U
>>>>>> +#define MALI_C55_DEFAULT_WIDTH        1920U
>>>>>> +#define MALI_C55_DEFAULT_HEIGHT        1080U
>>>>>> +
>>>>>> +#define MALI_C55_DEFAULT_MEDIA_BUS_FMT MEDIA_BUS_FMT_RGB121212_1X36
>>>>>> +
>>>>>> +struct mali_c55;
>>>>>> +struct mali_c55_cap_dev;
>>>>>> +struct platform_device;
>>>>> You should also forward-declare
>>>>>
>>>>> struct device;
>>>>> struct dma_chan;
>>>>> struct resource;
>>>>>
>>>>>> +
>>>>>> +static const char * const mali_c55_clk_names[] = {
>>>>>> +    "aclk",
>>>>>> +    "hclk",
>>>>>> +};
>>>>> This will end up duplicating the array in each compilation unit, not
>>>>> great. Move it to mali-c55-core.c. You use it in this file just for its
>>>>> size, replace that with a macro that defines the size, or allocate
>>>>> mali_c55.clks dynamically with devm_kcalloc().
>>>>>
>>>>>> +
>>>>>> +enum mali_c55_interrupts {
>>>>>> +    MALI_C55_IRQ_ISP_START,
>>>>>> +    MALI_C55_IRQ_ISP_DONE,
>>>>>> +    MALI_C55_IRQ_MCM_ERROR,
>>>>>> +    MALI_C55_IRQ_BROKEN_FRAME_ERROR,
>>>>>> +    MALI_C55_IRQ_MET_AF_DONE,
>>>>>> +    MALI_C55_IRQ_MET_AEXP_DONE,
>>>>>> +    MALI_C55_IRQ_MET_AWB_DONE,
>>>>>> +    MALI_C55_IRQ_AEXP_1024_DONE,
>>>>>> +    MALI_C55_IRQ_IRIDIX_MET_DONE,
>>>>>> +    MALI_C55_IRQ_LUT_INIT_DONE,
>>>>>> +    MALI_C55_IRQ_FR_Y_DONE,
>>>>>> +    MALI_C55_IRQ_FR_UV_DONE,
>>>>>> +    MALI_C55_IRQ_DS_Y_DONE,
>>>>>> +    MALI_C55_IRQ_DS_UV_DONE,
>>>>>> +    MALI_C55_IRQ_LINEARIZATION_DONE,
>>>>>> +    MALI_C55_IRQ_RAW_FRONTEND_DONE,
>>>>>> +    MALI_C55_IRQ_NOISE_REDUCTION_DONE,
>>>>>> +    MALI_C55_IRQ_IRIDIX_DONE,
>>>>>> +    MALI_C55_IRQ_BAYER2RGB_DONE,
>>>>>> +    MALI_C55_IRQ_WATCHDOG_TIMER,
>>>>>> +    MALI_C55_IRQ_FRAME_COLLISION,
>>>>>> +    MALI_C55_IRQ_UNUSED,
>>>>>> +    MALI_C55_IRQ_DMA_ERROR,
>>>>>> +    MALI_C55_IRQ_INPUT_STOPPED,
>>>>>> +    MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
>>>>>> +    MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
>>>>>> +    MALI_C55_NUM_IRQ_BITS
>>>>> Those are register bits, I think they belong to mali-c55-registers.h,
>>>>> and should probably be macros instead of an enum.
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +enum mali_c55_isp_pads {
>>>>>> +    MALI_C55_ISP_PAD_SINK_VIDEO,
>>>>> As there's a single sink pad, maybe MALI_C55_ISP_PAD_SINK ? Ah, you're
>>>>> probably preparing for ISP parameters support. It's fine.
>>>>>
>>>>>> +    MALI_C55_ISP_PAD_SOURCE,
>>>>> Then maybe this should be named MALI_C55_ISP_PAD_SOURCE_VIDEO as I
>>>>> assume there will be a stats source pad.
>>>>>
>>>>>> +    MALI_C55_ISP_PAD_SOURCE_BYPASS,
>>>>>> +    MALI_C55_ISP_NUM_PADS,
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_tpg {
>>>>>> +    struct mali_c55 *mali_c55;
>>>>>> +    struct v4l2_subdev sd;
>>>>>> +    struct media_pad pad;
>>>>>> +    struct mutex lock;
>>>>>> +    struct mali_c55_tpg_ctrls {
>>>>>> +        struct v4l2_ctrl_handler handler;
>>>>>> +        struct v4l2_ctrl *test_pattern;
>>>>> Set but never used. You can drop it.
>>>>>
>>>>>> +        struct v4l2_ctrl *hblank;
>>>>> Set and used only once, in the same function. You can make it a local
>>>>> variable.
>>>>>
>>>>>> +        struct v4l2_ctrl *vblank;
>>>>>> +    } ctrls;
>>>>>> +};
>>>>> I wonder if this file should be split, with mali-c55-capture.h,
>>>>> mali-c55-core.h, mali-c55-isp.h, ... I think it could increase
>>>>> readability by clearly separating the different elements. Up to you.
>>>>>
>>>>>> +
>>>>>> +struct mali_c55_isp {
>>>>>> +    struct mali_c55 *mali_c55;
>>>>>> +    struct v4l2_subdev sd;
>>>>>> +    struct media_pad pads[MALI_C55_ISP_NUM_PADS];
>>>>>> +    struct media_pad *remote_src;
>>>>>> +    struct v4l2_async_notifier notifier;
>>>>> I'm tempted to move the notifier to mali_c55, as it's related to
>>>>> components external to the whole ISP, not to the ISP subdev itself.
>>>>> Could you give it a try, to see if it could be done without any drawback
>>>>> ?
>>>>>
>>>>>> +    struct mutex lock;
>>>>> Locks require a comment to explain what they protect. Same below where
>>>>> applicable (for both mutexes and spinlocks).
>>>>>
>>>>>> +    unsigned int frame_sequence;
>>>>>> +};
>>>>>> +
>>>>>> +enum mali_c55_resizer_ids {
>>>>>> +    MALI_C55_RZR_FR,
>>>>>> +    MALI_C55_RZR_DS,
>>>>>> +    MALI_C55_NUM_RZRS,
>>>>> The usual abbreviation for "resizer" in drivers/media/ is "rsz", not
>>>>> "rzr". I would have said we can leave it as-is as changing it would be a
>>>>> bit annoying, but I then realized that "rzr" is not just unusual, it's
>>>>> actually not used at all. Would you mind applying a sed globally ? :-)
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +enum mali_c55_rzr_pads {
>>>>> Same enums/structs use abbreviations, some don't. Consistency would
>>>>> help.
>>>>>
>>>>>> +    MALI_C55_RZR_SINK_PAD,
>>>>>> +    MALI_C55_RZR_SOURCE_PAD,
>>>>>> +    MALI_C55_RZR_SINK_BYPASS_PAD,
>>>>>> +    MALI_C55_RZR_NUM_PADS
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_resizer {
>>>>>> +    struct mali_c55 *mali_c55;
>>>>>> +    struct mali_c55_cap_dev *cap_dev;
>>>>>> +    enum mali_c55_resizer_ids id;
>>>>>> +    struct v4l2_subdev sd;
>>>>>> +    struct media_pad pads[MALI_C55_RZR_NUM_PADS];
>>>>>> +    unsigned int num_routes;
>>>>>> +    bool streaming;
>>>>>> +};
>>>>>> +
>>>>>> +enum mali_c55_cap_devs {
>>>>>> +    MALI_C55_CAP_DEV_FR,
>>>>>> +    MALI_C55_CAP_DEV_DS,
>>>>>> +    MALI_C55_NUM_CAP_DEVS
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_fmt {
>>>>> mali_c55_format_info would be a better name I think, as this stores
>>>>> format information, not formats.
>>>>>
>>>>>> +    u32 fourcc;
>>>>>> +    unsigned int mbus_codes[2];
>>>>> A comment to explain why we have two media bus codes would be useful.
>>>>> You can document the whole structure if desired :-)
>>>>>
>>>>>> +    bool is_raw;
>>>>>> +    struct mali_c55_fmt_registers {
>>>>> Make it an anonymous structure, it's never used anywhere else.
>>>>>
>>>>>> +        unsigned int base_mode;
>>>>>> +        unsigned int uv_plane;
>>>>> If those are register field values, use u32 instead of unsigned int.
>>>>>
>>>>>> +    } registers;
>>>>> It's funny, we tend to abbreviate different things, I would have used
>>>>> "regs" here but written "format" in full in the structure name :-)
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +enum mali_c55_isp_bayer_order {
>>>>>> +    MALI_C55_BAYER_ORDER_RGGB,
>>>>>> +    MALI_C55_BAYER_ORDER_GRBG,
>>>>>> +    MALI_C55_BAYER_ORDER_GBRG,
>>>>>> +    MALI_C55_BAYER_ORDER_BGGR
>>>>> These are registers values too, they belong to mali-c55-registers.h.
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_isp_fmt {
>>>>> mali_c55_isp_format_info
>>>>>
>>>>>> +    u32 code;
>>>>>> +    enum v4l2_pixel_encoding encoding;
>>>>> Here you use v4l2_pixel_encoding, for mali_c55_fmt you use is_raw. Maybe
>>>>> pick the same option for both structures ?
>>>>>
>>>>>> +    enum mali_c55_isp_bayer_order order;
>>>>>> +};
>>>>>> +
>>>>>> +enum mali_c55_planes {
>>>>>> +    MALI_C55_PLANE_Y,
>>>>>> +    MALI_C55_PLANE_UV,
>>>>>> +    MALI_C55_NUM_PLANES
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_buffer {
>>>>>> +    struct vb2_v4l2_buffer vb;
>>>>>> +    bool plane_done[MALI_C55_NUM_PLANES];
>>>>> I think tracking the pending state would simplify the logic in
>>>>> mali_c55_set_plane_done(), which would become
>>>>>
>>>>>      curr_buf->plane_pending[plane] = false;
>>>>>
>>>>>      if (!curr_buf->plane_pending[0] && !curr_buf->plane_pending[1]) {
>>>>>          mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
>>>>>          cap_dev->buffers.curr = NULL;
>>>>>          isp->frame_sequence++;
>>>>>      }
>>>>>
>>>>> Or a counter may be even easier (and would consume less memory).
>>>>>
>>>>>> +    struct list_head queue;
>>>>>> +    u32 addrs[MALI_C55_NUM_PLANES];
>>>>> This stores DMA addresses, use dma_addr_t.
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_cap_dev {
>>>>>> +    struct mali_c55 *mali_c55;
>>>>>> +    struct mali_c55_resizer *rzr;
>>>>>> +    struct video_device vdev;
>>>>>> +    struct media_pad pad;
>>>>>> +    struct vb2_queue queue;
>>>>>> +    struct mutex lock;
>>>>>> +    unsigned int reg_offset;
>>>>> Manual handling of the offset everywhere, with parametric macros for the
>>>>> resizer register addresses, isn't very nice. Introduce resizer-specific
>>>>> accessors and capture-specific accessors (mali_c55_rsz_write(), ...)
>>>>> that take a mali_c55_resizer or mali_c55_cap_dev pointer, and handle the
>>>>> offset there. The register macros should loose their offset parameter.
>>>>>
>>>>> You could also use a single set of accessors that would become
>>>>> path/output-specific (mali_c55_path_write() ? mali_c55_output_write()
>>>>> ?), that may make the code easier to read.
>>>>>
>>>>> You can also replace reg_offset with a void __iomem * base, which would
>>>>> avoid the computation at runtime.
>>>>>
>>>>>> +
>>>>>> +    struct mali_c55_mode {
>>>>> Make the structure anonymous.
>>>>>
>>>>>> +        const struct mali_c55_fmt *capture_fmt;
>>>>>> +        struct v4l2_pix_format_mplane pix_mp;
>>>>>> +    } mode;
>>>>> What's a "mode" ? I think I'd name this
>>>>>
>>>>>      struct {
>>>>>          const struct mali_c55_fmt *info;
>>>>>          struct v4l2_pix_format_mplane format;
>>>>>      } format;
>>>>>
>>>>> Or you could just drop the structure and have
>>>>>
>>>>>      const struct mali_c55_fmt *format_info;
>>>>>      struct v4l2_pix_format_mplane format;
>>>>>
>>>>> or something similar.
>>>>>
>>>>>> +
>>>>>> +    struct {
>>>>>> +        spinlock_t lock;
>>>>>> +        struct list_head queue;
>>>>>> +        struct mali_c55_buffer *curr;
>>>>>> +        struct mali_c55_buffer *next;
>>>>>> +    } buffers;
>>>>>> +
>>>>>> +    bool streaming;
>>>>>> +};
>>>>>> +
>>>>>> +enum mali_c55_config_spaces {
>>>>>> +    MALI_C55_CONFIG_PING,
>>>>>> +    MALI_C55_CONFIG_PONG,
>>>>>> +    MALI_C55_NUM_CONFIG_SPACES
>>>>> The last enumerator is not used.
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_ctx {
>>>>> mali_c55_context ?
>>>>>
>>>>>> +    struct mali_c55 *mali_c55;
>>>>>> +    void *registers;
>>>>> Please document this structure and explain that this field points to a
>>>>> copy of the register space in system memory, I was about to write you're
>>>>> missing __iomem :-)
>>>>>
>>>>>> +    phys_addr_t base;
>>>>>> +    spinlock_t lock;
>>>>>> +    struct list_head list;
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55 {
>>>>>> +    struct device *dev;
>>>>>> +    struct resource *res;
>>>>> You could possibly drop this field by passing the physical address of
>>>>> the register space from mali_c55_probe() to mali_c55_init_context() as a
>>>>> function parameter.
>>>>>
>>>>>> +    void __iomem *base;
>>>>>> +    struct dma_chan *channel;
>>>>>> +    struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
>>>>>> +
>>>>>> +    u16 capabilities;
>>>>>> +    struct media_device media_dev;
>>>>>> +    struct v4l2_device v4l2_dev;
>>>>>> +    struct media_pipeline pipe;
>>>>>> +
>>>>>> +    struct mali_c55_tpg tpg;
>>>>>> +    struct mali_c55_isp isp;
>>>>>> +    struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
>>>>>> +    struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
>>>>>> +
>>>>>> +    struct list_head contexts;
>>>>>> +    enum mali_c55_config_spaces next_config;
>>>>>> +};
>>>>>> +
>>>>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
>>>>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
>>>>>> +          bool force_hardware);
>>>>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
>>>>>> +              u32 mask, u32 val);
>>>>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
>>>>>> +              enum mali_c55_config_spaces cfg_space);
>>>>>> +
>>>>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55);
>>>>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55);
>>>>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
>>>>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
>>>>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55);
>>>>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
>>>>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
>>>>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
>>>>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
>>>>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>>>>>> +                 enum mali_c55_planes plane);
>>>>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
>>>>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
>>>>>> +
>>>>>> +bool mali_c55_format_is_raw(unsigned int mbus_code);
>>>>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
>>>>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
>>>>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
>>>>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
>>>>>> +
>>>>>> +const struct mali_c55_isp_fmt *
>>>>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
>>>>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
>>>>>> +#define for_each_mali_isp_fmt(fmt)\
>>>>> #define for_each_mali_isp_fmt(fmt) \
>>>>>
>>>>>> +    for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
>>>>> Looks like parentheses were on sale :-)
>>>>>
>>>>>      for ((fmt) = NULL; (fmt) = mali_c55_isp_fmt_next(fmt); )
>>>>>
>>>>> This macro is used in two places only, in the mali-c55-isp.c file where
>>>>> open-coding the loop without using mali_c55_isp_fmt_next() would be more
>>>>> efficient, and in mali-c55-resizer.c where a function to return format
>>>>> i'th would be more efficient. I think you can drop the macro and the
>>>>> mali_c55_isp_fmt_next() function.
>>>>>
>>>>>> +
>>>>>> +#endif /* _MALI_C55_COMMON_H */
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>>>> new file mode 100644
>>>>>> index 000000000000..50caf5ee7474
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>>>> @@ -0,0 +1,767 @@
>>>>>> +// SPDX-License-Identifier: GPL-2.0
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Core driver code
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#include <linux/bitops.h>
>>>>>> +#include <linux/cleanup.h>
>>>>>> +#include <linux/clk.h>
>>>>>> +#include <linux/delay.h>
>>>>>> +#include <linux/device.h>
>>>>>> +#include <linux/dmaengine.h>
>>>>>> +#include <linux/dma-mapping.h>
>>>>>> +#include <linux/interrupt.h>
>>>>>> +#include <linux/iopoll.h>
>>>>>> +#include <linux/ioport.h>
>>>>>> +#include <linux/mod_devicetable.h>
>>>>>> +#include <linux/of.h>
>>>>>> +#include <linux/of_reserved_mem.h>
>>>>>> +#include <linux/platform_device.h>
>>>>>> +#include <linux/pm_runtime.h>
>>>>>> +#include <linux/scatterlist.h>
>>>>> I don't think this is needed.
>>>>>
>>>>> Missing slab.h.
>>>>>
>>>>>> +#include <linux/string.h>
>>>>>> +
>>>>>> +#include <media/media-entity.h>
>>>>>> +#include <media/v4l2-device.h>
>>>>>> +#include <media/videobuf2-dma-contig.h>
>>>>>> +
>>>>>> +#include "mali-c55-common.h"
>>>>>> +#include "mali-c55-registers.h"
>>>>>> +
>>>>>> +static const char * const mali_c55_interrupt_names[] = {
>>>>>> +    [MALI_C55_IRQ_ISP_START] = "ISP start",
>>>>>> +    [MALI_C55_IRQ_ISP_DONE] = "ISP done",
>>>>>> +    [MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
>>>>>> +    [MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
>>>>>> +    [MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
>>>>>> +    [MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
>>>>>> +    [MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
>>>>>> +    [MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
>>>>>> +    [MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
>>>>>> +    [MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
>>>>>> +    [MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
>>>>>> +    [MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
>>>>>> +    [MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
>>>>>> +    [MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
>>>>>> +    [MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
>>>>>> +    [MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
>>>>>> +    [MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
>>>>>> +    [MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
>>>>>> +    [MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
>>>>>> +    [MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
>>>>>> +    [MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
>>>>>> +    [MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
>>>>>> +    [MALI_C55_IRQ_DMA_ERROR] = "DMA error",
>>>>>> +    [MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
>>>>>> +    [MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
>>>>>> +    [MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
>>>>>> +};
>>>>>> +
>>>>>> +static unsigned int config_space_addrs[] = {
>>>>> const
>>>>>
>>>>>> +    [MALI_C55_CONFIG_PING] = 0x0AB6C,
>>>>>> +    [MALI_C55_CONFIG_PONG] = 0x22B2C,
>>>>> Lowercase hex constants.
>>>>>
>>>>> Don't the values belong to mali-c55-registers.h ?
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +/* System IO
>>>>> /*
>>>>>    * System IO
>>>>>
>>>>>> + *
>>>>>> + * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
>>>>>> + * and 'pong'), with the  expectation that the 'active' space will be left
>>>>> s/the  /the /
>>>>>
>>>>>> + * untouched whilst a frame is being processed and the 'inactive' space
>>>>>> + * configured ready to be passed during the blanking period before the next
>>>>> s/to be passed/to be switched to/ ?
>>>>>
>>>>>> + * frame processing starts. These spaces should ideally be set via DMA transfer
>>>>>> + * from a buffer rather than through individual register set operations. There
>>>>>> + * is also a shared global register space which should be set normally. Of
>>>>>> + * course, the ISP might be included in a system which lacks a suitable DMA
>>>>>> + * engine, and the second configuration space might not be fitted at all, which
>>>>>> + * means we need to support four scenarios:
>>>>>> + *
>>>>>> + * 1. Multi config space, with DMA engine.
>>>>>> + * 2. Multi config space, no DMA engine.
>>>>>> + * 3. Single config space, with DMA engine.
>>>>>> + * 4. Single config space, no DMA engine.
>>>>>> + *
>>>>>> + * The first case is very easy, but the rest present annoying problems. The best
>>>>>> + * way to solve them seems to be simply to replicate the concept of DMAing over
>>>>>> + * the configuration buffer even if there's no DMA engine on the board, for
>>>>>> + * which we rely on memcpy. To facilitate this any read/write call that is made
>>>>>> + * to an address within those config spaces should infact be directed to a
>>>>>> + * buffer that was allocated to hold them rather than the IO memory itself. The
>>>>>> + * actual copy of that buffer to IO mem will happen on interrupt.
>>>>>> + */
>>>>>> +
>>>>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
>>>>>> +{
>>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>>>> +
>>>>>> +    if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
>>>>>> +        spin_lock(&ctx->lock);
>>>>>> +        addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
>>>>>> +        ((u32 *)ctx->registers)[addr] = val;
>>>>>> +        spin_unlock(&ctx->lock);
>>>>>> +
>>>>>> +        return;
>>>>>> +    }
>>>>> Ouch. This is likely the second comment you really won't like (after the
>>>>> comment regarding mali_c55_update_bits() at the very top). I apologize
>>>>> in advance.
>>>>>
>>>>> I really don't like this. Directing writes either to hardware registers
>>>>> or to the shadow registers in the context makes the callers of the
>>>>> read/write accessors very hard to read. The probe code, for instance,
>>>>> mixes writes to hardware registers and writes to the context shadow
>>>>> registers to initialize the value of some of the shadow registers.
>>>>>
>>>>> I'd like to split the read/write accessors into functions that access
>>>>> the hardware registers (that's easy) and functions that access the
>>>>> shadow registers. I think the latter should receive a mali_c55_ctx
>>>>> pointer instead of a mali_c55 pointer to prepare for multi-context
>>>>> support.
>>>>>
>>>>> You can add WARN_ON() guards to the two sets of functions, to ensure
>>>>> that no register from the "other" space gets passed to the wrong
>>>>> function by mistake.
>>>>>
>>>>>> +
>>>>>> +    writel(val, mali_c55->base + addr);
>>>>>> +}
>>>>>> +
>>>>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
>>>>>> +          bool force_hardware)
>>>>> force_hardware is never set to true.
>>>>>
>>>>>> +{
>>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>>>> +    u32 val;
>>>>>> +
>>>>>> +    if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
>>>>>> +        spin_lock(&ctx->lock);
>>>>>> +        addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
>>>>>> +        val = ((u32 *)ctx->registers)[addr];
>>>>>> +        spin_unlock(&ctx->lock);
>>>>>> +
>>>>>> +        return val;
>>>>>> +    }
>>>>>> +
>>>>>> +    return readl(mali_c55->base + addr);
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
>>>>>> +              u32 mask, u32 val)
>>>>>> +{
>>>>>> +    u32 orig, tmp;
>>>>>> +
>>>>>> +    orig = mali_c55_read(mali_c55, addr, false);
>>>>>> +
>>>>>> +    tmp = orig & ~mask;
>>>>>> +    tmp |= (val << (ffs(mask) - 1)) & mask;
>>>>>> +
>>>>>> +    if (tmp != orig)
>>>>>> +        mali_c55_write(mali_c55, addr, tmp);
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
>>>>>> +                 dma_addr_t dst, enum dma_data_direction dir)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>>>> +    struct dma_async_tx_descriptor *tx;
>>>>>> +    enum dma_status status;
>>>>>> +    dma_cookie_t cookie;
>>>>>> +
>>>>>> +    tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
>>>>>> +                       MALI_C55_CONFIG_SPACE_SIZE, 0);
>>>>>> +    if (!tx) {
>>>>>> +        dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
>>>>>> +        return -EIO;
>>>>>> +    }
>>>>>> +
>>>>>> +    cookie = dmaengine_submit(tx);
>>>>>> +    if (dma_submit_error(cookie)) {
>>>>>> +        dev_err(mali_c55->dev, "error submitting dma transfer\n");
>>>>>> +        return -EIO;
>>>>>> +    }
>>>>>> +
>>>>>> +    status = dma_sync_wait(mali_c55->channel, cookie);
>>>>> I've just realized this performs a busy-wait :-S See the comment in the
>>>>> probe function about the threaded IRQ handler. I think we'll need to
>>>>> rework all this. It could be done on top though.
>>>>>
>>>>>> +    if (status != DMA_COMPLETE) {
>>>>>> +        dev_err(mali_c55->dev, "dma transfer failed\n");
>>>>>> +        return -EIO;
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
>>>>>> +                 enum mali_c55_config_spaces cfg_space)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>>>> +    struct device *dma_dev = mali_c55->channel->device->dev;
>>>>>> +    dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
>>>>>> +    dma_addr_t dst;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    guard(spinlock)(&ctx->lock);
>>>>>> +
>>>>>> +    dst = dma_map_single(dma_dev, ctx->registers,
>>>>>> +                 MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
>>>>>> +    if (dma_mapping_error(dma_dev, dst)) {
>>>>>> +        dev_err(mali_c55->dev, "failed to map DMA addr\n");
>>>>>> +        return -EIO;
>>>>>> +    }
>>>>>> +
>>>>>> +    ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
>>>>>> +    dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
>>>>>> +             DMA_FROM_DEVICE);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
>>>>>> +               enum mali_c55_config_spaces cfg_space)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>>>> +    struct device *dma_dev = mali_c55->channel->device->dev;
>>>>>> +    dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
>>>>>> +    dma_addr_t src;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    guard(spinlock)(&ctx->lock);
>>>>> The code below can take a large amount of time, holding a spinlock will
>>>>> disable interrupts on the local CPU, that's not good :-(
>>>>>
>>>>>> +
>>>>>> +    src = dma_map_single(dma_dev, ctx->registers,
>>>>>> +                 MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
>>>>>> +    if (dma_mapping_error(dma_dev, src)) {
>>>>>> +        dev_err(mali_c55->dev, "failed to map DMA addr\n");
>>>>>> +        return -EIO;
>>>>>> +    }
>>>>>> +
>>>>>> +    ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
>>>>>> +    dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
>>>>>> +             DMA_TO_DEVICE);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_config_read(struct mali_c55_ctx *ctx,
>>>>>> +                enum mali_c55_config_spaces cfg_space)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>>>> +
>>>>>> +    if (mali_c55->channel) {
>>>>>> +        return mali_c55_dma_read(ctx, cfg_space);
>>>>> As this function is used at probe time only, to initialize the context,
>>>>> I think DMA is overkill.
>>>>>
>>>>>> +    } else {
>>>>>> +        memcpy_fromio(ctx->registers,
>>>>>> +                  mali_c55->base + config_space_addrs[cfg_space],
>>>>>> +                  MALI_C55_CONFIG_SPACE_SIZE);
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +}
>>>>>> +
>>>>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
>>>>>> +              enum mali_c55_config_spaces cfg_space)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
>>>>>> +
>>>>>> +    if (mali_c55->channel) {
>>>>>> +        return mali_c55_dma_write(ctx, cfg_space);
>>>>>> +    } else {
>>>>>> +        memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
>>>>>> +                ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
>>>>>> +        return 0;
>>>>>> +    }
>>>>> Could you measure the time it typically takes to write the registers
>>>>> using DMA compared to using memcpy_toio() ?
>>>>>
>>>>>> +}
>>>>>> +
>>>>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);
>>>>> I think it's too early to tell how multi-context support will look like.
>>>>> I'm fine keeping mali_c55_get_active_context() as changing that would be
>>>>> very intrusive (even if I think it will need to be changed), but the
>>>>> list of contexts is neither the mechanism we'll use, nor something we
>>>>> need now. Drop the list, embed the context in struct mali_c55, and
>>>>> return the pointer to that single context from this function.
>>>>>
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_remove_links(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> + media_entity_remove_links(&mali_c55->tpg.sd.entity);
>>>>>> + media_entity_remove_links(&mali_c55->isp.sd.entity);
>>>>>> +
>>>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; i++)
>>>>>> + media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
>>>>>> +
>>>>>> +    for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
>>>>>> + media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_create_links(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    /* Test pattern generator to ISP */
>>>>>> +    ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
>>>>>> +                    &mali_c55->isp.sd.entity,
>>>>>> +                    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
>>>>>> +        goto err_remove_links;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* Full resolution resizer pipe. */
>>>>>> +    ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>>>>> +            MALI_C55_ISP_PAD_SOURCE,
>>>>>> + &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
>>>>>> +            MALI_C55_RZR_SINK_PAD,
>>>>>> +            MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
>>>>>> +        goto err_remove_links;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* Full resolution bypass. */
>>>>>> +    ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>>>>> +                    MALI_C55_ISP_PAD_SOURCE_BYPASS,
>>>>>> + &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
>>>>>> +                    MALI_C55_RZR_SINK_BYPASS_PAD,
>>>>>> +                    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
>>>>>> +        goto err_remove_links;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* Resizer pipe to video capture nodes. */
>>>>>> +    ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
>>>>>> +            MALI_C55_RZR_SOURCE_PAD,
>>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
>>>>>> +            0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev,
>>>>>> +            "failed to link FR resizer and video device\n");
>>>>>> +        goto err_remove_links;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* The downscale pipe is an optional hardware block */
>>>>>> +    if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
>>>>>> +        ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>>>>> +            MALI_C55_ISP_PAD_SOURCE,
>>>>>> + &mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
>>>>>> +            MALI_C55_RZR_SINK_PAD,
>>>>>> +            MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>>>> +        if (ret) {
>>>>>> +            dev_err(mali_c55->dev,
>>>>>> +                "failed to link ISP and DS resizer\n");
>>>>>> +            goto err_remove_links;
>>>>>> +        }
>>>>>> +
>>>>>> +        ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
>>>>>> +            MALI_C55_RZR_SOURCE_PAD,
>>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
>>>>>> +            0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>>>> +        if (ret) {
>>>>>> +            dev_err(mali_c55->dev,
>>>>>> +                "failed to link DS resizer and video device\n");
>>>>>> +            goto err_remove_links;
>>>>>> +        }
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_remove_links:
>>>>>> +    mali_c55_remove_links(mali_c55);
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    mali_c55_unregister_tpg(mali_c55);
>>>>>> +    mali_c55_unregister_isp(mali_c55);
>>>>>> +    mali_c55_unregister_resizers(mali_c55);
>>>>>> +    mali_c55_unregister_capture_devs(mali_c55);
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    ret = mali_c55_register_tpg(mali_c55);
>>>>>> +    if (ret)
>>>>>> +        return ret;
>>>>>> +
>>>>>> +    ret = mali_c55_register_isp(mali_c55);
>>>>>> +    if (ret)
>>>>>> +        goto err_unregister_entities;
>>>>>> +
>>>>>> +    ret = mali_c55_register_resizers(mali_c55);
>>>>>> +    if (ret)
>>>>>> +        goto err_unregister_entities;
>>>>>> +
>>>>>> +    ret = mali_c55_register_capture_devs(mali_c55);
>>>>>> +    if (ret)
>>>>>> +        goto err_unregister_entities;
>>>>>> +
>>>>>> +    ret = mali_c55_create_links(mali_c55);
>>>>>> +    if (ret)
>>>>>> +        goto err_unregister_entities;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_unregister_entities:
>>>>>> +    mali_c55_unregister_entities(mali_c55);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    u32 product, version, revision, capabilities;
>>>>>> +
>>>>>> +    product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
>>>>>> +    version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
>>>>>> +    revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
>>>>>> +
>>>>>> +    dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
>>>>>> +         product, version, revision);
>>>>>> +
>>>>>> +    capabilities = mali_c55_read(mali_c55,
>>>>>> +                     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
>>>>>> +                     false);
>>>>>> +    mali_c55->capabilities = (capabilities & 0xffff);
>>>>>> +
>>>>>> +    /* TODO: Might as well start some debugfs */
>>>>> If it's just to expose the version and capabilities, I think that's
>>>>> overkill. It's not needed for debug purpose (you can get it from the
>>>>> kernel log already). debugfs isn't meant to be accessible in production,
>>>>> so an application that would need access to the information wouldn't be
>>>>> able to use it.
>>>>>
>>>>>> +    dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);
>>>>> Combine the two messages into one.
>>>>>
>>>>>> +    return version;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>>>> +    u32 curr_config, next_config;
>>>>>> +
>>>>>> +    curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
>>>>>> +    curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
>>>>>> +              >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
>>>>>> +    next_config = curr_config ^ 1;
>>>>>> +
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>>>> +                 MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
>>>>>> +    mali_c55_config_write(ctx, next_config ?
>>>>>> +                  MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
>>>>>> +}
>>>>>> +
>>>>>> +static irqreturn_t mali_c55_isr(int irq, void *context)
>>>>>> +{
>>>>>> +    struct device *dev = context;
>>>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>>>>> +    u32 interrupt_status;
>>>>>> +    unsigned int i, j;
>>>>>> +
>>>>>> +    interrupt_status = mali_c55_read(mali_c55,
>>>>>> +                     MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
>>>>>> +                     false);
>>>>>> +    if (!interrupt_status)
>>>>>> +        return IRQ_NONE;
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
>>>>>> +               interrupt_status);
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
>>>>>> +
>>>>>> +    for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
>>>>>> +        if (!(interrupt_status & (1 << i)))
>>>>>> +            continue;
>>>>>> +
>>>>>> +        switch (i) {
>>>>>> +        case MALI_C55_IRQ_ISP_START:
>>>>>> +            mali_c55_isp_queue_event_sof(mali_c55);
>>>>>> +
>>>>>> +            for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
>>>>>> + mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
>>>>>> +
>>>>>> +            mali_c55_swap_next_config(mali_c55);
>>>>>> +
>>>>>> +            break;
>>>>>> +        case MALI_C55_IRQ_ISP_DONE:
>>>>>> +            /*
>>>>>> +             * TODO: Where the ISP has no Pong config fitted, we'd
>>>>>> +             * have to do the mali_c55_swap_next_config() call here.
>>>>>> +             */
>>>>>> +            break;
>>>>>> +        case MALI_C55_IRQ_FR_Y_DONE:
>>>>>> +            mali_c55_set_plane_done(
>>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
>>>>>> +                MALI_C55_PLANE_Y);
>>>>>> +            break;
>>>>>> +        case MALI_C55_IRQ_FR_UV_DONE:
>>>>>> +            mali_c55_set_plane_done(
>>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
>>>>>> +                MALI_C55_PLANE_UV);
>>>>>> +            break;
>>>>>> +        case MALI_C55_IRQ_DS_Y_DONE:
>>>>>> +            mali_c55_set_plane_done(
>>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
>>>>>> +                MALI_C55_PLANE_Y);
>>>>>> +            break;
>>>>>> +        case MALI_C55_IRQ_DS_UV_DONE:
>>>>>> +            mali_c55_set_plane_done(
>>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
>>>>>> +                MALI_C55_PLANE_UV);
>>>>>> +            break;
>>>>>> +        default:
>>>>>> +            /*
>>>>>> +             * Only the above interrupts are currently unmasked. If
>>>>>> +             * we receive anything else here then something weird
>>>>>> +             * has gone on.
>>>>>> +             */
>>>>>> +            dev_err(dev, "masked interrupt %s triggered\n",
>>>>>> +                mali_c55_interrupt_names[i]);
>>>>>> +        }
>>>>>> +    }
>>>>>> +
>>>>>> +    return IRQ_HANDLED;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_init_context(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_ctx *ctx;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
>>>>>> +    if (!ctx) {
>>>>>> +        dev_err(mali_c55->dev, "failed to allocate new context\n");
>>>>> No need for an error message when memory allocation fails.
>>>>>
>>>>>> +        return -ENOMEM;
>>>>>> +    }
>>>>>> +
>>>>>> +    ctx->base = mali_c55->res->start;
>>>>>> +    ctx->mali_c55 = mali_c55;
>>>>>> +
>>>>>> +    ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
>>>>>> +                 GFP_KERNEL | GFP_DMA);
>>>>>> +    if (!ctx->registers) {
>>>>>> +        ret = -ENOMEM;
>>>>>> +        goto err_free_ctx;
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The allocated memory is empty, we need to load the default
>>>>>> +     * register settings. We just read Ping; it's identical to Pong.
>>>>>> +     */
>>>>>> +    ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
>>>>>> +    if (ret)
>>>>>> +        goto err_free_registers;
>>>>>> +
>>>>>> +    list_add_tail(&ctx->list, &mali_c55->contexts);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Some features of the ISP need to be disabled by default and only
>>>>>> +     * enabled at the same time as they're configured by a parameters buffer
>>>>>> +     */
>>>>>> +
>>>>>> +    /* Bypass the sqrt and square compression and expansion modules */
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
>>>>>> +                 MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
>>>>>> +                 MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
>>>>>> +
>>>>>> +    /* Bypass the temper module */
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
>>>>>> +               MALI_C55_REG_BYPASS_2_TEMPER);
>>>>>> +
>>>>>> +    /* Bypass the colour noise reduction  */
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
>>>>>> +               MALI_C55_REG_BYPASS_4_CNR);
>>>>>> +
>>>>>> +    /* Disable the sinter module */
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
>>>>>> +                 MALI_C55_SINTER_ENABLE_MASK, 0x00);
>>>>>> +
>>>>>> +    /* Disable the RGB Gamma module for each output */
>>>>>> +    mali_c55_write(
>>>>>> +        mali_c55,
>>>>>> + MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
>>>>>> +        0x00);
>>>>>> +    mali_c55_write(
>>>>>> +        mali_c55,
>>>>>> + MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
>>>>>> +        0x00);
>>>>>> +
>>>>>> +    /* Disable the colour correction matrix */
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_free_registers:
>>>>>> +    kfree(ctx->registers);
>>>>>> +err_free_ctx:
>>>>>> +    kfree(ctx);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_runtime_resume(struct device *dev)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
>>>>>> +                      mali_c55->clks);
>>>>>> +    if (ret)
>>>>>> +        dev_err(mali_c55->dev, "failed to enable clocks\n");
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_runtime_suspend(struct device *dev)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
>>>>>> +
>>>>>> + clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct dev_pm_ops mali_c55_pm_ops = {
>>>>>> +    SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
>>>>>> +                pm_runtime_force_resume)
>>>>>> +    SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
>>>>>> +               NULL)
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_probe(struct platform_device *pdev)
>>>>>> +{
>>>>>> +    struct device *dev = &pdev->dev;
>>>>>> +    struct mali_c55 *mali_c55;
>>>>>> +    dma_cap_mask_t mask;
>>>>>> +    u32 version;
>>>>>> +    int ret;
>>>>>> +    u32 val;
>>>>>> +
>>>>>> +    mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
>>>>>> +    if (!mali_c55)
>>>>>> +        return dev_err_probe(dev, -ENOMEM,
>>>>>> +                     "failed to allocate memory\n");
>>>>>          return -ENOMEM;
>>>>>
>>>>> There's no need to print messages for memory allocation failures.
>>>>>
>>>>>> +
>>>>>> +    mali_c55->dev = dev;
>>>>>> +    platform_set_drvdata(pdev, mali_c55);
>>>>>> +
>>>>>> +    mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
>>>>>> +                                &mali_c55->res);
>>>>>> +    if (IS_ERR(mali_c55->base))
>>>>>> +        return dev_err_probe(dev, PTR_ERR(mali_c55->base),
>>>>>> +                     "failed to map IO memory\n");
>>>>>> +
>>>>>> +    ret = platform_get_irq(pdev, 0);
>>>>>> +    if (ret < 0)
>>>>>> +        return dev_err_probe(dev, ret, "failed to get interrupt num\n");
>>>>> s/ num// or s/num/number/
>>>>>
>>>>>> +
>>>>>> +    ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
>>>>>> +                    mali_c55_isr, IRQF_ONESHOT,
>>>>>> +                    dev_driver_string(&pdev->dev),
>>>>>> +                    &pdev->dev);
>>>>> Requested the IRQ should be done much lower, after you have initialized
>>>>> everything, or an IRQ that would fire early would have really bad
>>>>> consequences.
>>>>>
>>>>> A comment to explain why you need a threaded interrupt handler would be
>>>>> good. I assume it is due only to the need to transfer the registers
>>>>> using DMA. I wonder if we should then split the interrupt handler in
>>>>> two, with a non-threaded part for the operations that can run quickly,
>>>>> and a threaded part for the reprogramming.
>>>>>
>>>>> It may also be that we could just start the DMA transfer in the
>>>>> non-threaded handler without waiting synchronously for it to complete.
>>>>> That would be a bigger change, and would require checking race
>>>>> conditions carefully. On the other hand, I'm a bit concerned about the
>>>>> current implementation, have you tested what happens if the DMA transfer
>>>>> takes too long to complete, and spans frame boundaries ? This part could
>>>>> be addressed by a patch on top of this one.
>>>>>
>>>>>> +    if (ret)
>>>>>> +        return dev_err_probe(dev, ret, "failed to request irq\n");
>>>>>> +
>>>>>> +    for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
>>>>>> +        mali_c55->clks[i].id = mali_c55_clk_names[i];
>>>>>> +
>>>>>> +    ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
>>>>>> +    if (ret)
>>>>>> +        return dev_err_probe(dev, ret, "failed to acquire clocks\n");
>>>>>> +
>>>>>> +    pm_runtime_enable(&pdev->dev);
>>>>>> +
>>>>>> +    ret = pm_runtime_resume_and_get(&pdev->dev);
>>>>>> +    if (ret)
>>>>>> +        goto err_pm_runtime_disable;
>>>>>> +
>>>>>> +    of_reserved_mem_device_init(dev);
>>>>> I'd move this, the vb2_dma_contig_set_max_seg_size() call and the
>>>>> dma_cap_* calls before pm_runtime_enable() as they don't need the device
>>>>> to be powered.
>>>>>
>>>>>> +    version = mali_c55_check_hwcfg(mali_c55);
>>>>>> +    vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
>>>>>> +
>>>>>> +    /* Use "software only" context management. */
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>>>> +                 MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
>>>>>> +                 MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
>>>>> You handle that in mali_c55_isp_start(), does the register have to be
>>>>> set here too ?
>>>>>
>>>>>> +
>>>>>> +    dma_cap_zero(mask);
>>>>>> +    dma_cap_set(DMA_MEMCPY, mask);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * No error check, because we will just fallback on memcpy if there is
>>>>>> +     * no usable DMA channel on the system.
>>>>>> +     */
>>>>>> +    mali_c55->channel = dma_request_channel(mask, NULL, NULL);
>>>>>> +
>>>>>> +    INIT_LIST_HEAD(&mali_c55->contexts);
>>>>>> +    ret = mali_c55_init_context(mali_c55);
>>>>>> +    if (ret)
>>>>>> +        goto err_release_dma_channel;
>>>>>> +
>>>>> I'd move all the code from here ...
>>>>>
>>>>>> +    mali_c55->media_dev.dev = dev;
>>>>>> +    strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
>>>>>> +        sizeof(mali_c55->media_dev.model));
>>>>>> +    mali_c55->media_dev.hw_revision = version;
>>>>>> +
>>>>>> +    media_device_init(&mali_c55->media_dev);
>>>>>> +    ret = media_device_register(&mali_c55->media_dev);
>>>>>> +    if (ret)
>>>>>> +        goto err_cleanup_media_device;
>>>>>> +
>>>>>> +    mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
>>>>>> +    ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(dev, "failed to register V4L2 device\n");
>>>>>> +        goto err_unregister_media_device;
>>>>>> +    };
>>>>>> +
>>>>>> +    ret = mali_c55_register_entities(mali_c55);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(dev, "failed to register entities\n");
>>>>>> +        goto err_unregister_v4l2_device;
>>>>>> +    }
>>>>> ... to here to a separate function, or maybe fold it all in
>>>>> mali_c55_register_entities() (which should the be renamed). Same thing
>>>>> for the cleanup code.
>>>>>
>>>>>> +
>>>>>> +    /* Set safe stop to ensure we're in a non-streaming state */
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>>>>>> +               MALI_C55_INPUT_SAFE_STOP);
>>>>>> +    readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * We're ready to process interrupts. Clear any that are set and then
>>>>>> +     * unmask them for processing.
>>>>>> +     */
>>>>>> +    mali_c55_write(mali_c55, 0x30, 0xffffffff);
>>>>>> +    mali_c55_write(mali_c55, 0x34, 0xffffffff);
>>>>>> +    mali_c55_write(mali_c55, 0x40, 0x01);
>>>>>> +    mali_c55_write(mali_c55, 0x40, 0x00);
>>>>> Please replace the register addresses with macros.
>>>>>
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);
>>>>> The value should use the interrupt bits macros.
>>>>>
>>>>>> +
>>>>>> +    pm_runtime_put(&pdev->dev);
>>>>> Once power gets cut, the registers your programmed above may be lost. I
>>>>> think you should programe them in the runtime PM resume handler.
>>>>>
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_unregister_v4l2_device:
>>>>>> +    v4l2_device_unregister(&mali_c55->v4l2_dev);
>>>>>> +err_unregister_media_device:
>>>>>> +    media_device_unregister(&mali_c55->media_dev);
>>>>>> +err_cleanup_media_device:
>>>>>> +    media_device_cleanup(&mali_c55->media_dev);
>>>>>> +err_release_dma_channel:
>>>>>> +    dma_release_channel(mali_c55->channel);
>>>>>> +err_pm_runtime_disable:
>>>>>> +    pm_runtime_disable(&pdev->dev);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_remove(struct platform_device *pdev)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
>>>>>> +    struct mali_c55_ctx *ctx, *tmp;
>>>>>> +
>>>>>> +    list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
>>>>>> +        list_del(&ctx->list);
>>>>>> +        kfree(ctx->registers);
>>>>>> +        kfree(ctx);
>>>>>> +    }
>>>>>> +
>>>>>> +    mali_c55_remove_links(mali_c55);
>>>>>> +    mali_c55_unregister_entities(mali_c55);
>>>>>> +    v4l2_device_put(&mali_c55->v4l2_dev);
>>>>>> +    media_device_unregister(&mali_c55->media_dev);
>>>>>> +    media_device_cleanup(&mali_c55->media_dev);
>>>>>> +    dma_release_channel(mali_c55->channel);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct of_device_id mali_c55_of_match[] = {
>>>>>> +    { .compatible = "arm,mali-c55", },
>>>>>> +    {},
>>>>>      { /* Sentinel */ },
>>>>>
>>>>>> +};
>>>>>> +MODULE_DEVICE_TABLE(of, mali_c55_of_match);
>>>>>> +
>>>>>> +static struct platform_driver mali_c55_driver = {
>>>>>> +    .driver = {
>>>>>> +        .name = "mali-c55",
>>>>>> +        .of_match_table = of_match_ptr(mali_c55_of_match),
>>>>> Drop of_match_ptr().
>>>>>
>>>>>> +        .pm = &mali_c55_pm_ops,
>>>>>> +    },
>>>>>> +    .probe = mali_c55_probe,
>>>>>> +    .remove_new = mali_c55_remove,
>>>>>> +};
>>>>>> +
>>>>>> +module_platform_driver(mali_c55_driver);
>>>>> Blank line.
>>>>>
>>>>>> +MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
>>>>>> +MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
>>>>>> +MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
>>>>>> +MODULE_LICENSE("GPL");
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>>>> new file mode 100644
>>>>>> index 000000000000..ea8b7b866e7a
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>>>> @@ -0,0 +1,611 @@
>>>>>> +// SPDX-License-Identifier: GPL-2.0
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Image signal processor
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#include <linux/delay.h>
>>>>>> +#include <linux/iopoll.h>
>>>>>> +#include <linux/property.h>
>>>>>> +#include <linux/string.h>
>>>>>> +
>>>>>> +#include <media/media-entity.h>
>>>>>> +#include <media/v4l2-common.h>
>>>>>> +#include <media/v4l2-event.h>
>>>>>> +#include <media/v4l2-mc.h>
>>>>>> +#include <media/v4l2-subdev.h>
>>>>>> +
>>>>>> +#include "mali-c55-common.h"
>>>>>> +#include "mali-c55-registers.h"
>>>>>> +
>>>>>> +static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
>>>>>> +    {
>>>>>> +        .code = MEDIA_BUS_FMT_SRGGB20_1X20,
>>>>>> +        .order = MALI_C55_BAYER_ORDER_RGGB,
>>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .code = MEDIA_BUS_FMT_SGRBG20_1X20,
>>>>>> +        .order = MALI_C55_BAYER_ORDER_GRBG,
>>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .code = MEDIA_BUS_FMT_SGBRG20_1X20,
>>>>>> +        .order = MALI_C55_BAYER_ORDER_GBRG,
>>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .code = MEDIA_BUS_FMT_SBGGR20_1X20,
>>>>>> +        .order = MALI_C55_BAYER_ORDER_BGGR,
>>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .code = MEDIA_BUS_FMT_RGB202020_1X60,
>>>>>> +        .order = 0, /* Not relevant for this format */
>>>>>> +        .encoding = V4L2_PIXEL_ENC_RGB,
>>>>>> +    }
>>>>>> +    /*
>>>>>> +     * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
>>>>>> +     * also support YUV input from a sensor passed-through to the output. At
>>>>>> +     * present we have no mechanism to test that though so it may have to
>>>>>> +     * wait a while...
>>>>>> +     */
>>>>>> +};
>>>>>> +
>>>>>> +const struct mali_c55_isp_fmt *
>>>>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
>>>>>> +{
>>>>>> +    if (!fmt)
>>>>>> +        fmt = &mali_c55_isp_fmts[0];
>>>>>> +    else
>>>>>> +        fmt++;
>>>>>> +
>>>>>> +    for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
>>>>>> +        return fmt;
>>>>> That's peculiar.
>>>>>
>>>>>      if (!fmt)
>>>>>          fmt = &mali_c55_isp_fmts[0];
>>>>>      else if (fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)])
>>>>>          return ++fmt;
>>>>>      else
>>>>>          return NULL;
>>>>>
>>>>>> +
>>>>>> +    return NULL;
>>>>>> +}
>>>>>> +
>>>>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
>>>>>> +{
>>>>>> +    const struct mali_c55_isp_fmt *isp_fmt;
>>>>>> +
>>>>>> +    for_each_mali_isp_fmt(isp_fmt) {
>>>>> I would open-code the loop instead of using the macro, like you do
>>>>> below. It will be more efficient.
>>>>>
>>>>>> +        if (isp_fmt->code == mbus_code)
>>>>>> +            return true;
>>>>>> +    }
>>>>>> +
>>>>>> +    return false;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct mali_c55_isp_fmt *
>>>>>> +mali_c55_isp_get_mbus_config_by_code(u32 code)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
>>>>>> +        if (mali_c55_isp_fmts[i].code == code)
>>>>>> +            return &mali_c55_isp_fmts[i];
>>>>>> +    }
>>>>>> +
>>>>>> +    return NULL;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    u32 val;
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);
>>>>>      mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>>>>>                 MALI_C55_INPUT_SAFE_STOP);
>>>>>
>>>>>> + readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_isp_start(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>>>> +    const struct mali_c55_isp_fmt *cfg;
>>>>>> +    struct v4l2_mbus_framefmt *format;
>>>>> const
>>>>>
>>>>>> +    struct v4l2_subdev_state *state;
>>>>>> +    struct v4l2_rect *crop;
>>>>> const
>>>>>
>>>>>> +    struct v4l2_subdev *sd;
>>>>>> +    u32 val;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    sd = &mali_c55->isp.sd;
>>>>> Assign when declaring the variable.
>>>>>
>>>>>> +
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
>>>>>> +                 MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
>>>>>> +
>>>>>> +    /* Apply input windowing */
>>>>>> +    state = v4l2_subdev_get_locked_active_state(sd);
>>>>> Using .enable_streams() (see below) you'll get this for free.
>>>>>
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +    format = v4l2_subdev_state_get_format(state,
>>>>>> +                          MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +    cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
>>>>>> +               MALI_C55_HC_START(crop->left));
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
>>>>>> +               MALI_C55_HC_SIZE(crop->width));
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
>>>>>> +               MALI_C55_VC_START(crop->top) |
>>>>>> +               MALI_C55_VC_SIZE(crop->height));
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
>>>>>> +                 MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
>>>>>> +                 MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
>>>>>> +                 MALI_C55_BAYER_ORDER_MASK, cfg->order);
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
>>>>>> +                 MALI_C55_INPUT_WIDTH_MASK,
>>>>>> +                 MALI_C55_INPUT_WIDTH_20BIT);
>>>>>> +
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>>>>> +                 MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
>>>>>> +                 cfg->encoding == V4L2_PIXEL_ENC_RGB ?
>>>>>> +                 MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
>>>>>> +
>>>>>> +    ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "failed to DMA config\n");
>>>>>> +        return ret;
>>>>>> +    }
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
>>>>>> +               MALI_C55_INPUT_SAFE_START);
>>>>>> +    readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
>>>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
>>>>> Should you return an error in case of timeout ?
>>>>>
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)
>>>>> Why is this not handled wired to .s_stream() ? Or better,
>>>>> .enable_streams() and .disable_streams().
>>>>>
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>>>> +    struct v4l2_subdev *sd;
>>>>>> +
>>>>>> +    if (isp->remote_src) {
>>>>>> +        sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
>>>>>> +        v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
>>>>>> +    }
>>>>>> +    isp->remote_src = NULL;
>>>>>> +
>>>>>> +    mali_c55_isp_stop(mali_c55);
>>>>>> +}
>>>>>> +
>>>>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>>>> +    struct media_pad *sink_pad;
>>>>>> +    struct v4l2_subdev *sd;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
>>>>>> +    isp->remote_src = media_pad_remote_pad_unique(sink_pad);
>>>>>> +    if (IS_ERR(isp->remote_src)) {
>>>>> If you set the MUST_CONNECT flag for the MALI_C55_ISP_PAD_SINK_VIDEO pad
>>>>> I think you can drop this check.
>>>>>
>>>>>> +        dev_err(mali_c55->dev, "Failed to get source for ISP\n");
>>>>>> +        return PTR_ERR(isp->remote_src);
>>>>>> +    }
>>>>>> +
>>>>>> +    sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
>>>>>> +
>>>>>> +    isp->frame_sequence = 0;
>>>>>> +    ret = mali_c55_isp_start(mali_c55);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "Failed to start ISP\n");
>>>>>> +        isp->remote_src = NULL;
>>>>>> +        return ret;
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * We only support a single input stream, so we can just enable the 1st
>>>>>> +     * entry in the streams mask.
>>>>>> +     */
>>>>>> +    ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "Failed to start ISP source\n");
>>>>>> +        mali_c55_isp_stop(mali_c55);
>>>>>> +        return ret;
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>> +                       struct v4l2_subdev_state *state,
>>>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
>>>>>> +{
>>>>>> +    /*
>>>>>> +     * Only the internal RGB processed format is allowed on the regular
>>>>>> +     * processing source pad.
>>>>>> +     */
>>>>>> +    if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
>>>>>> +        if (code->index)
>>>>>> +            return -EINVAL;
>>>>>> +
>>>>>> +        code->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* On the sink and bypass pads all the supported formats are allowed. */
>>>>>> +    if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    code->code = mali_c55_isp_fmts[code->index].code;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
>>>>>> +                    struct v4l2_subdev_state *state,
>>>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
>>>>>> +{
>>>>>> +    const struct mali_c55_isp_fmt *cfg;
>>>>>> +
>>>>>> +    if (fse->index > 0)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Only the internal RGB processed format is allowed on the regular
>>>>>> +     * processing source pad.
>>>>>> +     *
>>>>>> +     * On the sink and bypass pads all the supported formats are allowed.
>>>>>> +     */
>>>>>> +    if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
>>>>>> +        if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
>>>>>> +            return -EINVAL;
>>>>>> +    } else {
>>>>>> +        cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
>>>>>> +        if (!cfg)
>>>>>> +            return -EINVAL;
>>>>>> +    }
>>>>>> +
>>>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
>>>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
>>>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
>>>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
>>>>>> +                struct v4l2_subdev_state *state,
>>>>>> +                struct v4l2_subdev_format *format)
>>>>>> +{
>>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
>>>>>> +    struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
>>>>>> +    const struct mali_c55_isp_fmt *cfg;
>>>>>> +    struct v4l2_rect *crop;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Disallow set_fmt on the source pads; format is fixed and the sizes
>>>>>> +     * are the result of applying the sink crop rectangle to the sink
>>>>>> +     * format.
>>>>>> +     */
>>>>>> +    if (format->pad)
>>>>>      if (format->pad != MALI_C55_ISP_PAD_SINK_VIDEO)
>>>>>
>>>>>> +        return v4l2_subdev_get_fmt(sd, state, format);
>>>>>> +
>>>>>> +    cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
>>>>>> +    if (!cfg)
>>>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>>>> +    fmt->field = V4L2_FIELD_NONE;
>>>>> Do you intentionally allow the colorspace fields to be overwritten to
>>>>> any value ? rkisp1_isp_set_src_fmt() and rkisp1_isp_set_sink_fmt() may
>>>>> show you how this could be handled.
>>>>>
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Clamp sizes in the accepted limits and clamp the crop rectangle in
>>>>>> +     * the new sizes.
>>>>>> +     */
>>>>>> +    clamp_t(unsigned int, fmt->width,
>>>>>> +        MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>>>>> +    clamp_t(unsigned int, fmt->width,
>>>>>> +        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>>>> clamp_t() returns a value, which you ignore. Those are no-ops. You meant
>>>>>
>>>>>      fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
>>>>>                   MALI_C55_MAX_WIDTH);
>>>>>      fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
>>>>>                    MALI_C55_MAX_HEIGHT);
>>>>>
>>>>> Same for every use of clamp_t() through the whole driver.
>>>>>
>>>>> Also, do you need clamp_t() ? I think all values are unsigned int, you
>>>>> can use clamp().
>>>>>
>>>>> Are there any alignment constraints, such a multiples of two for bayer
>>>>> formats ? Same in all the other locations where applicable.
>>>>>
>>>>>> +
>>>>>> +    sink_fmt = v4l2_subdev_state_get_format(state,
>>>>>> +                        MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +    *sink_fmt = *fmt;
>>>>>> +
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +    crop->left = 0;
>>>>>> +    crop->top = 0;
>>>>>> +    crop->width = fmt->width;
>>>>>> +    crop->height = fmt->height;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Propagate format to source pads. On the 'regular' output pad use
>>>>>> +     * the internal RGB processed format, while on the bypass pad simply
>>>>>> +     * replicate the ISP sink format. The sizes on both pads are the same as
>>>>>> +     * the ISP sink crop rectangle.
>>>>>> +     */
>>>>> Colorspace information will need to be propagated too.
>>>>>
>>>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
>>>>>> +    src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>>>> +    src_fmt->width = crop->width;
>>>>>> +    src_fmt->height = crop->height;
>>>>>> +
>>>>>> +    src_fmt = v4l2_subdev_state_get_format(state,
>>>>>> +                           MALI_C55_ISP_PAD_SOURCE_BYPASS);
>>>>>> +    src_fmt->code = fmt->code;
>>>>>> +    src_fmt->width = crop->width;
>>>>>> +    src_fmt->height = crop->height;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
>>>>>> +                      struct v4l2_subdev_state *state,
>>>>>> +                      struct v4l2_subdev_selection *sel)
>>>>>> +{
>>>>>> +    if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
>>>>>      sel->pad != MALI_C55_ISP_PAD_SINK_VIDEO
>>>>>
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
>>>>>> +                      struct v4l2_subdev_state *state,
>>>>>> +                      struct v4l2_subdev_selection *sel)
>>>>>> +{
>>>>>> +    struct v4l2_mbus_framefmt *src_fmt;
>>>>>> +    struct v4l2_mbus_framefmt *fmt;
>>>>> const
>>>>>
>>>>>> +    struct v4l2_rect *crop;
>>>>>> +
>>>>>> +    if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
>>>>> Ditto.
>>>>>
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +
>>>>>> +    clamp_t(unsigned int, sel->r.left, 0, fmt->width);
>>>>>> +    clamp_t(unsigned int, sel->r.top, 0, fmt->height);
>>>>>> +    clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
>>>>>> +        fmt->width - sel->r.left);
>>>>>> +    clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
>>>>>> +        fmt->height - sel->r.top);
>>>>>> +
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +    *crop = sel->r;
>>>>>> +
>>>>>> +    /* Propagate the crop rectangle sizes to the source pad format. */
>>>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
>>>>>> +    src_fmt->width = crop->width;
>>>>>> +    src_fmt->height = crop->height;
>>>>> Can you confirm that cropping doesn't affect the bypass path ? And maybe
>>>>> add a comment to mention it.
>>>>>
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
>>>>>> +    .enum_mbus_code        = mali_c55_isp_enum_mbus_code,
>>>>>> +    .enum_frame_size    = mali_c55_isp_enum_frame_size,
>>>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
>>>>>> +    .set_fmt        = mali_c55_isp_set_fmt,
>>>>>> +    .get_selection        = mali_c55_isp_get_selection,
>>>>>> +    .set_selection        = mali_c55_isp_set_selection,
>>>>>> +    .link_validate        = v4l2_subdev_link_validate_default,
>>>>>> +};
>>>>>> +
>>>>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct v4l2_event event = {
>>>>>> +        .type = V4L2_EVENT_FRAME_SYNC,
>>>>>> +    };
>>>>>> +
>>>>>> +    event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
>>>>>> +    v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
>>>>>> +}
>>>>>> +
>>>>>> +static int
>>>>>> +mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
>>>>>> +                 struct v4l2_event_subscription *sub)
>>>>>> +{
>>>>>> +    if (sub->type != V4L2_EVENT_FRAME_SYNC)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    /* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
>>>>>> +    if (sub->id != 0)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    return v4l2_event_subscribe(fh, sub, 0, NULL);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
>>>>>> +    .subscribe_event = mali_c55_isp_subscribe_event,
>>>>>> +    .unsubscribe_event = v4l2_event_subdev_unsubscribe,
>>>>>> +};
>>>>>> +
>>>>>> +static const struct v4l2_subdev_ops mali_c55_isp_ops = {
>>>>>> +    .pad    = &mali_c55_isp_pad_ops,
>>>>>> +    .core    = &mali_c55_isp_core_ops,
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
>>>>>> +                   struct v4l2_subdev_state *sd_state)
>>>>> You name this variable state in every other subdev operation handler.
>>>>>
>>>>>> +{
>>>>>> +    struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>> +    struct v4l2_rect *in_crop;
>>>>>> +
>>>>>> +    sink_fmt = v4l2_subdev_state_get_format(sd_state,
>>>>>> +                        MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +    src_fmt = v4l2_subdev_state_get_format(sd_state,
>>>>>> +                           MALI_C55_ISP_PAD_SOURCE);
>>>>>> +    in_crop = v4l2_subdev_state_get_crop(sd_state,
>>>>>> +                         MALI_C55_ISP_PAD_SINK_VIDEO);
>>>>>> +
>>>>>> +    sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>>>> +    sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>>>> +    sink_fmt->field = V4L2_FIELD_NONE;
>>>>>> +    sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>>> You should initialize the colorspace fields too. Same below.
>>>>>
>>>>>> +
>>>>>> +    *v4l2_subdev_state_get_format(sd_state,
>>>>>> +                  MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
>>>>>> +
>>>>>> +    src_fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>>>> +    src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>>>> +    src_fmt->field = V4L2_FIELD_NONE;
>>>>>> +    src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>>>> +
>>>>>> +    in_crop->top = 0;
>>>>>> +    in_crop->left = 0;
>>>>>> +    in_crop->width = MALI_C55_DEFAULT_WIDTH;
>>>>>> +    in_crop->height = MALI_C55_DEFAULT_HEIGHT;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
>>>>>> +    .init_state = mali_c55_isp_init_state,
>>>>>> +};
>>>>>> +
>>>>>> +static const struct media_entity_operations mali_c55_isp_media_ops = {
>>>>>> +    .link_validate        = v4l2_subdev_link_validate,
>>>>>      .link_validate = v4l2_subdev_link_validate,
>>>>>
>>>>> to match mali_c55_isp_internal_ops.
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
>>>>>> +                       struct v4l2_subdev *subdev,
>>>>>> +                       struct v4l2_async_connection *asc)
>>>>>> +{
>>>>>> +    struct mali_c55_isp *isp = container_of(notifier,
>>>>>> +                        struct mali_c55_isp, notifier);
>>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>>>> +    struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * By default we'll flag this link enabled and the TPG disabled, but
>>>>>> +     * no immutable flag because we need to be able to switch between the
>>>>>> +     * two.
>>>>>> +     */
>>>>>> +    ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
>>>>>> +                          MEDIA_LNK_FL_ENABLED);
>>>>>> +    if (ret)
>>>>>> +        dev_err(mali_c55->dev, "failed to create link for %s\n",
>>>>>> +            subdev->name);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
>>>>>> +{
>>>>>> +    struct mali_c55_isp *isp = container_of(notifier,
>>>>>> +                        struct mali_c55_isp, notifier);
>>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>>>> +
>>>>>> +    return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
>>>>>> +    .bound = mali_c55_isp_notifier_bound,
>>>>>> +    .complete = mali_c55_isp_notifier_complete,
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
>>>>>> +    struct v4l2_async_connection *asc;
>>>>>> +    struct fwnode_handle *ep;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The ISP should have a single endpoint pointing to some flavour of
>>>>>> +     * CSI-2 receiver...but for now at least we do want everything to work
>>>>>> +     * normally even with no sensors connected, as we have the TPG. If we
>>>>>> +     * don't find a sensor just warn and return success.
>>>>>> +     */
>>>>>> +    ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
>>>>>> +                         0, 0, 0);
>>>>>> +    if (!ep) {
>>>>>> +        dev_warn(mali_c55->dev, "no local endpoint found\n");
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
>>>>>> +                          struct v4l2_async_connection);
>>>>>> +    if (IS_ERR(asc)) {
>>>>>> +        dev_err(mali_c55->dev, "failed to add remote fwnode\n");
>>>>>> +        ret = PTR_ERR(asc);
>>>>>> +        goto err_put_ep;
>>>>>> +    }
>>>>>> +
>>>>>> +    isp->notifier.ops = &mali_c55_isp_notifier_ops;
>>>>>> +    ret = v4l2_async_nf_register(&isp->notifier);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "failed to register notifier\n");
>>>>>> +        goto err_cleanup_nf;
>>>>>> +    }
>>>>>> +
>>>>>> +    fwnode_handle_put(ep);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_cleanup_nf:
>>>>>> +    v4l2_async_nf_cleanup(&isp->notifier);
>>>>>> +err_put_ep:
>>>>>> +    fwnode_handle_put(ep);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
>>>>>> +    struct v4l2_subdev *sd = &isp->sd;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    isp->mali_c55 = mali_c55;
>>>>>> +
>>>>>> +    v4l2_subdev_init(sd, &mali_c55_isp_ops);
>>>>>> +    sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
>>>>>> +    sd->entity.ops = &mali_c55_isp_media_ops;
>>>>>> +    sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
>>>>>> +    sd->internal_ops = &mali_c55_isp_internal_ops;
>>>>>> +    strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
>>>>>> +
>>>>>> +    isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
>>>>> The MUST_CONNECT flag would make sense here.
>>>>>
>>>>>> + isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>>>>>> +    isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
>>>>>> +
>>>>>> +    ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
>>>>>> +                     isp->pads);
>>>>>> +    if (ret)
>>>>>> +        return ret;
>>>>>> +
>>>>>> +    ret = v4l2_subdev_init_finalize(sd);
>>>>>> +    if (ret)
>>>>>> +        goto err_cleanup_media_entity;
>>>>>> +
>>>>>> +    ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>>>>> +    if (ret)
>>>>>> +        goto err_cleanup_subdev;
>>>>>> +
>>>>>> +    ret = mali_c55_isp_parse_endpoint(isp);
>>>>>> +    if (ret)
>>>>>> +        goto err_cleanup_subdev;
>>>>> As noted elsewhere, I think this belongs to mali-c55-core.c.
>>>>>
>>>>>> +
>>>>>> +    mutex_init(&isp->lock);
>>>>> This lock is used in mali-c55-capture.c only, that seems weird.
>>>>>
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_cleanup_subdev:
>>>>>> +    v4l2_subdev_cleanup(sd);
>>>>>> +err_cleanup_media_entity:
>>>>>> +    media_entity_cleanup(&sd->entity);
>>>>>> +    isp->mali_c55 = NULL;
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
>>>>>> +
>>>>>> +    if (!isp->mali_c55)
>>>>>> +        return;
>>>>>> +
>>>>>> +    mutex_destroy(&isp->lock);
>>>>>> +    v4l2_async_nf_unregister(&isp->notifier);
>>>>>> +    v4l2_async_nf_cleanup(&isp->notifier);
>>>>>> +    v4l2_device_unregister_subdev(&isp->sd);
>>>>>> +    v4l2_subdev_cleanup(&isp->sd);
>>>>>> +    media_entity_cleanup(&isp->sd.entity);
>>>>>> +}
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>>>> new file mode 100644
>>>>>> index 000000000000..cb27abde2aa5
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>>>> @@ -0,0 +1,258 @@
>>>>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Register definitions
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#ifndef _MALI_C55_REGISTERS_H
>>>>>> +#define _MALI_C55_REGISTERS_H
>>>>>> +
>>>>>> +#include <linux/bits.h>
>>>>>> +
>>>>>> +/* ISP Common 0x00000 - 0x000ff */
>>>>>> +
>>>>>> +#define MALI_C55_REG_API                0x00000
>>>>>> +#define MALI_C55_REG_PRODUCT                0x00004
>>>>>> +#define MALI_C55_REG_VERSION                0x00008
>>>>>> +#define MALI_C55_REG_REVISION                0x0000c
>>>>>> +#define MALI_C55_REG_PULSE_MODE                0x0003c
>>>>>> +#define MALI_C55_REG_INPUT_MODE_REQUEST            0x0009c
>>>>>> +#define MALI_C55_INPUT_SAFE_STOP            0x00
>>>>>> +#define MALI_C55_INPUT_SAFE_START            0x01
>>>>>> +#define MALI_C55_REG_MODE_STATUS            0x000a0
>>>>>> +#define MALI_C55_REG_INTERRUPT_MASK_VECTOR        0x00030
>>>>>> +#define MALI_C55_INTERRUPT_MASK_ALL            GENMASK(31, 0)
>>>>>> +
>>>>>> +#define MALI_C55_REG_GLOBAL_MONITOR            0x00050
>>>>>> +
>>>>>> +#define MALI_C55_REG_GEN_VIDEO                0x00080
>>>>>> +#define MALI_C55_REG_GEN_VIDEO_ON_MASK            BIT(0)
>>>>>> +#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK        BIT(1)
>>>>>> +#define MALI_C55_REG_GEN_PREFETCH_MASK GENMASK(31, 16)
>>>>>> +
>>>>>> +#define MALI_C55_REG_MCU_CONFIG                0x00020
>>>>>> +#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK        BIT(0)
>>>>> #define MALI_C55_REG_MCU_CONFIG_OVERRIDE        BIT(0)
>>>>>
>>>>> Same in other places where applicable.
>>>>>
>>>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK        BIT(1)
>>>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PING        BIT(1)
>>>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG        0x00
>>>>>> +#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK        BIT(8)
>>>>>> +#define MALI_C55_REG_PING_PONG_READ            0x00024
>>>>>> +#define MALI_C55_REG_PING_PONG_READ_MASK        BIT(2)
>>>>>> +
>>>>>> +#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR        0x00034
>>>>>> +#define MALI_C55_REG_INTERRUPT_CLEAR            0x00040
>>>>>> +#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR        0x00044
>>>>>> +#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS        0x00068
>>>>>> +#define MALI_C55_GPS_PONG_FITTED            BIT(0)
>>>>>> +#define MALI_C55_GPS_WDR_FITTED                BIT(1)
>>>>>> +#define MALI_C55_GPS_COMPRESSION_FITTED            BIT(2)
>>>>>> +#define MALI_C55_GPS_TEMPER_FITTED            BIT(3)
>>>>>> +#define MALI_C55_GPS_SINTER_LITE_FITTED            BIT(4)
>>>>>> +#define MALI_C55_GPS_SINTER_FITTED            BIT(5)
>>>>>> +#define MALI_C55_GPS_IRIDIX_LTM_FITTED            BIT(6)
>>>>>> +#define MALI_C55_GPS_IRIDIX_GTM_FITTED            BIT(7)
>>>>>> +#define MALI_C55_GPS_CNR_FITTED                BIT(8)
>>>>>> +#define MALI_C55_GPS_FRSCALER_FITTED            BIT(9)
>>>>>> +#define MALI_C55_GPS_DS_PIPE_FITTED            BIT(10)
>>>>>> +
>>>>>> +#define MALI_C55_REG_BLANKING                0x00084
>>>>>> +#define MALI_C55_REG_HBLANK_MASK            GENMASK(15, 0)
>>>>>> +#define MALI_C55_REG_VBLANK_MASK            GENMASK(31, 16)
>>>>>> +
>>>>>> +#define MALI_C55_REG_HC_START                0x00088
>>>>>> +#define MALI_C55_HC_START(h)                (((h) & 0xffff) << 16)
>>>>>> +#define MALI_C55_REG_HC_SIZE                0x0008c
>>>>>> +#define MALI_C55_HC_SIZE(h)                ((h) & 0xffff)
>>>>>> +#define MALI_C55_REG_VC_START_SIZE            0x00094
>>>>>> +#define MALI_C55_VC_START(v)                ((v) & 0xffff)
>>>>>> +#define MALI_C55_VC_SIZE(v)                (((v) & 0xffff) << 16)
>>>>>> +
>>>>>> +/* Ping/Pong Configuration Space */
>>>>>> +#define MALI_C55_REG_BASE_ADDR                0x18e88
>>>>>> +#define MALI_C55_REG_BYPASS_0                0x18eac
>>>>>> +#define MALI_C55_REG_BYPASS_0_VIDEO_TEST        BIT(0)
>>>>>> +#define MALI_C55_REG_BYPASS_0_INPUT_FMT            BIT(1)
>>>>>> +#define MALI_C55_REG_BYPASS_0_DECOMPANDER        BIT(2)
>>>>>> +#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR BIT(3)
>>>>>> +#define MALI_C55_REG_BYPASS_0_GAIN_WDR            BIT(4)
>>>>>> +#define MALI_C55_REG_BYPASS_0_FRAME_STITCH        BIT(5)
>>>>>> +#define MALI_C55_REG_BYPASS_1                0x18eb0
>>>>>> +#define MALI_C55_REG_BYPASS_1_DIGI_GAIN            BIT(0)
>>>>>> +#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS        BIT(1)
>>>>>> +#define MALI_C55_REG_BYPASS_1_FE_SQRT            BIT(2)
>>>>>> +#define MALI_C55_REG_BYPASS_1_RAW_FE            BIT(3)
>>>>>> +#define MALI_C55_REG_BYPASS_2                0x18eb8
>>>>>> +#define MALI_C55_REG_BYPASS_2_SINTER            BIT(0)
>>>>>> +#define MALI_C55_REG_BYPASS_2_TEMPER            BIT(1)
>>>>>> +#define MALI_C55_REG_BYPASS_3                0x18ebc
>>>>>> +#define MALI_C55_REG_BYPASS_3_SQUARE_BE            BIT(0)
>>>>>> +#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH BIT(1)
>>>>>> +#define MALI_C55_REG_BYPASS_3_MESH_SHADING        BIT(3)
>>>>>> +#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE        BIT(4)
>>>>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX            BIT(5)
>>>>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN        BIT(6)
>>>>>> +#define MALI_C55_REG_BYPASS_4                0x18ec0
>>>>>> +#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB        BIT(1)
>>>>>> +#define MALI_C55_REG_BYPASS_4_PF_CORRECTION        BIT(3)
>>>>>> +#define MALI_C55_REG_BYPASS_4_CCM            BIT(4)
>>>>>> +#define MALI_C55_REG_BYPASS_4_CNR            BIT(5)
>>>>>> +#define MALI_C55_REG_FR_BYPASS                0x18ec4
>>>>>> +#define MALI_C55_REG_DS_BYPASS                0x18ec8
>>>>>> +#define MALI_C55_BYPASS_CROP                BIT(0)
>>>>>> +#define MALI_C55_BYPASS_SCALER                BIT(1)
>>>>>> +#define MALI_C55_BYPASS_GAMMA_RGB            BIT(2)
>>>>>> +#define MALI_C55_BYPASS_SHARPEN                BIT(3)
>>>>>> +#define MALI_C55_BYPASS_CS_CONV                BIT(4)
>>>>>> +#define MALI_C55_REG_ISP_RAW_BYPASS            0x18ecc
>>>>>> +#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK        BIT(0)
>>>>>> +#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK GENMASK(9, 8)
>>>>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS        2
>>>>>> +#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS        1
>>>>>> +#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE BIT(1)
>>>>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS        BIT(0)
>>>>>> +
>>>>>> +#define MALI_C55_REG_ACTIVE_WIDTH_MASK            0xffff
>>>>>> +#define MALI_C55_REG_ACTIVE_HEIGHT_MASK 0xffff0000
>>>>>> +#define MALI_C55_REG_BAYER_ORDER            0x18e8c
>>>>>> +#define MALI_C55_BAYER_ORDER_MASK            GENMASK(1, 0)
>>>>>> +#define MALI_C55_REG_TPG_CH0                0x18ed8
>>>>>> +#define MALI_C55_TEST_PATTERN_ON_OFF            BIT(0)
>>>>>> +#define MALI_C55_TEST_PATTERN_RGB_MASK            BIT(1)
>>>>>> +#define MALI_C55_REG_TPG_R_BACKGROUND            0x18ee0
>>>>>> +#define MALI_C55_REG_TPG_G_BACKGROUND            0x18ee4
>>>>>> +#define MALI_C55_REG_TPG_B_BACKGROUND            0x18ee8
>>>>>> +#define MALI_C55_TPG_BACKGROUND_MAX            0xfffff
>>>>>> +#define MALI_C55_REG_INPUT_WIDTH            0x18f98
>>>>>> +#define MALI_C55_INPUT_WIDTH_MASK            GENMASK(18, 16)
>>>>>> +#define MALI_C55_INPUT_WIDTH_8BIT            0
>>>>>> +#define MALI_C55_INPUT_WIDTH_10BIT            1
>>>>>> +#define MALI_C55_INPUT_WIDTH_12BIT            2
>>>>>> +#define MALI_C55_INPUT_WIDTH_14BIT            3
>>>>>> +#define MALI_C55_INPUT_WIDTH_16BIT            4
>>>>>> +#define MALI_C55_INPUT_WIDTH_20BIT            5
>>>>>> +#define MALI_C55_REG_SPACE_SIZE                0x4000
>>>>>> +#define MALI_C55_REG_CONFIG_SPACES_OFFSET        0x0ab6c
>>>>>> +#define MALI_C55_CONFIG_SPACE_SIZE            0x1231c
>>>>>> +
>>>>>> +#define MALI_C55_REG_SINTER_CONFIG            0x19348
>>>>>> +#define MALI_C55_SINTER_VIEW_FILTER_MASK        GENMASK(1, 0)
>>>>>> +#define MALI_C55_SINTER_SCALE_MODE_MASK GENMASK(3, 2)
>>>>>> +#define MALI_C55_SINTER_ENABLE_MASK            BIT(4)
>>>>>> +#define MALI_C55_SINTER_FILTER_SELECT_MASK        BIT(5)
>>>>>> +#define MALI_C55_SINTER_INT_SELECT_MASK            BIT(6)
>>>>>> +#define MALI_C55_SINTER_RM_ENABLE_MASK            BIT(7)
>>>>>> +
>>>>>> +/* Colour Correction Matrix Configuration */
>>>>>> +#define MALI_C55_REG_CCM_ENABLE                0x1b07c
>>>>>> +#define MALI_C55_CCM_ENABLE_MASK            BIT(0)
>>>>>> +#define MALI_C55_REG_CCM_COEF_R_R            0x1b080
>>>>>> +#define MALI_C55_REG_CCM_COEF_R_G            0x1b084
>>>>>> +#define MALI_C55_REG_CCM_COEF_R_B            0x1b088
>>>>>> +#define MALI_C55_REG_CCM_COEF_G_R            0x1b090
>>>>>> +#define MALI_C55_REG_CCM_COEF_G_G            0x1b094
>>>>>> +#define MALI_C55_REG_CCM_COEF_G_B            0x1b098
>>>>>> +#define MALI_C55_REG_CCM_COEF_B_R            0x1b0a0
>>>>>> +#define MALI_C55_REG_CCM_COEF_B_G            0x1b0a4
>>>>>> +#define MALI_C55_REG_CCM_COEF_B_B            0x1b0a8
>>>>>> +#define MALI_C55_CCM_COEF_MASK                GENMASK(12, 0)
>>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R            0x1b0b0
>>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G            0x1b0b4
>>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B            0x1b0b8
>>>>>> +#define MALI_C55_CCM_ANTIFOG_GAIN_MASK GENMASK(11, 0)
>>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R        0x1b0c0
>>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G        0x1b0c4
>>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B        0x1b0c8
>>>>>> +#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK        GENMASK(11, 0)
>>>>>> +
>>>>>> +/*
>>>>>> + * The Mali-C55 ISP has up to two output pipes; known as full resolution and
>>>>>> + * down scaled. The register space for these is laid out identically, but offset
>>>>>> + * by 372 bytes.
>>>>>> + */
>>>>>> +#define MALI_C55_CAP_DEV_FR_REG_OFFSET        0x0
>>>>>> +#define MALI_C55_CAP_DEV_DS_REG_OFFSET        0x174
>>>>>> +
>>>>>> +#define MALI_C55_REG_CS_CONV_CONFIG(offset)        (0x1c098 + (offset))
>>>>>> +#define MALI_C55_CS_CONV_MATRIX_MASK            BIT(0)
>>>>>> +#define MALI_C55_CS_CONV_FILTER_MASK            BIT(1)
>>>>>> +#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK        BIT(2)
>>>>>> +#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK        BIT(3)
>>>>>> +#define MALI_C55_REG_Y_WRITER_MODE(offset)        (0x1c0ec + (offset))
>>>>>> +#define MALI_C55_REG_UV_WRITER_MODE(offset)        (0x1c144 + (offset))
>>>>>> +#define MALI_C55_WRITER_MODE_MASK            GENMASK(4, 0)
>>>>>> +#define MALI_C55_WRITER_SUBMODE_MASK            GENMASK(7, 6)
>>>>>> +#define MALI_C55_WRITER_FRAME_WRITE_MASK        BIT(9)
>>>>>> +#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset) (0x1c0f0 + (offset))
>>>>>> +#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset) (0x1c148 + (offset))
>>>>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)        ((w) << 0)
>>>>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)        ((h) << 16)
>>>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset) (0x1c0f4 + (offset))
>>>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset) (0x1c108 + (offset))
>>>>>> +#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK GENMASK(2, 0)
>>>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_RESTART        BIT(3)
>>>>>> +#define MALI_C55_REG_Y_WRITER_OFFSET(offset) (0x1c10c + (offset))
>>>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset) (0x1c14c + (offset))
>>>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset) (0x1c160 + (offset))
>>>>>> +#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK GENMASK(2, 0)
>>>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_RESTART        BIT(3)
>>>>>> +#define MALI_C55_REG_UV_WRITER_OFFSET(offset) (0x1c164 + (offset))
>>>>>> +
>>>>>> +#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
>>>>>> +#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE 0x18edc
>>>>>> +
>>>>>> +#define MALI_C55_REG_CROP_EN(offset)            (0x1c028 + (offset))
>>>>>> +#define MALI_C55_CROP_ENABLE                BIT(0)
>>>>>> +#define MALI_C55_REG_CROP_X_START(offset)        (0x1c02c + (offset))
>>>>>> +#define MALI_C55_REG_CROP_Y_START(offset)        (0x1c030 + (offset))
>>>>>> +#define MALI_C55_REG_CROP_X_SIZE(offset)        (0x1c034 + (offset))
>>>>>> +#define MALI_C55_REG_CROP_Y_SIZE(offset)        (0x1c038 + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset) (0x1c040 + (offset))
>>>>>> +#define MALI_C55_SCALER_TIMEOUT_EN            BIT(4)
>>>>>> +#define MALI_C55_SCALER_TIMEOUT(t)            ((t) << 16)
>>>>>> +#define MALI_C55_REG_SCALER_IN_WIDTH(offset) (0x1c044 + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_IN_HEIGHT(offset) (0x1c048 + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_OUT_WIDTH(offset) (0x1c04c + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset) (0x1c050 + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_HFILT_TINC(offset) (0x1c054 + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_HFILT_COEF(offset) (0x1c058 + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_VFILT_TINC(offset) (0x1c05c + (offset))
>>>>>> +#define MALI_C55_REG_SCALER_VFILT_COEF(offset) (0x1c060 + (offset))
>>>>>> +
>>>>>> +#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset) (0x1c064 + (offset))
>>>>>> +#define MALI_C55_GAMMA_ENABLE_MASK            BIT(0)
>>>>>> +#define MALI_C55_REG_GAMMA_GAINS_1(offset)        (0x1c068 + (offset))
>>>>>> +#define MALI_C55_GAMMA_GAIN_R_MASK            GENMASK(11, 0)
>>>>>> +#define MALI_C55_GAMMA_GAIN_G_MASK            GENMASK(27, 16)
>>>>>> +#define MALI_C55_REG_GAMMA_GAINS_2(offset)        (0x1c06c + (offset))
>>>>>> +#define MALI_C55_GAMMA_GAIN_B_MASK            GENMASK(11, 0)
>>>>>> +#define MALI_C55_REG_GAMMA_OFFSETS_1(offset) (0x1c070 + (offset))
>>>>>> +#define MALI_C55_GAMMA_OFFSET_R_MASK            GENMASK(11, 0)
>>>>>> +#define MALI_C55_GAMMA_OFFSET_G_MASK            GENMASK(27, 16)
>>>>>> +#define MALI_C55_REG_GAMMA_OFFSETS_2(offset) (0x1c074 + (offset))
>>>>>> +#define MALI_C55_GAMMA_OFFSET_B_MASK            GENMASK(11, 0)
>>>>>> +
>>>>>> +/* Output DMA Writer */
>>>>>> +
>>>>>> +#define MALI_C55_OUTPUT_DISABLED        0
>>>>>> +#define MALI_C55_OUTPUT_RGB32            1
>>>>>> +#define MALI_C55_OUTPUT_A2R10G10B10        2
>>>>>> +#define MALI_C55_OUTPUT_RGB565            3
>>>>>> +#define MALI_C55_OUTPUT_RGB24            4
>>>>>> +#define MALI_C55_OUTPUT_GEN32            5
>>>>>> +#define MALI_C55_OUTPUT_RAW16            6
>>>>>> +#define MALI_C55_OUTPUT_AYUV            8
>>>>>> +#define MALI_C55_OUTPUT_Y410            9
>>>>>> +#define MALI_C55_OUTPUT_YUY2            10
>>>>>> +#define MALI_C55_OUTPUT_UYVY            11
>>>>>> +#define MALI_C55_OUTPUT_Y210            12
>>>>>> +#define MALI_C55_OUTPUT_NV12_21            13
>>>>>> +#define MALI_C55_OUTPUT_YUV_420_422        17
>>>>>> +#define MALI_C55_OUTPUT_P210_P010        19
>>>>>> +#define MALI_C55_OUTPUT_YUV422            20
>>>>> I'd define this just below the MALI_C55_REG_UV_WRITER_MODE register
>>>>> macro.
>>>>>
>>>>>> +
>>>>>> +#define MALI_C55_OUTPUT_PLANE_ALT0        0
>>>>>> +#define MALI_C55_OUTPUT_PLANE_ALT1        1
>>>>>> +#define MALI_C55_OUTPUT_PLANE_ALT2        2
>>>>> Same here ?
>>>>>
>>>>>> +
>>>>>> +#endif /* _MALI_C55_REGISTERS_H */
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>>>> new file mode 100644
>>>>>> index 000000000000..8edae87f1e5f
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
>>>>>> @@ -0,0 +1,382 @@
>>>>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Resizer Coefficients
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#ifndef _MALI_C55_RESIZER_COEFS_H
>>>>>> +#define _MALI_C55_RESIZER_COEFS_H
>>>>>> +
>>>>>> +#include "mali-c55-common.h"
>>>>>> +
>>>>>> +#define MALI_C55_RESIZER_COEFS_NUM_BANKS    8
>>>>>> +#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES    64
>>>>> Do these belongs to mali-c55-registers.h ?
>>>>>
>>>>>> +
>>>>>> +static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
>>>>>> +    {    /* Bank 0 */
>>>>>> +        0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
>>>>>> +        0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
>>>>>> +        0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
>>>>>> +        0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
>>>>>> +        0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
>>>>>> +        0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
>>>>>> +        0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
>>>>>> +        0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
>>>>>> +        0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
>>>>>> +        0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
>>>>>> +        0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
>>>>>> +        0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
>>>>>> +        0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
>>>>>> +        0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
>>>>>> +        0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
>>>>>> +        0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
>>>>>> +    },
>>>>>> +    {    /* Bank 1 */
>>>>>> +        0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
>>>>>> +        0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
>>>>>> +        0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
>>>>>> +        0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
>>>>>> +        0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
>>>>>> +        0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
>>>>>> +        0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
>>>>>> +        0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
>>>>>> +        0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
>>>>>> +        0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
>>>>>> +        0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
>>>>>> +        0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
>>>>>> +        0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
>>>>>> +        0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
>>>>>> +        0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
>>>>>> +        0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
>>>>>> +    },
>>>>>> +    {    /* Bank 2 */
>>>>>> +        0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
>>>>>> +        0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
>>>>>> +        0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
>>>>>> +        0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
>>>>>> +        0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
>>>>>> +        0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
>>>>>> +        0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
>>>>>> +        0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
>>>>>> +        0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
>>>>>> +        0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
>>>>>> +        0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
>>>>>> +        0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
>>>>>> +        0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
>>>>>> +        0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
>>>>>> +        0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
>>>>>> +        0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
>>>>>> +    },
>>>>>> +    {    /* Bank 3 */
>>>>>> +        0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
>>>>>> +        0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
>>>>>> +        0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
>>>>>> +        0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
>>>>>> +        0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
>>>>>> +        0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
>>>>>> +        0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
>>>>>> +        0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
>>>>>> +        0x20100000, 0x00000010, 0x1f110000, 0x00000010,
>>>>>> +        0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
>>>>>> +        0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
>>>>>> +        0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
>>>>>> +        0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
>>>>>> +        0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
>>>>>> +        0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
>>>>>> +        0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
>>>>>> +    },
>>>>>> +    {    /* Bank 4 */
>>>>>> +        0x17090000, 0x00000917, 0x18090000, 0x00000916,
>>>>>> +        0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
>>>>>> +        0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
>>>>>> +        0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
>>>>>> +        0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
>>>>>> +        0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
>>>>>> +        0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
>>>>>> +        0x190f0300, 0x00000411, 0x18100300, 0x00000411,
>>>>>> +        0x1a100300, 0x00000310, 0x18110400, 0x00000310,
>>>>>> +        0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
>>>>>> +        0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
>>>>>> +        0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
>>>>>> +        0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
>>>>>> +        0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
>>>>>> +        0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
>>>>>> +        0x17160800, 0x0000010a, 0x18160900, 0x00000009,
>>>>>> +    },
>>>>>> +    {    /* Bank 5 */
>>>>>> +        0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
>>>>>> +        0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
>>>>>> +        0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
>>>>>> +        0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
>>>>>> +        0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
>>>>>> +        0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
>>>>>> +        0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
>>>>>> +        0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
>>>>>> +        0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
>>>>>> +        0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
>>>>>> +        0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
>>>>>> +        0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
>>>>>> +        0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
>>>>>> +        0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
>>>>>> +        0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
>>>>>> +        0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
>>>>>> +    },
>>>>>> +    {    /* Bank 6 */
>>>>>> +        0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
>>>>>> +        0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
>>>>>> +        0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
>>>>>> +        0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
>>>>>> +        0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
>>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>>>> +        0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
>>>>>> +        0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>>>> +        0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
>>>>>> +        0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
>>>>>> +        0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
>>>>>> +        0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>>>>> +        0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>>>>> +    },
>>>>>> +    {    /* Bank 7 */
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
>>>>>> +    }
>>>>>> +};
>>>>>> +
>>>>>> +static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
>>>>>> +    {    /* Bank 0 */
>>>>>> +        0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
>>>>>> +        0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
>>>>>> +        0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
>>>>>> +        0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
>>>>>> +        0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
>>>>>> +        0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
>>>>>> +        0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
>>>>>> +        0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
>>>>>> +        0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
>>>>>> +        0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
>>>>>> +        0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
>>>>>> +        0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
>>>>>> +        0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
>>>>>> +        0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
>>>>>> +        0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
>>>>>> +        0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
>>>>>> +    },
>>>>>> +    {    /* Bank 1 */
>>>>>> +        0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
>>>>>> +        0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
>>>>>> +        0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
>>>>>> +        0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
>>>>>> +        0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
>>>>>> +        0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
>>>>>> +        0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
>>>>>> +        0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
>>>>>> +        0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
>>>>>> +        0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
>>>>>> +        0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
>>>>>> +        0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
>>>>>> +        0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
>>>>>> +        0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
>>>>>> +        0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
>>>>>> +        0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
>>>>>> +    },
>>>>>> +    {    /* Bank 2 */
>>>>>> +        0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
>>>>>> +        0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
>>>>>> +        0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
>>>>>> +        0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
>>>>>> +        0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
>>>>>> +        0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
>>>>>> +        0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
>>>>>> +        0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
>>>>>> +        0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
>>>>>> +        0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
>>>>>> +        0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
>>>>>> +        0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
>>>>>> +        0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
>>>>>> +        0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
>>>>>> +        0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
>>>>>> +        0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
>>>>>> +    },
>>>>>> +    {    /* Bank 3 */
>>>>>> +        0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
>>>>>> +        0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
>>>>>> +        0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
>>>>>> +        0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
>>>>>> +        0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
>>>>>> +        0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
>>>>>> +        0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
>>>>>> +        0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
>>>>>> +        0x20100000, 0x00000010, 0x1f100000, 0x00000011,
>>>>>> +        0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
>>>>>> +        0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
>>>>>> +        0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
>>>>>> +        0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
>>>>>> +        0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
>>>>>> +        0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
>>>>>> +        0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
>>>>>> +    },
>>>>>> +    {    /* Bank 4 */
>>>>>> +        0x17170900, 0x00000009, 0x18160900, 0x00000009,
>>>>>> +        0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
>>>>>> +        0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
>>>>>> +        0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
>>>>>> +        0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
>>>>>> +        0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
>>>>>> +        0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
>>>>>> +        0x19110400, 0x0000030f, 0x18110400, 0x00000310,
>>>>>> +        0x1a100300, 0x00000310, 0x18100300, 0x00000411,
>>>>>> +        0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
>>>>>> +        0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
>>>>>> +        0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
>>>>>> +        0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
>>>>>> +        0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
>>>>>> +        0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
>>>>>> +        0x170a0100, 0x00000816, 0x18090000, 0x00000916,
>>>>>> +    },
>>>>>> +    {    /* Bank 5 */
>>>>>> +        0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
>>>>>> +        0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
>>>>>> +        0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
>>>>>> +        0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
>>>>>> +        0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
>>>>>> +        0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
>>>>>> +        0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
>>>>>> +        0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
>>>>>> +        0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
>>>>>> +        0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
>>>>>> +        0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
>>>>>> +        0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
>>>>>> +        0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
>>>>>> +        0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
>>>>>> +        0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
>>>>>> +        0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
>>>>>> +    },
>>>>>> +    {    /* Bank 6 */
>>>>>> +        0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
>>>>>> +        0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
>>>>>> +        0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
>>>>>> +        0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
>>>>>> +        0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
>>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
>>>>>> +        0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>>>>> +        0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
>>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
>>>>>> +        0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
>>>>>> +        0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
>>>>>> +        0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
>>>>>> +        0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
>>>>>> +        0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
>>>>>> +    },
>>>>>> +    {    /* Bank 7 */
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
>>>>>> +    }
>>>>>> +};
>>>>>> +
>>>>>> +struct mali_c55_resizer_coef_bank {
>>>>>> +    unsigned int bank;
>>>>> This is always equal to the index of the entry in the
>>>>> mali_c55_coefficient_banks array, you can drop it.
>>>>>
>>>>>> +    unsigned int top;
>>>>>> +    unsigned int bottom;
>>>>> The bottom value of bank N is always equal to the top value of bank N+1.
>>>>> You can simplify this by storing a single value.
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
>>>>>> +    {
>>>>>> +        .bank = 0,
>>>>>> +        .top = 1000,
>>>>>> +        .bottom = 770,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .bank = 1,
>>>>>> +        .top = 769,
>>>>>> +        .bottom = 600,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .bank = 2,
>>>>>> +        .top = 599,
>>>>>> +        .bottom = 460,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .bank = 3,
>>>>>> +        .top = 459,
>>>>>> +        .bottom = 354,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .bank = 4,
>>>>>> +        .top = 353,
>>>>>> +        .bottom = 273,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .bank = 5,
>>>>>> +        .top = 272,
>>>>>> +        .bottom = 210,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .bank = 6,
>>>>>> +        .top = 209,
>>>>>> +        .bottom = 162,
>>>>>> +    },
>>>>>> +    {
>>>>>> +        .bank = 7,
>>>>>> +        .top = 161,
>>>>>> +        .bottom = 125,
>>>>>> +    },
>>>>>> +};
>>>>>> +
>>>>> A small comment would be nice, such as
>>>>>
>>>>> /* Select a bank of resizer coefficients, based on the scaling ratio. */
>>>>>
>>>>>> +static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,
>>>>> This function is related to the resizers. Add "rsz" somewhere in the
>>>>> function name, and pass a resizer pointer.
>>>>>
>>>>>> +                        unsigned int crop,
>>>>>> +                        unsigned int scale)
>>>>> I think those are the input and output sizes to the scaler. Rename them
>>>>> to make it clearer.
>>>>>
>>>>>> +{
>>>>>> +    unsigned int tmp;
>>>>> tmp is almost always a bad variable name. Please use a more descriptive
>>>>> name, size as rsz_ratio.
>>>>>
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    tmp = (scale * 1000U) / crop;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
>>>>>> +        if (tmp >= mali_c55_coefficient_banks[i].bottom &&
>>>>>> +            tmp <= mali_c55_coefficient_banks[i].top)
>>>>>> +            return mali_c55_coefficient_banks[i].bank;
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * We shouldn't ever get here, in theory. As we have no good choices
>>>>>> +     * simply warn the user and use the first bank of coefficients.
>>>>>> +     */
>>>>>> +    dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
>>>>>> +    return 0;
>>>>>> +}
>>>>> And everything else belongs to mali-c55-resizer.c. Drop this header
>>>>> file.
>>>>>
>>>>>> +
>>>>>> +#endif /* _MALI_C55_RESIZER_COEFS_H */
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>>>> new file mode 100644
>>>>>> index 000000000000..0a5a2969d3ce
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
>>>>>> @@ -0,0 +1,779 @@
>>>>>> +// SPDX-License-Identifier: GPL-2.0
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Image signal processor
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#include <linux/math.h>
>>>>>> +#include <linux/minmax.h>
>>>>>> +
>>>>>> +#include <media/media-entity.h>
>>>>>> +#include <media/v4l2-subdev.h>
>>>>>> +
>>>>>> +#include "mali-c55-common.h"
>>>>>> +#include "mali-c55-registers.h"
>>>>>> +#include "mali-c55-resizer-coefs.h"
>>>>>> +
>>>>>> +/* Scaling factor in Q4.20 format. */
>>>>>> +#define MALI_C55_RZR_SCALER_FACTOR    (1U << 20)
>>>>>> +
>>>>>> +static const u32 rzr_non_bypass_src_fmts[] = {
>>>>>> +    MEDIA_BUS_FMT_RGB121212_1X36,
>>>>>> +    MEDIA_BUS_FMT_YUV10_1X30
>>>>>> +};
>>>>>> +
>>>>>> +static const char * const mali_c55_resizer_names[] = {
>>>>>> +    [MALI_C55_RZR_FR] = "resizer fr",
>>>>>> +    [MALI_C55_RZR_DS] = "resizer ds",
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
>>>>>> +                     struct v4l2_subdev_state *state)
>>>>>> +{
>>>>>> +    unsigned int reg_offset = rzr->cap_dev->reg_offset;
>>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>>>> +    struct v4l2_mbus_framefmt *fmt;
>>>>>> +    struct v4l2_rect *crop;
>>>> const
>>>>
>>>>>> +
>>>>>> +    /* Verify if crop should be enabled. */
>>>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
>>>>>> +
>>>>>> +    if (fmt->width == crop->width && fmt->height == crop->height)
>>>>>> +        return MALI_C55_BYPASS_CROP;
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
>>>>>> +               crop->left);
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
>>>>>> +               crop->top);
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
>>>>>> +               crop->width);
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
>>>>>> +               crop->height);
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
>>>>>> +               MALI_C55_CROP_ENABLE);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
>>>>>> +                    struct v4l2_subdev_state *state)
>>>>>> +{
>>>>>> +    unsigned int reg_offset = rzr->cap_dev->reg_offset;
>>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>>>> +    struct v4l2_rect *crop, *scale;
>>>> const
>>>>
>>>> Once "[PATCH v4 0/3] media: v4l2-subdev: Support const-awareness in
>>>> state accessors" gets merged, the state argument to this function can be
>>>> made const too. Same for other functions, as applicable.
>>>>
>>>>>> +    unsigned int h_bank, v_bank;
>>>>>> +    u64 h_scale, v_scale;
>>>>>> +
>>>>>> +    /* Verify if scaling should be enabled. */
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
>>>>>> +    scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
>>>>>> +
>>>>>> +    if (crop->width == scale->width && crop->height == scale->height)
>>>>>> +        return MALI_C55_BYPASS_SCALER;
>>>>>> +
>>>>>> +    /* Program the V/H scaling factor in Q4.20 format. */
>>>>>> +    h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
>>>>>> +    v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
>>>>>> +
>>>>>> +    do_div(h_scale, scale->width);
>>>>>> +    do_div(v_scale, scale->height);
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
>>>>>> +               crop->width);
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
>>>>>> +               crop->height);
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
>>>>>> +               scale->width);
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
>>>>>> +               scale->height);
>>>>>> +
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
>>>>>> +               h_scale);
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
>>>>>> +               v_scale);
>>>>>> +
>>>>>> +    h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
>>>>>> +                         scale->width);
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
>>>>>> +               h_bank);
>>>>>> +
>>>>>> +    v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
>>>>>> +                         scale->height);
>>>>>> +    mali_c55_write(mali_c55,
>>>>>> +               MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
>>>>>> +               v_bank);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
>>>>>> +                 struct v4l2_subdev_state *state)
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>>>> +    u32 bypass = 0;
>>>>>> +
>>>>>> +    /* Verify if cropping and scaling should be enabled. */
>>>>>> +    bypass |= mali_c55_rzr_program_crop(rzr, state);
>>>>>> +    bypass |= mali_c55_rzr_program_resizer(rzr, state);
>>>>>> +
>>>>>> +    mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
>>>>>> +                 MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
>>>>>> +                 MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
>>>>>> +                 bypass);
>>>>>> +}
>>>>>> +
>>>>>> +/*
>>>>>> + * Inspect the routing table to know which of the two (mutually exclusive)
>>>>>> + * routes is enabled and return the sink pad id of the active route.
>>>>>> + */
>>>>>> +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
>>>>>> +{
>>>>>> +    struct v4l2_subdev_krouting *routing = &state->routing;
>>>>>> +    struct v4l2_subdev_route *route;
>>>>>> +
>>>>>> +    /* A single route is enabled at a time. */
>>>>>> +    for_each_active_route(routing, route)
>>>>>> +        return route->sink_pad;
>>>>>> +
>>>>>> +    return MALI_C55_RZR_SINK_PAD;
>>>>>> +}
>>>>>> +
>>>>>> +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
>>>>>> +{
>>>>>> +    u32 corrected_code = 0;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The ISP takes input in a 20-bit format, but can only output 16-bit
>>>>>> +     * RAW bayer data (with the 4 least significant bits from the input
>>>>>> +     * being lost). Return the 16-bit version of the 20-bit input formats.
>>>>>> +     */
>>>>>> +    switch (mbus_code) {
>>>>>> +    case MEDIA_BUS_FMT_SBGGR20_1X20:
>>>>>> +        corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
>>>>>> +        break;
>>>>>> +    case MEDIA_BUS_FMT_SGBRG20_1X20:
>>>>>> +        corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
>>>>>> +        break;
>>>>>> +    case MEDIA_BUS_FMT_SGRBG20_1X20:
>>>>>> +        corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
>>>>>> +        break;
>>>>>> +    case MEDIA_BUS_FMT_SRGGB20_1X20:
>>>>>> +        corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
>>>>>> +        break;
>>>> Would it make sense to add the shifted code to mali_c55_isp_fmt ?
>>>>
>>>>>> +    }
>>>>>> +
>>>>>> +    return corrected_code;
>>>>>> +}
>>>>>> +
>>>>>> +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
>>>>>> +                      struct v4l2_subdev_state *state,
>>>>>> +                      struct v4l2_subdev_krouting *routing)
>>>> I think the last argument can be const.
>>>>
>>>>>> +{
>>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>>>> +                            sd);
>>>> A to_mali_c55_resizer() static inline function would be useful. Same for
>>>> other components, where applicable.
>>>>
>>>>>> +    unsigned int active_sink = UINT_MAX;
>>>>>> +    struct v4l2_mbus_framefmt *src_fmt;
>>>>>> +    struct v4l2_rect *crop, *compose;
>>>>>> +    struct v4l2_subdev_route *route;
>>>>>> +    unsigned int active_routes = 0;
>>>>>> +    struct v4l2_mbus_framefmt *fmt;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    ret = v4l2_subdev_routing_validate(sd, routing, 0);
>>>>>> +    if (ret)
>>>>>> +        return ret;
>>>>>> +
>>>>>> +    /* Only a single route can be enabled at a time. */
>>>>>> +    for_each_active_route(routing, route) {
>>>>>> +        if (++active_routes > 1) {
>>>>>> +            dev_err(rzr->mali_c55->dev,
>>>>>> +                "Only one route can be active");
>>>> No kernel log message with a level higher than dev_dbg() from
>>>> user-controlled paths please, here and where applicable. This is to
>>>> avoid giving applications an easy way to flood the kernel log.
>>>>
>>>>>> +            return -EINVAL;
>>>>>> +        }
>>>>>> +
>>>>>> +        active_sink = route->sink_pad;
>>>>>> +    }
>>>>>> +    if (active_sink == UINT_MAX) {
>>>>>> +        dev_err(rzr->mali_c55->dev, "One route has to be active");
>>>>>> +        return -EINVAL;
>>>>>> +    }
>>>> The recommended handling of invalid routing is to adjust the routing
>>>> table, not to return errors.
>>>>
>>>>>> +
>>>>>> +    ret = v4l2_subdev_set_routing(sd, state, routing);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
>>>>>> +        return ret;
>>>>>> +    }
>>>>>> +
>>>>>> +    fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
>>>>>> +    compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
>>>>>> +
>>>>>> +    fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>>>> +    fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>>>> +    fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>> There are other colorspace-related fields.
>>>>
>>>>>> +    fmt->field = V4L2_FIELD_NONE;
>>>> I wonder if we should really update the sink pad format, or just
>>>> propagate it. If we update it, I think it should be set to defaults on
>>>> both sink pads, not just the active sink pad.
>>>>
>>>>>> +
>>>>>> +    if (active_sink == MALI_C55_RZR_SINK_PAD) {
>>>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>>>> +
>>>>>> +        crop->left = crop->top = 0;
>>>>          crop->left = 0;
>>>>          crop->top = 0;
>>>>
>>>>>> +        crop->width = MALI_C55_DEFAULT_WIDTH;
>>>>>> +        crop->height = MALI_C55_DEFAULT_HEIGHT;
>>>>>> +
>>>>>> +        *compose = *crop;
>>>>>> +    } else {
>>>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* Propagate the format to the source pad */
>>>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
>>>>>> +                           0);
>>>>>> +    *src_fmt = *fmt;
>>>>>> +
>>>>>> +    /* In the event this is the bypass pad the mbus code needs correcting */
>>>>>> +    if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
>>>>>> +        src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>> +                       struct v4l2_subdev_state *state,
>>>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
>>>>>> +{
>>>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
>>>>>> +    const struct mali_c55_isp_fmt *fmt;
>>>>>> +    unsigned int index = 0;
>>>>>> +    u32 sink_pad;
>>>>>> +
>>>>>> +    switch (code->pad) {
>>>>>> +    case MALI_C55_RZR_SINK_PAD:
>>>>>> +        if (code->index)
>>>>>> +            return -EINVAL;
>>>>>> +
>>>>>> +        code->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    case MALI_C55_RZR_SOURCE_PAD:
>>>>>> +        sink_pad = mali_c55_rzr_get_active_sink(state);
>>>>>> +        sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
>>>>>> +
>>>>>> +        /*
>>>>>> +         * If the active route is from the Bypass sink pad, then the
>>>>>> +         * source pad is a simple passthrough of the sink format,
>>>>>> +         * downshifted to 16-bits.
>>>>>> +         */
>>>>>> +
>>>>>> +        if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>>>>> +            if (code->index)
>>>>>> +                return -EINVAL;
>>>>>> +
>>>>>> +            code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
>>>>>> +            if (!code->code)
>>>>>> +                return -EINVAL;
>>>>>> +
>>>>>> +            return 0;
>>>>>> +        }
>>>>>> +
>>>>>> +        /*
>>>>>> +         * If the active route is from the non-bypass sink then we can
>>>>>> +         * select either RGB or conversion to YUV.
>>>>>> +         */
>>>>>> +
>>>>>> +        if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
>>>>>> +            return -EINVAL;
>>>>>> +
>>>>>> +        code->code = rzr_non_bypass_src_fmts[code->index];
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    case MALI_C55_RZR_SINK_BYPASS_PAD:
>>>>>> +        for_each_mali_isp_fmt(fmt) {
>>>>>> +            if (index++ == code->index) {
>>>>>> +                code->code = fmt->code;
>>>>>> +                return 0;
>>>>>> +            }
>>>>>> +        }
>>>>>> +
>>>>>> +        break;
>>>>>> +    }
>>>>>> +
>>>>>> +    return -EINVAL;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
>>>>>> +                    struct v4l2_subdev_state *state,
>>>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
>>>>>> +{
>>>>>> +    if (fse->index)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
>>>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
>>>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
>>>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
>>>>>> +                     struct v4l2_subdev_state *state,
>>>>>> +                     struct v4l2_subdev_format *format)
>>>>>> +{
>>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
>>>>>> +    struct v4l2_rect *rect;
>>>>>> +    unsigned int sink_pad;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Clamp to min/max and then reset crop and compose rectangles to the
>>>>>> +     * newly applied size.
>>>>>> +     */
>>>>>> +    clamp_t(unsigned int, fmt->width,
>>>>>> +        MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>>>>> +    clamp_t(unsigned int, fmt->height,
>>>>>> +        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>>> Please check comments for other components related to the colorspace
>>>> fields, to decide how to handle them here.
>>>>
>>>>>> +
>>>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
>>>>>> +    if (sink_pad == MALI_C55_RZR_SINK_PAD) {
>>>> The selection here should depend on format->pad, not the active sink
>>>> pad.
>>>>
>>>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>>>> +
>>>>>> +        rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
>>>>>> +        rect->left = 0;
>>>>>> +        rect->top = 0;
>>>>>> +        rect->width = fmt->width;
>>>>>> +        rect->height = fmt->height;
>>>>>> +
>>>>>> +        rect = v4l2_subdev_state_get_compose(state,
>>>>>> +                             MALI_C55_RZR_SINK_PAD);
>>>>>> +        rect->left = 0;
>>>>>> +        rect->top = 0;
>>>>>> +        rect->width = fmt->width;
>>>>>> +        rect->height = fmt->height;
>>>>>> +    } else {
>>>>>> +        /*
>>>>>> +         * Make sure the media bus code is one of the supported
>>>>>> +         * ISP input media bus codes.
>>>>>> +         */
>>>>>> +        if (!mali_c55_isp_is_format_supported(fmt->code))
>>>>>> +            fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
>>>>>> +    }
>>>>>> +
>>>>>> +    *v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
>>>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
>>>> Propagation to the source pad, however, should depend on the active
>>>> route. If format->pad is routed to the source pad, you should propagate,
>>>> otherwise, you shouldn't.
>>>>
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
>>>>>> +                       struct v4l2_subdev_state *state,
>>>>>> +                       struct v4l2_subdev_format *format)
>>>>>> +{
>>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>>>> +                            sd);
>>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
>>>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
>>>>>> +    struct v4l2_rect *crop, *compose;
>>>>>> +    unsigned int sink_pad;
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
>>>>>> +    sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
>>>>>> +    compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
>>>>>> +
>>>>>> +    /* FR Bypass pipe. */
>>>>>> +
>>>>>> +    if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>>>>> +        /*
>>>>>> +         * Format on the source pad is the same as the one on the
>>>>>> +         * sink pad, downshifted to 16-bits.
>>>>>> +         */
>>>>>> +        fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
>>>>>> +        if (!fmt->code)
>>>>>> +            return -EINVAL;
>>>>>> +
>>>>>> +        /* RAW bypass disables scaling and cropping. */
>>>>>> +        crop->top = compose->top = 0;
>>>>>> +        crop->left = compose->left = 0;
>>>>>> +        fmt->width = crop->width = compose->width = sink_fmt->width;
>>>>>> +        fmt->height = crop->height = compose->height = sink_fmt->height;
>>>> I don't think this is right. This function sets the format on the source
>>>> pad. Subdevs should propagate formats from the sink to the source, not
>>>> the other way around.
>>>>
>>>> The only parameter that can be modified on the source pad (as far as I
>>>> understand) is the media bus code. In the bypass path, I understand it's
>>>> fixed, while in the other path, you can select between RGB and YUV. I
>>>> think the following code is what you need to implement this function.
>>>>
>>>> static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
>>>>                         struct v4l2_subdev_state *state,
>>>>                         struct v4l2_subdev_format *format)
>>>> {
>>>>      struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>>                              sd);
>>>>      struct v4l2_mbus_framefmt *fmt;
>>>>
>>>>      fmt = v4l2_subdev_state_get_format(state, format->pad);
>>>>
>>>>      /* In the non-bypass path the output format can be selected. */
>>>>      if (mali_c55_rzr_get_active_sink(state) == MALI_C55_RZR_SINK_PAD) {
>>>>          unsigned int i;
>>>>
>>>>          fmt->code = format->format.code;
>>>>
>>>>          for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
>>>>              if (fmt->code == rzr_non_bypass_src_fmts[i])
>>>>                  break;
>>>>          }
>>>>
>>>>          if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts))
>>>>              fmt->code = rzr_non_bypass_src_fmts[0];
>>>>      }
>>>>
>>>>      format->format = *fmt;
>>>>
>>>>      return 0;
>>>> }
>>>>
>>>>>> +
>>>>>> +        *v4l2_subdev_state_get_format(state,
>>>>>> +                          MALI_C55_RZR_SOURCE_PAD) = *fmt;
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* Regular processing pipe. */
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
>>>>>> +        if (fmt->code == rzr_non_bypass_src_fmts[i])
>>>>>> +            break;
>>>>>> +    }
>>>>>> +
>>>>>> +    if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
>>>>>> +        dev_dbg(rzr->mali_c55->dev,
>>>>>> +            "Unsupported mbus code 0x%x: using default\n",
>>>>>> +            fmt->code);
>>>> I think you can drop this message.
>>>>
>>>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The source pad format size comes directly from the sink pad
>>>>>> +     * compose rectangle.
>>>>>> +     */
>>>>>> +    fmt->width = compose->width;
>>>>>> +    fmt->height = compose->height;
>>>>>> +
>>>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
>>>>>> +                struct v4l2_subdev_state *state,
>>>>>> +                struct v4l2_subdev_format *format)
>>>>>> +{
>>>>>> +    /*
>>>>>> +     * On sink pads fmt is either fixed for the 'regular' processing
>>>>>> +     * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
>>>>>> +     * pad.
>>>>>> +     *
>>>>>> +     * On source pad sizes are the result of crop+compose on the sink
>>>>>> +     * pad sizes, while the format depends on the active route.
>>>>>> +     */
>>>>>> +
>>>>>> +    if (format->pad != MALI_C55_RZR_SOURCE_PAD)
>>>>>> +        return mali_c55_rzr_set_sink_fmt(sd, state, format);
>>>>>> +
>>>>>> +    return mali_c55_rzr_set_source_fmt(sd, state, format);
>>>> Nitpicking,
>>>>
>>>>      if (format->pad == MALI_C55_RZR_SOURCE_PAD)
>>>>          return mali_c55_rzr_set_source_fmt(sd, state, format);
>>>>
>>>>      return mali_c55_rzr_set_sink_fmt(sd, state, format);
>>>>
>>>> to match SOURCE_PAD and source_fmt.
>>>>
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
>>>>>> +                      struct v4l2_subdev_state *state,
>>>>>> +                      struct v4l2_subdev_selection *sel)
>>>>>> +{
>>>>>> +    if (sel->pad != MALI_C55_RZR_SINK_PAD)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    if (sel->target != V4L2_SEL_TGT_CROP &&
>>>>>> +        sel->target != V4L2_SEL_TGT_COMPOSE)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    sel->r = sel->target == V4L2_SEL_TGT_CROP
>>>>>> +           ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
>>>>>> +           : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
>>>>>> +                      struct v4l2_subdev_state *state,
>>>>>> +                      struct v4l2_subdev_selection *sel)
>>>>>> +{
>>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>>>> +                            sd);
>>>>>> +    struct v4l2_mbus_framefmt *source_fmt;
>>>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
>>>>>> +    struct v4l2_rect *crop, *compose;
>>>>>> +
>>>>>> +    if (sel->pad != MALI_C55_RZR_SINK_PAD)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    if (sel->target != V4L2_SEL_TGT_CROP &&
>>>>>> +        sel->target != V4L2_SEL_TGT_COMPOSE)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    source_fmt = v4l2_subdev_state_get_format(state,
>>>>>> +                          MALI_C55_RZR_SOURCE_PAD);
>>>>>> +    sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
>>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
>>>>>> +    compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
>>>>>> +
>>>>>> +    /* RAW bypass disables crop/scaling. */
>>>>>> +    if (mali_c55_format_is_raw(source_fmt->code)) {
>>>>>> +        crop->top = compose->top = 0;
>>>>>> +        crop->left = compose->left = 0;
>>>>>> +        crop->width = compose->width = sink_fmt->width;
>>>>>> +        crop->height = compose->height = sink_fmt->height;
>>>>>> +
>>>>>> +        sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* During streaming, it is allowed to only change the crop rectangle. */
>>>>>> +    if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +     /*
>>>>>> +      * Update the desired target and then clamp the crop rectangle to the
>>>>>> +      * sink format sizes and the compose size to the crop sizes.
>>>>>> +      */
>>>>>> +    if (sel->target == V4L2_SEL_TGT_CROP)
>>>>>> +        *crop = sel->r;
>>>>>> +    else
>>>>>> +        *compose = sel->r;
>>>>>> +
>>>>>> +    clamp_t(unsigned int, crop->left, 0, sink_fmt->width);
>>>>>> +    clamp_t(unsigned int, crop->top, 0, sink_fmt->height);
>>>>>> +    clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
>>>>>> +        sink_fmt->width - crop->left);
>>>>>> +    clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
>>>>>> +        sink_fmt->height - crop->top);
>>>>>> +
>>>>>> +    if (rzr->streaming) {
>>>>>> +        /*
>>>>>> +         * Apply at runtime a crop rectangle on the resizer's sink only
>>>>>> +         * if it doesn't require re-programming the scaler output sizes
>>>>>> +         * as it would require changing the output buffer sizes as well.
>>>>>> +         */
>>>>>> +        if (sel->r.width < compose->width ||
>>>>>> +            sel->r.height < compose->height)
>>>>>> +            return -EINVAL;
>>>>>> +
>>>>>> +        *crop = sel->r;
>>>>>> +        mali_c55_rzr_program(rzr, state);
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    compose->left = 0;
>>>>>> +    compose->top = 0;
>>>>>> +    clamp_t(unsigned int, compose->left, 0, sink_fmt->width);
>>>>>> +    clamp_t(unsigned int, compose->top, 0, sink_fmt->height);
>>>>>> +    clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
>>>>>> +    clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
>>>>>> +
>>>>>> +    sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
>>>>>> +                    struct v4l2_subdev_state *state,
>>>>>> +                    enum v4l2_subdev_format_whence which,
>>>>>> +                    struct v4l2_subdev_krouting *routing)
>>>>>> +{
>>>>>> +    if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
>>>>>> +        media_entity_is_streaming(&sd->entity))
>>>>>> +        return -EBUSY;
>>>>>> +
>>>>>> +    return __mali_c55_rzr_set_routing(sd, state, routing);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
>>>>>> +    .enum_mbus_code        = mali_c55_rzr_enum_mbus_code,
>>>>>> +    .enum_frame_size    = mali_c55_rzr_enum_frame_size,
>>>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
>>>>>> +    .set_fmt        = mali_c55_rzr_set_fmt,
>>>>>> +    .get_selection        = mali_c55_rzr_get_selection,
>>>>>> +    .set_selection        = mali_c55_rzr_set_selection,
>>>>>> +    .set_routing        = mali_c55_rzr_set_routing,
>>>>>> +};
>>>>>> +
>>>>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
>>>> Could this be handled through the .enable_streams() and
>>>> .disable_streams() operations ? They ensure that the stream state stored
>>>> internal is correct. That may not matter much today, but I think it will
>>>> become increasingly important in the future for the V4L2 core.
>>>>
>>>>>> +{
>>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
>>>>>> +    struct v4l2_subdev *sd = &rzr->sd;
>>>>>> +    struct v4l2_subdev_state *state;
>>>>>> +    unsigned int sink_pad;
>>>>>> +
>>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
>>>>>> +
>>>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
>>>>>> +    if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
>>>>>> +        /* Bypass FR pipe processing if the bypass route is active. */
>>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>>>>> + MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
>>>>>> + MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
>>>>>> +        goto unlock_state;
>>>>>> +    }
>>>>>> +
>>>>>> +    /* Disable bypass and use regular processing. */
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
>>>>>> +                 MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
>>>>>> +    mali_c55_rzr_program(rzr, state);
>>>>>> +
>>>>>> +unlock_state:
>>>>>> +    rzr->streaming = true;
>>>> And hopefully you'll be able to replace this with
>>>> v4l2_subdev_is_streaming(), introduced in "[PATCH v6 00/11] media:
>>>> subdev: Improve stream enable/disable machinery" (Sakari has sent a pull
>>>> request for v6.11 yesterday).
>>>>
>>>>>> +    v4l2_subdev_unlock_state(state);
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
>>>>>> +{
>>>>>> +    struct v4l2_subdev *sd = &rzr->sd;
>>>>>> +    struct v4l2_subdev_state *state;
>>>>>> +
>>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
>>>>>> +    rzr->streaming = false;
>>>>>> +    v4l2_subdev_unlock_state(state);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
>>>>>> +    .pad    = &mali_c55_resizer_pad_ops,
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
>>>>>> +                   struct v4l2_subdev_state *state)
>>>>>> +{
>>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
>>>>>> +                            sd);
>>>>>> +    struct v4l2_subdev_krouting routing = { };
>>>>>> +    struct v4l2_subdev_route *routes;
>>>>>> +    unsigned int i;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
>>>>>> +    if (!routes)
>>>>>> +        return -ENOMEM;
>>>>>> +
>>>>>> +    for (i = 0; i < rzr->num_routes; ++i) {
>>>>>> +        struct v4l2_subdev_route *route = &routes[i];
>>>>>> +
>>>>>> +        route->sink_pad = i
>>>>>> +                ? MALI_C55_RZR_SINK_BYPASS_PAD
>>>>>> +                : MALI_C55_RZR_SINK_PAD;
>>>>>> +        route->source_pad = MALI_C55_RZR_SOURCE_PAD;
>>>>>> +        if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
>>>>>> +            route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
>>>>>> +    }
>>>>>> +
>>>>>> +    routing.num_routes = rzr->num_routes;
>>>>>> +    routing.routes = routes;
>>>>>> +
>>>>>> +    ret = __mali_c55_rzr_set_routing(sd, state, &routing);
>>>>>> +    kfree(routes);
>>>>>> +
>>>>>> +    return ret;
>>>> I think this could be simplified.
>>>>
>>>>      struct v4l2_subdev_route routes[2] = {
>>>>          {
>>>>              .sink_pad = MALI_C55_RZR_SINK_PAD,
>>>>              .source_pad = MALI_C55_RZR_SOURCE_PAD,
>>>>              .flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
>>>>          }, {
>>>>              .sink_pad = MALI_C55_RZR_SINK_BYPASS_PAD,
>>>>              .source_pad = MALI_C55_RZR_SOURCE_PAD,
>>>>          },
>>>>      };
>>>>      struct v4l2_subdev_krouting routing = {
>>>>          .num_routes = rzr->num_routes,
>>>>          .routes = routes,
>>>>      };
>>>>
>>>>      return __mali_c55_rzr_set_routing(sd, state, &routing);
>>>>
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
>>>>>> +    .init_state = mali_c55_rzr_init_state,
>>>>>> +};
>>>>>> +
>>>>>> +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
>>>>>> +                          unsigned int index)
>>>>>> +{
>>>>>> +    const unsigned int scaler_filt_coefmem_addrs[][2] = {
>>>>>> +        [MALI_C55_RZR_FR] = {
>>>>>> +            0x034A8, /* hfilt */
>>>>>> +            0x044A8  /* vfilt */
>>>>> Lowercase hex constants.
>>>> And addresses belong to the mali-c55-registers.h file.
>>>>
>>>>>> +        },
>>>>>> +        [MALI_C55_RZR_DS] = {
>>>>>> +            0x014A8, /* hfilt */
>>>>>> +            0x024A8  /* vfilt */
>>>>>> +        },
>>>>>> +    };
>>>>>> +    unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
>>>>>> +    unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
>>>>>> +    unsigned int i, j;
>>>>>> +
>>>>>> +    for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
>>>>>> +        for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
>>>>>> +            mali_c55_write(mali_c55, haddr,
>>>>>> + mali_c55_scaler_h_filter_coefficients[i][j]);
>>>>>> +            mali_c55_write(mali_c55, vaddr,
>>>>>> + mali_c55_scaler_v_filter_coefficients[i][j]);
>>>>>> +
>>>>>> +            haddr += sizeof(u32);
>>>>>> +            vaddr += sizeof(u32);
>>>>>> +        }
>>>>>> +    }
>>>> How about memcpy_toio() ? I suppose this function isn't
>>>> performance sensitive, so maybe usage of mali_c55_write() is better from
>>>> a consistency point of view.
>>>>
>>>>>> +}
>>>>>> +
>>>>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
>>>> Moving the inner content to a separate mali_c55_register_resizer()
>>>> function would increase readability I think, and remove usage of gotos.
>>>> I would probably do the same for unregistration too, for consistency.
>>>>
>>>>>> +        struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
>>>>>> +        struct v4l2_subdev *sd = &rzr->sd;
>>>>>> +        unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
>>>>>> +
>>>>>> +        rzr->id = i;
>>>>>> +        rzr->streaming = false;
>>>>>> +
>>>>>> +        if (rzr->id == MALI_C55_RZR_FR)
>>>>>> +            rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
>>>>>> +        else
>>>>>> +            rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
>>>>>> +
>>>>>> +        mali_c55_resizer_program_coefficients(mali_c55, i);
>>>> Should this be done at stream start, given that power may be cut off
>>>> between streaming sessions ?
>>>>
>>>>>> +
>>>>>> +        v4l2_subdev_init(sd, &mali_c55_resizer_ops);
>>>>>> +        sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
>>>>>> +                 | V4L2_SUBDEV_FL_STREAMS;
>>>>>> +        sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
>>>>>> +        sd->internal_ops = &mali_c55_resizer_internal_ops;
>>>>>> +        snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
>>>>          snprintf(sd->name, ARRAY_SIZE(sd->name), "%s resizer %s",
>>>>
>>>> and drop the "resizer " prefix from mali_c55_resizer_names. You can also
>>>> make mali_c55_resizer_names a local static const variable.
>>>>
>>>>>> +             MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
>>>>>> +
>>>>>> +        rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
>>>>>> +        rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
>>>>>> +
>>>>>> +        /* Only the FR pipe has a bypass pad. */
>>>>>> +        if (rzr->id == MALI_C55_RZR_FR) {
>>>>>> + rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
>>>>>> +                            MEDIA_PAD_FL_SINK;
>>>>>> +            rzr->num_routes = 2;
>>>>>> +        } else {
>>>>>> +            num_pads -= 1;
>>>>>> +            rzr->num_routes = 1;
>>>>>> +        }
>>>>>> +
>>>>>> +        ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
>>>>>> +        if (ret)
>>>>>> +            return ret;
>>>>>> +
>>>>>> +        ret = v4l2_subdev_init_finalize(sd);
>>>>>> +        if (ret)
>>>>>> +            goto err_cleanup;
>>>>>> +
>>>>>> +        ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>>>>> +        if (ret)
>>>>>> +            goto err_cleanup;
>>>>>> +
>>>>>> +        rzr->mali_c55 = mali_c55;
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_cleanup:
>>>>>> +    for (; i >= 0; --i) {
>>>>>> +        struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
>>>>>> +        struct v4l2_subdev *sd = &rzr->sd;
>>>>>> +
>>>>>> +        v4l2_subdev_cleanup(sd);
>>>>>> +        media_entity_cleanup(&sd->entity);
>>>>>> +    }
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
>>>>>> +        struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
>>>>>> +
>>>>>> +        if (!resizer->mali_c55)
>>>>>> +            continue;
>>>>>> +
>>>>>> +        v4l2_device_unregister_subdev(&resizer->sd);
>>>>>> +        v4l2_subdev_cleanup(&resizer->sd);
>>>>>> +        media_entity_cleanup(&resizer->sd.entity);
>>>>>> +    }
>>>>>> +}
>>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>>>>> new file mode 100644
>>>>>> index 000000000000..c7e699741c6d
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
>>>>>> @@ -0,0 +1,402 @@
>>>>>> +// SPDX-License-Identifier: GPL-2.0
>>>>>> +/*
>>>>>> + * ARM Mali-C55 ISP Driver - Test pattern generator
>>>>>> + *
>>>>>> + * Copyright (C) 2024 Ideas on Board Oy
>>>>>> + */
>>>>>> +
>>>>>> +#include <linux/minmax.h>
>>>>>> +#include <linux/string.h>
>>>>>> +
>>>>>> +#include <media/media-entity.h>
>>>>>> +#include <media/v4l2-ctrls.h>
>>>>>> +#include <media/v4l2-subdev.h>
>>>>>> +
>>>>>> +#include "mali-c55-common.h"
>>>>>> +#include "mali-c55-registers.h"
>>>>>> +
>>>>>> +#define MALI_C55_TPG_SRC_PAD        0
>>>>>> +#define MALI_C55_TPG_FIXED_HBLANK    0x20
>>>>>> +#define MALI_C55_TPG_MAX_VBLANK        0xFFFF
>>>>> Lowercase hex constants.
>>>>>
>>>>>> +#define MALI_C55_TPG_PIXEL_RATE        100000000
>>>>> This should be exposed to applications using the V4L2_CID_PIXEL_RATE
>>>>> control (read-only).
>>>>>
>>>>>> +
>>>>>> +static const char * const mali_c55_tpg_test_pattern_menu[] = {
>>>>>> +    "Flat field",
>>>>>> +    "Horizontal gradient",
>>>>>> +    "Vertical gradient",
>>>>>> +    "Vertical bars",
>>>>>> +    "Arbitrary rectangle",
>>>>>> +    "White frame on black field"
>>>>>> +};
>>>>>> +
>>>>>> +static const u32 mali_c55_tpg_mbus_codes[] = {
>>>>>> +    MEDIA_BUS_FMT_SRGGB20_1X20,
>>>>>> +    MEDIA_BUS_FMT_RGB202020_1X60,
>>>>>> +};
>>>>>> +
>>>>>> +static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
>>>>>> +                       int *def_vblank, int *min_vblank)
>>>>> unsigned int ?
>>>>>
>>>>>> +{
>>>>>> +    unsigned int hts;
>>>>>> +    int tgt_fps;
>>>>>> +    int vblank;
>>>>>> +
>>>>>> +    hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The ISP has minimum vertical blanking requirements that must be
>>>>>> +     * adhered to by the TPG. The minimum is a function of the Iridix blocks
>>>>>> +     * clocking requirements and the width of the image and horizontal
>>>>>> +     * blanking, but if we assume the worst case iVariance and sVariance
>>>>>> +     * values then it boils down to the below.
>>>>>> +     */
>>>>>> +    *min_vblank = 15 + (120500 / hts);
>>>>> I wonder if this should round up.
>>>>>
>>>>>> +
>>>>>> +    /*
>>>>>> +     * We need to set a sensible default vblank for whatever format height
>>>>>> +     * we happen to be given from set_fmt(). This function just targets
>>>>>> +     * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
>>>>>> +     * If we can't get 5fps we'll take whatever the minimum vblank gives us.
>>>>>> +     */
>>>>>> +    tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
>>>>>> +
>>>>>> +    if (tgt_fps < 5)
>>>>>> +        vblank = *min_vblank;
>>>>>> +    else
>>>>>> +        vblank = MALI_C55_TPG_PIXEL_RATE / hts
>>>>>> +               / max(rounddown(tgt_fps, 15), 5);
>>>>>> +
>>>>>> +    *def_vblank = ALIGN_DOWN(vblank, 2) - format->height;
>>>>> "vblank = vblank - height" doesn't seem right. The "else" branch stores
>>>>> a vts in vblank, which doesn't seem right either. Maybe you meant
>>>>> something like
>>>>>
>>>>>      if (tgt_fps < 5)
>>>>>          def_vts = *min_vblank + format->height;
>>>>>      else
>>>>>          def_vts = MALI_C55_TPG_PIXEL_RATE / hts
>>>>>              / max(rounddown(tgt_fps, 15), 5);
>>>>>
>>>>>      *def_vblank = ALIGN_DOWN(def_vts, 2) - format->height;
>>>>>
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
>>>>>> +{
>>>>>> +    struct mali_c55_tpg *tpg = container_of(ctrl->handler,
>>>>>> +                        struct mali_c55_tpg,
>>>>>> +                        ctrls.handler);
>>>>>> +    struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
>>>>>> +
>>>>> Should you return here if the pipeline isn't streaming ?
>>>>>
>>>>>> +    switch (ctrl->id) {
>>>>>> +    case V4L2_CID_TEST_PATTERN:
>>>>>> +        mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
>>>>>> +                   ctrl->val);
>>>>>> +        break;
>>>>>> +    case V4L2_CID_VBLANK:
>>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
>>>>>> +                     MALI_C55_REG_VBLANK_MASK, ctrl->val);
>>>>>> +        break;
>>>>>> +    default:
>>>>>> +        dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
>>>>>> +        return -EINVAL;
>>>>> Can this happen ?
>>>>>
>>>>>> +    }
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
>>>>>> +    .s_ctrl = &mali_c55_tpg_s_ctrl,
>>>>>> +};
>>>>>> +
>>>>>> +static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
>>>>>> +                   struct v4l2_subdev *sd)
>>>>>> +{
>>>>>> +    struct v4l2_subdev_state *state;
>>>>>> +    struct v4l2_mbus_framefmt *fmt;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * hblank needs setting, but is a read-only control and thus won't be
>>>>>> +     * called during __v4l2_ctrl_handler_setup(). Do it here instead.
>>>>>> +     */
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
>>>>>> +                 MALI_C55_REG_HBLANK_MASK,
>>>>>> +                 MALI_C55_TPG_FIXED_HBLANK);
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>>>>> +                 MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
>>>>>> +
>>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
>>>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>>>> +
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>>>>> +                 MALI_C55_TEST_PATTERN_RGB_MASK,
>>>>>> +                 fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
>>>>>> +                      0x01 : 0x0);
>>>>>> +
>>>>>> +    v4l2_subdev_unlock_state(state);
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
>>>>>> +{
>>>>>> +    struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
>>>>>> +    struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
>>>>>> +
>>>>>> +    if (!enable) {
>>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>>>>> +                MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
>>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>>>>> +                MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * One might reasonably expect the framesize to be set here
>>>>>> +     * given it's configurable in .set_fmt(), but it's done in the
>>>>>> +     * ISP subdevice's stream on func instead, as the same register
>>>>> s/func/function/
>>>>>
>>>>>> +     * is also used to indicate the size of the data coming from the
>>>>>> +     * sensor.
>>>>>> +     */
>>>>>> +    mali_c55_tpg_configure(mali_c55, sd);
>>>>>      mali_c55_tpg_configure(tpg);
>>>>>
>>>>>> + __v4l2_ctrl_handler_setup(sd->ctrl_handler);
>>>>>> +
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
>>>>>> +                 MALI_C55_TEST_PATTERN_ON_OFF,
>>>>>> +                 MALI_C55_TEST_PATTERN_ON_OFF);
>>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
>>>>>> +                 MALI_C55_REG_GEN_VIDEO_ON_MASK,
>>>>>> +                 MALI_C55_REG_GEN_VIDEO_ON_MASK);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
>>>>>> +    .s_stream = &mali_c55_tpg_s_stream,
>>>>> Can we use .enable_streams() and .disable_streams() ?
>>>>>
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>> +                       struct v4l2_subdev_state *state,
>>>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
>>>>>> +{
>>>>>> +    if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    code->code = mali_c55_tpg_mbus_codes[code->index];
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
>>>>>> +                    struct v4l2_subdev_state *state,
>>>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
>>>>>> +{
>>>>> You sohuld verify here that fse->code is a supported value and return
>>>>> -EINVAL otherwise.
>>>>>
>>>>>> +    if (fse->index > 0 || fse->pad > sd->entity.num_pads)
>>>>> Drop the pad check, it's done in the subdev core already.
>>>>>
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
>>>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
>>>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
>>>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
>>>>>> +                struct v4l2_subdev_state *state,
>>>>>> +                struct v4l2_subdev_format *format)
>>>>>> +{
>>>>>> +    struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
>>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
>>>>>> +    int vblank_def, vblank_min;
>>>>>> +    unsigned int i;
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>>>>>> +        if (fmt->code == mali_c55_tpg_mbus_codes[i])
>>>>>> +            break;
>>>>>> +    }
>>>>>> +
>>>>>> +    if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The TPG says that the test frame timing generation logic expects a
>>>>>> +     * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
>>>>>> +     * handle anything smaller than 128x128 it seems pointless to allow a
>>>>>> +     * smaller frame.
>>>>>> +     */
>>>>>> +    clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
>>>>>> +        MALI_C55_MAX_WIDTH);
>>>>>> +    clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
>>>>>> +        MALI_C55_MAX_HEIGHT);
>>>>>> +
>>>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;
>>>>> You're allowing userspace to set fmt->field, as well as all the
>>>>> colorspace parameters, to random values. I would instead do something
>>>>> like
>>>>>
>>>>>      for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>>>>>          if (format->format.code == mali_c55_tpg_mbus_codes[i])
>>>>>              break;
>>>>>      }
>>>>>
>>>>>      if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>>>>          format->format.code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>>>
>>>>>      format->format.width = clamp(format->format.width,
>>>>>                       MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>>>>      format->format.height = clamp(format->format.height,
>>>>>                        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>>>>
>>>>>      fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>>>      fmt->code = format->format.code;
>>>>>      fmt->width = format->format.width;
>>>>>      fmt->height = format->format.height;
>>>>>
>>>>>      format->format = *fmt;
>>>>>
>>>>> Alternatively (which I think I like better),
>>>>>
>>>>>      fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>>>
>>>>>      fmt->code = format->format.code;
>>>>>
>>>>>      for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
>>>>>          if (fmt->code == mali_c55_tpg_mbus_codes[i])
>>>>>              break;
>>>>>      }
>>>>>
>>>>>      if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
>>>>>          fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>>>
>>>>>      fmt->width = clamp(format->format.width,
>>>>>                 MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
>>>>>      fmt->height = clamp(format->format.height,
>>>>>                  MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
>>>>>
>>>>>      format->format = *fmt;
>>>>>
>>>>>> +
>>>>>> +    if (format->which == V4L2_SUBDEV_FORMAT_TRY)
>>>>>> +        return 0;
>>>>>> +
>>>>>> +    __mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
>>>>>> +    __v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
>>>>>> +                 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
>>>>>> +    __v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);
>>>>> Move those three calls to a separate function, it will be reused below.
>>>>> I'd name is mali_c55_tpg_update_vblank(). You can fold
>>>>> __mali_c55_tpg_calc_vblank() in it.
>>>>>
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
>>>>>> +    .enum_mbus_code        = mali_c55_tpg_enum_mbus_code,
>>>>>> +    .enum_frame_size    = mali_c55_tpg_enum_frame_size,
>>>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
>>>>>> +    .set_fmt        = mali_c55_tpg_set_fmt,
>>>>>> +};
>>>>>> +
>>>>>> +static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
>>>>>> +    .video    = &mali_c55_tpg_video_ops,
>>>>>> +    .pad    = &mali_c55_tpg_pad_ops,
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
>>>>>> +                   struct v4l2_subdev_state *sd_state)
>>>>> You name this variable state in every other subdev operation handler.
>>>>>
>>>>>> +{
>>>>>> +    struct v4l2_mbus_framefmt *fmt =
>>>>>> +        v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
>>>>>> +
>>>>>> +    fmt->width = MALI_C55_DEFAULT_WIDTH;
>>>>>> +    fmt->height = MALI_C55_DEFAULT_HEIGHT;
>>>>>> +    fmt->field = V4L2_FIELD_NONE;
>>>>>> +    fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
>>>>> Initialize the colorspace fields too.
>>>>>
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
>>>>>> +    .init_state = mali_c55_tpg_init_state,
>>>>>> +};
>>>>>> +
>>>>>> +static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
>>>>>> +    struct v4l2_subdev *sd = &mali_c55->tpg.sd;
>>>>>> +    struct v4l2_mbus_framefmt *format;
>>>>>> +    struct v4l2_subdev_state *state;
>>>>>> +    int vblank_def, vblank_min;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
>>>>>> +    format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
>>>>>> +
>>>>>> +    ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);
>>>>> You have 3 controls.
>>>>>
>>>>>> +    if (ret)
>>>>>> +        goto err_unlock;
>>>>>> +
>>>>>> +    ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
>>>>>> +                &mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
>>>>>> +                ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
>>>>>> +                0, 3, mali_c55_tpg_test_pattern_menu);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * We fix hblank at the minimum allowed value and control framerate
>>>>>> +     * solely through the vblank control.
>>>>>> +     */
>>>>>> +    ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
>>>>>> +                &mali_c55_tpg_ctrl_ops,
>>>>>> +                V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
>>>>>> +                MALI_C55_TPG_FIXED_HBLANK, 1,
>>>>>> +                MALI_C55_TPG_FIXED_HBLANK);
>>>>>> +    if (ctrls->hblank)
>>>>>> +        ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
>>>>>> +
>>>>>> +    __mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);
>>>>> Drop this and initialize the control with default values. You can then
>>>>> update the value by calling mali_c55_tpg_update_vblank() in
>>>>> mali_c55_register_tpg().
>>>>>
>>>>> The reason is to share the same mutex between the control handler and
>>>>> the subdev active state without having to add a separate mutex in the
>>>>> mali_c55_tpg structure. The simplest way to do so is to initialize the
>>>>> controls first, set sd->state_lock to point to the control handler lock,
>>>>> and call v4l2_subdev_init_finalize() as the last step. As a consequence,
>>>>> you can't access the active state when initializing controls.
>>>>>
>>>>> You can alternatively keep the lock in mali_c55_tpg and set
>>>>> sd->state_lock to point to it, but I think that's more complex.
>>>>>
>>>>>> +    ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
>>>>>> +                      &mali_c55_tpg_ctrl_ops,
>>>>>> +                      V4L2_CID_VBLANK, vblank_min,
>>>>>> +                      MALI_C55_TPG_MAX_VBLANK, 1,
>>>>>> +                      vblank_def);
>>>>>> +
>>>>>> +    if (ctrls->handler.error) {
>>>>>> +        dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
>>>>>> +        ret = ctrls->handler.error;
>>>>>> +        goto err_free_handler;
>>>>>> +    }
>>>>>> +
>>>>>> +    ctrls->handler.lock = &mali_c55->tpg.lock;
>>>>> Drop this and drop the mutex. The control handler will use its internal
>>>>> mutex.
>>>>>
>>>>>> +    mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
>>>>>> +
>>>>>> +    v4l2_subdev_unlock_state(state);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_free_handler:
>>>>>> +    v4l2_ctrl_handler_free(&ctrls->handler);
>>>>>> +err_unlock:
>>>>>> +    v4l2_subdev_unlock_state(state);
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_tpg *tpg = &mali_c55->tpg;
>>>>>> +    struct v4l2_subdev *sd = &tpg->sd;
>>>>>> +    struct media_pad *pad = &tpg->pad;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    mutex_init(&tpg->lock);
>>>>>> +
>>>>>> +    v4l2_subdev_init(sd, &mali_c55_tpg_ops);
>>>>>> +    sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
>>>>>> +    sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
>>>>> Should we introduce a TPG function ?
>>>>>
>>>>>> +    sd->internal_ops = &mali_c55_tpg_internal_ops;
>>>>>> +    strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
>>>>>> +
>>>>>> +    pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;
>>>>> I don't think MEDIA_PAD_FL_MUST_CONNECT is right.
>>>>>
>>>>>> +    ret = media_entity_pads_init(&sd->entity, 1, pad);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev,
>>>>>> +            "Failed to initialize media entity pads\n");
>>>>>> +        goto err_destroy_mutex;
>>>>>> +    }
>>>>>> +
>>>>>      sd->state_lock = sd->ctrl_handler->lock;
>>>>>
>>>>> to use the same lock for the controls and the active state. You need to
>>>>> move this line and the v4l2_subdev_init_finalize() call after
>>>>> mali_c55_tpg_init_controls() to get the control handler lock initialized
>>>>> first.
>>>>>
>>>>>> +    ret = v4l2_subdev_init_finalize(sd);
>>>>>> +    if (ret)
>>>>>> +        goto err_cleanup_media_entity;
>>>>>> +
>>>>>> +    ret = mali_c55_tpg_init_controls(mali_c55);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev,
>>>>>> +            "Error initialising controls\n");
>>>>>> +        goto err_cleanup_subdev;
>>>>>> +    }
>>>>>> +
>>>>>> +    ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
>>>>>> +    if (ret) {
>>>>>> +        dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
>>>>>> +        goto err_free_ctrl_handler;
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * By default the colour settings lead to a very dim image that is
>>>>>> +     * nearly indistinguishable from black on some monitor settings. Ramp
>>>>>> +     * them up a bit so the image is brighter.
>>>>>> +     */
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
>>>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
>>>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
>>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
>>>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
>>>>>> +
>>>>>> +    tpg->mali_c55 = mali_c55;
>>>>>> +
>>>>>> +    return 0;
>>>>>> +
>>>>>> +err_free_ctrl_handler:
>>>>>> +    v4l2_ctrl_handler_free(&tpg->ctrls.handler);
>>>>>> +err_cleanup_subdev:
>>>>>> +    v4l2_subdev_cleanup(sd);
>>>>>> +err_cleanup_media_entity:
>>>>>> +    media_entity_cleanup(&sd->entity);
>>>>>> +err_destroy_mutex:
>>>>>> +    mutex_destroy(&tpg->lock);
>>>>>> +
>>>>>> +    return ret;
>>>>>> +}
>>>>>> +
>>>>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
>>>>>> +{
>>>>>> +    struct mali_c55_tpg *tpg = &mali_c55->tpg;
>>>>>> +
>>>>>> +    if (!tpg->mali_c55)
>>>>>> +        return;
>>>>>> +
>>>>>> +    v4l2_device_unregister_subdev(&tpg->sd);
>>>>>> +    v4l2_subdev_cleanup(&tpg->sd);
>>>>>> +    media_entity_cleanup(&tpg->sd.entity);
>>>>>> +    v4l2_ctrl_handler_free(&tpg->ctrls.handler);
>>>>> Free the control handler just after v4l2_device_unregister_subdev() to
>>>>> match the order in mali_c55_register_tpg().
>>>>>
>>>>>> +    mutex_destroy(&tpg->lock);
>>>>>> +}

Re: [PATCH v5 10/16] media: platform: Add mali-c55 3a stats devnode

[Laurent Pinchart]

Hi Dan,

On Thu, Jun 20, 2024 at 04:10:01PM +0100, Daniel Scally wrote:
> On 16/06/2024 22:19, Laurent Pinchart wrote:
> > On Wed, May 29, 2024 at 04:28:52PM +0100, Daniel Scally wrote:
> >> Add a new code file to govern the 3a statistics capture node.
> >>
> >> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
> >> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> >> ---
> >> Changes in v5:
> >>
> >> 	- New patch
> >>
> >>   drivers/media/platform/arm/mali-c55/Makefile  |   3 +-
> >>   .../platform/arm/mali-c55/mali-c55-common.h   |  28 ++
> >>   .../platform/arm/mali-c55/mali-c55-core.c     |  15 +
> >>   .../platform/arm/mali-c55/mali-c55-isp.c      |   1 +
> >>   .../arm/mali-c55/mali-c55-registers.h         |   3 +
> >>   .../platform/arm/mali-c55/mali-c55-stats.c    | 350 ++++++++++++++++++
> >>   6 files changed, 399 insertions(+), 1 deletion(-)
> >>   create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-stats.c
> >>
> >> diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
> >> index 77dcb2fbf0f4..cd5a64bf0c62 100644
> >> --- a/drivers/media/platform/arm/mali-c55/Makefile
> >> +++ b/drivers/media/platform/arm/mali-c55/Makefile
> >> @@ -4,6 +4,7 @@ mali-c55-y := mali-c55-capture.o \
> >>   	      mali-c55-core.o \
> >>   	      mali-c55-isp.o \
> >>   	      mali-c55-tpg.o \
> >> -	      mali-c55-resizer.o
> >> +	      mali-c55-resizer.o \
> >> +	      mali-c55-stats.o
> >>   
> >>   obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
> >> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >> index 2d0c4d152beb..44119e04009b 100644
> >> --- a/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >> @@ -79,6 +79,7 @@ enum mali_c55_isp_pads {
> >>   	MALI_C55_ISP_PAD_SINK_VIDEO,
> >>   	MALI_C55_ISP_PAD_SOURCE,
> >>   	MALI_C55_ISP_PAD_SOURCE_BYPASS,
> >> +	MALI_C55_ISP_PAD_SOURCE_3A,
> >
> > Functions and structures are named with a "stats" suffix, let's call
> > this MALI_C55_ISP_PAD_SOURCE_STATS.
> >
> >>   	MALI_C55_ISP_NUM_PADS,
> >>   };
> >>   
> >> @@ -194,6 +195,28 @@ struct mali_c55_cap_dev {
> >>   	bool streaming;
> >>   };
> >>   
> >> +struct mali_c55_stats_buf {
> >> +	struct vb2_v4l2_buffer vb;
> >> +	spinlock_t lock;
> > 
> > All locks require a comment to document what they protect. Same below.
> >
> >> +	unsigned int segments_remaining;
> >> +	struct list_head queue;
> >> +	bool failed;
> >> +};
> >> +
> >> +struct mali_c55_stats {
> >> +	struct mali_c55 *mali_c55;
> >> +	struct video_device vdev;
> >> +	struct dma_chan *channel;
> >> +	struct vb2_queue queue;
> >> +	struct media_pad pad;
> >> +	struct mutex lock;
> >> +
> >> +	struct {
> >> +		spinlock_t lock;
> >> +		struct list_head queue;
> >> +	} buffers;
> >> +};
> >> +
> >>   enum mali_c55_config_spaces {
> >>   	MALI_C55_CONFIG_PING,
> >>   	MALI_C55_CONFIG_PONG,
> >> @@ -224,6 +247,7 @@ struct mali_c55 {
> >>   	struct mali_c55_isp isp;
> >>   	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
> >>   	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
> >> +	struct mali_c55_stats stats;
> >>   
> >>   	struct list_head contexts;
> >>   	enum mali_c55_config_spaces next_config;
> >> @@ -245,6 +269,8 @@ int mali_c55_register_resizers(struct mali_c55 *mali_c55);
> >>   void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
> >>   int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
> >>   void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
> >> +int mali_c55_register_stats(struct mali_c55 *mali_c55);
> >> +void mali_c55_unregister_stats(struct mali_c55 *mali_c55);
> >>   struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
> >>   void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> >>   			     enum mali_c55_planes plane);
> >> @@ -262,5 +288,7 @@ mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
> >>   bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
> >>   #define for_each_mali_isp_fmt(fmt)\
> >>   	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
> >> +void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
> >> +				enum mali_c55_config_spaces cfg_space);
> >>   
> >>   #endif /* _MALI_C55_COMMON_H */
> >> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >> index 50caf5ee7474..9ea70010876c 100644
> >> --- a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >> @@ -337,6 +337,16 @@ static int mali_c55_create_links(struct mali_c55 *mali_c55)
> >>   		}
> >>   	}
> >>   
> >> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >> +			MALI_C55_ISP_PAD_SOURCE_3A,
> >> +			&mali_c55->stats.vdev.entity, 0,
> >> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev,
> >> +			"failed to link ISP and 3a stats node\n");
> > 
> > s/3a stats/stats/
> >
> >> +		goto err_remove_links;
> >> +	}
> >> +
> >>   	return 0;
> >>   
> >>   err_remove_links:
> >> @@ -350,6 +360,7 @@ static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
> >>   	mali_c55_unregister_isp(mali_c55);
> >>   	mali_c55_unregister_resizers(mali_c55);
> >>   	mali_c55_unregister_capture_devs(mali_c55);
> >> +	mali_c55_unregister_stats(mali_c55);
> >>   }
> >>   
> >>   static int mali_c55_register_entities(struct mali_c55 *mali_c55)
> >> @@ -372,6 +383,10 @@ static int mali_c55_register_entities(struct mali_c55 *mali_c55)
> >>   	if (ret)
> >>   		goto err_unregister_entities;
> >>   
> >> +	ret = mali_c55_register_stats(mali_c55);
> >> +	if (ret)
> >> +		goto err_unregister_entities;
> >> +
> >>   	ret = mali_c55_create_links(mali_c55);
> >>   	if (ret)
> >>   		goto err_unregister_entities;
> >> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >> index ea8b7b866e7a..94876fba3353 100644
> >> --- a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >> @@ -564,6 +564,7 @@ int mali_c55_register_isp(struct mali_c55 *mali_c55)
> >>   	isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
> >>   	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> >>   	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
> >> +	isp->pads[MALI_C55_ISP_PAD_SOURCE_3A].flags = MEDIA_PAD_FL_SOURCE;
> >>   
> >>   	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
> >>   				     isp->pads);
> >> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >> index cb27abde2aa5..eb3719245ec3 100644
> >> --- a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >> @@ -68,6 +68,9 @@
> >>   #define MALI_C55_VC_START(v)				((v) & 0xffff)
> >>   #define MALI_C55_VC_SIZE(v)				(((v) & 0xffff) << 16)
> >>   
> >> +#define MALI_C55_REG_1024BIN_HIST			0x054a8
> >> +#define MALI_C55_1024BIN_HIST_SIZE			4096
> >> +
> >>   /* Ping/Pong Configuration Space */
> >>   #define MALI_C55_REG_BASE_ADDR				0x18e88
> >>   #define MALI_C55_REG_BYPASS_0				0x18eac
> >> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-stats.c b/drivers/media/platform/arm/mali-c55/mali-c55-stats.c
> >> new file mode 100644
> >> index 000000000000..aa40480ed814
> >> --- /dev/null
> >> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-stats.c
> >> @@ -0,0 +1,350 @@
> >> +// SPDX-License-Identifier: GPL-2.0
> >> +/*
> >> + * ARM Mali-C55 ISP Driver - 3A Statistics capture device
> >> + *
> >> + * Copyright (C) 2023 Ideas on Board Oy
> >> + */
> >> +
> >> +#include <linux/dmaengine.h>
> >> +#include <linux/media/arm/mali-c55-config.h>
> >> +#include <linux/spinlock.h>
> > 
> > You're missing some headers here, for
> >
> > container_of()
> > dev_err()
> > list_*()
> > mutex_init()
> > strscpy()
> > strscpy()
> >
> >> +
> >> +#include <media/media-entity.h>
> >> +#include <media/v4l2-dev.h>
> >> +#include <media/v4l2-event.h>
> >> +#include <media/v4l2-fh.h>
> >> +#include <media/v4l2-ioctl.h>
> >> +#include <media/videobuf2-core.h>
> >> +#include <media/videobuf2-dma-contig.h>
> >> +
> >> +#include "mali-c55-common.h"
> >> +#include "mali-c55-registers.h"
> >> +
> >> +static unsigned int metering_space_addrs[] = {
> > 
> > const
> >
> >> +	[MALI_C55_CONFIG_PING] = 0x095AC,
> >> +	[MALI_C55_CONFIG_PONG] = 0x2156C,
> > 
> > Lower-case hex constants.
> >
> >> +};
> >> +
> >> +static int mali_c55_stats_enum_fmt_meta_cap(struct file *file, void *fh,
> >> +					    struct v4l2_fmtdesc *f)
> >> +{
> >> +	if (f->index || f->type != V4L2_BUF_TYPE_META_CAPTURE)
> >> +		return -EINVAL;
> >> +
> >> +	f->pixelformat = V4L2_META_FMT_MALI_C55_3A_STATS;
> > 
> > The format could be called V4L2_META_FMT_MALI_C55_STATS. While most
> > statistics are related to one of the 3A algorithms, I think it would be
> > better to name this generically. It's name bikeshedding only of course.
> >
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_stats_g_fmt_meta_cap(struct file *file, void *fh,
> >> +					 struct v4l2_format *f)
> >> +{
> >> +	static const struct v4l2_meta_format mfmt = {
> >> +		.dataformat = V4L2_META_FMT_MALI_C55_3A_STATS,
> >> +		.buffersize = sizeof(struct mali_c55_stats_buffer)
> >> +	};
> >> +
> >> +	if (f->type != V4L2_BUF_TYPE_META_CAPTURE)
> >> +		return -EINVAL;
> >> +
> >> +	f->fmt.meta = mfmt;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int mali_c55_stats_querycap(struct file *file,
> >> +				   void *priv, struct v4l2_capability *cap)
> >> +{
> >> +	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
> >> +	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static const struct v4l2_ioctl_ops mali_c55_stats_v4l2_ioctl_ops = {
> >> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
> >> +	.vidioc_querybuf = vb2_ioctl_querybuf,
> >> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
> >> +	.vidioc_qbuf = vb2_ioctl_qbuf,
> >> +	.vidioc_expbuf = vb2_ioctl_expbuf,
> >> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
> >> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> >> +	.vidioc_streamon = vb2_ioctl_streamon,
> >> +	.vidioc_streamoff = vb2_ioctl_streamoff,
> >> +	.vidioc_enum_fmt_meta_cap = mali_c55_stats_enum_fmt_meta_cap,
> >> +	.vidioc_g_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
> >> +	.vidioc_s_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
> >> +	.vidioc_try_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
> >> +	.vidioc_querycap = mali_c55_stats_querycap,
> >> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
> >> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
> >> +};
> >> +
> >> +static const struct v4l2_file_operations mali_c55_stats_v4l2_fops = {
> >> +	.owner = THIS_MODULE,
> >> +	.unlocked_ioctl = video_ioctl2,
> >> +	.open = v4l2_fh_open,
> >> +	.release = vb2_fop_release,
> >> +	.poll = vb2_fop_poll,
> >> +	.mmap = vb2_fop_mmap,
> >> +};
> >> +
> >> +static int
> >> +mali_c55_stats_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
> >> +			   unsigned int *num_planes, unsigned int sizes[],
> >> +			   struct device *alloc_devs[])
> >> +{
> >> +	struct mali_c55_stats *stats = vb2_get_drv_priv(q);
> >> +
> >> +	if (*num_planes && *num_planes > 1)
> >> +		return -EINVAL;
> >> +
> >> +	if (sizes[0] && sizes[0] != sizeof(struct mali_c55_stats_buffer))
> >> +		return -EINVAL;
> >> +
> >> +	*num_planes = 1;
> >> +	sizes[0] = sizeof(struct mali_c55_stats_buffer);
> >> +
> >> +	if (stats->channel)
> >> +		alloc_devs[0] = stats->channel->device->dev;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static void mali_c55_stats_buf_queue(struct vb2_buffer *vb)
> >> +{
> >> +	struct mali_c55_stats *stats = vb2_get_drv_priv(vb->vb2_queue);
> >> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> >> +	struct mali_c55_stats_buf *buf = container_of(vbuf,
> >> +						struct mali_c55_stats_buf, vb);
> >> +
> >> +	vb2_set_plane_payload(vb, 0, sizeof(struct mali_c55_stats_buffer));
> >> +	buf->segments_remaining = 2;
> >> +	buf->failed = false;
> >> +
> >> +	spin_lock(&stats->buffers.lock);
> > 
> > Isn't the DMA completion handler run from IRQ context ? If so you'll
> > need to use spin_lock_irq() here and in the other function that are
> > not called with interrupts disabled.
> 
> They're run in the bottom half of the interrupt handler; I'm under the
> impression that that means the interrupts aren't disabled, and it's
> safe to do...is that mistaken?

I'm talking about the DMA completion handler, called by the DMA engine
subsystem, not the IRQs of the C55 itself.

> >> +	list_add_tail(&buf->queue, &stats->buffers.queue);
> >> +	spin_unlock(&stats->buffers.lock);
> >> +}
> >> +
> >> +static void mali_c55_stats_stop_streaming(struct vb2_queue *q)
> >> +{
> >> +	struct mali_c55_stats *stats = vb2_get_drv_priv(q);
> >> +	struct mali_c55_stats_buf *buf, *tmp;
> >> +
> >> +	spin_lock(&stats->buffers.lock);
> >> +
> >> +	list_for_each_entry_safe(buf, tmp, &stats->buffers.queue, queue) {
> >> +		list_del(&buf->queue);
> >> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
> >> +	}
> >> +
> >> +	spin_unlock(&stats->buffers.lock);
> >> +}
> >> +
> >> +static const struct vb2_ops mali_c55_stats_vb2_ops = {
> >> +	.queue_setup = mali_c55_stats_queue_setup,
> >> +	.buf_queue = mali_c55_stats_buf_queue,
> >> +	.wait_prepare = vb2_ops_wait_prepare,
> >> +	.wait_finish = vb2_ops_wait_finish,
> >> +	.stop_streaming = mali_c55_stats_stop_streaming,
> >> +};
> >> +
> >> +static void
> >> +mali_c55_stats_metering_complete(void *param,
> >> +				 const struct dmaengine_result *result)
> >> +{
> >> +	struct mali_c55_stats_buf *buf = param;
> >> +
> >> +	spin_lock(&buf->lock);
> > 
> > I wonder if this is needed. Can the DMA engine call the completion
> > handlers of two sequential DMA transfers in parallel ?
> 
> The DMA engine that's on the system we have can't...I wasn't sure
> whether that was generically true.

I think it is, but please double-check.

> >> +
> >> +	if (buf->failed)
> >> +		goto out_unlock;
> >> +
> >> +	buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> >> +
> >> +	if (result->result != DMA_TRANS_NOERROR) {
> >> +		buf->failed = true;
> >> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
> > 
> > This will possibly return the buffer to userspace after the first DMA
> > transfer. Userspace could then requeue the buffer to the kernel before
> > the completion of the second DMA transfer. That will cause trouble. I
> > think you should instead do something like
> >
> > 	spin_lock(&buf->lock);
> >
> > 	if (result->result != DMA_TRANS_NOERROR)
> > 		buf->failed = true;
> >
> > 	if (!--buf->segments_remaining) {
> > 		buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> > 		vb2_buffer_done(&buf->vb.vb2_buf, buf->failed ?
> > 				VB2_BUF_STATE_ERROR : VB2_BUF_STATE_DONE);
> > 	}
> >
> > 	spin_unlock(&buf->lock);
> >
> > The
> >
> > 	buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> >
> > line could also be moved to mali_c55_stats_fill_buffer(), which would
> > make sure the timestamp is filled in case of DMA submission failures.
> 
> Okedokey
> 
> >
> >> +		goto out_unlock;
> >> +	}
> >> +
> >> +	if (!--buf->segments_remaining)
> >> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
> >> +
> >> +out_unlock:
> >> +	spin_unlock(&buf->lock);
> >> +}
> >> +
> >> +static int mali_c55_stats_dma_xfer(struct mali_c55_stats *stats, dma_addr_t src,
> >> +				   dma_addr_t dst,
> >> +				   struct mali_c55_stats_buf *buf,
> >> +				   size_t length,
> >> +				   void (*callback)(void *, const struct dmaengine_result *result))
> > 
> > The same callback is used for both invocations of this function, you can
> > drop the parameter and hardcode it below.
> 
> Yeah, not even sure now why I had a parameter.
> 
> >> +{
> >> +	struct dma_async_tx_descriptor *tx;
> >> +	dma_cookie_t cookie;
> >> +
> >> +	tx = dmaengine_prep_dma_memcpy(stats->channel, dst, src, length, 0);
> >> +	if (!tx) {
> >> +		dev_err(stats->mali_c55->dev, "failed to prep stats DMA\n");
> >> +		return -EIO;
> >> +	}
> >> +
> >> +	tx->callback_result = callback;
> >> +	tx->callback_param = buf;
> >> +
> >> +	cookie = dmaengine_submit(tx);
> >> +	if (dma_submit_error(cookie)) {
> >> +		dev_err(stats->mali_c55->dev, "failed to submit stats DMA\n");
> >> +		return -EIO;
> >> +	}
> >> +
> >> +	dma_async_issue_pending(stats->channel);
> >> +	return 0;
> >> +}
> >> +
> >> +void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
> >> +				enum mali_c55_config_spaces cfg_space)
> >> +{
> >> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >> +	struct mali_c55_stats *stats = &mali_c55->stats;
> >> +	struct mali_c55_stats_buf *buf = NULL;
> >> +	dma_addr_t src, dst;
> >> +	int ret;
> >> +
> >> +	spin_lock(&stats->buffers.lock);
> >> +	if (!list_empty(&stats->buffers.queue)) {
> >> +		buf = list_first_entry(&stats->buffers.queue,
> >> +				       struct mali_c55_stats_buf, queue);
> >> +		list_del(&buf->queue);
> >> +	}
> >> +	spin_unlock(&stats->buffers.lock);
> >> +
> >> +	if (!buf)
> >> +		return;
> >> +
> >> +	buf->vb.sequence = mali_c55->isp.frame_sequence;
> >> +
> >> +	/*
> >> +	 * There are infact two noncontiguous sections of the ISP's
> > 
> > s/infact/in fact/
> >
> >> +	 * memory space that hold statistics for 3a algorithms to use. A
> > 
> > s/use. A/use: a/
> >
> >> +	 * section in each config space and a global section holding
> >> +	 * histograms which is double buffered and so holds data for the
> >> +	 * last frame. We need to read both.
> >> +	 */
> >> +	src = ctx->base + MALI_C55_REG_1024BIN_HIST;
> >> +	dst = vb2_dma_contig_plane_dma_addr(&buf->vb.vb2_buf, 0);
> >> +
> >> +	ret = mali_c55_stats_dma_xfer(stats, src, dst, buf,
> >> +				      MALI_C55_1024BIN_HIST_SIZE,
> >> +				      mali_c55_stats_metering_complete);
> >> +	if (ret)
> >> +		goto err_fail_buffer;
> >> +
> >> +	src = ctx->base + metering_space_addrs[cfg_space];
> >> +	dst += MALI_C55_1024BIN_HIST_SIZE;
> >> +
> >> +	ret = mali_c55_stats_dma_xfer(
> >> +		stats, src, dst, buf,
> >> +		sizeof(struct mali_c55_stats_buffer) - MALI_C55_1024BIN_HIST_SIZE,
> >> +		mali_c55_stats_metering_complete);
> >> +	if (ret) {
> >> +		dmaengine_terminate_sync(stats->channel);
> >> +		goto err_fail_buffer;
> >> +	}
> > 
> > I think you will need to terminate DMA transfers at stream off time.
> >
> >> +
> >> +	return;
> >> +
> >> +err_fail_buffer:
> >> +	vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
> >> +}
> >> +
> >> +void mali_c55_unregister_stats(struct mali_c55 *mali_c55)
> >> +{
> >> +	struct mali_c55_stats *stats = &mali_c55->stats;
> >> +
> >> +	if (!video_is_registered(&stats->vdev))
> >> +		return;
> >> +
> >> +	vb2_video_unregister_device(&stats->vdev);
> >> +	media_entity_cleanup(&stats->vdev.entity);
> >> +	dma_release_channel(stats->channel);
> >> +	mutex_destroy(&stats->lock);
> >> +}
> >> +
> >> +int mali_c55_register_stats(struct mali_c55 *mali_c55)
> >> +{
> >> +	struct mali_c55_stats *stats = &mali_c55->stats;
> >> +	struct video_device *vdev = &stats->vdev;
> >> +	struct vb2_queue *vb2q = &stats->queue;
> >> +	dma_cap_mask_t mask;
> >> +	int ret;
> >> +
> >> +	mutex_init(&stats->lock);
> >> +	INIT_LIST_HEAD(&stats->buffers.queue);
> >> +
> >> +	dma_cap_zero(mask);
> >> +	dma_cap_set(DMA_MEMCPY, mask);
> >> +
> >> +	stats->channel = dma_request_channel(mask, 0, NULL);
> > 
> > Do we need a CPU fallback in case no DMA is available ?
> 
> Yes, actually.
> 
> > I'm still very curious to know how long it takes to perform the DMA
> > transfer, compared to copying the data with the CPU, and especially
> > compared to the frame duration.
> 
> On my list of things to test and report :)

Looking forward to it :-)

> >> +	if (!stats->channel) {
> >> +		ret = -ENODEV;
> >> +		goto err_destroy_mutex;
> >> +	}
> >> +
> >> +	stats->pad.flags = MEDIA_PAD_FL_SINK;
> >> +	ret = media_entity_pads_init(&stats->vdev.entity, 1, &stats->pad);
> >> +	if (ret)
> >> +		goto err_release_dma_channel;
> >> +
> >> +	vb2q->type = V4L2_BUF_TYPE_META_CAPTURE;
> >> +	vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
> >> +	vb2q->drv_priv = stats;
> >> +	vb2q->mem_ops = &vb2_dma_contig_memops;
> >> +	vb2q->ops = &mali_c55_stats_vb2_ops;
> >> +	vb2q->buf_struct_size = sizeof(struct mali_c55_stats_buf);
> >> +	vb2q->min_queued_buffers = 1;
> >> +	vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> >> +	vb2q->lock = &stats->lock;
> >> +	vb2q->dev = mali_c55->dev;
> > 
> > That's not the right device. The device that performs the DMA operation
> > is the DMA engine, and that's what you need to pass to vb2. Otherwise
> > the DMA address returned by vb2_dma_contig_plane_dma_addr() will be
> > mapped to the ISP device, not the DMA engine. In practice, if neither
> > are behind an IOMMU, things will likely work, but when that's not the
> > case, run into problems.
> >
> >> +
> >> +	ret = vb2_queue_init(vb2q);
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev, "stats vb2 queue init failed\n");
> >> +		goto err_cleanup_entity;
> >> +	}
> >> +
> >> +	strscpy(stats->vdev.name, "mali-c55 3a stats", sizeof(stats->vdev.name));
> > 
> > s/3a //
> >
> >> +	vdev->release = video_device_release_empty;
> > 
> > That's never right. You should refcount the data structures to ensure
> > proper lifetime management.
> >
> >> +	vdev->fops = &mali_c55_stats_v4l2_fops;
> >> +	vdev->ioctl_ops = &mali_c55_stats_v4l2_ioctl_ops;
> >> +	vdev->lock = &stats->lock;
> >> +	vdev->v4l2_dev = &mali_c55->v4l2_dev;
> >> +	vdev->queue = &stats->queue;
> >> +	vdev->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING;
> >> +	vdev->vfl_dir = VFL_DIR_RX;
> >> +	video_set_drvdata(vdev, stats);
> >> +
> >> +	ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
> >> +	if (ret) {
> >> +		dev_err(mali_c55->dev,
> >> +			"failed to register stats video device\n");
> >> +		goto err_release_vb2q;
> >> +	}
> >> +
> >> +	stats->mali_c55 = mali_c55;
> >> +
> >> +	return 0;
> >> +
> >> +err_release_vb2q:
> >> +	vb2_queue_release(vb2q);
> >> +err_cleanup_entity:
> >> +	media_entity_cleanup(&stats->vdev.entity);
> >> +err_release_dma_channel:
> >> +	dma_release_channel(stats->channel);
> >> +err_destroy_mutex:
> >> +	mutex_destroy(&stats->lock);
> >> +
> >> +	return ret;
> >> +}

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Laurent Pinchart]

Hi Dan,

On Wed, Jun 19, 2024 at 04:43:15PM +0100, Daniel Scally wrote:
> On 18/06/2024 00:04, Laurent Pinchart wrote:
> > On Mon, Jun 17, 2024 at 12:41:11PM +0100, Daniel Scally wrote:
> >> Hi Laurent - sorry, should have included everything in the last reply rather than responding
> >> piecemeal. Some more responses and questions below
> > 
> > No worries.
> >
> >> On 30/05/2024 01:15, Laurent Pinchart wrote:
> >>> Hi Dan,
> >>>
> >>> Thank you for the patch.
> >>>
> >>> On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
> >>>> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
> >>>> V4L2 and Media Controller compliant and creates subdevices to manage
> >>>> the ISP itself, its internal test pattern generator as well as the
> >>>> crop, scaler and output format functionality for each of its two
> >>>> output devices.
> >>>>
> >>>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
> >>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> >>>> ---
> >>>> Changes in v5:
> >>>>
> >>>> 	- Reworked input formats - previously we allowed representing input data
> >>>> 	  as any 8-16 bit format. Now we only allow input data to be represented
> >>>> 	  by the new 20-bit bayer formats, which is corrected to the equivalent
> >>>> 	  16-bit format in RAW bypass mode.
> >>>> 	- Stopped bypassing blocks that we haven't added supporting parameters
> >>>> 	  for yet.
> >>>> 	- Addressed most of Sakari's comments from the list
> >>>>
> >>>> Changes not yet made in v5:
> >>>>
> >>>> 	- The output pipelines can still be started and stopped independently of
> >>>> 	  one another - I'd like to discuss that more.
> >>>> 	- the TPG subdev still uses .s_stream() - I need to rebase onto a tree
> >>>> 	  with working .enable_streams() for a single-source-pad subdevice.
> >>>>
> >>>> Changes in v4:
> >>>>
> >>>> 	- Reworked mali_c55_update_bits() to internally perform the bit-shift
> >>> 
> >>> I really don't like that, it makes the code very confusing, even more so
> >>> as it differs from regmap_update_bits().
> >>>
> >>> Look at this for instance:
> >>>
> >>> 	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>> 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> >>> 			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> >>>
> >>> It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
> >>> BIT(0).
> >>>
> >>> Sorry, I know it will be painful, but this change needs to be reverted.
> >>>
> >>>> 	- Reworked the resizer to allow cropping during streaming
> >>>> 	- Fixed a bug in NV12 output
> >>>>
> >>>> Changes in v3:
> >>>>
> >>>> 	- Mostly minor fixes suggested by Sakari
> >>>> 	- Fixed the sequencing of vb2 buffers to be synchronised across the two
> >>>> 	  capture devices.
> >>>>
> >>>> Changes in v2:
> >>>>
> >>>> 	- Clock handling
> >>>> 	- Fixed the warnings raised by the kernel test robot
> >>>>
> >>>>    drivers/media/platform/Kconfig                |   1 +
> >>>>    drivers/media/platform/Makefile               |   1 +
> >>>>    drivers/media/platform/arm/Kconfig            |   5 +
> >>>>    drivers/media/platform/arm/Makefile           |   2 +
> >>>>    drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
> >>>>    drivers/media/platform/arm/mali-c55/Makefile  |   9 +
> >>>>    .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
> >>>>    .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
> >>>>    .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
> >>>>    .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
> >>>>    .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
> >>>>    .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
> >>>>    .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
> >>>>    .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
> >>>>    14 files changed, 4452 insertions(+)
> >>>>    create mode 100644 drivers/media/platform/arm/Kconfig
> >>>>    create mode 100644 drivers/media/platform/arm/Makefile
> >>>>    create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
> >>>>    create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
> >>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >>> 
> >>> I've skipped review of capture.c and resizer.c as I already have plenty
> >>> of comments for the other files, and it's getting late. I'll try to
> >>> review the rest tomorrow.
> >>>
> >>>> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> >>>> index 2d79bfc68c15..c929169766aa 100644
> >>>> --- a/drivers/media/platform/Kconfig
> >>>> +++ b/drivers/media/platform/Kconfig
> >>>> @@ -65,6 +65,7 @@ config VIDEO_MUX
> >>>>    source "drivers/media/platform/allegro-dvt/Kconfig"
> >>>>    source "drivers/media/platform/amlogic/Kconfig"
> >>>>    source "drivers/media/platform/amphion/Kconfig"
> >>>> +source "drivers/media/platform/arm/Kconfig"
> >>>>    source "drivers/media/platform/aspeed/Kconfig"
> >>>>    source "drivers/media/platform/atmel/Kconfig"
> >>>>    source "drivers/media/platform/broadcom/Kconfig"
> >>>> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> >>>> index da17301f7439..9a647abd5218 100644
> >>>> --- a/drivers/media/platform/Makefile
> >>>> +++ b/drivers/media/platform/Makefile
> >>>> @@ -8,6 +8,7 @@
> >>>>    obj-y += allegro-dvt/
> >>>>    obj-y += amlogic/
> >>>>    obj-y += amphion/
> >>>> +obj-y += arm/
> >>>>    obj-y += aspeed/
> >>>>    obj-y += atmel/
> >>>>    obj-y += broadcom/
> >>>> diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
> >>>> new file mode 100644
> >>>> index 000000000000..4f0764c329c7
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/Kconfig
> >>>> @@ -0,0 +1,5 @@
> >>>> +# SPDX-License-Identifier: GPL-2.0-only
> >>>> +
> >>>> +comment "ARM media platform drivers"
> >>>> +
> >>>> +source "drivers/media/platform/arm/mali-c55/Kconfig"
> >>>> diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
> >>>> new file mode 100644
> >>>> index 000000000000..8cc4918725ef
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/Makefile
> >>>> @@ -0,0 +1,2 @@
> >>>> +# SPDX-License-Identifier: GPL-2.0-only
> >>>> +obj-y += mali-c55/
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/Kconfig b/drivers/media/platform/arm/mali-c55/Kconfig
> >>>> new file mode 100644
> >>>> index 000000000000..602085e28b01
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/Kconfig
> >>>> @@ -0,0 +1,18 @@
> >>>> +# SPDX-License-Identifier: GPL-2.0-only
> >>>> +config VIDEO_MALI_C55
> >>>> +	tristate "ARM Mali-C55 Image Signal Processor driver"
> >>>> +	depends on V4L_PLATFORM_DRIVERS
> >>>> +	depends on VIDEO_DEV && OF
> >>>> +	depends on ARCH_VEXPRESS || COMPILE_TEST
> >>>> +	select MEDIA_CONTROLLER
> >>>> +	select VIDEO_V4L2_SUBDEV_API
> >>>> +	select VIDEOBUF2_DMA_CONTIG
> >>>> +	select VIDEOBUF2_VMALLOC
> >>>> +	select V4L2_FWNODE
> >>>> +	select GENERIC_PHY_MIPI_DPHY
> >>> 
> >>> Alphabetical order ?
> >>>
> >>>> +	default n
> >>> 
> >>> That's the default, you don't have to specify ti.
> >>>
> >>>> +	help
> >>>> +	  Enable this to support Arm's Mali-C55 Image Signal Processor.
> >>>> +
> >>>> +	  To compile this driver as a module, choose M here: the module
> >>>> +	  will be called mali-c55.
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
> >>>> new file mode 100644
> >>>> index 000000000000..77dcb2fbf0f4
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
> >>>> @@ -0,0 +1,9 @@
> >>>> +# SPDX-License-Identifier: GPL-2.0
> >>>> +
> >>>> +mali-c55-y := mali-c55-capture.o \
> >>>> +	      mali-c55-core.o \
> >>>> +	      mali-c55-isp.o \
> >>>> +	      mali-c55-tpg.o \
> >>>> +	      mali-c55-resizer.o
> >>> 
> >>> Alphabetical order here too.
> >>>
> >>>> +
> >>>> +obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>>> new file mode 100644
> >>>> index 000000000000..1d539ac9c498
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>>> @@ -0,0 +1,951 @@
> >>>> +// SPDX-License-Identifier: GPL-2.0
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Video capture devices
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#include <linux/cleanup.h>
> >>>> +#include <linux/minmax.h>
> >>>> +#include <linux/pm_runtime.h>
> >>>> +#include <linux/string.h>
> >>>> +#include <linux/videodev2.h>
> >>>> +
> >>>> +#include <media/v4l2-dev.h>
> >>>> +#include <media/v4l2-event.h>
> >>>> +#include <media/v4l2-ioctl.h>
> >>>> +#include <media/v4l2-subdev.h>
> >>>> +#include <media/videobuf2-core.h>
> >>>> +#include <media/videobuf2-dma-contig.h>
> >>>> +
> >>>> +#include "mali-c55-common.h"
> >>>> +#include "mali-c55-registers.h"
> >>>> +
> >>>> +static const struct mali_c55_fmt mali_c55_fmts[] = {
> >>>> +	/*
> >>>> +	 * This table is missing some entries which need further work or
> >>>> +	 * investigation:
> >>>> +	 *
> >>>> +	 * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
> >>>> +	 * Base mode 5 is "Generic Data"
> >>>> +	 * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
> >>>> +	 * Base mode 9 seems to have no V4L2 equivalent
> >>>> +	 * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
> >>>> +	 * equivalent
> >>>> +	 */
> >>>> +	{
> >>>> +		.fourcc = V4L2_PIX_FMT_ARGB2101010,
> >>>> +		.mbus_codes = {
> >>>> +			MEDIA_BUS_FMT_RGB121212_1X36,
> >>>> +			MEDIA_BUS_FMT_RGB202020_1X60,
> >>>> +		},
> >>>> +		.is_raw = false,
> >>>> +		.registers = {
> >>>> +			.base_mode = MALI_C55_OUTPUT_A2R10G10B10,
> >>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +		}
> >>>> +	},
> >>>> +	{
> >>>> +		.fourcc = V4L2_PIX_FMT_RGB565,
> >>>> +		.mbus_codes = {
> >>>> +			MEDIA_BUS_FMT_RGB121212_1X36,
> >>>> +			MEDIA_BUS_FMT_RGB202020_1X60,
> >>>> +		},
> >>>> +		.is_raw = false,
> >>>> +		.registers = {
> >>>> +			.base_mode = MALI_C55_OUTPUT_RGB565,
> >>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +		}
> >>>> +	},
> >>>> +	{
> >>>> +		.fourcc = V4L2_PIX_FMT_BGR24,
> >>>> +		.mbus_codes = {
> >>>> +			MEDIA_BUS_FMT_RGB121212_1X36,
> >>>> +			MEDIA_BUS_FMT_RGB202020_1X60,
> >>>> +		},
> >>>> +		.is_raw = false,
> >>>> +		.registers = {
> >>>> +			.base_mode = MALI_C55_OUTPUT_RGB24,
> >>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +		}
> >>>> +	},
> >>>> +	{
> >>>> +		.fourcc = V4L2_PIX_FMT_YUYV,
> >>>> +		.mbus_codes = {
> >>>> +			MEDIA_BUS_FMT_YUV10_1X30,
> >>>> +		},
> >>>> +		.is_raw = false,
> >>>> +		.registers = {
> >>>> +			.base_mode = MALI_C55_OUTPUT_YUY2,
> >>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +		}
> >>>> +	},
> >>>> +	{
> >>>> +		.fourcc = V4L2_PIX_FMT_UYVY,
> >>>> +		.mbus_codes = {
> >>>> +			MEDIA_BUS_FMT_YUV10_1X30,
> >>>> +		},
> >>>> +		.is_raw = false,
> >>>> +		.registers = {
> >>>> +			.base_mode = MALI_C55_OUTPUT_UYVY,
> >>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +		}
> >>>> +	},
> >>>> +	{
> >>>> +		.fourcc = V4L2_PIX_FMT_Y210,
> >>>> +		.mbus_codes = {
> >>>> +			MEDIA_BUS_FMT_YUV10_1X30,
> >>>> +		},
> >>>> +		.is_raw = false,
> >>>> +		.registers = {
> >>>> +			.base_mode = MALI_C55_OUTPUT_Y210,
> >>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +		}
> >>>> +	},
> >>>> +	/*
> >>>> +	 * This is something of a hack, the ISP thinks it's running NV12M but
> >>>> +	 * by setting uv_plane = 0 we simply discard that planes and only output
> >>>> +	 * the Y-plane.
> >>>> +	 */
> >>>> +	{
> >>>> +		.fourcc = V4L2_PIX_FMT_GREY,
> >>>> +		.mbus_codes = {
> >>>> +			MEDIA_BUS_FMT_YUV10_1X30,
> >>>> +		},
> >>>> +		.is_raw = false,
> >>>> +		.registers = {
> >>>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
> >>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +		}
> >>>> +	},
> >>>> +	{
> >>>> +		.fourcc = V4L2_PIX_FMT_NV12M,
> >>>> +		.mbus_codes = {
> >>>> +			MEDIA_BUS_FMT_YUV10_1X30,
> >>>> +		},
> >>>> +		.is_raw = false,
> >>>> +		.registers = {
> >>>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
> >>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
> >>>> +		}
> >>>> +	},
> >>>> +	{
> >>>> +		.fourcc = V4L2_PIX_FMT_NV21M,
> >>>> +		.mbus_codes = {
> >>>> +			MEDIA_BUS_FMT_YUV10_1X30,
> >>>> +		},
> >>>> +		.is_raw = false,
> >>>> +		.registers = {
> >>>> +			.base_mode = MALI_C55_OUTPUT_NV12_21,
> >>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
> >>>> +		}
> >>>> +	},
> >>>> +	/*
> >>>> +	 * RAW uncompressed formats are all packed in 16 bpp.
> >>>> +	 * TODO: Expand this list to encompass all possible RAW formats.
> >>>> +	 */
> >>>> +	{
> >>>> +		.fourcc = V4L2_PIX_FMT_SRGGB16,
> >>>> +		.mbus_codes = {
> >>>> +			MEDIA_BUS_FMT_SRGGB16_1X16,
> >>>> +		},
> >>>> +		.is_raw = true,
> >>>> +		.registers = {
> >>>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
> >>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +		}
> >>>> +	},
> >>>> +	{
> >>>> +		.fourcc = V4L2_PIX_FMT_SBGGR16,
> >>>> +		.mbus_codes = {
> >>>> +			MEDIA_BUS_FMT_SBGGR16_1X16,
> >>>> +		},
> >>>> +		.is_raw = true,
> >>>> +		.registers = {
> >>>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
> >>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +		}
> >>>> +	},
> >>>> +	{
> >>>> +		.fourcc = V4L2_PIX_FMT_SGBRG16,
> >>>> +		.mbus_codes = {
> >>>> +			MEDIA_BUS_FMT_SGBRG16_1X16,
> >>>> +		},
> >>>> +		.is_raw = true,
> >>>> +		.registers = {
> >>>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
> >>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +		}
> >>>> +	},
> >>>> +	{
> >>>> +		.fourcc = V4L2_PIX_FMT_SGRBG16,
> >>>> +		.mbus_codes = {
> >>>> +			MEDIA_BUS_FMT_SGRBG16_1X16,
> >>>> +		},
> >>>> +		.is_raw = true,
> >>>> +		.registers = {
> >>>> +			.base_mode = MALI_C55_OUTPUT_RAW16,
> >>>> +			.uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>> +		}
> >>>> +	},
> >>>> +};
> >>>> +
> >>>> +static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
> >>>> +					       u32 code)
> >>>> +{
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
> >>>> +		if (fmt->mbus_codes[i] == code)
> >>>> +			return true;
> >>>> +	}
> >>>> +
> >>>> +	return false;
> >>>> +}
> >>>> +
> >>>> +bool mali_c55_format_is_raw(unsigned int mbus_code)
> >>>> +{
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >>>> +		if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
> >>>> +			return mali_c55_fmts[i].is_raw;
> >>>> +	}
> >>>> +
> >>>> +	return false;
> >>>> +}
> >>>> +
> >>>> +static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
> >>>> +{
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >>>> +		if (mali_c55_fmts[i].fourcc == pixelformat)
> >>>> +			return &mali_c55_fmts[i];
> >>>> +	}
> >>>> +
> >>>> +	/*
> >>>> +	 * If we find no matching pixelformat, we'll just default to the first
> >>>> +	 * one for now.
> >>>> +	 */
> >>>> +
> >>>> +	return &mali_c55_fmts[0];
> >>>> +}
> >>>> +
> >>>> +static const char * const capture_device_names[] = {
> >>>> +	"mali-c55 fr",
> >>>> +	"mali-c55 ds",
> >>>> +	"mali-c55 3a stats",
> >>>> +	"mali-c55 params",
> >>>> +};
> >>>> +
> >>>> +static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
> >>>> +{
> >>>> +	if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
> >>>> +		return capture_device_names[0];
> >>>> +
> >>>> +	if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
> >>>> +		return capture_device_names[1];
> >>>> +
> >>>> +	return "params/stat not supported yet";
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_link_validate(struct media_link *link)
> >>>> +{
> >>>> +	struct video_device *vdev =
> >>>> +		media_entity_to_video_device(link->sink->entity);
> >>>> +	struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
> >>>> +	struct v4l2_subdev *sd =
> >>>> +		media_entity_to_v4l2_subdev(link->source->entity);
> >>>> +	const struct v4l2_pix_format_mplane *pix_mp;
> >>>> +	const struct mali_c55_fmt *cap_fmt;
> >>>> +	struct v4l2_subdev_format sd_fmt = {
> >>>> +		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
> >>>> +		.pad = link->source->index,
> >>>> +	};
> >>>> +	int ret;
> >>>> +
> >>>> +	ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
> >>>> +	if (ret)
> >>>> +		return ret;
> >>>> +
> >>>> +	pix_mp = &cap_dev->mode.pix_mp;
> >>>> +	cap_fmt = cap_dev->mode.capture_fmt;
> >>>> +
> >>>> +	if (sd_fmt.format.width != pix_mp->width ||
> >>>> +	    sd_fmt.format.height != pix_mp->height) {
> >>>> +		dev_dbg(cap_dev->mali_c55->dev,
> >>>> +			"link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
> >>>> +			link->source->entity->name, link->source->index,
> >>>> +			link->sink->entity->name, link->sink->index,
> >>>> +			sd_fmt.format.width, sd_fmt.format.height,
> >>>> +			pix_mp->width, pix_mp->height);
> >>>> +		return -EPIPE;
> >>>> +	}
> >>>> +
> >>>> +	if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
> >>>> +		dev_dbg(cap_dev->mali_c55->dev,
> >>>> +			"link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format %p4cc\n",
> >>>> +			link->source->entity->name, link->source->index,
> >>>> +			link->sink->entity->name, link->sink->index,
> >>>> +			sd_fmt.format.code, &pix_mp->pixelformat);
> >>>> +		return -EPIPE;
> >>>> +	}
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct media_entity_operations mali_c55_media_ops = {
> >>>> +	.link_validate = mali_c55_link_validate,
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
> >>>> +				    unsigned int *num_planes, unsigned int sizes[],
> >>>> +				    struct device *alloc_devs[])
> >>>> +{
> >>>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	if (*num_planes) {
> >>>> +		if (*num_planes != cap_dev->mode.pix_mp.num_planes)
> >>>> +			return -EINVAL;
> >>>> +
> >>>> +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >>>> +			if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
> >>>> +				return -EINVAL;
> >>>> +	} else {
> >>>> +		*num_planes = cap_dev->mode.pix_mp.num_planes;
> >>>> +		for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >>>> +			sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
> >>>> +	}
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_buf_queue(struct vb2_buffer *vb)
> >>>> +{
> >>>> +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
> >>>> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> >>>> +	struct mali_c55_buffer *buf = container_of(vbuf,
> >>>> +						   struct mali_c55_buffer, vb);
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	buf->plane_done[MALI_C55_PLANE_Y] = false;
> >>>> +
> >>>> +	/*
> >>>> +	 * If we're in a single-plane format we flag the other plane as done
> >>>> +	 * already so it's dequeued appropriately later
> >>>> +	 */
> >>>> +	buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
> >>>> +
> >>>> +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
> >>>> +		unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
> >>>> +
> >>>> +		vb2_set_plane_payload(vb, i, size);
> >>>> +	}
> >>>> +
> >>>> +	spin_lock(&cap_dev->buffers.lock);
> >>>> +	list_add_tail(&buf->queue, &cap_dev->buffers.queue);
> >>>> +	spin_unlock(&cap_dev->buffers.lock);
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_buf_init(struct vb2_buffer *vb)
> >>>> +{
> >>>> +	struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
> >>>> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> >>>> +	struct mali_c55_buffer *buf = container_of(vbuf,
> >>>> +						   struct mali_c55_buffer, vb);
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >>>> +		buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>> +
> >>>> +	guard(spinlock)(&cap_dev->buffers.lock);
> >>>> +
> >>>> +	cap_dev->buffers.curr = cap_dev->buffers.next;
> >>>> +	cap_dev->buffers.next = NULL;
> >>>> +
> >>>> +	if (!list_empty(&cap_dev->buffers.queue)) {
> >>>> +		struct v4l2_pix_format_mplane *pix_mp;
> >>>> +		const struct v4l2_format_info *info;
> >>>> +		u32 *addrs;
> >>>> +
> >>>> +		pix_mp = &cap_dev->mode.pix_mp;
> >>>> +		info = v4l2_format_info(pix_mp->pixelformat);
> >>>> +
> >>>> +		mali_c55_update_bits(mali_c55,
> >>>> +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
> >>>> +		if (cap_dev->mode.capture_fmt->registers.uv_plane)
> >>>> +			mali_c55_update_bits(mali_c55,
> >>>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
> >>>> +
> >>>> +		cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
> >>>> +							 struct mali_c55_buffer,
> >>>> +							 queue);
> >>>> +		list_del(&cap_dev->buffers.next->queue);
> >>>> +
> >>>> +		addrs = cap_dev->buffers.next->addrs;
> >>>> +		mali_c55_write(mali_c55,
> >>>> +			MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
> >>>> +			addrs[MALI_C55_PLANE_Y]);
> >>>> +		mali_c55_write(mali_c55,
> >>>> +			MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
> >>>> +			addrs[MALI_C55_PLANE_UV]);
> >>>> +		mali_c55_write(mali_c55,
> >>>> +			MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
> >>>> +			pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
> >>>> +		mali_c55_write(mali_c55,
> >>>> +			MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
> >>>> +			pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
> >>>> +			/ info->hdiv);
> >>>> +	} else {
> >>>> +		/*
> >>>> +		 * If we underflow then we can tell the ISP that we don't want
> >>>> +		 * to write out the next frame.
> >>>> +		 */
> >>>> +		mali_c55_update_bits(mali_c55,
> >>>> +				MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>> +		mali_c55_update_bits(mali_c55,
> >>>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>> +				MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>> +	}
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
> >>>> +				   unsigned int framecount)
> >>>> +{
> >>>> +	curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> >>>> +	curr_buf->vb.field = V4L2_FIELD_NONE;
> >>>> +	curr_buf->vb.sequence = framecount;
> >>>> +	vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
> >>>> +}
> >>>> +
> >>>> +/**
> >>>> + * mali_c55_set_plane_done - mark the plane as written and process the buffer if
> >>>> + *			     both planes are finished.
> >>>> + * @cap_dev:  pointer to the fr or ds pipe output
> >>>> + * @plane:    the plane to mark as completed
> >>>> + *
> >>>> + * The Mali C55 ISP has muliplanar outputs for some formats that come with two
> >>>> + * separate "buffer write completed" interrupts - we need to flag each plane's
> >>>> + * completion and check whether both planes are done - if so, complete the buf
> >>>> + * in vb2.
> >>>> + */
> >>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> >>>> +			     enum mali_c55_planes plane)
> >>>> +{
> >>>> +	struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
> >>>> +	struct mali_c55_buffer *curr_buf;
> >>>> +
> >>>> +	guard(spinlock)(&cap_dev->buffers.lock);
> >>>> +	curr_buf = cap_dev->buffers.curr;
> >>>> +
> >>>> +	/*
> >>>> +	 * This _should_ never happen. If no buffer was available from vb2 then
> >>>> +	 * we tell the ISP not to bother writing the next frame, which means the
> >>>> +	 * interrupts that call this function should never trigger. If it does
> >>>> +	 * happen then one of our assumptions is horribly wrong - complain
> >>>> +	 * loudly and do nothing.
> >>>> +	 */
> >>>> +	if (!curr_buf) {
> >>>> +		dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
> >>>> +			mali_c55_cap_dev_to_name(cap_dev), __func__);
> >>>> +		return;
> >>>> +	}
> >>>> +
> >>>> +	/* If the other plane is also done... */
> >>>> +	if (curr_buf->plane_done[~plane & 1]) {
> >>>> +		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> >>>> +		cap_dev->buffers.curr = NULL;
> >>>> +		isp->frame_sequence++;
> >>>> +	} else {
> >>>> +		curr_buf->plane_done[plane] = true;
> >>>> +	}
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>> +
> >>>> +	mali_c55_update_bits(mali_c55,
> >>>> +			     MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>> +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>> +	mali_c55_update_bits(mali_c55,
> >>>> +			     MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>> +			     MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>> +
> >>>> +	/*
> >>>> +	 * The Mali ISP can hold up to 5 buffer addresses and simply cycle
> >>>> +	 * through them, but it's not clear to me that the vb2 queue _guarantees_
> >>>> +	 * it will queue buffers to the driver in a fixed order, and ensuring
> >>>> +	 * we call vb2_buffer_done() for the right buffer seems to me to add
> >>>> +	 * pointless complexity given in multi-context mode we'd need to
> >>>> +	 * re-write those registers every frame anyway...so we tell the ISP to
> >>>> +	 * use a single register and update it for each frame.
> >>>> +	 */
> >>>> +	mali_c55_update_bits(mali_c55,
> >>>> +			MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
> >>>> +			MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
> >>>> +	mali_c55_update_bits(mali_c55,
> >>>> +			MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
> >>>> +			MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
> >>>> +
> >>>> +	/*
> >>>> +	 * We only queue a buffer in the streamon path if this is the first of
> >>>> +	 * the capture devices to start streaming. If the ISP is already running
> >>>> +	 * then we rely on the ISP_START interrupt to queue the first buffer for
> >>>> +	 * this capture device.
> >>>> +	 */
> >>>> +	if (mali_c55->pipe.start_count == 1)
> >>>> +		mali_c55_set_next_buffer(cap_dev);
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
> >>>> +					    enum vb2_buffer_state state)
> >>>> +{
> >>>> +	struct mali_c55_buffer *buf, *tmp;
> >>>> +
> >>>> +	guard(spinlock)(&cap_dev->buffers.lock);
> >>>> +
> >>>> +	if (cap_dev->buffers.curr) {
> >>>> +		vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
> >>>> +				state);
> >>>> +		cap_dev->buffers.curr = NULL;
> >>>> +	}
> >>>> +
> >>>> +	if (cap_dev->buffers.next) {
> >>>> +		vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
> >>>> +				state);
> >>>> +		cap_dev->buffers.next = NULL;
> >>>> +	}
> >>>> +
> >>>> +	list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
> >>>> +		list_del(&buf->queue);
> >>>> +		vb2_buffer_done(&buf->vb.vb2_buf, state);
> >>>> +	}
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
> >>>> +{
> >>>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>> +	struct mali_c55_resizer *rzr = cap_dev->rzr;
> >>>> +	struct mali_c55_isp *isp = &mali_c55->isp;
> >>>> +	int ret;
> >>>> +
> >>>> +	guard(mutex)(&isp->lock);
> >>>> +
> >>>> +	ret = pm_runtime_resume_and_get(mali_c55->dev);
> >>>> +	if (ret)
> >>>> +		return ret;
> >>>> +
> >>>> +	ret = video_device_pipeline_start(&cap_dev->vdev,
> >>>> +					  &cap_dev->mali_c55->pipe);
> >>>> +	if (ret) {
> >>>> +		dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
> >>>> +			mali_c55_cap_dev_to_name(cap_dev));
> >>>> +		goto err_pm_put;
> >>>> +	}
> >>>> +
> >>>> +	mali_c55_cap_dev_stream_enable(cap_dev);
> >>>> +	mali_c55_rzr_start_stream(rzr);
> >>>> +
> >>>> +	/*
> >>>> +	 * We only start the ISP if we're the only capture device that's
> >>>> +	 * streaming. Otherwise, it'll already be active.
> >>>> +	 */
> >>>> +	if (mali_c55->pipe.start_count == 1) {
> >>>> +		ret = mali_c55_isp_start_stream(isp);
> >>>> +		if (ret)
> >>>> +			goto err_disable_cap_dev;
> >>>> +	}
> >>>> +
> >>>> +	return 0;
> >>>> +
> >>>> +err_disable_cap_dev:
> >>>> +	mali_c55_cap_dev_stream_disable(cap_dev);
> >>>> +	video_device_pipeline_stop(&cap_dev->vdev);
> >>>> +err_pm_put:
> >>>> +	pm_runtime_put(mali_c55->dev);
> >>>> +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
> >>>> +
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
> >>>> +{
> >>>> +	struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>> +	struct mali_c55_resizer *rzr = cap_dev->rzr;
> >>>> +	struct mali_c55_isp *isp = &mali_c55->isp;
> >>>> +
> >>>> +	guard(mutex)(&isp->lock);
> >>>> +
> >>>> +	/*
> >>>> +	 * If one of the other capture nodes is streaming, we shouldn't
> >>>> +	 * disable the ISP here.
> >>>> +	 */
> >>>> +	if (mali_c55->pipe.start_count == 1)
> >>>> +		mali_c55_isp_stop_stream(&mali_c55->isp);
> >>>> +
> >>>> +	mali_c55_rzr_stop_stream(rzr);
> >>>> +	mali_c55_cap_dev_stream_disable(cap_dev);
> >>>> +	mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
> >>>> +	video_device_pipeline_stop(&cap_dev->vdev);
> >>>> +	pm_runtime_put(mali_c55->dev);
> >>>> +}
> >>>> +
> >>>> +static const struct vb2_ops mali_c55_vb2_ops = {
> >>>> +	.queue_setup		= &mali_c55_vb2_queue_setup,
> >>>> +	.buf_queue		= &mali_c55_buf_queue,
> >>>> +	.buf_init		= &mali_c55_buf_init,
> >>>> +	.wait_prepare		= vb2_ops_wait_prepare,
> >>>> +	.wait_finish		= vb2_ops_wait_finish,
> >>>> +	.start_streaming	= &mali_c55_vb2_start_streaming,
> >>>> +	.stop_streaming		= &mali_c55_vb2_stop_streaming,
> >>>> +};
> >>>> +
> >>>> +static const struct v4l2_file_operations mali_c55_v4l2_fops = {
> >>>> +	.owner = THIS_MODULE,
> >>>> +	.unlocked_ioctl = video_ioctl2,
> >>>> +	.open = v4l2_fh_open,
> >>>> +	.release = vb2_fop_release,
> >>>> +	.poll = vb2_fop_poll,
> >>>> +	.mmap = vb2_fop_mmap,
> >>>> +};
> >>>> +
> >>>> +static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
> >>>> +{
> >>>> +	const struct mali_c55_fmt *capture_format;
> >>>> +	const struct v4l2_format_info *info;
> >>>> +	struct v4l2_plane_pix_format *plane;
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
> >>>> +	pix_mp->pixelformat = capture_format->fourcc;
> >>>> +
> >>>> +	pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
> >>>> +			      MALI_C55_MAX_WIDTH);
> >>>> +	pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
> >>>> +			       MALI_C55_MAX_HEIGHT);
> >>>> +
> >>>> +	pix_mp->field = V4L2_FIELD_NONE;
> >>>> +	pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>> +	pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>> +	pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
> >>>> +
> >>>> +	info = v4l2_format_info(pix_mp->pixelformat);
> >>>> +	pix_mp->num_planes = info->mem_planes;
> >>>> +	memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
> >>>> +
> >>>> +	pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;
> >>>> +	pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
> >>>> +				       * pix_mp->height;
> >>>> +
> >>>> +	for (i = 1; i < info->comp_planes; i++) {
> >>>> +		plane = &pix_mp->plane_fmt[i];
> >>>> +
> >>>> +		plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
> >>>> +						   info->hdiv);
> >>>> +		plane->sizeimage = DIV_ROUND_UP(
> >>>> +					plane->bytesperline * pix_mp->height,
> >>>> +					info->vdiv);
> >>>> +	}
> >>>> +
> >>>> +	if (info->mem_planes == 1) {
> >>>> +		for (i = 1; i < info->comp_planes; i++) {
> >>>> +			plane = &pix_mp->plane_fmt[i];
> >>>> +			pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
> >>>> +		}
> >>>> +	}
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>> +					   struct v4l2_format *f)
> >>>> +{
> >>>> +	mali_c55_try_fmt(&f->fmt.pix_mp);
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
> >>>> +				struct v4l2_pix_format_mplane *pix_mp)
> >>>> +{
> >>>> +	const struct mali_c55_fmt *capture_format;
> >>>> +	struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>> +	const struct v4l2_format_info *info;
> >>>> +
> >>>> +	mali_c55_try_fmt(pix_mp);
> >>>> +	capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
> >>>> +	info = v4l2_format_info(pix_mp->pixelformat);
> >>>> +	if (WARN_ON(!info))
> >>>> +		return;
> >>>> +
> >>>> +	mali_c55_write(mali_c55,
> >>>> +		       MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>> +		       capture_format->registers.base_mode);
> >>>> +	mali_c55_write(mali_c55,
> >>>> +		       MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
> >>>> +		       MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
> >>>> +		       MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
> >>>> +
> >>>> +	if (info->mem_planes > 1) {
> >>>> +		mali_c55_write(mali_c55,
> >>>> +			       MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>> +			       capture_format->registers.base_mode);
> >>>> +		mali_c55_update_bits(mali_c55,
> >>>> +				MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>> +				MALI_C55_WRITER_SUBMODE_MASK,
> >>>> +				capture_format->registers.uv_plane);
> >>>> +
> >>>> +		mali_c55_write(mali_c55,
> >>>> +			MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
> >>>> +			MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
> >>>> +			MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
> >>>> +	}
> >>>> +
> >>>> +	if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
> >>>> +		/*
> >>>> +		 * TODO: Figure out the colour matrix coefficients and calculate
> >>>> +		 * and write them here.
> >>>> +		 */
> >>>> +
> >>>> +		mali_c55_write(mali_c55,
> >>>> +			       MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>> +			       MALI_C55_CS_CONV_MATRIX_MASK);
> >>>> +
> >>>> +		if (info->hdiv > 1)
> >>>> +			mali_c55_update_bits(mali_c55,
> >>>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>> +				MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
> >>>> +		if (info->vdiv > 1)
> >>>> +			mali_c55_update_bits(mali_c55,
> >>>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>> +				MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
> >>>> +		if (info->hdiv > 1 || info->vdiv > 1)
> >>>> +			mali_c55_update_bits(mali_c55,
> >>>> +				MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>> +				MALI_C55_CS_CONV_FILTER_MASK, 0x01);
> >>>> +	}
> >>>> +
> >>>> +	cap_dev->mode.pix_mp = *pix_mp;
> >>>> +	cap_dev->mode.capture_fmt = capture_format;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>> +					 struct v4l2_format *f)
> >>>> +{
> >>>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >>>> +
> >>>> +	if (vb2_is_busy(&cap_dev->queue))
> >>>> +		return -EBUSY;
> >>>> +
> >>>> +	mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>> +					 struct v4l2_format *f)
> >>>> +{
> >>>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >>>> +
> >>>> +	f->fmt.pix_mp = cap_dev->mode.pix_mp;
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>> +					    struct v4l2_fmtdesc *f)
> >>>> +{
> >>>> +	struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >>>> +	unsigned int j = 0;
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >>>> +		if (f->mbus_code &&
> >>>> +		    !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
> >>>> +						       f->mbus_code))
> >>>> +			continue;
> >>>> +
> >>>> +		/* Downscale pipe can't output RAW formats */
> >>>> +		if (mali_c55_fmts[i].is_raw &&
> >>>> +		    cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
> >>>> +			continue;
> >>>> +
> >>>> +		if (j++ == f->index) {
> >>>> +			f->pixelformat = mali_c55_fmts[i].fourcc;
> >>>> +			return 0;
> >>>> +		}
> >>>> +	}
> >>>> +
> >>>> +	return -EINVAL;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_querycap(struct file *file, void *fh,
> >>>> +			     struct v4l2_capability *cap)
> >>>> +{
> >>>> +	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
> >>>> +	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
> >>>> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
> >>>> +	.vidioc_querybuf = vb2_ioctl_querybuf,
> >>>> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
> >>>> +	.vidioc_qbuf = vb2_ioctl_qbuf,
> >>>> +	.vidioc_expbuf = vb2_ioctl_expbuf,
> >>>> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
> >>>> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> >>>> +	.vidioc_streamon = vb2_ioctl_streamon,
> >>>> +	.vidioc_streamoff = vb2_ioctl_streamoff,
> >>>> +	.vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
> >>>> +	.vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
> >>>> +	.vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
> >>>> +	.vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
> >>>> +	.vidioc_querycap = mali_c55_querycap,
> >>>> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
> >>>> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
> >>>> +};
> >>>> +
> >>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	struct v4l2_pix_format_mplane pix_mp;
> >>>> +	struct mali_c55_cap_dev *cap_dev;
> >>>> +	struct video_device *vdev;
> >>>> +	struct vb2_queue *vb2q;
> >>>> +	unsigned int i;
> >>>> +	int ret;
> >>>> +
> >>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
> >>>> +		cap_dev = &mali_c55->cap_devs[i];
> >>>> +		vdev = &cap_dev->vdev;
> >>>> +		vb2q = &cap_dev->queue;
> >>>> +
> >>>> +		/*
> >>>> +		 * The downscale output pipe is an optional block within the ISP
> >>>> +		 * so we need to check whether it's actually been fitted or not.
> >>>> +		 */
> >>>> +
> >>>> +		if (i == MALI_C55_CAP_DEV_DS &&
> >>>> +		    !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
> >>>> +			continue;
> >>>> +
> >>>> +		cap_dev->mali_c55 = mali_c55;
> >>>> +		mutex_init(&cap_dev->lock);
> >>>> +		INIT_LIST_HEAD(&cap_dev->buffers.queue);
> >>>> +
> >>>> +		switch (i) {
> >>>> +		case MALI_C55_CAP_DEV_FR:
> >>>> +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
> >>>> +			cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
> >>>> +			break;
> >>>> +		case MALI_C55_CAP_DEV_DS:
> >>>> +			cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
> >>>> +			cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
> >>>> +			break;
> >>>> +		default:
> >>>> +			mutex_destroy(&cap_dev->lock);
> >>>> +			ret = -EINVAL;
> >>>> +			goto err_destroy_mutex;
> >>>> +		}
> >>>> +
> >>>> +		cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
> >>>> +		ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
> >>>> +		if (ret) {
> >>>> +			mutex_destroy(&cap_dev->lock);
> >>>> +			goto err_destroy_mutex;
> >>>> +		}
> >>>> +
> >>>> +		vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> >>>> +		vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
> >>>> +		vb2q->drv_priv = cap_dev;
> >>>> +		vb2q->mem_ops = &vb2_dma_contig_memops;
> >>>> +		vb2q->ops = &mali_c55_vb2_ops;
> >>>> +		vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
> >>>> +		vb2q->min_queued_buffers = 1;
> >>>> +		vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> >>>> +		vb2q->lock = &cap_dev->lock;
> >>>> +		vb2q->dev = mali_c55->dev;
> >>>> +
> >>>> +		ret = vb2_queue_init(vb2q);
> >>>> +		if (ret) {
> >>>> +			dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
> >>>> +				mali_c55_cap_dev_to_name(cap_dev));
> >>>> +			goto err_cleanup_media_entity;
> >>>> +		}
> >>>> +
> >>>> +		strscpy(cap_dev->vdev.name, capture_device_names[i],
> >>>> +			sizeof(cap_dev->vdev.name));
> >>>> +		vdev->release = video_device_release_empty;
> >>>> +		vdev->fops = &mali_c55_v4l2_fops;
> >>>> +		vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
> >>>> +		vdev->lock = &cap_dev->lock;
> >>>> +		vdev->v4l2_dev = &mali_c55->v4l2_dev;
> >>>> +		vdev->queue = &cap_dev->queue;
> >>>> +		vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> >>>> +				    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
> >>>> +		vdev->entity.ops = &mali_c55_media_ops;
> >>>> +		video_set_drvdata(vdev, cap_dev);
> >>>> +
> >>>> +		memset(&pix_mp, 0, sizeof(pix_mp));
> >>>> +		pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
> >>>> +		pix_mp.width = MALI_C55_DEFAULT_WIDTH;
> >>>> +		pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
> >>>> +		mali_c55_set_format(cap_dev, &pix_mp);
> >>>> +
> >>>> +		ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
> >>>> +		if (ret) {
> >>>> +			dev_err(mali_c55->dev,
> >>>> +				"%s failed to register video device\n",
> >>>> +				mali_c55_cap_dev_to_name(cap_dev));
> >>>> +			goto err_release_vb2q;
> >>>> +		}
> >>>> +	}
> >>>> +
> >>>> +	return 0;
> >>>> +
> >>>> +err_release_vb2q:
> >>>> +	vb2_queue_release(vb2q);
> >>>> +err_cleanup_media_entity:
> >>>> +	media_entity_cleanup(&cap_dev->vdev.entity);
> >>>> +err_destroy_mutex:
> >>>> +	mutex_destroy(&cap_dev->lock);
> >>>> +	mali_c55_unregister_capture_devs(mali_c55);
> >>>> +
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	struct mali_c55_cap_dev *cap_dev;
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
> >>>> +		cap_dev = &mali_c55->cap_devs[i];
> >>>> +
> >>>> +		if (!video_is_registered(&cap_dev->vdev))
> >>>> +			continue;
> >>>> +
> >>>> +		vb2_video_unregister_device(&cap_dev->vdev);
> >>>> +		media_entity_cleanup(&cap_dev->vdev.entity);
> >>>> +		mutex_destroy(&cap_dev->lock);
> >>>> +	}
> >>>> +}
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>> new file mode 100644
> >>>> index 000000000000..2d0c4d152beb
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>> @@ -0,0 +1,266 @@
> >>>> +/* SPDX-License-Identifier: GPL-2.0 */
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Common definitions
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#ifndef _MALI_C55_COMMON_H
> >>>> +#define _MALI_C55_COMMON_H
> >>>> +
> >>>> +#include <linux/clk.h>
> >>>> +#include <linux/io.h>
> >>>> +#include <linux/list.h>
> >>>> +#include <linux/mutex.h>
> >>>> +#include <linux/scatterlist.h>
> >>> 
> >>> I don't think this is needed. You're however missing spinlock.h.
> >>>
> >>>> +#include <linux/videodev2.h>
> >>>> +
> >>>> +#include <media/media-device.h>
> >>>> +#include <media/v4l2-async.h>
> >>>> +#include <media/v4l2-ctrls.h>
> >>>> +#include <media/v4l2-dev.h>
> >>>> +#include <media/v4l2-device.h>
> >>>> +#include <media/v4l2-subdev.h>
> >>>> +#include <media/videobuf2-core.h>
> >>>> +#include <media/videobuf2-v4l2.h>
> >>>> +
> >>>> +#define MALI_C55_DRIVER_NAME		"mali-c55"
> >>>> +
> >>>> +/* min and max values for the image sizes */
> >>>> +#define MALI_C55_MIN_WIDTH		640U
> >>>> +#define MALI_C55_MIN_HEIGHT		480U
> >>>> +#define MALI_C55_MAX_WIDTH		8192U
> >>>> +#define MALI_C55_MAX_HEIGHT		8192U
> >>>> +#define MALI_C55_DEFAULT_WIDTH		1920U
> >>>> +#define MALI_C55_DEFAULT_HEIGHT		1080U
> >>>> +
> >>>> +#define MALI_C55_DEFAULT_MEDIA_BUS_FMT	MEDIA_BUS_FMT_RGB121212_1X36
> >>>> +
> >>>> +struct mali_c55;
> >>>> +struct mali_c55_cap_dev;
> >>>> +struct platform_device;
> >>> 
> >>> You should also forward-declare
> >>>
> >>> struct device;
> >>> struct dma_chan;
> >>> struct resource;
> >>>
> >>>> +
> >>>> +static const char * const mali_c55_clk_names[] = {
> >>>> +	"aclk",
> >>>> +	"hclk",
> >>>> +};
> >>> 
> >>> This will end up duplicating the array in each compilation unit, not
> >>> great. Move it to mali-c55-core.c. You use it in this file just for its
> >>> size, replace that with a macro that defines the size, or allocate
> >>> mali_c55.clks dynamically with devm_kcalloc().
> >>>
> >>>> +
> >>>> +enum mali_c55_interrupts {
> >>>> +	MALI_C55_IRQ_ISP_START,
> >>>> +	MALI_C55_IRQ_ISP_DONE,
> >>>> +	MALI_C55_IRQ_MCM_ERROR,
> >>>> +	MALI_C55_IRQ_BROKEN_FRAME_ERROR,
> >>>> +	MALI_C55_IRQ_MET_AF_DONE,
> >>>> +	MALI_C55_IRQ_MET_AEXP_DONE,
> >>>> +	MALI_C55_IRQ_MET_AWB_DONE,
> >>>> +	MALI_C55_IRQ_AEXP_1024_DONE,
> >>>> +	MALI_C55_IRQ_IRIDIX_MET_DONE,
> >>>> +	MALI_C55_IRQ_LUT_INIT_DONE,
> >>>> +	MALI_C55_IRQ_FR_Y_DONE,
> >>>> +	MALI_C55_IRQ_FR_UV_DONE,
> >>>> +	MALI_C55_IRQ_DS_Y_DONE,
> >>>> +	MALI_C55_IRQ_DS_UV_DONE,
> >>>> +	MALI_C55_IRQ_LINEARIZATION_DONE,
> >>>> +	MALI_C55_IRQ_RAW_FRONTEND_DONE,
> >>>> +	MALI_C55_IRQ_NOISE_REDUCTION_DONE,
> >>>> +	MALI_C55_IRQ_IRIDIX_DONE,
> >>>> +	MALI_C55_IRQ_BAYER2RGB_DONE,
> >>>> +	MALI_C55_IRQ_WATCHDOG_TIMER,
> >>>> +	MALI_C55_IRQ_FRAME_COLLISION,
> >>>> +	MALI_C55_IRQ_UNUSED,
> >>>> +	MALI_C55_IRQ_DMA_ERROR,
> >>>> +	MALI_C55_IRQ_INPUT_STOPPED,
> >>>> +	MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
> >>>> +	MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
> >>>> +	MALI_C55_NUM_IRQ_BITS
> >>> 
> >>> Those are register bits, I think they belong to mali-c55-registers.h,
> >>> and should probably be macros instead of an enum.
> >>>
> >>>> +};
> >>>> +
> >>>> +enum mali_c55_isp_pads {
> >>>> +	MALI_C55_ISP_PAD_SINK_VIDEO,
> >>> 
> >>> As there's a single sink pad, maybe MALI_C55_ISP_PAD_SINK ? Ah, you're
> >>> probably preparing for ISP parameters support. It's fine.
> >>>
> >>>> +	MALI_C55_ISP_PAD_SOURCE,
> >>> 
> >>> Then maybe this should be named MALI_C55_ISP_PAD_SOURCE_VIDEO as I
> >>> assume there will be a stats source pad.
> >>>
> >>>> +	MALI_C55_ISP_PAD_SOURCE_BYPASS,
> >>>> +	MALI_C55_ISP_NUM_PADS,
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_tpg {
> >>>> +	struct mali_c55 *mali_c55;
> >>>> +	struct v4l2_subdev sd;
> >>>> +	struct media_pad pad;
> >>>> +	struct mutex lock;
> >>>> +	struct mali_c55_tpg_ctrls {
> >>>> +		struct v4l2_ctrl_handler handler;
> >>>> +		struct v4l2_ctrl *test_pattern;
> >>> 
> >>> Set but never used. You can drop it.
> >>>
> >>>> +		struct v4l2_ctrl *hblank;
> >>> 
> >>> Set and used only once, in the same function. You can make it a local
> >>> variable.
> >>>
> >>>> +		struct v4l2_ctrl *vblank;
> >>>> +	} ctrls;
> >>>> +};
> >>> 
> >>> I wonder if this file should be split, with mali-c55-capture.h,
> >>> mali-c55-core.h, mali-c55-isp.h, ... I think it could increase
> >>> readability by clearly separating the different elements. Up to you.
> >>>
> >>>> +
> >>>> +struct mali_c55_isp {
> >>>> +	struct mali_c55 *mali_c55;
> >>>> +	struct v4l2_subdev sd;
> >>>> +	struct media_pad pads[MALI_C55_ISP_NUM_PADS];
> >>>> +	struct media_pad *remote_src;
> >>>> +	struct v4l2_async_notifier notifier;
> >>> 
> >>> I'm tempted to move the notifier to mali_c55, as it's related to
> >>> components external to the whole ISP, not to the ISP subdev itself.
> >>> Could you give it a try, to see if it could be done without any drawback
> >>> ?
> >> 
> >> This seems to work fine.
> >>
> >>>> +	struct mutex lock;
> >>> 
> >>> Locks require a comment to explain what they protect. Same below where
> >>> applicable (for both mutexes and spinlocks).
> >>>
> >>>> +	unsigned int frame_sequence;
> >>>> +};
> >>>> +
> >>>> +enum mali_c55_resizer_ids {
> >>>> +	MALI_C55_RZR_FR,
> >>>> +	MALI_C55_RZR_DS,
> >>>> +	MALI_C55_NUM_RZRS,
> >>> 
> >>> The usual abbreviation for "resizer" in drivers/media/ is "rsz", not
> >>> "rzr". I would have said we can leave it as-is as changing it would be a
> >>> bit annoying, but I then realized that "rzr" is not just unusual, it's
> >>> actually not used at all. Would you mind applying a sed globally ? :-)
> >>>
> >>>> +};
> >>>> +
> >>>> +enum mali_c55_rzr_pads {
> >>> 
> >>> Same enums/structs use abbreviations, some don't. Consistency would
> >>> help.
> >>>
> >>>> +	MALI_C55_RZR_SINK_PAD,
> >>>> +	MALI_C55_RZR_SOURCE_PAD,
> >>>> +	MALI_C55_RZR_SINK_BYPASS_PAD,
> >>>> +	MALI_C55_RZR_NUM_PADS
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_resizer {
> >>>> +	struct mali_c55 *mali_c55;
> >>>> +	struct mali_c55_cap_dev *cap_dev;
> >>>> +	enum mali_c55_resizer_ids id;
> >>>> +	struct v4l2_subdev sd;
> >>>> +	struct media_pad pads[MALI_C55_RZR_NUM_PADS];
> >>>> +	unsigned int num_routes;
> >>>> +	bool streaming;
> >>>> +};
> >>>> +
> >>>> +enum mali_c55_cap_devs {
> >>>> +	MALI_C55_CAP_DEV_FR,
> >>>> +	MALI_C55_CAP_DEV_DS,
> >>>> +	MALI_C55_NUM_CAP_DEVS
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_fmt {
> >>> 
> >>> mali_c55_format_info would be a better name I think, as this stores
> >>> format information, not formats.
> >>>
> >>>> +	u32 fourcc;
> >>>> +	unsigned int mbus_codes[2];
> >>> 
> >>> A comment to explain why we have two media bus codes would be useful.
> >>> You can document the whole structure if desired :-)
> >>>
> >>>> +	bool is_raw;
> >>>> +	struct mali_c55_fmt_registers {
> >>> 
> >>> Make it an anonymous structure, it's never used anywhere else.
> >>>
> >>>> +		unsigned int base_mode;
> >>>> +		unsigned int uv_plane;
> >>> 
> >>> If those are register field values, use u32 instead of unsigned int.
> >>>
> >>>> +	} registers;
> >>> 
> >>> It's funny, we tend to abbreviate different things, I would have used
> >>> "regs" here but written "format" in full in the structure name :-)
> >>>
> >>>> +};
> >>>> +
> >>>> +enum mali_c55_isp_bayer_order {
> >>>> +	MALI_C55_BAYER_ORDER_RGGB,
> >>>> +	MALI_C55_BAYER_ORDER_GRBG,
> >>>> +	MALI_C55_BAYER_ORDER_GBRG,
> >>>> +	MALI_C55_BAYER_ORDER_BGGR
> >>> 
> >>> These are registers values too, they belong to mali-c55-registers.h.
> >>>
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_isp_fmt {
> >>> 
> >>> mali_c55_isp_format_info
> >>>
> >>>> +	u32 code;
> >>>> +	enum v4l2_pixel_encoding encoding;
> >>> 
> >>> Here you use v4l2_pixel_encoding, for mali_c55_fmt you use is_raw. Maybe
> >>> pick the same option for both structures ?
> >>>
> >>>> +	enum mali_c55_isp_bayer_order order;
> >>>> +};
> >>>> +
> >>>> +enum mali_c55_planes {
> >>>> +	MALI_C55_PLANE_Y,
> >>>> +	MALI_C55_PLANE_UV,
> >>>> +	MALI_C55_NUM_PLANES
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_buffer {
> >>>> +	struct vb2_v4l2_buffer vb;
> >>>> +	bool plane_done[MALI_C55_NUM_PLANES];
> >>> 
> >>> I think tracking the pending state would simplify the logic in
> >>> mali_c55_set_plane_done(), which would become
> >>>
> >>> 	curr_buf->plane_pending[plane] = false;
> >>>
> >>> 	if (!curr_buf->plane_pending[0] && !curr_buf->plane_pending[1]) {
> >>> 		mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> >>> 		cap_dev->buffers.curr = NULL;
> >>> 		isp->frame_sequence++;
> >>> 	}
> >>>
> >>> Or a counter may be even easier (and would consume less memory).
> >> 
> >> I'll do the counter; a  similar function in the stats code does so already.
> >>
> >>>> +	struct list_head queue;
> >>>> +	u32 addrs[MALI_C55_NUM_PLANES];
> >>> 
> >>> This stores DMA addresses, use dma_addr_t.
> >>>
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_cap_dev {
> >>>> +	struct mali_c55 *mali_c55;
> >>>> +	struct mali_c55_resizer *rzr;
> >>>> +	struct video_device vdev;
> >>>> +	struct media_pad pad;
> >>>> +	struct vb2_queue queue;
> >>>> +	struct mutex lock;
> >>>> +	unsigned int reg_offset;
> >>> 
> >>> Manual handling of the offset everywhere, with parametric macros for the
> >>> resizer register addresses, isn't very nice. Introduce resizer-specific
> >>> accessors and capture-specific accessors (mali_c55_rsz_write(), ...)
> >>> that take a mali_c55_resizer or mali_c55_cap_dev pointer, and handle the
> >>> offset there. The register macros should loose their offset parameter.
> >>>
> >>> You could also use a single set of accessors that would become
> >>> path/output-specific (mali_c55_path_write() ? mali_c55_output_write()
> >>> ?), that may make the code easier to read.
> >>>
> >>> You can also replace reg_offset with a void __iomem * base, which would
> >>> avoid the computation at runtime.
> >>>
> >>>> +
> >>>> +	struct mali_c55_mode {
> >>> 
> >>> Make the structure anonymous.
> >>>
> >>>> +		const struct mali_c55_fmt *capture_fmt;
> >>>> +		struct v4l2_pix_format_mplane pix_mp;
> >>>> +	} mode;
> >>> 
> >>> What's a "mode" ? I think I'd name this
> >>>
> >>> 	struct {
> >>> 		const struct mali_c55_fmt *info;
> >>> 		struct v4l2_pix_format_mplane format;
> >>> 	} format;
> >>>
> >>> Or you could just drop the structure and have
> >>>
> >>> 	const struct mali_c55_fmt *format_info;
> >>> 	struct v4l2_pix_format_mplane format;
> >>>
> >>> or something similar.
> >>>
> >>>> +
> >>>> +	struct {
> >>>> +		spinlock_t lock;
> >>>> +		struct list_head queue;
> >>>> +		struct mali_c55_buffer *curr;
> >>>> +		struct mali_c55_buffer *next;
> >>>> +	} buffers;
> >>>> +
> >>>> +	bool streaming;
> >>>> +};
> >>>> +
> >>>> +enum mali_c55_config_spaces {
> >>>> +	MALI_C55_CONFIG_PING,
> >>>> +	MALI_C55_CONFIG_PONG,
> >>>> +	MALI_C55_NUM_CONFIG_SPACES
> >>> 
> >>> The last enumerator is not used.
> >>>
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_ctx {
> >>> 
> >>> mali_c55_context ?
> >>>
> >>>> +	struct mali_c55 *mali_c55;
> >>>> +	void *registers;
> >>> 
> >>> Please document this structure and explain that this field points to a
> >>> copy of the register space in system memory, I was about to write you're
> >>> missing __iomem :-)
> >> 
> >> Will do
> >>
> >>>> +	phys_addr_t base;
> >>>> +	spinlock_t lock;
> >>>> +	struct list_head list;
> >>>> +};
> >>>> +
> >>>> +struct mali_c55 {
> >>>> +	struct device *dev;
> >>>> +	struct resource *res;
> >>> 
> >>> You could possibly drop this field by passing the physical address of
> >>> the register space from mali_c55_probe() to mali_c55_init_context() as a
> >>> function parameter.
> >>>
> >>>> +	void __iomem *base;
> >>>> +	struct dma_chan *channel;
> >>>> +	struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
> >>>> +
> >>>> +	u16 capabilities;
> >>>> +	struct media_device media_dev;
> >>>> +	struct v4l2_device v4l2_dev;
> >>>> +	struct media_pipeline pipe;
> >>>> +
> >>>> +	struct mali_c55_tpg tpg;
> >>>> +	struct mali_c55_isp isp;
> >>>> +	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
> >>>> +	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
> >>>> +
> >>>> +	struct list_head contexts;
> >>>> +	enum mali_c55_config_spaces next_config;
> >>>> +};
> >>>> +
> >>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
> >>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
> >>>> +		  bool force_hardware);
> >>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
> >>>> +			  u32 mask, u32 val);
> >>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
> >>>> +			  enum mali_c55_config_spaces cfg_space);
> >>>> +
> >>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55);
> >>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55);
> >>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
> >>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
> >>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55);
> >>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
> >>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
> >>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
> >>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
> >>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> >>>> +			     enum mali_c55_planes plane);
> >>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
> >>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
> >>>> +
> >>>> +bool mali_c55_format_is_raw(unsigned int mbus_code);
> >>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
> >>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
> >>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
> >>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
> >>>> +
> >>>> +const struct mali_c55_isp_fmt *
> >>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
> >>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
> >>>> +#define for_each_mali_isp_fmt(fmt)\
> >>> 
> >>> #define for_each_mali_isp_fmt(fmt) \
> >>>
> >>>> +	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
> >>> 
> >>> Looks like parentheses were on sale :-)
> >> 
> >> Hah
> >>
> >>> 	for ((fmt) = NULL; (fmt) = mali_c55_isp_fmt_next(fmt); )
> >>>
> >>> This macro is used in two places only, in the mali-c55-isp.c file where
> >>> open-coding the loop without using mali_c55_isp_fmt_next() would be more
> >>> efficient, and in mali-c55-resizer.c where a function to return format
> >>> i'th would be more efficient. I think you can drop the macro and the
> >>> mali_c55_isp_fmt_next() function.
> >>>
> >>>> +
> >>>> +#endif /* _MALI_C55_COMMON_H */
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>> new file mode 100644
> >>>> index 000000000000..50caf5ee7474
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>> @@ -0,0 +1,767 @@
> >>>> +// SPDX-License-Identifier: GPL-2.0
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Core driver code
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#include <linux/bitops.h>
> >>>> +#include <linux/cleanup.h>
> >>>> +#include <linux/clk.h>
> >>>> +#include <linux/delay.h>
> >>>> +#include <linux/device.h>
> >>>> +#include <linux/dmaengine.h>
> >>>> +#include <linux/dma-mapping.h>
> >>>> +#include <linux/interrupt.h>
> >>>> +#include <linux/iopoll.h>
> >>>> +#include <linux/ioport.h>
> >>>> +#include <linux/mod_devicetable.h>
> >>>> +#include <linux/of.h>
> >>>> +#include <linux/of_reserved_mem.h>
> >>>> +#include <linux/platform_device.h>
> >>>> +#include <linux/pm_runtime.h>
> >>>> +#include <linux/scatterlist.h>
> >>> 
> >>> I don't think this is needed.
> >>>
> >>> Missing slab.h.
> >>>
> >>>> +#include <linux/string.h>
> >>>> +
> >>>> +#include <media/media-entity.h>
> >>>> +#include <media/v4l2-device.h>
> >>>> +#include <media/videobuf2-dma-contig.h>
> >>>> +
> >>>> +#include "mali-c55-common.h"
> >>>> +#include "mali-c55-registers.h"
> >>>> +
> >>>> +static const char * const mali_c55_interrupt_names[] = {
> >>>> +	[MALI_C55_IRQ_ISP_START] = "ISP start",
> >>>> +	[MALI_C55_IRQ_ISP_DONE] = "ISP done",
> >>>> +	[MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
> >>>> +	[MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
> >>>> +	[MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
> >>>> +	[MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
> >>>> +	[MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
> >>>> +	[MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
> >>>> +	[MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
> >>>> +	[MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
> >>>> +	[MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
> >>>> +	[MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
> >>>> +	[MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
> >>>> +	[MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
> >>>> +	[MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
> >>>> +	[MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
> >>>> +	[MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
> >>>> +	[MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
> >>>> +	[MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
> >>>> +	[MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
> >>>> +	[MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
> >>>> +	[MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
> >>>> +	[MALI_C55_IRQ_DMA_ERROR] = "DMA error",
> >>>> +	[MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
> >>>> +	[MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
> >>>> +	[MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
> >>>> +};
> >>>> +
> >>>> +static unsigned int config_space_addrs[] = {
> >>> 
> >>> const
> >>>
> >>>> +	[MALI_C55_CONFIG_PING] = 0x0AB6C,
> >>>> +	[MALI_C55_CONFIG_PONG] = 0x22B2C,
> >>> 
> >>> Lowercase hex constants.
> >>>
> >>> Don't the values belong to mali-c55-registers.h ?
> >>>
> >>>> +};
> >>>> +
> >>>> +/* System IO
> >>> 
> >>> /*
> >>>    * System IO
> >>>
> >>>> + *
> >>>> + * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
> >>>> + * and 'pong'), with the  expectation that the 'active' space will be left
> >>> 
> >>> s/the  /the /
> >>>
> >>>> + * untouched whilst a frame is being processed and the 'inactive' space
> >>>> + * configured ready to be passed during the blanking period before the next
> >>> 
> >>> s/to be passed/to be switched to/ ?
> >>>
> >>>> + * frame processing starts. These spaces should ideally be set via DMA transfer
> >>>> + * from a buffer rather than through individual register set operations. There
> >>>> + * is also a shared global register space which should be set normally. Of
> >>>> + * course, the ISP might be included in a system which lacks a suitable DMA
> >>>> + * engine, and the second configuration space might not be fitted at all, which
> >>>> + * means we need to support four scenarios:
> >>>> + *
> >>>> + * 1. Multi config space, with DMA engine.
> >>>> + * 2. Multi config space, no DMA engine.
> >>>> + * 3. Single config space, with DMA engine.
> >>>> + * 4. Single config space, no DMA engine.
> >>>> + *
> >>>> + * The first case is very easy, but the rest present annoying problems. The best
> >>>> + * way to solve them seems to be simply to replicate the concept of DMAing over
> >>>> + * the configuration buffer even if there's no DMA engine on the board, for
> >>>> + * which we rely on memcpy. To facilitate this any read/write call that is made
> >>>> + * to an address within those config spaces should infact be directed to a
> >>>> + * buffer that was allocated to hold them rather than the IO memory itself. The
> >>>> + * actual copy of that buffer to IO mem will happen on interrupt.
> >>>> + */
> >>>> +
> >>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
> >>>> +{
> >>>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>> +
> >>>> +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
> >>>> +		spin_lock(&ctx->lock);
> >>>> +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
> >>>> +		((u32 *)ctx->registers)[addr] = val;
> >>>> +		spin_unlock(&ctx->lock);
> >>>> +
> >>>> +		return;
> >>>> +	}
> >>> 
> >>> Ouch. This is likely the second comment you really won't like (after the
> >>> comment regarding mali_c55_update_bits() at the very top). I apologize
> >>> in advance.
> >> 
> >> All good - it's the thinking that is painful, changing the code is easy :)
> >>
> >>> I really don't like this. Directing writes either to hardware registers
> >>> or to the shadow registers in the context makes the callers of the
> >>> read/write accessors very hard to read. The probe code, for instance,
> >>> mixes writes to hardware registers and writes to the context shadow
> >>> registers to initialize the value of some of the shadow registers.
> >>>
> >>> I'd like to split the read/write accessors into functions that access
> >>> the hardware registers (that's easy) and functions that access the
> >>> shadow registers. I think the latter should receive a mali_c55_ctx
> >>> pointer instead of a mali_c55 pointer to prepare for multi-context
> >>> support.
> >>>
> >>> You can add WARN_ON() guards to the two sets of functions, to ensure
> >>> that no register from the "other" space gets passed to the wrong
> >>> function by mistake.
> >>>
> >>>> +
> >>>> +	writel(val, mali_c55->base + addr);
> >>>> +}
> >>>> +
> >>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
> >>>> +		  bool force_hardware)
> >>> 
> >>> force_hardware is never set to true.
> >>>
> >>>> +{
> >>>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>> +	u32 val;
> >>>> +
> >>>> +	if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
> >>>> +		spin_lock(&ctx->lock);
> >>>> +		addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
> >>>> +		val = ((u32 *)ctx->registers)[addr];
> >>>> +		spin_unlock(&ctx->lock);
> >>>> +
> >>>> +		return val;
> >>>> +	}
> >>>> +
> >>>> +	return readl(mali_c55->base + addr);
> >>>> +}
> >>>> +
> >>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
> >>>> +			  u32 mask, u32 val)
> >>>> +{
> >>>> +	u32 orig, tmp;
> >>>> +
> >>>> +	orig = mali_c55_read(mali_c55, addr, false);
> >>>> +
> >>>> +	tmp = orig & ~mask;
> >>>> +	tmp |= (val << (ffs(mask) - 1)) & mask;
> >>>> +
> >>>> +	if (tmp != orig)
> >>>> +		mali_c55_write(mali_c55, addr, tmp);
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
> >>>> +			     dma_addr_t dst, enum dma_data_direction dir)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>> +	struct dma_async_tx_descriptor *tx;
> >>>> +	enum dma_status status;
> >>>> +	dma_cookie_t cookie;
> >>>> +
> >>>> +	tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
> >>>> +				       MALI_C55_CONFIG_SPACE_SIZE, 0);
> >>>> +	if (!tx) {
> >>>> +		dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
> >>>> +		return -EIO;
> >>>> +	}
> >>>> +
> >>>> +	cookie = dmaengine_submit(tx);
> >>>> +	if (dma_submit_error(cookie)) {
> >>>> +		dev_err(mali_c55->dev, "error submitting dma transfer\n");
> >>>> +		return -EIO;
> >>>> +	}
> >>>> +
> >>>> +	status = dma_sync_wait(mali_c55->channel, cookie);
> >>> 
> >>> I've just realized this performs a busy-wait :-S See the comment in the
> >>> probe function about the threaded IRQ handler. I think we'll need to
> >>> rework all this. It could be done on top though.
> >> 
> >> It can be switched to an asynchronous transfer quite easily.
> >>
> >>>> +	if (status != DMA_COMPLETE) {
> >>>> +		dev_err(mali_c55->dev, "dma transfer failed\n");
> >>>> +		return -EIO;
> >>>> +	}
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
> >>>> +			     enum mali_c55_config_spaces cfg_space)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>> +	struct device *dma_dev = mali_c55->channel->device->dev;
> >>>> +	dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
> >>>> +	dma_addr_t dst;
> >>>> +	int ret;
> >>>> +
> >>>> +	guard(spinlock)(&ctx->lock);
> >>>> +
> >>>> +	dst = dma_map_single(dma_dev, ctx->registers,
> >>>> +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
> >>>> +	if (dma_mapping_error(dma_dev, dst)) {
> >>>> +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
> >>>> +		return -EIO;
> >>>> +	}
> >>>> +
> >>>> +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
> >>>> +	dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
> >>>> +			 DMA_FROM_DEVICE);
> >>>> +
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
> >>>> +		       enum mali_c55_config_spaces cfg_space)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>> +	struct device *dma_dev = mali_c55->channel->device->dev;
> >>>> +	dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
> >>>> +	dma_addr_t src;
> >>>> +	int ret;
> >>>> +
> >>>> +	guard(spinlock)(&ctx->lock);
> >>> 
> >>> The code below can take a large amount of time, holding a spinlock will
> >>> disable interrupts on the local CPU, that's not good :-(
> >> 
> >> The intention here is just to prevent the rest of the driver writing into the register space whilst
> >> it's being DMA transferred to the hardware - perhaps a different means to signal it's safe is more
> >> appropriate?
> > 
> > Are there other parts of the driver that can write to the config space,
> > or do all writes to the config space go through this function ?
> 
> All writes to the hardware go through this function, but what I meant
> was to prevent other parts of the driver writing into the buffer at
> ctx->registers whilst this write is occurring

I'm not sure how we can solve that, but it still feels very wrong. It's
related to tbe busy wait comment above.

> >>>> +
> >>>> +	src = dma_map_single(dma_dev, ctx->registers,
> >>>> +			     MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
> >>>> +	if (dma_mapping_error(dma_dev, src)) {
> >>>> +		dev_err(mali_c55->dev, "failed to map DMA addr\n");
> >>>> +		return -EIO;
> >>>> +	}
> >>>> +
> >>>> +	ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
> >>>> +	dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
> >>>> +			 DMA_TO_DEVICE);
> >>>> +
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_config_read(struct mali_c55_ctx *ctx,
> >>>> +				enum mali_c55_config_spaces cfg_space)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>> +
> >>>> +	if (mali_c55->channel) {
> >>>> +		return mali_c55_dma_read(ctx, cfg_space);
> >>> 
> >>> As this function is used at probe time only, to initialize the context,
> >>> I think DMA is overkill.
> >> 
> >> Agreed - I'll switch this to memcpy_fromio()
> >>
> >>>> +	} else {
> >>>> +		memcpy_fromio(ctx->registers,
> >>>> +			      mali_c55->base + config_space_addrs[cfg_space],
> >>>> +			      MALI_C55_CONFIG_SPACE_SIZE);
> >>>> +		return 0;
> >>>> +	}
> >>>> +}
> >>>> +
> >>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
> >>>> +			  enum mali_c55_config_spaces cfg_space)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>> +
> >>>> +	if (mali_c55->channel) {
> >>>> +		return mali_c55_dma_write(ctx, cfg_space);
> >>>> +	} else {
> >>>> +		memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
> >>>> +			    ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
> >>>> +		return 0;
> >>>> +	}
> >>> 
> >>> Could you measure the time it typically takes to write the registers
> >>> using DMA compared to using memcpy_toio() ?
> >> 
> >> I will test this and come back to you.
> >>
> >>>> +}
> >>>> +
> >>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);
> >>> 
> >>> I think it's too early to tell how multi-context support will look like.
> >>> I'm fine keeping mali_c55_get_active_context() as changing that would be
> >>> very intrusive (even if I think it will need to be changed), but the
> >>> list of contexts is neither the mechanism we'll use, nor something we
> >>> need now. Drop the list, embed the context in struct mali_c55, and
> >>> return the pointer to that single context from this function.
> >>>
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_remove_links(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	media_entity_remove_links(&mali_c55->tpg.sd.entity);
> >>>> +	media_entity_remove_links(&mali_c55->isp.sd.entity);
> >>>> +
> >>>> +	for (i = 0; i < MALI_C55_NUM_RZRS; i++)
> >>>> +		media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
> >>>> +
> >>>> +	for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
> >>>> +		media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_create_links(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	int ret;
> >>>> +
> >>>> +	/* Test pattern generator to ISP */
> >>>> +	ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
> >>>> +				    &mali_c55->isp.sd.entity,
> >>>> +				    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
> >>>> +	if (ret) {
> >>>> +		dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
> >>>> +		goto err_remove_links;
> >>>> +	}
> >>>> +
> >>>> +	/* Full resolution resizer pipe. */
> >>>> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >>>> +			MALI_C55_ISP_PAD_SOURCE,
> >>>> +			&mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
> >>>> +			MALI_C55_RZR_SINK_PAD,
> >>>> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>> +	if (ret) {
> >>>> +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
> >>>> +		goto err_remove_links;
> >>>> +	}
> >>>> +
> >>>> +	/* Full resolution bypass. */
> >>>> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >>>> +				    MALI_C55_ISP_PAD_SOURCE_BYPASS,
> >>>> +				    &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
> >>>> +				    MALI_C55_RZR_SINK_BYPASS_PAD,
> >>>> +				    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>> +	if (ret) {
> >>>> +		dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
> >>>> +		goto err_remove_links;
> >>>> +	}
> >>>> +
> >>>> +	/* Resizer pipe to video capture nodes. */
> >>>> +	ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
> >>>> +			MALI_C55_RZR_SOURCE_PAD,
> >>>> +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
> >>>> +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>> +	if (ret) {
> >>>> +		dev_err(mali_c55->dev,
> >>>> +			"failed to link FR resizer and video device\n");
> >>>> +		goto err_remove_links;
> >>>> +	}
> >>>> +
> >>>> +	/* The downscale pipe is an optional hardware block */
> >>>> +	if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
> >>>> +		ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >>>> +			MALI_C55_ISP_PAD_SOURCE,
> >>>> +			&mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
> >>>> +			MALI_C55_RZR_SINK_PAD,
> >>>> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>> +		if (ret) {
> >>>> +			dev_err(mali_c55->dev,
> >>>> +				"failed to link ISP and DS resizer\n");
> >>>> +			goto err_remove_links;
> >>>> +		}
> >>>> +
> >>>> +		ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
> >>>> +			MALI_C55_RZR_SOURCE_PAD,
> >>>> +			&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
> >>>> +			0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>> +		if (ret) {
> >>>> +			dev_err(mali_c55->dev,
> >>>> +				"failed to link DS resizer and video device\n");
> >>>> +			goto err_remove_links;
> >>>> +		}
> >>>> +	}
> >>>> +
> >>>> +	return 0;
> >>>> +
> >>>> +err_remove_links:
> >>>> +	mali_c55_remove_links(mali_c55);
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	mali_c55_unregister_tpg(mali_c55);
> >>>> +	mali_c55_unregister_isp(mali_c55);
> >>>> +	mali_c55_unregister_resizers(mali_c55);
> >>>> +	mali_c55_unregister_capture_devs(mali_c55);
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_register_entities(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	int ret;
> >>>> +
> >>>> +	ret = mali_c55_register_tpg(mali_c55);
> >>>> +	if (ret)
> >>>> +		return ret;
> >>>> +
> >>>> +	ret = mali_c55_register_isp(mali_c55);
> >>>> +	if (ret)
> >>>> +		goto err_unregister_entities;
> >>>> +
> >>>> +	ret = mali_c55_register_resizers(mali_c55);
> >>>> +	if (ret)
> >>>> +		goto err_unregister_entities;
> >>>> +
> >>>> +	ret = mali_c55_register_capture_devs(mali_c55);
> >>>> +	if (ret)
> >>>> +		goto err_unregister_entities;
> >>>> +
> >>>> +	ret = mali_c55_create_links(mali_c55);
> >>>> +	if (ret)
> >>>> +		goto err_unregister_entities;
> >>>> +
> >>>> +	return 0;
> >>>> +
> >>>> +err_unregister_entities:
> >>>> +	mali_c55_unregister_entities(mali_c55);
> >>>> +
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	u32 product, version, revision, capabilities;
> >>>> +
> >>>> +	product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
> >>>> +	version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
> >>>> +	revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
> >>>> +
> >>>> +	dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
> >>>> +		 product, version, revision);
> >>>> +
> >>>> +	capabilities = mali_c55_read(mali_c55,
> >>>> +				     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
> >>>> +				     false);
> >>>> +	mali_c55->capabilities = (capabilities & 0xffff);
> >>>> +
> >>>> +	/* TODO: Might as well start some debugfs */
> >>> 
> >>> If it's just to expose the version and capabilities, I think that's
> >>> overkill. It's not needed for debug purpose (you can get it from the
> >>> kernel log already). debugfs isn't meant to be accessible in production,
> >>> so an application that would need access to the information wouldn't be
> >>> able to use it.
> >>>
> >>>> +	dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);
> >>> 
> >>> Combine the two messages into one.
> >>>
> >>>> +	return version;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>> +	u32 curr_config, next_config;
> >>>> +
> >>>> +	curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
> >>>> +	curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
> >>>> +		      >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
> >>>> +	next_config = curr_config ^ 1;
> >>>> +
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>> +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
> >>>> +	mali_c55_config_write(ctx, next_config ?
> >>>> +			      MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
> >>>> +}
> >>>> +
> >>>> +static irqreturn_t mali_c55_isr(int irq, void *context)
> >>>> +{
> >>>> +	struct device *dev = context;
> >>>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >>>> +	u32 interrupt_status;
> >>>> +	unsigned int i, j;
> >>>> +
> >>>> +	interrupt_status = mali_c55_read(mali_c55,
> >>>> +					 MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
> >>>> +					 false);
> >>>> +	if (!interrupt_status)
> >>>> +		return IRQ_NONE;
> >>>> +
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
> >>>> +		       interrupt_status);
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
> >>>> +
> >>>> +	for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
> >>>> +		if (!(interrupt_status & (1 << i)))
> >>>> +			continue;
> >>>> +
> >>>> +		switch (i) {
> >>>> +		case MALI_C55_IRQ_ISP_START:
> >>>> +			mali_c55_isp_queue_event_sof(mali_c55);
> >>>> +
> >>>> +			for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
> >>>> +				mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
> >>>> +
> >>>> +			mali_c55_swap_next_config(mali_c55);
> >>>> +
> >>>> +			break;
> >>>> +		case MALI_C55_IRQ_ISP_DONE:
> >>>> +			/*
> >>>> +			 * TODO: Where the ISP has no Pong config fitted, we'd
> >>>> +			 * have to do the mali_c55_swap_next_config() call here.
> >>>> +			 */
> >>>> +			break;
> >>>> +		case MALI_C55_IRQ_FR_Y_DONE:
> >>>> +			mali_c55_set_plane_done(
> >>>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
> >>>> +				MALI_C55_PLANE_Y);
> >>>> +			break;
> >>>> +		case MALI_C55_IRQ_FR_UV_DONE:
> >>>> +			mali_c55_set_plane_done(
> >>>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
> >>>> +				MALI_C55_PLANE_UV);
> >>>> +			break;
> >>>> +		case MALI_C55_IRQ_DS_Y_DONE:
> >>>> +			mali_c55_set_plane_done(
> >>>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
> >>>> +				MALI_C55_PLANE_Y);
> >>>> +			break;
> >>>> +		case MALI_C55_IRQ_DS_UV_DONE:
> >>>> +			mali_c55_set_plane_done(
> >>>> +				&mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
> >>>> +				MALI_C55_PLANE_UV);
> >>>> +			break;
> >>>> +		default:
> >>>> +			/*
> >>>> +			 * Only the above interrupts are currently unmasked. If
> >>>> +			 * we receive anything else here then something weird
> >>>> +			 * has gone on.
> >>>> +			 */
> >>>> +			dev_err(dev, "masked interrupt %s triggered\n",
> >>>> +				mali_c55_interrupt_names[i]);
> >>>> +		}
> >>>> +	}
> >>>> +
> >>>> +	return IRQ_HANDLED;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_init_context(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	struct mali_c55_ctx *ctx;
> >>>> +	int ret;
> >>>> +
> >>>> +	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
> >>>> +	if (!ctx) {
> >>>> +		dev_err(mali_c55->dev, "failed to allocate new context\n");
> >>> 
> >>> No need for an error message when memory allocation fails.
> >>>
> >>>> +		return -ENOMEM;
> >>>> +	}
> >>>> +
> >>>> +	ctx->base = mali_c55->res->start;
> >>>> +	ctx->mali_c55 = mali_c55;
> >>>> +
> >>>> +	ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
> >>>> +				 GFP_KERNEL | GFP_DMA);
> >>>> +	if (!ctx->registers) {
> >>>> +		ret = -ENOMEM;
> >>>> +		goto err_free_ctx;
> >>>> +	}
> >>>> +
> >>>> +	/*
> >>>> +	 * The allocated memory is empty, we need to load the default
> >>>> +	 * register settings. We just read Ping; it's identical to Pong.
> >>>> +	 */
> >>>> +	ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
> >>>> +	if (ret)
> >>>> +		goto err_free_registers;
> >>>> +
> >>>> +	list_add_tail(&ctx->list, &mali_c55->contexts);
> >>>> +
> >>>> +	/*
> >>>> +	 * Some features of the ISP need to be disabled by default and only
> >>>> +	 * enabled at the same time as they're configured by a parameters buffer
> >>>> +	 */
> >>>> +
> >>>> +	/* Bypass the sqrt and square compression and expansion modules */
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
> >>>> +			     MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
> >>>> +			     MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
> >>>> +
> >>>> +	/* Bypass the temper module */
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
> >>>> +		       MALI_C55_REG_BYPASS_2_TEMPER);
> >>>> +
> >>>> +	/* Bypass the colour noise reduction  */
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
> >>>> +		       MALI_C55_REG_BYPASS_4_CNR);
> >>>> +
> >>>> +	/* Disable the sinter module */
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
> >>>> +			     MALI_C55_SINTER_ENABLE_MASK, 0x00);
> >>>> +
> >>>> +	/* Disable the RGB Gamma module for each output */
> >>>> +	mali_c55_write(
> >>>> +		mali_c55,
> >>>> +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
> >>>> +		0x00);
> >>>> +	mali_c55_write(
> >>>> +		mali_c55,
> >>>> +		MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
> >>>> +		0x00);
> >>>> +
> >>>> +	/* Disable the colour correction matrix */
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
> >>>> +
> >>>> +	return 0;
> >>>> +
> >>>> +err_free_registers:
> >>>> +	kfree(ctx->registers);
> >>>> +err_free_ctx:
> >>>> +	kfree(ctx);
> >>>> +
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_runtime_resume(struct device *dev)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >>>> +	int ret;
> >>>> +
> >>>> +	ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
> >>>> +				      mali_c55->clks);
> >>>> +	if (ret)
> >>>> +		dev_err(mali_c55->dev, "failed to enable clocks\n");
> >>>> +
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_runtime_suspend(struct device *dev)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >>>> +
> >>>> +	clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct dev_pm_ops mali_c55_pm_ops = {
> >>>> +	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> >>>> +				pm_runtime_force_resume)
> >>>> +	SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
> >>>> +			   NULL)
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_probe(struct platform_device *pdev)
> >>>> +{
> >>>> +	struct device *dev = &pdev->dev;
> >>>> +	struct mali_c55 *mali_c55;
> >>>> +	dma_cap_mask_t mask;
> >>>> +	u32 version;
> >>>> +	int ret;
> >>>> +	u32 val;
> >>>> +
> >>>> +	mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
> >>>> +	if (!mali_c55)
> >>>> +		return dev_err_probe(dev, -ENOMEM,
> >>>> +				     "failed to allocate memory\n");
> >>> 
> >>> 		return -ENOMEM;
> >>>
> >>> There's no need to print messages for memory allocation failures.
> >>>
> >>>> +
> >>>> +	mali_c55->dev = dev;
> >>>> +	platform_set_drvdata(pdev, mali_c55);
> >>>> +
> >>>> +	mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
> >>>> +								&mali_c55->res);
> >>>> +	if (IS_ERR(mali_c55->base))
> >>>> +		return dev_err_probe(dev, PTR_ERR(mali_c55->base),
> >>>> +				     "failed to map IO memory\n");
> >>>> +
> >>>> +	ret = platform_get_irq(pdev, 0);
> >>>> +	if (ret < 0)
> >>>> +		return dev_err_probe(dev, ret, "failed to get interrupt num\n");
> >>> 
> >>> s/ num// or s/num/number/
> >>>
> >>>> +
> >>>> +	ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
> >>>> +					mali_c55_isr, IRQF_ONESHOT,
> >>>> +					dev_driver_string(&pdev->dev),
> >>>> +					&pdev->dev);
> >>> 
> >>> Requested the IRQ should be done much lower, after you have initialized
> >>> everything, or an IRQ that would fire early would have really bad
> >>> consequences.
> >>>
> >>> A comment to explain why you need a threaded interrupt handler would be
> >>> good. I assume it is due only to the need to transfer the registers
> >>> using DMA. I wonder if we should then split the interrupt handler in
> >>> two, with a non-threaded part for the operations that can run quickly,
> >>> and a threaded part for the reprogramming.
> >> 
> >> Instead of passing NULL for the top handler you mean?
> > 
> > Yes.
> >
> >>> It may also be that we could just start the DMA transfer in the
> >>> non-threaded handler without waiting synchronously for it to complete.
> >>> That would be a bigger change, and would require checking race
> >>> conditions carefully. On the other hand, I'm a bit concerned about the
> >>> current implementation, have you tested what happens if the DMA transfer
> >>> takes too long to complete, and spans frame boundaries ?
> >> 
> >> No; I can force a delay in the DMA engine and see how it behaves. The stats buffers are currently
> >> DMAd asynchronously; I don't think it'd be a huge change to make the configuration buffer handling
> >> work that way too.
> > 
> > That would be nice. Race conditions will need to be carefully considered
> > (but they should, anyway).
> >
> >>>> +	if (ret)
> >>>> +		return dev_err_probe(dev, ret, "failed to request irq\n");
> >>>> +
> >>>> +	for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
> >>>> +		mali_c55->clks[i].id = mali_c55_clk_names[i];
> >>>> +
> >>>> +	ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
> >>>> +	if (ret)
> >>>> +		return dev_err_probe(dev, ret, "failed to acquire clocks\n");
> >>>> +
> >>>> +	pm_runtime_enable(&pdev->dev);
> >>>> +
> >>>> +	ret = pm_runtime_resume_and_get(&pdev->dev);
> >>>> +	if (ret)
> >>>> +		goto err_pm_runtime_disable;
> >>>> +
> >>>> +	of_reserved_mem_device_init(dev);
> >>> 
> >>> I'd move this, the vb2_dma_contig_set_max_seg_size() call and the
> >>> dma_cap_* calls before pm_runtime_enable() as they don't need the device
> >>> to be powered.
> >>>
> >>>> +	version = mali_c55_check_hwcfg(mali_c55);
> >>>> +	vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
> >>>> +
> >>>> +	/* Use "software only" context management. */
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>> +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> >>>> +			     MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> >>> 
> >>> You handle that in mali_c55_isp_start(), does the register have to be
> >>> set here too ?
> >>>
> >>>> +
> >>>> +	dma_cap_zero(mask);
> >>>> +	dma_cap_set(DMA_MEMCPY, mask);
> >>>> +
> >>>> +	/*
> >>>> +	 * No error check, because we will just fallback on memcpy if there is
> >>>> +	 * no usable DMA channel on the system.
> >>>> +	 */
> >>>> +	mali_c55->channel = dma_request_channel(mask, NULL, NULL);
> >>>> +
> >>>> +	INIT_LIST_HEAD(&mali_c55->contexts);
> >>>> +	ret = mali_c55_init_context(mali_c55);
> >>>> +	if (ret)
> >>>> +		goto err_release_dma_channel;
> >>>> +
> >>> 
> >>> I'd move all the code from here ...
> >>>
> >>>> +	mali_c55->media_dev.dev = dev;
> >>>> +	strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
> >>>> +		sizeof(mali_c55->media_dev.model));
> >>>> +	mali_c55->media_dev.hw_revision = version;
> >>>> +
> >>>> +	media_device_init(&mali_c55->media_dev);
> >>>> +	ret = media_device_register(&mali_c55->media_dev);
> >>>> +	if (ret)
> >>>> +		goto err_cleanup_media_device;
> >>>> +
> >>>> +	mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
> >>>> +	ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
> >>>> +	if (ret) {
> >>>> +		dev_err(dev, "failed to register V4L2 device\n");
> >>>> +		goto err_unregister_media_device;
> >>>> +	};
> >>>> +
> >>>> +	ret = mali_c55_register_entities(mali_c55);
> >>>> +	if (ret) {
> >>>> +		dev_err(dev, "failed to register entities\n");
> >>>> +		goto err_unregister_v4l2_device;
> >>>> +	}
> >>> 
> >>> ... to here to a separate function, or maybe fold it all in
> >>> mali_c55_register_entities() (which should the be renamed). Same thing
> >>> for the cleanup code.
> >>>
> >>>> +
> >>>> +	/* Set safe stop to ensure we're in a non-streaming state */
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> >>>> +		       MALI_C55_INPUT_SAFE_STOP);
> >>>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >>>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >>>> +
> >>>> +	/*
> >>>> +	 * We're ready to process interrupts. Clear any that are set and then
> >>>> +	 * unmask them for processing.
> >>>> +	 */
> >>>> +	mali_c55_write(mali_c55, 0x30, 0xffffffff);
> >>>> +	mali_c55_write(mali_c55, 0x34, 0xffffffff);
> >>>> +	mali_c55_write(mali_c55, 0x40, 0x01);
> >>>> +	mali_c55_write(mali_c55, 0x40, 0x00);
> >>> 
> >>> Please replace the register addresses with macros.
> >>>
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);
> >>> 
> >>> The value should use the interrupt bits macros.
> >>>
> >>>> +
> >>>> +	pm_runtime_put(&pdev->dev);
> >>> 
> >>> Once power gets cut, the registers your programmed above may be lost. I
> >>> think you should programe them in the runtime PM resume handler.
> >>>
> >>>> +
> >>>> +	return 0;
> >>>> +
> >>>> +err_unregister_v4l2_device:
> >>>> +	v4l2_device_unregister(&mali_c55->v4l2_dev);
> >>>> +err_unregister_media_device:
> >>>> +	media_device_unregister(&mali_c55->media_dev);
> >>>> +err_cleanup_media_device:
> >>>> +	media_device_cleanup(&mali_c55->media_dev);
> >>>> +err_release_dma_channel:
> >>>> +	dma_release_channel(mali_c55->channel);
> >>>> +err_pm_runtime_disable:
> >>>> +	pm_runtime_disable(&pdev->dev);
> >>>> +
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_remove(struct platform_device *pdev)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
> >>>> +	struct mali_c55_ctx *ctx, *tmp;
> >>>> +
> >>>> +	list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
> >>>> +		list_del(&ctx->list);
> >>>> +		kfree(ctx->registers);
> >>>> +		kfree(ctx);
> >>>> +	}
> >>>> +
> >>>> +	mali_c55_remove_links(mali_c55);
> >>>> +	mali_c55_unregister_entities(mali_c55);
> >>>> +	v4l2_device_put(&mali_c55->v4l2_dev);
> >>>> +	media_device_unregister(&mali_c55->media_dev);
> >>>> +	media_device_cleanup(&mali_c55->media_dev);
> >>>> +	dma_release_channel(mali_c55->channel);
> >>>> +}
> >>>> +
> >>>> +static const struct of_device_id mali_c55_of_match[] = {
> >>>> +	{ .compatible = "arm,mali-c55", },
> >>>> +	{},
> >>> 
> >>> 	{ /* Sentinel */ },
> >>>
> >>>> +};
> >>>> +MODULE_DEVICE_TABLE(of, mali_c55_of_match);
> >>>> +
> >>>> +static struct platform_driver mali_c55_driver = {
> >>>> +	.driver = {
> >>>> +		.name = "mali-c55",
> >>>> +		.of_match_table = of_match_ptr(mali_c55_of_match),
> >>> 
> >>> Drop of_match_ptr().
> >>>
> >>>> +		.pm = &mali_c55_pm_ops,
> >>>> +	},
> >>>> +	.probe = mali_c55_probe,
> >>>> +	.remove_new = mali_c55_remove,
> >>>> +};
> >>>> +
> >>>> +module_platform_driver(mali_c55_driver);
> >>> 
> >>> Blank line.
> >>>
> >>>> +MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
> >>>> +MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
> >>>> +MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
> >>>> +MODULE_LICENSE("GPL");
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>> new file mode 100644
> >>>> index 000000000000..ea8b7b866e7a
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>> @@ -0,0 +1,611 @@
> >>>> +// SPDX-License-Identifier: GPL-2.0
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Image signal processor
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#include <linux/delay.h>
> >>>> +#include <linux/iopoll.h>
> >>>> +#include <linux/property.h>
> >>>> +#include <linux/string.h>
> >>>> +
> >>>> +#include <media/media-entity.h>
> >>>> +#include <media/v4l2-common.h>
> >>>> +#include <media/v4l2-event.h>
> >>>> +#include <media/v4l2-mc.h>
> >>>> +#include <media/v4l2-subdev.h>
> >>>> +
> >>>> +#include "mali-c55-common.h"
> >>>> +#include "mali-c55-registers.h"
> >>>> +
> >>>> +static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
> >>>> +	{
> >>>> +		.code = MEDIA_BUS_FMT_SRGGB20_1X20,
> >>>> +		.order = MALI_C55_BAYER_ORDER_RGGB,
> >>>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
> >>>> +	},
> >>>> +	{
> >>>> +		.code = MEDIA_BUS_FMT_SGRBG20_1X20,
> >>>> +		.order = MALI_C55_BAYER_ORDER_GRBG,
> >>>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
> >>>> +	},
> >>>> +	{
> >>>> +		.code = MEDIA_BUS_FMT_SGBRG20_1X20,
> >>>> +		.order = MALI_C55_BAYER_ORDER_GBRG,
> >>>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
> >>>> +	},
> >>>> +	{
> >>>> +		.code = MEDIA_BUS_FMT_SBGGR20_1X20,
> >>>> +		.order = MALI_C55_BAYER_ORDER_BGGR,
> >>>> +		.encoding = V4L2_PIXEL_ENC_BAYER,
> >>>> +	},
> >>>> +	{
> >>>> +		.code = MEDIA_BUS_FMT_RGB202020_1X60,
> >>>> +		.order = 0, /* Not relevant for this format */
> >>>> +		.encoding = V4L2_PIXEL_ENC_RGB,
> >>>> +	}
> >>>> +	/*
> >>>> +	 * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
> >>>> +	 * also support YUV input from a sensor passed-through to the output. At
> >>>> +	 * present we have no mechanism to test that though so it may have to
> >>>> +	 * wait a while...
> >>>> +	 */
> >>>> +};
> >>>> +
> >>>> +const struct mali_c55_isp_fmt *
> >>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
> >>>> +{
> >>>> +	if (!fmt)
> >>>> +		fmt = &mali_c55_isp_fmts[0];
> >>>> +	else
> >>>> +		fmt++;
> >>>> +
> >>>> +	for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
> >>>> +		return fmt;
> >>> 
> >>> That's peculiar.
> >>>
> >>> 	if (!fmt)
> >>> 		fmt = &mali_c55_isp_fmts[0];
> >>> 	else if (fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)])
> >>> 		return ++fmt;
> >>> 	else
> >>> 		return NULL;
> >>>
> >>>> +
> >>>> +	return NULL;
> >>>> +}
> >>>> +
> >>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
> >>>> +{
> >>>> +	const struct mali_c55_isp_fmt *isp_fmt;
> >>>> +
> >>>> +	for_each_mali_isp_fmt(isp_fmt) {
> >>> 
> >>> I would open-code the loop instead of using the macro, like you do
> >>> below. It will be more efficient.
> >>>
> >>>> +		if (isp_fmt->code == mbus_code)
> >>>> +			return true;
> >>>> +	}
> >>>> +
> >>>> +	return false;
> >>>> +}
> >>>> +
> >>>> +static const struct mali_c55_isp_fmt *
> >>>> +mali_c55_isp_get_mbus_config_by_code(u32 code)
> >>>> +{
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
> >>>> +		if (mali_c55_isp_fmts[i].code == code)
> >>>> +			return &mali_c55_isp_fmts[i];
> >>>> +	}
> >>>> +
> >>>> +	return NULL;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	u32 val;
> >>>> +
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);
> >>> 
> >>> 	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> >>> 		       MALI_C55_INPUT_SAFE_STOP);
> >>>
> >>>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >>>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_isp_start(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>> +	const struct mali_c55_isp_fmt *cfg;
> >>>> +	struct v4l2_mbus_framefmt *format;
> >>> 
> >>> const
> >>>
> >>>> +	struct v4l2_subdev_state *state;
> >>>> +	struct v4l2_rect *crop;
> >>> 
> >>> const
> >>>
> >>>> +	struct v4l2_subdev *sd;
> >>>> +	u32 val;
> >>>> +	int ret;
> >>>> +
> >>>> +	sd = &mali_c55->isp.sd;
> >>> 
> >>> Assign when declaring the variable.
> >>>
> >>>> +
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>> +			     MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
> >>>> +
> >>>> +	/* Apply input windowing */
> >>>> +	state = v4l2_subdev_get_locked_active_state(sd);
> >>> 
> >>> Using .enable_streams() (see below) you'll get this for free.
> >>>
> >>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +	format = v4l2_subdev_state_get_format(state,
> >>>> +					      MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +	cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
> >>>> +
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
> >>>> +		       MALI_C55_HC_START(crop->left));
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
> >>>> +		       MALI_C55_HC_SIZE(crop->width));
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
> >>>> +		       MALI_C55_VC_START(crop->top) |
> >>>> +		       MALI_C55_VC_SIZE(crop->height));
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
> >>>> +			     MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
> >>>> +			     MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
> >>>> +			     MALI_C55_BAYER_ORDER_MASK, cfg->order);
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
> >>>> +			     MALI_C55_INPUT_WIDTH_MASK,
> >>>> +			     MALI_C55_INPUT_WIDTH_20BIT);
> >>>> +
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >>>> +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
> >>>> +			     cfg->encoding == V4L2_PIXEL_ENC_RGB ?
> >>>> +			     MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
> >>>> +
> >>>> +	ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
> >>>> +	if (ret) {
> >>>> +		dev_err(mali_c55->dev, "failed to DMA config\n");
> >>>> +		return ret;
> >>>> +	}
> >>>> +
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> >>>> +		       MALI_C55_INPUT_SAFE_START);
> >>>> +	readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >>>> +			   val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >>> 
> >>> Should you return an error in case of timeout ?
> >>>
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)
> >>> 
> >>> Why is this not handled wired to .s_stream() ? Or better,
> >>> .enable_streams() and .disable_streams().
> >> 
> >> There didn't really seem to be much point, since nothing outside this driver is ever going to start
> >> the subdev streaming...it's not like the sensor case where a separate driver might have to call some
> >> operation to start it.
> > 
> > True, but we're moving towards recording per-stream and per-pad
> > streaming state in the subdev through v4l2_subdev_enable_streams() and
> > v4l2_subdev_disable_streams(). We will likely expand usage of that
> > information, so it would be nice to keep it up-to-date already.
> 
> I'll move all the subdevs in the driver to .enable_streams() and .disable_streams()
> 
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>> +	struct v4l2_subdev *sd;
> >>>> +
> >>>> +	if (isp->remote_src) {
> >>>> +		sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
> >>>> +		v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
> >>>> +	}
> >>>> +	isp->remote_src = NULL;
> >>>> +
> >>>> +	mali_c55_isp_stop(mali_c55);
> >>>> +}
> >>>> +
> >>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>> +	struct media_pad *sink_pad;
> >>>> +	struct v4l2_subdev *sd;
> >>>> +	int ret;
> >>>> +
> >>>> +	sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
> >>>> +	isp->remote_src = media_pad_remote_pad_unique(sink_pad);
> >>>> +	if (IS_ERR(isp->remote_src)) {
> >>> 
> >>> If you set the MUST_CONNECT flag for the MALI_C55_ISP_PAD_SINK_VIDEO pad
> >>> I think you can drop this check.
> >>>
> >>>> +		dev_err(mali_c55->dev, "Failed to get source for ISP\n");
> >>>> +		return PTR_ERR(isp->remote_src);
> >>>> +	}
> >>>> +
> >>>> +	sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
> >>>> +
> >>>> +	isp->frame_sequence = 0;
> >>>> +	ret = mali_c55_isp_start(mali_c55);
> >>>> +	if (ret) {
> >>>> +		dev_err(mali_c55->dev, "Failed to start ISP\n");
> >>>> +		isp->remote_src = NULL;
> >>>> +		return ret;
> >>>> +	}
> >>>> +
> >>>> +	/*
> >>>> +	 * We only support a single input stream, so we can just enable the 1st
> >>>> +	 * entry in the streams mask.
> >>>> +	 */
> >>>> +	ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
> >>>> +	if (ret) {
> >>>> +		dev_err(mali_c55->dev, "Failed to start ISP source\n");
> >>>> +		mali_c55_isp_stop(mali_c55);
> >>>> +		return ret;
> >>>> +	}
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >>>> +				       struct v4l2_subdev_state *state,
> >>>> +				       struct v4l2_subdev_mbus_code_enum *code)
> >>>> +{
> >>>> +	/*
> >>>> +	 * Only the internal RGB processed format is allowed on the regular
> >>>> +	 * processing source pad.
> >>>> +	 */
> >>>> +	if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
> >>>> +		if (code->index)
> >>>> +			return -EINVAL;
> >>>> +
> >>>> +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>> +		return 0;
> >>>> +	}
> >>>> +
> >>>> +	/* On the sink and bypass pads all the supported formats are allowed. */
> >>>> +	if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	code->code = mali_c55_isp_fmts[code->index].code;
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
> >>>> +					struct v4l2_subdev_state *state,
> >>>> +					struct v4l2_subdev_frame_size_enum *fse)
> >>>> +{
> >>>> +	const struct mali_c55_isp_fmt *cfg;
> >>>> +
> >>>> +	if (fse->index > 0)
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	/*
> >>>> +	 * Only the internal RGB processed format is allowed on the regular
> >>>> +	 * processing source pad.
> >>>> +	 *
> >>>> +	 * On the sink and bypass pads all the supported formats are allowed.
> >>>> +	 */
> >>>> +	if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
> >>>> +		if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
> >>>> +			return -EINVAL;
> >>>> +	} else {
> >>>> +		cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
> >>>> +		if (!cfg)
> >>>> +			return -EINVAL;
> >>>> +	}
> >>>> +
> >>>> +	fse->min_width = MALI_C55_MIN_WIDTH;
> >>>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
> >>>> +	fse->max_width = MALI_C55_MAX_WIDTH;
> >>>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
> >>>> +				struct v4l2_subdev_state *state,
> >>>> +				struct v4l2_subdev_format *format)
> >>>> +{
> >>>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>> +	struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
> >>>> +	const struct mali_c55_isp_fmt *cfg;
> >>>> +	struct v4l2_rect *crop;
> >>>> +
> >>>> +	/*
> >>>> +	 * Disallow set_fmt on the source pads; format is fixed and the sizes
> >>>> +	 * are the result of applying the sink crop rectangle to the sink
> >>>> +	 * format.
> >>>> +	 */
> >>>> +	if (format->pad)
> >>> 
> >>> 	if (format->pad != MALI_C55_ISP_PAD_SINK_VIDEO)
> >>>
> >>>> +		return v4l2_subdev_get_fmt(sd, state, format);
> >>>> +
> >>>> +	cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
> >>>> +	if (!cfg)
> >>>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>> +	fmt->field = V4L2_FIELD_NONE;
> >>> 
> >>> Do you intentionally allow the colorspace fields to be overwritten to
> >>> any value ? rkisp1_isp_set_src_fmt() and rkisp1_isp_set_sink_fmt() may
> >>> show you how this could be handled.
> >>>
> >>>> +
> >>>> +	/*
> >>>> +	 * Clamp sizes in the accepted limits and clamp the crop rectangle in
> >>>> +	 * the new sizes.
> >>>> +	 */
> >>>> +	clamp_t(unsigned int, fmt->width,
> >>>> +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>>> +	clamp_t(unsigned int, fmt->width,
> >>>> +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>> 
> >>> clamp_t() returns a value, which you ignore. Those are no-ops. You meant
> >>>
> >>> 	fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> >>> 			     MALI_C55_MAX_WIDTH);
> >>> 	fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> >>> 			      MALI_C55_MAX_HEIGHT);
> >>>
> >>> Same for every use of clamp_t() through the whole driver.
> >>>
> >>> Also, do you need clamp_t() ? I think all values are unsigned int, you
> >>> can use clamp().
> >>>
> >>> Are there any alignment constraints, such a multiples of two for bayer
> >>> formats ? Same in all the other locations where applicable.
> >>>
> >>>> +
> >>>> +	sink_fmt = v4l2_subdev_state_get_format(state,
> >>>> +						MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +	*sink_fmt = *fmt;
> >>>> +
> >>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +	crop->left = 0;
> >>>> +	crop->top = 0;
> >>>> +	crop->width = fmt->width;
> >>>> +	crop->height = fmt->height;
> >>>> +
> >>>> +	/*
> >>>> +	 * Propagate format to source pads. On the 'regular' output pad use
> >>>> +	 * the internal RGB processed format, while on the bypass pad simply
> >>>> +	 * replicate the ISP sink format. The sizes on both pads are the same as
> >>>> +	 * the ISP sink crop rectangle.
> >>>> +	 */
> >>> 
> >>> Colorspace information will need to be propagated too.
> >>>
> >>>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
> >>>> +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>> +	src_fmt->width = crop->width;
> >>>> +	src_fmt->height = crop->height;
> >>>> +
> >>>> +	src_fmt = v4l2_subdev_state_get_format(state,
> >>>> +					       MALI_C55_ISP_PAD_SOURCE_BYPASS);
> >>>> +	src_fmt->code = fmt->code;
> >>>> +	src_fmt->width = crop->width;
> >>>> +	src_fmt->height = crop->height;
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
> >>>> +				      struct v4l2_subdev_state *state,
> >>>> +				      struct v4l2_subdev_selection *sel)
> >>>> +{
> >>>> +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> >>> 
> >>> 	sel->pad != MALI_C55_ISP_PAD_SINK_VIDEO
> >>>
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
> >>>> +				      struct v4l2_subdev_state *state,
> >>>> +				      struct v4l2_subdev_selection *sel)
> >>>> +{
> >>>> +	struct v4l2_mbus_framefmt *src_fmt;
> >>>> +	struct v4l2_mbus_framefmt *fmt;
> >>> 
> >>> const
> >>>
> >>>> +	struct v4l2_rect *crop;
> >>>> +
> >>>> +	if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> >>> 
> >>> Ditto.
> >>>
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +
> >>>> +	clamp_t(unsigned int, sel->r.left, 0, fmt->width);
> >>>> +	clamp_t(unsigned int, sel->r.top, 0, fmt->height);
> >>>> +	clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
> >>>> +		fmt->width - sel->r.left);
> >>>> +	clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
> >>>> +		fmt->height - sel->r.top);
> >>>> +
> >>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +	*crop = sel->r;
> >>>> +
> >>>> +	/* Propagate the crop rectangle sizes to the source pad format. */
> >>>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
> >>>> +	src_fmt->width = crop->width;
> >>>> +	src_fmt->height = crop->height;
> >>> 
> >>> Can you confirm that cropping doesn't affect the bypass path ?
> >> 
> >> Yes
> >>
> >>> And maybe
> >>> add a comment to mention it.
> >> 
> >> Will
> >>
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
> >>>> +	.enum_mbus_code		= mali_c55_isp_enum_mbus_code,
> >>>> +	.enum_frame_size	= mali_c55_isp_enum_frame_size,
> >>>> +	.get_fmt		= v4l2_subdev_get_fmt,
> >>>> +	.set_fmt		= mali_c55_isp_set_fmt,
> >>>> +	.get_selection		= mali_c55_isp_get_selection,
> >>>> +	.set_selection		= mali_c55_isp_set_selection,
> >>>> +	.link_validate		= v4l2_subdev_link_validate_default,
> >>>> +};
> >>>> +
> >>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	struct v4l2_event event = {
> >>>> +		.type = V4L2_EVENT_FRAME_SYNC,
> >>>> +	};
> >>>> +
> >>>> +	event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
> >>>> +	v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
> >>>> +}
> >>>> +
> >>>> +static int
> >>>> +mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
> >>>> +			     struct v4l2_event_subscription *sub)
> >>>> +{
> >>>> +	if (sub->type != V4L2_EVENT_FRAME_SYNC)
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	/* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
> >>>> +	if (sub->id != 0)
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	return v4l2_event_subscribe(fh, sub, 0, NULL);
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
> >>>> +	.subscribe_event = mali_c55_isp_subscribe_event,
> >>>> +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
> >>>> +};
> >>>> +
> >>>> +static const struct v4l2_subdev_ops mali_c55_isp_ops = {
> >>>> +	.pad	= &mali_c55_isp_pad_ops,
> >>>> +	.core	= &mali_c55_isp_core_ops,
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
> >>>> +				   struct v4l2_subdev_state *sd_state)
> >>> 
> >>> You name this variable state in every other subdev operation handler.
> >>>
> >>>> +{
> >>>> +	struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >>>> +	struct v4l2_rect *in_crop;
> >>>> +
> >>>> +	sink_fmt = v4l2_subdev_state_get_format(sd_state,
> >>>> +						MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +	src_fmt = v4l2_subdev_state_get_format(sd_state,
> >>>> +					       MALI_C55_ISP_PAD_SOURCE);
> >>>> +	in_crop = v4l2_subdev_state_get_crop(sd_state,
> >>>> +					     MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>> +
> >>>> +	sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>> +	sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>> +	sink_fmt->field = V4L2_FIELD_NONE;
> >>>> +	sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>> 
> >>> You should initialize the colorspace fields too. Same below.
> >>>
> >>>> +
> >>>> +	*v4l2_subdev_state_get_format(sd_state,
> >>>> +			      MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
> >>>> +
> >>>> +	src_fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>> +	src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>> +	src_fmt->field = V4L2_FIELD_NONE;
> >>>> +	src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>> +
> >>>> +	in_crop->top = 0;
> >>>> +	in_crop->left = 0;
> >>>> +	in_crop->width = MALI_C55_DEFAULT_WIDTH;
> >>>> +	in_crop->height = MALI_C55_DEFAULT_HEIGHT;
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
> >>>> +	.init_state = mali_c55_isp_init_state,
> >>>> +};
> >>>> +
> >>>> +static const struct media_entity_operations mali_c55_isp_media_ops = {
> >>>> +	.link_validate		= v4l2_subdev_link_validate,
> >>> 
> >>> 	.link_validate = v4l2_subdev_link_validate,
> >>>
> >>> to match mali_c55_isp_internal_ops.
> >>>
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
> >>>> +				       struct v4l2_subdev *subdev,
> >>>> +				       struct v4l2_async_connection *asc)
> >>>> +{
> >>>> +	struct mali_c55_isp *isp = container_of(notifier,
> >>>> +						struct mali_c55_isp, notifier);
> >>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>> +	struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
> >>>> +	int ret;
> >>>> +
> >>>> +	/*
> >>>> +	 * By default we'll flag this link enabled and the TPG disabled, but
> >>>> +	 * no immutable flag because we need to be able to switch between the
> >>>> +	 * two.
> >>>> +	 */
> >>>> +	ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
> >>>> +					      MEDIA_LNK_FL_ENABLED);
> >>>> +	if (ret)
> >>>> +		dev_err(mali_c55->dev, "failed to create link for %s\n",
> >>>> +			subdev->name);
> >>>> +
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
> >>>> +{
> >>>> +	struct mali_c55_isp *isp = container_of(notifier,
> >>>> +						struct mali_c55_isp, notifier);
> >>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>> +
> >>>> +	return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
> >>>> +	.bound = mali_c55_isp_notifier_bound,
> >>>> +	.complete = mali_c55_isp_notifier_complete,
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>> +	struct v4l2_async_connection *asc;
> >>>> +	struct fwnode_handle *ep;
> >>>> +	int ret;
> >>>> +
> >>>> +	v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
> >>>> +
> >>>> +	/*
> >>>> +	 * The ISP should have a single endpoint pointing to some flavour of
> >>>> +	 * CSI-2 receiver...but for now at least we do want everything to work
> >>>> +	 * normally even with no sensors connected, as we have the TPG. If we
> >>>> +	 * don't find a sensor just warn and return success.
> >>>> +	 */
> >>>> +	ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
> >>>> +					     0, 0, 0);
> >>>> +	if (!ep) {
> >>>> +		dev_warn(mali_c55->dev, "no local endpoint found\n");
> >>>> +		return 0;
> >>>> +	}
> >>>> +
> >>>> +	asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
> >>>> +					      struct v4l2_async_connection);
> >>>> +	if (IS_ERR(asc)) {
> >>>> +		dev_err(mali_c55->dev, "failed to add remote fwnode\n");
> >>>> +		ret = PTR_ERR(asc);
> >>>> +		goto err_put_ep;
> >>>> +	}
> >>>> +
> >>>> +	isp->notifier.ops = &mali_c55_isp_notifier_ops;
> >>>> +	ret = v4l2_async_nf_register(&isp->notifier);
> >>>> +	if (ret) {
> >>>> +		dev_err(mali_c55->dev, "failed to register notifier\n");
> >>>> +		goto err_cleanup_nf;
> >>>> +	}
> >>>> +
> >>>> +	fwnode_handle_put(ep);
> >>>> +
> >>>> +	return 0;
> >>>> +
> >>>> +err_cleanup_nf:
> >>>> +	v4l2_async_nf_cleanup(&isp->notifier);
> >>>> +err_put_ep:
> >>>> +	fwnode_handle_put(ep);
> >>>> +
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	struct mali_c55_isp *isp = &mali_c55->isp;
> >>>> +	struct v4l2_subdev *sd = &isp->sd;
> >>>> +	int ret;
> >>>> +
> >>>> +	isp->mali_c55 = mali_c55;
> >>>> +
> >>>> +	v4l2_subdev_init(sd, &mali_c55_isp_ops);
> >>>> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> >>>> +	sd->entity.ops = &mali_c55_isp_media_ops;
> >>>> +	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
> >>>> +	sd->internal_ops = &mali_c55_isp_internal_ops;
> >>>> +	strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
> >>>> +
> >>>> +	isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
> >>> 
> >>> The MUST_CONNECT flag would make sense here.
> >>>
> >>>> +	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> >>>> +	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
> >>>> +
> >>>> +	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
> >>>> +				     isp->pads);
> >>>> +	if (ret)
> >>>> +		return ret;
> >>>> +
> >>>> +	ret = v4l2_subdev_init_finalize(sd);
> >>>> +	if (ret)
> >>>> +		goto err_cleanup_media_entity;
> >>>> +
> >>>> +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >>>> +	if (ret)
> >>>> +		goto err_cleanup_subdev;
> >>>> +
> >>>> +	ret = mali_c55_isp_parse_endpoint(isp);
> >>>> +	if (ret)
> >>>> +		goto err_cleanup_subdev;
> >>> 
> >>> As noted elsewhere, I think this belongs to mali-c55-core.c.
> >>>
> >>>> +
> >>>> +	mutex_init(&isp->lock);
> >>> 
> >>> This lock is used in mali-c55-capture.c only, that seems weird.
> >> 
> >> It's because we have two separate capture devices, who's start/stop streaming path calls into the
> >> ISP subdevice's start streaming function, but has to do it after a bunch of other writes to the cap
> >> device or resizer specific sections...the intention was to delay doing any of that until the ISP was
> >> in a known state.
> > 
> > OK. Naming it capture_lock or something similar, with a comment, would
> > help.
> 
> Ack
> 
> >>>> +
> >>>> +	return 0;
> >>>> +
> >>>> +err_cleanup_subdev:
> >>>> +	v4l2_subdev_cleanup(sd);
> >>>> +err_cleanup_media_entity:
> >>>> +	media_entity_cleanup(&sd->entity);
> >>>> +	isp->mali_c55 = NULL;
> >>>> +
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	struct mali_c55_isp *isp = &mali_c55->isp;
> >>>> +
> >>>> +	if (!isp->mali_c55)
> >>>> +		return;
> >>>> +
> >>>> +	mutex_destroy(&isp->lock);
> >>>> +	v4l2_async_nf_unregister(&isp->notifier);
> >>>> +	v4l2_async_nf_cleanup(&isp->notifier);
> >>>> +	v4l2_device_unregister_subdev(&isp->sd);
> >>>> +	v4l2_subdev_cleanup(&isp->sd);
> >>>> +	media_entity_cleanup(&isp->sd.entity);
> >>>> +}
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>> new file mode 100644
> >>>> index 000000000000..cb27abde2aa5
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>> @@ -0,0 +1,258 @@
> >>>> +/* SPDX-License-Identifier: GPL-2.0 */
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Register definitions
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#ifndef _MALI_C55_REGISTERS_H
> >>>> +#define _MALI_C55_REGISTERS_H
> >>>> +
> >>>> +#include <linux/bits.h>
> >>>> +
> >>>> +/* ISP Common 0x00000 - 0x000ff */
> >>>> +
> >>>> +#define MALI_C55_REG_API				0x00000
> >>>> +#define MALI_C55_REG_PRODUCT				0x00004
> >>>> +#define MALI_C55_REG_VERSION				0x00008
> >>>> +#define MALI_C55_REG_REVISION				0x0000c
> >>>> +#define MALI_C55_REG_PULSE_MODE				0x0003c
> >>>> +#define MALI_C55_REG_INPUT_MODE_REQUEST			0x0009c
> >>>> +#define MALI_C55_INPUT_SAFE_STOP			0x00
> >>>> +#define MALI_C55_INPUT_SAFE_START			0x01
> >>>> +#define MALI_C55_REG_MODE_STATUS			0x000a0
> >>>> +#define MALI_C55_REG_INTERRUPT_MASK_VECTOR		0x00030
> >>>> +#define MALI_C55_INTERRUPT_MASK_ALL			GENMASK(31, 0)
> >>>> +
> >>>> +#define MALI_C55_REG_GLOBAL_MONITOR			0x00050
> >>>> +
> >>>> +#define MALI_C55_REG_GEN_VIDEO				0x00080
> >>>> +#define MALI_C55_REG_GEN_VIDEO_ON_MASK			BIT(0)
> >>>> +#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK		BIT(1)
> >>>> +#define MALI_C55_REG_GEN_PREFETCH_MASK			GENMASK(31, 16)
> >>>> +
> >>>> +#define MALI_C55_REG_MCU_CONFIG				0x00020
> >>>> +#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK		BIT(0)
> >>> 
> >>> #define MALI_C55_REG_MCU_CONFIG_OVERRIDE		BIT(0)
> >>>
> >>> Same in other places where applicable.
> >>>
> >>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK		BIT(1)
> >>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PING		BIT(1)
> >>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG		0x00
> >>>> +#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK		BIT(8)
> >>>> +#define MALI_C55_REG_PING_PONG_READ			0x00024
> >>>> +#define MALI_C55_REG_PING_PONG_READ_MASK		BIT(2)
> >>>> +
> >>>> +#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR		0x00034
> >>>> +#define MALI_C55_REG_INTERRUPT_CLEAR			0x00040
> >>>> +#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR		0x00044
> >>>> +#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS		0x00068
> >>>> +#define MALI_C55_GPS_PONG_FITTED			BIT(0)
> >>>> +#define MALI_C55_GPS_WDR_FITTED				BIT(1)
> >>>> +#define MALI_C55_GPS_COMPRESSION_FITTED			BIT(2)
> >>>> +#define MALI_C55_GPS_TEMPER_FITTED			BIT(3)
> >>>> +#define MALI_C55_GPS_SINTER_LITE_FITTED			BIT(4)
> >>>> +#define MALI_C55_GPS_SINTER_FITTED			BIT(5)
> >>>> +#define MALI_C55_GPS_IRIDIX_LTM_FITTED			BIT(6)
> >>>> +#define MALI_C55_GPS_IRIDIX_GTM_FITTED			BIT(7)
> >>>> +#define MALI_C55_GPS_CNR_FITTED				BIT(8)
> >>>> +#define MALI_C55_GPS_FRSCALER_FITTED			BIT(9)
> >>>> +#define MALI_C55_GPS_DS_PIPE_FITTED			BIT(10)
> >>>> +
> >>>> +#define MALI_C55_REG_BLANKING				0x00084
> >>>> +#define MALI_C55_REG_HBLANK_MASK			GENMASK(15, 0)
> >>>> +#define MALI_C55_REG_VBLANK_MASK			GENMASK(31, 16)
> >>>> +
> >>>> +#define MALI_C55_REG_HC_START				0x00088
> >>>> +#define MALI_C55_HC_START(h)				(((h) & 0xffff) << 16)
> >>>> +#define MALI_C55_REG_HC_SIZE				0x0008c
> >>>> +#define MALI_C55_HC_SIZE(h)				((h) & 0xffff)
> >>>> +#define MALI_C55_REG_VC_START_SIZE			0x00094
> >>>> +#define MALI_C55_VC_START(v)				((v) & 0xffff)
> >>>> +#define MALI_C55_VC_SIZE(v)				(((v) & 0xffff) << 16)
> >>>> +
> >>>> +/* Ping/Pong Configuration Space */
> >>>> +#define MALI_C55_REG_BASE_ADDR				0x18e88
> >>>> +#define MALI_C55_REG_BYPASS_0				0x18eac
> >>>> +#define MALI_C55_REG_BYPASS_0_VIDEO_TEST		BIT(0)
> >>>> +#define MALI_C55_REG_BYPASS_0_INPUT_FMT			BIT(1)
> >>>> +#define MALI_C55_REG_BYPASS_0_DECOMPANDER		BIT(2)
> >>>> +#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR		BIT(3)
> >>>> +#define MALI_C55_REG_BYPASS_0_GAIN_WDR			BIT(4)
> >>>> +#define MALI_C55_REG_BYPASS_0_FRAME_STITCH		BIT(5)
> >>>> +#define MALI_C55_REG_BYPASS_1				0x18eb0
> >>>> +#define MALI_C55_REG_BYPASS_1_DIGI_GAIN			BIT(0)
> >>>> +#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS		BIT(1)
> >>>> +#define MALI_C55_REG_BYPASS_1_FE_SQRT			BIT(2)
> >>>> +#define MALI_C55_REG_BYPASS_1_RAW_FE			BIT(3)
> >>>> +#define MALI_C55_REG_BYPASS_2				0x18eb8
> >>>> +#define MALI_C55_REG_BYPASS_2_SINTER			BIT(0)
> >>>> +#define MALI_C55_REG_BYPASS_2_TEMPER			BIT(1)
> >>>> +#define MALI_C55_REG_BYPASS_3				0x18ebc
> >>>> +#define MALI_C55_REG_BYPASS_3_SQUARE_BE			BIT(0)
> >>>> +#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH	BIT(1)
> >>>> +#define MALI_C55_REG_BYPASS_3_MESH_SHADING		BIT(3)
> >>>> +#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE		BIT(4)
> >>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX			BIT(5)
> >>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN		BIT(6)
> >>>> +#define MALI_C55_REG_BYPASS_4				0x18ec0
> >>>> +#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB		BIT(1)
> >>>> +#define MALI_C55_REG_BYPASS_4_PF_CORRECTION		BIT(3)
> >>>> +#define MALI_C55_REG_BYPASS_4_CCM			BIT(4)
> >>>> +#define MALI_C55_REG_BYPASS_4_CNR			BIT(5)
> >>>> +#define MALI_C55_REG_FR_BYPASS				0x18ec4
> >>>> +#define MALI_C55_REG_DS_BYPASS				0x18ec8
> >>>> +#define MALI_C55_BYPASS_CROP				BIT(0)
> >>>> +#define MALI_C55_BYPASS_SCALER				BIT(1)
> >>>> +#define MALI_C55_BYPASS_GAMMA_RGB			BIT(2)
> >>>> +#define MALI_C55_BYPASS_SHARPEN				BIT(3)
> >>>> +#define MALI_C55_BYPASS_CS_CONV				BIT(4)
> >>>> +#define MALI_C55_REG_ISP_RAW_BYPASS			0x18ecc
> >>>> +#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK		BIT(0)
> >>>> +#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK		GENMASK(9, 8)
> >>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS		2
> >>>> +#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS		1
> >>>> +#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE		BIT(1)
> >>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS		BIT(0)
> >>>> +
> >>>> +#define MALI_C55_REG_ACTIVE_WIDTH_MASK			0xffff
> >>>> +#define MALI_C55_REG_ACTIVE_HEIGHT_MASK			0xffff0000
> >>>> +#define MALI_C55_REG_BAYER_ORDER			0x18e8c
> >>>> +#define MALI_C55_BAYER_ORDER_MASK			GENMASK(1, 0)
> >>>> +#define MALI_C55_REG_TPG_CH0				0x18ed8
> >>>> +#define MALI_C55_TEST_PATTERN_ON_OFF			BIT(0)
> >>>> +#define MALI_C55_TEST_PATTERN_RGB_MASK			BIT(1)
> >>>> +#define MALI_C55_REG_TPG_R_BACKGROUND			0x18ee0
> >>>> +#define MALI_C55_REG_TPG_G_BACKGROUND			0x18ee4
> >>>> +#define MALI_C55_REG_TPG_B_BACKGROUND			0x18ee8
> >>>> +#define MALI_C55_TPG_BACKGROUND_MAX			0xfffff
> >>>> +#define MALI_C55_REG_INPUT_WIDTH			0x18f98
> >>>> +#define MALI_C55_INPUT_WIDTH_MASK			GENMASK(18, 16)
> >>>> +#define MALI_C55_INPUT_WIDTH_8BIT			0
> >>>> +#define MALI_C55_INPUT_WIDTH_10BIT			1
> >>>> +#define MALI_C55_INPUT_WIDTH_12BIT			2
> >>>> +#define MALI_C55_INPUT_WIDTH_14BIT			3
> >>>> +#define MALI_C55_INPUT_WIDTH_16BIT			4
> >>>> +#define MALI_C55_INPUT_WIDTH_20BIT			5
> >>>> +#define MALI_C55_REG_SPACE_SIZE				0x4000
> >>>> +#define MALI_C55_REG_CONFIG_SPACES_OFFSET		0x0ab6c
> >>>> +#define MALI_C55_CONFIG_SPACE_SIZE			0x1231c
> >>>> +
> >>>> +#define MALI_C55_REG_SINTER_CONFIG			0x19348
> >>>> +#define MALI_C55_SINTER_VIEW_FILTER_MASK		GENMASK(1, 0)
> >>>> +#define MALI_C55_SINTER_SCALE_MODE_MASK			GENMASK(3, 2)
> >>>> +#define MALI_C55_SINTER_ENABLE_MASK			BIT(4)
> >>>> +#define MALI_C55_SINTER_FILTER_SELECT_MASK		BIT(5)
> >>>> +#define MALI_C55_SINTER_INT_SELECT_MASK			BIT(6)
> >>>> +#define MALI_C55_SINTER_RM_ENABLE_MASK			BIT(7)
> >>>> +
> >>>> +/* Colour Correction Matrix Configuration */
> >>>> +#define MALI_C55_REG_CCM_ENABLE				0x1b07c
> >>>> +#define MALI_C55_CCM_ENABLE_MASK			BIT(0)
> >>>> +#define MALI_C55_REG_CCM_COEF_R_R			0x1b080
> >>>> +#define MALI_C55_REG_CCM_COEF_R_G			0x1b084
> >>>> +#define MALI_C55_REG_CCM_COEF_R_B			0x1b088
> >>>> +#define MALI_C55_REG_CCM_COEF_G_R			0x1b090
> >>>> +#define MALI_C55_REG_CCM_COEF_G_G			0x1b094
> >>>> +#define MALI_C55_REG_CCM_COEF_G_B			0x1b098
> >>>> +#define MALI_C55_REG_CCM_COEF_B_R			0x1b0a0
> >>>> +#define MALI_C55_REG_CCM_COEF_B_G			0x1b0a4
> >>>> +#define MALI_C55_REG_CCM_COEF_B_B			0x1b0a8
> >>>> +#define MALI_C55_CCM_COEF_MASK				GENMASK(12, 0)
> >>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R			0x1b0b0
> >>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G			0x1b0b4
> >>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B			0x1b0b8
> >>>> +#define MALI_C55_CCM_ANTIFOG_GAIN_MASK			GENMASK(11, 0)
> >>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R		0x1b0c0
> >>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G		0x1b0c4
> >>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B		0x1b0c8
> >>>> +#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK		GENMASK(11, 0)
> >>>> +
> >>>> +/*
> >>>> + * The Mali-C55 ISP has up to two output pipes; known as full resolution and
> >>>> + * down scaled. The register space for these is laid out identically, but offset
> >>>> + * by 372 bytes.
> >>>> + */
> >>>> +#define MALI_C55_CAP_DEV_FR_REG_OFFSET		0x0
> >>>> +#define MALI_C55_CAP_DEV_DS_REG_OFFSET		0x174
> >>>> +
> >>>> +#define MALI_C55_REG_CS_CONV_CONFIG(offset)		(0x1c098 + (offset))
> >>>> +#define MALI_C55_CS_CONV_MATRIX_MASK			BIT(0)
> >>>> +#define MALI_C55_CS_CONV_FILTER_MASK			BIT(1)
> >>>> +#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK		BIT(2)
> >>>> +#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK		BIT(3)
> >>>> +#define MALI_C55_REG_Y_WRITER_MODE(offset)		(0x1c0ec + (offset))
> >>>> +#define MALI_C55_REG_UV_WRITER_MODE(offset)		(0x1c144 + (offset))
> >>>> +#define MALI_C55_WRITER_MODE_MASK			GENMASK(4, 0)
> >>>> +#define MALI_C55_WRITER_SUBMODE_MASK			GENMASK(7, 6)
> >>>> +#define MALI_C55_WRITER_FRAME_WRITE_MASK		BIT(9)
> >>>> +#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset)		(0x1c0f0 + (offset))
> >>>> +#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset)		(0x1c148 + (offset))
> >>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)		((w) << 0)
> >>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)		((h) << 16)
> >>>> +#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset)	(0x1c0f4 + (offset))
> >>>> +#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset)	(0x1c108 + (offset))
> >>>> +#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
> >>>> +#define MALI_C55_REG_Y_WRITER_BANKS_RESTART		BIT(3)
> >>>> +#define MALI_C55_REG_Y_WRITER_OFFSET(offset)		(0x1c10c + (offset))
> >>>> +#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset)	(0x1c14c + (offset))
> >>>> +#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset)	(0x1c160 + (offset))
> >>>> +#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK		GENMASK(2, 0)
> >>>> +#define MALI_C55_REG_UV_WRITER_BANKS_RESTART		BIT(3)
> >>>> +#define MALI_C55_REG_UV_WRITER_OFFSET(offset)		(0x1c164 + (offset))
> >>>> +
> >>>> +#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
> >>>> +#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE		0x18edc
> >>>> +
> >>>> +#define MALI_C55_REG_CROP_EN(offset)			(0x1c028 + (offset))
> >>>> +#define MALI_C55_CROP_ENABLE				BIT(0)
> >>>> +#define MALI_C55_REG_CROP_X_START(offset)		(0x1c02c + (offset))
> >>>> +#define MALI_C55_REG_CROP_Y_START(offset)		(0x1c030 + (offset))
> >>>> +#define MALI_C55_REG_CROP_X_SIZE(offset)		(0x1c034 + (offset))
> >>>> +#define MALI_C55_REG_CROP_Y_SIZE(offset)		(0x1c038 + (offset))
> >>>> +#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset)		(0x1c040 + (offset))
> >>>> +#define MALI_C55_SCALER_TIMEOUT_EN			BIT(4)
> >>>> +#define MALI_C55_SCALER_TIMEOUT(t)			((t) << 16)
> >>>> +#define MALI_C55_REG_SCALER_IN_WIDTH(offset)		(0x1c044 + (offset))
> >>>> +#define MALI_C55_REG_SCALER_IN_HEIGHT(offset)		(0x1c048 + (offset))
> >>>> +#define MALI_C55_REG_SCALER_OUT_WIDTH(offset)		(0x1c04c + (offset))
> >>>> +#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset)		(0x1c050 + (offset))
> >>>> +#define MALI_C55_REG_SCALER_HFILT_TINC(offset)		(0x1c054 + (offset))
> >>>> +#define MALI_C55_REG_SCALER_HFILT_COEF(offset)		(0x1c058 + (offset))
> >>>> +#define MALI_C55_REG_SCALER_VFILT_TINC(offset)		(0x1c05c + (offset))
> >>>> +#define MALI_C55_REG_SCALER_VFILT_COEF(offset)		(0x1c060 + (offset))
> >>>> +
> >>>> +#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset)		(0x1c064 + (offset))
> >>>> +#define MALI_C55_GAMMA_ENABLE_MASK			BIT(0)
> >>>> +#define MALI_C55_REG_GAMMA_GAINS_1(offset)		(0x1c068 + (offset))
> >>>> +#define MALI_C55_GAMMA_GAIN_R_MASK			GENMASK(11, 0)
> >>>> +#define MALI_C55_GAMMA_GAIN_G_MASK			GENMASK(27, 16)
> >>>> +#define MALI_C55_REG_GAMMA_GAINS_2(offset)		(0x1c06c + (offset))
> >>>> +#define MALI_C55_GAMMA_GAIN_B_MASK			GENMASK(11, 0)
> >>>> +#define MALI_C55_REG_GAMMA_OFFSETS_1(offset)		(0x1c070 + (offset))
> >>>> +#define MALI_C55_GAMMA_OFFSET_R_MASK			GENMASK(11, 0)
> >>>> +#define MALI_C55_GAMMA_OFFSET_G_MASK			GENMASK(27, 16)
> >>>> +#define MALI_C55_REG_GAMMA_OFFSETS_2(offset)		(0x1c074 + (offset))
> >>>> +#define MALI_C55_GAMMA_OFFSET_B_MASK			GENMASK(11, 0)
> >>>> +
> >>>> +/* Output DMA Writer */
> >>>> +
> >>>> +#define MALI_C55_OUTPUT_DISABLED		0
> >>>> +#define MALI_C55_OUTPUT_RGB32			1
> >>>> +#define MALI_C55_OUTPUT_A2R10G10B10		2
> >>>> +#define MALI_C55_OUTPUT_RGB565			3
> >>>> +#define MALI_C55_OUTPUT_RGB24			4
> >>>> +#define MALI_C55_OUTPUT_GEN32			5
> >>>> +#define MALI_C55_OUTPUT_RAW16			6
> >>>> +#define MALI_C55_OUTPUT_AYUV			8
> >>>> +#define MALI_C55_OUTPUT_Y410			9
> >>>> +#define MALI_C55_OUTPUT_YUY2			10
> >>>> +#define MALI_C55_OUTPUT_UYVY			11
> >>>> +#define MALI_C55_OUTPUT_Y210			12
> >>>> +#define MALI_C55_OUTPUT_NV12_21			13
> >>>> +#define MALI_C55_OUTPUT_YUV_420_422		17
> >>>> +#define MALI_C55_OUTPUT_P210_P010		19
> >>>> +#define MALI_C55_OUTPUT_YUV422			20
> >>> 
> >>> I'd define this just below the MALI_C55_REG_UV_WRITER_MODE register
> >>> macro.
> >>>
> >>>> +
> >>>> +#define MALI_C55_OUTPUT_PLANE_ALT0		0
> >>>> +#define MALI_C55_OUTPUT_PLANE_ALT1		1
> >>>> +#define MALI_C55_OUTPUT_PLANE_ALT2		2
> >>> 
> >>> Same here ?
> >>>
> >>>> +
> >>>> +#endif /* _MALI_C55_REGISTERS_H */
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>>> new file mode 100644
> >>>> index 000000000000..8edae87f1e5f
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>>> @@ -0,0 +1,382 @@
> >>>> +/* SPDX-License-Identifier: GPL-2.0 */
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Resizer Coefficients
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#ifndef _MALI_C55_RESIZER_COEFS_H
> >>>> +#define _MALI_C55_RESIZER_COEFS_H
> >>>> +
> >>>> +#include "mali-c55-common.h"
> >>>> +
> >>>> +#define MALI_C55_RESIZER_COEFS_NUM_BANKS	8
> >>>> +#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES	64
> >>> 
> >>> Do these belongs to mali-c55-registers.h ?
> >>>
> >>>> +
> >>>> +static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
> >>>> +	{	/* Bank 0 */
> >>>> +		0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
> >>>> +		0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
> >>>> +		0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
> >>>> +		0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
> >>>> +		0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
> >>>> +		0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
> >>>> +		0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
> >>>> +		0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
> >>>> +		0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
> >>>> +		0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
> >>>> +		0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
> >>>> +		0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
> >>>> +		0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
> >>>> +		0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
> >>>> +		0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
> >>>> +		0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
> >>>> +	},
> >>>> +	{	/* Bank 1 */
> >>>> +		0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
> >>>> +		0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
> >>>> +		0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
> >>>> +		0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
> >>>> +		0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
> >>>> +		0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
> >>>> +		0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
> >>>> +		0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
> >>>> +		0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
> >>>> +		0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
> >>>> +		0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
> >>>> +		0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
> >>>> +		0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
> >>>> +		0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
> >>>> +		0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
> >>>> +		0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
> >>>> +	},
> >>>> +	{	/* Bank 2 */
> >>>> +		0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
> >>>> +		0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
> >>>> +		0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
> >>>> +		0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
> >>>> +		0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
> >>>> +		0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
> >>>> +		0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
> >>>> +		0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
> >>>> +		0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
> >>>> +		0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
> >>>> +		0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
> >>>> +		0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
> >>>> +		0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
> >>>> +		0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
> >>>> +		0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
> >>>> +		0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
> >>>> +	},
> >>>> +	{	/* Bank 3 */
> >>>> +		0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
> >>>> +		0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
> >>>> +		0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
> >>>> +		0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
> >>>> +		0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
> >>>> +		0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
> >>>> +		0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
> >>>> +		0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
> >>>> +		0x20100000, 0x00000010, 0x1f110000, 0x00000010,
> >>>> +		0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
> >>>> +		0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
> >>>> +		0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
> >>>> +		0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
> >>>> +		0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
> >>>> +		0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
> >>>> +		0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
> >>>> +	},
> >>>> +	{	/* Bank 4 */
> >>>> +		0x17090000, 0x00000917, 0x18090000, 0x00000916,
> >>>> +		0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
> >>>> +		0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
> >>>> +		0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
> >>>> +		0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
> >>>> +		0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
> >>>> +		0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
> >>>> +		0x190f0300, 0x00000411, 0x18100300, 0x00000411,
> >>>> +		0x1a100300, 0x00000310, 0x18110400, 0x00000310,
> >>>> +		0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
> >>>> +		0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
> >>>> +		0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
> >>>> +		0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
> >>>> +		0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
> >>>> +		0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
> >>>> +		0x17160800, 0x0000010a, 0x18160900, 0x00000009,
> >>>> +	},
> >>>> +	{	/* Bank 5 */
> >>>> +		0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
> >>>> +		0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
> >>>> +		0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
> >>>> +		0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
> >>>> +		0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
> >>>> +		0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
> >>>> +		0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
> >>>> +		0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
> >>>> +		0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
> >>>> +		0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
> >>>> +		0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
> >>>> +		0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
> >>>> +		0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
> >>>> +		0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
> >>>> +		0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
> >>>> +		0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
> >>>> +	},
> >>>> +	{	/* Bank 6 */
> >>>> +		0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
> >>>> +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
> >>>> +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
> >>>> +		0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
> >>>> +		0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
> >>>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>> +		0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
> >>>> +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >>>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>> +		0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
> >>>> +		0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
> >>>> +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
> >>>> +		0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >>>> +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >>>> +	},
> >>>> +	{	/* Bank 7 */
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +		0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>> +	}
> >>>> +};
> >>>> +
> >>>> +static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
> >>>> +	{	/* Bank 0 */
> >>>> +		0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
> >>>> +		0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
> >>>> +		0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
> >>>> +		0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
> >>>> +		0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
> >>>> +		0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
> >>>> +		0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
> >>>> +		0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
> >>>> +		0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
> >>>> +		0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
> >>>> +		0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
> >>>> +		0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
> >>>> +		0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
> >>>> +		0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
> >>>> +		0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
> >>>> +		0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
> >>>> +	},
> >>>> +	{	/* Bank 1 */
> >>>> +		0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
> >>>> +		0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
> >>>> +		0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
> >>>> +		0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
> >>>> +		0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
> >>>> +		0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
> >>>> +		0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
> >>>> +		0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
> >>>> +		0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
> >>>> +		0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
> >>>> +		0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
> >>>> +		0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
> >>>> +		0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
> >>>> +		0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
> >>>> +		0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
> >>>> +		0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
> >>>> +	},
> >>>> +	{	/* Bank 2 */
> >>>> +		0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
> >>>> +		0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
> >>>> +		0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
> >>>> +		0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
> >>>> +		0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
> >>>> +		0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
> >>>> +		0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
> >>>> +		0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
> >>>> +		0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
> >>>> +		0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
> >>>> +		0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
> >>>> +		0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
> >>>> +		0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
> >>>> +		0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
> >>>> +		0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
> >>>> +		0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
> >>>> +	},
> >>>> +	{	/* Bank 3 */
> >>>> +		0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
> >>>> +		0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
> >>>> +		0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
> >>>> +		0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
> >>>> +		0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
> >>>> +		0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
> >>>> +		0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
> >>>> +		0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
> >>>> +		0x20100000, 0x00000010, 0x1f100000, 0x00000011,
> >>>> +		0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
> >>>> +		0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
> >>>> +		0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
> >>>> +		0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
> >>>> +		0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
> >>>> +		0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
> >>>> +		0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
> >>>> +	},
> >>>> +	{	/* Bank 4 */
> >>>> +		0x17170900, 0x00000009, 0x18160900, 0x00000009,
> >>>> +		0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
> >>>> +		0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
> >>>> +		0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
> >>>> +		0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
> >>>> +		0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
> >>>> +		0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
> >>>> +		0x19110400, 0x0000030f, 0x18110400, 0x00000310,
> >>>> +		0x1a100300, 0x00000310, 0x18100300, 0x00000411,
> >>>> +		0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
> >>>> +		0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
> >>>> +		0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
> >>>> +		0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
> >>>> +		0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
> >>>> +		0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
> >>>> +		0x170a0100, 0x00000816, 0x18090000, 0x00000916,
> >>>> +	},
> >>>> +	{	/* Bank 5 */
> >>>> +		0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
> >>>> +		0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
> >>>> +		0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
> >>>> +		0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
> >>>> +		0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
> >>>> +		0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
> >>>> +		0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
> >>>> +		0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
> >>>> +		0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
> >>>> +		0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
> >>>> +		0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
> >>>> +		0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
> >>>> +		0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
> >>>> +		0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
> >>>> +		0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
> >>>> +		0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
> >>>> +	},
> >>>> +	{	/* Bank 6 */
> >>>> +		0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
> >>>> +		0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >>>> +		0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
> >>>> +		0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
> >>>> +		0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
> >>>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>> +		0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>> +		0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >>>> +		0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >>>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>> +		0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>> +		0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
> >>>> +		0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
> >>>> +		0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
> >>>> +		0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
> >>>> +		0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
> >>>> +	},
> >>>> +	{	/* Bank 7 */
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +		0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>> +	}
> >>>> +};
> >>>> +
> >>>> +struct mali_c55_resizer_coef_bank {
> >>>> +	unsigned int bank;
> >>> 
> >>> This is always equal to the index of the entry in the
> >>> mali_c55_coefficient_banks array, you can drop it.
> >>>
> >>>> +	unsigned int top;
> >>>> +	unsigned int bottom;
> >>> 
> >>> The bottom value of bank N is always equal to the top value of bank N+1.
> >>> You can simplify this by storing a single value.
> >>>
> >>>> +};
> >>>> +
> >>>> +static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
> >>>> +	{
> >>>> +		.bank = 0,
> >>>> +		.top = 1000,
> >>>> +		.bottom = 770,
> >>>> +	},
> >>>> +	{
> >>>> +		.bank = 1,
> >>>> +		.top = 769,
> >>>> +		.bottom = 600,
> >>>> +	},
> >>>> +	{
> >>>> +		.bank = 2,
> >>>> +		.top = 599,
> >>>> +		.bottom = 460,
> >>>> +	},
> >>>> +	{
> >>>> +		.bank = 3,
> >>>> +		.top = 459,
> >>>> +		.bottom = 354,
> >>>> +	},
> >>>> +	{
> >>>> +		.bank = 4,
> >>>> +		.top = 353,
> >>>> +		.bottom = 273,
> >>>> +	},
> >>>> +	{
> >>>> +		.bank = 5,
> >>>> +		.top = 272,
> >>>> +		.bottom = 210,
> >>>> +	},
> >>>> +	{
> >>>> +		.bank = 6,
> >>>> +		.top = 209,
> >>>> +		.bottom = 162,
> >>>> +	},
> >>>> +	{
> >>>> +		.bank = 7,
> >>>> +		.top = 161,
> >>>> +		.bottom = 125,
> >>>> +	},
> >>>> +};
> >>>> +
> >>> 
> >>> A small comment would be nice, such as
> >>>
> >>> /* Select a bank of resizer coefficients, based on the scaling ratio. */
> >>>
> >>>> +static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,
> >>> 
> >>> This function is related to the resizers. Add "rsz" somewhere in the
> >>> function name, and pass a resizer pointer.
> >>>
> >>>> +						unsigned int crop,
> >>>> +						unsigned int scale)
> >>> 
> >>> I think those are the input and output sizes to the scaler. Rename them
> >>> to make it clearer.
> >>>
> >>>> +{
> >>>> +	unsigned int tmp;
> >>> 
> >>> tmp is almost always a bad variable name. Please use a more descriptive
> >>> name, size as rsz_ratio.
> >>>
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	tmp = (scale * 1000U) / crop;
> >>>> +
> >>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
> >>>> +		if (tmp >= mali_c55_coefficient_banks[i].bottom &&
> >>>> +		    tmp <= mali_c55_coefficient_banks[i].top)
> >>>> +			return mali_c55_coefficient_banks[i].bank;
> >>>> +	}
> >>>> +
> >>>> +	/*
> >>>> +	 * We shouldn't ever get here, in theory. As we have no good choices
> >>>> +	 * simply warn the user and use the first bank of coefficients.
> >>>> +	 */
> >>>> +	dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
> >>>> +	return 0;
> >>>> +}
> >>> 
> >>> And everything else belongs to mali-c55-resizer.c. Drop this header
> >>> file.
> >>>
> >>>> +
> >>>> +#endif /* _MALI_C55_RESIZER_COEFS_H */
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>>> new file mode 100644
> >>>> index 000000000000..0a5a2969d3ce
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>>> @@ -0,0 +1,779 @@
> >>>> +// SPDX-License-Identifier: GPL-2.0
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Image signal processor
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#include <linux/math.h>
> >>>> +#include <linux/minmax.h>
> >>>> +
> >>>> +#include <media/media-entity.h>
> >>>> +#include <media/v4l2-subdev.h>
> >>>> +
> >>>> +#include "mali-c55-common.h"
> >>>> +#include "mali-c55-registers.h"
> >>>> +#include "mali-c55-resizer-coefs.h"
> >>>> +
> >>>> +/* Scaling factor in Q4.20 format. */
> >>>> +#define MALI_C55_RZR_SCALER_FACTOR	(1U << 20)
> >>>> +
> >>>> +static const u32 rzr_non_bypass_src_fmts[] = {
> >>>> +	MEDIA_BUS_FMT_RGB121212_1X36,
> >>>> +	MEDIA_BUS_FMT_YUV10_1X30
> >>>> +};
> >>>> +
> >>>> +static const char * const mali_c55_resizer_names[] = {
> >>>> +	[MALI_C55_RZR_FR] = "resizer fr",
> >>>> +	[MALI_C55_RZR_DS] = "resizer ds",
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
> >>>> +				     struct v4l2_subdev_state *state)
> >>>> +{
> >>>> +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
> >>>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>> +	struct v4l2_mbus_framefmt *fmt;
> >>>> +	struct v4l2_rect *crop;
> >>>> +
> >>>> +	/* Verify if crop should be enabled. */
> >>>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>> +
> >>>> +	if (fmt->width == crop->width && fmt->height == crop->height)
> >>>> +		return MALI_C55_BYPASS_CROP;
> >>>> +
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
> >>>> +		       crop->left);
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
> >>>> +		       crop->top);
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
> >>>> +		       crop->width);
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
> >>>> +		       crop->height);
> >>>> +
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
> >>>> +		       MALI_C55_CROP_ENABLE);
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
> >>>> +					struct v4l2_subdev_state *state)
> >>>> +{
> >>>> +	unsigned int reg_offset = rzr->cap_dev->reg_offset;
> >>>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>> +	struct v4l2_rect *crop, *scale;
> >>>> +	unsigned int h_bank, v_bank;
> >>>> +	u64 h_scale, v_scale;
> >>>> +
> >>>> +	/* Verify if scaling should be enabled. */
> >>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>> +	scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>> +
> >>>> +	if (crop->width == scale->width && crop->height == scale->height)
> >>>> +		return MALI_C55_BYPASS_SCALER;
> >>>> +
> >>>> +	/* Program the V/H scaling factor in Q4.20 format. */
> >>>> +	h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
> >>>> +	v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
> >>>> +
> >>>> +	do_div(h_scale, scale->width);
> >>>> +	do_div(v_scale, scale->height);
> >>>> +
> >>>> +	mali_c55_write(mali_c55,
> >>>> +		       MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
> >>>> +		       crop->width);
> >>>> +	mali_c55_write(mali_c55,
> >>>> +		       MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
> >>>> +		       crop->height);
> >>>> +
> >>>> +	mali_c55_write(mali_c55,
> >>>> +		       MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
> >>>> +		       scale->width);
> >>>> +	mali_c55_write(mali_c55,
> >>>> +		       MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
> >>>> +		       scale->height);
> >>>> +
> >>>> +	mali_c55_write(mali_c55,
> >>>> +		       MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
> >>>> +		       h_scale);
> >>>> +	mali_c55_write(mali_c55,
> >>>> +		       MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
> >>>> +		       v_scale);
> >>>> +
> >>>> +	h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
> >>>> +					     scale->width);
> >>>> +	mali_c55_write(mali_c55,
> >>>> +		       MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
> >>>> +		       h_bank);
> >>>> +
> >>>> +	v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
> >>>> +					     scale->height);
> >>>> +	mali_c55_write(mali_c55,
> >>>> +		       MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
> >>>> +		       v_bank);
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
> >>>> +				 struct v4l2_subdev_state *state)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>> +	u32 bypass = 0;
> >>>> +
> >>>> +	/* Verify if cropping and scaling should be enabled. */
> >>>> +	bypass |= mali_c55_rzr_program_crop(rzr, state);
> >>>> +	bypass |= mali_c55_rzr_program_resizer(rzr, state);
> >>>> +
> >>>> +	mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
> >>>> +			     MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
> >>>> +			     MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
> >>>> +			     bypass);
> >>>> +}
> >>>> +
> >>>> +/*
> >>>> + * Inspect the routing table to know which of the two (mutually exclusive)
> >>>> + * routes is enabled and return the sink pad id of the active route.
> >>>> + */
> >>>> +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
> >>>> +{
> >>>> +	struct v4l2_subdev_krouting *routing = &state->routing;
> >>>> +	struct v4l2_subdev_route *route;
> >>>> +
> >>>> +	/* A single route is enabled at a time. */
> >>>> +	for_each_active_route(routing, route)
> >>>> +		return route->sink_pad;
> >>>> +
> >>>> +	return MALI_C55_RZR_SINK_PAD;
> >>>> +}
> >>>> +
> >>>> +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
> >>>> +{
> >>>> +	u32 corrected_code = 0;
> >>>> +
> >>>> +	/*
> >>>> +	 * The ISP takes input in a 20-bit format, but can only output 16-bit
> >>>> +	 * RAW bayer data (with the 4 least significant bits from the input
> >>>> +	 * being lost). Return the 16-bit version of the 20-bit input formats.
> >>>> +	 */
> >>>> +	switch (mbus_code) {
> >>>> +	case MEDIA_BUS_FMT_SBGGR20_1X20:
> >>>> +		corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
> >>>> +		break;
> >>>> +	case MEDIA_BUS_FMT_SGBRG20_1X20:
> >>>> +		corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
> >>>> +		break;
> >>>> +	case MEDIA_BUS_FMT_SGRBG20_1X20:
> >>>> +		corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
> >>>> +		break;
> >>>> +	case MEDIA_BUS_FMT_SRGGB20_1X20:
> >>>> +		corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
> >>>> +		break;
> >>>> +	}
> >>>> +
> >>>> +	return corrected_code;
> >>>> +}
> >>>> +
> >>>> +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> >>>> +				      struct v4l2_subdev_state *state,
> >>>> +				      struct v4l2_subdev_krouting *routing)
> >>>> +{
> >>>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>> +						    sd);
> >>>> +	unsigned int active_sink = UINT_MAX;
> >>>> +	struct v4l2_mbus_framefmt *src_fmt;
> >>>> +	struct v4l2_rect *crop, *compose;
> >>>> +	struct v4l2_subdev_route *route;
> >>>> +	unsigned int active_routes = 0;
> >>>> +	struct v4l2_mbus_framefmt *fmt;
> >>>> +	int ret;
> >>>> +
> >>>> +	ret = v4l2_subdev_routing_validate(sd, routing, 0);
> >>>> +	if (ret)
> >>>> +		return ret;
> >>>> +
> >>>> +	/* Only a single route can be enabled at a time. */
> >>>> +	for_each_active_route(routing, route) {
> >>>> +		if (++active_routes > 1) {
> >>>> +			dev_err(rzr->mali_c55->dev,
> >>>> +				"Only one route can be active");
> >>>> +			return -EINVAL;
> >>>> +		}
> >>>> +
> >>>> +		active_sink = route->sink_pad;
> >>>> +	}
> >>>> +	if (active_sink == UINT_MAX) {
> >>>> +		dev_err(rzr->mali_c55->dev, "One route has to be active");
> >>>> +		return -EINVAL;
> >>>> +	}
> >>>> +
> >>>> +	ret = v4l2_subdev_set_routing(sd, state, routing);
> >>>> +	if (ret) {
> >>>> +		dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
> >>>> +		return ret;
> >>>> +	}
> >>>> +
> >>>> +	fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
> >>>> +	crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
> >>>> +	compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
> >>>> +
> >>>> +	fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>> +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>> +	fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>> +	fmt->field = V4L2_FIELD_NONE;
> >>>> +
> >>>> +	if (active_sink == MALI_C55_RZR_SINK_PAD) {
> >>>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>> +
> >>>> +		crop->left = crop->top = 0;
> >>>> +		crop->width = MALI_C55_DEFAULT_WIDTH;
> >>>> +		crop->height = MALI_C55_DEFAULT_HEIGHT;
> >>>> +
> >>>> +		*compose = *crop;
> >>>> +	} else {
> >>>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>> +	}
> >>>> +
> >>>> +	/* Propagate the format to the source pad */
> >>>> +	src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
> >>>> +					       0);
> >>>> +	*src_fmt = *fmt;
> >>>> +
> >>>> +	/* In the event this is the bypass pad the mbus code needs correcting */
> >>>> +	if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
> >>>> +		src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
> >>>> +				       struct v4l2_subdev_state *state,
> >>>> +				       struct v4l2_subdev_mbus_code_enum *code)
> >>>> +{
> >>>> +	struct v4l2_mbus_framefmt *sink_fmt;
> >>>> +	const struct mali_c55_isp_fmt *fmt;
> >>>> +	unsigned int index = 0;
> >>>> +	u32 sink_pad;
> >>>> +
> >>>> +	switch (code->pad) {
> >>>> +	case MALI_C55_RZR_SINK_PAD:
> >>>> +		if (code->index)
> >>>> +			return -EINVAL;
> >>>> +
> >>>> +		code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>> +
> >>>> +		return 0;
> >>>> +	case MALI_C55_RZR_SOURCE_PAD:
> >>>> +		sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>> +		sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> >>>> +
> >>>> +		/*
> >>>> +		 * If the active route is from the Bypass sink pad, then the
> >>>> +		 * source pad is a simple passthrough of the sink format,
> >>>> +		 * downshifted to 16-bits.
> >>>> +		 */
> >>>> +
> >>>> +		if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >>>> +			if (code->index)
> >>>> +				return -EINVAL;
> >>>> +
> >>>> +			code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> >>>> +			if (!code->code)
> >>>> +				return -EINVAL;
> >>>> +
> >>>> +			return 0;
> >>>> +		}
> >>>> +
> >>>> +		/*
> >>>> +		 * If the active route is from the non-bypass sink then we can
> >>>> +		 * select either RGB or conversion to YUV.
> >>>> +		 */
> >>>> +
> >>>> +		if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
> >>>> +			return -EINVAL;
> >>>> +
> >>>> +		code->code = rzr_non_bypass_src_fmts[code->index];
> >>>> +
> >>>> +		return 0;
> >>>> +	case MALI_C55_RZR_SINK_BYPASS_PAD:
> >>>> +		for_each_mali_isp_fmt(fmt) {
> >>>> +			if (index++ == code->index) {
> >>>> +				code->code = fmt->code;
> >>>> +				return 0;
> >>>> +			}
> >>>> +		}
> >>>> +
> >>>> +		break;
> >>>> +	}
> >>>> +
> >>>> +	return -EINVAL;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
> >>>> +					struct v4l2_subdev_state *state,
> >>>> +					struct v4l2_subdev_frame_size_enum *fse)
> >>>> +{
> >>>> +	if (fse->index)
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	fse->max_width = MALI_C55_MAX_WIDTH;
> >>>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
> >>>> +	fse->min_width = MALI_C55_MIN_WIDTH;
> >>>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
> >>>> +				     struct v4l2_subdev_state *state,
> >>>> +				     struct v4l2_subdev_format *format)
> >>>> +{
> >>>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>> +	struct v4l2_rect *rect;
> >>>> +	unsigned int sink_pad;
> >>>> +
> >>>> +	/*
> >>>> +	 * Clamp to min/max and then reset crop and compose rectangles to the
> >>>> +	 * newly applied size.
> >>>> +	 */
> >>>> +	clamp_t(unsigned int, fmt->width,
> >>>> +		MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>>> +	clamp_t(unsigned int, fmt->height,
> >>>> +		MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>>> +
> >>>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>> +	if (sink_pad == MALI_C55_RZR_SINK_PAD) {
> >>>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>> +
> >>>> +		rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> >>>> +		rect->left = 0;
> >>>> +		rect->top = 0;
> >>>> +		rect->width = fmt->width;
> >>>> +		rect->height = fmt->height;
> >>>> +
> >>>> +		rect = v4l2_subdev_state_get_compose(state,
> >>>> +						     MALI_C55_RZR_SINK_PAD);
> >>>> +		rect->left = 0;
> >>>> +		rect->top = 0;
> >>>> +		rect->width = fmt->width;
> >>>> +		rect->height = fmt->height;
> >>>> +	} else {
> >>>> +		/*
> >>>> +		 * Make sure the media bus code is one of the supported
> >>>> +		 * ISP input media bus codes.
> >>>> +		 */
> >>>> +		if (!mali_c55_isp_is_format_supported(fmt->code))
> >>>> +			fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
> >>>> +	}
> >>>> +
> >>>> +	*v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
> >>>> +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> >>>> +				       struct v4l2_subdev_state *state,
> >>>> +				       struct v4l2_subdev_format *format)
> >>>> +{
> >>>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>> +						    sd);
> >>>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>> +	struct v4l2_mbus_framefmt *sink_fmt;
> >>>> +	struct v4l2_rect *crop, *compose;
> >>>> +	unsigned int sink_pad;
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>> +	sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> >>>> +	crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
> >>>> +	compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
> >>>> +
> >>>> +	/* FR Bypass pipe. */
> >>>> +
> >>>> +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >>>> +		/*
> >>>> +		 * Format on the source pad is the same as the one on the
> >>>> +		 * sink pad, downshifted to 16-bits.
> >>>> +		 */
> >>>> +		fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> >>>> +		if (!fmt->code)
> >>>> +			return -EINVAL;
> >>>> +
> >>>> +		/* RAW bypass disables scaling and cropping. */
> >>>> +		crop->top = compose->top = 0;
> >>>> +		crop->left = compose->left = 0;
> >>>> +		fmt->width = crop->width = compose->width = sink_fmt->width;
> >>>> +		fmt->height = crop->height = compose->height = sink_fmt->height;
> >>>> +
> >>>> +		*v4l2_subdev_state_get_format(state,
> >>>> +					      MALI_C55_RZR_SOURCE_PAD) = *fmt;
> >>>> +
> >>>> +		return 0;
> >>>> +	}
> >>>> +
> >>>> +	/* Regular processing pipe. */
> >>>> +
> >>>> +	for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> >>>> +		if (fmt->code == rzr_non_bypass_src_fmts[i])
> >>>> +			break;
> >>>> +	}
> >>>> +
> >>>> +	if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
> >>>> +		dev_dbg(rzr->mali_c55->dev,
> >>>> +			"Unsupported mbus code 0x%x: using default\n",
> >>>> +			fmt->code);
> >>>> +		fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>> +	}
> >>>> +
> >>>> +	/*
> >>>> +	 * The source pad format size comes directly from the sink pad
> >>>> +	 * compose rectangle.
> >>>> +	 */
> >>>> +	fmt->width = compose->width;
> >>>> +	fmt->height = compose->height;
> >>>> +
> >>>> +	*v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
> >>>> +				struct v4l2_subdev_state *state,
> >>>> +				struct v4l2_subdev_format *format)
> >>>> +{
> >>>> +	/*
> >>>> +	 * On sink pads fmt is either fixed for the 'regular' processing
> >>>> +	 * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
> >>>> +	 * pad.
> >>>> +	 *
> >>>> +	 * On source pad sizes are the result of crop+compose on the sink
> >>>> +	 * pad sizes, while the format depends on the active route.
> >>>> +	 */
> >>>> +
> >>>> +	if (format->pad != MALI_C55_RZR_SOURCE_PAD)
> >>>> +		return mali_c55_rzr_set_sink_fmt(sd, state, format);
> >>>> +
> >>>> +	return mali_c55_rzr_set_source_fmt(sd, state, format);
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
> >>>> +				      struct v4l2_subdev_state *state,
> >>>> +				      struct v4l2_subdev_selection *sel)
> >>>> +{
> >>>> +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	if (sel->target != V4L2_SEL_TGT_CROP &&
> >>>> +	    sel->target != V4L2_SEL_TGT_COMPOSE)
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	sel->r = sel->target == V4L2_SEL_TGT_CROP
> >>>> +	       ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
> >>>> +	       : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
> >>>> +				      struct v4l2_subdev_state *state,
> >>>> +				      struct v4l2_subdev_selection *sel)
> >>>> +{
> >>>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>> +						    sd);
> >>>> +	struct v4l2_mbus_framefmt *source_fmt;
> >>>> +	struct v4l2_mbus_framefmt *sink_fmt;
> >>>> +	struct v4l2_rect *crop, *compose;
> >>>> +
> >>>> +	if (sel->pad != MALI_C55_RZR_SINK_PAD)
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	if (sel->target != V4L2_SEL_TGT_CROP &&
> >>>> +	    sel->target != V4L2_SEL_TGT_COMPOSE)
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	source_fmt = v4l2_subdev_state_get_format(state,
> >>>> +						  MALI_C55_RZR_SOURCE_PAD);
> >>>> +	sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
> >>>> +	crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> >>>> +	compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> >>>> +
> >>>> +	/* RAW bypass disables crop/scaling. */
> >>>> +	if (mali_c55_format_is_raw(source_fmt->code)) {
> >>>> +		crop->top = compose->top = 0;
> >>>> +		crop->left = compose->left = 0;
> >>>> +		crop->width = compose->width = sink_fmt->width;
> >>>> +		crop->height = compose->height = sink_fmt->height;
> >>>> +
> >>>> +		sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> >>>> +
> >>>> +		return 0;
> >>>> +	}
> >>>> +
> >>>> +	/* During streaming, it is allowed to only change the crop rectangle. */
> >>>> +	if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	 /*
> >>>> +	  * Update the desired target and then clamp the crop rectangle to the
> >>>> +	  * sink format sizes and the compose size to the crop sizes.
> >>>> +	  */
> >>>> +	if (sel->target == V4L2_SEL_TGT_CROP)
> >>>> +		*crop = sel->r;
> >>>> +	else
> >>>> +		*compose = sel->r;
> >>>> +
> >>>> +	clamp_t(unsigned int, crop->left, 0,  sink_fmt->width);
> >>>> +	clamp_t(unsigned int, crop->top, 0,  sink_fmt->height);
> >>>> +	clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
> >>>> +		sink_fmt->width - crop->left);
> >>>> +	clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
> >>>> +		sink_fmt->height - crop->top);
> >>>> +
> >>>> +	if (rzr->streaming) {
> >>>> +		/*
> >>>> +		 * Apply at runtime a crop rectangle on the resizer's sink only
> >>>> +		 * if it doesn't require re-programming the scaler output sizes
> >>>> +		 * as it would require changing the output buffer sizes as well.
> >>>> +		 */
> >>>> +		if (sel->r.width < compose->width ||
> >>>> +		    sel->r.height < compose->height)
> >>>> +			return -EINVAL;
> >>>> +
> >>>> +		*crop = sel->r;
> >>>> +		mali_c55_rzr_program(rzr, state);
> >>>> +
> >>>> +		return 0;
> >>>> +	}
> >>>> +
> >>>> +	compose->left = 0;
> >>>> +	compose->top = 0;
> >>>> +	clamp_t(unsigned int, compose->left, 0,  sink_fmt->width);
> >>>> +	clamp_t(unsigned int, compose->top, 0,  sink_fmt->height);
> >>>> +	clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
> >>>> +	clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
> >>>> +
> >>>> +	sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> >>>> +				    struct v4l2_subdev_state *state,
> >>>> +				    enum v4l2_subdev_format_whence which,
> >>>> +				    struct v4l2_subdev_krouting *routing)
> >>>> +{
> >>>> +	if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
> >>>> +	    media_entity_is_streaming(&sd->entity))
> >>>> +		return -EBUSY;
> >>>> +
> >>>> +	return __mali_c55_rzr_set_routing(sd, state, routing);
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
> >>>> +	.enum_mbus_code		= mali_c55_rzr_enum_mbus_code,
> >>>> +	.enum_frame_size	= mali_c55_rzr_enum_frame_size,
> >>>> +	.get_fmt		= v4l2_subdev_get_fmt,
> >>>> +	.set_fmt		= mali_c55_rzr_set_fmt,
> >>>> +	.get_selection		= mali_c55_rzr_get_selection,
> >>>> +	.set_selection		= mali_c55_rzr_set_selection,
> >>>> +	.set_routing		= mali_c55_rzr_set_routing,
> >>>> +};
> >>>> +
> >>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
> >>>> +{
> >>>> +	struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>> +	struct v4l2_subdev *sd = &rzr->sd;
> >>>> +	struct v4l2_subdev_state *state;
> >>>> +	unsigned int sink_pad;
> >>>> +
> >>>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>> +
> >>>> +	sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>> +	if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >>>> +		/* Bypass FR pipe processing if the bypass route is active. */
> >>>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >>>> +				     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
> >>>> +				     MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
> >>>> +		goto unlock_state;
> >>>> +	}
> >>>> +
> >>>> +	/* Disable bypass and use regular processing. */
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >>>> +			     MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
> >>>> +	mali_c55_rzr_program(rzr, state);
> >>>> +
> >>>> +unlock_state:
> >>>> +	rzr->streaming = true;
> >>>> +	v4l2_subdev_unlock_state(state);
> >>>> +}
> >>>> +
> >>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
> >>>> +{
> >>>> +	struct v4l2_subdev *sd = &rzr->sd;
> >>>> +	struct v4l2_subdev_state *state;
> >>>> +
> >>>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>> +	rzr->streaming = false;
> >>>> +	v4l2_subdev_unlock_state(state);
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
> >>>> +	.pad	= &mali_c55_resizer_pad_ops,
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
> >>>> +				   struct v4l2_subdev_state *state)
> >>>> +{
> >>>> +	struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>> +						    sd);
> >>>> +	struct v4l2_subdev_krouting routing = { };
> >>>> +	struct v4l2_subdev_route *routes;
> >>>> +	unsigned int i;
> >>>> +	int ret;
> >>>> +
> >>>> +	routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
> >>>> +	if (!routes)
> >>>> +		return -ENOMEM;
> >>>> +
> >>>> +	for (i = 0; i < rzr->num_routes; ++i) {
> >>>> +		struct v4l2_subdev_route *route = &routes[i];
> >>>> +
> >>>> +		route->sink_pad = i
> >>>> +				? MALI_C55_RZR_SINK_BYPASS_PAD
> >>>> +				: MALI_C55_RZR_SINK_PAD;
> >>>> +		route->source_pad = MALI_C55_RZR_SOURCE_PAD;
> >>>> +		if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
> >>>> +			route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
> >>>> +	}
> >>>> +
> >>>> +	routing.num_routes = rzr->num_routes;
> >>>> +	routing.routes = routes;
> >>>> +
> >>>> +	ret = __mali_c55_rzr_set_routing(sd, state, &routing);
> >>>> +	kfree(routes);
> >>>> +
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
> >>>> +	.init_state = mali_c55_rzr_init_state,
> >>>> +};
> >>>> +
> >>>> +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
> >>>> +						  unsigned int index)
> >>>> +{
> >>>> +	const unsigned int scaler_filt_coefmem_addrs[][2] = {
> >>>> +		[MALI_C55_RZR_FR] = {
> >>>> +			0x034A8, /* hfilt */
> >>>> +			0x044A8  /* vfilt */
> >>> 
> >>> Lowercase hex constants.
> >>>
> >>>> +		},
> >>>> +		[MALI_C55_RZR_DS] = {
> >>>> +			0x014A8, /* hfilt */
> >>>> +			0x024A8  /* vfilt */
> >>>> +		},
> >>>> +	};
> >>>> +	unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
> >>>> +	unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
> >>>> +	unsigned int i, j;
> >>>> +
> >>>> +	for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
> >>>> +		for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
> >>>> +			mali_c55_write(mali_c55, haddr,
> >>>> +				mali_c55_scaler_h_filter_coefficients[i][j]);
> >>>> +			mali_c55_write(mali_c55, vaddr,
> >>>> +				mali_c55_scaler_v_filter_coefficients[i][j]);
> >>>> +
> >>>> +			haddr += sizeof(u32);
> >>>> +			vaddr += sizeof(u32);
> >>>> +		}
> >>>> +	}
> >>>> +}
> >>>> +
> >>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	unsigned int i;
> >>>> +	int ret;
> >>>> +
> >>>> +	for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
> >>>> +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> >>>> +		struct v4l2_subdev *sd = &rzr->sd;
> >>>> +		unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
> >>>> +
> >>>> +		rzr->id = i;
> >>>> +		rzr->streaming = false;
> >>>> +
> >>>> +		if (rzr->id == MALI_C55_RZR_FR)
> >>>> +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
> >>>> +		else
> >>>> +			rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
> >>>> +
> >>>> +		mali_c55_resizer_program_coefficients(mali_c55, i);
> >>>> +
> >>>> +		v4l2_subdev_init(sd, &mali_c55_resizer_ops);
> >>>> +		sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
> >>>> +			     | V4L2_SUBDEV_FL_STREAMS;
> >>>> +		sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
> >>>> +		sd->internal_ops = &mali_c55_resizer_internal_ops;
> >>>> +		snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
> >>>> +			 MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
> >>>> +
> >>>> +		rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
> >>>> +		rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
> >>>> +
> >>>> +		/* Only the FR pipe has a bypass pad. */
> >>>> +		if (rzr->id == MALI_C55_RZR_FR) {
> >>>> +			rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
> >>>> +							MEDIA_PAD_FL_SINK;
> >>>> +			rzr->num_routes = 2;
> >>>> +		} else {
> >>>> +			num_pads -= 1;
> >>>> +			rzr->num_routes = 1;
> >>>> +		}
> >>>> +
> >>>> +		ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
> >>>> +		if (ret)
> >>>> +			return ret;
> >>>> +
> >>>> +		ret = v4l2_subdev_init_finalize(sd);
> >>>> +		if (ret)
> >>>> +			goto err_cleanup;
> >>>> +
> >>>> +		ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >>>> +		if (ret)
> >>>> +			goto err_cleanup;
> >>>> +
> >>>> +		rzr->mali_c55 = mali_c55;
> >>>> +	}
> >>>> +
> >>>> +	return 0;
> >>>> +
> >>>> +err_cleanup:
> >>>> +	for (; i >= 0; --i) {
> >>>> +		struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> >>>> +		struct v4l2_subdev *sd = &rzr->sd;
> >>>> +
> >>>> +		v4l2_subdev_cleanup(sd);
> >>>> +		media_entity_cleanup(&sd->entity);
> >>>> +	}
> >>>> +
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
> >>>> +		struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
> >>>> +
> >>>> +		if (!resizer->mali_c55)
> >>>> +			continue;
> >>>> +
> >>>> +		v4l2_device_unregister_subdev(&resizer->sd);
> >>>> +		v4l2_subdev_cleanup(&resizer->sd);
> >>>> +		media_entity_cleanup(&resizer->sd.entity);
> >>>> +	}
> >>>> +}
> >>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >>>> new file mode 100644
> >>>> index 000000000000..c7e699741c6d
> >>>> --- /dev/null
> >>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >>>> @@ -0,0 +1,402 @@
> >>>> +// SPDX-License-Identifier: GPL-2.0
> >>>> +/*
> >>>> + * ARM Mali-C55 ISP Driver - Test pattern generator
> >>>> + *
> >>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>> + */
> >>>> +
> >>>> +#include <linux/minmax.h>
> >>>> +#include <linux/string.h>
> >>>> +
> >>>> +#include <media/media-entity.h>
> >>>> +#include <media/v4l2-ctrls.h>
> >>>> +#include <media/v4l2-subdev.h>
> >>>> +
> >>>> +#include "mali-c55-common.h"
> >>>> +#include "mali-c55-registers.h"
> >>>> +
> >>>> +#define MALI_C55_TPG_SRC_PAD		0
> >>>> +#define MALI_C55_TPG_FIXED_HBLANK	0x20
> >>>> +#define MALI_C55_TPG_MAX_VBLANK		0xFFFF
> >>> 
> >>> Lowercase hex constants.
> >>>
> >>>> +#define MALI_C55_TPG_PIXEL_RATE		100000000
> >>> 
> >>> This should be exposed to applications using the V4L2_CID_PIXEL_RATE
> >>> control (read-only).
> >>>
> >>>> +
> >>>> +static const char * const mali_c55_tpg_test_pattern_menu[] = {
> >>>> +	"Flat field",
> >>>> +	"Horizontal gradient",
> >>>> +	"Vertical gradient",
> >>>> +	"Vertical bars",
> >>>> +	"Arbitrary rectangle",
> >>>> +	"White frame on black field"
> >>>> +};
> >>>> +
> >>>> +static const u32 mali_c55_tpg_mbus_codes[] = {
> >>>> +	MEDIA_BUS_FMT_SRGGB20_1X20,
> >>>> +	MEDIA_BUS_FMT_RGB202020_1X60,
> >>>> +};
> >>>> +
> >>>> +static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
> >>>> +				       int *def_vblank, int *min_vblank)
> >>> 
> >>> unsigned int ?
> >>>
> >>>> +{
> >>>> +	unsigned int hts;
> >>>> +	int tgt_fps;
> >>>> +	int vblank;
> >>>> +
> >>>> +	hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
> >>>> +
> >>>> +	/*
> >>>> +	 * The ISP has minimum vertical blanking requirements that must be
> >>>> +	 * adhered to by the TPG. The minimum is a function of the Iridix blocks
> >>>> +	 * clocking requirements and the width of the image and horizontal
> >>>> +	 * blanking, but if we assume the worst case iVariance and sVariance
> >>>> +	 * values then it boils down to the below.
> >>>> +	 */
> >>>> +	*min_vblank = 15 + (120500 / hts);
> >>> 
> >>> I wonder if this should round up.
> >> 
> >> Maybe? I think the difference is probably too minor to have a practical effect
> > 
> > Just to make sure we won't have any problem by having a just too short
> > minimum vblank when rounding down.
> >
> >>>> +
> >>>> +	/*
> >>>> +	 * We need to set a sensible default vblank for whatever format height
> >>>> +	 * we happen to be given from set_fmt(). This function just targets
> >>>> +	 * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
> >>>> +	 * If we can't get 5fps we'll take whatever the minimum vblank gives us.
> >>>> +	 */
> >>>> +	tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
> >>>> +
> >>>> +	if (tgt_fps < 5)
> >>>> +		vblank = *min_vblank;
> >>>> +	else
> >>>> +		vblank = MALI_C55_TPG_PIXEL_RATE / hts
> >>>> +		       / max(rounddown(tgt_fps, 15), 5);
> >>>> +
> >>>> +	*def_vblank = ALIGN_DOWN(vblank, 2) - format->height;
> >>> 
> >>> "vblank = vblank - height" doesn't seem right. The "else" branch stores
> >>> a vts in vblank, which doesn't seem right either. Maybe you meant
> >>> something like
> >>>
> >>> 	if (tgt_fps < 5)
> >>> 		def_vts = *min_vblank + format->height;
> >>> 	else
> >>> 		def_vts = MALI_C55_TPG_PIXEL_RATE / hts
> >>> 			/ max(rounddown(tgt_fps, 15), 5);
> >>>
> >>> 	*def_vblank = ALIGN_DOWN(def_vts, 2) - format->height;
> >> 
> >> I did, thank you.
> >>
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
> >>>> +{
> >>>> +	struct mali_c55_tpg *tpg = container_of(ctrl->handler,
> >>>> +						struct mali_c55_tpg,
> >>>> +						ctrls.handler);
> >>>> +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
> >>>> +
> >>> 
> >>> Should you return here if the pipeline isn't streaming ?
> >> 
> >> Yes probably; or if (pm_runtime_get_if_in_use()) would be the usual model I guess.
> > 
> > I think so yes.
> >
> >>>> +	switch (ctrl->id) {
> >>>> +	case V4L2_CID_TEST_PATTERN:
> >>>> +		mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
> >>>> +			       ctrl->val);
> >>>> +		break;
> >>>> +	case V4L2_CID_VBLANK:
> >>>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
> >>>> +				     MALI_C55_REG_VBLANK_MASK, ctrl->val);
> >>>> +		break;
> >>>> +	default:
> >>>> +		dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
> >>>> +		return -EINVAL;
> >>> 
> >>> Can this happen ?
> >> 
> >> Not unless somebody breaks something
> > 
> > Then I think you can drop the error message.
> 
> Ack
> 
> >>>> +	}
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
> >>>> +	.s_ctrl = &mali_c55_tpg_s_ctrl,
> >>>> +};
> >>>> +
> >>>> +static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
> >>>> +				   struct v4l2_subdev *sd)
> >>>> +{
> >>>> +	struct v4l2_subdev_state *state;
> >>>> +	struct v4l2_mbus_framefmt *fmt;
> >>>> +
> >>>> +	/*
> >>>> +	 * hblank needs setting, but is a read-only control and thus won't be
> >>>> +	 * called during __v4l2_ctrl_handler_setup(). Do it here instead.
> >>>> +	 */
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
> >>>> +			     MALI_C55_REG_HBLANK_MASK,
> >>>> +			     MALI_C55_TPG_FIXED_HBLANK);
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >>>> +			     MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
> >>>> +
> >>>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>> +	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>> +
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >>>> +			     MALI_C55_TEST_PATTERN_RGB_MASK,
> >>>> +			     fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
> >>>> +					  0x01 : 0x0);
> >>>> +
> >>>> +	v4l2_subdev_unlock_state(state);
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
> >>>> +{
> >>>> +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
> >>>> +	struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
> >>>> +
> >>>> +	if (!enable) {
> >>>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >>>> +				MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
> >>>> +		mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >>>> +				MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
> >>>> +		return 0;
> >>>> +	}
> >>>> +
> >>>> +	/*
> >>>> +	 * One might reasonably expect the framesize to be set here
> >>>> +	 * given it's configurable in .set_fmt(), but it's done in the
> >>>> +	 * ISP subdevice's stream on func instead, as the same register
> >>> 
> >>> s/func/function/
> >>>
> >>>> +	 * is also used to indicate the size of the data coming from the
> >>>> +	 * sensor.
> >>>> +	 */
> >>>> +	mali_c55_tpg_configure(mali_c55, sd);
> >>> 
> >>> 	mali_c55_tpg_configure(tpg);
> >>>
> >>>> +	__v4l2_ctrl_handler_setup(sd->ctrl_handler);
> >>>> +
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >>>> +			     MALI_C55_TEST_PATTERN_ON_OFF,
> >>>> +			     MALI_C55_TEST_PATTERN_ON_OFF);
> >>>> +	mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >>>> +			     MALI_C55_REG_GEN_VIDEO_ON_MASK,
> >>>> +			     MALI_C55_REG_GEN_VIDEO_ON_MASK);
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
> >>>> +	.s_stream = &mali_c55_tpg_s_stream,
> >>> 
> >>> Can we use .enable_streams() and .disable_streams() ?
> >> 
> >> Yes, with some extra patches from Tomi that Sakari is picking - I'll base on top of those for the v6.
> >>
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
> >>>> +				       struct v4l2_subdev_state *state,
> >>>> +				       struct v4l2_subdev_mbus_code_enum *code)
> >>>> +{
> >>>> +	if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	code->code = mali_c55_tpg_mbus_codes[code->index];
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
> >>>> +					struct v4l2_subdev_state *state,
> >>>> +					struct v4l2_subdev_frame_size_enum *fse)
> >>>> +{
> >>> 
> >>> You sohuld verify here that fse->code is a supported value and return
> >>> -EINVAL otherwise.
> >>>
> >>>> +	if (fse->index > 0 || fse->pad > sd->entity.num_pads)
> >>> 
> >>> Drop the pad check, it's done in the subdev core already.
> >>>
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	fse->min_width = MALI_C55_MIN_WIDTH;
> >>>> +	fse->max_width = MALI_C55_MAX_WIDTH;
> >>>> +	fse->min_height = MALI_C55_MIN_HEIGHT;
> >>>> +	fse->max_height = MALI_C55_MAX_HEIGHT;
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
> >>>> +				struct v4l2_subdev_state *state,
> >>>> +				struct v4l2_subdev_format *format)
> >>>> +{
> >>>> +	struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
> >>>> +	struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>> +	int vblank_def, vblank_min;
> >>>> +	unsigned int i;
> >>>> +
> >>>> +	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> >>>> +		if (fmt->code == mali_c55_tpg_mbus_codes[i])
> >>>> +			break;
> >>>> +	}
> >>>> +
> >>>> +	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>>> +		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>> +
> >>>> +	/*
> >>>> +	 * The TPG says that the test frame timing generation logic expects a
> >>>> +	 * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
> >>>> +	 * handle anything smaller than 128x128 it seems pointless to allow a
> >>>> +	 * smaller frame.
> >>>> +	 */
> >>>> +	clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> >>>> +		MALI_C55_MAX_WIDTH);
> >>>> +	clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> >>>> +		MALI_C55_MAX_HEIGHT);
> >>>> +
> >>>> +	*v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;
> >>> 
> >>> You're allowing userspace to set fmt->field, as well as all the
> >>> colorspace parameters, to random values. I would instead do something
> >>> like
> >>>
> >>> 	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> >>> 		if (format->format.code == mali_c55_tpg_mbus_codes[i])
> >>> 			break;
> >>> 	}
> >>>
> >>> 	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>> 		format->format.code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>
> >>> 	format->format.width = clamp(format->format.width,
> >>> 				     MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>> 	format->format.height = clamp(format->format.height,
> >>> 				      MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>>
> >>> 	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>> 	fmt->code = format->format.code;
> >>> 	fmt->width = format->format.width;
> >>> 	fmt->height = format->format.height;
> >>>
> >>> 	format->format = *fmt;
> >>>
> >>> Alternatively (which I think I like better),
> >>>
> >>> 	fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>
> >>> 	fmt->code = format->format.code;
> >>>
> >>> 	for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> >>> 		if (fmt->code == mali_c55_tpg_mbus_codes[i])
> >>> 			break;
> >>> 	}
> >>>
> >>> 	if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>> 		fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>
> >>> 	fmt->width = clamp(format->format.width,
> >>> 			   MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>> 	fmt->height = clamp(format->format.height,
> >>> 			    MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>>
> >>> 	format->format = *fmt;
> >>>
> >>>> +
> >>>> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY)
> >>>> +		return 0;
> >>>> +
> >>>> +	__mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
> >>>> +	__v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
> >>>> +				 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
> >>>> +	__v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);
> >>> 
> >>> Move those three calls to a separate function, it will be reused below.
> >>> I'd name is mali_c55_tpg_update_vblank(). You can fold
> >>> __mali_c55_tpg_calc_vblank() in it.
> >>>
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
> >>>> +	.enum_mbus_code		= mali_c55_tpg_enum_mbus_code,
> >>>> +	.enum_frame_size	= mali_c55_tpg_enum_frame_size,
> >>>> +	.get_fmt		= v4l2_subdev_get_fmt,
> >>>> +	.set_fmt		= mali_c55_tpg_set_fmt,
> >>>> +};
> >>>> +
> >>>> +static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
> >>>> +	.video	= &mali_c55_tpg_video_ops,
> >>>> +	.pad	= &mali_c55_tpg_pad_ops,
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
> >>>> +				   struct v4l2_subdev_state *sd_state)
> >>> 
> >>> You name this variable state in every other subdev operation handler.
> >>>
> >>>> +{
> >>>> +	struct v4l2_mbus_framefmt *fmt =
> >>>> +		v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
> >>>> +
> >>>> +	fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>> +	fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>> +	fmt->field = V4L2_FIELD_NONE;
> >>>> +	fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>> 
> >>> Initialize the colorspace fields too.
> >>>
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
> >>>> +	.init_state = mali_c55_tpg_init_state,
> >>>> +};
> >>>> +
> >>>> +static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
> >>>> +	struct v4l2_subdev *sd = &mali_c55->tpg.sd;
> >>>> +	struct v4l2_mbus_framefmt *format;
> >>>> +	struct v4l2_subdev_state *state;
> >>>> +	int vblank_def, vblank_min;
> >>>> +	int ret;
> >>>> +
> >>>> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>> +	format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>> +
> >>>> +	ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);
> >>> 
> >>> You have 3 controls.
> >>>
> >>>> +	if (ret)
> >>>> +		goto err_unlock;
> >>>> +
> >>>> +	ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
> >>>> +				&mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
> >>>> +				ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
> >>>> +				0, 3, mali_c55_tpg_test_pattern_menu);
> >>>> +
> >>>> +	/*
> >>>> +	 * We fix hblank at the minimum allowed value and control framerate
> >>>> +	 * solely through the vblank control.
> >>>> +	 */
> >>>> +	ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
> >>>> +				&mali_c55_tpg_ctrl_ops,
> >>>> +				V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
> >>>> +				MALI_C55_TPG_FIXED_HBLANK, 1,
> >>>> +				MALI_C55_TPG_FIXED_HBLANK);
> >>>> +	if (ctrls->hblank)
> >>>> +		ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
> >>>> +
> >>>> +	__mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);
> >>> 
> >>> Drop this and initialize the control with default values. You can then
> >>> update the value by calling mali_c55_tpg_update_vblank() in
> >>> mali_c55_register_tpg().
> >>>
> >>> The reason is to share the same mutex between the control handler and
> >>> the subdev active state without having to add a separate mutex in the
> >>> mali_c55_tpg structure. The simplest way to do so is to initialize the
> >>> controls first, set sd->state_lock to point to the control handler lock,
> >>> and call v4l2_subdev_init_finalize() as the last step. As a consequence,
> >>> you can't access the active state when initializing controls.
> >>>
> >>> You can alternatively keep the lock in mali_c55_tpg and set
> >>> sd->state_lock to point to it, but I think that's more complex.
> >>>
> >>>> +	ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
> >>>> +					  &mali_c55_tpg_ctrl_ops,
> >>>> +					  V4L2_CID_VBLANK, vblank_min,
> >>>> +					  MALI_C55_TPG_MAX_VBLANK, 1,
> >>>> +					  vblank_def);
> >>>> +
> >>>> +	if (ctrls->handler.error) {
> >>>> +		dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
> >>>> +		ret = ctrls->handler.error;
> >>>> +		goto err_free_handler;
> >>>> +	}
> >>>> +
> >>>> +	ctrls->handler.lock = &mali_c55->tpg.lock;
> >>> 
> >>> Drop this and drop the mutex. The control handler will use its internal
> >>> mutex.
> >>>
> >>>> +	mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
> >>>> +
> >>>> +	v4l2_subdev_unlock_state(state);
> >>>> +
> >>>> +	return 0;
> >>>> +
> >>>> +err_free_handler:
> >>>> +	v4l2_ctrl_handler_free(&ctrls->handler);
> >>>> +err_unlock:
> >>>> +	v4l2_subdev_unlock_state(state);
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
> >>>> +	struct v4l2_subdev *sd = &tpg->sd;
> >>>> +	struct media_pad *pad = &tpg->pad;
> >>>> +	int ret;
> >>>> +
> >>>> +	mutex_init(&tpg->lock);
> >>>> +
> >>>> +	v4l2_subdev_init(sd, &mali_c55_tpg_ops);
> >>>> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> >>>> +	sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
> >>> 
> >>> Should we introduce a TPG function ?
> >> 
> >> Hmmm I vacillate a bit. I don't see that it would hurt right now, but on the other hand I think
> >> there's some value in pretending they're sensors - on the grounds that they should be handled
> >> identically as much as possible. I'd be quite wary if we ever saw "if (sd->entity.function ==
> >> MEDIA_ENT_F_TPG)" somewhere.
> > 
> > The TPG doesn't expose all the API elements we expect from raw sensors,
> > so we'll have userspace code that is TPG-specific. I suppose it will
> > look for th TPG entity by name, so we don't need a separate function.
> > The other part to consider is generic code looking for a raw sensor that
> > would pick the TPG by mistake. I'm not sure how big of a risk that would
> > be.
> 
> That'll always be a risk though, unless they specifically search by
> function, which isn't guaranteed...we could provide a parameter to
> prevent the TPG from being enumerated if that was a worry?

I see pros and cons for a separate function and can't really decide. I
suppose we can ignore it for now.

> >>>> +	sd->internal_ops = &mali_c55_tpg_internal_ops;
> >>>> +	strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
> >>>> +
> >>>> +	pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;
> >>> 
> >>> I don't think MEDIA_PAD_FL_MUST_CONNECT is right.
> >> 
> >> The docs say "If this flag is set and the pad is linked to any other pad, then at least one of those
> >> links must be enabled for the entity to be able to stream.", and that's the case here right?
> > 
> > The MEDIA_PAD_FL_MUST_CONNECT flag indicates that at least one link must
> > be enabled for that pad in order for the entity to be included in a
> > pipeline. The intent is to check in the core if, for instance, a CSI-2
> > RX input is connected to something, and refuse streaming if it isn't.
> >
> > The TPG output needs to be connected to something for the TPG to be used
> > in a pipeline, but the TPG won't be in the pipeline in the first place
> > if its output is not connected. The flag is therefore not necessary.
> 
> Okedokey - thanks
> 
> >>>> +	ret = media_entity_pads_init(&sd->entity, 1, pad);
> >>>> +	if (ret) {
> >>>> +		dev_err(mali_c55->dev,
> >>>> +			"Failed to initialize media entity pads\n");
> >>>> +		goto err_destroy_mutex;
> >>>> +	}
> >>>> +
> >>> 
> >>> 	sd->state_lock = sd->ctrl_handler->lock;
> >>>
> >>> to use the same lock for the controls and the active state. You need to
> >>> move this line and the v4l2_subdev_init_finalize() call after
> >>> mali_c55_tpg_init_controls() to get the control handler lock initialized
> >>> first.
> >>>
> >>>> +	ret = v4l2_subdev_init_finalize(sd);
> >>>> +	if (ret)
> >>>> +		goto err_cleanup_media_entity;
> >>>> +
> >>>> +	ret = mali_c55_tpg_init_controls(mali_c55);
> >>>> +	if (ret) {
> >>>> +		dev_err(mali_c55->dev,
> >>>> +			"Error initialising controls\n");
> >>>> +		goto err_cleanup_subdev;
> >>>> +	}
> >>>> +
> >>>> +	ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >>>> +	if (ret) {
> >>>> +		dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
> >>>> +		goto err_free_ctrl_handler;
> >>>> +	}
> >>>> +
> >>>> +	/*
> >>>> +	 * By default the colour settings lead to a very dim image that is
> >>>> +	 * nearly indistinguishable from black on some monitor settings. Ramp
> >>>> +	 * them up a bit so the image is brighter.
> >>>> +	 */
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
> >>>> +		       MALI_C55_TPG_BACKGROUND_MAX);
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
> >>>> +		       MALI_C55_TPG_BACKGROUND_MAX);
> >>>> +	mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
> >>>> +		       MALI_C55_TPG_BACKGROUND_MAX);
> >>>> +
> >>>> +	tpg->mali_c55 = mali_c55;
> >>>> +
> >>>> +	return 0;
> >>>> +
> >>>> +err_free_ctrl_handler:
> >>>> +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> >>>> +err_cleanup_subdev:
> >>>> +	v4l2_subdev_cleanup(sd);
> >>>> +err_cleanup_media_entity:
> >>>> +	media_entity_cleanup(&sd->entity);
> >>>> +err_destroy_mutex:
> >>>> +	mutex_destroy(&tpg->lock);
> >>>> +
> >>>> +	return ret;
> >>>> +}
> >>>> +
> >>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
> >>>> +{
> >>>> +	struct mali_c55_tpg *tpg = &mali_c55->tpg;
> >>>> +
> >>>> +	if (!tpg->mali_c55)
> >>>> +		return;
> >>>> +
> >>>> +	v4l2_device_unregister_subdev(&tpg->sd);
> >>>> +	v4l2_subdev_cleanup(&tpg->sd);
> >>>> +	media_entity_cleanup(&tpg->sd.entity);
> >>>> +	v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> >>> 
> >>> Free the control handler just after v4l2_device_unregister_subdev() to
> >>> match the order in mali_c55_register_tpg().
> >>>
> >>>> +	mutex_destroy(&tpg->lock);
> >>>> +}

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Laurent Pinchart]

Hi Dan,

On Fri, Jun 21, 2024 at 11:42:59AM +0100, Daniel Scally wrote:
> On 20/06/2024 16:23, Laurent Pinchart wrote:
> > On Thu, Jun 20, 2024 at 03:49:23PM +0100, Daniel Scally wrote:
> >> On 20/06/2024 15:33, Dan Scally wrote:
> >>> On 30/05/2024 22:43, Laurent Pinchart wrote:
> >>>> And now the second part of the review, addressing mali-c55-capture.c and
> >>>> mali-c55-resizer.c. I've reviewed the code from the bottom up, so some
> >>>> messages may be repeated in an order that seems weird. Sorry about that.
> >>>>
> >>>> On Thu, May 30, 2024 at 03:15:10AM +0300, Laurent Pinchart wrote:
> >>>>> On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
> >>>>>> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
> >>>>>> V4L2 and Media Controller compliant and creates subdevices to manage
> >>>>>> the ISP itself, its internal test pattern generator as well as the
> >>>>>> crop, scaler and output format functionality for each of its two
> >>>>>> output devices.
> >>>>>>
> >>>>>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
> >>>>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >>>>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >>>>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> >>>>>> ---
> >>>>>> Changes in v5:
> >>>>>>
> >>>>>>      - Reworked input formats - previously we allowed representing input data
> >>>>>>        as any 8-16 bit format. Now we only allow input data to be represented
> >>>>>>        by the new 20-bit bayer formats, which is corrected to the equivalent
> >>>>>>        16-bit format in RAW bypass mode.
> >>>>>>      - Stopped bypassing blocks that we haven't added supporting parameters
> >>>>>>        for yet.
> >>>>>>      - Addressed most of Sakari's comments from the list
> >>>>>>
> >>>>>> Changes not yet made in v5:
> >>>>>>
> >>>>>>      - The output pipelines can still be started and stopped independently of
> >>>>>>        one another - I'd like to discuss that more.
> >>>>>>      - the TPG subdev still uses .s_stream() - I need to rebase onto a tree
> >>>>>>        with working .enable_streams() for a single-source-pad subdevice.
> >>>>>>
> >>>>>> Changes in v4:
> >>>>>>
> >>>>>>      - Reworked mali_c55_update_bits() to internally perform the bit-shift
> >>>>> 
> >>>>> I really don't like that, it makes the code very confusing, even more so
> >>>>> as it differs from regmap_update_bits().
> >>>>>
> >>>>> Look at this for instance:
> >>>>>
> >>>>>      mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>>>                   MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> >>>>>                   MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> >>>>>
> >>>>> It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
> >>>>> BIT(0).
> >>>>>
> >>>>> Sorry, I know it will be painful, but this change needs to be reverted.
> >>>>>
> >>>>>>      - Reworked the resizer to allow cropping during streaming
> >>>>>>      - Fixed a bug in NV12 output
> >>>>>>
> >>>>>> Changes in v3:
> >>>>>>
> >>>>>>      - Mostly minor fixes suggested by Sakari
> >>>>>>      - Fixed the sequencing of vb2 buffers to be synchronised across the two
> >>>>>>        capture devices.
> >>>>>>
> >>>>>> Changes in v2:
> >>>>>>
> >>>>>>      - Clock handling
> >>>>>>      - Fixed the warnings raised by the kernel test robot
> >>>>>>
> >>>>>>    drivers/media/platform/Kconfig                |   1 +
> >>>>>>    drivers/media/platform/Makefile               |   1 +
> >>>>>>    drivers/media/platform/arm/Kconfig            |   5 +
> >>>>>>    drivers/media/platform/arm/Makefile           |   2 +
> >>>>>>    drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
> >>>>>>    drivers/media/platform/arm/mali-c55/Makefile  |   9 +
> >>>>>>    .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
> >>>>>>    .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
> >>>>>>    .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
> >>>>>>    .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
> >>>>>>    .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
> >>>>>>    .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
> >>>>>>    .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
> >>>>>>    .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
> >>>>>>    14 files changed, 4452 insertions(+)
> >>>>>>    create mode 100644 drivers/media/platform/arm/Kconfig
> >>>>>>    create mode 100644 drivers/media/platform/arm/Makefile
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >>>>> 
> >>>>> I've skipped review of capture.c and resizer.c as I already have plenty
> >>>>> of comments for the other files, and it's getting late. I'll try to
> >>>>> review the rest tomorrow.
> >>>>>
> >>>>>> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> >>>>>> index 2d79bfc68c15..c929169766aa 100644
> >>>>>> --- a/drivers/media/platform/Kconfig
> >>>>>> +++ b/drivers/media/platform/Kconfig
> >>>>>> @@ -65,6 +65,7 @@ config VIDEO_MUX
> >>>>>>    source "drivers/media/platform/allegro-dvt/Kconfig"
> >>>>>>    source "drivers/media/platform/amlogic/Kconfig"
> >>>>>>    source "drivers/media/platform/amphion/Kconfig"
> >>>>>> +source "drivers/media/platform/arm/Kconfig"
> >>>>>>    source "drivers/media/platform/aspeed/Kconfig"
> >>>>>>    source "drivers/media/platform/atmel/Kconfig"
> >>>>>>    source "drivers/media/platform/broadcom/Kconfig"
> >>>>>> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> >>>>>> index da17301f7439..9a647abd5218 100644
> >>>>>> --- a/drivers/media/platform/Makefile
> >>>>>> +++ b/drivers/media/platform/Makefile
> >>>>>> @@ -8,6 +8,7 @@
> >>>>>>    obj-y += allegro-dvt/
> >>>>>>    obj-y += amlogic/
> >>>>>>    obj-y += amphion/
> >>>>>> +obj-y += arm/
> >>>>>>    obj-y += aspeed/
> >>>>>>    obj-y += atmel/
> >>>>>>    obj-y += broadcom/
> >>>>>> diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..4f0764c329c7
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/Kconfig
> >>>>>> @@ -0,0 +1,5 @@
> >>>>>> +# SPDX-License-Identifier: GPL-2.0-only
> >>>>>> +
> >>>>>> +comment "ARM media platform drivers"
> >>>>>> +
> >>>>>> +source "drivers/media/platform/arm/mali-c55/Kconfig"
> >>>>>> diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..8cc4918725ef
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/Makefile
> >>>>>> @@ -0,0 +1,2 @@
> >>>>>> +# SPDX-License-Identifier: GPL-2.0-only
> >>>>>> +obj-y += mali-c55/
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/Kconfig
> >>>>>> b/drivers/media/platform/arm/mali-c55/Kconfig
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..602085e28b01
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/Kconfig
> >>>>>> @@ -0,0 +1,18 @@
> >>>>>> +# SPDX-License-Identifier: GPL-2.0-only
> >>>>>> +config VIDEO_MALI_C55
> >>>>>> +    tristate "ARM Mali-C55 Image Signal Processor driver"
> >>>>>> +    depends on V4L_PLATFORM_DRIVERS
> >>>>>> +    depends on VIDEO_DEV && OF
> >>>>>> +    depends on ARCH_VEXPRESS || COMPILE_TEST
> >>>>>> +    select MEDIA_CONTROLLER
> >>>>>> +    select VIDEO_V4L2_SUBDEV_API
> >>>>>> +    select VIDEOBUF2_DMA_CONTIG
> >>>>>> +    select VIDEOBUF2_VMALLOC
> >>>>>> +    select V4L2_FWNODE
> >>>>>> +    select GENERIC_PHY_MIPI_DPHY
> >>>>> 
> >>>>> Alphabetical order ?
> >>>>>
> >>>>>> +    default n
> >>>>> 
> >>>>> That's the default, you don't have to specify ti.
> >>>>>
> >>>>>> +    help
> >>>>>> +      Enable this to support Arm's Mali-C55 Image Signal Processor.
> >>>>>> +
> >>>>>> +      To compile this driver as a module, choose M here: the module
> >>>>>> +      will be called mali-c55.
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile
> >>>>>> b/drivers/media/platform/arm/mali-c55/Makefile
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..77dcb2fbf0f4
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
> >>>>>> @@ -0,0 +1,9 @@
> >>>>>> +# SPDX-License-Identifier: GPL-2.0
> >>>>>> +
> >>>>>> +mali-c55-y := mali-c55-capture.o \
> >>>>>> +          mali-c55-core.o \
> >>>>>> +          mali-c55-isp.o \
> >>>>>> +          mali-c55-tpg.o \
> >>>>>> +          mali-c55-resizer.o
> >>>>> 
> >>>>> Alphabetical order here too.
> >>>>>
> >>>>>> +
> >>>>>> +obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..1d539ac9c498
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>>>>> @@ -0,0 +1,951 @@
> >>>>>> +// SPDX-License-Identifier: GPL-2.0
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Video capture devices
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#include <linux/cleanup.h>
> >>>>>> +#include <linux/minmax.h>
> >>>>>> +#include <linux/pm_runtime.h>
> >>>>>> +#include <linux/string.h>
> >>>>>> +#include <linux/videodev2.h>
> >>>>>> +
> >>>>>> +#include <media/v4l2-dev.h>
> >>>>>> +#include <media/v4l2-event.h>
> >>>>>> +#include <media/v4l2-ioctl.h>
> >>>>>> +#include <media/v4l2-subdev.h>
> >>>>>> +#include <media/videobuf2-core.h>
> >>>>>> +#include <media/videobuf2-dma-contig.h>
> >>>>>> +
> >>>>>> +#include "mali-c55-common.h"
> >>>>>> +#include "mali-c55-registers.h"
> >>>>>> +
> >>>>>> +static const struct mali_c55_fmt mali_c55_fmts[] = {
> >>>>>> +    /*
> >>>>>> +     * This table is missing some entries which need further work or
> >>>>>> +     * investigation:
> >>>>>> +     *
> >>>>>> +     * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
> >>>>>> +     * Base mode 5 is "Generic Data"
> >>>>>> +     * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
> >>>>>> +     * Base mode 9 seems to have no V4L2 equivalent
> >>>>>> +     * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
> >>>>>> +     * equivalent
> >>>>>> +     */
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_ARGB2101010,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
> >>>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_A2R10G10B10,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_RGB565,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
> >>>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_RGB565,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_BGR24,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
> >>>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_RGB24,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_YUYV,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_YUY2,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_UYVY,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_UYVY,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_Y210,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_Y210,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    /*
> >>>>>> +     * This is something of a hack, the ISP thinks it's running NV12M but
> >>>>>> +     * by setting uv_plane = 0 we simply discard that planes and only output
> >>>>>> +     * the Y-plane.
> >>>>>> +     */
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_GREY,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_NV12M,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_NV21M,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    /*
> >>>>>> +     * RAW uncompressed formats are all packed in 16 bpp.
> >>>>>> +     * TODO: Expand this list to encompass all possible RAW formats.
> >>>>>> +     */
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_SRGGB16,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_SRGGB16_1X16,
> >>>>>> +        },
> >>>>>> +        .is_raw = true,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_SBGGR16,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_SBGGR16_1X16,
> >>>>>> +        },
> >>>>>> +        .is_raw = true,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_SGBRG16,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_SGBRG16_1X16,
> >>>>>> +        },
> >>>>>> +        .is_raw = true,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_SGRBG16,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_SGRBG16_1X16,
> >>>>>> +        },
> >>>>>> +        .is_raw = true,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +};
> >>>>>> +
> >>>>>> +static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
> >>>>>> +                           u32 code)
> >>>>>> +{
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
> >>>>>> +        if (fmt->mbus_codes[i] == code)
> >>>>>> +            return true;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return false;
> >>>>>> +}
> >>>>>> +
> >>>>>> +bool mali_c55_format_is_raw(unsigned int mbus_code)
> >>>>>> +{
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >>>>>> +        if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
> >>>>>> +            return mali_c55_fmts[i].is_raw;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return false;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
> >>>>>> +{
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >>>>>> +        if (mali_c55_fmts[i].fourcc == pixelformat)
> >>>>>> +            return &mali_c55_fmts[i];
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * If we find no matching pixelformat, we'll just default to the first
> >>>>>> +     * one for now.
> >>>>>> +     */
> >>>>>> +
> >>>>>> +    return &mali_c55_fmts[0];
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const char * const capture_device_names[] = {
> >>>>>> +    "mali-c55 fr",
> >>>>>> +    "mali-c55 ds",
> >>>>>> +    "mali-c55 3a stats",
> >>>>>> +    "mali-c55 params",
> >>>> 
> >>>> The last two entries are not used AFAICT, neither here, nor in
> >>>> subsequent patches.
> >>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
> >>>>>> +{
> >>>>>> +    if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
> >>>>>> +        return capture_device_names[0];
> >>>>>> +
> >>>>>> +    if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
> >>>>>> +        return capture_device_names[1];
> >>>>>> +
> >>>>>> +    return "params/stat not supported yet";
> >>>>>> +}
> >>>> 
> >>>> Use cap_dev->vdev.name instead of mali_c55_cap_dev_to_name(cap_dev) and
> >>>> drop this function.
> >>>>
> >>>>>> +
> >>>>>> +static int mali_c55_link_validate(struct media_link *link)
> >>>>>> +{
> >>>>>> +    struct video_device *vdev =
> >>>>>> + media_entity_to_video_device(link->sink->entity);
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
> >>>>>> +    struct v4l2_subdev *sd =
> >>>>>> + media_entity_to_v4l2_subdev(link->source->entity);
> >>>>>> +    const struct v4l2_pix_format_mplane *pix_mp;
> >>>>>> +    const struct mali_c55_fmt *cap_fmt;
> >>>>>> +    struct v4l2_subdev_format sd_fmt = {
> >>>>>> +        .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> >>>>>> +        .pad = link->source->index,
> >>>>>> +    };
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
> >>>>>> +    if (ret)
> >>>>>> +        return ret;
> >>>>>> +
> >>>>>> +    pix_mp = &cap_dev->mode.pix_mp;
> >>>>>> +    cap_fmt = cap_dev->mode.capture_fmt;
> >>>>>> +
> >>>>>> +    if (sd_fmt.format.width != pix_mp->width ||
> >>>>>> +        sd_fmt.format.height != pix_mp->height) {
> >>>>>> +        dev_dbg(cap_dev->mali_c55->dev,
> >>>>>> +            "link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
> >>>>>> +            link->source->entity->name, link->source->index,
> >>>>>> +            link->sink->entity->name, link->sink->index,
> >>>>>> +            sd_fmt.format.width, sd_fmt.format.height,
> >>>>>> +            pix_mp->width, pix_mp->height);
> >>>>>> +        return -EPIPE;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
> >>>>>> +        dev_dbg(cap_dev->mali_c55->dev,
> >>>>>> +            "link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format
> >>>>>> %p4cc\n",
> >>>>>> +            link->source->entity->name, link->source->index,
> >>>>>> +            link->sink->entity->name, link->sink->index,
> >>>>>> +            sd_fmt.format.code, &pix_mp->pixelformat);
> >>>>>> +        return -EPIPE;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct media_entity_operations mali_c55_media_ops = {
> >>>>>> +    .link_validate = mali_c55_link_validate,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
> >>>>>> +                    unsigned int *num_planes, unsigned int sizes[],
> >>>>>> +                    struct device *alloc_devs[])
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    if (*num_planes) {
> >>>>>> +        if (*num_planes != cap_dev->mode.pix_mp.num_planes)
> >>>>>> +            return -EINVAL;
> >>>>>> +
> >>>>>> +        for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >>>>>> +            if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
> >>>>>> +                return -EINVAL;
> >>>>>> +    } else {
> >>>>>> +        *num_planes = cap_dev->mode.pix_mp.num_planes;
> >>>>>> +        for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >>>>>> +            sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_buf_queue(struct vb2_buffer *vb)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
> >>>>>> +    struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> >>>>>> +    struct mali_c55_buffer *buf = container_of(vbuf,
> >>>>>> +                           struct mali_c55_buffer, vb);
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    buf->plane_done[MALI_C55_PLANE_Y] = false;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * If we're in a single-plane format we flag the other plane as done
> >>>>>> +     * already so it's dequeued appropriately later
> >>>>>> +     */
> >>>>>> +    buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
> >>>>>> +
> >>>>>> +    for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
> >>>>>> +        unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
> >>>>>> +
> >>>>>> +        vb2_set_plane_payload(vb, i, size);
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    spin_lock(&cap_dev->buffers.lock);
> >>>>>> +    list_add_tail(&buf->queue, &cap_dev->buffers.queue);
> >>>>>> +    spin_unlock(&cap_dev->buffers.lock);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_buf_init(struct vb2_buffer *vb)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
> >>>>>> +    struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> >>>>>> +    struct mali_c55_buffer *buf = container_of(vbuf,
> >>>>>> +                           struct mali_c55_buffer, vb);
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >>>>>> +        buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>>>> +
> >>>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
> >>>>>> +
> >>>>>> +    cap_dev->buffers.curr = cap_dev->buffers.next;
> >>>>>> +    cap_dev->buffers.next = NULL;
> >>>>>> +
> >>>>>> +    if (!list_empty(&cap_dev->buffers.queue)) {
> >>>>>> +        struct v4l2_pix_format_mplane *pix_mp;
> >>>>>> +        const struct v4l2_format_info *info;
> >>>>>> +        u32 *addrs;
> >>>>>> +
> >>>>>> +        pix_mp = &cap_dev->mode.pix_mp;
> >>>>>> +        info = v4l2_format_info(pix_mp->pixelformat);
> >>>>>> +
> >>>>>> +        mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
> >>>>>> +        if (cap_dev->mode.capture_fmt->registers.uv_plane)
> >>>>>> +            mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
> >>>>>> +
> >>>>>> +        cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
> >>>>>> +                             struct mali_c55_buffer,
> >>>>>> +                             queue);
> >>>>>> +        list_del(&cap_dev->buffers.next->queue);
> >>>>>> +
> >>>>>> +        addrs = cap_dev->buffers.next->addrs;
> >>>>>> +        mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
> >>>>>> +            addrs[MALI_C55_PLANE_Y]);
> >>>>>> +        mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
> >>>>>> +            addrs[MALI_C55_PLANE_UV]);
> >>>>>> +        mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
> >>>>>> +            pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
> >>>>>> +        mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
> >>>>>> +            pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
> >>>>>> +            / info->hdiv);
> >>>>>> +    } else {
> >>>>>> +        /*
> >>>>>> +         * If we underflow then we can tell the ISP that we don't want
> >>>>>> +         * to write out the next frame.
> >>>>>> +         */
> >>>>>> +        mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>>>> +        mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>>>> +    }
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
> >>>>>> +                   unsigned int framecount)
> >>>>>> +{
> >>>>>> +    curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> >>>>>> +    curr_buf->vb.field = V4L2_FIELD_NONE;
> >>>> 
> >>>> The could be set already when the buffer is queued.
> >>>>
> >>>>>> +    curr_buf->vb.sequence = framecount;
> >>>>>> +    vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
> >>>>>> +}
> >>>>>> +
> >>>>>> +/**
> >>>>>> + * mali_c55_set_plane_done - mark the plane as written and process the buffer if
> >>>>>> + *                 both planes are finished.
> >>>>>> + * @cap_dev:  pointer to the fr or ds pipe output
> >>>>>> + * @plane:    the plane to mark as completed
> >>>>>> + *
> >>>>>> + * The Mali C55 ISP has muliplanar outputs for some formats that come with two
> >>>>>> + * separate "buffer write completed" interrupts - we need to flag each plane's
> >>>>>> + * completion and check whether both planes are done - if so, complete the buf
> >>>>>> + * in vb2.
> >>>>>> + */
> >>>>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> >>>>>> +                 enum mali_c55_planes plane)
> >>>>>> +{
> >>>>>> +    struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
> >>>>>> +    struct mali_c55_buffer *curr_buf;
> >>>>>> +
> >>>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
> >>>>>> +    curr_buf = cap_dev->buffers.curr;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * This _should_ never happen. If no buffer was available from vb2 then
> >>>>>> +     * we tell the ISP not to bother writing the next frame, which means the
> >>>>>> +     * interrupts that call this function should never trigger. If it does
> >>>>>> +     * happen then one of our assumptions is horribly wrong - complain
> >>>>>> +     * loudly and do nothing.
> >>>>>> +     */
> >>>>>> +    if (!curr_buf) {
> >>>>>> +        dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
> >>>>>> +            mali_c55_cap_dev_to_name(cap_dev), __func__);
> >>>>>> +        return;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* If the other plane is also done... */
> >>>>>> +    if (curr_buf->plane_done[~plane & 1]) {
> >>>>>> +        mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> >>>>>> +        cap_dev->buffers.curr = NULL;
> >>>>>> +        isp->frame_sequence++;
> >>>>>> +    } else {
> >>>>>> +        curr_buf->plane_done[plane] = true;
> >>>>>> +    }
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>>>> +
> >>>>>> +    mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                 MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>>>> +    mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                 MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * The Mali ISP can hold up to 5 buffer addresses and simply cycle
> >>>>>> +     * through them, but it's not clear to me that the vb2 queue _guarantees_
> >>>>>> +     * it will queue buffers to the driver in a fixed order, and ensuring
> >>>>>> +     * we call vb2_buffer_done() for the right buffer seems to me to add
> >>>>>> +     * pointless complexity given in multi-context mode we'd need to
> >>>>>> +     * re-write those registers every frame anyway...so we tell the ISP to
> >>>>>> +     * use a single register and update it for each frame.
> >>>>>> +     */
> >>>> 
> >>>> A single register sounds prone to error conditions. Is it at least
> >>>> shadowed in the hardware, or do you have to make sure you reprogram it
> >>>> during the vertical blanking only ?
> >>> 
> >>> It would have to be reprogrammed during the vertical blanking if we were running in a
> >>> configuration with a single config space, otherwise you have the time it takes to process a frame
> >>> plus vertical blanking. As I say, it'll have to work like this in multi-context mode anyway.
> >>>
> >>> If we want to use the cycling...is it guaranteed that vb2 buffers will always be queued in order?
> > 
> > In which order ?
> 
> vb2_buffer.index order...

No that's not guaranteed.

> the ISP cycles through writing to the addresses (called "banks") you give 
> it automatically, so if we're initially queued 4 buffers and write their addresses to banks 0-3, the 
> ISP would expect to just keep writing to those addresses cyclically in that order. The problem is I 
> don't think that there's necessarily a guarantee that when a buffer that was returned to userspace 
> and processed is re-queued to the driver that it is queued in the order it was allocated, right? So 
> we'd need to keep track of which bank was currently being written to and when a buffer is queued, 
> populate its address to bankN+1 or N+2...or wrap around to 0.

Correct, buffer addresses would be set in banks based on the order in
which buffers are queued.

> Or perhaps we could just use two banks instead of one and alternate them, and that would at least 
> provide the shadowing functionality in the event that the Pong config wasn't fitted?

I think that could be a good first step.

For devices with two config spaces we're probably fine with a single
bank (I haven't checked in details though), but if you have a single
config space and operate the ISP inline, then things will be racy.

Maybe we can simply start by erroring out at probe time if the ISP is
inline and has a single config space, with a comment that explains what
would need to be implemented to support that configuration.

> >>>> I'll mostly skip buffer handling in this review, I need to first
> >>>> understand how the hardware operates to make an informed opinion.
> >>>>
> >>>>>> +    mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
> >>>>>> +            MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
> >>>>>> +    mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
> >>>>>> +            MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * We only queue a buffer in the streamon path if this is the first of
> >>>>>> +     * the capture devices to start streaming. If the ISP is already running
> >>>>>> +     * then we rely on the ISP_START interrupt to queue the first buffer for
> >>>>>> +     * this capture device.
> >>>>>> +     */
> >>>>>> +    if (mali_c55->pipe.start_count == 1)
> >>>>>> +        mali_c55_set_next_buffer(cap_dev);
> >>>> 
> >>>> I think we'll have to revisit buffer handling to make sure it's 100%
> >>>> race-free.
> >>>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
> >>>>>> +                        enum vb2_buffer_state state)
> >>>>>> +{
> >>>>>> +    struct mali_c55_buffer *buf, *tmp;
> >>>>>> +
> >>>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
> >>>>>> +
> >>>>>> +    if (cap_dev->buffers.curr) {
> >>>>>> + vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
> >>>>>> +                state);
> >>>>>> +        cap_dev->buffers.curr = NULL;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    if (cap_dev->buffers.next) {
> >>>>>> + vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
> >>>>>> +                state);
> >>>>>> +        cap_dev->buffers.next = NULL;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
> >>>>>> +        list_del(&buf->queue);
> >>>>>> +        vb2_buffer_done(&buf->vb.vb2_buf, state);
> >>>>>> +    }
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>>>> +    struct mali_c55_resizer *rzr = cap_dev->rzr;
> >>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    guard(mutex)(&isp->lock);
> >>>> 
> >>>> What's the reason for using the isp lock here and in
> >>>> mali_c55_vb2_stop_streaming() ? If you need a lock that covers all video
> >>>> nodes in order to synchronize start/stop, you may want to use the
> >>>> graph_mutex of the media device instead.
> >>> 
> >>> It's because I wanted to make sure that the ISP was in a known started/stopped state before
> >>> possibly trying to start/stop it, which can be done from either of the two capture devices. This
> >>> would go away if we were synchronising with the links anyway.
> > 
> > OK.
> >
> >>>>>> +
> >>>>>> +    ret = pm_runtime_resume_and_get(mali_c55->dev);
> >>>>>> +    if (ret)
> >>>>>> +        return ret;
> >>>>>> +
> >>>>>> +    ret = video_device_pipeline_start(&cap_dev->vdev,
> >>>>>> +                      &cap_dev->mali_c55->pipe);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
> >>>>>> +            mali_c55_cap_dev_to_name(cap_dev));
> >>>> 
> >>>> Drop the message or make it dev_dbg() as it can be triggered by
> >>>> userspace.
> >>>>
> >>>>>> +        goto err_pm_put;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    mali_c55_cap_dev_stream_enable(cap_dev);
> >>>>>> +    mali_c55_rzr_start_stream(rzr);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * We only start the ISP if we're the only capture device that's
> >>>>>> +     * streaming. Otherwise, it'll already be active.
> >>>>>> +     */
> >>>> 
> >>>> I still think we should use link setup to indicate which video devices
> >>>> userspace plans to use, and then only start when they're all started.
> >>>> That includes stats and parameters buffers. We can continue this
> >>>> discussion in the context of the previous version of the patch series,
> >>>> or here, up to you.
> >>> 
> >>> Let's just continue here. I think I called it "clunky" before; from my perspective it's an
> >>> unnecessary extra step - we can already signal to the driver that we don't want to use the video
> >>> devices by not queuing buffers to them or starting the stream on them and although I understand
> > 
> > By not starting streaming, perhaps, but by not queuing buffers, no. The
> > reason is that there's no synchronization between buffer queues. If you
> > queue
> >
> > Frame	FR	DS
> > --------------------
> > 1	x	x
> > 2	x
> > 3	x	x
> > 4	x	x
> >
> > it will not be distinguishable by the driver from
> >
> > Frame	FR	DS
> > --------------------
> > 1	x	x
> > 2	x	x
> > 3	x	x
> > 4	x
> >
> >>> that that means that one of the two image data capture devices will receive data before the other,
> >>> I don't understand why that's considered to be a problem. Possibly that last part is the stickler;
> >>> can you explain a bit why it's an issue for one capture queue to start earlier than the other?
> > 
> > Because from a userspace point of view, if you want to capture frames
> > from both pipelines, you will expect to receive a buffer from each
> > pipeline for every frame. If that's not guaranteed at stream start, you
> > will then need to implement synchronization code that will drop buffers
> > on one pipeline until you get the first buffer on the other pipeline
> > (assuming you can synchronize them by sequence number). That will be
> > more work, and can introduce latency.
> 
> Alright, that makes sense. I'll move to this method then...probably
> drawing on the IPU6 isys code since Sakari mentioned it worked that
> way already.
> 
> >>>>>> +    if (mali_c55->pipe.start_count == 1) {
> >>>>>> +        ret = mali_c55_isp_start_stream(isp);
> >>>>>> +        if (ret)
> >>>>>> +            goto err_disable_cap_dev;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_disable_cap_dev:
> >>>>>> +    mali_c55_cap_dev_stream_disable(cap_dev);
> >>>>>> +    video_device_pipeline_stop(&cap_dev->vdev);
> >>>>>> +err_pm_put:
> >>>>>> +    pm_runtime_put(mali_c55->dev);
> >>>>>> +    mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>>>> +    struct mali_c55_resizer *rzr = cap_dev->rzr;
> >>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
> >>>>>> +
> >>>>>> +    guard(mutex)(&isp->lock);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * If one of the other capture nodes is streaming, we shouldn't
> >>>>>> +     * disable the ISP here.
> >>>>>> +     */
> >>>>>> +    if (mali_c55->pipe.start_count == 1)
> >>>>>> +        mali_c55_isp_stop_stream(&mali_c55->isp);
> >>>>>> +
> >>>>>> +    mali_c55_rzr_stop_stream(rzr);
> >>>>>> +    mali_c55_cap_dev_stream_disable(cap_dev);
> >>>>>> +    mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
> >>>>>> +    video_device_pipeline_stop(&cap_dev->vdev);
> >>>>>> +    pm_runtime_put(mali_c55->dev);
> >>>> 
> >>>> I think runtime PM autosuspend would be very useful, as it will ensure
> >>>> that stop-reconfigure-start cycles get handled as efficiently as
> >>>> possible without powering the device down. It could be done on top as a
> >>>> separate patch.
> >>> 
> >>> Alright
> >>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct vb2_ops mali_c55_vb2_ops = {
> >>>>>> +    .queue_setup        = &mali_c55_vb2_queue_setup,
> >>>>>> +    .buf_queue        = &mali_c55_buf_queue,
> >>>>>> +    .buf_init        = &mali_c55_buf_init,
> >>>>>> +    .wait_prepare        = vb2_ops_wait_prepare,
> >>>>>> +    .wait_finish        = vb2_ops_wait_finish,
> >>>>>> +    .start_streaming    = &mali_c55_vb2_start_streaming,
> >>>>>> +    .stop_streaming        = &mali_c55_vb2_stop_streaming,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const struct v4l2_file_operations mali_c55_v4l2_fops = {
> >>>>>> +    .owner = THIS_MODULE,
> >>>>>> +    .unlocked_ioctl = video_ioctl2,
> >>>>>> +    .open = v4l2_fh_open,
> >>>>>> +    .release = vb2_fop_release,
> >>>>>> +    .poll = vb2_fop_poll,
> >>>>>> +    .mmap = vb2_fop_mmap,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
> >>>>>> +{
> >>>>>> +    const struct mali_c55_fmt *capture_format;
> >>>>>> +    const struct v4l2_format_info *info;
> >>>>>> +    struct v4l2_plane_pix_format *plane;
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
> >>>>>> +    pix_mp->pixelformat = capture_format->fourcc;
> >>>>>> +
> >>>>>> +    pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
> >>>>>> +                  MALI_C55_MAX_WIDTH);
> >>>>>> +    pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
> >>>>>> +                   MALI_C55_MAX_HEIGHT);
> >>>> 
> >>>> Ah, these clamps are right :-)
> >>> 
> >>> Hurrah!
> >>>
> >>>>>> +
> >>>>>> +    pix_mp->field = V4L2_FIELD_NONE;
> >>>>>> +    pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>>>> +    pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>>>> +    pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
> >>>>>> +
> >>>>>> +    info = v4l2_format_info(pix_mp->pixelformat);
> >>>> 
> >>>> This function may return NULL. That shouldn't be the case as long as it
> >>>> supports all formats that the C55 driver supports, so I suppose it's
> >>>> safe.
> >>>>
> >>>>>> +    pix_mp->num_planes = info->mem_planes;
> >>>>>> +    memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
> >>>>>> +
> >>>>>> +    pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;
> >>>> 
> >>>> Does the hardware support configurable line strides ? If so we should
> >>>> support it.
> >>> 
> >>> You have to set the line stride in the DMA writer registers, which we do using this same
> >>> value...might userspace have set bytesperline already then or something? Or is there some other
> >>> place it could be configured?
> > 
> > Userspace can request a specific stride by setting bytesperline, yes. If
> > that's set, you should honour it (and of course adjust it to a
> > reasonable [min, max] range as well as align it based on hardware
> > constraints).
> 
> That seems pretty weird...the bytes per pixel is a fixed value
> dependent on the format,

Correct.

> if the stride is changed from (bpp * width)
> then the width of the image won't match what was requested, so we
> couldn't honour both things at the same time. How is that supposed to
> work? Or am I misunderstanding something?

The width of the image won't change. What will change is the amount of
padding at the end of the line. While the ISP may not have specific
restrictions, other hardware blocks that would consume the frames may
have different constraints. For instance a display controller may
require all lines to be multiples of 32 bytes. If that constraint isn't
met by the image width, padding must be added at the end of each line,
and that's what the configurable stride is for.

> >>>>>> +    pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
> >>>>>> +                       * pix_mp->height;
> >>>> 
> >>>>      pix_mp->plane_fmt[0].sizeimage = pix_mp->plane_fmt[0].bytesperline
> >>>>                         * pix_mp->height;
> >>>>
> >>>>>> +
> >>>>>> +    for (i = 1; i < info->comp_planes; i++) {
> >>>>>> +        plane = &pix_mp->plane_fmt[i];
> >>>>>> +
> >>>>>> +        plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
> >>>>>> +                           info->hdiv);
> >>>>>> +        plane->sizeimage = DIV_ROUND_UP(
> >>>>>> +                    plane->bytesperline * pix_mp->height,
> >>>>>> +                    info->vdiv);
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    if (info->mem_planes == 1) {
> >>>>>> +        for (i = 1; i < info->comp_planes; i++) {
> >>>>>> +            plane = &pix_mp->plane_fmt[i];
> >>>>>> +            pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
> >>>>>> +        }
> >>>>>> +    }
> >>>> 
> >>>> I'm wondering, could v4l2_fill_pixfmt_mp() help ? It doesn't support
> >>>> configurable strides though :-S Maybe the helper could be improved, if
> >>>> it's close enough to what you need ?
> >>> 
> >>> I'll take a look
> >>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>>>> +                       struct v4l2_format *f)
> >>>>>> +{
> >>>>>> +    mali_c55_try_fmt(&f->fmt.pix_mp);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
> >>>>>> +                struct v4l2_pix_format_mplane *pix_mp)
> >>>>>> +{
> >>>>>> +    const struct mali_c55_fmt *capture_format;
> >>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>>>> +    const struct v4l2_format_info *info;
> >>>>>> +
> >>>>>> +    mali_c55_try_fmt(pix_mp);
> >>>>>> +    capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
> >>>>>> +    info = v4l2_format_info(pix_mp->pixelformat);
> >>>>>> +    if (WARN_ON(!info))
> >>>>>> +        return;
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +               capture_format->registers.base_mode);
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
> >>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
> >>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
> >>>> 
> >>>> Could the register writes be moved to stream start time ?
> >> 
> >> Sorry missed this one. These are writes to the context's registers
> >> buffer, not to the hardware. Does it matter that they're not done at
> >> stream on time?
> > 
> > Writing them here means you'll have to call pm_runtime_resume_and_get()
> > here. If power is then cut off, registers may or may not lose their
> > contents, so you would need to write them at stream on time anyway. I
> > think it's best to move all the hardware configuration at stream on
> > time.
> >
> >>>>>> +
> >>>>>> +    if (info->mem_planes > 1) {
> >>>>>> +        mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                   capture_format->registers.base_mode);
> >>>>>> +        mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_WRITER_SUBMODE_MASK,
> >>>>>> +                capture_format->registers.uv_plane);
> >>>>>> +
> >>>>>> +        mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
> >>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
> >>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
> >>>>>> +        /*
> >>>>>> +         * TODO: Figure out the colour matrix coefficients and calculate
> >>>>>> +         * and write them here.
> >>>>>> +         */
> >>>> 
> >>>> Ideally they should also be exposed directly to userspace as ISP
> >>>> parameters. I would probably go as far as saying that they should come
> >>>> directly from userspace, and not derived from the colorspace fields.
> >>> 
> >>> Yes I think I agree, I'll drop the todo from here.
> >>>
> >>>>>> +
> >>>>>> +        mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>>>> +                   MALI_C55_CS_CONV_MATRIX_MASK);
> >>>>>> +
> >>>>>> +        if (info->hdiv > 1)
> >>>>>> +            mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
> >>>>>> +        if (info->vdiv > 1)
> >>>>>> +            mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
> >>>>>> +        if (info->hdiv > 1 || info->vdiv > 1)
> >>>>>> +            mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_CS_CONV_FILTER_MASK, 0x01);
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    cap_dev->mode.pix_mp = *pix_mp;
> >>>>>> +    cap_dev->mode.capture_fmt = capture_format;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>>>> +                     struct v4l2_format *f)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >>>>>> +
> >>>>>> +    if (vb2_is_busy(&cap_dev->queue))
> >>>>>> +        return -EBUSY;
> >>>>>> +
> >>>>>> +    mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>>>> +                     struct v4l2_format *f)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >>>>>> +
> >>>>>> +    f->fmt.pix_mp = cap_dev->mode.pix_mp;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>>>> +                        struct v4l2_fmtdesc *f)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >>>>>> +    unsigned int j = 0;
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >>>>>> +        if (f->mbus_code &&
> >>>>>> + !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
> >>>>>> +                               f->mbus_code))
> >>>> 
> >>>> Small indentation mistake.
> >>>>
> >>>>>> +            continue;
> >>>>>> +
> >>>>>> +        /* Downscale pipe can't output RAW formats */
> >>>>>> +        if (mali_c55_fmts[i].is_raw &&
> >>>>>> +            cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
> >>>>>> +            continue;
> >>>>>> +
> >>>>>> +        if (j++ == f->index) {
> >>>>>> +            f->pixelformat = mali_c55_fmts[i].fourcc;
> >>>>>> +            return 0;
> >>>>>> +        }
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return -EINVAL;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_querycap(struct file *file, void *fh,
> >>>>>> +                 struct v4l2_capability *cap)
> >>>>>> +{
> >>>>>> +    strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
> >>>>>> +    strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
> >>>>>> +    .vidioc_reqbufs = vb2_ioctl_reqbufs,
> >>>>>> +    .vidioc_querybuf = vb2_ioctl_querybuf,
> >>>>>> +    .vidioc_create_bufs = vb2_ioctl_create_bufs,
> >>>>>> +    .vidioc_qbuf = vb2_ioctl_qbuf,
> >>>>>> +    .vidioc_expbuf = vb2_ioctl_expbuf,
> >>>>>> +    .vidioc_dqbuf = vb2_ioctl_dqbuf,
> >>>>>> +    .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> >>>>>> +    .vidioc_streamon = vb2_ioctl_streamon,
> >>>>>> +    .vidioc_streamoff = vb2_ioctl_streamoff,
> >>>>>> +    .vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
> >>>>>> +    .vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
> >>>>>> +    .vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
> >>>>>> +    .vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
> >>>>>> +    .vidioc_querycap = mali_c55_querycap,
> >>>>>> +    .vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
> >>>>>> +    .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
> >>>>>> +};
> >>>>>> +
> >>>>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct v4l2_pix_format_mplane pix_mp;
> >>>>>> +    struct mali_c55_cap_dev *cap_dev;
> >>>>>> +    struct video_device *vdev;
> >>>>>> +    struct vb2_queue *vb2q;
> >>>>>> +    unsigned int i;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
> >>>> 
> >>>> Moving the inner content to a separate mali_c55_register_capture_dev()
> >>>> function would increase readability I think, and remove usage of gotos.
> >>>> I would probably do the same for unregistration too, for consistency.
> >>>>
> >>>>>> +        cap_dev = &mali_c55->cap_devs[i];
> >>>>>> +        vdev = &cap_dev->vdev;
> >>>>>> +        vb2q = &cap_dev->queue;
> >>>>>> +
> >>>>>> +        /*
> >>>>>> +         * The downscale output pipe is an optional block within the ISP
> >>>>>> +         * so we need to check whether it's actually been fitted or not.
> >>>>>> +         */
> >>>>>> +
> >>>>>> +        if (i == MALI_C55_CAP_DEV_DS &&
> >>>>>> +            !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
> >>>>>> +            continue;
> >>>> 
> >>>> Given that there's only two capture devices, and one is optional, when
> >>>> moving the inner code to a separate function you could unroll the loop.
> >>>> Up to you.
> >>>>
> >>>>>> +
> >>>>>> +        cap_dev->mali_c55 = mali_c55;
> >>>>>> +        mutex_init(&cap_dev->lock);
> >>>>>> +        INIT_LIST_HEAD(&cap_dev->buffers.queue);
> >>>>>> +
> >>>>>> +        switch (i) {
> >>>>>> +        case MALI_C55_CAP_DEV_FR:
> >>>>>> +            cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
> >>>>>> +            cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
> >>>>>> +            break;
> >>>>>> +        case MALI_C55_CAP_DEV_DS:
> >>>>>> +            cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
> >>>>>> +            cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
> >>>>>> +            break;
> >>>>>> +        default:
> >>>> 
> >>>> That can't happen.
> >>>>
> >>>>>> + mutex_destroy(&cap_dev->lock);
> >>>>>> +            ret = -EINVAL;
> >>>>>> +            goto err_destroy_mutex;
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
> >>>>>> +        ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
> >>>>>> +        if (ret) {
> >>>>>> +            mutex_destroy(&cap_dev->lock);
> >>>>>> +            goto err_destroy_mutex;
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> >>>>>> +        vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
> >>>>>> +        vb2q->drv_priv = cap_dev;
> >>>>>> +        vb2q->mem_ops = &vb2_dma_contig_memops;
> >>>>>> +        vb2q->ops = &mali_c55_vb2_ops;
> >>>>>> +        vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
> >>>>>> +        vb2q->min_queued_buffers = 1;
> >>>>>> +        vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> >>>>>> +        vb2q->lock = &cap_dev->lock;
> >>>>>> +        vb2q->dev = mali_c55->dev;
> >>>>>> +
> >>>>>> +        ret = vb2_queue_init(vb2q);
> >>>>>> +        if (ret) {
> >>>>>> +            dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
> >>>>>> +                mali_c55_cap_dev_to_name(cap_dev));
> >>>>>> +            goto err_cleanup_media_entity;
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        strscpy(cap_dev->vdev.name, capture_device_names[i],
> >>>>>> +            sizeof(cap_dev->vdev.name));
> >>>>>> +        vdev->release = video_device_release_empty;
> >>>>>> +        vdev->fops = &mali_c55_v4l2_fops;
> >>>>>> +        vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
> >>>>>> +        vdev->lock = &cap_dev->lock;
> >>>>>> +        vdev->v4l2_dev = &mali_c55->v4l2_dev;
> >>>>>> +        vdev->queue = &cap_dev->queue;
> >>>>>> +        vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> >>>>>> +                    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
> >>>>>> +        vdev->entity.ops = &mali_c55_media_ops;
> >>>>>> +        video_set_drvdata(vdev, cap_dev);
> >>>>>> +
> >>>>>> +        memset(&pix_mp, 0, sizeof(pix_mp));
> >>>>>> +        pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
> >>>>>> +        pix_mp.width = MALI_C55_DEFAULT_WIDTH;
> >>>>>> +        pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
> >>>>>> +        mali_c55_set_format(cap_dev, &pix_mp);
> >>>>>> +
> >>>>>> +        ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
> >>>>>> +        if (ret) {
> >>>>>> +            dev_err(mali_c55->dev,
> >>>>>> +                "%s failed to register video device\n",
> >>>>>> +                mali_c55_cap_dev_to_name(cap_dev));
> >>>>>> +            goto err_release_vb2q;
> >>>>>> +        }
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_release_vb2q:
> >>>>>> +    vb2_queue_release(vb2q);
> >>>>>> +err_cleanup_media_entity:
> >>>>>> +    media_entity_cleanup(&cap_dev->vdev.entity);
> >>>>>> +err_destroy_mutex:
> >>>>>> +    mutex_destroy(&cap_dev->lock);
> >>>>>> +    mali_c55_unregister_capture_devs(mali_c55);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev;
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
> >>>>>> +        cap_dev = &mali_c55->cap_devs[i];
> >>>>>> +
> >>>>>> +        if (!video_is_registered(&cap_dev->vdev))
> >>>>>> +            continue;
> >>>>>> +
> >>>>>> +        vb2_video_unregister_device(&cap_dev->vdev);
> >>>>>> +        media_entity_cleanup(&cap_dev->vdev.entity);
> >>>>>> +        mutex_destroy(&cap_dev->lock);
> >>>>>> +    }
> >>>>>> +}
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..2d0c4d152beb
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>>>> @@ -0,0 +1,266 @@
> >>>>>> +/* SPDX-License-Identifier: GPL-2.0 */
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Common definitions
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#ifndef _MALI_C55_COMMON_H
> >>>>>> +#define _MALI_C55_COMMON_H
> >>>>>> +
> >>>>>> +#include <linux/clk.h>
> >>>>>> +#include <linux/io.h>
> >>>>>> +#include <linux/list.h>
> >>>>>> +#include <linux/mutex.h>
> >>>>>> +#include <linux/scatterlist.h>
> >>>>> 
> >>>>> I don't think this is needed. You're however missing spinlock.h.
> >>>>>
> >>>>>> +#include <linux/videodev2.h>
> >>>>>> +
> >>>>>> +#include <media/media-device.h>
> >>>>>> +#include <media/v4l2-async.h>
> >>>>>> +#include <media/v4l2-ctrls.h>
> >>>>>> +#include <media/v4l2-dev.h>
> >>>>>> +#include <media/v4l2-device.h>
> >>>>>> +#include <media/v4l2-subdev.h>
> >>>>>> +#include <media/videobuf2-core.h>
> >>>>>> +#include <media/videobuf2-v4l2.h>
> >>>>>> +
> >>>>>> +#define MALI_C55_DRIVER_NAME        "mali-c55"
> >>>>>> +
> >>>>>> +/* min and max values for the image sizes */
> >>>>>> +#define MALI_C55_MIN_WIDTH        640U
> >>>>>> +#define MALI_C55_MIN_HEIGHT        480U
> >>>>>> +#define MALI_C55_MAX_WIDTH        8192U
> >>>>>> +#define MALI_C55_MAX_HEIGHT        8192U
> >>>>>> +#define MALI_C55_DEFAULT_WIDTH        1920U
> >>>>>> +#define MALI_C55_DEFAULT_HEIGHT        1080U
> >>>>>> +
> >>>>>> +#define MALI_C55_DEFAULT_MEDIA_BUS_FMT MEDIA_BUS_FMT_RGB121212_1X36
> >>>>>> +
> >>>>>> +struct mali_c55;
> >>>>>> +struct mali_c55_cap_dev;
> >>>>>> +struct platform_device;
> >>>>> 
> >>>>> You should also forward-declare
> >>>>>
> >>>>> struct device;
> >>>>> struct dma_chan;
> >>>>> struct resource;
> >>>>>
> >>>>>> +
> >>>>>> +static const char * const mali_c55_clk_names[] = {
> >>>>>> +    "aclk",
> >>>>>> +    "hclk",
> >>>>>> +};
> >>>>> 
> >>>>> This will end up duplicating the array in each compilation unit, not
> >>>>> great. Move it to mali-c55-core.c. You use it in this file just for its
> >>>>> size, replace that with a macro that defines the size, or allocate
> >>>>> mali_c55.clks dynamically with devm_kcalloc().
> >>>>>
> >>>>>> +
> >>>>>> +enum mali_c55_interrupts {
> >>>>>> +    MALI_C55_IRQ_ISP_START,
> >>>>>> +    MALI_C55_IRQ_ISP_DONE,
> >>>>>> +    MALI_C55_IRQ_MCM_ERROR,
> >>>>>> +    MALI_C55_IRQ_BROKEN_FRAME_ERROR,
> >>>>>> +    MALI_C55_IRQ_MET_AF_DONE,
> >>>>>> +    MALI_C55_IRQ_MET_AEXP_DONE,
> >>>>>> +    MALI_C55_IRQ_MET_AWB_DONE,
> >>>>>> +    MALI_C55_IRQ_AEXP_1024_DONE,
> >>>>>> +    MALI_C55_IRQ_IRIDIX_MET_DONE,
> >>>>>> +    MALI_C55_IRQ_LUT_INIT_DONE,
> >>>>>> +    MALI_C55_IRQ_FR_Y_DONE,
> >>>>>> +    MALI_C55_IRQ_FR_UV_DONE,
> >>>>>> +    MALI_C55_IRQ_DS_Y_DONE,
> >>>>>> +    MALI_C55_IRQ_DS_UV_DONE,
> >>>>>> +    MALI_C55_IRQ_LINEARIZATION_DONE,
> >>>>>> +    MALI_C55_IRQ_RAW_FRONTEND_DONE,
> >>>>>> +    MALI_C55_IRQ_NOISE_REDUCTION_DONE,
> >>>>>> +    MALI_C55_IRQ_IRIDIX_DONE,
> >>>>>> +    MALI_C55_IRQ_BAYER2RGB_DONE,
> >>>>>> +    MALI_C55_IRQ_WATCHDOG_TIMER,
> >>>>>> +    MALI_C55_IRQ_FRAME_COLLISION,
> >>>>>> +    MALI_C55_IRQ_UNUSED,
> >>>>>> +    MALI_C55_IRQ_DMA_ERROR,
> >>>>>> +    MALI_C55_IRQ_INPUT_STOPPED,
> >>>>>> +    MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
> >>>>>> +    MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
> >>>>>> +    MALI_C55_NUM_IRQ_BITS
> >>>>> 
> >>>>> Those are register bits, I think they belong to mali-c55-registers.h,
> >>>>> and should probably be macros instead of an enum.
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +enum mali_c55_isp_pads {
> >>>>>> +    MALI_C55_ISP_PAD_SINK_VIDEO,
> >>>>> 
> >>>>> As there's a single sink pad, maybe MALI_C55_ISP_PAD_SINK ? Ah, you're
> >>>>> probably preparing for ISP parameters support. It's fine.
> >>>>>
> >>>>>> +    MALI_C55_ISP_PAD_SOURCE,
> >>>>> 
> >>>>> Then maybe this should be named MALI_C55_ISP_PAD_SOURCE_VIDEO as I
> >>>>> assume there will be a stats source pad.
> >>>>>
> >>>>>> +    MALI_C55_ISP_PAD_SOURCE_BYPASS,
> >>>>>> +    MALI_C55_ISP_NUM_PADS,
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_tpg {
> >>>>>> +    struct mali_c55 *mali_c55;
> >>>>>> +    struct v4l2_subdev sd;
> >>>>>> +    struct media_pad pad;
> >>>>>> +    struct mutex lock;
> >>>>>> +    struct mali_c55_tpg_ctrls {
> >>>>>> +        struct v4l2_ctrl_handler handler;
> >>>>>> +        struct v4l2_ctrl *test_pattern;
> >>>>> 
> >>>>> Set but never used. You can drop it.
> >>>>>
> >>>>>> +        struct v4l2_ctrl *hblank;
> >>>>> 
> >>>>> Set and used only once, in the same function. You can make it a local
> >>>>> variable.
> >>>>>
> >>>>>> +        struct v4l2_ctrl *vblank;
> >>>>>> +    } ctrls;
> >>>>>> +};
> >>>>> 
> >>>>> I wonder if this file should be split, with mali-c55-capture.h,
> >>>>> mali-c55-core.h, mali-c55-isp.h, ... I think it could increase
> >>>>> readability by clearly separating the different elements. Up to you.
> >>>>>
> >>>>>> +
> >>>>>> +struct mali_c55_isp {
> >>>>>> +    struct mali_c55 *mali_c55;
> >>>>>> +    struct v4l2_subdev sd;
> >>>>>> +    struct media_pad pads[MALI_C55_ISP_NUM_PADS];
> >>>>>> +    struct media_pad *remote_src;
> >>>>>> +    struct v4l2_async_notifier notifier;
> >>>>> 
> >>>>> I'm tempted to move the notifier to mali_c55, as it's related to
> >>>>> components external to the whole ISP, not to the ISP subdev itself.
> >>>>> Could you give it a try, to see if it could be done without any drawback
> >>>>> ?
> >>>>>
> >>>>>> +    struct mutex lock;
> >>>>> 
> >>>>> Locks require a comment to explain what they protect. Same below where
> >>>>> applicable (for both mutexes and spinlocks).
> >>>>>
> >>>>>> +    unsigned int frame_sequence;
> >>>>>> +};
> >>>>>> +
> >>>>>> +enum mali_c55_resizer_ids {
> >>>>>> +    MALI_C55_RZR_FR,
> >>>>>> +    MALI_C55_RZR_DS,
> >>>>>> +    MALI_C55_NUM_RZRS,
> >>>>> 
> >>>>> The usual abbreviation for "resizer" in drivers/media/ is "rsz", not
> >>>>> "rzr". I would have said we can leave it as-is as changing it would be a
> >>>>> bit annoying, but I then realized that "rzr" is not just unusual, it's
> >>>>> actually not used at all. Would you mind applying a sed globally ? :-)
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +enum mali_c55_rzr_pads {
> >>>>> 
> >>>>> Same enums/structs use abbreviations, some don't. Consistency would
> >>>>> help.
> >>>>>
> >>>>>> +    MALI_C55_RZR_SINK_PAD,
> >>>>>> +    MALI_C55_RZR_SOURCE_PAD,
> >>>>>> +    MALI_C55_RZR_SINK_BYPASS_PAD,
> >>>>>> +    MALI_C55_RZR_NUM_PADS
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_resizer {
> >>>>>> +    struct mali_c55 *mali_c55;
> >>>>>> +    struct mali_c55_cap_dev *cap_dev;
> >>>>>> +    enum mali_c55_resizer_ids id;
> >>>>>> +    struct v4l2_subdev sd;
> >>>>>> +    struct media_pad pads[MALI_C55_RZR_NUM_PADS];
> >>>>>> +    unsigned int num_routes;
> >>>>>> +    bool streaming;
> >>>>>> +};
> >>>>>> +
> >>>>>> +enum mali_c55_cap_devs {
> >>>>>> +    MALI_C55_CAP_DEV_FR,
> >>>>>> +    MALI_C55_CAP_DEV_DS,
> >>>>>> +    MALI_C55_NUM_CAP_DEVS
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_fmt {
> >>>>> 
> >>>>> mali_c55_format_info would be a better name I think, as this stores
> >>>>> format information, not formats.
> >>>>>
> >>>>>> +    u32 fourcc;
> >>>>>> +    unsigned int mbus_codes[2];
> >>>>> 
> >>>>> A comment to explain why we have two media bus codes would be useful.
> >>>>> You can document the whole structure if desired :-)
> >>>>>
> >>>>>> +    bool is_raw;
> >>>>>> +    struct mali_c55_fmt_registers {
> >>>>> 
> >>>>> Make it an anonymous structure, it's never used anywhere else.
> >>>>>
> >>>>>> +        unsigned int base_mode;
> >>>>>> +        unsigned int uv_plane;
> >>>>> 
> >>>>> If those are register field values, use u32 instead of unsigned int.
> >>>>>
> >>>>>> +    } registers;
> >>>>> 
> >>>>> It's funny, we tend to abbreviate different things, I would have used
> >>>>> "regs" here but written "format" in full in the structure name :-)
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +enum mali_c55_isp_bayer_order {
> >>>>>> +    MALI_C55_BAYER_ORDER_RGGB,
> >>>>>> +    MALI_C55_BAYER_ORDER_GRBG,
> >>>>>> +    MALI_C55_BAYER_ORDER_GBRG,
> >>>>>> +    MALI_C55_BAYER_ORDER_BGGR
> >>>>> 
> >>>>> These are registers values too, they belong to mali-c55-registers.h.
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_isp_fmt {
> >>>>> 
> >>>>> mali_c55_isp_format_info
> >>>>>
> >>>>>> +    u32 code;
> >>>>>> +    enum v4l2_pixel_encoding encoding;
> >>>>> 
> >>>>> Here you use v4l2_pixel_encoding, for mali_c55_fmt you use is_raw. Maybe
> >>>>> pick the same option for both structures ?
> >>>>>
> >>>>>> +    enum mali_c55_isp_bayer_order order;
> >>>>>> +};
> >>>>>> +
> >>>>>> +enum mali_c55_planes {
> >>>>>> +    MALI_C55_PLANE_Y,
> >>>>>> +    MALI_C55_PLANE_UV,
> >>>>>> +    MALI_C55_NUM_PLANES
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_buffer {
> >>>>>> +    struct vb2_v4l2_buffer vb;
> >>>>>> +    bool plane_done[MALI_C55_NUM_PLANES];
> >>>>> 
> >>>>> I think tracking the pending state would simplify the logic in
> >>>>> mali_c55_set_plane_done(), which would become
> >>>>>
> >>>>>      curr_buf->plane_pending[plane] = false;
> >>>>>
> >>>>>      if (!curr_buf->plane_pending[0] && !curr_buf->plane_pending[1]) {
> >>>>>          mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> >>>>>          cap_dev->buffers.curr = NULL;
> >>>>>          isp->frame_sequence++;
> >>>>>      }
> >>>>>
> >>>>> Or a counter may be even easier (and would consume less memory).
> >>>>>
> >>>>>> +    struct list_head queue;
> >>>>>> +    u32 addrs[MALI_C55_NUM_PLANES];
> >>>>> 
> >>>>> This stores DMA addresses, use dma_addr_t.
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_cap_dev {
> >>>>>> +    struct mali_c55 *mali_c55;
> >>>>>> +    struct mali_c55_resizer *rzr;
> >>>>>> +    struct video_device vdev;
> >>>>>> +    struct media_pad pad;
> >>>>>> +    struct vb2_queue queue;
> >>>>>> +    struct mutex lock;
> >>>>>> +    unsigned int reg_offset;
> >>>>> 
> >>>>> Manual handling of the offset everywhere, with parametric macros for the
> >>>>> resizer register addresses, isn't very nice. Introduce resizer-specific
> >>>>> accessors and capture-specific accessors (mali_c55_rsz_write(), ...)
> >>>>> that take a mali_c55_resizer or mali_c55_cap_dev pointer, and handle the
> >>>>> offset there. The register macros should loose their offset parameter.
> >>>>>
> >>>>> You could also use a single set of accessors that would become
> >>>>> path/output-specific (mali_c55_path_write() ? mali_c55_output_write()
> >>>>> ?), that may make the code easier to read.
> >>>>>
> >>>>> You can also replace reg_offset with a void __iomem * base, which would
> >>>>> avoid the computation at runtime.
> >>>>>
> >>>>>> +
> >>>>>> +    struct mali_c55_mode {
> >>>>> 
> >>>>> Make the structure anonymous.
> >>>>>
> >>>>>> +        const struct mali_c55_fmt *capture_fmt;
> >>>>>> +        struct v4l2_pix_format_mplane pix_mp;
> >>>>>> +    } mode;
> >>>>> 
> >>>>> What's a "mode" ? I think I'd name this
> >>>>>
> >>>>>      struct {
> >>>>>          const struct mali_c55_fmt *info;
> >>>>>          struct v4l2_pix_format_mplane format;
> >>>>>      } format;
> >>>>>
> >>>>> Or you could just drop the structure and have
> >>>>>
> >>>>>      const struct mali_c55_fmt *format_info;
> >>>>>      struct v4l2_pix_format_mplane format;
> >>>>>
> >>>>> or something similar.
> >>>>>
> >>>>>> +
> >>>>>> +    struct {
> >>>>>> +        spinlock_t lock;
> >>>>>> +        struct list_head queue;
> >>>>>> +        struct mali_c55_buffer *curr;
> >>>>>> +        struct mali_c55_buffer *next;
> >>>>>> +    } buffers;
> >>>>>> +
> >>>>>> +    bool streaming;
> >>>>>> +};
> >>>>>> +
> >>>>>> +enum mali_c55_config_spaces {
> >>>>>> +    MALI_C55_CONFIG_PING,
> >>>>>> +    MALI_C55_CONFIG_PONG,
> >>>>>> +    MALI_C55_NUM_CONFIG_SPACES
> >>>>> 
> >>>>> The last enumerator is not used.
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_ctx {
> >>>>> 
> >>>>> mali_c55_context ?
> >>>>>
> >>>>>> +    struct mali_c55 *mali_c55;
> >>>>>> +    void *registers;
> >>>>> 
> >>>>> Please document this structure and explain that this field points to a
> >>>>> copy of the register space in system memory, I was about to write you're
> >>>>> missing __iomem :-)
> >>>>>
> >>>>>> +    phys_addr_t base;
> >>>>>> +    spinlock_t lock;
> >>>>>> +    struct list_head list;
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55 {
> >>>>>> +    struct device *dev;
> >>>>>> +    struct resource *res;
> >>>>> 
> >>>>> You could possibly drop this field by passing the physical address of
> >>>>> the register space from mali_c55_probe() to mali_c55_init_context() as a
> >>>>> function parameter.
> >>>>>
> >>>>>> +    void __iomem *base;
> >>>>>> +    struct dma_chan *channel;
> >>>>>> +    struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
> >>>>>> +
> >>>>>> +    u16 capabilities;
> >>>>>> +    struct media_device media_dev;
> >>>>>> +    struct v4l2_device v4l2_dev;
> >>>>>> +    struct media_pipeline pipe;
> >>>>>> +
> >>>>>> +    struct mali_c55_tpg tpg;
> >>>>>> +    struct mali_c55_isp isp;
> >>>>>> +    struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
> >>>>>> +    struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
> >>>>>> +
> >>>>>> +    struct list_head contexts;
> >>>>>> +    enum mali_c55_config_spaces next_config;
> >>>>>> +};
> >>>>>> +
> >>>>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
> >>>>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
> >>>>>> +          bool force_hardware);
> >>>>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
> >>>>>> +              u32 mask, u32 val);
> >>>>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
> >>>>>> +              enum mali_c55_config_spaces cfg_space);
> >>>>>> +
> >>>>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55);
> >>>>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55);
> >>>>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
> >>>>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
> >>>>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55);
> >>>>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
> >>>>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
> >>>>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
> >>>>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
> >>>>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> >>>>>> +                 enum mali_c55_planes plane);
> >>>>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
> >>>>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
> >>>>>> +
> >>>>>> +bool mali_c55_format_is_raw(unsigned int mbus_code);
> >>>>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
> >>>>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
> >>>>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
> >>>>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
> >>>>>> +
> >>>>>> +const struct mali_c55_isp_fmt *
> >>>>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
> >>>>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
> >>>>>> +#define for_each_mali_isp_fmt(fmt)\
> >>>>> 
> >>>>> #define for_each_mali_isp_fmt(fmt) \
> >>>>>
> >>>>>> +    for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
> >>>>> 
> >>>>> Looks like parentheses were on sale :-)
> >>>>>
> >>>>>      for ((fmt) = NULL; (fmt) = mali_c55_isp_fmt_next(fmt); )
> >>>>>
> >>>>> This macro is used in two places only, in the mali-c55-isp.c file where
> >>>>> open-coding the loop without using mali_c55_isp_fmt_next() would be more
> >>>>> efficient, and in mali-c55-resizer.c where a function to return format
> >>>>> i'th would be more efficient. I think you can drop the macro and the
> >>>>> mali_c55_isp_fmt_next() function.
> >>>>>
> >>>>>> +
> >>>>>> +#endif /* _MALI_C55_COMMON_H */
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..50caf5ee7474
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>>>> @@ -0,0 +1,767 @@
> >>>>>> +// SPDX-License-Identifier: GPL-2.0
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Core driver code
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#include <linux/bitops.h>
> >>>>>> +#include <linux/cleanup.h>
> >>>>>> +#include <linux/clk.h>
> >>>>>> +#include <linux/delay.h>
> >>>>>> +#include <linux/device.h>
> >>>>>> +#include <linux/dmaengine.h>
> >>>>>> +#include <linux/dma-mapping.h>
> >>>>>> +#include <linux/interrupt.h>
> >>>>>> +#include <linux/iopoll.h>
> >>>>>> +#include <linux/ioport.h>
> >>>>>> +#include <linux/mod_devicetable.h>
> >>>>>> +#include <linux/of.h>
> >>>>>> +#include <linux/of_reserved_mem.h>
> >>>>>> +#include <linux/platform_device.h>
> >>>>>> +#include <linux/pm_runtime.h>
> >>>>>> +#include <linux/scatterlist.h>
> >>>>> 
> >>>>> I don't think this is needed.
> >>>>>
> >>>>> Missing slab.h.
> >>>>>
> >>>>>> +#include <linux/string.h>
> >>>>>> +
> >>>>>> +#include <media/media-entity.h>
> >>>>>> +#include <media/v4l2-device.h>
> >>>>>> +#include <media/videobuf2-dma-contig.h>
> >>>>>> +
> >>>>>> +#include "mali-c55-common.h"
> >>>>>> +#include "mali-c55-registers.h"
> >>>>>> +
> >>>>>> +static const char * const mali_c55_interrupt_names[] = {
> >>>>>> +    [MALI_C55_IRQ_ISP_START] = "ISP start",
> >>>>>> +    [MALI_C55_IRQ_ISP_DONE] = "ISP done",
> >>>>>> +    [MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
> >>>>>> +    [MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
> >>>>>> +    [MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
> >>>>>> +    [MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
> >>>>>> +    [MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
> >>>>>> +    [MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
> >>>>>> +    [MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
> >>>>>> +    [MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
> >>>>>> +    [MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
> >>>>>> +    [MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
> >>>>>> +    [MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
> >>>>>> +    [MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
> >>>>>> +    [MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
> >>>>>> +    [MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
> >>>>>> +    [MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
> >>>>>> +    [MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
> >>>>>> +    [MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
> >>>>>> +    [MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
> >>>>>> +    [MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
> >>>>>> +    [MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
> >>>>>> +    [MALI_C55_IRQ_DMA_ERROR] = "DMA error",
> >>>>>> +    [MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
> >>>>>> +    [MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
> >>>>>> +    [MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
> >>>>>> +};
> >>>>>> +
> >>>>>> +static unsigned int config_space_addrs[] = {
> >>>>> 
> >>>>> const
> >>>>>
> >>>>>> +    [MALI_C55_CONFIG_PING] = 0x0AB6C,
> >>>>>> +    [MALI_C55_CONFIG_PONG] = 0x22B2C,
> >>>>> 
> >>>>> Lowercase hex constants.
> >>>>>
> >>>>> Don't the values belong to mali-c55-registers.h ?
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +/* System IO
> >>>>> 
> >>>>> /*
> >>>>>    * System IO
> >>>>>
> >>>>>> + *
> >>>>>> + * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
> >>>>>> + * and 'pong'), with the  expectation that the 'active' space will be left
> >>>>> 
> >>>>> s/the  /the /
> >>>>>
> >>>>>> + * untouched whilst a frame is being processed and the 'inactive' space
> >>>>>> + * configured ready to be passed during the blanking period before the next
> >>>>> 
> >>>>> s/to be passed/to be switched to/ ?
> >>>>>
> >>>>>> + * frame processing starts. These spaces should ideally be set via DMA transfer
> >>>>>> + * from a buffer rather than through individual register set operations. There
> >>>>>> + * is also a shared global register space which should be set normally. Of
> >>>>>> + * course, the ISP might be included in a system which lacks a suitable DMA
> >>>>>> + * engine, and the second configuration space might not be fitted at all, which
> >>>>>> + * means we need to support four scenarios:
> >>>>>> + *
> >>>>>> + * 1. Multi config space, with DMA engine.
> >>>>>> + * 2. Multi config space, no DMA engine.
> >>>>>> + * 3. Single config space, with DMA engine.
> >>>>>> + * 4. Single config space, no DMA engine.
> >>>>>> + *
> >>>>>> + * The first case is very easy, but the rest present annoying problems. The best
> >>>>>> + * way to solve them seems to be simply to replicate the concept of DMAing over
> >>>>>> + * the configuration buffer even if there's no DMA engine on the board, for
> >>>>>> + * which we rely on memcpy. To facilitate this any read/write call that is made
> >>>>>> + * to an address within those config spaces should infact be directed to a
> >>>>>> + * buffer that was allocated to hold them rather than the IO memory itself. The
> >>>>>> + * actual copy of that buffer to IO mem will happen on interrupt.
> >>>>>> + */
> >>>>>> +
> >>>>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
> >>>>>> +{
> >>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>>>> +
> >>>>>> +    if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
> >>>>>> +        spin_lock(&ctx->lock);
> >>>>>> +        addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
> >>>>>> +        ((u32 *)ctx->registers)[addr] = val;
> >>>>>> +        spin_unlock(&ctx->lock);
> >>>>>> +
> >>>>>> +        return;
> >>>>>> +    }
> >>>>> 
> >>>>> Ouch. This is likely the second comment you really won't like (after the
> >>>>> comment regarding mali_c55_update_bits() at the very top). I apologize
> >>>>> in advance.
> >>>>>
> >>>>> I really don't like this. Directing writes either to hardware registers
> >>>>> or to the shadow registers in the context makes the callers of the
> >>>>> read/write accessors very hard to read. The probe code, for instance,
> >>>>> mixes writes to hardware registers and writes to the context shadow
> >>>>> registers to initialize the value of some of the shadow registers.
> >>>>>
> >>>>> I'd like to split the read/write accessors into functions that access
> >>>>> the hardware registers (that's easy) and functions that access the
> >>>>> shadow registers. I think the latter should receive a mali_c55_ctx
> >>>>> pointer instead of a mali_c55 pointer to prepare for multi-context
> >>>>> support.
> >>>>>
> >>>>> You can add WARN_ON() guards to the two sets of functions, to ensure
> >>>>> that no register from the "other" space gets passed to the wrong
> >>>>> function by mistake.
> >>>>>
> >>>>>> +
> >>>>>> +    writel(val, mali_c55->base + addr);
> >>>>>> +}
> >>>>>> +
> >>>>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
> >>>>>> +          bool force_hardware)
> >>>>> 
> >>>>> force_hardware is never set to true.
> >>>>>
> >>>>>> +{
> >>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>>>> +    u32 val;
> >>>>>> +
> >>>>>> +    if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
> >>>>>> +        spin_lock(&ctx->lock);
> >>>>>> +        addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
> >>>>>> +        val = ((u32 *)ctx->registers)[addr];
> >>>>>> +        spin_unlock(&ctx->lock);
> >>>>>> +
> >>>>>> +        return val;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return readl(mali_c55->base + addr);
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
> >>>>>> +              u32 mask, u32 val)
> >>>>>> +{
> >>>>>> +    u32 orig, tmp;
> >>>>>> +
> >>>>>> +    orig = mali_c55_read(mali_c55, addr, false);
> >>>>>> +
> >>>>>> +    tmp = orig & ~mask;
> >>>>>> +    tmp |= (val << (ffs(mask) - 1)) & mask;
> >>>>>> +
> >>>>>> +    if (tmp != orig)
> >>>>>> +        mali_c55_write(mali_c55, addr, tmp);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
> >>>>>> +                 dma_addr_t dst, enum dma_data_direction dir)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>>>> +    struct dma_async_tx_descriptor *tx;
> >>>>>> +    enum dma_status status;
> >>>>>> +    dma_cookie_t cookie;
> >>>>>> +
> >>>>>> +    tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
> >>>>>> +                       MALI_C55_CONFIG_SPACE_SIZE, 0);
> >>>>>> +    if (!tx) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
> >>>>>> +        return -EIO;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    cookie = dmaengine_submit(tx);
> >>>>>> +    if (dma_submit_error(cookie)) {
> >>>>>> +        dev_err(mali_c55->dev, "error submitting dma transfer\n");
> >>>>>> +        return -EIO;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    status = dma_sync_wait(mali_c55->channel, cookie);
> >>>>> 
> >>>>> I've just realized this performs a busy-wait :-S See the comment in the
> >>>>> probe function about the threaded IRQ handler. I think we'll need to
> >>>>> rework all this. It could be done on top though.
> >>>>>
> >>>>>> +    if (status != DMA_COMPLETE) {
> >>>>>> +        dev_err(mali_c55->dev, "dma transfer failed\n");
> >>>>>> +        return -EIO;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
> >>>>>> +                 enum mali_c55_config_spaces cfg_space)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>>>> +    struct device *dma_dev = mali_c55->channel->device->dev;
> >>>>>> +    dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
> >>>>>> +    dma_addr_t dst;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    guard(spinlock)(&ctx->lock);
> >>>>>> +
> >>>>>> +    dst = dma_map_single(dma_dev, ctx->registers,
> >>>>>> +                 MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
> >>>>>> +    if (dma_mapping_error(dma_dev, dst)) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to map DMA addr\n");
> >>>>>> +        return -EIO;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
> >>>>>> +    dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
> >>>>>> +             DMA_FROM_DEVICE);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
> >>>>>> +               enum mali_c55_config_spaces cfg_space)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>>>> +    struct device *dma_dev = mali_c55->channel->device->dev;
> >>>>>> +    dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
> >>>>>> +    dma_addr_t src;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    guard(spinlock)(&ctx->lock);
> >>>>> 
> >>>>> The code below can take a large amount of time, holding a spinlock will
> >>>>> disable interrupts on the local CPU, that's not good :-(
> >>>>>
> >>>>>> +
> >>>>>> +    src = dma_map_single(dma_dev, ctx->registers,
> >>>>>> +                 MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
> >>>>>> +    if (dma_mapping_error(dma_dev, src)) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to map DMA addr\n");
> >>>>>> +        return -EIO;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
> >>>>>> +    dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
> >>>>>> +             DMA_TO_DEVICE);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_config_read(struct mali_c55_ctx *ctx,
> >>>>>> +                enum mali_c55_config_spaces cfg_space)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>>>> +
> >>>>>> +    if (mali_c55->channel) {
> >>>>>> +        return mali_c55_dma_read(ctx, cfg_space);
> >>>>> 
> >>>>> As this function is used at probe time only, to initialize the context,
> >>>>> I think DMA is overkill.
> >>>>>
> >>>>>> +    } else {
> >>>>>> +        memcpy_fromio(ctx->registers,
> >>>>>> +                  mali_c55->base + config_space_addrs[cfg_space],
> >>>>>> +                  MALI_C55_CONFIG_SPACE_SIZE);
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>>> +}
> >>>>>> +
> >>>>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
> >>>>>> +              enum mali_c55_config_spaces cfg_space)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>>>> +
> >>>>>> +    if (mali_c55->channel) {
> >>>>>> +        return mali_c55_dma_write(ctx, cfg_space);
> >>>>>> +    } else {
> >>>>>> +        memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
> >>>>>> +                ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>> 
> >>>>> Could you measure the time it typically takes to write the registers
> >>>>> using DMA compared to using memcpy_toio() ?
> >>>>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);
> >>>>> 
> >>>>> I think it's too early to tell how multi-context support will look like.
> >>>>> I'm fine keeping mali_c55_get_active_context() as changing that would be
> >>>>> very intrusive (even if I think it will need to be changed), but the
> >>>>> list of contexts is neither the mechanism we'll use, nor something we
> >>>>> need now. Drop the list, embed the context in struct mali_c55, and
> >>>>> return the pointer to that single context from this function.
> >>>>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_remove_links(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> + media_entity_remove_links(&mali_c55->tpg.sd.entity);
> >>>>>> + media_entity_remove_links(&mali_c55->isp.sd.entity);
> >>>>>> +
> >>>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; i++)
> >>>>>> + media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
> >>>>>> +
> >>>>>> +    for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
> >>>>>> + media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_create_links(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    /* Test pattern generator to ISP */
> >>>>>> +    ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
> >>>>>> +                    &mali_c55->isp.sd.entity,
> >>>>>> +                    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
> >>>>>> +        goto err_remove_links;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* Full resolution resizer pipe. */
> >>>>>> +    ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >>>>>> +            MALI_C55_ISP_PAD_SOURCE,
> >>>>>> + &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
> >>>>>> +            MALI_C55_RZR_SINK_PAD,
> >>>>>> +            MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
> >>>>>> +        goto err_remove_links;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* Full resolution bypass. */
> >>>>>> +    ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >>>>>> +                    MALI_C55_ISP_PAD_SOURCE_BYPASS,
> >>>>>> + &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
> >>>>>> +                    MALI_C55_RZR_SINK_BYPASS_PAD,
> >>>>>> +                    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
> >>>>>> +        goto err_remove_links;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* Resizer pipe to video capture nodes. */
> >>>>>> +    ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
> >>>>>> +            MALI_C55_RZR_SOURCE_PAD,
> >>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
> >>>>>> +            0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev,
> >>>>>> +            "failed to link FR resizer and video device\n");
> >>>>>> +        goto err_remove_links;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* The downscale pipe is an optional hardware block */
> >>>>>> +    if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
> >>>>>> +        ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >>>>>> +            MALI_C55_ISP_PAD_SOURCE,
> >>>>>> + &mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
> >>>>>> +            MALI_C55_RZR_SINK_PAD,
> >>>>>> +            MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>>>> +        if (ret) {
> >>>>>> +            dev_err(mali_c55->dev,
> >>>>>> +                "failed to link ISP and DS resizer\n");
> >>>>>> +            goto err_remove_links;
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
> >>>>>> +            MALI_C55_RZR_SOURCE_PAD,
> >>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
> >>>>>> +            0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>>>> +        if (ret) {
> >>>>>> +            dev_err(mali_c55->dev,
> >>>>>> +                "failed to link DS resizer and video device\n");
> >>>>>> +            goto err_remove_links;
> >>>>>> +        }
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_remove_links:
> >>>>>> +    mali_c55_remove_links(mali_c55);
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    mali_c55_unregister_tpg(mali_c55);
> >>>>>> +    mali_c55_unregister_isp(mali_c55);
> >>>>>> +    mali_c55_unregister_resizers(mali_c55);
> >>>>>> +    mali_c55_unregister_capture_devs(mali_c55);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_register_entities(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    ret = mali_c55_register_tpg(mali_c55);
> >>>>>> +    if (ret)
> >>>>>> +        return ret;
> >>>>>> +
> >>>>>> +    ret = mali_c55_register_isp(mali_c55);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_unregister_entities;
> >>>>>> +
> >>>>>> +    ret = mali_c55_register_resizers(mali_c55);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_unregister_entities;
> >>>>>> +
> >>>>>> +    ret = mali_c55_register_capture_devs(mali_c55);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_unregister_entities;
> >>>>>> +
> >>>>>> +    ret = mali_c55_create_links(mali_c55);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_unregister_entities;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_unregister_entities:
> >>>>>> +    mali_c55_unregister_entities(mali_c55);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    u32 product, version, revision, capabilities;
> >>>>>> +
> >>>>>> +    product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
> >>>>>> +    version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
> >>>>>> +    revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
> >>>>>> +
> >>>>>> +    dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
> >>>>>> +         product, version, revision);
> >>>>>> +
> >>>>>> +    capabilities = mali_c55_read(mali_c55,
> >>>>>> +                     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
> >>>>>> +                     false);
> >>>>>> +    mali_c55->capabilities = (capabilities & 0xffff);
> >>>>>> +
> >>>>>> +    /* TODO: Might as well start some debugfs */
> >>>>> 
> >>>>> If it's just to expose the version and capabilities, I think that's
> >>>>> overkill. It's not needed for debug purpose (you can get it from the
> >>>>> kernel log already). debugfs isn't meant to be accessible in production,
> >>>>> so an application that would need access to the information wouldn't be
> >>>>> able to use it.
> >>>>>
> >>>>>> +    dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);
> >>>>> 
> >>>>> Combine the two messages into one.
> >>>>>
> >>>>>> +    return version;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>>>> +    u32 curr_config, next_config;
> >>>>>> +
> >>>>>> +    curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
> >>>>>> +    curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
> >>>>>> +              >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
> >>>>>> +    next_config = curr_config ^ 1;
> >>>>>> +
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>>>> +                 MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
> >>>>>> +    mali_c55_config_write(ctx, next_config ?
> >>>>>> +                  MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static irqreturn_t mali_c55_isr(int irq, void *context)
> >>>>>> +{
> >>>>>> +    struct device *dev = context;
> >>>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >>>>>> +    u32 interrupt_status;
> >>>>>> +    unsigned int i, j;
> >>>>>> +
> >>>>>> +    interrupt_status = mali_c55_read(mali_c55,
> >>>>>> +                     MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
> >>>>>> +                     false);
> >>>>>> +    if (!interrupt_status)
> >>>>>> +        return IRQ_NONE;
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
> >>>>>> +               interrupt_status);
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
> >>>>>> +
> >>>>>> +    for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
> >>>>>> +        if (!(interrupt_status & (1 << i)))
> >>>>>> +            continue;
> >>>>>> +
> >>>>>> +        switch (i) {
> >>>>>> +        case MALI_C55_IRQ_ISP_START:
> >>>>>> +            mali_c55_isp_queue_event_sof(mali_c55);
> >>>>>> +
> >>>>>> +            for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
> >>>>>> + mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
> >>>>>> +
> >>>>>> +            mali_c55_swap_next_config(mali_c55);
> >>>>>> +
> >>>>>> +            break;
> >>>>>> +        case MALI_C55_IRQ_ISP_DONE:
> >>>>>> +            /*
> >>>>>> +             * TODO: Where the ISP has no Pong config fitted, we'd
> >>>>>> +             * have to do the mali_c55_swap_next_config() call here.
> >>>>>> +             */
> >>>>>> +            break;
> >>>>>> +        case MALI_C55_IRQ_FR_Y_DONE:
> >>>>>> +            mali_c55_set_plane_done(
> >>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
> >>>>>> +                MALI_C55_PLANE_Y);
> >>>>>> +            break;
> >>>>>> +        case MALI_C55_IRQ_FR_UV_DONE:
> >>>>>> +            mali_c55_set_plane_done(
> >>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
> >>>>>> +                MALI_C55_PLANE_UV);
> >>>>>> +            break;
> >>>>>> +        case MALI_C55_IRQ_DS_Y_DONE:
> >>>>>> +            mali_c55_set_plane_done(
> >>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
> >>>>>> +                MALI_C55_PLANE_Y);
> >>>>>> +            break;
> >>>>>> +        case MALI_C55_IRQ_DS_UV_DONE:
> >>>>>> +            mali_c55_set_plane_done(
> >>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
> >>>>>> +                MALI_C55_PLANE_UV);
> >>>>>> +            break;
> >>>>>> +        default:
> >>>>>> +            /*
> >>>>>> +             * Only the above interrupts are currently unmasked. If
> >>>>>> +             * we receive anything else here then something weird
> >>>>>> +             * has gone on.
> >>>>>> +             */
> >>>>>> +            dev_err(dev, "masked interrupt %s triggered\n",
> >>>>>> +                mali_c55_interrupt_names[i]);
> >>>>>> +        }
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return IRQ_HANDLED;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_init_context(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_ctx *ctx;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
> >>>>>> +    if (!ctx) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to allocate new context\n");
> >>>>> 
> >>>>> No need for an error message when memory allocation fails.
> >>>>>
> >>>>>> +        return -ENOMEM;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    ctx->base = mali_c55->res->start;
> >>>>>> +    ctx->mali_c55 = mali_c55;
> >>>>>> +
> >>>>>> +    ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
> >>>>>> +                 GFP_KERNEL | GFP_DMA);
> >>>>>> +    if (!ctx->registers) {
> >>>>>> +        ret = -ENOMEM;
> >>>>>> +        goto err_free_ctx;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * The allocated memory is empty, we need to load the default
> >>>>>> +     * register settings. We just read Ping; it's identical to Pong.
> >>>>>> +     */
> >>>>>> +    ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_free_registers;
> >>>>>> +
> >>>>>> +    list_add_tail(&ctx->list, &mali_c55->contexts);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * Some features of the ISP need to be disabled by default and only
> >>>>>> +     * enabled at the same time as they're configured by a parameters buffer
> >>>>>> +     */
> >>>>>> +
> >>>>>> +    /* Bypass the sqrt and square compression and expansion modules */
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
> >>>>>> +                 MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
> >>>>>> +                 MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
> >>>>>> +
> >>>>>> +    /* Bypass the temper module */
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
> >>>>>> +               MALI_C55_REG_BYPASS_2_TEMPER);
> >>>>>> +
> >>>>>> +    /* Bypass the colour noise reduction  */
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
> >>>>>> +               MALI_C55_REG_BYPASS_4_CNR);
> >>>>>> +
> >>>>>> +    /* Disable the sinter module */
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
> >>>>>> +                 MALI_C55_SINTER_ENABLE_MASK, 0x00);
> >>>>>> +
> >>>>>> +    /* Disable the RGB Gamma module for each output */
> >>>>>> +    mali_c55_write(
> >>>>>> +        mali_c55,
> >>>>>> + MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
> >>>>>> +        0x00);
> >>>>>> +    mali_c55_write(
> >>>>>> +        mali_c55,
> >>>>>> + MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
> >>>>>> +        0x00);
> >>>>>> +
> >>>>>> +    /* Disable the colour correction matrix */
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_free_registers:
> >>>>>> +    kfree(ctx->registers);
> >>>>>> +err_free_ctx:
> >>>>>> +    kfree(ctx);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_runtime_resume(struct device *dev)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
> >>>>>> +                      mali_c55->clks);
> >>>>>> +    if (ret)
> >>>>>> +        dev_err(mali_c55->dev, "failed to enable clocks\n");
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_runtime_suspend(struct device *dev)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >>>>>> +
> >>>>>> + clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct dev_pm_ops mali_c55_pm_ops = {
> >>>>>> +    SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> >>>>>> +                pm_runtime_force_resume)
> >>>>>> +    SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
> >>>>>> +               NULL)
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_probe(struct platform_device *pdev)
> >>>>>> +{
> >>>>>> +    struct device *dev = &pdev->dev;
> >>>>>> +    struct mali_c55 *mali_c55;
> >>>>>> +    dma_cap_mask_t mask;
> >>>>>> +    u32 version;
> >>>>>> +    int ret;
> >>>>>> +    u32 val;
> >>>>>> +
> >>>>>> +    mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
> >>>>>> +    if (!mali_c55)
> >>>>>> +        return dev_err_probe(dev, -ENOMEM,
> >>>>>> +                     "failed to allocate memory\n");
> >>>>> 
> >>>>>          return -ENOMEM;
> >>>>>
> >>>>> There's no need to print messages for memory allocation failures.
> >>>>>
> >>>>>> +
> >>>>>> +    mali_c55->dev = dev;
> >>>>>> +    platform_set_drvdata(pdev, mali_c55);
> >>>>>> +
> >>>>>> +    mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
> >>>>>> +                                &mali_c55->res);
> >>>>>> +    if (IS_ERR(mali_c55->base))
> >>>>>> +        return dev_err_probe(dev, PTR_ERR(mali_c55->base),
> >>>>>> +                     "failed to map IO memory\n");
> >>>>>> +
> >>>>>> +    ret = platform_get_irq(pdev, 0);
> >>>>>> +    if (ret < 0)
> >>>>>> +        return dev_err_probe(dev, ret, "failed to get interrupt num\n");
> >>>>> 
> >>>>> s/ num// or s/num/number/
> >>>>>
> >>>>>> +
> >>>>>> +    ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
> >>>>>> +                    mali_c55_isr, IRQF_ONESHOT,
> >>>>>> +                    dev_driver_string(&pdev->dev),
> >>>>>> +                    &pdev->dev);
> >>>>> 
> >>>>> Requested the IRQ should be done much lower, after you have initialized
> >>>>> everything, or an IRQ that would fire early would have really bad
> >>>>> consequences.
> >>>>>
> >>>>> A comment to explain why you need a threaded interrupt handler would be
> >>>>> good. I assume it is due only to the need to transfer the registers
> >>>>> using DMA. I wonder if we should then split the interrupt handler in
> >>>>> two, with a non-threaded part for the operations that can run quickly,
> >>>>> and a threaded part for the reprogramming.
> >>>>>
> >>>>> It may also be that we could just start the DMA transfer in the
> >>>>> non-threaded handler without waiting synchronously for it to complete.
> >>>>> That would be a bigger change, and would require checking race
> >>>>> conditions carefully. On the other hand, I'm a bit concerned about the
> >>>>> current implementation, have you tested what happens if the DMA transfer
> >>>>> takes too long to complete, and spans frame boundaries ? This part could
> >>>>> be addressed by a patch on top of this one.
> >>>>>
> >>>>>> +    if (ret)
> >>>>>> +        return dev_err_probe(dev, ret, "failed to request irq\n");
> >>>>>> +
> >>>>>> +    for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
> >>>>>> +        mali_c55->clks[i].id = mali_c55_clk_names[i];
> >>>>>> +
> >>>>>> +    ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
> >>>>>> +    if (ret)
> >>>>>> +        return dev_err_probe(dev, ret, "failed to acquire clocks\n");
> >>>>>> +
> >>>>>> +    pm_runtime_enable(&pdev->dev);
> >>>>>> +
> >>>>>> +    ret = pm_runtime_resume_and_get(&pdev->dev);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_pm_runtime_disable;
> >>>>>> +
> >>>>>> +    of_reserved_mem_device_init(dev);
> >>>>> 
> >>>>> I'd move this, the vb2_dma_contig_set_max_seg_size() call and the
> >>>>> dma_cap_* calls before pm_runtime_enable() as they don't need the device
> >>>>> to be powered.
> >>>>>
> >>>>>> +    version = mali_c55_check_hwcfg(mali_c55);
> >>>>>> +    vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
> >>>>>> +
> >>>>>> +    /* Use "software only" context management. */
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>>>> +                 MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> >>>>>> +                 MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> >>>>> 
> >>>>> You handle that in mali_c55_isp_start(), does the register have to be
> >>>>> set here too ?
> >>>>>
> >>>>>> +
> >>>>>> +    dma_cap_zero(mask);
> >>>>>> +    dma_cap_set(DMA_MEMCPY, mask);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * No error check, because we will just fallback on memcpy if there is
> >>>>>> +     * no usable DMA channel on the system.
> >>>>>> +     */
> >>>>>> +    mali_c55->channel = dma_request_channel(mask, NULL, NULL);
> >>>>>> +
> >>>>>> +    INIT_LIST_HEAD(&mali_c55->contexts);
> >>>>>> +    ret = mali_c55_init_context(mali_c55);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_release_dma_channel;
> >>>>>> +
> >>>>> 
> >>>>> I'd move all the code from here ...
> >>>>>
> >>>>>> +    mali_c55->media_dev.dev = dev;
> >>>>>> +    strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
> >>>>>> +        sizeof(mali_c55->media_dev.model));
> >>>>>> +    mali_c55->media_dev.hw_revision = version;
> >>>>>> +
> >>>>>> +    media_device_init(&mali_c55->media_dev);
> >>>>>> +    ret = media_device_register(&mali_c55->media_dev);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_cleanup_media_device;
> >>>>>> +
> >>>>>> +    mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
> >>>>>> +    ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(dev, "failed to register V4L2 device\n");
> >>>>>> +        goto err_unregister_media_device;
> >>>>>> +    };
> >>>>>> +
> >>>>>> +    ret = mali_c55_register_entities(mali_c55);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(dev, "failed to register entities\n");
> >>>>>> +        goto err_unregister_v4l2_device;
> >>>>>> +    }
> >>>>> 
> >>>>> ... to here to a separate function, or maybe fold it all in
> >>>>> mali_c55_register_entities() (which should the be renamed). Same thing
> >>>>> for the cleanup code.
> >>>>>
> >>>>>> +
> >>>>>> +    /* Set safe stop to ensure we're in a non-streaming state */
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> >>>>>> +               MALI_C55_INPUT_SAFE_STOP);
> >>>>>> +    readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >>>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * We're ready to process interrupts. Clear any that are set and then
> >>>>>> +     * unmask them for processing.
> >>>>>> +     */
> >>>>>> +    mali_c55_write(mali_c55, 0x30, 0xffffffff);
> >>>>>> +    mali_c55_write(mali_c55, 0x34, 0xffffffff);
> >>>>>> +    mali_c55_write(mali_c55, 0x40, 0x01);
> >>>>>> +    mali_c55_write(mali_c55, 0x40, 0x00);
> >>>>> 
> >>>>> Please replace the register addresses with macros.
> >>>>>
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);
> >>>>> 
> >>>>> The value should use the interrupt bits macros.
> >>>>>
> >>>>>> +
> >>>>>> +    pm_runtime_put(&pdev->dev);
> >>>>> 
> >>>>> Once power gets cut, the registers your programmed above may be lost. I
> >>>>> think you should programe them in the runtime PM resume handler.
> >>>>>
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_unregister_v4l2_device:
> >>>>>> +    v4l2_device_unregister(&mali_c55->v4l2_dev);
> >>>>>> +err_unregister_media_device:
> >>>>>> +    media_device_unregister(&mali_c55->media_dev);
> >>>>>> +err_cleanup_media_device:
> >>>>>> +    media_device_cleanup(&mali_c55->media_dev);
> >>>>>> +err_release_dma_channel:
> >>>>>> +    dma_release_channel(mali_c55->channel);
> >>>>>> +err_pm_runtime_disable:
> >>>>>> +    pm_runtime_disable(&pdev->dev);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_remove(struct platform_device *pdev)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
> >>>>>> +    struct mali_c55_ctx *ctx, *tmp;
> >>>>>> +
> >>>>>> +    list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
> >>>>>> +        list_del(&ctx->list);
> >>>>>> +        kfree(ctx->registers);
> >>>>>> +        kfree(ctx);
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    mali_c55_remove_links(mali_c55);
> >>>>>> +    mali_c55_unregister_entities(mali_c55);
> >>>>>> +    v4l2_device_put(&mali_c55->v4l2_dev);
> >>>>>> +    media_device_unregister(&mali_c55->media_dev);
> >>>>>> +    media_device_cleanup(&mali_c55->media_dev);
> >>>>>> +    dma_release_channel(mali_c55->channel);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct of_device_id mali_c55_of_match[] = {
> >>>>>> +    { .compatible = "arm,mali-c55", },
> >>>>>> +    {},
> >>>>> 
> >>>>>      { /* Sentinel */ },
> >>>>>
> >>>>>> +};
> >>>>>> +MODULE_DEVICE_TABLE(of, mali_c55_of_match);
> >>>>>> +
> >>>>>> +static struct platform_driver mali_c55_driver = {
> >>>>>> +    .driver = {
> >>>>>> +        .name = "mali-c55",
> >>>>>> +        .of_match_table = of_match_ptr(mali_c55_of_match),
> >>>>> 
> >>>>> Drop of_match_ptr().
> >>>>>
> >>>>>> +        .pm = &mali_c55_pm_ops,
> >>>>>> +    },
> >>>>>> +    .probe = mali_c55_probe,
> >>>>>> +    .remove_new = mali_c55_remove,
> >>>>>> +};
> >>>>>> +
> >>>>>> +module_platform_driver(mali_c55_driver);
> >>>>> 
> >>>>> Blank line.
> >>>>>
> >>>>>> +MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
> >>>>>> +MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
> >>>>>> +MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
> >>>>>> +MODULE_LICENSE("GPL");
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..ea8b7b866e7a
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>>>> @@ -0,0 +1,611 @@
> >>>>>> +// SPDX-License-Identifier: GPL-2.0
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Image signal processor
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#include <linux/delay.h>
> >>>>>> +#include <linux/iopoll.h>
> >>>>>> +#include <linux/property.h>
> >>>>>> +#include <linux/string.h>
> >>>>>> +
> >>>>>> +#include <media/media-entity.h>
> >>>>>> +#include <media/v4l2-common.h>
> >>>>>> +#include <media/v4l2-event.h>
> >>>>>> +#include <media/v4l2-mc.h>
> >>>>>> +#include <media/v4l2-subdev.h>
> >>>>>> +
> >>>>>> +#include "mali-c55-common.h"
> >>>>>> +#include "mali-c55-registers.h"
> >>>>>> +
> >>>>>> +static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
> >>>>>> +    {
> >>>>>> +        .code = MEDIA_BUS_FMT_SRGGB20_1X20,
> >>>>>> +        .order = MALI_C55_BAYER_ORDER_RGGB,
> >>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .code = MEDIA_BUS_FMT_SGRBG20_1X20,
> >>>>>> +        .order = MALI_C55_BAYER_ORDER_GRBG,
> >>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .code = MEDIA_BUS_FMT_SGBRG20_1X20,
> >>>>>> +        .order = MALI_C55_BAYER_ORDER_GBRG,
> >>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .code = MEDIA_BUS_FMT_SBGGR20_1X20,
> >>>>>> +        .order = MALI_C55_BAYER_ORDER_BGGR,
> >>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .code = MEDIA_BUS_FMT_RGB202020_1X60,
> >>>>>> +        .order = 0, /* Not relevant for this format */
> >>>>>> +        .encoding = V4L2_PIXEL_ENC_RGB,
> >>>>>> +    }
> >>>>>> +    /*
> >>>>>> +     * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
> >>>>>> +     * also support YUV input from a sensor passed-through to the output. At
> >>>>>> +     * present we have no mechanism to test that though so it may have to
> >>>>>> +     * wait a while...
> >>>>>> +     */
> >>>>>> +};
> >>>>>> +
> >>>>>> +const struct mali_c55_isp_fmt *
> >>>>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
> >>>>>> +{
> >>>>>> +    if (!fmt)
> >>>>>> +        fmt = &mali_c55_isp_fmts[0];
> >>>>>> +    else
> >>>>>> +        fmt++;
> >>>>>> +
> >>>>>> +    for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
> >>>>>> +        return fmt;
> >>>>> 
> >>>>> That's peculiar.
> >>>>>
> >>>>>      if (!fmt)
> >>>>>          fmt = &mali_c55_isp_fmts[0];
> >>>>>      else if (fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)])
> >>>>>          return ++fmt;
> >>>>>      else
> >>>>>          return NULL;
> >>>>>
> >>>>>> +
> >>>>>> +    return NULL;
> >>>>>> +}
> >>>>>> +
> >>>>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
> >>>>>> +{
> >>>>>> +    const struct mali_c55_isp_fmt *isp_fmt;
> >>>>>> +
> >>>>>> +    for_each_mali_isp_fmt(isp_fmt) {
> >>>>> 
> >>>>> I would open-code the loop instead of using the macro, like you do
> >>>>> below. It will be more efficient.
> >>>>>
> >>>>>> +        if (isp_fmt->code == mbus_code)
> >>>>>> +            return true;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return false;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct mali_c55_isp_fmt *
> >>>>>> +mali_c55_isp_get_mbus_config_by_code(u32 code)
> >>>>>> +{
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
> >>>>>> +        if (mali_c55_isp_fmts[i].code == code)
> >>>>>> +            return &mali_c55_isp_fmts[i];
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return NULL;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    u32 val;
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);
> >>>>> 
> >>>>>      mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> >>>>>                 MALI_C55_INPUT_SAFE_STOP);
> >>>>>
> >>>>>> + readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >>>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_isp_start(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>>>> +    const struct mali_c55_isp_fmt *cfg;
> >>>>>> +    struct v4l2_mbus_framefmt *format;
> >>>>> 
> >>>>> const
> >>>>>
> >>>>>> +    struct v4l2_subdev_state *state;
> >>>>>> +    struct v4l2_rect *crop;
> >>>>> 
> >>>>> const
> >>>>>
> >>>>>> +    struct v4l2_subdev *sd;
> >>>>>> +    u32 val;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    sd = &mali_c55->isp.sd;
> >>>>> 
> >>>>> Assign when declaring the variable.
> >>>>>
> >>>>>> +
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>>>> +                 MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
> >>>>>> +
> >>>>>> +    /* Apply input windowing */
> >>>>>> +    state = v4l2_subdev_get_locked_active_state(sd);
> >>>>> 
> >>>>> Using .enable_streams() (see below) you'll get this for free.
> >>>>>
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +    format = v4l2_subdev_state_get_format(state,
> >>>>>> +                          MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +    cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
> >>>>>> +               MALI_C55_HC_START(crop->left));
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
> >>>>>> +               MALI_C55_HC_SIZE(crop->width));
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
> >>>>>> +               MALI_C55_VC_START(crop->top) |
> >>>>>> +               MALI_C55_VC_SIZE(crop->height));
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
> >>>>>> +                 MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
> >>>>>> +                 MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
> >>>>>> +                 MALI_C55_BAYER_ORDER_MASK, cfg->order);
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
> >>>>>> +                 MALI_C55_INPUT_WIDTH_MASK,
> >>>>>> +                 MALI_C55_INPUT_WIDTH_20BIT);
> >>>>>> +
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >>>>>> +                 MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
> >>>>>> +                 cfg->encoding == V4L2_PIXEL_ENC_RGB ?
> >>>>>> +                 MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
> >>>>>> +
> >>>>>> +    ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to DMA config\n");
> >>>>>> +        return ret;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> >>>>>> +               MALI_C55_INPUT_SAFE_START);
> >>>>>> +    readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >>>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >>>>> 
> >>>>> Should you return an error in case of timeout ?
> >>>>>
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)
> >>>>> 
> >>>>> Why is this not handled wired to .s_stream() ? Or better,
> >>>>> .enable_streams() and .disable_streams().
> >>>>>
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>>>> +    struct v4l2_subdev *sd;
> >>>>>> +
> >>>>>> +    if (isp->remote_src) {
> >>>>>> +        sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
> >>>>>> +        v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
> >>>>>> +    }
> >>>>>> +    isp->remote_src = NULL;
> >>>>>> +
> >>>>>> +    mali_c55_isp_stop(mali_c55);
> >>>>>> +}
> >>>>>> +
> >>>>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>>>> +    struct media_pad *sink_pad;
> >>>>>> +    struct v4l2_subdev *sd;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
> >>>>>> +    isp->remote_src = media_pad_remote_pad_unique(sink_pad);
> >>>>>> +    if (IS_ERR(isp->remote_src)) {
> >>>>> 
> >>>>> If you set the MUST_CONNECT flag for the MALI_C55_ISP_PAD_SINK_VIDEO pad
> >>>>> I think you can drop this check.
> >>>>>
> >>>>>> +        dev_err(mali_c55->dev, "Failed to get source for ISP\n");
> >>>>>> +        return PTR_ERR(isp->remote_src);
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
> >>>>>> +
> >>>>>> +    isp->frame_sequence = 0;
> >>>>>> +    ret = mali_c55_isp_start(mali_c55);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "Failed to start ISP\n");
> >>>>>> +        isp->remote_src = NULL;
> >>>>>> +        return ret;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * We only support a single input stream, so we can just enable the 1st
> >>>>>> +     * entry in the streams mask.
> >>>>>> +     */
> >>>>>> +    ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "Failed to start ISP source\n");
> >>>>>> +        mali_c55_isp_stop(mali_c55);
> >>>>>> +        return ret;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>> +                       struct v4l2_subdev_state *state,
> >>>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
> >>>>>> +{
> >>>>>> +    /*
> >>>>>> +     * Only the internal RGB processed format is allowed on the regular
> >>>>>> +     * processing source pad.
> >>>>>> +     */
> >>>>>> +    if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
> >>>>>> +        if (code->index)
> >>>>>> +            return -EINVAL;
> >>>>>> +
> >>>>>> +        code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* On the sink and bypass pads all the supported formats are allowed. */
> >>>>>> +    if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    code->code = mali_c55_isp_fmts[code->index].code;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
> >>>>>> +                    struct v4l2_subdev_state *state,
> >>>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
> >>>>>> +{
> >>>>>> +    const struct mali_c55_isp_fmt *cfg;
> >>>>>> +
> >>>>>> +    if (fse->index > 0)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * Only the internal RGB processed format is allowed on the regular
> >>>>>> +     * processing source pad.
> >>>>>> +     *
> >>>>>> +     * On the sink and bypass pads all the supported formats are allowed.
> >>>>>> +     */
> >>>>>> +    if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
> >>>>>> +        if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
> >>>>>> +            return -EINVAL;
> >>>>>> +    } else {
> >>>>>> +        cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
> >>>>>> +        if (!cfg)
> >>>>>> +            return -EINVAL;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
> >>>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
> >>>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
> >>>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
> >>>>>> +                struct v4l2_subdev_state *state,
> >>>>>> +                struct v4l2_subdev_format *format)
> >>>>>> +{
> >>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>>>> +    struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
> >>>>>> +    const struct mali_c55_isp_fmt *cfg;
> >>>>>> +    struct v4l2_rect *crop;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * Disallow set_fmt on the source pads; format is fixed and the sizes
> >>>>>> +     * are the result of applying the sink crop rectangle to the sink
> >>>>>> +     * format.
> >>>>>> +     */
> >>>>>> +    if (format->pad)
> >>>>> 
> >>>>>      if (format->pad != MALI_C55_ISP_PAD_SINK_VIDEO)
> >>>>>
> >>>>>> +        return v4l2_subdev_get_fmt(sd, state, format);
> >>>>>> +
> >>>>>> +    cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
> >>>>>> +    if (!cfg)
> >>>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>>>> +    fmt->field = V4L2_FIELD_NONE;
> >>>>> 
> >>>>> Do you intentionally allow the colorspace fields to be overwritten to
> >>>>> any value ? rkisp1_isp_set_src_fmt() and rkisp1_isp_set_sink_fmt() may
> >>>>> show you how this could be handled.
> >>>>>
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * Clamp sizes in the accepted limits and clamp the crop rectangle in
> >>>>>> +     * the new sizes.
> >>>>>> +     */
> >>>>>> +    clamp_t(unsigned int, fmt->width,
> >>>>>> +        MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>>>>> +    clamp_t(unsigned int, fmt->width,
> >>>>>> +        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>>>> 
> >>>>> clamp_t() returns a value, which you ignore. Those are no-ops. You meant
> >>>>>
> >>>>>      fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> >>>>>                   MALI_C55_MAX_WIDTH);
> >>>>>      fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> >>>>>                    MALI_C55_MAX_HEIGHT);
> >>>>>
> >>>>> Same for every use of clamp_t() through the whole driver.
> >>>>>
> >>>>> Also, do you need clamp_t() ? I think all values are unsigned int, you
> >>>>> can use clamp().
> >>>>>
> >>>>> Are there any alignment constraints, such a multiples of two for bayer
> >>>>> formats ? Same in all the other locations where applicable.
> >>>>>
> >>>>>> +
> >>>>>> +    sink_fmt = v4l2_subdev_state_get_format(state,
> >>>>>> +                        MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +    *sink_fmt = *fmt;
> >>>>>> +
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +    crop->left = 0;
> >>>>>> +    crop->top = 0;
> >>>>>> +    crop->width = fmt->width;
> >>>>>> +    crop->height = fmt->height;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * Propagate format to source pads. On the 'regular' output pad use
> >>>>>> +     * the internal RGB processed format, while on the bypass pad simply
> >>>>>> +     * replicate the ISP sink format. The sizes on both pads are the same as
> >>>>>> +     * the ISP sink crop rectangle.
> >>>>>> +     */
> >>>>> 
> >>>>> Colorspace information will need to be propagated too.
> >>>>>
> >>>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
> >>>>>> +    src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>>>> +    src_fmt->width = crop->width;
> >>>>>> +    src_fmt->height = crop->height;
> >>>>>> +
> >>>>>> +    src_fmt = v4l2_subdev_state_get_format(state,
> >>>>>> +                           MALI_C55_ISP_PAD_SOURCE_BYPASS);
> >>>>>> +    src_fmt->code = fmt->code;
> >>>>>> +    src_fmt->width = crop->width;
> >>>>>> +    src_fmt->height = crop->height;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
> >>>>>> +                      struct v4l2_subdev_state *state,
> >>>>>> +                      struct v4l2_subdev_selection *sel)
> >>>>>> +{
> >>>>>> +    if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> >>>>> 
> >>>>>      sel->pad != MALI_C55_ISP_PAD_SINK_VIDEO
> >>>>>
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
> >>>>>> +                      struct v4l2_subdev_state *state,
> >>>>>> +                      struct v4l2_subdev_selection *sel)
> >>>>>> +{
> >>>>>> +    struct v4l2_mbus_framefmt *src_fmt;
> >>>>>> +    struct v4l2_mbus_framefmt *fmt;
> >>>>> 
> >>>>> const
> >>>>>
> >>>>>> +    struct v4l2_rect *crop;
> >>>>>> +
> >>>>>> +    if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> >>>>> 
> >>>>> Ditto.
> >>>>>
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +
> >>>>>> +    clamp_t(unsigned int, sel->r.left, 0, fmt->width);
> >>>>>> +    clamp_t(unsigned int, sel->r.top, 0, fmt->height);
> >>>>>> +    clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
> >>>>>> +        fmt->width - sel->r.left);
> >>>>>> +    clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
> >>>>>> +        fmt->height - sel->r.top);
> >>>>>> +
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +    *crop = sel->r;
> >>>>>> +
> >>>>>> +    /* Propagate the crop rectangle sizes to the source pad format. */
> >>>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
> >>>>>> +    src_fmt->width = crop->width;
> >>>>>> +    src_fmt->height = crop->height;
> >>>>> 
> >>>>> Can you confirm that cropping doesn't affect the bypass path ? And maybe
> >>>>> add a comment to mention it.
> >>>>>
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
> >>>>>> +    .enum_mbus_code        = mali_c55_isp_enum_mbus_code,
> >>>>>> +    .enum_frame_size    = mali_c55_isp_enum_frame_size,
> >>>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
> >>>>>> +    .set_fmt        = mali_c55_isp_set_fmt,
> >>>>>> +    .get_selection        = mali_c55_isp_get_selection,
> >>>>>> +    .set_selection        = mali_c55_isp_set_selection,
> >>>>>> +    .link_validate        = v4l2_subdev_link_validate_default,
> >>>>>> +};
> >>>>>> +
> >>>>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct v4l2_event event = {
> >>>>>> +        .type = V4L2_EVENT_FRAME_SYNC,
> >>>>>> +    };
> >>>>>> +
> >>>>>> +    event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
> >>>>>> +    v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int
> >>>>>> +mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
> >>>>>> +                 struct v4l2_event_subscription *sub)
> >>>>>> +{
> >>>>>> +    if (sub->type != V4L2_EVENT_FRAME_SYNC)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    /* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
> >>>>>> +    if (sub->id != 0)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    return v4l2_event_subscribe(fh, sub, 0, NULL);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
> >>>>>> +    .subscribe_event = mali_c55_isp_subscribe_event,
> >>>>>> +    .unsubscribe_event = v4l2_event_subdev_unsubscribe,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_ops mali_c55_isp_ops = {
> >>>>>> +    .pad    = &mali_c55_isp_pad_ops,
> >>>>>> +    .core    = &mali_c55_isp_core_ops,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
> >>>>>> +                   struct v4l2_subdev_state *sd_state)
> >>>>> 
> >>>>> You name this variable state in every other subdev operation handler.
> >>>>>
> >>>>>> +{
> >>>>>> +    struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >>>>>> +    struct v4l2_rect *in_crop;
> >>>>>> +
> >>>>>> +    sink_fmt = v4l2_subdev_state_get_format(sd_state,
> >>>>>> +                        MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +    src_fmt = v4l2_subdev_state_get_format(sd_state,
> >>>>>> +                           MALI_C55_ISP_PAD_SOURCE);
> >>>>>> +    in_crop = v4l2_subdev_state_get_crop(sd_state,
> >>>>>> +                         MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +
> >>>>>> +    sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>>>> +    sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>>>> +    sink_fmt->field = V4L2_FIELD_NONE;
> >>>>>> +    sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>>> 
> >>>>> You should initialize the colorspace fields too. Same below.
> >>>>>
> >>>>>> +
> >>>>>> +    *v4l2_subdev_state_get_format(sd_state,
> >>>>>> +                  MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
> >>>>>> +
> >>>>>> +    src_fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>>>> +    src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>>>> +    src_fmt->field = V4L2_FIELD_NONE;
> >>>>>> +    src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>>>> +
> >>>>>> +    in_crop->top = 0;
> >>>>>> +    in_crop->left = 0;
> >>>>>> +    in_crop->width = MALI_C55_DEFAULT_WIDTH;
> >>>>>> +    in_crop->height = MALI_C55_DEFAULT_HEIGHT;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
> >>>>>> +    .init_state = mali_c55_isp_init_state,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const struct media_entity_operations mali_c55_isp_media_ops = {
> >>>>>> +    .link_validate        = v4l2_subdev_link_validate,
> >>>>> 
> >>>>>      .link_validate = v4l2_subdev_link_validate,
> >>>>>
> >>>>> to match mali_c55_isp_internal_ops.
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
> >>>>>> +                       struct v4l2_subdev *subdev,
> >>>>>> +                       struct v4l2_async_connection *asc)
> >>>>>> +{
> >>>>>> +    struct mali_c55_isp *isp = container_of(notifier,
> >>>>>> +                        struct mali_c55_isp, notifier);
> >>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>>>> +    struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * By default we'll flag this link enabled and the TPG disabled, but
> >>>>>> +     * no immutable flag because we need to be able to switch between the
> >>>>>> +     * two.
> >>>>>> +     */
> >>>>>> +    ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
> >>>>>> +                          MEDIA_LNK_FL_ENABLED);
> >>>>>> +    if (ret)
> >>>>>> +        dev_err(mali_c55->dev, "failed to create link for %s\n",
> >>>>>> +            subdev->name);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
> >>>>>> +{
> >>>>>> +    struct mali_c55_isp *isp = container_of(notifier,
> >>>>>> +                        struct mali_c55_isp, notifier);
> >>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>>>> +
> >>>>>> +    return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
> >>>>>> +    .bound = mali_c55_isp_notifier_bound,
> >>>>>> +    .complete = mali_c55_isp_notifier_complete,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>>>> +    struct v4l2_async_connection *asc;
> >>>>>> +    struct fwnode_handle *ep;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * The ISP should have a single endpoint pointing to some flavour of
> >>>>>> +     * CSI-2 receiver...but for now at least we do want everything to work
> >>>>>> +     * normally even with no sensors connected, as we have the TPG. If we
> >>>>>> +     * don't find a sensor just warn and return success.
> >>>>>> +     */
> >>>>>> +    ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
> >>>>>> +                         0, 0, 0);
> >>>>>> +    if (!ep) {
> >>>>>> +        dev_warn(mali_c55->dev, "no local endpoint found\n");
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
> >>>>>> +                          struct v4l2_async_connection);
> >>>>>> +    if (IS_ERR(asc)) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to add remote fwnode\n");
> >>>>>> +        ret = PTR_ERR(asc);
> >>>>>> +        goto err_put_ep;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    isp->notifier.ops = &mali_c55_isp_notifier_ops;
> >>>>>> +    ret = v4l2_async_nf_register(&isp->notifier);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to register notifier\n");
> >>>>>> +        goto err_cleanup_nf;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    fwnode_handle_put(ep);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_cleanup_nf:
> >>>>>> +    v4l2_async_nf_cleanup(&isp->notifier);
> >>>>>> +err_put_ep:
> >>>>>> +    fwnode_handle_put(ep);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
> >>>>>> +    struct v4l2_subdev *sd = &isp->sd;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    isp->mali_c55 = mali_c55;
> >>>>>> +
> >>>>>> +    v4l2_subdev_init(sd, &mali_c55_isp_ops);
> >>>>>> +    sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> >>>>>> +    sd->entity.ops = &mali_c55_isp_media_ops;
> >>>>>> +    sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
> >>>>>> +    sd->internal_ops = &mali_c55_isp_internal_ops;
> >>>>>> +    strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
> >>>>>> +
> >>>>>> +    isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
> >>>>> 
> >>>>> The MUST_CONNECT flag would make sense here.
> >>>>>
> >>>>>> + isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> >>>>>> +    isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
> >>>>>> +
> >>>>>> +    ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
> >>>>>> +                     isp->pads);
> >>>>>> +    if (ret)
> >>>>>> +        return ret;
> >>>>>> +
> >>>>>> +    ret = v4l2_subdev_init_finalize(sd);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_cleanup_media_entity;
> >>>>>> +
> >>>>>> +    ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_cleanup_subdev;
> >>>>>> +
> >>>>>> +    ret = mali_c55_isp_parse_endpoint(isp);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_cleanup_subdev;
> >>>>> 
> >>>>> As noted elsewhere, I think this belongs to mali-c55-core.c.
> >>>>>
> >>>>>> +
> >>>>>> +    mutex_init(&isp->lock);
> >>>>> 
> >>>>> This lock is used in mali-c55-capture.c only, that seems weird.
> >>>>>
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_cleanup_subdev:
> >>>>>> +    v4l2_subdev_cleanup(sd);
> >>>>>> +err_cleanup_media_entity:
> >>>>>> +    media_entity_cleanup(&sd->entity);
> >>>>>> +    isp->mali_c55 = NULL;
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
> >>>>>> +
> >>>>>> +    if (!isp->mali_c55)
> >>>>>> +        return;
> >>>>>> +
> >>>>>> +    mutex_destroy(&isp->lock);
> >>>>>> +    v4l2_async_nf_unregister(&isp->notifier);
> >>>>>> +    v4l2_async_nf_cleanup(&isp->notifier);
> >>>>>> +    v4l2_device_unregister_subdev(&isp->sd);
> >>>>>> +    v4l2_subdev_cleanup(&isp->sd);
> >>>>>> +    media_entity_cleanup(&isp->sd.entity);
> >>>>>> +}
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..cb27abde2aa5
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>>>> @@ -0,0 +1,258 @@
> >>>>>> +/* SPDX-License-Identifier: GPL-2.0 */
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Register definitions
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#ifndef _MALI_C55_REGISTERS_H
> >>>>>> +#define _MALI_C55_REGISTERS_H
> >>>>>> +
> >>>>>> +#include <linux/bits.h>
> >>>>>> +
> >>>>>> +/* ISP Common 0x00000 - 0x000ff */
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_API                0x00000
> >>>>>> +#define MALI_C55_REG_PRODUCT                0x00004
> >>>>>> +#define MALI_C55_REG_VERSION                0x00008
> >>>>>> +#define MALI_C55_REG_REVISION                0x0000c
> >>>>>> +#define MALI_C55_REG_PULSE_MODE                0x0003c
> >>>>>> +#define MALI_C55_REG_INPUT_MODE_REQUEST            0x0009c
> >>>>>> +#define MALI_C55_INPUT_SAFE_STOP            0x00
> >>>>>> +#define MALI_C55_INPUT_SAFE_START            0x01
> >>>>>> +#define MALI_C55_REG_MODE_STATUS            0x000a0
> >>>>>> +#define MALI_C55_REG_INTERRUPT_MASK_VECTOR        0x00030
> >>>>>> +#define MALI_C55_INTERRUPT_MASK_ALL            GENMASK(31, 0)
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_GLOBAL_MONITOR            0x00050
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_GEN_VIDEO                0x00080
> >>>>>> +#define MALI_C55_REG_GEN_VIDEO_ON_MASK            BIT(0)
> >>>>>> +#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK        BIT(1)
> >>>>>> +#define MALI_C55_REG_GEN_PREFETCH_MASK GENMASK(31, 16)
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_MCU_CONFIG                0x00020
> >>>>>> +#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK        BIT(0)
> >>>>> 
> >>>>> #define MALI_C55_REG_MCU_CONFIG_OVERRIDE        BIT(0)
> >>>>>
> >>>>> Same in other places where applicable.
> >>>>>
> >>>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK        BIT(1)
> >>>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PING        BIT(1)
> >>>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG        0x00
> >>>>>> +#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK        BIT(8)
> >>>>>> +#define MALI_C55_REG_PING_PONG_READ            0x00024
> >>>>>> +#define MALI_C55_REG_PING_PONG_READ_MASK        BIT(2)
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR        0x00034
> >>>>>> +#define MALI_C55_REG_INTERRUPT_CLEAR            0x00040
> >>>>>> +#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR        0x00044
> >>>>>> +#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS        0x00068
> >>>>>> +#define MALI_C55_GPS_PONG_FITTED            BIT(0)
> >>>>>> +#define MALI_C55_GPS_WDR_FITTED                BIT(1)
> >>>>>> +#define MALI_C55_GPS_COMPRESSION_FITTED            BIT(2)
> >>>>>> +#define MALI_C55_GPS_TEMPER_FITTED            BIT(3)
> >>>>>> +#define MALI_C55_GPS_SINTER_LITE_FITTED            BIT(4)
> >>>>>> +#define MALI_C55_GPS_SINTER_FITTED            BIT(5)
> >>>>>> +#define MALI_C55_GPS_IRIDIX_LTM_FITTED            BIT(6)
> >>>>>> +#define MALI_C55_GPS_IRIDIX_GTM_FITTED            BIT(7)
> >>>>>> +#define MALI_C55_GPS_CNR_FITTED                BIT(8)
> >>>>>> +#define MALI_C55_GPS_FRSCALER_FITTED            BIT(9)
> >>>>>> +#define MALI_C55_GPS_DS_PIPE_FITTED            BIT(10)
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_BLANKING                0x00084
> >>>>>> +#define MALI_C55_REG_HBLANK_MASK            GENMASK(15, 0)
> >>>>>> +#define MALI_C55_REG_VBLANK_MASK            GENMASK(31, 16)
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_HC_START                0x00088
> >>>>>> +#define MALI_C55_HC_START(h)                (((h) & 0xffff) << 16)
> >>>>>> +#define MALI_C55_REG_HC_SIZE                0x0008c
> >>>>>> +#define MALI_C55_HC_SIZE(h)                ((h) & 0xffff)
> >>>>>> +#define MALI_C55_REG_VC_START_SIZE            0x00094
> >>>>>> +#define MALI_C55_VC_START(v)                ((v) & 0xffff)
> >>>>>> +#define MALI_C55_VC_SIZE(v)                (((v) & 0xffff) << 16)
> >>>>>> +
> >>>>>> +/* Ping/Pong Configuration Space */
> >>>>>> +#define MALI_C55_REG_BASE_ADDR                0x18e88
> >>>>>> +#define MALI_C55_REG_BYPASS_0                0x18eac
> >>>>>> +#define MALI_C55_REG_BYPASS_0_VIDEO_TEST        BIT(0)
> >>>>>> +#define MALI_C55_REG_BYPASS_0_INPUT_FMT            BIT(1)
> >>>>>> +#define MALI_C55_REG_BYPASS_0_DECOMPANDER        BIT(2)
> >>>>>> +#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR BIT(3)
> >>>>>> +#define MALI_C55_REG_BYPASS_0_GAIN_WDR            BIT(4)
> >>>>>> +#define MALI_C55_REG_BYPASS_0_FRAME_STITCH        BIT(5)
> >>>>>> +#define MALI_C55_REG_BYPASS_1                0x18eb0
> >>>>>> +#define MALI_C55_REG_BYPASS_1_DIGI_GAIN            BIT(0)
> >>>>>> +#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS        BIT(1)
> >>>>>> +#define MALI_C55_REG_BYPASS_1_FE_SQRT            BIT(2)
> >>>>>> +#define MALI_C55_REG_BYPASS_1_RAW_FE            BIT(3)
> >>>>>> +#define MALI_C55_REG_BYPASS_2                0x18eb8
> >>>>>> +#define MALI_C55_REG_BYPASS_2_SINTER            BIT(0)
> >>>>>> +#define MALI_C55_REG_BYPASS_2_TEMPER            BIT(1)
> >>>>>> +#define MALI_C55_REG_BYPASS_3                0x18ebc
> >>>>>> +#define MALI_C55_REG_BYPASS_3_SQUARE_BE            BIT(0)
> >>>>>> +#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH BIT(1)
> >>>>>> +#define MALI_C55_REG_BYPASS_3_MESH_SHADING        BIT(3)
> >>>>>> +#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE        BIT(4)
> >>>>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX            BIT(5)
> >>>>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN        BIT(6)
> >>>>>> +#define MALI_C55_REG_BYPASS_4                0x18ec0
> >>>>>> +#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB        BIT(1)
> >>>>>> +#define MALI_C55_REG_BYPASS_4_PF_CORRECTION        BIT(3)
> >>>>>> +#define MALI_C55_REG_BYPASS_4_CCM            BIT(4)
> >>>>>> +#define MALI_C55_REG_BYPASS_4_CNR            BIT(5)
> >>>>>> +#define MALI_C55_REG_FR_BYPASS                0x18ec4
> >>>>>> +#define MALI_C55_REG_DS_BYPASS                0x18ec8
> >>>>>> +#define MALI_C55_BYPASS_CROP                BIT(0)
> >>>>>> +#define MALI_C55_BYPASS_SCALER                BIT(1)
> >>>>>> +#define MALI_C55_BYPASS_GAMMA_RGB            BIT(2)
> >>>>>> +#define MALI_C55_BYPASS_SHARPEN                BIT(3)
> >>>>>> +#define MALI_C55_BYPASS_CS_CONV                BIT(4)
> >>>>>> +#define MALI_C55_REG_ISP_RAW_BYPASS            0x18ecc
> >>>>>> +#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK        BIT(0)
> >>>>>> +#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK GENMASK(9, 8)
> >>>>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS        2
> >>>>>> +#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS        1
> >>>>>> +#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE BIT(1)
> >>>>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS        BIT(0)
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_ACTIVE_WIDTH_MASK            0xffff
> >>>>>> +#define MALI_C55_REG_ACTIVE_HEIGHT_MASK 0xffff0000
> >>>>>> +#define MALI_C55_REG_BAYER_ORDER            0x18e8c
> >>>>>> +#define MALI_C55_BAYER_ORDER_MASK            GENMASK(1, 0)
> >>>>>> +#define MALI_C55_REG_TPG_CH0                0x18ed8
> >>>>>> +#define MALI_C55_TEST_PATTERN_ON_OFF            BIT(0)
> >>>>>> +#define MALI_C55_TEST_PATTERN_RGB_MASK            BIT(1)
> >>>>>> +#define MALI_C55_REG_TPG_R_BACKGROUND            0x18ee0
> >>>>>> +#define MALI_C55_REG_TPG_G_BACKGROUND            0x18ee4
> >>>>>> +#define MALI_C55_REG_TPG_B_BACKGROUND            0x18ee8
> >>>>>> +#define MALI_C55_TPG_BACKGROUND_MAX            0xfffff
> >>>>>> +#define MALI_C55_REG_INPUT_WIDTH            0x18f98
> >>>>>> +#define MALI_C55_INPUT_WIDTH_MASK            GENMASK(18, 16)
> >>>>>> +#define MALI_C55_INPUT_WIDTH_8BIT            0
> >>>>>> +#define MALI_C55_INPUT_WIDTH_10BIT            1
> >>>>>> +#define MALI_C55_INPUT_WIDTH_12BIT            2
> >>>>>> +#define MALI_C55_INPUT_WIDTH_14BIT            3
> >>>>>> +#define MALI_C55_INPUT_WIDTH_16BIT            4
> >>>>>> +#define MALI_C55_INPUT_WIDTH_20BIT            5
> >>>>>> +#define MALI_C55_REG_SPACE_SIZE                0x4000
> >>>>>> +#define MALI_C55_REG_CONFIG_SPACES_OFFSET        0x0ab6c
> >>>>>> +#define MALI_C55_CONFIG_SPACE_SIZE            0x1231c
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_SINTER_CONFIG            0x19348
> >>>>>> +#define MALI_C55_SINTER_VIEW_FILTER_MASK        GENMASK(1, 0)
> >>>>>> +#define MALI_C55_SINTER_SCALE_MODE_MASK GENMASK(3, 2)
> >>>>>> +#define MALI_C55_SINTER_ENABLE_MASK            BIT(4)
> >>>>>> +#define MALI_C55_SINTER_FILTER_SELECT_MASK        BIT(5)
> >>>>>> +#define MALI_C55_SINTER_INT_SELECT_MASK            BIT(6)
> >>>>>> +#define MALI_C55_SINTER_RM_ENABLE_MASK            BIT(7)
> >>>>>> +
> >>>>>> +/* Colour Correction Matrix Configuration */
> >>>>>> +#define MALI_C55_REG_CCM_ENABLE                0x1b07c
> >>>>>> +#define MALI_C55_CCM_ENABLE_MASK            BIT(0)
> >>>>>> +#define MALI_C55_REG_CCM_COEF_R_R            0x1b080
> >>>>>> +#define MALI_C55_REG_CCM_COEF_R_G            0x1b084
> >>>>>> +#define MALI_C55_REG_CCM_COEF_R_B            0x1b088
> >>>>>> +#define MALI_C55_REG_CCM_COEF_G_R            0x1b090
> >>>>>> +#define MALI_C55_REG_CCM_COEF_G_G            0x1b094
> >>>>>> +#define MALI_C55_REG_CCM_COEF_G_B            0x1b098
> >>>>>> +#define MALI_C55_REG_CCM_COEF_B_R            0x1b0a0
> >>>>>> +#define MALI_C55_REG_CCM_COEF_B_G            0x1b0a4
> >>>>>> +#define MALI_C55_REG_CCM_COEF_B_B            0x1b0a8
> >>>>>> +#define MALI_C55_CCM_COEF_MASK                GENMASK(12, 0)
> >>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R            0x1b0b0
> >>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G            0x1b0b4
> >>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B            0x1b0b8
> >>>>>> +#define MALI_C55_CCM_ANTIFOG_GAIN_MASK GENMASK(11, 0)
> >>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R        0x1b0c0
> >>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G        0x1b0c4
> >>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B        0x1b0c8
> >>>>>> +#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK        GENMASK(11, 0)
> >>>>>> +
> >>>>>> +/*
> >>>>>> + * The Mali-C55 ISP has up to two output pipes; known as full resolution and
> >>>>>> + * down scaled. The register space for these is laid out identically, but offset
> >>>>>> + * by 372 bytes.
> >>>>>> + */
> >>>>>> +#define MALI_C55_CAP_DEV_FR_REG_OFFSET        0x0
> >>>>>> +#define MALI_C55_CAP_DEV_DS_REG_OFFSET        0x174
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_CS_CONV_CONFIG(offset)        (0x1c098 + (offset))
> >>>>>> +#define MALI_C55_CS_CONV_MATRIX_MASK            BIT(0)
> >>>>>> +#define MALI_C55_CS_CONV_FILTER_MASK            BIT(1)
> >>>>>> +#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK        BIT(2)
> >>>>>> +#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK        BIT(3)
> >>>>>> +#define MALI_C55_REG_Y_WRITER_MODE(offset)        (0x1c0ec + (offset))
> >>>>>> +#define MALI_C55_REG_UV_WRITER_MODE(offset)        (0x1c144 + (offset))
> >>>>>> +#define MALI_C55_WRITER_MODE_MASK            GENMASK(4, 0)
> >>>>>> +#define MALI_C55_WRITER_SUBMODE_MASK            GENMASK(7, 6)
> >>>>>> +#define MALI_C55_WRITER_FRAME_WRITE_MASK        BIT(9)
> >>>>>> +#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset) (0x1c0f0 + (offset))
> >>>>>> +#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset) (0x1c148 + (offset))
> >>>>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)        ((w) << 0)
> >>>>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)        ((h) << 16)
> >>>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset) (0x1c0f4 + (offset))
> >>>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset) (0x1c108 + (offset))
> >>>>>> +#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK GENMASK(2, 0)
> >>>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_RESTART        BIT(3)
> >>>>>> +#define MALI_C55_REG_Y_WRITER_OFFSET(offset) (0x1c10c + (offset))
> >>>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset) (0x1c14c + (offset))
> >>>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset) (0x1c160 + (offset))
> >>>>>> +#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK GENMASK(2, 0)
> >>>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_RESTART        BIT(3)
> >>>>>> +#define MALI_C55_REG_UV_WRITER_OFFSET(offset) (0x1c164 + (offset))
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
> >>>>>> +#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE 0x18edc
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_CROP_EN(offset)            (0x1c028 + (offset))
> >>>>>> +#define MALI_C55_CROP_ENABLE                BIT(0)
> >>>>>> +#define MALI_C55_REG_CROP_X_START(offset)        (0x1c02c + (offset))
> >>>>>> +#define MALI_C55_REG_CROP_Y_START(offset)        (0x1c030 + (offset))
> >>>>>> +#define MALI_C55_REG_CROP_X_SIZE(offset)        (0x1c034 + (offset))
> >>>>>> +#define MALI_C55_REG_CROP_Y_SIZE(offset)        (0x1c038 + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset) (0x1c040 + (offset))
> >>>>>> +#define MALI_C55_SCALER_TIMEOUT_EN            BIT(4)
> >>>>>> +#define MALI_C55_SCALER_TIMEOUT(t)            ((t) << 16)
> >>>>>> +#define MALI_C55_REG_SCALER_IN_WIDTH(offset) (0x1c044 + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_IN_HEIGHT(offset) (0x1c048 + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_OUT_WIDTH(offset) (0x1c04c + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset) (0x1c050 + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_HFILT_TINC(offset) (0x1c054 + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_HFILT_COEF(offset) (0x1c058 + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_VFILT_TINC(offset) (0x1c05c + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_VFILT_COEF(offset) (0x1c060 + (offset))
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset) (0x1c064 + (offset))
> >>>>>> +#define MALI_C55_GAMMA_ENABLE_MASK            BIT(0)
> >>>>>> +#define MALI_C55_REG_GAMMA_GAINS_1(offset)        (0x1c068 + (offset))
> >>>>>> +#define MALI_C55_GAMMA_GAIN_R_MASK            GENMASK(11, 0)
> >>>>>> +#define MALI_C55_GAMMA_GAIN_G_MASK            GENMASK(27, 16)
> >>>>>> +#define MALI_C55_REG_GAMMA_GAINS_2(offset)        (0x1c06c + (offset))
> >>>>>> +#define MALI_C55_GAMMA_GAIN_B_MASK            GENMASK(11, 0)
> >>>>>> +#define MALI_C55_REG_GAMMA_OFFSETS_1(offset) (0x1c070 + (offset))
> >>>>>> +#define MALI_C55_GAMMA_OFFSET_R_MASK            GENMASK(11, 0)
> >>>>>> +#define MALI_C55_GAMMA_OFFSET_G_MASK            GENMASK(27, 16)
> >>>>>> +#define MALI_C55_REG_GAMMA_OFFSETS_2(offset) (0x1c074 + (offset))
> >>>>>> +#define MALI_C55_GAMMA_OFFSET_B_MASK            GENMASK(11, 0)
> >>>>>> +
> >>>>>> +/* Output DMA Writer */
> >>>>>> +
> >>>>>> +#define MALI_C55_OUTPUT_DISABLED        0
> >>>>>> +#define MALI_C55_OUTPUT_RGB32            1
> >>>>>> +#define MALI_C55_OUTPUT_A2R10G10B10        2
> >>>>>> +#define MALI_C55_OUTPUT_RGB565            3
> >>>>>> +#define MALI_C55_OUTPUT_RGB24            4
> >>>>>> +#define MALI_C55_OUTPUT_GEN32            5
> >>>>>> +#define MALI_C55_OUTPUT_RAW16            6
> >>>>>> +#define MALI_C55_OUTPUT_AYUV            8
> >>>>>> +#define MALI_C55_OUTPUT_Y410            9
> >>>>>> +#define MALI_C55_OUTPUT_YUY2            10
> >>>>>> +#define MALI_C55_OUTPUT_UYVY            11
> >>>>>> +#define MALI_C55_OUTPUT_Y210            12
> >>>>>> +#define MALI_C55_OUTPUT_NV12_21            13
> >>>>>> +#define MALI_C55_OUTPUT_YUV_420_422        17
> >>>>>> +#define MALI_C55_OUTPUT_P210_P010        19
> >>>>>> +#define MALI_C55_OUTPUT_YUV422            20
> >>>>> 
> >>>>> I'd define this just below the MALI_C55_REG_UV_WRITER_MODE register
> >>>>> macro.
> >>>>>
> >>>>>> +
> >>>>>> +#define MALI_C55_OUTPUT_PLANE_ALT0        0
> >>>>>> +#define MALI_C55_OUTPUT_PLANE_ALT1        1
> >>>>>> +#define MALI_C55_OUTPUT_PLANE_ALT2        2
> >>>>> 
> >>>>> Same here ?
> >>>>>
> >>>>>> +
> >>>>>> +#endif /* _MALI_C55_REGISTERS_H */
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..8edae87f1e5f
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>>>>> @@ -0,0 +1,382 @@
> >>>>>> +/* SPDX-License-Identifier: GPL-2.0 */
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Resizer Coefficients
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#ifndef _MALI_C55_RESIZER_COEFS_H
> >>>>>> +#define _MALI_C55_RESIZER_COEFS_H
> >>>>>> +
> >>>>>> +#include "mali-c55-common.h"
> >>>>>> +
> >>>>>> +#define MALI_C55_RESIZER_COEFS_NUM_BANKS    8
> >>>>>> +#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES    64
> >>>>> 
> >>>>> Do these belongs to mali-c55-registers.h ?
> >>>>>
> >>>>>> +
> >>>>>> +static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
> >>>>>> +    {    /* Bank 0 */
> >>>>>> +        0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
> >>>>>> +        0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
> >>>>>> +        0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
> >>>>>> +        0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
> >>>>>> +        0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
> >>>>>> +        0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
> >>>>>> +        0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
> >>>>>> +        0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
> >>>>>> +        0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
> >>>>>> +        0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
> >>>>>> +        0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
> >>>>>> +        0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
> >>>>>> +        0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
> >>>>>> +        0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
> >>>>>> +        0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
> >>>>>> +        0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 1 */
> >>>>>> +        0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
> >>>>>> +        0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
> >>>>>> +        0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
> >>>>>> +        0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
> >>>>>> +        0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
> >>>>>> +        0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
> >>>>>> +        0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
> >>>>>> +        0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
> >>>>>> +        0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
> >>>>>> +        0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
> >>>>>> +        0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
> >>>>>> +        0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
> >>>>>> +        0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
> >>>>>> +        0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
> >>>>>> +        0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
> >>>>>> +        0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 2 */
> >>>>>> +        0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
> >>>>>> +        0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
> >>>>>> +        0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
> >>>>>> +        0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
> >>>>>> +        0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
> >>>>>> +        0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
> >>>>>> +        0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
> >>>>>> +        0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
> >>>>>> +        0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
> >>>>>> +        0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
> >>>>>> +        0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
> >>>>>> +        0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
> >>>>>> +        0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
> >>>>>> +        0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
> >>>>>> +        0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
> >>>>>> +        0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 3 */
> >>>>>> +        0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
> >>>>>> +        0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
> >>>>>> +        0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
> >>>>>> +        0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
> >>>>>> +        0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
> >>>>>> +        0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
> >>>>>> +        0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
> >>>>>> +        0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
> >>>>>> +        0x20100000, 0x00000010, 0x1f110000, 0x00000010,
> >>>>>> +        0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
> >>>>>> +        0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
> >>>>>> +        0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
> >>>>>> +        0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
> >>>>>> +        0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
> >>>>>> +        0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
> >>>>>> +        0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 4 */
> >>>>>> +        0x17090000, 0x00000917, 0x18090000, 0x00000916,
> >>>>>> +        0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
> >>>>>> +        0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
> >>>>>> +        0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
> >>>>>> +        0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
> >>>>>> +        0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
> >>>>>> +        0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
> >>>>>> +        0x190f0300, 0x00000411, 0x18100300, 0x00000411,
> >>>>>> +        0x1a100300, 0x00000310, 0x18110400, 0x00000310,
> >>>>>> +        0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
> >>>>>> +        0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
> >>>>>> +        0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
> >>>>>> +        0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
> >>>>>> +        0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
> >>>>>> +        0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
> >>>>>> +        0x17160800, 0x0000010a, 0x18160900, 0x00000009,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 5 */
> >>>>>> +        0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
> >>>>>> +        0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
> >>>>>> +        0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
> >>>>>> +        0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
> >>>>>> +        0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
> >>>>>> +        0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
> >>>>>> +        0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
> >>>>>> +        0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
> >>>>>> +        0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
> >>>>>> +        0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
> >>>>>> +        0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
> >>>>>> +        0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
> >>>>>> +        0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
> >>>>>> +        0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
> >>>>>> +        0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
> >>>>>> +        0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 6 */
> >>>>>> +        0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
> >>>>>> +        0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
> >>>>>> +        0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
> >>>>>> +        0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
> >>>>>> +        0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
> >>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>>>> +        0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
> >>>>>> +        0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>>>> +        0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
> >>>>>> +        0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
> >>>>>> +        0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
> >>>>>> +        0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >>>>>> +        0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 7 */
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +    }
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
> >>>>>> +    {    /* Bank 0 */
> >>>>>> +        0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
> >>>>>> +        0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
> >>>>>> +        0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
> >>>>>> +        0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
> >>>>>> +        0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
> >>>>>> +        0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
> >>>>>> +        0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
> >>>>>> +        0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
> >>>>>> +        0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
> >>>>>> +        0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
> >>>>>> +        0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
> >>>>>> +        0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
> >>>>>> +        0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
> >>>>>> +        0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
> >>>>>> +        0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
> >>>>>> +        0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 1 */
> >>>>>> +        0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
> >>>>>> +        0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
> >>>>>> +        0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
> >>>>>> +        0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
> >>>>>> +        0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
> >>>>>> +        0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
> >>>>>> +        0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
> >>>>>> +        0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
> >>>>>> +        0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
> >>>>>> +        0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
> >>>>>> +        0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
> >>>>>> +        0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
> >>>>>> +        0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
> >>>>>> +        0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
> >>>>>> +        0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
> >>>>>> +        0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 2 */
> >>>>>> +        0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
> >>>>>> +        0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
> >>>>>> +        0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
> >>>>>> +        0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
> >>>>>> +        0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
> >>>>>> +        0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
> >>>>>> +        0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
> >>>>>> +        0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
> >>>>>> +        0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
> >>>>>> +        0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
> >>>>>> +        0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
> >>>>>> +        0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
> >>>>>> +        0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
> >>>>>> +        0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
> >>>>>> +        0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
> >>>>>> +        0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 3 */
> >>>>>> +        0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
> >>>>>> +        0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
> >>>>>> +        0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
> >>>>>> +        0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
> >>>>>> +        0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
> >>>>>> +        0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
> >>>>>> +        0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
> >>>>>> +        0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
> >>>>>> +        0x20100000, 0x00000010, 0x1f100000, 0x00000011,
> >>>>>> +        0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
> >>>>>> +        0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
> >>>>>> +        0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
> >>>>>> +        0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
> >>>>>> +        0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
> >>>>>> +        0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
> >>>>>> +        0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 4 */
> >>>>>> +        0x17170900, 0x00000009, 0x18160900, 0x00000009,
> >>>>>> +        0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
> >>>>>> +        0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
> >>>>>> +        0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
> >>>>>> +        0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
> >>>>>> +        0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
> >>>>>> +        0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
> >>>>>> +        0x19110400, 0x0000030f, 0x18110400, 0x00000310,
> >>>>>> +        0x1a100300, 0x00000310, 0x18100300, 0x00000411,
> >>>>>> +        0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
> >>>>>> +        0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
> >>>>>> +        0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
> >>>>>> +        0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
> >>>>>> +        0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
> >>>>>> +        0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
> >>>>>> +        0x170a0100, 0x00000816, 0x18090000, 0x00000916,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 5 */
> >>>>>> +        0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
> >>>>>> +        0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
> >>>>>> +        0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
> >>>>>> +        0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
> >>>>>> +        0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
> >>>>>> +        0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
> >>>>>> +        0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
> >>>>>> +        0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
> >>>>>> +        0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
> >>>>>> +        0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
> >>>>>> +        0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
> >>>>>> +        0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
> >>>>>> +        0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
> >>>>>> +        0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
> >>>>>> +        0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
> >>>>>> +        0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 6 */
> >>>>>> +        0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
> >>>>>> +        0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >>>>>> +        0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
> >>>>>> +        0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
> >>>>>> +        0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
> >>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>>>> +        0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >>>>>> +        0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>>>> +        0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
> >>>>>> +        0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
> >>>>>> +        0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
> >>>>>> +        0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
> >>>>>> +        0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 7 */
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +    }
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_resizer_coef_bank {
> >>>>>> +    unsigned int bank;
> >>>>> 
> >>>>> This is always equal to the index of the entry in the
> >>>>> mali_c55_coefficient_banks array, you can drop it.
> >>>>>
> >>>>>> +    unsigned int top;
> >>>>>> +    unsigned int bottom;
> >>>>> 
> >>>>> The bottom value of bank N is always equal to the top value of bank N+1.
> >>>>> You can simplify this by storing a single value.
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
> >>>>>> +    {
> >>>>>> +        .bank = 0,
> >>>>>> +        .top = 1000,
> >>>>>> +        .bottom = 770,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .bank = 1,
> >>>>>> +        .top = 769,
> >>>>>> +        .bottom = 600,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .bank = 2,
> >>>>>> +        .top = 599,
> >>>>>> +        .bottom = 460,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .bank = 3,
> >>>>>> +        .top = 459,
> >>>>>> +        .bottom = 354,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .bank = 4,
> >>>>>> +        .top = 353,
> >>>>>> +        .bottom = 273,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .bank = 5,
> >>>>>> +        .top = 272,
> >>>>>> +        .bottom = 210,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .bank = 6,
> >>>>>> +        .top = 209,
> >>>>>> +        .bottom = 162,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .bank = 7,
> >>>>>> +        .top = 161,
> >>>>>> +        .bottom = 125,
> >>>>>> +    },
> >>>>>> +};
> >>>>>> +
> >>>>> 
> >>>>> A small comment would be nice, such as
> >>>>>
> >>>>> /* Select a bank of resizer coefficients, based on the scaling ratio. */
> >>>>>
> >>>>>> +static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,
> >>>>> 
> >>>>> This function is related to the resizers. Add "rsz" somewhere in the
> >>>>> function name, and pass a resizer pointer.
> >>>>>
> >>>>>> +                        unsigned int crop,
> >>>>>> +                        unsigned int scale)
> >>>>> 
> >>>>> I think those are the input and output sizes to the scaler. Rename them
> >>>>> to make it clearer.
> >>>>>
> >>>>>> +{
> >>>>>> +    unsigned int tmp;
> >>>>> 
> >>>>> tmp is almost always a bad variable name. Please use a more descriptive
> >>>>> name, size as rsz_ratio.
> >>>>>
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    tmp = (scale * 1000U) / crop;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
> >>>>>> +        if (tmp >= mali_c55_coefficient_banks[i].bottom &&
> >>>>>> +            tmp <= mali_c55_coefficient_banks[i].top)
> >>>>>> +            return mali_c55_coefficient_banks[i].bank;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * We shouldn't ever get here, in theory. As we have no good choices
> >>>>>> +     * simply warn the user and use the first bank of coefficients.
> >>>>>> +     */
> >>>>>> +    dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>> 
> >>>>> And everything else belongs to mali-c55-resizer.c. Drop this header
> >>>>> file.
> >>>>>
> >>>>>> +
> >>>>>> +#endif /* _MALI_C55_RESIZER_COEFS_H */
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..0a5a2969d3ce
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>>>>> @@ -0,0 +1,779 @@
> >>>>>> +// SPDX-License-Identifier: GPL-2.0
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Image signal processor
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#include <linux/math.h>
> >>>>>> +#include <linux/minmax.h>
> >>>>>> +
> >>>>>> +#include <media/media-entity.h>
> >>>>>> +#include <media/v4l2-subdev.h>
> >>>>>> +
> >>>>>> +#include "mali-c55-common.h"
> >>>>>> +#include "mali-c55-registers.h"
> >>>>>> +#include "mali-c55-resizer-coefs.h"
> >>>>>> +
> >>>>>> +/* Scaling factor in Q4.20 format. */
> >>>>>> +#define MALI_C55_RZR_SCALER_FACTOR    (1U << 20)
> >>>>>> +
> >>>>>> +static const u32 rzr_non_bypass_src_fmts[] = {
> >>>>>> +    MEDIA_BUS_FMT_RGB121212_1X36,
> >>>>>> +    MEDIA_BUS_FMT_YUV10_1X30
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const char * const mali_c55_resizer_names[] = {
> >>>>>> +    [MALI_C55_RZR_FR] = "resizer fr",
> >>>>>> +    [MALI_C55_RZR_DS] = "resizer ds",
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
> >>>>>> +                     struct v4l2_subdev_state *state)
> >>>>>> +{
> >>>>>> +    unsigned int reg_offset = rzr->cap_dev->reg_offset;
> >>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>>>> +    struct v4l2_mbus_framefmt *fmt;
> >>>>>> +    struct v4l2_rect *crop;
> >>>> 
> >>>> const
> >>>>
> >>>>>> +
> >>>>>> +    /* Verify if crop should be enabled. */
> >>>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>>>> +
> >>>>>> +    if (fmt->width == crop->width && fmt->height == crop->height)
> >>>>>> +        return MALI_C55_BYPASS_CROP;
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
> >>>>>> +               crop->left);
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
> >>>>>> +               crop->top);
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
> >>>>>> +               crop->width);
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
> >>>>>> +               crop->height);
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
> >>>>>> +               MALI_C55_CROP_ENABLE);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
> >>>>>> +                    struct v4l2_subdev_state *state)
> >>>>>> +{
> >>>>>> +    unsigned int reg_offset = rzr->cap_dev->reg_offset;
> >>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>>>> +    struct v4l2_rect *crop, *scale;
> >>>> 
> >>>> const
> >>>>
> >>>> Once "[PATCH v4 0/3] media: v4l2-subdev: Support const-awareness in
> >>>> state accessors" gets merged, the state argument to this function can be
> >>>> made const too. Same for other functions, as applicable.
> >>>>
> >>>>>> +    unsigned int h_bank, v_bank;
> >>>>>> +    u64 h_scale, v_scale;
> >>>>>> +
> >>>>>> +    /* Verify if scaling should be enabled. */
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>>>> +    scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>>>> +
> >>>>>> +    if (crop->width == scale->width && crop->height == scale->height)
> >>>>>> +        return MALI_C55_BYPASS_SCALER;
> >>>>>> +
> >>>>>> +    /* Program the V/H scaling factor in Q4.20 format. */
> >>>>>> +    h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
> >>>>>> +    v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
> >>>>>> +
> >>>>>> +    do_div(h_scale, scale->width);
> >>>>>> +    do_div(v_scale, scale->height);
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
> >>>>>> +               crop->width);
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
> >>>>>> +               crop->height);
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
> >>>>>> +               scale->width);
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
> >>>>>> +               scale->height);
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
> >>>>>> +               h_scale);
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
> >>>>>> +               v_scale);
> >>>>>> +
> >>>>>> +    h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
> >>>>>> +                         scale->width);
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
> >>>>>> +               h_bank);
> >>>>>> +
> >>>>>> +    v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
> >>>>>> +                         scale->height);
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
> >>>>>> +               v_bank);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
> >>>>>> +                 struct v4l2_subdev_state *state)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>>>> +    u32 bypass = 0;
> >>>>>> +
> >>>>>> +    /* Verify if cropping and scaling should be enabled. */
> >>>>>> +    bypass |= mali_c55_rzr_program_crop(rzr, state);
> >>>>>> +    bypass |= mali_c55_rzr_program_resizer(rzr, state);
> >>>>>> +
> >>>>>> +    mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
> >>>>>> +                 MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
> >>>>>> +                 MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
> >>>>>> +                 bypass);
> >>>>>> +}
> >>>>>> +
> >>>>>> +/*
> >>>>>> + * Inspect the routing table to know which of the two (mutually exclusive)
> >>>>>> + * routes is enabled and return the sink pad id of the active route.
> >>>>>> + */
> >>>>>> +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
> >>>>>> +{
> >>>>>> +    struct v4l2_subdev_krouting *routing = &state->routing;
> >>>>>> +    struct v4l2_subdev_route *route;
> >>>>>> +
> >>>>>> +    /* A single route is enabled at a time. */
> >>>>>> +    for_each_active_route(routing, route)
> >>>>>> +        return route->sink_pad;
> >>>>>> +
> >>>>>> +    return MALI_C55_RZR_SINK_PAD;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
> >>>>>> +{
> >>>>>> +    u32 corrected_code = 0;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * The ISP takes input in a 20-bit format, but can only output 16-bit
> >>>>>> +     * RAW bayer data (with the 4 least significant bits from the input
> >>>>>> +     * being lost). Return the 16-bit version of the 20-bit input formats.
> >>>>>> +     */
> >>>>>> +    switch (mbus_code) {
> >>>>>> +    case MEDIA_BUS_FMT_SBGGR20_1X20:
> >>>>>> +        corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
> >>>>>> +        break;
> >>>>>> +    case MEDIA_BUS_FMT_SGBRG20_1X20:
> >>>>>> +        corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
> >>>>>> +        break;
> >>>>>> +    case MEDIA_BUS_FMT_SGRBG20_1X20:
> >>>>>> +        corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
> >>>>>> +        break;
> >>>>>> +    case MEDIA_BUS_FMT_SRGGB20_1X20:
> >>>>>> +        corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
> >>>>>> +        break;
> >>>> 
> >>>> Would it make sense to add the shifted code to mali_c55_isp_fmt ?
> >>>>
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return corrected_code;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> >>>>>> +                      struct v4l2_subdev_state *state,
> >>>>>> +                      struct v4l2_subdev_krouting *routing)
> >>>> 
> >>>> I think the last argument can be const.
> >>>>
> >>>>>> +{
> >>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>>>> +                            sd);
> >>>> 
> >>>> A to_mali_c55_resizer() static inline function would be useful. Same for
> >>>> other components, where applicable.
> >>>>
> >>>>>> +    unsigned int active_sink = UINT_MAX;
> >>>>>> +    struct v4l2_mbus_framefmt *src_fmt;
> >>>>>> +    struct v4l2_rect *crop, *compose;
> >>>>>> +    struct v4l2_subdev_route *route;
> >>>>>> +    unsigned int active_routes = 0;
> >>>>>> +    struct v4l2_mbus_framefmt *fmt;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    ret = v4l2_subdev_routing_validate(sd, routing, 0);
> >>>>>> +    if (ret)
> >>>>>> +        return ret;
> >>>>>> +
> >>>>>> +    /* Only a single route can be enabled at a time. */
> >>>>>> +    for_each_active_route(routing, route) {
> >>>>>> +        if (++active_routes > 1) {
> >>>>>> +            dev_err(rzr->mali_c55->dev,
> >>>>>> +                "Only one route can be active");
> >>>> 
> >>>> No kernel log message with a level higher than dev_dbg() from
> >>>> user-controlled paths please, here and where applicable. This is to
> >>>> avoid giving applications an easy way to flood the kernel log.
> >>>>
> >>>>>> +            return -EINVAL;
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        active_sink = route->sink_pad;
> >>>>>> +    }
> >>>>>> +    if (active_sink == UINT_MAX) {
> >>>>>> +        dev_err(rzr->mali_c55->dev, "One route has to be active");
> >>>>>> +        return -EINVAL;
> >>>>>> +    }
> >>>> 
> >>>> The recommended handling of invalid routing is to adjust the routing
> >>>> table, not to return errors.
> >>>>
> >>>>>> +
> >>>>>> +    ret = v4l2_subdev_set_routing(sd, state, routing);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
> >>>>>> +        return ret;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
> >>>>>> +    compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
> >>>>>> +
> >>>>>> +    fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>>>> +    fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>>>> +    fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>> 
> >>>> There are other colorspace-related fields.
> >>>>
> >>>>>> +    fmt->field = V4L2_FIELD_NONE;
> >>>> 
> >>>> I wonder if we should really update the sink pad format, or just
> >>>> propagate it. If we update it, I think it should be set to defaults on
> >>>> both sink pads, not just the active sink pad.
> >>>>
> >>>>>> +
> >>>>>> +    if (active_sink == MALI_C55_RZR_SINK_PAD) {
> >>>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>>>> +
> >>>>>> +        crop->left = crop->top = 0;
> >>>> 
> >>>>          crop->left = 0;
> >>>>          crop->top = 0;
> >>>>
> >>>>>> +        crop->width = MALI_C55_DEFAULT_WIDTH;
> >>>>>> +        crop->height = MALI_C55_DEFAULT_HEIGHT;
> >>>>>> +
> >>>>>> +        *compose = *crop;
> >>>>>> +    } else {
> >>>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* Propagate the format to the source pad */
> >>>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
> >>>>>> +                           0);
> >>>>>> +    *src_fmt = *fmt;
> >>>>>> +
> >>>>>> +    /* In the event this is the bypass pad the mbus code needs correcting */
> >>>>>> +    if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
> >>>>>> +        src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>> +                       struct v4l2_subdev_state *state,
> >>>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
> >>>>>> +{
> >>>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
> >>>>>> +    const struct mali_c55_isp_fmt *fmt;
> >>>>>> +    unsigned int index = 0;
> >>>>>> +    u32 sink_pad;
> >>>>>> +
> >>>>>> +    switch (code->pad) {
> >>>>>> +    case MALI_C55_RZR_SINK_PAD:
> >>>>>> +        if (code->index)
> >>>>>> +            return -EINVAL;
> >>>>>> +
> >>>>>> +        code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>>>> +
> >>>>>> +        return 0;
> >>>>>> +    case MALI_C55_RZR_SOURCE_PAD:
> >>>>>> +        sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>>>> +        sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> >>>>>> +
> >>>>>> +        /*
> >>>>>> +         * If the active route is from the Bypass sink pad, then the
> >>>>>> +         * source pad is a simple passthrough of the sink format,
> >>>>>> +         * downshifted to 16-bits.
> >>>>>> +         */
> >>>>>> +
> >>>>>> +        if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >>>>>> +            if (code->index)
> >>>>>> +                return -EINVAL;
> >>>>>> +
> >>>>>> +            code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> >>>>>> +            if (!code->code)
> >>>>>> +                return -EINVAL;
> >>>>>> +
> >>>>>> +            return 0;
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        /*
> >>>>>> +         * If the active route is from the non-bypass sink then we can
> >>>>>> +         * select either RGB or conversion to YUV.
> >>>>>> +         */
> >>>>>> +
> >>>>>> +        if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
> >>>>>> +            return -EINVAL;
> >>>>>> +
> >>>>>> +        code->code = rzr_non_bypass_src_fmts[code->index];
> >>>>>> +
> >>>>>> +        return 0;
> >>>>>> +    case MALI_C55_RZR_SINK_BYPASS_PAD:
> >>>>>> +        for_each_mali_isp_fmt(fmt) {
> >>>>>> +            if (index++ == code->index) {
> >>>>>> +                code->code = fmt->code;
> >>>>>> +                return 0;
> >>>>>> +            }
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        break;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return -EINVAL;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
> >>>>>> +                    struct v4l2_subdev_state *state,
> >>>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
> >>>>>> +{
> >>>>>> +    if (fse->index)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
> >>>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
> >>>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
> >>>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
> >>>>>> +                     struct v4l2_subdev_state *state,
> >>>>>> +                     struct v4l2_subdev_format *format)
> >>>>>> +{
> >>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>>>> +    struct v4l2_rect *rect;
> >>>>>> +    unsigned int sink_pad;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * Clamp to min/max and then reset crop and compose rectangles to the
> >>>>>> +     * newly applied size.
> >>>>>> +     */
> >>>>>> +    clamp_t(unsigned int, fmt->width,
> >>>>>> +        MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>>>>> +    clamp_t(unsigned int, fmt->height,
> >>>>>> +        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>>> 
> >>>> Please check comments for other components related to the colorspace
> >>>> fields, to decide how to handle them here.
> >>>>
> >>>>>> +
> >>>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>>>> +    if (sink_pad == MALI_C55_RZR_SINK_PAD) {
> >>>> 
> >>>> The selection here should depend on format->pad, not the active sink
> >>>> pad.
> >>>>
> >>>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>>>> +
> >>>>>> +        rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> >>>>>> +        rect->left = 0;
> >>>>>> +        rect->top = 0;
> >>>>>> +        rect->width = fmt->width;
> >>>>>> +        rect->height = fmt->height;
> >>>>>> +
> >>>>>> +        rect = v4l2_subdev_state_get_compose(state,
> >>>>>> +                             MALI_C55_RZR_SINK_PAD);
> >>>>>> +        rect->left = 0;
> >>>>>> +        rect->top = 0;
> >>>>>> +        rect->width = fmt->width;
> >>>>>> +        rect->height = fmt->height;
> >>>>>> +    } else {
> >>>>>> +        /*
> >>>>>> +         * Make sure the media bus code is one of the supported
> >>>>>> +         * ISP input media bus codes.
> >>>>>> +         */
> >>>>>> +        if (!mali_c55_isp_is_format_supported(fmt->code))
> >>>>>> +            fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    *v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
> >>>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
> >>>> 
> >>>> Propagation to the source pad, however, should depend on the active
> >>>> route. If format->pad is routed to the source pad, you should propagate,
> >>>> otherwise, you shouldn't.
> >>>>
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> >>>>>> +                       struct v4l2_subdev_state *state,
> >>>>>> +                       struct v4l2_subdev_format *format)
> >>>>>> +{
> >>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>>>> +                            sd);
> >>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
> >>>>>> +    struct v4l2_rect *crop, *compose;
> >>>>>> +    unsigned int sink_pad;
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>>>> +    sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
> >>>>>> +    compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
> >>>>>> +
> >>>>>> +    /* FR Bypass pipe. */
> >>>>>> +
> >>>>>> +    if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >>>>>> +        /*
> >>>>>> +         * Format on the source pad is the same as the one on the
> >>>>>> +         * sink pad, downshifted to 16-bits.
> >>>>>> +         */
> >>>>>> +        fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> >>>>>> +        if (!fmt->code)
> >>>>>> +            return -EINVAL;
> >>>>>> +
> >>>>>> +        /* RAW bypass disables scaling and cropping. */
> >>>>>> +        crop->top = compose->top = 0;
> >>>>>> +        crop->left = compose->left = 0;
> >>>>>> +        fmt->width = crop->width = compose->width = sink_fmt->width;
> >>>>>> +        fmt->height = crop->height = compose->height = sink_fmt->height;
> >>>> 
> >>>> I don't think this is right. This function sets the format on the source
> >>>> pad. Subdevs should propagate formats from the sink to the source, not
> >>>> the other way around.
> >>>>
> >>>> The only parameter that can be modified on the source pad (as far as I
> >>>> understand) is the media bus code. In the bypass path, I understand it's
> >>>> fixed, while in the other path, you can select between RGB and YUV. I
> >>>> think the following code is what you need to implement this function.
> >>>>
> >>>> static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> >>>>                         struct v4l2_subdev_state *state,
> >>>>                         struct v4l2_subdev_format *format)
> >>>> {
> >>>>      struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>>                              sd);
> >>>>      struct v4l2_mbus_framefmt *fmt;
> >>>>
> >>>>      fmt = v4l2_subdev_state_get_format(state, format->pad);
> >>>>
> >>>>      /* In the non-bypass path the output format can be selected. */
> >>>>      if (mali_c55_rzr_get_active_sink(state) == MALI_C55_RZR_SINK_PAD) {
> >>>>          unsigned int i;
> >>>>
> >>>>          fmt->code = format->format.code;
> >>>>
> >>>>          for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> >>>>              if (fmt->code == rzr_non_bypass_src_fmts[i])
> >>>>                  break;
> >>>>          }
> >>>>
> >>>>          if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts))
> >>>>              fmt->code = rzr_non_bypass_src_fmts[0];
> >>>>      }
> >>>>
> >>>>      format->format = *fmt;
> >>>>
> >>>>      return 0;
> >>>> }
> >>>>
> >>>>>> +
> >>>>>> +        *v4l2_subdev_state_get_format(state,
> >>>>>> +                          MALI_C55_RZR_SOURCE_PAD) = *fmt;
> >>>>>> +
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* Regular processing pipe. */
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> >>>>>> +        if (fmt->code == rzr_non_bypass_src_fmts[i])
> >>>>>> +            break;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
> >>>>>> +        dev_dbg(rzr->mali_c55->dev,
> >>>>>> +            "Unsupported mbus code 0x%x: using default\n",
> >>>>>> +            fmt->code);
> >>>> 
> >>>> I think you can drop this message.
> >>>>
> >>>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * The source pad format size comes directly from the sink pad
> >>>>>> +     * compose rectangle.
> >>>>>> +     */
> >>>>>> +    fmt->width = compose->width;
> >>>>>> +    fmt->height = compose->height;
> >>>>>> +
> >>>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
> >>>>>> +                struct v4l2_subdev_state *state,
> >>>>>> +                struct v4l2_subdev_format *format)
> >>>>>> +{
> >>>>>> +    /*
> >>>>>> +     * On sink pads fmt is either fixed for the 'regular' processing
> >>>>>> +     * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
> >>>>>> +     * pad.
> >>>>>> +     *
> >>>>>> +     * On source pad sizes are the result of crop+compose on the sink
> >>>>>> +     * pad sizes, while the format depends on the active route.
> >>>>>> +     */
> >>>>>> +
> >>>>>> +    if (format->pad != MALI_C55_RZR_SOURCE_PAD)
> >>>>>> +        return mali_c55_rzr_set_sink_fmt(sd, state, format);
> >>>>>> +
> >>>>>> +    return mali_c55_rzr_set_source_fmt(sd, state, format);
> >>>> 
> >>>> Nitpicking,
> >>>>
> >>>>      if (format->pad == MALI_C55_RZR_SOURCE_PAD)
> >>>>          return mali_c55_rzr_set_source_fmt(sd, state, format);
> >>>>
> >>>>      return mali_c55_rzr_set_sink_fmt(sd, state, format);
> >>>>
> >>>> to match SOURCE_PAD and source_fmt.
> >>>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
> >>>>>> +                      struct v4l2_subdev_state *state,
> >>>>>> +                      struct v4l2_subdev_selection *sel)
> >>>>>> +{
> >>>>>> +    if (sel->pad != MALI_C55_RZR_SINK_PAD)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    if (sel->target != V4L2_SEL_TGT_CROP &&
> >>>>>> +        sel->target != V4L2_SEL_TGT_COMPOSE)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    sel->r = sel->target == V4L2_SEL_TGT_CROP
> >>>>>> +           ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
> >>>>>> +           : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
> >>>>>> +                      struct v4l2_subdev_state *state,
> >>>>>> +                      struct v4l2_subdev_selection *sel)
> >>>>>> +{
> >>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>>>> +                            sd);
> >>>>>> +    struct v4l2_mbus_framefmt *source_fmt;
> >>>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
> >>>>>> +    struct v4l2_rect *crop, *compose;
> >>>>>> +
> >>>>>> +    if (sel->pad != MALI_C55_RZR_SINK_PAD)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    if (sel->target != V4L2_SEL_TGT_CROP &&
> >>>>>> +        sel->target != V4L2_SEL_TGT_COMPOSE)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    source_fmt = v4l2_subdev_state_get_format(state,
> >>>>>> +                          MALI_C55_RZR_SOURCE_PAD);
> >>>>>> +    sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> >>>>>> +    compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> >>>>>> +
> >>>>>> +    /* RAW bypass disables crop/scaling. */
> >>>>>> +    if (mali_c55_format_is_raw(source_fmt->code)) {
> >>>>>> +        crop->top = compose->top = 0;
> >>>>>> +        crop->left = compose->left = 0;
> >>>>>> +        crop->width = compose->width = sink_fmt->width;
> >>>>>> +        crop->height = compose->height = sink_fmt->height;
> >>>>>> +
> >>>>>> +        sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> >>>>>> +
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* During streaming, it is allowed to only change the crop rectangle. */
> >>>>>> +    if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +     /*
> >>>>>> +      * Update the desired target and then clamp the crop rectangle to the
> >>>>>> +      * sink format sizes and the compose size to the crop sizes.
> >>>>>> +      */
> >>>>>> +    if (sel->target == V4L2_SEL_TGT_CROP)
> >>>>>> +        *crop = sel->r;
> >>>>>> +    else
> >>>>>> +        *compose = sel->r;
> >>>>>> +
> >>>>>> +    clamp_t(unsigned int, crop->left, 0, sink_fmt->width);
> >>>>>> +    clamp_t(unsigned int, crop->top, 0, sink_fmt->height);
> >>>>>> +    clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
> >>>>>> +        sink_fmt->width - crop->left);
> >>>>>> +    clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
> >>>>>> +        sink_fmt->height - crop->top);
> >>>>>> +
> >>>>>> +    if (rzr->streaming) {
> >>>>>> +        /*
> >>>>>> +         * Apply at runtime a crop rectangle on the resizer's sink only
> >>>>>> +         * if it doesn't require re-programming the scaler output sizes
> >>>>>> +         * as it would require changing the output buffer sizes as well.
> >>>>>> +         */
> >>>>>> +        if (sel->r.width < compose->width ||
> >>>>>> +            sel->r.height < compose->height)
> >>>>>> +            return -EINVAL;
> >>>>>> +
> >>>>>> +        *crop = sel->r;
> >>>>>> +        mali_c55_rzr_program(rzr, state);
> >>>>>> +
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    compose->left = 0;
> >>>>>> +    compose->top = 0;
> >>>>>> +    clamp_t(unsigned int, compose->left, 0, sink_fmt->width);
> >>>>>> +    clamp_t(unsigned int, compose->top, 0, sink_fmt->height);
> >>>>>> +    clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
> >>>>>> +    clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
> >>>>>> +
> >>>>>> +    sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> >>>>>> +                    struct v4l2_subdev_state *state,
> >>>>>> +                    enum v4l2_subdev_format_whence which,
> >>>>>> +                    struct v4l2_subdev_krouting *routing)
> >>>>>> +{
> >>>>>> +    if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
> >>>>>> +        media_entity_is_streaming(&sd->entity))
> >>>>>> +        return -EBUSY;
> >>>>>> +
> >>>>>> +    return __mali_c55_rzr_set_routing(sd, state, routing);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
> >>>>>> +    .enum_mbus_code        = mali_c55_rzr_enum_mbus_code,
> >>>>>> +    .enum_frame_size    = mali_c55_rzr_enum_frame_size,
> >>>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
> >>>>>> +    .set_fmt        = mali_c55_rzr_set_fmt,
> >>>>>> +    .get_selection        = mali_c55_rzr_get_selection,
> >>>>>> +    .set_selection        = mali_c55_rzr_set_selection,
> >>>>>> +    .set_routing        = mali_c55_rzr_set_routing,
> >>>>>> +};
> >>>>>> +
> >>>>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
> >>>> 
> >>>> Could this be handled through the .enable_streams() and
> >>>> .disable_streams() operations ? They ensure that the stream state stored
> >>>> internal is correct. That may not matter much today, but I think it will
> >>>> become increasingly important in the future for the V4L2 core.
> >>>>
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>>>> +    struct v4l2_subdev *sd = &rzr->sd;
> >>>>>> +    struct v4l2_subdev_state *state;
> >>>>>> +    unsigned int sink_pad;
> >>>>>> +
> >>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>>>> +
> >>>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>>>> +    if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >>>>>> +        /* Bypass FR pipe processing if the bypass route is active. */
> >>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >>>>>> + MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
> >>>>>> + MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
> >>>>>> +        goto unlock_state;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* Disable bypass and use regular processing. */
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >>>>>> +                 MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
> >>>>>> +    mali_c55_rzr_program(rzr, state);
> >>>>>> +
> >>>>>> +unlock_state:
> >>>>>> +    rzr->streaming = true;
> >>>> 
> >>>> And hopefully you'll be able to replace this with
> >>>> v4l2_subdev_is_streaming(), introduced in "[PATCH v6 00/11] media:
> >>>> subdev: Improve stream enable/disable machinery" (Sakari has sent a pull
> >>>> request for v6.11 yesterday).
> >>>>
> >>>>>> +    v4l2_subdev_unlock_state(state);
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
> >>>>>> +{
> >>>>>> +    struct v4l2_subdev *sd = &rzr->sd;
> >>>>>> +    struct v4l2_subdev_state *state;
> >>>>>> +
> >>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>>>> +    rzr->streaming = false;
> >>>>>> +    v4l2_subdev_unlock_state(state);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
> >>>>>> +    .pad    = &mali_c55_resizer_pad_ops,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
> >>>>>> +                   struct v4l2_subdev_state *state)
> >>>>>> +{
> >>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>>>> +                            sd);
> >>>>>> +    struct v4l2_subdev_krouting routing = { };
> >>>>>> +    struct v4l2_subdev_route *routes;
> >>>>>> +    unsigned int i;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
> >>>>>> +    if (!routes)
> >>>>>> +        return -ENOMEM;
> >>>>>> +
> >>>>>> +    for (i = 0; i < rzr->num_routes; ++i) {
> >>>>>> +        struct v4l2_subdev_route *route = &routes[i];
> >>>>>> +
> >>>>>> +        route->sink_pad = i
> >>>>>> +                ? MALI_C55_RZR_SINK_BYPASS_PAD
> >>>>>> +                : MALI_C55_RZR_SINK_PAD;
> >>>>>> +        route->source_pad = MALI_C55_RZR_SOURCE_PAD;
> >>>>>> +        if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
> >>>>>> +            route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    routing.num_routes = rzr->num_routes;
> >>>>>> +    routing.routes = routes;
> >>>>>> +
> >>>>>> +    ret = __mali_c55_rzr_set_routing(sd, state, &routing);
> >>>>>> +    kfree(routes);
> >>>>>> +
> >>>>>> +    return ret;
> >>>> 
> >>>> I think this could be simplified.
> >>>>
> >>>>      struct v4l2_subdev_route routes[2] = {
> >>>>          {
> >>>>              .sink_pad = MALI_C55_RZR_SINK_PAD,
> >>>>              .source_pad = MALI_C55_RZR_SOURCE_PAD,
> >>>>              .flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> >>>>          }, {
> >>>>              .sink_pad = MALI_C55_RZR_SINK_BYPASS_PAD,
> >>>>              .source_pad = MALI_C55_RZR_SOURCE_PAD,
> >>>>          },
> >>>>      };
> >>>>      struct v4l2_subdev_krouting routing = {
> >>>>          .num_routes = rzr->num_routes,
> >>>>          .routes = routes,
> >>>>      };
> >>>>
> >>>>      return __mali_c55_rzr_set_routing(sd, state, &routing);
> >>>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
> >>>>>> +    .init_state = mali_c55_rzr_init_state,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
> >>>>>> +                          unsigned int index)
> >>>>>> +{
> >>>>>> +    const unsigned int scaler_filt_coefmem_addrs[][2] = {
> >>>>>> +        [MALI_C55_RZR_FR] = {
> >>>>>> +            0x034A8, /* hfilt */
> >>>>>> +            0x044A8  /* vfilt */
> >>>>> 
> >>>>> Lowercase hex constants.
> >>>> 
> >>>> And addresses belong to the mali-c55-registers.h file.
> >>>>
> >>>>>> +        },
> >>>>>> +        [MALI_C55_RZR_DS] = {
> >>>>>> +            0x014A8, /* hfilt */
> >>>>>> +            0x024A8  /* vfilt */
> >>>>>> +        },
> >>>>>> +    };
> >>>>>> +    unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
> >>>>>> +    unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
> >>>>>> +    unsigned int i, j;
> >>>>>> +
> >>>>>> +    for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
> >>>>>> +        for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
> >>>>>> +            mali_c55_write(mali_c55, haddr,
> >>>>>> + mali_c55_scaler_h_filter_coefficients[i][j]);
> >>>>>> +            mali_c55_write(mali_c55, vaddr,
> >>>>>> + mali_c55_scaler_v_filter_coefficients[i][j]);
> >>>>>> +
> >>>>>> +            haddr += sizeof(u32);
> >>>>>> +            vaddr += sizeof(u32);
> >>>>>> +        }
> >>>>>> +    }
> >>>> 
> >>>> How about memcpy_toio() ? I suppose this function isn't
> >>>> performance sensitive, so maybe usage of mali_c55_write() is better from
> >>>> a consistency point of view.
> >>>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    unsigned int i;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
> >>>> 
> >>>> Moving the inner content to a separate mali_c55_register_resizer()
> >>>> function would increase readability I think, and remove usage of gotos.
> >>>> I would probably do the same for unregistration too, for consistency.
> >>>>
> >>>>>> +        struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> >>>>>> +        struct v4l2_subdev *sd = &rzr->sd;
> >>>>>> +        unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
> >>>>>> +
> >>>>>> +        rzr->id = i;
> >>>>>> +        rzr->streaming = false;
> >>>>>> +
> >>>>>> +        if (rzr->id == MALI_C55_RZR_FR)
> >>>>>> +            rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
> >>>>>> +        else
> >>>>>> +            rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
> >>>>>> +
> >>>>>> +        mali_c55_resizer_program_coefficients(mali_c55, i);
> >>>> 
> >>>> Should this be done at stream start, given that power may be cut off
> >>>> between streaming sessions ?
> >>>>
> >>>>>> +
> >>>>>> +        v4l2_subdev_init(sd, &mali_c55_resizer_ops);
> >>>>>> +        sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
> >>>>>> +                 | V4L2_SUBDEV_FL_STREAMS;
> >>>>>> +        sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
> >>>>>> +        sd->internal_ops = &mali_c55_resizer_internal_ops;
> >>>>>> +        snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
> >>>> 
> >>>>          snprintf(sd->name, ARRAY_SIZE(sd->name), "%s resizer %s",
> >>>>
> >>>> and drop the "resizer " prefix from mali_c55_resizer_names. You can also
> >>>> make mali_c55_resizer_names a local static const variable.
> >>>>
> >>>>>> +             MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
> >>>>>> +
> >>>>>> +        rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
> >>>>>> +        rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
> >>>>>> +
> >>>>>> +        /* Only the FR pipe has a bypass pad. */
> >>>>>> +        if (rzr->id == MALI_C55_RZR_FR) {
> >>>>>> + rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
> >>>>>> +                            MEDIA_PAD_FL_SINK;
> >>>>>> +            rzr->num_routes = 2;
> >>>>>> +        } else {
> >>>>>> +            num_pads -= 1;
> >>>>>> +            rzr->num_routes = 1;
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
> >>>>>> +        if (ret)
> >>>>>> +            return ret;
> >>>>>> +
> >>>>>> +        ret = v4l2_subdev_init_finalize(sd);
> >>>>>> +        if (ret)
> >>>>>> +            goto err_cleanup;
> >>>>>> +
> >>>>>> +        ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >>>>>> +        if (ret)
> >>>>>> +            goto err_cleanup;
> >>>>>> +
> >>>>>> +        rzr->mali_c55 = mali_c55;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_cleanup:
> >>>>>> +    for (; i >= 0; --i) {
> >>>>>> +        struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> >>>>>> +        struct v4l2_subdev *sd = &rzr->sd;
> >>>>>> +
> >>>>>> +        v4l2_subdev_cleanup(sd);
> >>>>>> +        media_entity_cleanup(&sd->entity);
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
> >>>>>> +        struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
> >>>>>> +
> >>>>>> +        if (!resizer->mali_c55)
> >>>>>> +            continue;
> >>>>>> +
> >>>>>> +        v4l2_device_unregister_subdev(&resizer->sd);
> >>>>>> +        v4l2_subdev_cleanup(&resizer->sd);
> >>>>>> +        media_entity_cleanup(&resizer->sd.entity);
> >>>>>> +    }
> >>>>>> +}
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..c7e699741c6d
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >>>>>> @@ -0,0 +1,402 @@
> >>>>>> +// SPDX-License-Identifier: GPL-2.0
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Test pattern generator
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#include <linux/minmax.h>
> >>>>>> +#include <linux/string.h>
> >>>>>> +
> >>>>>> +#include <media/media-entity.h>
> >>>>>> +#include <media/v4l2-ctrls.h>
> >>>>>> +#include <media/v4l2-subdev.h>
> >>>>>> +
> >>>>>> +#include "mali-c55-common.h"
> >>>>>> +#include "mali-c55-registers.h"
> >>>>>> +
> >>>>>> +#define MALI_C55_TPG_SRC_PAD        0
> >>>>>> +#define MALI_C55_TPG_FIXED_HBLANK    0x20
> >>>>>> +#define MALI_C55_TPG_MAX_VBLANK        0xFFFF
> >>>>> 
> >>>>> Lowercase hex constants.
> >>>>>
> >>>>>> +#define MALI_C55_TPG_PIXEL_RATE        100000000
> >>>>> 
> >>>>> This should be exposed to applications using the V4L2_CID_PIXEL_RATE
> >>>>> control (read-only).
> >>>>>
> >>>>>> +
> >>>>>> +static const char * const mali_c55_tpg_test_pattern_menu[] = {
> >>>>>> +    "Flat field",
> >>>>>> +    "Horizontal gradient",
> >>>>>> +    "Vertical gradient",
> >>>>>> +    "Vertical bars",
> >>>>>> +    "Arbitrary rectangle",
> >>>>>> +    "White frame on black field"
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const u32 mali_c55_tpg_mbus_codes[] = {
> >>>>>> +    MEDIA_BUS_FMT_SRGGB20_1X20,
> >>>>>> +    MEDIA_BUS_FMT_RGB202020_1X60,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
> >>>>>> +                       int *def_vblank, int *min_vblank)
> >>>>> 
> >>>>> unsigned int ?
> >>>>>
> >>>>>> +{
> >>>>>> +    unsigned int hts;
> >>>>>> +    int tgt_fps;
> >>>>>> +    int vblank;
> >>>>>> +
> >>>>>> +    hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * The ISP has minimum vertical blanking requirements that must be
> >>>>>> +     * adhered to by the TPG. The minimum is a function of the Iridix blocks
> >>>>>> +     * clocking requirements and the width of the image and horizontal
> >>>>>> +     * blanking, but if we assume the worst case iVariance and sVariance
> >>>>>> +     * values then it boils down to the below.
> >>>>>> +     */
> >>>>>> +    *min_vblank = 15 + (120500 / hts);
> >>>>> 
> >>>>> I wonder if this should round up.
> >>>>>
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * We need to set a sensible default vblank for whatever format height
> >>>>>> +     * we happen to be given from set_fmt(). This function just targets
> >>>>>> +     * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
> >>>>>> +     * If we can't get 5fps we'll take whatever the minimum vblank gives us.
> >>>>>> +     */
> >>>>>> +    tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
> >>>>>> +
> >>>>>> +    if (tgt_fps < 5)
> >>>>>> +        vblank = *min_vblank;
> >>>>>> +    else
> >>>>>> +        vblank = MALI_C55_TPG_PIXEL_RATE / hts
> >>>>>> +               / max(rounddown(tgt_fps, 15), 5);
> >>>>>> +
> >>>>>> +    *def_vblank = ALIGN_DOWN(vblank, 2) - format->height;
> >>>>> 
> >>>>> "vblank = vblank - height" doesn't seem right. The "else" branch stores
> >>>>> a vts in vblank, which doesn't seem right either. Maybe you meant
> >>>>> something like
> >>>>>
> >>>>>      if (tgt_fps < 5)
> >>>>>          def_vts = *min_vblank + format->height;
> >>>>>      else
> >>>>>          def_vts = MALI_C55_TPG_PIXEL_RATE / hts
> >>>>>              / max(rounddown(tgt_fps, 15), 5);
> >>>>>
> >>>>>      *def_vblank = ALIGN_DOWN(def_vts, 2) - format->height;
> >>>>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
> >>>>>> +{
> >>>>>> +    struct mali_c55_tpg *tpg = container_of(ctrl->handler,
> >>>>>> +                        struct mali_c55_tpg,
> >>>>>> +                        ctrls.handler);
> >>>>>> +    struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
> >>>>>> +
> >>>>> 
> >>>>> Should you return here if the pipeline isn't streaming ?
> >>>>>
> >>>>>> +    switch (ctrl->id) {
> >>>>>> +    case V4L2_CID_TEST_PATTERN:
> >>>>>> +        mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
> >>>>>> +                   ctrl->val);
> >>>>>> +        break;
> >>>>>> +    case V4L2_CID_VBLANK:
> >>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
> >>>>>> +                     MALI_C55_REG_VBLANK_MASK, ctrl->val);
> >>>>>> +        break;
> >>>>>> +    default:
> >>>>>> +        dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
> >>>>>> +        return -EINVAL;
> >>>>> 
> >>>>> Can this happen ?
> >>>>>
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
> >>>>>> +    .s_ctrl = &mali_c55_tpg_s_ctrl,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
> >>>>>> +                   struct v4l2_subdev *sd)
> >>>>>> +{
> >>>>>> +    struct v4l2_subdev_state *state;
> >>>>>> +    struct v4l2_mbus_framefmt *fmt;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * hblank needs setting, but is a read-only control and thus won't be
> >>>>>> +     * called during __v4l2_ctrl_handler_setup(). Do it here instead.
> >>>>>> +     */
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
> >>>>>> +                 MALI_C55_REG_HBLANK_MASK,
> >>>>>> +                 MALI_C55_TPG_FIXED_HBLANK);
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >>>>>> +                 MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
> >>>>>> +
> >>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>>>> +
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >>>>>> +                 MALI_C55_TEST_PATTERN_RGB_MASK,
> >>>>>> +                 fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
> >>>>>> +                      0x01 : 0x0);
> >>>>>> +
> >>>>>> +    v4l2_subdev_unlock_state(state);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
> >>>>>> +{
> >>>>>> +    struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
> >>>>>> +    struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
> >>>>>> +
> >>>>>> +    if (!enable) {
> >>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >>>>>> +                MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
> >>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >>>>>> +                MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * One might reasonably expect the framesize to be set here
> >>>>>> +     * given it's configurable in .set_fmt(), but it's done in the
> >>>>>> +     * ISP subdevice's stream on func instead, as the same register
> >>>>> 
> >>>>> s/func/function/
> >>>>>
> >>>>>> +     * is also used to indicate the size of the data coming from the
> >>>>>> +     * sensor.
> >>>>>> +     */
> >>>>>> +    mali_c55_tpg_configure(mali_c55, sd);
> >>>>> 
> >>>>>      mali_c55_tpg_configure(tpg);
> >>>>>
> >>>>>> + __v4l2_ctrl_handler_setup(sd->ctrl_handler);
> >>>>>> +
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >>>>>> +                 MALI_C55_TEST_PATTERN_ON_OFF,
> >>>>>> +                 MALI_C55_TEST_PATTERN_ON_OFF);
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >>>>>> +                 MALI_C55_REG_GEN_VIDEO_ON_MASK,
> >>>>>> +                 MALI_C55_REG_GEN_VIDEO_ON_MASK);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
> >>>>>> +    .s_stream = &mali_c55_tpg_s_stream,
> >>>>> 
> >>>>> Can we use .enable_streams() and .disable_streams() ?
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>> +                       struct v4l2_subdev_state *state,
> >>>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
> >>>>>> +{
> >>>>>> +    if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    code->code = mali_c55_tpg_mbus_codes[code->index];
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
> >>>>>> +                    struct v4l2_subdev_state *state,
> >>>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
> >>>>>> +{
> >>>>> 
> >>>>> You sohuld verify here that fse->code is a supported value and return
> >>>>> -EINVAL otherwise.
> >>>>>
> >>>>>> +    if (fse->index > 0 || fse->pad > sd->entity.num_pads)
> >>>>> 
> >>>>> Drop the pad check, it's done in the subdev core already.
> >>>>>
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
> >>>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
> >>>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
> >>>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
> >>>>>> +                struct v4l2_subdev_state *state,
> >>>>>> +                struct v4l2_subdev_format *format)
> >>>>>> +{
> >>>>>> +    struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
> >>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>>>> +    int vblank_def, vblank_min;
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> >>>>>> +        if (fmt->code == mali_c55_tpg_mbus_codes[i])
> >>>>>> +            break;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * The TPG says that the test frame timing generation logic expects a
> >>>>>> +     * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
> >>>>>> +     * handle anything smaller than 128x128 it seems pointless to allow a
> >>>>>> +     * smaller frame.
> >>>>>> +     */
> >>>>>> +    clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> >>>>>> +        MALI_C55_MAX_WIDTH);
> >>>>>> +    clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> >>>>>> +        MALI_C55_MAX_HEIGHT);
> >>>>>> +
> >>>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;
> >>>>> 
> >>>>> You're allowing userspace to set fmt->field, as well as all the
> >>>>> colorspace parameters, to random values. I would instead do something
> >>>>> like
> >>>>>
> >>>>>      for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> >>>>>          if (format->format.code == mali_c55_tpg_mbus_codes[i])
> >>>>>              break;
> >>>>>      }
> >>>>>
> >>>>>      if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>>>>          format->format.code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>>>
> >>>>>      format->format.width = clamp(format->format.width,
> >>>>>                       MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>>>>      format->format.height = clamp(format->format.height,
> >>>>>                        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>>>>
> >>>>>      fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>>>      fmt->code = format->format.code;
> >>>>>      fmt->width = format->format.width;
> >>>>>      fmt->height = format->format.height;
> >>>>>
> >>>>>      format->format = *fmt;
> >>>>>
> >>>>> Alternatively (which I think I like better),
> >>>>>
> >>>>>      fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>>>
> >>>>>      fmt->code = format->format.code;
> >>>>>
> >>>>>      for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> >>>>>          if (fmt->code == mali_c55_tpg_mbus_codes[i])
> >>>>>              break;
> >>>>>      }
> >>>>>
> >>>>>      if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>>>>          fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>>>
> >>>>>      fmt->width = clamp(format->format.width,
> >>>>>                 MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>>>>      fmt->height = clamp(format->format.height,
> >>>>>                  MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>>>>
> >>>>>      format->format = *fmt;
> >>>>>
> >>>>>> +
> >>>>>> +    if (format->which == V4L2_SUBDEV_FORMAT_TRY)
> >>>>>> +        return 0;
> >>>>>> +
> >>>>>> +    __mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
> >>>>>> +    __v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
> >>>>>> +                 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
> >>>>>> +    __v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);
> >>>>> 
> >>>>> Move those three calls to a separate function, it will be reused below.
> >>>>> I'd name is mali_c55_tpg_update_vblank(). You can fold
> >>>>> __mali_c55_tpg_calc_vblank() in it.
> >>>>>
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
> >>>>>> +    .enum_mbus_code        = mali_c55_tpg_enum_mbus_code,
> >>>>>> +    .enum_frame_size    = mali_c55_tpg_enum_frame_size,
> >>>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
> >>>>>> +    .set_fmt        = mali_c55_tpg_set_fmt,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
> >>>>>> +    .video    = &mali_c55_tpg_video_ops,
> >>>>>> +    .pad    = &mali_c55_tpg_pad_ops,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
> >>>>>> +                   struct v4l2_subdev_state *sd_state)
> >>>>> 
> >>>>> You name this variable state in every other subdev operation handler.
> >>>>>
> >>>>>> +{
> >>>>>> +    struct v4l2_mbus_framefmt *fmt =
> >>>>>> +        v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
> >>>>>> +
> >>>>>> +    fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>>>> +    fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>>>> +    fmt->field = V4L2_FIELD_NONE;
> >>>>>> +    fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>>> 
> >>>>> Initialize the colorspace fields too.
> >>>>>
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
> >>>>>> +    .init_state = mali_c55_tpg_init_state,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
> >>>>>> +    struct v4l2_subdev *sd = &mali_c55->tpg.sd;
> >>>>>> +    struct v4l2_mbus_framefmt *format;
> >>>>>> +    struct v4l2_subdev_state *state;
> >>>>>> +    int vblank_def, vblank_min;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>>>> +    format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>>>> +
> >>>>>> +    ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);
> >>>>> 
> >>>>> You have 3 controls.
> >>>>>
> >>>>>> +    if (ret)
> >>>>>> +        goto err_unlock;
> >>>>>> +
> >>>>>> +    ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
> >>>>>> +                &mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
> >>>>>> +                ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
> >>>>>> +                0, 3, mali_c55_tpg_test_pattern_menu);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * We fix hblank at the minimum allowed value and control framerate
> >>>>>> +     * solely through the vblank control.
> >>>>>> +     */
> >>>>>> +    ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
> >>>>>> +                &mali_c55_tpg_ctrl_ops,
> >>>>>> +                V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
> >>>>>> +                MALI_C55_TPG_FIXED_HBLANK, 1,
> >>>>>> +                MALI_C55_TPG_FIXED_HBLANK);
> >>>>>> +    if (ctrls->hblank)
> >>>>>> +        ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
> >>>>>> +
> >>>>>> +    __mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);
> >>>>> 
> >>>>> Drop this and initialize the control with default values. You can then
> >>>>> update the value by calling mali_c55_tpg_update_vblank() in
> >>>>> mali_c55_register_tpg().
> >>>>>
> >>>>> The reason is to share the same mutex between the control handler and
> >>>>> the subdev active state without having to add a separate mutex in the
> >>>>> mali_c55_tpg structure. The simplest way to do so is to initialize the
> >>>>> controls first, set sd->state_lock to point to the control handler lock,
> >>>>> and call v4l2_subdev_init_finalize() as the last step. As a consequence,
> >>>>> you can't access the active state when initializing controls.
> >>>>>
> >>>>> You can alternatively keep the lock in mali_c55_tpg and set
> >>>>> sd->state_lock to point to it, but I think that's more complex.
> >>>>>
> >>>>>> +    ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
> >>>>>> +                      &mali_c55_tpg_ctrl_ops,
> >>>>>> +                      V4L2_CID_VBLANK, vblank_min,
> >>>>>> +                      MALI_C55_TPG_MAX_VBLANK, 1,
> >>>>>> +                      vblank_def);
> >>>>>> +
> >>>>>> +    if (ctrls->handler.error) {
> >>>>>> +        dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
> >>>>>> +        ret = ctrls->handler.error;
> >>>>>> +        goto err_free_handler;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    ctrls->handler.lock = &mali_c55->tpg.lock;
> >>>>> 
> >>>>> Drop this and drop the mutex. The control handler will use its internal
> >>>>> mutex.
> >>>>>
> >>>>>> +    mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
> >>>>>> +
> >>>>>> +    v4l2_subdev_unlock_state(state);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_free_handler:
> >>>>>> +    v4l2_ctrl_handler_free(&ctrls->handler);
> >>>>>> +err_unlock:
> >>>>>> +    v4l2_subdev_unlock_state(state);
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_tpg *tpg = &mali_c55->tpg;
> >>>>>> +    struct v4l2_subdev *sd = &tpg->sd;
> >>>>>> +    struct media_pad *pad = &tpg->pad;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    mutex_init(&tpg->lock);
> >>>>>> +
> >>>>>> +    v4l2_subdev_init(sd, &mali_c55_tpg_ops);
> >>>>>> +    sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> >>>>>> +    sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
> >>>>> 
> >>>>> Should we introduce a TPG function ?
> >>>>>
> >>>>>> +    sd->internal_ops = &mali_c55_tpg_internal_ops;
> >>>>>> +    strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
> >>>>>> +
> >>>>>> +    pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;
> >>>>> 
> >>>>> I don't think MEDIA_PAD_FL_MUST_CONNECT is right.
> >>>>>
> >>>>>> +    ret = media_entity_pads_init(&sd->entity, 1, pad);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev,
> >>>>>> +            "Failed to initialize media entity pads\n");
> >>>>>> +        goto err_destroy_mutex;
> >>>>>> +    }
> >>>>>> +
> >>>>> 
> >>>>>      sd->state_lock = sd->ctrl_handler->lock;
> >>>>>
> >>>>> to use the same lock for the controls and the active state. You need to
> >>>>> move this line and the v4l2_subdev_init_finalize() call after
> >>>>> mali_c55_tpg_init_controls() to get the control handler lock initialized
> >>>>> first.
> >>>>>
> >>>>>> +    ret = v4l2_subdev_init_finalize(sd);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_cleanup_media_entity;
> >>>>>> +
> >>>>>> +    ret = mali_c55_tpg_init_controls(mali_c55);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev,
> >>>>>> +            "Error initialising controls\n");
> >>>>>> +        goto err_cleanup_subdev;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
> >>>>>> +        goto err_free_ctrl_handler;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * By default the colour settings lead to a very dim image that is
> >>>>>> +     * nearly indistinguishable from black on some monitor settings. Ramp
> >>>>>> +     * them up a bit so the image is brighter.
> >>>>>> +     */
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
> >>>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
> >>>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
> >>>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
> >>>>>> +
> >>>>>> +    tpg->mali_c55 = mali_c55;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_free_ctrl_handler:
> >>>>>> +    v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> >>>>>> +err_cleanup_subdev:
> >>>>>> +    v4l2_subdev_cleanup(sd);
> >>>>>> +err_cleanup_media_entity:
> >>>>>> +    media_entity_cleanup(&sd->entity);
> >>>>>> +err_destroy_mutex:
> >>>>>> +    mutex_destroy(&tpg->lock);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_tpg *tpg = &mali_c55->tpg;
> >>>>>> +
> >>>>>> +    if (!tpg->mali_c55)
> >>>>>> +        return;
> >>>>>> +
> >>>>>> +    v4l2_device_unregister_subdev(&tpg->sd);
> >>>>>> +    v4l2_subdev_cleanup(&tpg->sd);
> >>>>>> +    media_entity_cleanup(&tpg->sd.entity);
> >>>>>> +    v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> >>>>> 
> >>>>> Free the control handler just after v4l2_device_unregister_subdev() to
> >>>>> match the order in mali_c55_register_tpg().
> >>>>>
> >>>>>> +    mutex_destroy(&tpg->lock);
> >>>>>> +}

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 05/16] media: mali-c55: Add Mali-C55 ISP driver

[Laurent Pinchart]

Hi Dan,

On Fri, Jun 21, 2024 at 10:28:56AM +0100, Daniel Scally wrote:
> On 20/06/2024 16:23, Laurent Pinchart wrote:
> > On Thu, Jun 20, 2024 at 03:49:23PM +0100, Daniel Scally wrote:
> >> On 20/06/2024 15:33, Dan Scally wrote:
> >>> On 30/05/2024 22:43, Laurent Pinchart wrote:
> >>>> And now the second part of the review, addressing mali-c55-capture.c and
> >>>> mali-c55-resizer.c. I've reviewed the code from the bottom up, so some
> >>>> messages may be repeated in an order that seems weird. Sorry about that.
> >>>>
> >>>> On Thu, May 30, 2024 at 03:15:10AM +0300, Laurent Pinchart wrote:
> >>>>> On Wed, May 29, 2024 at 04:28:47PM +0100, Daniel Scally wrote:
> >>>>>> Add a driver for Arm's Mali-C55 Image Signal Processor. The driver is
> >>>>>> V4L2 and Media Controller compliant and creates subdevices to manage
> >>>>>> the ISP itself, its internal test pattern generator as well as the
> >>>>>> crop, scaler and output format functionality for each of its two
> >>>>>> output devices.
> >>>>>>
> >>>>>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
> >>>>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >>>>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >>>>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> >>>>>> ---
> >>>>>> Changes in v5:
> >>>>>>
> >>>>>>      - Reworked input formats - previously we allowed representing input data
> >>>>>>        as any 8-16 bit format. Now we only allow input data to be represented
> >>>>>>        by the new 20-bit bayer formats, which is corrected to the equivalent
> >>>>>>        16-bit format in RAW bypass mode.
> >>>>>>      - Stopped bypassing blocks that we haven't added supporting parameters
> >>>>>>        for yet.
> >>>>>>      - Addressed most of Sakari's comments from the list
> >>>>>>
> >>>>>> Changes not yet made in v5:
> >>>>>>
> >>>>>>      - The output pipelines can still be started and stopped independently of
> >>>>>>        one another - I'd like to discuss that more.
> >>>>>>      - the TPG subdev still uses .s_stream() - I need to rebase onto a tree
> >>>>>>        with working .enable_streams() for a single-source-pad subdevice.
> >>>>>>
> >>>>>> Changes in v4:
> >>>>>>
> >>>>>>      - Reworked mali_c55_update_bits() to internally perform the bit-shift
> >>>>> 
> >>>>> I really don't like that, it makes the code very confusing, even more so
> >>>>> as it differs from regmap_update_bits().
> >>>>>
> >>>>> Look at this for instance:
> >>>>>
> >>>>>      mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>>>                   MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> >>>>>                   MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> >>>>>
> >>>>> It only works by change because MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK is
> >>>>> BIT(0).
> >>>>>
> >>>>> Sorry, I know it will be painful, but this change needs to be reverted.
> >>>>>
> >>>>>>      - Reworked the resizer to allow cropping during streaming
> >>>>>>      - Fixed a bug in NV12 output
> >>>>>>
> >>>>>> Changes in v3:
> >>>>>>
> >>>>>>      - Mostly minor fixes suggested by Sakari
> >>>>>>      - Fixed the sequencing of vb2 buffers to be synchronised across the two
> >>>>>>        capture devices.
> >>>>>>
> >>>>>> Changes in v2:
> >>>>>>
> >>>>>>      - Clock handling
> >>>>>>      - Fixed the warnings raised by the kernel test robot
> >>>>>>
> >>>>>>    drivers/media/platform/Kconfig                |   1 +
> >>>>>>    drivers/media/platform/Makefile               |   1 +
> >>>>>>    drivers/media/platform/arm/Kconfig            |   5 +
> >>>>>>    drivers/media/platform/arm/Makefile           |   2 +
> >>>>>>    drivers/media/platform/arm/mali-c55/Kconfig   |  18 +
> >>>>>>    drivers/media/platform/arm/mali-c55/Makefile  |   9 +
> >>>>>>    .../platform/arm/mali-c55/mali-c55-capture.c  | 951 ++++++++++++++++++
> >>>>>>    .../platform/arm/mali-c55/mali-c55-common.h   | 266 +++++
> >>>>>>    .../platform/arm/mali-c55/mali-c55-core.c     | 767 ++++++++++++++
> >>>>>>    .../platform/arm/mali-c55/mali-c55-isp.c      | 611 +++++++++++
> >>>>>>    .../arm/mali-c55/mali-c55-registers.h         | 258 +++++
> >>>>>>    .../arm/mali-c55/mali-c55-resizer-coefs.h     | 382 +++++++
> >>>>>>    .../platform/arm/mali-c55/mali-c55-resizer.c  | 779 ++++++++++++++
> >>>>>>    .../platform/arm/mali-c55/mali-c55-tpg.c      | 402 ++++++++
> >>>>>>    14 files changed, 4452 insertions(+)
> >>>>>>    create mode 100644 drivers/media/platform/arm/Kconfig
> >>>>>>    create mode 100644 drivers/media/platform/arm/Makefile
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/Kconfig
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/Makefile
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >>>>> 
> >>>>> I've skipped review of capture.c and resizer.c as I already have plenty
> >>>>> of comments for the other files, and it's getting late. I'll try to
> >>>>> review the rest tomorrow.
> >>>>>
> >>>>>> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> >>>>>> index 2d79bfc68c15..c929169766aa 100644
> >>>>>> --- a/drivers/media/platform/Kconfig
> >>>>>> +++ b/drivers/media/platform/Kconfig
> >>>>>> @@ -65,6 +65,7 @@ config VIDEO_MUX
> >>>>>>    source "drivers/media/platform/allegro-dvt/Kconfig"
> >>>>>>    source "drivers/media/platform/amlogic/Kconfig"
> >>>>>>    source "drivers/media/platform/amphion/Kconfig"
> >>>>>> +source "drivers/media/platform/arm/Kconfig"
> >>>>>>    source "drivers/media/platform/aspeed/Kconfig"
> >>>>>>    source "drivers/media/platform/atmel/Kconfig"
> >>>>>>    source "drivers/media/platform/broadcom/Kconfig"
> >>>>>> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> >>>>>> index da17301f7439..9a647abd5218 100644
> >>>>>> --- a/drivers/media/platform/Makefile
> >>>>>> +++ b/drivers/media/platform/Makefile
> >>>>>> @@ -8,6 +8,7 @@
> >>>>>>    obj-y += allegro-dvt/
> >>>>>>    obj-y += amlogic/
> >>>>>>    obj-y += amphion/
> >>>>>> +obj-y += arm/
> >>>>>>    obj-y += aspeed/
> >>>>>>    obj-y += atmel/
> >>>>>>    obj-y += broadcom/
> >>>>>> diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..4f0764c329c7
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/Kconfig
> >>>>>> @@ -0,0 +1,5 @@
> >>>>>> +# SPDX-License-Identifier: GPL-2.0-only
> >>>>>> +
> >>>>>> +comment "ARM media platform drivers"
> >>>>>> +
> >>>>>> +source "drivers/media/platform/arm/mali-c55/Kconfig"
> >>>>>> diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..8cc4918725ef
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/Makefile
> >>>>>> @@ -0,0 +1,2 @@
> >>>>>> +# SPDX-License-Identifier: GPL-2.0-only
> >>>>>> +obj-y += mali-c55/
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/Kconfig
> >>>>>> b/drivers/media/platform/arm/mali-c55/Kconfig
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..602085e28b01
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/Kconfig
> >>>>>> @@ -0,0 +1,18 @@
> >>>>>> +# SPDX-License-Identifier: GPL-2.0-only
> >>>>>> +config VIDEO_MALI_C55
> >>>>>> +    tristate "ARM Mali-C55 Image Signal Processor driver"
> >>>>>> +    depends on V4L_PLATFORM_DRIVERS
> >>>>>> +    depends on VIDEO_DEV && OF
> >>>>>> +    depends on ARCH_VEXPRESS || COMPILE_TEST
> >>>>>> +    select MEDIA_CONTROLLER
> >>>>>> +    select VIDEO_V4L2_SUBDEV_API
> >>>>>> +    select VIDEOBUF2_DMA_CONTIG
> >>>>>> +    select VIDEOBUF2_VMALLOC
> >>>>>> +    select V4L2_FWNODE
> >>>>>> +    select GENERIC_PHY_MIPI_DPHY
> >>>>> 
> >>>>> Alphabetical order ?
> >>>>>
> >>>>>> +    default n
> >>>>> 
> >>>>> That's the default, you don't have to specify ti.
> >>>>>
> >>>>>> +    help
> >>>>>> +      Enable this to support Arm's Mali-C55 Image Signal Processor.
> >>>>>> +
> >>>>>> +      To compile this driver as a module, choose M here: the module
> >>>>>> +      will be called mali-c55.
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile
> >>>>>> b/drivers/media/platform/arm/mali-c55/Makefile
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..77dcb2fbf0f4
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
> >>>>>> @@ -0,0 +1,9 @@
> >>>>>> +# SPDX-License-Identifier: GPL-2.0
> >>>>>> +
> >>>>>> +mali-c55-y := mali-c55-capture.o \
> >>>>>> +          mali-c55-core.o \
> >>>>>> +          mali-c55-isp.o \
> >>>>>> +          mali-c55-tpg.o \
> >>>>>> +          mali-c55-resizer.o
> >>>>> 
> >>>>> Alphabetical order here too.
> >>>>>
> >>>>>> +
> >>>>>> +obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..1d539ac9c498
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-capture.c
> >>>>>> @@ -0,0 +1,951 @@
> >>>>>> +// SPDX-License-Identifier: GPL-2.0
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Video capture devices
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#include <linux/cleanup.h>
> >>>>>> +#include <linux/minmax.h>
> >>>>>> +#include <linux/pm_runtime.h>
> >>>>>> +#include <linux/string.h>
> >>>>>> +#include <linux/videodev2.h>
> >>>>>> +
> >>>>>> +#include <media/v4l2-dev.h>
> >>>>>> +#include <media/v4l2-event.h>
> >>>>>> +#include <media/v4l2-ioctl.h>
> >>>>>> +#include <media/v4l2-subdev.h>
> >>>>>> +#include <media/videobuf2-core.h>
> >>>>>> +#include <media/videobuf2-dma-contig.h>
> >>>>>> +
> >>>>>> +#include "mali-c55-common.h"
> >>>>>> +#include "mali-c55-registers.h"
> >>>>>> +
> >>>>>> +static const struct mali_c55_fmt mali_c55_fmts[] = {
> >>>>>> +    /*
> >>>>>> +     * This table is missing some entries which need further work or
> >>>>>> +     * investigation:
> >>>>>> +     *
> >>>>>> +     * Base mode 1 is a backwards V4L2_PIX_FMT_XRGB32 with no V4L2 equivalent
> >>>>>> +     * Base mode 5 is "Generic Data"
> >>>>>> +     * Base mode 8 is a backwards V4L2_PIX_FMT_XYUV32 - no V4L2 equivalent
> >>>>>> +     * Base mode 9 seems to have no V4L2 equivalent
> >>>>>> +     * Base mode 17, 19 and 20 describe formats which seem to have no V4L2
> >>>>>> +     * equivalent
> >>>>>> +     */
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_ARGB2101010,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
> >>>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_A2R10G10B10,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_RGB565,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
> >>>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_RGB565,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_BGR24,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_RGB121212_1X36,
> >>>>>> +            MEDIA_BUS_FMT_RGB202020_1X60,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_RGB24,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_YUYV,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_YUY2,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_UYVY,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_UYVY,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_Y210,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_Y210,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    /*
> >>>>>> +     * This is something of a hack, the ISP thinks it's running NV12M but
> >>>>>> +     * by setting uv_plane = 0 we simply discard that planes and only output
> >>>>>> +     * the Y-plane.
> >>>>>> +     */
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_GREY,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_NV12M,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT1
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_NV21M,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_YUV10_1X30,
> >>>>>> +        },
> >>>>>> +        .is_raw = false,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_NV12_21,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT2
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    /*
> >>>>>> +     * RAW uncompressed formats are all packed in 16 bpp.
> >>>>>> +     * TODO: Expand this list to encompass all possible RAW formats.
> >>>>>> +     */
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_SRGGB16,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_SRGGB16_1X16,
> >>>>>> +        },
> >>>>>> +        .is_raw = true,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_SBGGR16,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_SBGGR16_1X16,
> >>>>>> +        },
> >>>>>> +        .is_raw = true,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_SGBRG16,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_SGBRG16_1X16,
> >>>>>> +        },
> >>>>>> +        .is_raw = true,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .fourcc = V4L2_PIX_FMT_SGRBG16,
> >>>>>> +        .mbus_codes = {
> >>>>>> +            MEDIA_BUS_FMT_SGRBG16_1X16,
> >>>>>> +        },
> >>>>>> +        .is_raw = true,
> >>>>>> +        .registers = {
> >>>>>> +            .base_mode = MALI_C55_OUTPUT_RAW16,
> >>>>>> +            .uv_plane = MALI_C55_OUTPUT_PLANE_ALT0
> >>>>>> +        }
> >>>>>> +    },
> >>>>>> +};
> >>>>>> +
> >>>>>> +static bool mali_c55_mbus_code_can_produce_fmt(const struct mali_c55_fmt *fmt,
> >>>>>> +                           u32 code)
> >>>>>> +{
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(fmt->mbus_codes); i++) {
> >>>>>> +        if (fmt->mbus_codes[i] == code)
> >>>>>> +            return true;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return false;
> >>>>>> +}
> >>>>>> +
> >>>>>> +bool mali_c55_format_is_raw(unsigned int mbus_code)
> >>>>>> +{
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >>>>>> +        if (mali_c55_fmts[i].mbus_codes[0] == mbus_code)
> >>>>>> +            return mali_c55_fmts[i].is_raw;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return false;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct mali_c55_fmt *mali_c55_format_from_pix(const u32 pixelformat)
> >>>>>> +{
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >>>>>> +        if (mali_c55_fmts[i].fourcc == pixelformat)
> >>>>>> +            return &mali_c55_fmts[i];
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * If we find no matching pixelformat, we'll just default to the first
> >>>>>> +     * one for now.
> >>>>>> +     */
> >>>>>> +
> >>>>>> +    return &mali_c55_fmts[0];
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const char * const capture_device_names[] = {
> >>>>>> +    "mali-c55 fr",
> >>>>>> +    "mali-c55 ds",
> >>>>>> +    "mali-c55 3a stats",
> >>>>>> +    "mali-c55 params",
> >>>> 
> >>>> The last two entries are not used AFAICT, neither here, nor in
> >>>> subsequent patches.
> >>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const char *mali_c55_cap_dev_to_name(struct mali_c55_cap_dev *cap)
> >>>>>> +{
> >>>>>> +    if (cap->reg_offset == MALI_C55_CAP_DEV_FR_REG_OFFSET)
> >>>>>> +        return capture_device_names[0];
> >>>>>> +
> >>>>>> +    if (cap->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
> >>>>>> +        return capture_device_names[1];
> >>>>>> +
> >>>>>> +    return "params/stat not supported yet";
> >>>>>> +}
> >>>> 
> >>>> Use cap_dev->vdev.name instead of mali_c55_cap_dev_to_name(cap_dev) and
> >>>> drop this function.
> >>>>
> >>>>>> +
> >>>>>> +static int mali_c55_link_validate(struct media_link *link)
> >>>>>> +{
> >>>>>> +    struct video_device *vdev =
> >>>>>> + media_entity_to_video_device(link->sink->entity);
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = video_get_drvdata(vdev);
> >>>>>> +    struct v4l2_subdev *sd =
> >>>>>> + media_entity_to_v4l2_subdev(link->source->entity);
> >>>>>> +    const struct v4l2_pix_format_mplane *pix_mp;
> >>>>>> +    const struct mali_c55_fmt *cap_fmt;
> >>>>>> +    struct v4l2_subdev_format sd_fmt = {
> >>>>>> +        .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> >>>>>> +        .pad = link->source->index,
> >>>>>> +    };
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sd_fmt);
> >>>>>> +    if (ret)
> >>>>>> +        return ret;
> >>>>>> +
> >>>>>> +    pix_mp = &cap_dev->mode.pix_mp;
> >>>>>> +    cap_fmt = cap_dev->mode.capture_fmt;
> >>>>>> +
> >>>>>> +    if (sd_fmt.format.width != pix_mp->width ||
> >>>>>> +        sd_fmt.format.height != pix_mp->height) {
> >>>>>> +        dev_dbg(cap_dev->mali_c55->dev,
> >>>>>> +            "link '%s':%u -> '%s':%u not valid: %ux%u != %ux%u\n",
> >>>>>> +            link->source->entity->name, link->source->index,
> >>>>>> +            link->sink->entity->name, link->sink->index,
> >>>>>> +            sd_fmt.format.width, sd_fmt.format.height,
> >>>>>> +            pix_mp->width, pix_mp->height);
> >>>>>> +        return -EPIPE;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    if (!mali_c55_mbus_code_can_produce_fmt(cap_fmt, sd_fmt.format.code)) {
> >>>>>> +        dev_dbg(cap_dev->mali_c55->dev,
> >>>>>> +            "link '%s':%u -> '%s':%u not valid: mbus_code 0x%04x cannot produce pixel format
> >>>>>> %p4cc\n",
> >>>>>> +            link->source->entity->name, link->source->index,
> >>>>>> +            link->sink->entity->name, link->sink->index,
> >>>>>> +            sd_fmt.format.code, &pix_mp->pixelformat);
> >>>>>> +        return -EPIPE;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct media_entity_operations mali_c55_media_ops = {
> >>>>>> +    .link_validate = mali_c55_link_validate,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_vb2_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
> >>>>>> +                    unsigned int *num_planes, unsigned int sizes[],
> >>>>>> +                    struct device *alloc_devs[])
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    if (*num_planes) {
> >>>>>> +        if (*num_planes != cap_dev->mode.pix_mp.num_planes)
> >>>>>> +            return -EINVAL;
> >>>>>> +
> >>>>>> +        for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >>>>>> +            if (sizes[i] < cap_dev->mode.pix_mp.plane_fmt[i].sizeimage)
> >>>>>> +                return -EINVAL;
> >>>>>> +    } else {
> >>>>>> +        *num_planes = cap_dev->mode.pix_mp.num_planes;
> >>>>>> +        for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >>>>>> +            sizes[i] = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_buf_queue(struct vb2_buffer *vb)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
> >>>>>> +    struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> >>>>>> +    struct mali_c55_buffer *buf = container_of(vbuf,
> >>>>>> +                           struct mali_c55_buffer, vb);
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    buf->plane_done[MALI_C55_PLANE_Y] = false;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * If we're in a single-plane format we flag the other plane as done
> >>>>>> +     * already so it's dequeued appropriately later
> >>>>>> +     */
> >>>>>> +    buf->plane_done[MALI_C55_PLANE_UV] = cap_dev->mode.pix_mp.num_planes <= 1;
> >>>>>> +
> >>>>>> +    for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++) {
> >>>>>> +        unsigned long size = cap_dev->mode.pix_mp.plane_fmt[i].sizeimage;
> >>>>>> +
> >>>>>> +        vb2_set_plane_payload(vb, i, size);
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    spin_lock(&cap_dev->buffers.lock);
> >>>>>> +    list_add_tail(&buf->queue, &cap_dev->buffers.queue);
> >>>>>> +    spin_unlock(&cap_dev->buffers.lock);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_buf_init(struct vb2_buffer *vb)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = vb2_get_drv_priv(vb->vb2_queue);
> >>>>>> +    struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> >>>>>> +    struct mali_c55_buffer *buf = container_of(vbuf,
> >>>>>> +                           struct mali_c55_buffer, vb);
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < cap_dev->mode.pix_mp.num_planes; i++)
> >>>>>> +        buf->addrs[i] = vb2_dma_contig_plane_dma_addr(vb, i);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>>>> +
> >>>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
> >>>>>> +
> >>>>>> +    cap_dev->buffers.curr = cap_dev->buffers.next;
> >>>>>> +    cap_dev->buffers.next = NULL;
> >>>>>> +
> >>>>>> +    if (!list_empty(&cap_dev->buffers.queue)) {
> >>>>>> +        struct v4l2_pix_format_mplane *pix_mp;
> >>>>>> +        const struct v4l2_format_info *info;
> >>>>>> +        u32 *addrs;
> >>>>>> +
> >>>>>> +        pix_mp = &cap_dev->mode.pix_mp;
> >>>>>> +        info = v4l2_format_info(pix_mp->pixelformat);
> >>>>>> +
> >>>>>> +        mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
> >>>>>> +        if (cap_dev->mode.capture_fmt->registers.uv_plane)
> >>>>>> +            mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x01);
> >>>>>> +
> >>>>>> +        cap_dev->buffers.next = list_first_entry(&cap_dev->buffers.queue,
> >>>>>> +                             struct mali_c55_buffer,
> >>>>>> +                             queue);
> >>>>>> +        list_del(&cap_dev->buffers.next->queue);
> >>>>>> +
> >>>>>> +        addrs = cap_dev->buffers.next->addrs;
> >>>>>> +        mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_Y_WRITER_BANKS_BASE(cap_dev->reg_offset),
> >>>>>> +            addrs[MALI_C55_PLANE_Y]);
> >>>>>> +        mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_BANKS_BASE(cap_dev->reg_offset),
> >>>>>> +            addrs[MALI_C55_PLANE_UV]);
> >>>>>> +        mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_Y_WRITER_OFFSET(cap_dev->reg_offset),
> >>>>>> +            pix_mp->width * info->bpp[MALI_C55_PLANE_Y]);
> >>>>>> +        mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_OFFSET(cap_dev->reg_offset),
> >>>>>> +            pix_mp->width * info->bpp[MALI_C55_PLANE_UV]
> >>>>>> +            / info->hdiv);
> >>>>>> +    } else {
> >>>>>> +        /*
> >>>>>> +         * If we underflow then we can tell the ISP that we don't want
> >>>>>> +         * to write out the next frame.
> >>>>>> +         */
> >>>>>> +        mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>>>> +        mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>>>> +    }
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_handle_buffer(struct mali_c55_buffer *curr_buf,
> >>>>>> +                   unsigned int framecount)
> >>>>>> +{
> >>>>>> +    curr_buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> >>>>>> +    curr_buf->vb.field = V4L2_FIELD_NONE;
> >>>> 
> >>>> The could be set already when the buffer is queued.
> >>>>
> >>>>>> +    curr_buf->vb.sequence = framecount;
> >>>>>> +    vb2_buffer_done(&curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
> >>>>>> +}
> >>>>>> +
> >>>>>> +/**
> >>>>>> + * mali_c55_set_plane_done - mark the plane as written and process the buffer if
> >>>>>> + *                 both planes are finished.
> >>>>>> + * @cap_dev:  pointer to the fr or ds pipe output
> >>>>>> + * @plane:    the plane to mark as completed
> >>>>>> + *
> >>>>>> + * The Mali C55 ISP has muliplanar outputs for some formats that come with two
> >>>>>> + * separate "buffer write completed" interrupts - we need to flag each plane's
> >>>>>> + * completion and check whether both planes are done - if so, complete the buf
> >>>>>> + * in vb2.
> >>>>>> + */
> >>>>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> >>>>>> +                 enum mali_c55_planes plane)
> >>>>>> +{
> >>>>>> +    struct mali_c55_isp *isp = &cap_dev->mali_c55->isp;
> >>>>>> +    struct mali_c55_buffer *curr_buf;
> >>>>>> +
> >>>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
> >>>>>> +    curr_buf = cap_dev->buffers.curr;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * This _should_ never happen. If no buffer was available from vb2 then
> >>>>>> +     * we tell the ISP not to bother writing the next frame, which means the
> >>>>>> +     * interrupts that call this function should never trigger. If it does
> >>>>>> +     * happen then one of our assumptions is horribly wrong - complain
> >>>>>> +     * loudly and do nothing.
> >>>>>> +     */
> >>>>>> +    if (!curr_buf) {
> >>>>>> +        dev_err(cap_dev->mali_c55->dev, "%s null buffer in %s()\n",
> >>>>>> +            mali_c55_cap_dev_to_name(cap_dev), __func__);
> >>>>>> +        return;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* If the other plane is also done... */
> >>>>>> +    if (curr_buf->plane_done[~plane & 1]) {
> >>>>>> +        mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> >>>>>> +        cap_dev->buffers.curr = NULL;
> >>>>>> +        isp->frame_sequence++;
> >>>>>> +    } else {
> >>>>>> +        curr_buf->plane_done[plane] = true;
> >>>>>> +    }
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_cap_dev_stream_disable(struct mali_c55_cap_dev *cap_dev)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>>>> +
> >>>>>> +    mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                 MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>>>> +    mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                 MALI_C55_WRITER_FRAME_WRITE_MASK, 0x00);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_cap_dev_stream_enable(struct mali_c55_cap_dev *cap_dev)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * The Mali ISP can hold up to 5 buffer addresses and simply cycle
> >>>>>> +     * through them, but it's not clear to me that the vb2 queue _guarantees_
> >>>>>> +     * it will queue buffers to the driver in a fixed order, and ensuring
> >>>>>> +     * we call vb2_buffer_done() for the right buffer seems to me to add
> >>>>>> +     * pointless complexity given in multi-context mode we'd need to
> >>>>>> +     * re-write those registers every frame anyway...so we tell the ISP to
> >>>>>> +     * use a single register and update it for each frame.
> >>>>>> +     */
> >>>> 
> >>>> A single register sounds prone to error conditions. Is it at least
> >>>> shadowed in the hardware, or do you have to make sure you reprogram it
> >>>> during the vertical blanking only ?
> >>> 
> >>> It would have to be reprogrammed during the vertical blanking if we were running in a
> >>> configuration with a single config space, otherwise you have the time it takes to process a frame
> >>> plus vertical blanking. As I say, it'll have to work like this in multi-context mode anyway.
> >>>
> >>> If we want to use the cycling...is it guaranteed that vb2 buffers will always be queued in order?
> > 
> > In which order ?
> >
> >>>> I'll mostly skip buffer handling in this review, I need to first
> >>>> understand how the hardware operates to make an informed opinion.
> >>>>
> >>>>>> +    mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_Y_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
> >>>>>> +            MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK, 0);
> >>>>>> +    mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_BANKS_CONFIG(cap_dev->reg_offset),
> >>>>>> +            MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK, 0);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * We only queue a buffer in the streamon path if this is the first of
> >>>>>> +     * the capture devices to start streaming. If the ISP is already running
> >>>>>> +     * then we rely on the ISP_START interrupt to queue the first buffer for
> >>>>>> +     * this capture device.
> >>>>>> +     */
> >>>>>> +    if (mali_c55->pipe.start_count == 1)
> >>>>>> +        mali_c55_set_next_buffer(cap_dev);
> >>>> 
> >>>> I think we'll have to revisit buffer handling to make sure it's 100%
> >>>> race-free.
> >>>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_cap_dev_return_buffers(struct mali_c55_cap_dev *cap_dev,
> >>>>>> +                        enum vb2_buffer_state state)
> >>>>>> +{
> >>>>>> +    struct mali_c55_buffer *buf, *tmp;
> >>>>>> +
> >>>>>> +    guard(spinlock)(&cap_dev->buffers.lock);
> >>>>>> +
> >>>>>> +    if (cap_dev->buffers.curr) {
> >>>>>> + vb2_buffer_done(&cap_dev->buffers.curr->vb.vb2_buf,
> >>>>>> +                state);
> >>>>>> +        cap_dev->buffers.curr = NULL;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    if (cap_dev->buffers.next) {
> >>>>>> + vb2_buffer_done(&cap_dev->buffers.next->vb.vb2_buf,
> >>>>>> +                state);
> >>>>>> +        cap_dev->buffers.next = NULL;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    list_for_each_entry_safe(buf, tmp, &cap_dev->buffers.queue, queue) {
> >>>>>> +        list_del(&buf->queue);
> >>>>>> +        vb2_buffer_done(&buf->vb.vb2_buf, state);
> >>>>>> +    }
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_vb2_start_streaming(struct vb2_queue *q, unsigned int count)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>>>> +    struct mali_c55_resizer *rzr = cap_dev->rzr;
> >>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    guard(mutex)(&isp->lock);
> >>>> 
> >>>> What's the reason for using the isp lock here and in
> >>>> mali_c55_vb2_stop_streaming() ? If you need a lock that covers all video
> >>>> nodes in order to synchronize start/stop, you may want to use the
> >>>> graph_mutex of the media device instead.
> >>> 
> >>> It's because I wanted to make sure that the ISP was in a known started/stopped state before
> >>> possibly trying to start/stop it, which can be done from either of the two capture devices. This
> >>> would go away if we were synchronising with the links anyway.
> > 
> > OK.
> >
> >>>>>> +
> >>>>>> +    ret = pm_runtime_resume_and_get(mali_c55->dev);
> >>>>>> +    if (ret)
> >>>>>> +        return ret;
> >>>>>> +
> >>>>>> +    ret = video_device_pipeline_start(&cap_dev->vdev,
> >>>>>> +                      &cap_dev->mali_c55->pipe);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "%s failed to start media pipeline\n",
> >>>>>> +            mali_c55_cap_dev_to_name(cap_dev));
> >>>> 
> >>>> Drop the message or make it dev_dbg() as it can be triggered by
> >>>> userspace.
> >>>>
> >>>>>> +        goto err_pm_put;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    mali_c55_cap_dev_stream_enable(cap_dev);
> >>>>>> +    mali_c55_rzr_start_stream(rzr);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * We only start the ISP if we're the only capture device that's
> >>>>>> +     * streaming. Otherwise, it'll already be active.
> >>>>>> +     */
> >>>> 
> >>>> I still think we should use link setup to indicate which video devices
> >>>> userspace plans to use, and then only start when they're all started.
> >>>> That includes stats and parameters buffers. We can continue this
> >>>> discussion in the context of the previous version of the patch series,
> >>>> or here, up to you.
> >>> 
> >>> Let's just continue here. I think I called it "clunky" before; from my perspective it's an
> >>> unnecessary extra step - we can already signal to the driver that we don't want to use the video
> >>> devices by not queuing buffers to them or starting the stream on them and although I understand
> > 
> > By not starting streaming, perhaps, but by not queuing buffers, no. The
> > reason is that there's no synchronization between buffer queues. If you
> > queue
> >
> > Frame	FR	DS
> > --------------------
> > 1	x	x
> > 2	x
> > 3	x	x
> > 4	x	x
> >
> > it will not be distinguishable by the driver from
> >
> > Frame	FR	DS
> > --------------------
> > 1	x	x
> > 2	x	x
> > 3	x	x
> > 4	x
> >
> >>> that that means that one of the two image data capture devices will receive data before the other,
> >>> I don't understand why that's considered to be a problem. Possibly that last part is the stickler;
> >>> can you explain a bit why it's an issue for one capture queue to start earlier than the other?
> > 
> > Because from a userspace point of view, if you want to capture frames
> > from both pipelines, you will expect to receive a buffer from each
> > pipeline for every frame. If that's not guaranteed at stream start, you
> > will then need to implement synchronization code that will drop buffers
> > on one pipeline until you get the first buffer on the other pipeline
> > (assuming you can synchronize them by sequence number). That will be
> > more work, and can introduce latency.
> >
> >>>>>> +    if (mali_c55->pipe.start_count == 1) {
> >>>>>> +        ret = mali_c55_isp_start_stream(isp);
> >>>>>> +        if (ret)
> >>>>>> +            goto err_disable_cap_dev;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_disable_cap_dev:
> >>>>>> +    mali_c55_cap_dev_stream_disable(cap_dev);
> >>>>>> +    video_device_pipeline_stop(&cap_dev->vdev);
> >>>>>> +err_pm_put:
> >>>>>> +    pm_runtime_put(mali_c55->dev);
> >>>>>> +    mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_QUEUED);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_vb2_stop_streaming(struct vb2_queue *q)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = q->drv_priv;
> >>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>>>> +    struct mali_c55_resizer *rzr = cap_dev->rzr;
> >>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
> >>>>>> +
> >>>>>> +    guard(mutex)(&isp->lock);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * If one of the other capture nodes is streaming, we shouldn't
> >>>>>> +     * disable the ISP here.
> >>>>>> +     */
> >>>>>> +    if (mali_c55->pipe.start_count == 1)
> >>>>>> +        mali_c55_isp_stop_stream(&mali_c55->isp);
> >>>>>> +
> >>>>>> +    mali_c55_rzr_stop_stream(rzr);
> >>>>>> +    mali_c55_cap_dev_stream_disable(cap_dev);
> >>>>>> +    mali_c55_cap_dev_return_buffers(cap_dev, VB2_BUF_STATE_ERROR);
> >>>>>> +    video_device_pipeline_stop(&cap_dev->vdev);
> >>>>>> +    pm_runtime_put(mali_c55->dev);
> >>>> 
> >>>> I think runtime PM autosuspend would be very useful, as it will ensure
> >>>> that stop-reconfigure-start cycles get handled as efficiently as
> >>>> possible without powering the device down. It could be done on top as a
> >>>> separate patch.
> >>> 
> >>> Alright
> >>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct vb2_ops mali_c55_vb2_ops = {
> >>>>>> +    .queue_setup        = &mali_c55_vb2_queue_setup,
> >>>>>> +    .buf_queue        = &mali_c55_buf_queue,
> >>>>>> +    .buf_init        = &mali_c55_buf_init,
> >>>>>> +    .wait_prepare        = vb2_ops_wait_prepare,
> >>>>>> +    .wait_finish        = vb2_ops_wait_finish,
> >>>>>> +    .start_streaming    = &mali_c55_vb2_start_streaming,
> >>>>>> +    .stop_streaming        = &mali_c55_vb2_stop_streaming,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const struct v4l2_file_operations mali_c55_v4l2_fops = {
> >>>>>> +    .owner = THIS_MODULE,
> >>>>>> +    .unlocked_ioctl = video_ioctl2,
> >>>>>> +    .open = v4l2_fh_open,
> >>>>>> +    .release = vb2_fop_release,
> >>>>>> +    .poll = vb2_fop_poll,
> >>>>>> +    .mmap = vb2_fop_mmap,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static void mali_c55_try_fmt(struct v4l2_pix_format_mplane *pix_mp)
> >>>>>> +{
> >>>>>> +    const struct mali_c55_fmt *capture_format;
> >>>>>> +    const struct v4l2_format_info *info;
> >>>>>> +    struct v4l2_plane_pix_format *plane;
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
> >>>>>> +    pix_mp->pixelformat = capture_format->fourcc;
> >>>>>> +
> >>>>>> +    pix_mp->width = clamp(pix_mp->width, MALI_C55_MIN_WIDTH,
> >>>>>> +                  MALI_C55_MAX_WIDTH);
> >>>>>> +    pix_mp->height = clamp(pix_mp->height, MALI_C55_MIN_HEIGHT,
> >>>>>> +                   MALI_C55_MAX_HEIGHT);
> >>>> 
> >>>> Ah, these clamps are right :-)
> >>> 
> >>> Hurrah!
> >>>
> >>>>>> +
> >>>>>> +    pix_mp->field = V4L2_FIELD_NONE;
> >>>>>> +    pix_mp->colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>>>> +    pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>>>> +    pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT;
> >>>>>> +
> >>>>>> +    info = v4l2_format_info(pix_mp->pixelformat);
> >>>> 
> >>>> This function may return NULL. That shouldn't be the case as long as it
> >>>> supports all formats that the C55 driver supports, so I suppose it's
> >>>> safe.
> >>>>
> >>>>>> +    pix_mp->num_planes = info->mem_planes;
> >>>>>> +    memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
> >>>>>> +
> >>>>>> +    pix_mp->plane_fmt[0].bytesperline = info->bpp[0] * pix_mp->width;
> >>>> 
> >>>> Does the hardware support configurable line strides ? If so we should
> >>>> support it.
> >>> 
> >>> You have to set the line stride in the DMA writer registers, which we do using this same
> >>> value...might userspace have set bytesperline already then or something? Or is there some other
> >>> place it could be configured?
> > 
> > Userspace can request a specific stride by setting bytesperline, yes. If
> > that's set, you should honour it (and of course adjust it to a
> > reasonable [min, max] range as well as align it based on hardware
> > constraints).
> >
> >>>>>> +    pix_mp->plane_fmt[0].sizeimage = info->bpp[0] * pix_mp->width
> >>>>>> +                       * pix_mp->height;
> >>>> 
> >>>>      pix_mp->plane_fmt[0].sizeimage = pix_mp->plane_fmt[0].bytesperline
> >>>>                         * pix_mp->height;
> >>>>
> >>>>>> +
> >>>>>> +    for (i = 1; i < info->comp_planes; i++) {
> >>>>>> +        plane = &pix_mp->plane_fmt[i];
> >>>>>> +
> >>>>>> +        plane->bytesperline = DIV_ROUND_UP(info->bpp[i] * pix_mp->width,
> >>>>>> +                           info->hdiv);
> >>>>>> +        plane->sizeimage = DIV_ROUND_UP(
> >>>>>> +                    plane->bytesperline * pix_mp->height,
> >>>>>> +                    info->vdiv);
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    if (info->mem_planes == 1) {
> >>>>>> +        for (i = 1; i < info->comp_planes; i++) {
> >>>>>> +            plane = &pix_mp->plane_fmt[i];
> >>>>>> +            pix_mp->plane_fmt[0].sizeimage += plane->sizeimage;
> >>>>>> +        }
> >>>>>> +    }
> >>>> 
> >>>> I'm wondering, could v4l2_fill_pixfmt_mp() help ? It doesn't support
> >>>> configurable strides though :-S Maybe the helper could be improved, if
> >>>> it's close enough to what you need ?
> >>> 
> >>> I'll take a look
> >>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_try_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>>>> +                       struct v4l2_format *f)
> >>>>>> +{
> >>>>>> +    mali_c55_try_fmt(&f->fmt.pix_mp);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_set_format(struct mali_c55_cap_dev *cap_dev,
> >>>>>> +                struct v4l2_pix_format_mplane *pix_mp)
> >>>>>> +{
> >>>>>> +    const struct mali_c55_fmt *capture_format;
> >>>>>> +    struct mali_c55 *mali_c55 = cap_dev->mali_c55;
> >>>>>> +    const struct v4l2_format_info *info;
> >>>>>> +
> >>>>>> +    mali_c55_try_fmt(pix_mp);
> >>>>>> +    capture_format = mali_c55_format_from_pix(pix_mp->pixelformat);
> >>>>>> +    info = v4l2_format_info(pix_mp->pixelformat);
> >>>>>> +    if (WARN_ON(!info))
> >>>>>> +        return;
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_Y_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +               capture_format->registers.base_mode);
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_ACTIVE_OUT_Y_SIZE(cap_dev->reg_offset),
> >>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
> >>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
> >>>> 
> >>>> Could the register writes be moved to stream start time ?
> >> 
> >> Sorry missed this one. These are writes to the context's registers
> >> buffer, not to the hardware. Does it matter that they're not done at
> >> stream on time?
> > 
> > Writing them here means you'll have to call pm_runtime_resume_and_get()
> > here. If power is then cut off, registers may or may not lose their
> > contents, so you would need to write them at stream on time anyway. I
> > think it's best to move all the hardware configuration at stream on
> > time.
> 
> They won't be lost, because they're not written to the hardware here, only to the registers buffer 
> we allocate in mali_c55_init_context(). They're automatically written to the hardware at stream on 
> time when the config is DMAd over, so unless I'm missing something this is safe as an 
> operation...though possibly the confusion makes them worth moving anyway.

Aahhhh I thought this was writing to the hardware. We really need
separate functions for hardware write and context writes.

Still, I think configuring the hardware at stream on time is the best
option. It will simplify the format operations and make the code
clearer.

> >>>>>> +
> >>>>>> +    if (info->mem_planes > 1) {
> >>>>>> +        mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                   capture_format->registers.base_mode);
> >>>>>> +        mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_UV_WRITER_MODE(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_WRITER_SUBMODE_MASK,
> >>>>>> +                capture_format->registers.uv_plane);
> >>>>>> +
> >>>>>> +        mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_ACTIVE_OUT_UV_SIZE(cap_dev->reg_offset),
> >>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_W(pix_mp->width) |
> >>>>>> + MALI_C55_REG_ACTIVE_OUT_SIZE_H(pix_mp->height));
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    if (info->pixel_enc == V4L2_PIXEL_ENC_YUV) {
> >>>>>> +        /*
> >>>>>> +         * TODO: Figure out the colour matrix coefficients and calculate
> >>>>>> +         * and write them here.
> >>>>>> +         */
> >>>> 
> >>>> Ideally they should also be exposed directly to userspace as ISP
> >>>> parameters. I would probably go as far as saying that they should come
> >>>> directly from userspace, and not derived from the colorspace fields.
> >>> 
> >>> Yes I think I agree, I'll drop the todo from here.
> >>>
> >>>>>> +
> >>>>>> +        mali_c55_write(mali_c55,
> >>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>>>> +                   MALI_C55_CS_CONV_MATRIX_MASK);
> >>>>>> +
> >>>>>> +        if (info->hdiv > 1)
> >>>>>> +            mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK, 0x01);
> >>>>>> +        if (info->vdiv > 1)
> >>>>>> +            mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK, 0x01);
> >>>>>> +        if (info->hdiv > 1 || info->vdiv > 1)
> >>>>>> +            mali_c55_update_bits(mali_c55,
> >>>>>> + MALI_C55_REG_CS_CONV_CONFIG(cap_dev->reg_offset),
> >>>>>> +                MALI_C55_CS_CONV_FILTER_MASK, 0x01);
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    cap_dev->mode.pix_mp = *pix_mp;
> >>>>>> +    cap_dev->mode.capture_fmt = capture_format;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_s_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>>>> +                     struct v4l2_format *f)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >>>>>> +
> >>>>>> +    if (vb2_is_busy(&cap_dev->queue))
> >>>>>> +        return -EBUSY;
> >>>>>> +
> >>>>>> +    mali_c55_set_format(cap_dev, &f->fmt.pix_mp);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_g_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>>>> +                     struct v4l2_format *f)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >>>>>> +
> >>>>>> +    f->fmt.pix_mp = cap_dev->mode.pix_mp;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_enum_fmt_vid_cap_mplane(struct file *file, void *fh,
> >>>>>> +                        struct v4l2_fmtdesc *f)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev = video_drvdata(file);
> >>>>>> +    unsigned int j = 0;
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_fmts); i++) {
> >>>>>> +        if (f->mbus_code &&
> >>>>>> + !mali_c55_mbus_code_can_produce_fmt(&mali_c55_fmts[i],
> >>>>>> +                               f->mbus_code))
> >>>> 
> >>>> Small indentation mistake.
> >>>>
> >>>>>> +            continue;
> >>>>>> +
> >>>>>> +        /* Downscale pipe can't output RAW formats */
> >>>>>> +        if (mali_c55_fmts[i].is_raw &&
> >>>>>> +            cap_dev->reg_offset == MALI_C55_CAP_DEV_DS_REG_OFFSET)
> >>>>>> +            continue;
> >>>>>> +
> >>>>>> +        if (j++ == f->index) {
> >>>>>> +            f->pixelformat = mali_c55_fmts[i].fourcc;
> >>>>>> +            return 0;
> >>>>>> +        }
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return -EINVAL;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_querycap(struct file *file, void *fh,
> >>>>>> +                 struct v4l2_capability *cap)
> >>>>>> +{
> >>>>>> +    strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
> >>>>>> +    strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_ioctl_ops mali_c55_v4l2_ioctl_ops = {
> >>>>>> +    .vidioc_reqbufs = vb2_ioctl_reqbufs,
> >>>>>> +    .vidioc_querybuf = vb2_ioctl_querybuf,
> >>>>>> +    .vidioc_create_bufs = vb2_ioctl_create_bufs,
> >>>>>> +    .vidioc_qbuf = vb2_ioctl_qbuf,
> >>>>>> +    .vidioc_expbuf = vb2_ioctl_expbuf,
> >>>>>> +    .vidioc_dqbuf = vb2_ioctl_dqbuf,
> >>>>>> +    .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> >>>>>> +    .vidioc_streamon = vb2_ioctl_streamon,
> >>>>>> +    .vidioc_streamoff = vb2_ioctl_streamoff,
> >>>>>> +    .vidioc_try_fmt_vid_cap_mplane = mali_c55_try_fmt_vid_cap_mplane,
> >>>>>> +    .vidioc_s_fmt_vid_cap_mplane = mali_c55_s_fmt_vid_cap_mplane,
> >>>>>> +    .vidioc_g_fmt_vid_cap_mplane = mali_c55_g_fmt_vid_cap_mplane,
> >>>>>> +    .vidioc_enum_fmt_vid_cap = mali_c55_enum_fmt_vid_cap_mplane,
> >>>>>> +    .vidioc_querycap = mali_c55_querycap,
> >>>>>> +    .vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
> >>>>>> +    .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
> >>>>>> +};
> >>>>>> +
> >>>>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct v4l2_pix_format_mplane pix_mp;
> >>>>>> +    struct mali_c55_cap_dev *cap_dev;
> >>>>>> +    struct video_device *vdev;
> >>>>>> +    struct vb2_queue *vb2q;
> >>>>>> +    unsigned int i;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
> >>>> 
> >>>> Moving the inner content to a separate mali_c55_register_capture_dev()
> >>>> function would increase readability I think, and remove usage of gotos.
> >>>> I would probably do the same for unregistration too, for consistency.
> >>>>
> >>>>>> +        cap_dev = &mali_c55->cap_devs[i];
> >>>>>> +        vdev = &cap_dev->vdev;
> >>>>>> +        vb2q = &cap_dev->queue;
> >>>>>> +
> >>>>>> +        /*
> >>>>>> +         * The downscale output pipe is an optional block within the ISP
> >>>>>> +         * so we need to check whether it's actually been fitted or not.
> >>>>>> +         */
> >>>>>> +
> >>>>>> +        if (i == MALI_C55_CAP_DEV_DS &&
> >>>>>> +            !(mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED))
> >>>>>> +            continue;
> >>>> 
> >>>> Given that there's only two capture devices, and one is optional, when
> >>>> moving the inner code to a separate function you could unroll the loop.
> >>>> Up to you.
> >>>>
> >>>>>> +
> >>>>>> +        cap_dev->mali_c55 = mali_c55;
> >>>>>> +        mutex_init(&cap_dev->lock);
> >>>>>> +        INIT_LIST_HEAD(&cap_dev->buffers.queue);
> >>>>>> +
> >>>>>> +        switch (i) {
> >>>>>> +        case MALI_C55_CAP_DEV_FR:
> >>>>>> +            cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_FR];
> >>>>>> +            cap_dev->reg_offset = MALI_C55_CAP_DEV_FR_REG_OFFSET;
> >>>>>> +            break;
> >>>>>> +        case MALI_C55_CAP_DEV_DS:
> >>>>>> +            cap_dev->rzr = &mali_c55->resizers[MALI_C55_RZR_DS];
> >>>>>> +            cap_dev->reg_offset = MALI_C55_CAP_DEV_DS_REG_OFFSET;
> >>>>>> +            break;
> >>>>>> +        default:
> >>>> 
> >>>> That can't happen.
> >>>>
> >>>>>> + mutex_destroy(&cap_dev->lock);
> >>>>>> +            ret = -EINVAL;
> >>>>>> +            goto err_destroy_mutex;
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        cap_dev->pad.flags = MEDIA_PAD_FL_SINK;
> >>>>>> +        ret = media_entity_pads_init(&cap_dev->vdev.entity, 1, &cap_dev->pad);
> >>>>>> +        if (ret) {
> >>>>>> +            mutex_destroy(&cap_dev->lock);
> >>>>>> +            goto err_destroy_mutex;
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        vb2q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> >>>>>> +        vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
> >>>>>> +        vb2q->drv_priv = cap_dev;
> >>>>>> +        vb2q->mem_ops = &vb2_dma_contig_memops;
> >>>>>> +        vb2q->ops = &mali_c55_vb2_ops;
> >>>>>> +        vb2q->buf_struct_size = sizeof(struct mali_c55_buffer);
> >>>>>> +        vb2q->min_queued_buffers = 1;
> >>>>>> +        vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> >>>>>> +        vb2q->lock = &cap_dev->lock;
> >>>>>> +        vb2q->dev = mali_c55->dev;
> >>>>>> +
> >>>>>> +        ret = vb2_queue_init(vb2q);
> >>>>>> +        if (ret) {
> >>>>>> +            dev_err(mali_c55->dev, "%s vb2 queue init failed\n",
> >>>>>> +                mali_c55_cap_dev_to_name(cap_dev));
> >>>>>> +            goto err_cleanup_media_entity;
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        strscpy(cap_dev->vdev.name, capture_device_names[i],
> >>>>>> +            sizeof(cap_dev->vdev.name));
> >>>>>> +        vdev->release = video_device_release_empty;
> >>>>>> +        vdev->fops = &mali_c55_v4l2_fops;
> >>>>>> +        vdev->ioctl_ops = &mali_c55_v4l2_ioctl_ops;
> >>>>>> +        vdev->lock = &cap_dev->lock;
> >>>>>> +        vdev->v4l2_dev = &mali_c55->v4l2_dev;
> >>>>>> +        vdev->queue = &cap_dev->queue;
> >>>>>> +        vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> >>>>>> +                    V4L2_CAP_STREAMING | V4L2_CAP_IO_MC;
> >>>>>> +        vdev->entity.ops = &mali_c55_media_ops;
> >>>>>> +        video_set_drvdata(vdev, cap_dev);
> >>>>>> +
> >>>>>> +        memset(&pix_mp, 0, sizeof(pix_mp));
> >>>>>> +        pix_mp.pixelformat = V4L2_PIX_FMT_RGB565;
> >>>>>> +        pix_mp.width = MALI_C55_DEFAULT_WIDTH;
> >>>>>> +        pix_mp.height = MALI_C55_DEFAULT_HEIGHT;
> >>>>>> +        mali_c55_set_format(cap_dev, &pix_mp);
> >>>>>> +
> >>>>>> +        ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
> >>>>>> +        if (ret) {
> >>>>>> +            dev_err(mali_c55->dev,
> >>>>>> +                "%s failed to register video device\n",
> >>>>>> +                mali_c55_cap_dev_to_name(cap_dev));
> >>>>>> +            goto err_release_vb2q;
> >>>>>> +        }
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_release_vb2q:
> >>>>>> +    vb2_queue_release(vb2q);
> >>>>>> +err_cleanup_media_entity:
> >>>>>> +    media_entity_cleanup(&cap_dev->vdev.entity);
> >>>>>> +err_destroy_mutex:
> >>>>>> +    mutex_destroy(&cap_dev->lock);
> >>>>>> +    mali_c55_unregister_capture_devs(mali_c55);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_cap_dev *cap_dev;
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55->cap_devs); i++) {
> >>>>>> +        cap_dev = &mali_c55->cap_devs[i];
> >>>>>> +
> >>>>>> +        if (!video_is_registered(&cap_dev->vdev))
> >>>>>> +            continue;
> >>>>>> +
> >>>>>> +        vb2_video_unregister_device(&cap_dev->vdev);
> >>>>>> +        media_entity_cleanup(&cap_dev->vdev.entity);
> >>>>>> +        mutex_destroy(&cap_dev->lock);
> >>>>>> +    }
> >>>>>> +}
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..2d0c4d152beb
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>>>> @@ -0,0 +1,266 @@
> >>>>>> +/* SPDX-License-Identifier: GPL-2.0 */
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Common definitions
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#ifndef _MALI_C55_COMMON_H
> >>>>>> +#define _MALI_C55_COMMON_H
> >>>>>> +
> >>>>>> +#include <linux/clk.h>
> >>>>>> +#include <linux/io.h>
> >>>>>> +#include <linux/list.h>
> >>>>>> +#include <linux/mutex.h>
> >>>>>> +#include <linux/scatterlist.h>
> >>>>> 
> >>>>> I don't think this is needed. You're however missing spinlock.h.
> >>>>>
> >>>>>> +#include <linux/videodev2.h>
> >>>>>> +
> >>>>>> +#include <media/media-device.h>
> >>>>>> +#include <media/v4l2-async.h>
> >>>>>> +#include <media/v4l2-ctrls.h>
> >>>>>> +#include <media/v4l2-dev.h>
> >>>>>> +#include <media/v4l2-device.h>
> >>>>>> +#include <media/v4l2-subdev.h>
> >>>>>> +#include <media/videobuf2-core.h>
> >>>>>> +#include <media/videobuf2-v4l2.h>
> >>>>>> +
> >>>>>> +#define MALI_C55_DRIVER_NAME        "mali-c55"
> >>>>>> +
> >>>>>> +/* min and max values for the image sizes */
> >>>>>> +#define MALI_C55_MIN_WIDTH        640U
> >>>>>> +#define MALI_C55_MIN_HEIGHT        480U
> >>>>>> +#define MALI_C55_MAX_WIDTH        8192U
> >>>>>> +#define MALI_C55_MAX_HEIGHT        8192U
> >>>>>> +#define MALI_C55_DEFAULT_WIDTH        1920U
> >>>>>> +#define MALI_C55_DEFAULT_HEIGHT        1080U
> >>>>>> +
> >>>>>> +#define MALI_C55_DEFAULT_MEDIA_BUS_FMT MEDIA_BUS_FMT_RGB121212_1X36
> >>>>>> +
> >>>>>> +struct mali_c55;
> >>>>>> +struct mali_c55_cap_dev;
> >>>>>> +struct platform_device;
> >>>>> 
> >>>>> You should also forward-declare
> >>>>>
> >>>>> struct device;
> >>>>> struct dma_chan;
> >>>>> struct resource;
> >>>>>
> >>>>>> +
> >>>>>> +static const char * const mali_c55_clk_names[] = {
> >>>>>> +    "aclk",
> >>>>>> +    "hclk",
> >>>>>> +};
> >>>>> 
> >>>>> This will end up duplicating the array in each compilation unit, not
> >>>>> great. Move it to mali-c55-core.c. You use it in this file just for its
> >>>>> size, replace that with a macro that defines the size, or allocate
> >>>>> mali_c55.clks dynamically with devm_kcalloc().
> >>>>>
> >>>>>> +
> >>>>>> +enum mali_c55_interrupts {
> >>>>>> +    MALI_C55_IRQ_ISP_START,
> >>>>>> +    MALI_C55_IRQ_ISP_DONE,
> >>>>>> +    MALI_C55_IRQ_MCM_ERROR,
> >>>>>> +    MALI_C55_IRQ_BROKEN_FRAME_ERROR,
> >>>>>> +    MALI_C55_IRQ_MET_AF_DONE,
> >>>>>> +    MALI_C55_IRQ_MET_AEXP_DONE,
> >>>>>> +    MALI_C55_IRQ_MET_AWB_DONE,
> >>>>>> +    MALI_C55_IRQ_AEXP_1024_DONE,
> >>>>>> +    MALI_C55_IRQ_IRIDIX_MET_DONE,
> >>>>>> +    MALI_C55_IRQ_LUT_INIT_DONE,
> >>>>>> +    MALI_C55_IRQ_FR_Y_DONE,
> >>>>>> +    MALI_C55_IRQ_FR_UV_DONE,
> >>>>>> +    MALI_C55_IRQ_DS_Y_DONE,
> >>>>>> +    MALI_C55_IRQ_DS_UV_DONE,
> >>>>>> +    MALI_C55_IRQ_LINEARIZATION_DONE,
> >>>>>> +    MALI_C55_IRQ_RAW_FRONTEND_DONE,
> >>>>>> +    MALI_C55_IRQ_NOISE_REDUCTION_DONE,
> >>>>>> +    MALI_C55_IRQ_IRIDIX_DONE,
> >>>>>> +    MALI_C55_IRQ_BAYER2RGB_DONE,
> >>>>>> +    MALI_C55_IRQ_WATCHDOG_TIMER,
> >>>>>> +    MALI_C55_IRQ_FRAME_COLLISION,
> >>>>>> +    MALI_C55_IRQ_UNUSED,
> >>>>>> +    MALI_C55_IRQ_DMA_ERROR,
> >>>>>> +    MALI_C55_IRQ_INPUT_STOPPED,
> >>>>>> +    MALI_C55_IRQ_MET_AWB_TARGET1_HIT,
> >>>>>> +    MALI_C55_IRQ_MET_AWB_TARGET2_HIT,
> >>>>>> +    MALI_C55_NUM_IRQ_BITS
> >>>>> 
> >>>>> Those are register bits, I think they belong to mali-c55-registers.h,
> >>>>> and should probably be macros instead of an enum.
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +enum mali_c55_isp_pads {
> >>>>>> +    MALI_C55_ISP_PAD_SINK_VIDEO,
> >>>>> 
> >>>>> As there's a single sink pad, maybe MALI_C55_ISP_PAD_SINK ? Ah, you're
> >>>>> probably preparing for ISP parameters support. It's fine.
> >>>>>
> >>>>>> +    MALI_C55_ISP_PAD_SOURCE,
> >>>>> 
> >>>>> Then maybe this should be named MALI_C55_ISP_PAD_SOURCE_VIDEO as I
> >>>>> assume there will be a stats source pad.
> >>>>>
> >>>>>> +    MALI_C55_ISP_PAD_SOURCE_BYPASS,
> >>>>>> +    MALI_C55_ISP_NUM_PADS,
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_tpg {
> >>>>>> +    struct mali_c55 *mali_c55;
> >>>>>> +    struct v4l2_subdev sd;
> >>>>>> +    struct media_pad pad;
> >>>>>> +    struct mutex lock;
> >>>>>> +    struct mali_c55_tpg_ctrls {
> >>>>>> +        struct v4l2_ctrl_handler handler;
> >>>>>> +        struct v4l2_ctrl *test_pattern;
> >>>>> 
> >>>>> Set but never used. You can drop it.
> >>>>>
> >>>>>> +        struct v4l2_ctrl *hblank;
> >>>>> 
> >>>>> Set and used only once, in the same function. You can make it a local
> >>>>> variable.
> >>>>>
> >>>>>> +        struct v4l2_ctrl *vblank;
> >>>>>> +    } ctrls;
> >>>>>> +};
> >>>>> 
> >>>>> I wonder if this file should be split, with mali-c55-capture.h,
> >>>>> mali-c55-core.h, mali-c55-isp.h, ... I think it could increase
> >>>>> readability by clearly separating the different elements. Up to you.
> >>>>>
> >>>>>> +
> >>>>>> +struct mali_c55_isp {
> >>>>>> +    struct mali_c55 *mali_c55;
> >>>>>> +    struct v4l2_subdev sd;
> >>>>>> +    struct media_pad pads[MALI_C55_ISP_NUM_PADS];
> >>>>>> +    struct media_pad *remote_src;
> >>>>>> +    struct v4l2_async_notifier notifier;
> >>>>> 
> >>>>> I'm tempted to move the notifier to mali_c55, as it's related to
> >>>>> components external to the whole ISP, not to the ISP subdev itself.
> >>>>> Could you give it a try, to see if it could be done without any drawback
> >>>>> ?
> >>>>>
> >>>>>> +    struct mutex lock;
> >>>>> 
> >>>>> Locks require a comment to explain what they protect. Same below where
> >>>>> applicable (for both mutexes and spinlocks).
> >>>>>
> >>>>>> +    unsigned int frame_sequence;
> >>>>>> +};
> >>>>>> +
> >>>>>> +enum mali_c55_resizer_ids {
> >>>>>> +    MALI_C55_RZR_FR,
> >>>>>> +    MALI_C55_RZR_DS,
> >>>>>> +    MALI_C55_NUM_RZRS,
> >>>>> 
> >>>>> The usual abbreviation for "resizer" in drivers/media/ is "rsz", not
> >>>>> "rzr". I would have said we can leave it as-is as changing it would be a
> >>>>> bit annoying, but I then realized that "rzr" is not just unusual, it's
> >>>>> actually not used at all. Would you mind applying a sed globally ? :-)
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +enum mali_c55_rzr_pads {
> >>>>> 
> >>>>> Same enums/structs use abbreviations, some don't. Consistency would
> >>>>> help.
> >>>>>
> >>>>>> +    MALI_C55_RZR_SINK_PAD,
> >>>>>> +    MALI_C55_RZR_SOURCE_PAD,
> >>>>>> +    MALI_C55_RZR_SINK_BYPASS_PAD,
> >>>>>> +    MALI_C55_RZR_NUM_PADS
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_resizer {
> >>>>>> +    struct mali_c55 *mali_c55;
> >>>>>> +    struct mali_c55_cap_dev *cap_dev;
> >>>>>> +    enum mali_c55_resizer_ids id;
> >>>>>> +    struct v4l2_subdev sd;
> >>>>>> +    struct media_pad pads[MALI_C55_RZR_NUM_PADS];
> >>>>>> +    unsigned int num_routes;
> >>>>>> +    bool streaming;
> >>>>>> +};
> >>>>>> +
> >>>>>> +enum mali_c55_cap_devs {
> >>>>>> +    MALI_C55_CAP_DEV_FR,
> >>>>>> +    MALI_C55_CAP_DEV_DS,
> >>>>>> +    MALI_C55_NUM_CAP_DEVS
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_fmt {
> >>>>> 
> >>>>> mali_c55_format_info would be a better name I think, as this stores
> >>>>> format information, not formats.
> >>>>>
> >>>>>> +    u32 fourcc;
> >>>>>> +    unsigned int mbus_codes[2];
> >>>>> 
> >>>>> A comment to explain why we have two media bus codes would be useful.
> >>>>> You can document the whole structure if desired :-)
> >>>>>
> >>>>>> +    bool is_raw;
> >>>>>> +    struct mali_c55_fmt_registers {
> >>>>> 
> >>>>> Make it an anonymous structure, it's never used anywhere else.
> >>>>>
> >>>>>> +        unsigned int base_mode;
> >>>>>> +        unsigned int uv_plane;
> >>>>> 
> >>>>> If those are register field values, use u32 instead of unsigned int.
> >>>>>
> >>>>>> +    } registers;
> >>>>> 
> >>>>> It's funny, we tend to abbreviate different things, I would have used
> >>>>> "regs" here but written "format" in full in the structure name :-)
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +enum mali_c55_isp_bayer_order {
> >>>>>> +    MALI_C55_BAYER_ORDER_RGGB,
> >>>>>> +    MALI_C55_BAYER_ORDER_GRBG,
> >>>>>> +    MALI_C55_BAYER_ORDER_GBRG,
> >>>>>> +    MALI_C55_BAYER_ORDER_BGGR
> >>>>> 
> >>>>> These are registers values too, they belong to mali-c55-registers.h.
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_isp_fmt {
> >>>>> 
> >>>>> mali_c55_isp_format_info
> >>>>>
> >>>>>> +    u32 code;
> >>>>>> +    enum v4l2_pixel_encoding encoding;
> >>>>> 
> >>>>> Here you use v4l2_pixel_encoding, for mali_c55_fmt you use is_raw. Maybe
> >>>>> pick the same option for both structures ?
> >>>>>
> >>>>>> +    enum mali_c55_isp_bayer_order order;
> >>>>>> +};
> >>>>>> +
> >>>>>> +enum mali_c55_planes {
> >>>>>> +    MALI_C55_PLANE_Y,
> >>>>>> +    MALI_C55_PLANE_UV,
> >>>>>> +    MALI_C55_NUM_PLANES
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_buffer {
> >>>>>> +    struct vb2_v4l2_buffer vb;
> >>>>>> +    bool plane_done[MALI_C55_NUM_PLANES];
> >>>>> 
> >>>>> I think tracking the pending state would simplify the logic in
> >>>>> mali_c55_set_plane_done(), which would become
> >>>>>
> >>>>>      curr_buf->plane_pending[plane] = false;
> >>>>>
> >>>>>      if (!curr_buf->plane_pending[0] && !curr_buf->plane_pending[1]) {
> >>>>>          mali_c55_handle_buffer(curr_buf, isp->frame_sequence);
> >>>>>          cap_dev->buffers.curr = NULL;
> >>>>>          isp->frame_sequence++;
> >>>>>      }
> >>>>>
> >>>>> Or a counter may be even easier (and would consume less memory).
> >>>>>
> >>>>>> +    struct list_head queue;
> >>>>>> +    u32 addrs[MALI_C55_NUM_PLANES];
> >>>>> 
> >>>>> This stores DMA addresses, use dma_addr_t.
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_cap_dev {
> >>>>>> +    struct mali_c55 *mali_c55;
> >>>>>> +    struct mali_c55_resizer *rzr;
> >>>>>> +    struct video_device vdev;
> >>>>>> +    struct media_pad pad;
> >>>>>> +    struct vb2_queue queue;
> >>>>>> +    struct mutex lock;
> >>>>>> +    unsigned int reg_offset;
> >>>>> 
> >>>>> Manual handling of the offset everywhere, with parametric macros for the
> >>>>> resizer register addresses, isn't very nice. Introduce resizer-specific
> >>>>> accessors and capture-specific accessors (mali_c55_rsz_write(), ...)
> >>>>> that take a mali_c55_resizer or mali_c55_cap_dev pointer, and handle the
> >>>>> offset there. The register macros should loose their offset parameter.
> >>>>>
> >>>>> You could also use a single set of accessors that would become
> >>>>> path/output-specific (mali_c55_path_write() ? mali_c55_output_write()
> >>>>> ?), that may make the code easier to read.
> >>>>>
> >>>>> You can also replace reg_offset with a void __iomem * base, which would
> >>>>> avoid the computation at runtime.
> >>>>>
> >>>>>> +
> >>>>>> +    struct mali_c55_mode {
> >>>>> 
> >>>>> Make the structure anonymous.
> >>>>>
> >>>>>> +        const struct mali_c55_fmt *capture_fmt;
> >>>>>> +        struct v4l2_pix_format_mplane pix_mp;
> >>>>>> +    } mode;
> >>>>> 
> >>>>> What's a "mode" ? I think I'd name this
> >>>>>
> >>>>>      struct {
> >>>>>          const struct mali_c55_fmt *info;
> >>>>>          struct v4l2_pix_format_mplane format;
> >>>>>      } format;
> >>>>>
> >>>>> Or you could just drop the structure and have
> >>>>>
> >>>>>      const struct mali_c55_fmt *format_info;
> >>>>>      struct v4l2_pix_format_mplane format;
> >>>>>
> >>>>> or something similar.
> >>>>>
> >>>>>> +
> >>>>>> +    struct {
> >>>>>> +        spinlock_t lock;
> >>>>>> +        struct list_head queue;
> >>>>>> +        struct mali_c55_buffer *curr;
> >>>>>> +        struct mali_c55_buffer *next;
> >>>>>> +    } buffers;
> >>>>>> +
> >>>>>> +    bool streaming;
> >>>>>> +};
> >>>>>> +
> >>>>>> +enum mali_c55_config_spaces {
> >>>>>> +    MALI_C55_CONFIG_PING,
> >>>>>> +    MALI_C55_CONFIG_PONG,
> >>>>>> +    MALI_C55_NUM_CONFIG_SPACES
> >>>>> 
> >>>>> The last enumerator is not used.
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_ctx {
> >>>>> 
> >>>>> mali_c55_context ?
> >>>>>
> >>>>>> +    struct mali_c55 *mali_c55;
> >>>>>> +    void *registers;
> >>>>> 
> >>>>> Please document this structure and explain that this field points to a
> >>>>> copy of the register space in system memory, I was about to write you're
> >>>>> missing __iomem :-)
> >>>>>
> >>>>>> +    phys_addr_t base;
> >>>>>> +    spinlock_t lock;
> >>>>>> +    struct list_head list;
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55 {
> >>>>>> +    struct device *dev;
> >>>>>> +    struct resource *res;
> >>>>> 
> >>>>> You could possibly drop this field by passing the physical address of
> >>>>> the register space from mali_c55_probe() to mali_c55_init_context() as a
> >>>>> function parameter.
> >>>>>
> >>>>>> +    void __iomem *base;
> >>>>>> +    struct dma_chan *channel;
> >>>>>> +    struct clk_bulk_data clks[ARRAY_SIZE(mali_c55_clk_names)];
> >>>>>> +
> >>>>>> +    u16 capabilities;
> >>>>>> +    struct media_device media_dev;
> >>>>>> +    struct v4l2_device v4l2_dev;
> >>>>>> +    struct media_pipeline pipe;
> >>>>>> +
> >>>>>> +    struct mali_c55_tpg tpg;
> >>>>>> +    struct mali_c55_isp isp;
> >>>>>> +    struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
> >>>>>> +    struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
> >>>>>> +
> >>>>>> +    struct list_head contexts;
> >>>>>> +    enum mali_c55_config_spaces next_config;
> >>>>>> +};
> >>>>>> +
> >>>>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val);
> >>>>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
> >>>>>> +          bool force_hardware);
> >>>>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
> >>>>>> +              u32 mask, u32 val);
> >>>>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
> >>>>>> +              enum mali_c55_config_spaces cfg_space);
> >>>>>> +
> >>>>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55);
> >>>>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55);
> >>>>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55);
> >>>>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55);
> >>>>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55);
> >>>>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
> >>>>>> +int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
> >>>>>> +void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
> >>>>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
> >>>>>> +void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> >>>>>> +                 enum mali_c55_planes plane);
> >>>>>> +void mali_c55_set_next_buffer(struct mali_c55_cap_dev *cap_dev);
> >>>>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55);
> >>>>>> +
> >>>>>> +bool mali_c55_format_is_raw(unsigned int mbus_code);
> >>>>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr);
> >>>>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr);
> >>>>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp);
> >>>>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp);
> >>>>>> +
> >>>>>> +const struct mali_c55_isp_fmt *
> >>>>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
> >>>>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
> >>>>>> +#define for_each_mali_isp_fmt(fmt)\
> >>>>> 
> >>>>> #define for_each_mali_isp_fmt(fmt) \
> >>>>>
> >>>>>> +    for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
> >>>>> 
> >>>>> Looks like parentheses were on sale :-)
> >>>>>
> >>>>>      for ((fmt) = NULL; (fmt) = mali_c55_isp_fmt_next(fmt); )
> >>>>>
> >>>>> This macro is used in two places only, in the mali-c55-isp.c file where
> >>>>> open-coding the loop without using mali_c55_isp_fmt_next() would be more
> >>>>> efficient, and in mali-c55-resizer.c where a function to return format
> >>>>> i'th would be more efficient. I think you can drop the macro and the
> >>>>> mali_c55_isp_fmt_next() function.
> >>>>>
> >>>>>> +
> >>>>>> +#endif /* _MALI_C55_COMMON_H */
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..50caf5ee7474
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>>>> @@ -0,0 +1,767 @@
> >>>>>> +// SPDX-License-Identifier: GPL-2.0
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Core driver code
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#include <linux/bitops.h>
> >>>>>> +#include <linux/cleanup.h>
> >>>>>> +#include <linux/clk.h>
> >>>>>> +#include <linux/delay.h>
> >>>>>> +#include <linux/device.h>
> >>>>>> +#include <linux/dmaengine.h>
> >>>>>> +#include <linux/dma-mapping.h>
> >>>>>> +#include <linux/interrupt.h>
> >>>>>> +#include <linux/iopoll.h>
> >>>>>> +#include <linux/ioport.h>
> >>>>>> +#include <linux/mod_devicetable.h>
> >>>>>> +#include <linux/of.h>
> >>>>>> +#include <linux/of_reserved_mem.h>
> >>>>>> +#include <linux/platform_device.h>
> >>>>>> +#include <linux/pm_runtime.h>
> >>>>>> +#include <linux/scatterlist.h>
> >>>>> 
> >>>>> I don't think this is needed.
> >>>>>
> >>>>> Missing slab.h.
> >>>>>
> >>>>>> +#include <linux/string.h>
> >>>>>> +
> >>>>>> +#include <media/media-entity.h>
> >>>>>> +#include <media/v4l2-device.h>
> >>>>>> +#include <media/videobuf2-dma-contig.h>
> >>>>>> +
> >>>>>> +#include "mali-c55-common.h"
> >>>>>> +#include "mali-c55-registers.h"
> >>>>>> +
> >>>>>> +static const char * const mali_c55_interrupt_names[] = {
> >>>>>> +    [MALI_C55_IRQ_ISP_START] = "ISP start",
> >>>>>> +    [MALI_C55_IRQ_ISP_DONE] = "ISP done",
> >>>>>> +    [MALI_C55_IRQ_MCM_ERROR] = "Multi-context management error",
> >>>>>> +    [MALI_C55_IRQ_BROKEN_FRAME_ERROR] = "Broken frame error",
> >>>>>> +    [MALI_C55_IRQ_MET_AF_DONE] = "AF metering done",
> >>>>>> +    [MALI_C55_IRQ_MET_AEXP_DONE] = "AEXP metering done",
> >>>>>> +    [MALI_C55_IRQ_MET_AWB_DONE] = "AWB metering done",
> >>>>>> +    [MALI_C55_IRQ_AEXP_1024_DONE] = "AEXP 1024-bit histogram done",
> >>>>>> +    [MALI_C55_IRQ_IRIDIX_MET_DONE] = "Iridix metering done",
> >>>>>> +    [MALI_C55_IRQ_LUT_INIT_DONE] = "LUT memory init done",
> >>>>>> +    [MALI_C55_IRQ_FR_Y_DONE] = "Full resolution Y plane DMA done",
> >>>>>> +    [MALI_C55_IRQ_FR_UV_DONE] = "Full resolution U/V plane DMA done",
> >>>>>> +    [MALI_C55_IRQ_DS_Y_DONE] = "Downscale Y plane DMA done",
> >>>>>> +    [MALI_C55_IRQ_DS_UV_DONE] = "Downscale U/V plane DMA done",
> >>>>>> +    [MALI_C55_IRQ_LINEARIZATION_DONE] = "Linearisation done",
> >>>>>> +    [MALI_C55_IRQ_RAW_FRONTEND_DONE] = "Raw frontend processing done",
> >>>>>> +    [MALI_C55_IRQ_NOISE_REDUCTION_DONE] = "Noise reduction done",
> >>>>>> +    [MALI_C55_IRQ_IRIDIX_DONE] = "Iridix done",
> >>>>>> +    [MALI_C55_IRQ_BAYER2RGB_DONE] = "Bayer to RGB conversion done",
> >>>>>> +    [MALI_C55_IRQ_WATCHDOG_TIMER] = "Watchdog timer timed out",
> >>>>>> +    [MALI_C55_IRQ_FRAME_COLLISION] = "Frame collision error",
> >>>>>> +    [MALI_C55_IRQ_UNUSED] = "IRQ bit unused",
> >>>>>> +    [MALI_C55_IRQ_DMA_ERROR] = "DMA error",
> >>>>>> +    [MALI_C55_IRQ_INPUT_STOPPED] = "Input port safely stopped",
> >>>>>> +    [MALI_C55_IRQ_MET_AWB_TARGET1_HIT] = "AWB metering target 1 address hit",
> >>>>>> +    [MALI_C55_IRQ_MET_AWB_TARGET2_HIT] = "AWB metering target 2 address hit"
> >>>>>> +};
> >>>>>> +
> >>>>>> +static unsigned int config_space_addrs[] = {
> >>>>> 
> >>>>> const
> >>>>>
> >>>>>> +    [MALI_C55_CONFIG_PING] = 0x0AB6C,
> >>>>>> +    [MALI_C55_CONFIG_PONG] = 0x22B2C,
> >>>>> 
> >>>>> Lowercase hex constants.
> >>>>>
> >>>>> Don't the values belong to mali-c55-registers.h ?
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +/* System IO
> >>>>> 
> >>>>> /*
> >>>>>    * System IO
> >>>>>
> >>>>>> + *
> >>>>>> + * The Mali-C55 ISP has up to two configuration register spaces (called 'ping'
> >>>>>> + * and 'pong'), with the  expectation that the 'active' space will be left
> >>>>> 
> >>>>> s/the  /the /
> >>>>>
> >>>>>> + * untouched whilst a frame is being processed and the 'inactive' space
> >>>>>> + * configured ready to be passed during the blanking period before the next
> >>>>> 
> >>>>> s/to be passed/to be switched to/ ?
> >>>>>
> >>>>>> + * frame processing starts. These spaces should ideally be set via DMA transfer
> >>>>>> + * from a buffer rather than through individual register set operations. There
> >>>>>> + * is also a shared global register space which should be set normally. Of
> >>>>>> + * course, the ISP might be included in a system which lacks a suitable DMA
> >>>>>> + * engine, and the second configuration space might not be fitted at all, which
> >>>>>> + * means we need to support four scenarios:
> >>>>>> + *
> >>>>>> + * 1. Multi config space, with DMA engine.
> >>>>>> + * 2. Multi config space, no DMA engine.
> >>>>>> + * 3. Single config space, with DMA engine.
> >>>>>> + * 4. Single config space, no DMA engine.
> >>>>>> + *
> >>>>>> + * The first case is very easy, but the rest present annoying problems. The best
> >>>>>> + * way to solve them seems to be simply to replicate the concept of DMAing over
> >>>>>> + * the configuration buffer even if there's no DMA engine on the board, for
> >>>>>> + * which we rely on memcpy. To facilitate this any read/write call that is made
> >>>>>> + * to an address within those config spaces should infact be directed to a
> >>>>>> + * buffer that was allocated to hold them rather than the IO memory itself. The
> >>>>>> + * actual copy of that buffer to IO mem will happen on interrupt.
> >>>>>> + */
> >>>>>> +
> >>>>>> +void mali_c55_write(struct mali_c55 *mali_c55, unsigned int addr, u32 val)
> >>>>>> +{
> >>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>>>> +
> >>>>>> +    if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET) {
> >>>>>> +        spin_lock(&ctx->lock);
> >>>>>> +        addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
> >>>>>> +        ((u32 *)ctx->registers)[addr] = val;
> >>>>>> +        spin_unlock(&ctx->lock);
> >>>>>> +
> >>>>>> +        return;
> >>>>>> +    }
> >>>>> 
> >>>>> Ouch. This is likely the second comment you really won't like (after the
> >>>>> comment regarding mali_c55_update_bits() at the very top). I apologize
> >>>>> in advance.
> >>>>>
> >>>>> I really don't like this. Directing writes either to hardware registers
> >>>>> or to the shadow registers in the context makes the callers of the
> >>>>> read/write accessors very hard to read. The probe code, for instance,
> >>>>> mixes writes to hardware registers and writes to the context shadow
> >>>>> registers to initialize the value of some of the shadow registers.
> >>>>>
> >>>>> I'd like to split the read/write accessors into functions that access
> >>>>> the hardware registers (that's easy) and functions that access the
> >>>>> shadow registers. I think the latter should receive a mali_c55_ctx
> >>>>> pointer instead of a mali_c55 pointer to prepare for multi-context
> >>>>> support.
> >>>>>
> >>>>> You can add WARN_ON() guards to the two sets of functions, to ensure
> >>>>> that no register from the "other" space gets passed to the wrong
> >>>>> function by mistake.
> >>>>>
> >>>>>> +
> >>>>>> +    writel(val, mali_c55->base + addr);
> >>>>>> +}
> >>>>>> +
> >>>>>> +u32 mali_c55_read(struct mali_c55 *mali_c55, unsigned int addr,
> >>>>>> +          bool force_hardware)
> >>>>> 
> >>>>> force_hardware is never set to true.
> >>>>>
> >>>>>> +{
> >>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>>>> +    u32 val;
> >>>>>> +
> >>>>>> +    if (addr >= MALI_C55_REG_CONFIG_SPACES_OFFSET && !force_hardware) {
> >>>>>> +        spin_lock(&ctx->lock);
> >>>>>> +        addr = (addr - MALI_C55_REG_CONFIG_SPACES_OFFSET) / 4;
> >>>>>> +        val = ((u32 *)ctx->registers)[addr];
> >>>>>> +        spin_unlock(&ctx->lock);
> >>>>>> +
> >>>>>> +        return val;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return readl(mali_c55->base + addr);
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_update_bits(struct mali_c55 *mali_c55, unsigned int addr,
> >>>>>> +              u32 mask, u32 val)
> >>>>>> +{
> >>>>>> +    u32 orig, tmp;
> >>>>>> +
> >>>>>> +    orig = mali_c55_read(mali_c55, addr, false);
> >>>>>> +
> >>>>>> +    tmp = orig & ~mask;
> >>>>>> +    tmp |= (val << (ffs(mask) - 1)) & mask;
> >>>>>> +
> >>>>>> +    if (tmp != orig)
> >>>>>> +        mali_c55_write(mali_c55, addr, tmp);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_dma_xfer(struct mali_c55_ctx *ctx, dma_addr_t src,
> >>>>>> +                 dma_addr_t dst, enum dma_data_direction dir)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>>>> +    struct dma_async_tx_descriptor *tx;
> >>>>>> +    enum dma_status status;
> >>>>>> +    dma_cookie_t cookie;
> >>>>>> +
> >>>>>> +    tx = dmaengine_prep_dma_memcpy(mali_c55->channel, dst, src,
> >>>>>> +                       MALI_C55_CONFIG_SPACE_SIZE, 0);
> >>>>>> +    if (!tx) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to prep DMA memcpy\n");
> >>>>>> +        return -EIO;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    cookie = dmaengine_submit(tx);
> >>>>>> +    if (dma_submit_error(cookie)) {
> >>>>>> +        dev_err(mali_c55->dev, "error submitting dma transfer\n");
> >>>>>> +        return -EIO;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    status = dma_sync_wait(mali_c55->channel, cookie);
> >>>>> 
> >>>>> I've just realized this performs a busy-wait :-S See the comment in the
> >>>>> probe function about the threaded IRQ handler. I think we'll need to
> >>>>> rework all this. It could be done on top though.
> >>>>>
> >>>>>> +    if (status != DMA_COMPLETE) {
> >>>>>> +        dev_err(mali_c55->dev, "dma transfer failed\n");
> >>>>>> +        return -EIO;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_dma_read(struct mali_c55_ctx *ctx,
> >>>>>> +                 enum mali_c55_config_spaces cfg_space)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>>>> +    struct device *dma_dev = mali_c55->channel->device->dev;
> >>>>>> +    dma_addr_t src = ctx->base + config_space_addrs[cfg_space];
> >>>>>> +    dma_addr_t dst;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    guard(spinlock)(&ctx->lock);
> >>>>>> +
> >>>>>> +    dst = dma_map_single(dma_dev, ctx->registers,
> >>>>>> +                 MALI_C55_CONFIG_SPACE_SIZE, DMA_FROM_DEVICE);
> >>>>>> +    if (dma_mapping_error(dma_dev, dst)) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to map DMA addr\n");
> >>>>>> +        return -EIO;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    ret = mali_c55_dma_xfer(ctx, src, dst, DMA_FROM_DEVICE);
> >>>>>> +    dma_unmap_single(dma_dev, dst, MALI_C55_CONFIG_SPACE_SIZE,
> >>>>>> +             DMA_FROM_DEVICE);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_dma_write(struct mali_c55_ctx *ctx,
> >>>>>> +               enum mali_c55_config_spaces cfg_space)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>>>> +    struct device *dma_dev = mali_c55->channel->device->dev;
> >>>>>> +    dma_addr_t dst = ctx->base + config_space_addrs[cfg_space];
> >>>>>> +    dma_addr_t src;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    guard(spinlock)(&ctx->lock);
> >>>>> 
> >>>>> The code below can take a large amount of time, holding a spinlock will
> >>>>> disable interrupts on the local CPU, that's not good :-(
> >>>>>
> >>>>>> +
> >>>>>> +    src = dma_map_single(dma_dev, ctx->registers,
> >>>>>> +                 MALI_C55_CONFIG_SPACE_SIZE, DMA_TO_DEVICE);
> >>>>>> +    if (dma_mapping_error(dma_dev, src)) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to map DMA addr\n");
> >>>>>> +        return -EIO;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    ret = mali_c55_dma_xfer(ctx, src, dst, DMA_TO_DEVICE);
> >>>>>> +    dma_unmap_single(dma_dev, src, MALI_C55_CONFIG_SPACE_SIZE,
> >>>>>> +             DMA_TO_DEVICE);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_config_read(struct mali_c55_ctx *ctx,
> >>>>>> +                enum mali_c55_config_spaces cfg_space)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>>>> +
> >>>>>> +    if (mali_c55->channel) {
> >>>>>> +        return mali_c55_dma_read(ctx, cfg_space);
> >>>>> 
> >>>>> As this function is used at probe time only, to initialize the context,
> >>>>> I think DMA is overkill.
> >>>>>
> >>>>>> +    } else {
> >>>>>> +        memcpy_fromio(ctx->registers,
> >>>>>> +                  mali_c55->base + config_space_addrs[cfg_space],
> >>>>>> +                  MALI_C55_CONFIG_SPACE_SIZE);
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>>> +}
> >>>>>> +
> >>>>>> +int mali_c55_config_write(struct mali_c55_ctx *ctx,
> >>>>>> +              enum mali_c55_config_spaces cfg_space)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = ctx->mali_c55;
> >>>>>> +
> >>>>>> +    if (mali_c55->channel) {
> >>>>>> +        return mali_c55_dma_write(ctx, cfg_space);
> >>>>>> +    } else {
> >>>>>> +        memcpy_toio(mali_c55->base + config_space_addrs[cfg_space],
> >>>>>> +                ctx->registers, MALI_C55_CONFIG_SPACE_SIZE);
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>> 
> >>>>> Could you measure the time it typically takes to write the registers
> >>>>> using DMA compared to using memcpy_toio() ?
> >>>>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    return list_first_entry(&mali_c55->contexts, struct mali_c55_ctx, list);
> >>>>> 
> >>>>> I think it's too early to tell how multi-context support will look like.
> >>>>> I'm fine keeping mali_c55_get_active_context() as changing that would be
> >>>>> very intrusive (even if I think it will need to be changed), but the
> >>>>> list of contexts is neither the mechanism we'll use, nor something we
> >>>>> need now. Drop the list, embed the context in struct mali_c55, and
> >>>>> return the pointer to that single context from this function.
> >>>>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_remove_links(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> + media_entity_remove_links(&mali_c55->tpg.sd.entity);
> >>>>>> + media_entity_remove_links(&mali_c55->isp.sd.entity);
> >>>>>> +
> >>>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; i++)
> >>>>>> + media_entity_remove_links(&mali_c55->resizers[i].sd.entity);
> >>>>>> +
> >>>>>> +    for (i = 0; i < MALI_C55_NUM_CAP_DEVS; i++)
> >>>>>> + media_entity_remove_links(&mali_c55->cap_devs[i].vdev.entity);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_create_links(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    /* Test pattern generator to ISP */
> >>>>>> +    ret = media_create_pad_link(&mali_c55->tpg.sd.entity, 0,
> >>>>>> +                    &mali_c55->isp.sd.entity,
> >>>>>> +                    MALI_C55_ISP_PAD_SINK_VIDEO, 0);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to link TPG and ISP\n");
> >>>>>> +        goto err_remove_links;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* Full resolution resizer pipe. */
> >>>>>> +    ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >>>>>> +            MALI_C55_ISP_PAD_SOURCE,
> >>>>>> + &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
> >>>>>> +            MALI_C55_RZR_SINK_PAD,
> >>>>>> +            MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
> >>>>>> +        goto err_remove_links;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* Full resolution bypass. */
> >>>>>> +    ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >>>>>> +                    MALI_C55_ISP_PAD_SOURCE_BYPASS,
> >>>>>> + &mali_c55->resizers[MALI_C55_RZR_FR].sd.entity,
> >>>>>> +                    MALI_C55_RZR_SINK_BYPASS_PAD,
> >>>>>> +                    MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to link ISP and FR resizer\n");
> >>>>>> +        goto err_remove_links;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* Resizer pipe to video capture nodes. */
> >>>>>> +    ret = media_create_pad_link(&mali_c55->resizers[0].sd.entity,
> >>>>>> +            MALI_C55_RZR_SOURCE_PAD,
> >>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR].vdev.entity,
> >>>>>> +            0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev,
> >>>>>> +            "failed to link FR resizer and video device\n");
> >>>>>> +        goto err_remove_links;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* The downscale pipe is an optional hardware block */
> >>>>>> +    if (mali_c55->capabilities & MALI_C55_GPS_DS_PIPE_FITTED) {
> >>>>>> +        ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >>>>>> +            MALI_C55_ISP_PAD_SOURCE,
> >>>>>> + &mali_c55->resizers[MALI_C55_RZR_DS].sd.entity,
> >>>>>> +            MALI_C55_RZR_SINK_PAD,
> >>>>>> +            MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>>>> +        if (ret) {
> >>>>>> +            dev_err(mali_c55->dev,
> >>>>>> +                "failed to link ISP and DS resizer\n");
> >>>>>> +            goto err_remove_links;
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        ret = media_create_pad_link(&mali_c55->resizers[1].sd.entity,
> >>>>>> +            MALI_C55_RZR_SOURCE_PAD,
> >>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS].vdev.entity,
> >>>>>> +            0, MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>>>> +        if (ret) {
> >>>>>> +            dev_err(mali_c55->dev,
> >>>>>> +                "failed to link DS resizer and video device\n");
> >>>>>> +            goto err_remove_links;
> >>>>>> +        }
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_remove_links:
> >>>>>> +    mali_c55_remove_links(mali_c55);
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    mali_c55_unregister_tpg(mali_c55);
> >>>>>> +    mali_c55_unregister_isp(mali_c55);
> >>>>>> +    mali_c55_unregister_resizers(mali_c55);
> >>>>>> +    mali_c55_unregister_capture_devs(mali_c55);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_register_entities(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    ret = mali_c55_register_tpg(mali_c55);
> >>>>>> +    if (ret)
> >>>>>> +        return ret;
> >>>>>> +
> >>>>>> +    ret = mali_c55_register_isp(mali_c55);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_unregister_entities;
> >>>>>> +
> >>>>>> +    ret = mali_c55_register_resizers(mali_c55);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_unregister_entities;
> >>>>>> +
> >>>>>> +    ret = mali_c55_register_capture_devs(mali_c55);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_unregister_entities;
> >>>>>> +
> >>>>>> +    ret = mali_c55_create_links(mali_c55);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_unregister_entities;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_unregister_entities:
> >>>>>> +    mali_c55_unregister_entities(mali_c55);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static u32 mali_c55_check_hwcfg(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    u32 product, version, revision, capabilities;
> >>>>>> +
> >>>>>> +    product = mali_c55_read(mali_c55, MALI_C55_REG_PRODUCT, false);
> >>>>>> +    version = mali_c55_read(mali_c55, MALI_C55_REG_VERSION, false);
> >>>>>> +    revision = mali_c55_read(mali_c55, MALI_C55_REG_REVISION, false);
> >>>>>> +
> >>>>>> +    dev_info(mali_c55->dev, "Detected Mali-C55 ISP %u.%u.%u\n",
> >>>>>> +         product, version, revision);
> >>>>>> +
> >>>>>> +    capabilities = mali_c55_read(mali_c55,
> >>>>>> +                     MALI_C55_REG_GLOBAL_PARAMETER_STATUS,
> >>>>>> +                     false);
> >>>>>> +    mali_c55->capabilities = (capabilities & 0xffff);
> >>>>>> +
> >>>>>> +    /* TODO: Might as well start some debugfs */
> >>>>> 
> >>>>> If it's just to expose the version and capabilities, I think that's
> >>>>> overkill. It's not needed for debug purpose (you can get it from the
> >>>>> kernel log already). debugfs isn't meant to be accessible in production,
> >>>>> so an application that would need access to the information wouldn't be
> >>>>> able to use it.
> >>>>>
> >>>>>> +    dev_info(mali_c55->dev, "Mali-C55 capabilities: 0x%04x\n", capabilities);
> >>>>> 
> >>>>> Combine the two messages into one.
> >>>>>
> >>>>>> +    return version;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_swap_next_config(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>>>> +    u32 curr_config, next_config;
> >>>>>> +
> >>>>>> +    curr_config = mali_c55_read(mali_c55, MALI_C55_REG_PING_PONG_READ, false);
> >>>>>> +    curr_config = (curr_config & MALI_C55_REG_PING_PONG_READ_MASK)
> >>>>>> +              >> (ffs(MALI_C55_REG_PING_PONG_READ_MASK) - 1);
> >>>>>> +    next_config = curr_config ^ 1;
> >>>>>> +
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>>>> +                 MALI_C55_REG_MCU_CONFIG_WRITE_MASK, next_config);
> >>>>>> +    mali_c55_config_write(ctx, next_config ?
> >>>>>> +                  MALI_C55_CONFIG_PING : MALI_C55_CONFIG_PONG);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static irqreturn_t mali_c55_isr(int irq, void *context)
> >>>>>> +{
> >>>>>> +    struct device *dev = context;
> >>>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >>>>>> +    u32 interrupt_status;
> >>>>>> +    unsigned int i, j;
> >>>>>> +
> >>>>>> +    interrupt_status = mali_c55_read(mali_c55,
> >>>>>> +                     MALI_C55_REG_INTERRUPT_STATUS_VECTOR,
> >>>>>> +                     false);
> >>>>>> +    if (!interrupt_status)
> >>>>>> +        return IRQ_NONE;
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR_VECTOR,
> >>>>>> +               interrupt_status);
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 0);
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_CLEAR, 1);
> >>>>>> +
> >>>>>> +    for (i = 0; i < MALI_C55_NUM_IRQ_BITS; i++) {
> >>>>>> +        if (!(interrupt_status & (1 << i)))
> >>>>>> +            continue;
> >>>>>> +
> >>>>>> +        switch (i) {
> >>>>>> +        case MALI_C55_IRQ_ISP_START:
> >>>>>> +            mali_c55_isp_queue_event_sof(mali_c55);
> >>>>>> +
> >>>>>> +            for (j = i; j < MALI_C55_NUM_CAP_DEVS; j++)
> >>>>>> + mali_c55_set_next_buffer(&mali_c55->cap_devs[j]);
> >>>>>> +
> >>>>>> +            mali_c55_swap_next_config(mali_c55);
> >>>>>> +
> >>>>>> +            break;
> >>>>>> +        case MALI_C55_IRQ_ISP_DONE:
> >>>>>> +            /*
> >>>>>> +             * TODO: Where the ISP has no Pong config fitted, we'd
> >>>>>> +             * have to do the mali_c55_swap_next_config() call here.
> >>>>>> +             */
> >>>>>> +            break;
> >>>>>> +        case MALI_C55_IRQ_FR_Y_DONE:
> >>>>>> +            mali_c55_set_plane_done(
> >>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
> >>>>>> +                MALI_C55_PLANE_Y);
> >>>>>> +            break;
> >>>>>> +        case MALI_C55_IRQ_FR_UV_DONE:
> >>>>>> +            mali_c55_set_plane_done(
> >>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR],
> >>>>>> +                MALI_C55_PLANE_UV);
> >>>>>> +            break;
> >>>>>> +        case MALI_C55_IRQ_DS_Y_DONE:
> >>>>>> +            mali_c55_set_plane_done(
> >>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
> >>>>>> +                MALI_C55_PLANE_Y);
> >>>>>> +            break;
> >>>>>> +        case MALI_C55_IRQ_DS_UV_DONE:
> >>>>>> +            mali_c55_set_plane_done(
> >>>>>> + &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS],
> >>>>>> +                MALI_C55_PLANE_UV);
> >>>>>> +            break;
> >>>>>> +        default:
> >>>>>> +            /*
> >>>>>> +             * Only the above interrupts are currently unmasked. If
> >>>>>> +             * we receive anything else here then something weird
> >>>>>> +             * has gone on.
> >>>>>> +             */
> >>>>>> +            dev_err(dev, "masked interrupt %s triggered\n",
> >>>>>> +                mali_c55_interrupt_names[i]);
> >>>>>> +        }
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return IRQ_HANDLED;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_init_context(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_ctx *ctx;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
> >>>>>> +    if (!ctx) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to allocate new context\n");
> >>>>> 
> >>>>> No need for an error message when memory allocation fails.
> >>>>>
> >>>>>> +        return -ENOMEM;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    ctx->base = mali_c55->res->start;
> >>>>>> +    ctx->mali_c55 = mali_c55;
> >>>>>> +
> >>>>>> +    ctx->registers = kzalloc(MALI_C55_CONFIG_SPACE_SIZE,
> >>>>>> +                 GFP_KERNEL | GFP_DMA);
> >>>>>> +    if (!ctx->registers) {
> >>>>>> +        ret = -ENOMEM;
> >>>>>> +        goto err_free_ctx;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * The allocated memory is empty, we need to load the default
> >>>>>> +     * register settings. We just read Ping; it's identical to Pong.
> >>>>>> +     */
> >>>>>> +    ret = mali_c55_config_read(ctx, MALI_C55_CONFIG_PING);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_free_registers;
> >>>>>> +
> >>>>>> +    list_add_tail(&ctx->list, &mali_c55->contexts);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * Some features of the ISP need to be disabled by default and only
> >>>>>> +     * enabled at the same time as they're configured by a parameters buffer
> >>>>>> +     */
> >>>>>> +
> >>>>>> +    /* Bypass the sqrt and square compression and expansion modules */
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_1,
> >>>>>> +                 MALI_C55_REG_BYPASS_1_FE_SQRT, 0x01);
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BYPASS_3,
> >>>>>> +                 MALI_C55_REG_BYPASS_3_SQUARE_BE, 0x01);
> >>>>>> +
> >>>>>> +    /* Bypass the temper module */
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_2,
> >>>>>> +               MALI_C55_REG_BYPASS_2_TEMPER);
> >>>>>> +
> >>>>>> +    /* Bypass the colour noise reduction  */
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_BYPASS_4,
> >>>>>> +               MALI_C55_REG_BYPASS_4_CNR);
> >>>>>> +
> >>>>>> +    /* Disable the sinter module */
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_SINTER_CONFIG,
> >>>>>> +                 MALI_C55_SINTER_ENABLE_MASK, 0x00);
> >>>>>> +
> >>>>>> +    /* Disable the RGB Gamma module for each output */
> >>>>>> +    mali_c55_write(
> >>>>>> +        mali_c55,
> >>>>>> + MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_FR_REG_OFFSET),
> >>>>>> +        0x00);
> >>>>>> +    mali_c55_write(
> >>>>>> +        mali_c55,
> >>>>>> + MALI_C55_REG_GAMMA_RGB_ENABLE(MALI_C55_CAP_DEV_DS_REG_OFFSET),
> >>>>>> +        0x00);
> >>>>>> +
> >>>>>> +    /* Disable the colour correction matrix */
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CCM_ENABLE, 0x00);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_free_registers:
> >>>>>> +    kfree(ctx->registers);
> >>>>>> +err_free_ctx:
> >>>>>> +    kfree(ctx);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_runtime_resume(struct device *dev)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    ret = clk_bulk_prepare_enable(ARRAY_SIZE(mali_c55->clks),
> >>>>>> +                      mali_c55->clks);
> >>>>>> +    if (ret)
> >>>>>> +        dev_err(mali_c55->dev, "failed to enable clocks\n");
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_runtime_suspend(struct device *dev)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = dev_get_drvdata(dev);
> >>>>>> +
> >>>>>> + clk_bulk_disable_unprepare(ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct dev_pm_ops mali_c55_pm_ops = {
> >>>>>> +    SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> >>>>>> +                pm_runtime_force_resume)
> >>>>>> +    SET_RUNTIME_PM_OPS(mali_c55_runtime_suspend, mali_c55_runtime_resume,
> >>>>>> +               NULL)
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_probe(struct platform_device *pdev)
> >>>>>> +{
> >>>>>> +    struct device *dev = &pdev->dev;
> >>>>>> +    struct mali_c55 *mali_c55;
> >>>>>> +    dma_cap_mask_t mask;
> >>>>>> +    u32 version;
> >>>>>> +    int ret;
> >>>>>> +    u32 val;
> >>>>>> +
> >>>>>> +    mali_c55 = devm_kzalloc(dev, sizeof(*mali_c55), GFP_KERNEL);
> >>>>>> +    if (!mali_c55)
> >>>>>> +        return dev_err_probe(dev, -ENOMEM,
> >>>>>> +                     "failed to allocate memory\n");
> >>>>> 
> >>>>>          return -ENOMEM;
> >>>>>
> >>>>> There's no need to print messages for memory allocation failures.
> >>>>>
> >>>>>> +
> >>>>>> +    mali_c55->dev = dev;
> >>>>>> +    platform_set_drvdata(pdev, mali_c55);
> >>>>>> +
> >>>>>> +    mali_c55->base = devm_platform_get_and_ioremap_resource(pdev, 0,
> >>>>>> +                                &mali_c55->res);
> >>>>>> +    if (IS_ERR(mali_c55->base))
> >>>>>> +        return dev_err_probe(dev, PTR_ERR(mali_c55->base),
> >>>>>> +                     "failed to map IO memory\n");
> >>>>>> +
> >>>>>> +    ret = platform_get_irq(pdev, 0);
> >>>>>> +    if (ret < 0)
> >>>>>> +        return dev_err_probe(dev, ret, "failed to get interrupt num\n");
> >>>>> 
> >>>>> s/ num// or s/num/number/
> >>>>>
> >>>>>> +
> >>>>>> +    ret = devm_request_threaded_irq(&pdev->dev, ret, NULL,
> >>>>>> +                    mali_c55_isr, IRQF_ONESHOT,
> >>>>>> +                    dev_driver_string(&pdev->dev),
> >>>>>> +                    &pdev->dev);
> >>>>> 
> >>>>> Requested the IRQ should be done much lower, after you have initialized
> >>>>> everything, or an IRQ that would fire early would have really bad
> >>>>> consequences.
> >>>>>
> >>>>> A comment to explain why you need a threaded interrupt handler would be
> >>>>> good. I assume it is due only to the need to transfer the registers
> >>>>> using DMA. I wonder if we should then split the interrupt handler in
> >>>>> two, with a non-threaded part for the operations that can run quickly,
> >>>>> and a threaded part for the reprogramming.
> >>>>>
> >>>>> It may also be that we could just start the DMA transfer in the
> >>>>> non-threaded handler without waiting synchronously for it to complete.
> >>>>> That would be a bigger change, and would require checking race
> >>>>> conditions carefully. On the other hand, I'm a bit concerned about the
> >>>>> current implementation, have you tested what happens if the DMA transfer
> >>>>> takes too long to complete, and spans frame boundaries ? This part could
> >>>>> be addressed by a patch on top of this one.
> >>>>>
> >>>>>> +    if (ret)
> >>>>>> +        return dev_err_probe(dev, ret, "failed to request irq\n");
> >>>>>> +
> >>>>>> +    for (unsigned int i = 0; i < ARRAY_SIZE(mali_c55_clk_names); i++)
> >>>>>> +        mali_c55->clks[i].id = mali_c55_clk_names[i];
> >>>>>> +
> >>>>>> +    ret = devm_clk_bulk_get(dev, ARRAY_SIZE(mali_c55->clks), mali_c55->clks);
> >>>>>> +    if (ret)
> >>>>>> +        return dev_err_probe(dev, ret, "failed to acquire clocks\n");
> >>>>>> +
> >>>>>> +    pm_runtime_enable(&pdev->dev);
> >>>>>> +
> >>>>>> +    ret = pm_runtime_resume_and_get(&pdev->dev);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_pm_runtime_disable;
> >>>>>> +
> >>>>>> +    of_reserved_mem_device_init(dev);
> >>>>> 
> >>>>> I'd move this, the vb2_dma_contig_set_max_seg_size() call and the
> >>>>> dma_cap_* calls before pm_runtime_enable() as they don't need the device
> >>>>> to be powered.
> >>>>>
> >>>>>> +    version = mali_c55_check_hwcfg(mali_c55);
> >>>>>> +    vb2_dma_contig_set_max_seg_size(dev, UINT_MAX);
> >>>>>> +
> >>>>>> +    /* Use "software only" context management. */
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>>>> +                 MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK,
> >>>>>> +                 MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK);
> >>>>> 
> >>>>> You handle that in mali_c55_isp_start(), does the register have to be
> >>>>> set here too ?
> >>>>>
> >>>>>> +
> >>>>>> +    dma_cap_zero(mask);
> >>>>>> +    dma_cap_set(DMA_MEMCPY, mask);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * No error check, because we will just fallback on memcpy if there is
> >>>>>> +     * no usable DMA channel on the system.
> >>>>>> +     */
> >>>>>> +    mali_c55->channel = dma_request_channel(mask, NULL, NULL);
> >>>>>> +
> >>>>>> +    INIT_LIST_HEAD(&mali_c55->contexts);
> >>>>>> +    ret = mali_c55_init_context(mali_c55);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_release_dma_channel;
> >>>>>> +
> >>>>> 
> >>>>> I'd move all the code from here ...
> >>>>>
> >>>>>> +    mali_c55->media_dev.dev = dev;
> >>>>>> +    strscpy(mali_c55->media_dev.model, "ARM Mali-C55 ISP",
> >>>>>> +        sizeof(mali_c55->media_dev.model));
> >>>>>> +    mali_c55->media_dev.hw_revision = version;
> >>>>>> +
> >>>>>> +    media_device_init(&mali_c55->media_dev);
> >>>>>> +    ret = media_device_register(&mali_c55->media_dev);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_cleanup_media_device;
> >>>>>> +
> >>>>>> +    mali_c55->v4l2_dev.mdev = &mali_c55->media_dev;
> >>>>>> +    ret = v4l2_device_register(dev, &mali_c55->v4l2_dev);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(dev, "failed to register V4L2 device\n");
> >>>>>> +        goto err_unregister_media_device;
> >>>>>> +    };
> >>>>>> +
> >>>>>> +    ret = mali_c55_register_entities(mali_c55);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(dev, "failed to register entities\n");
> >>>>>> +        goto err_unregister_v4l2_device;
> >>>>>> +    }
> >>>>> 
> >>>>> ... to here to a separate function, or maybe fold it all in
> >>>>> mali_c55_register_entities() (which should the be renamed). Same thing
> >>>>> for the cleanup code.
> >>>>>
> >>>>>> +
> >>>>>> +    /* Set safe stop to ensure we're in a non-streaming state */
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> >>>>>> +               MALI_C55_INPUT_SAFE_STOP);
> >>>>>> +    readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >>>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * We're ready to process interrupts. Clear any that are set and then
> >>>>>> +     * unmask them for processing.
> >>>>>> +     */
> >>>>>> +    mali_c55_write(mali_c55, 0x30, 0xffffffff);
> >>>>>> +    mali_c55_write(mali_c55, 0x34, 0xffffffff);
> >>>>>> +    mali_c55_write(mali_c55, 0x40, 0x01);
> >>>>>> +    mali_c55_write(mali_c55, 0x40, 0x00);
> >>>>> 
> >>>>> Please replace the register addresses with macros.
> >>>>>
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INTERRUPT_MASK_VECTOR, 0x39fc3fc);
> >>>>> 
> >>>>> The value should use the interrupt bits macros.
> >>>>>
> >>>>>> +
> >>>>>> +    pm_runtime_put(&pdev->dev);
> >>>>> 
> >>>>> Once power gets cut, the registers your programmed above may be lost. I
> >>>>> think you should programe them in the runtime PM resume handler.
> >>>>>
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_unregister_v4l2_device:
> >>>>>> +    v4l2_device_unregister(&mali_c55->v4l2_dev);
> >>>>>> +err_unregister_media_device:
> >>>>>> +    media_device_unregister(&mali_c55->media_dev);
> >>>>>> +err_cleanup_media_device:
> >>>>>> +    media_device_cleanup(&mali_c55->media_dev);
> >>>>>> +err_release_dma_channel:
> >>>>>> +    dma_release_channel(mali_c55->channel);
> >>>>>> +err_pm_runtime_disable:
> >>>>>> +    pm_runtime_disable(&pdev->dev);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_remove(struct platform_device *pdev)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = platform_get_drvdata(pdev);
> >>>>>> +    struct mali_c55_ctx *ctx, *tmp;
> >>>>>> +
> >>>>>> +    list_for_each_entry_safe(ctx, tmp, &mali_c55->contexts, list) {
> >>>>>> +        list_del(&ctx->list);
> >>>>>> +        kfree(ctx->registers);
> >>>>>> +        kfree(ctx);
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    mali_c55_remove_links(mali_c55);
> >>>>>> +    mali_c55_unregister_entities(mali_c55);
> >>>>>> +    v4l2_device_put(&mali_c55->v4l2_dev);
> >>>>>> +    media_device_unregister(&mali_c55->media_dev);
> >>>>>> +    media_device_cleanup(&mali_c55->media_dev);
> >>>>>> +    dma_release_channel(mali_c55->channel);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct of_device_id mali_c55_of_match[] = {
> >>>>>> +    { .compatible = "arm,mali-c55", },
> >>>>>> +    {},
> >>>>> 
> >>>>>      { /* Sentinel */ },
> >>>>>
> >>>>>> +};
> >>>>>> +MODULE_DEVICE_TABLE(of, mali_c55_of_match);
> >>>>>> +
> >>>>>> +static struct platform_driver mali_c55_driver = {
> >>>>>> +    .driver = {
> >>>>>> +        .name = "mali-c55",
> >>>>>> +        .of_match_table = of_match_ptr(mali_c55_of_match),
> >>>>> 
> >>>>> Drop of_match_ptr().
> >>>>>
> >>>>>> +        .pm = &mali_c55_pm_ops,
> >>>>>> +    },
> >>>>>> +    .probe = mali_c55_probe,
> >>>>>> +    .remove_new = mali_c55_remove,
> >>>>>> +};
> >>>>>> +
> >>>>>> +module_platform_driver(mali_c55_driver);
> >>>>> 
> >>>>> Blank line.
> >>>>>
> >>>>>> +MODULE_AUTHOR("Daniel Scally <dan.scally@ideasonboard.com>");
> >>>>>> +MODULE_AUTHOR("Jacopo Mondi <jacopo.mondi@ideasonboard.com>");
> >>>>>> +MODULE_DESCRIPTION("ARM Mali-C55 ISP platform driver");
> >>>>>> +MODULE_LICENSE("GPL");
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..ea8b7b866e7a
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>>>> @@ -0,0 +1,611 @@
> >>>>>> +// SPDX-License-Identifier: GPL-2.0
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Image signal processor
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#include <linux/delay.h>
> >>>>>> +#include <linux/iopoll.h>
> >>>>>> +#include <linux/property.h>
> >>>>>> +#include <linux/string.h>
> >>>>>> +
> >>>>>> +#include <media/media-entity.h>
> >>>>>> +#include <media/v4l2-common.h>
> >>>>>> +#include <media/v4l2-event.h>
> >>>>>> +#include <media/v4l2-mc.h>
> >>>>>> +#include <media/v4l2-subdev.h>
> >>>>>> +
> >>>>>> +#include "mali-c55-common.h"
> >>>>>> +#include "mali-c55-registers.h"
> >>>>>> +
> >>>>>> +static const struct mali_c55_isp_fmt mali_c55_isp_fmts[] = {
> >>>>>> +    {
> >>>>>> +        .code = MEDIA_BUS_FMT_SRGGB20_1X20,
> >>>>>> +        .order = MALI_C55_BAYER_ORDER_RGGB,
> >>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .code = MEDIA_BUS_FMT_SGRBG20_1X20,
> >>>>>> +        .order = MALI_C55_BAYER_ORDER_GRBG,
> >>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .code = MEDIA_BUS_FMT_SGBRG20_1X20,
> >>>>>> +        .order = MALI_C55_BAYER_ORDER_GBRG,
> >>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .code = MEDIA_BUS_FMT_SBGGR20_1X20,
> >>>>>> +        .order = MALI_C55_BAYER_ORDER_BGGR,
> >>>>>> +        .encoding = V4L2_PIXEL_ENC_BAYER,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .code = MEDIA_BUS_FMT_RGB202020_1X60,
> >>>>>> +        .order = 0, /* Not relevant for this format */
> >>>>>> +        .encoding = V4L2_PIXEL_ENC_RGB,
> >>>>>> +    }
> >>>>>> +    /*
> >>>>>> +     * TODO: Support MEDIA_BUS_FMT_YUV20_1X60 here. This is so that we can
> >>>>>> +     * also support YUV input from a sensor passed-through to the output. At
> >>>>>> +     * present we have no mechanism to test that though so it may have to
> >>>>>> +     * wait a while...
> >>>>>> +     */
> >>>>>> +};
> >>>>>> +
> >>>>>> +const struct mali_c55_isp_fmt *
> >>>>>> +mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt)
> >>>>>> +{
> >>>>>> +    if (!fmt)
> >>>>>> +        fmt = &mali_c55_isp_fmts[0];
> >>>>>> +    else
> >>>>>> +        fmt++;
> >>>>>> +
> >>>>>> +    for (; fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)]; fmt++)
> >>>>>> +        return fmt;
> >>>>> 
> >>>>> That's peculiar.
> >>>>>
> >>>>>      if (!fmt)
> >>>>>          fmt = &mali_c55_isp_fmts[0];
> >>>>>      else if (fmt < &mali_c55_isp_fmts[ARRAY_SIZE(mali_c55_isp_fmts)])
> >>>>>          return ++fmt;
> >>>>>      else
> >>>>>          return NULL;
> >>>>>
> >>>>>> +
> >>>>>> +    return NULL;
> >>>>>> +}
> >>>>>> +
> >>>>>> +bool mali_c55_isp_is_format_supported(unsigned int mbus_code)
> >>>>>> +{
> >>>>>> +    const struct mali_c55_isp_fmt *isp_fmt;
> >>>>>> +
> >>>>>> +    for_each_mali_isp_fmt(isp_fmt) {
> >>>>> 
> >>>>> I would open-code the loop instead of using the macro, like you do
> >>>>> below. It will be more efficient.
> >>>>>
> >>>>>> +        if (isp_fmt->code == mbus_code)
> >>>>>> +            return true;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return false;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct mali_c55_isp_fmt *
> >>>>>> +mali_c55_isp_get_mbus_config_by_code(u32 code)
> >>>>>> +{
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_isp_fmts); i++) {
> >>>>>> +        if (mali_c55_isp_fmts[i].code == code)
> >>>>>> +            return &mali_c55_isp_fmts[i];
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return NULL;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_isp_stop(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    u32 val;
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST, MALI_C55_INPUT_SAFE_STOP);
> >>>>> 
> >>>>>      mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> >>>>>                 MALI_C55_INPUT_SAFE_STOP);
> >>>>>
> >>>>>> + readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >>>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_isp_start(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>>>> +    const struct mali_c55_isp_fmt *cfg;
> >>>>>> +    struct v4l2_mbus_framefmt *format;
> >>>>> 
> >>>>> const
> >>>>>
> >>>>>> +    struct v4l2_subdev_state *state;
> >>>>>> +    struct v4l2_rect *crop;
> >>>>> 
> >>>>> const
> >>>>>
> >>>>>> +    struct v4l2_subdev *sd;
> >>>>>> +    u32 val;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    sd = &mali_c55->isp.sd;
> >>>>> 
> >>>>> Assign when declaring the variable.
> >>>>>
> >>>>>> +
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_MCU_CONFIG,
> >>>>>> +                 MALI_C55_REG_MCU_CONFIG_WRITE_MASK, 0x01);
> >>>>>> +
> >>>>>> +    /* Apply input windowing */
> >>>>>> +    state = v4l2_subdev_get_locked_active_state(sd);
> >>>>> 
> >>>>> Using .enable_streams() (see below) you'll get this for free.
> >>>>>
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +    format = v4l2_subdev_state_get_format(state,
> >>>>>> +                          MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +    cfg = mali_c55_isp_get_mbus_config_by_code(format->code);
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_HC_START,
> >>>>>> +               MALI_C55_HC_START(crop->left));
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_HC_SIZE,
> >>>>>> +               MALI_C55_HC_SIZE(crop->width));
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_VC_START_SIZE,
> >>>>>> +               MALI_C55_VC_START(crop->top) |
> >>>>>> +               MALI_C55_VC_SIZE(crop->height));
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
> >>>>>> +                 MALI_C55_REG_ACTIVE_WIDTH_MASK, format->width);
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BASE_ADDR,
> >>>>>> +                 MALI_C55_REG_ACTIVE_HEIGHT_MASK, format->height);
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BAYER_ORDER,
> >>>>>> +                 MALI_C55_BAYER_ORDER_MASK, cfg->order);
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_INPUT_WIDTH,
> >>>>>> +                 MALI_C55_INPUT_WIDTH_MASK,
> >>>>>> +                 MALI_C55_INPUT_WIDTH_20BIT);
> >>>>>> +
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >>>>>> +                 MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK,
> >>>>>> +                 cfg->encoding == V4L2_PIXEL_ENC_RGB ?
> >>>>>> +                 MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK : 0x00);
> >>>>>> +
> >>>>>> +    ret = mali_c55_config_write(ctx, MALI_C55_CONFIG_PING);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to DMA config\n");
> >>>>>> +        return ret;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_INPUT_MODE_REQUEST,
> >>>>>> +               MALI_C55_INPUT_SAFE_START);
> >>>>>> +    readl_poll_timeout(mali_c55->base + MALI_C55_REG_MODE_STATUS,
> >>>>>> +               val, !val, 10 * USEC_PER_MSEC, 250 * USEC_PER_MSEC);
> >>>>> 
> >>>>> Should you return an error in case of timeout ?
> >>>>>
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_isp_stop_stream(struct mali_c55_isp *isp)
> >>>>> 
> >>>>> Why is this not handled wired to .s_stream() ? Or better,
> >>>>> .enable_streams() and .disable_streams().
> >>>>>
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>>>> +    struct v4l2_subdev *sd;
> >>>>>> +
> >>>>>> +    if (isp->remote_src) {
> >>>>>> +        sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
> >>>>>> +        v4l2_subdev_disable_streams(sd, isp->remote_src->index, BIT(0));
> >>>>>> +    }
> >>>>>> +    isp->remote_src = NULL;
> >>>>>> +
> >>>>>> +    mali_c55_isp_stop(mali_c55);
> >>>>>> +}
> >>>>>> +
> >>>>>> +int mali_c55_isp_start_stream(struct mali_c55_isp *isp)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>>>> +    struct media_pad *sink_pad;
> >>>>>> +    struct v4l2_subdev *sd;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    sink_pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
> >>>>>> +    isp->remote_src = media_pad_remote_pad_unique(sink_pad);
> >>>>>> +    if (IS_ERR(isp->remote_src)) {
> >>>>> 
> >>>>> If you set the MUST_CONNECT flag for the MALI_C55_ISP_PAD_SINK_VIDEO pad
> >>>>> I think you can drop this check.
> >>>>>
> >>>>>> +        dev_err(mali_c55->dev, "Failed to get source for ISP\n");
> >>>>>> +        return PTR_ERR(isp->remote_src);
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    sd = media_entity_to_v4l2_subdev(isp->remote_src->entity);
> >>>>>> +
> >>>>>> +    isp->frame_sequence = 0;
> >>>>>> +    ret = mali_c55_isp_start(mali_c55);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "Failed to start ISP\n");
> >>>>>> +        isp->remote_src = NULL;
> >>>>>> +        return ret;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * We only support a single input stream, so we can just enable the 1st
> >>>>>> +     * entry in the streams mask.
> >>>>>> +     */
> >>>>>> +    ret = v4l2_subdev_enable_streams(sd, isp->remote_src->index, BIT(0));
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "Failed to start ISP source\n");
> >>>>>> +        mali_c55_isp_stop(mali_c55);
> >>>>>> +        return ret;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>> +                       struct v4l2_subdev_state *state,
> >>>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
> >>>>>> +{
> >>>>>> +    /*
> >>>>>> +     * Only the internal RGB processed format is allowed on the regular
> >>>>>> +     * processing source pad.
> >>>>>> +     */
> >>>>>> +    if (code->pad == MALI_C55_ISP_PAD_SOURCE) {
> >>>>>> +        if (code->index)
> >>>>>> +            return -EINVAL;
> >>>>>> +
> >>>>>> +        code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* On the sink and bypass pads all the supported formats are allowed. */
> >>>>>> +    if (code->index >= ARRAY_SIZE(mali_c55_isp_fmts))
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    code->code = mali_c55_isp_fmts[code->index].code;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_isp_enum_frame_size(struct v4l2_subdev *sd,
> >>>>>> +                    struct v4l2_subdev_state *state,
> >>>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
> >>>>>> +{
> >>>>>> +    const struct mali_c55_isp_fmt *cfg;
> >>>>>> +
> >>>>>> +    if (fse->index > 0)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * Only the internal RGB processed format is allowed on the regular
> >>>>>> +     * processing source pad.
> >>>>>> +     *
> >>>>>> +     * On the sink and bypass pads all the supported formats are allowed.
> >>>>>> +     */
> >>>>>> +    if (fse->pad == MALI_C55_ISP_PAD_SOURCE) {
> >>>>>> +        if (fse->code != MEDIA_BUS_FMT_RGB121212_1X36)
> >>>>>> +            return -EINVAL;
> >>>>>> +    } else {
> >>>>>> +        cfg = mali_c55_isp_get_mbus_config_by_code(fse->code);
> >>>>>> +        if (!cfg)
> >>>>>> +            return -EINVAL;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
> >>>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
> >>>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
> >>>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_isp_set_fmt(struct v4l2_subdev *sd,
> >>>>>> +                struct v4l2_subdev_state *state,
> >>>>>> +                struct v4l2_subdev_format *format)
> >>>>>> +{
> >>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>>>> +    struct v4l2_mbus_framefmt *src_fmt, *sink_fmt;
> >>>>>> +    const struct mali_c55_isp_fmt *cfg;
> >>>>>> +    struct v4l2_rect *crop;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * Disallow set_fmt on the source pads; format is fixed and the sizes
> >>>>>> +     * are the result of applying the sink crop rectangle to the sink
> >>>>>> +     * format.
> >>>>>> +     */
> >>>>>> +    if (format->pad)
> >>>>> 
> >>>>>      if (format->pad != MALI_C55_ISP_PAD_SINK_VIDEO)
> >>>>>
> >>>>>> +        return v4l2_subdev_get_fmt(sd, state, format);
> >>>>>> +
> >>>>>> +    cfg = mali_c55_isp_get_mbus_config_by_code(fmt->code);
> >>>>>> +    if (!cfg)
> >>>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>>>> +    fmt->field = V4L2_FIELD_NONE;
> >>>>> 
> >>>>> Do you intentionally allow the colorspace fields to be overwritten to
> >>>>> any value ? rkisp1_isp_set_src_fmt() and rkisp1_isp_set_sink_fmt() may
> >>>>> show you how this could be handled.
> >>>>>
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * Clamp sizes in the accepted limits and clamp the crop rectangle in
> >>>>>> +     * the new sizes.
> >>>>>> +     */
> >>>>>> +    clamp_t(unsigned int, fmt->width,
> >>>>>> +        MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>>>>> +    clamp_t(unsigned int, fmt->width,
> >>>>>> +        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>>>> 
> >>>>> clamp_t() returns a value, which you ignore. Those are no-ops. You meant
> >>>>>
> >>>>>      fmt->width = clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> >>>>>                   MALI_C55_MAX_WIDTH);
> >>>>>      fmt->height = clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> >>>>>                    MALI_C55_MAX_HEIGHT);
> >>>>>
> >>>>> Same for every use of clamp_t() through the whole driver.
> >>>>>
> >>>>> Also, do you need clamp_t() ? I think all values are unsigned int, you
> >>>>> can use clamp().
> >>>>>
> >>>>> Are there any alignment constraints, such a multiples of two for bayer
> >>>>> formats ? Same in all the other locations where applicable.
> >>>>>
> >>>>>> +
> >>>>>> +    sink_fmt = v4l2_subdev_state_get_format(state,
> >>>>>> +                        MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +    *sink_fmt = *fmt;
> >>>>>> +
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +    crop->left = 0;
> >>>>>> +    crop->top = 0;
> >>>>>> +    crop->width = fmt->width;
> >>>>>> +    crop->height = fmt->height;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * Propagate format to source pads. On the 'regular' output pad use
> >>>>>> +     * the internal RGB processed format, while on the bypass pad simply
> >>>>>> +     * replicate the ISP sink format. The sizes on both pads are the same as
> >>>>>> +     * the ISP sink crop rectangle.
> >>>>>> +     */
> >>>>> 
> >>>>> Colorspace information will need to be propagated too.
> >>>>>
> >>>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
> >>>>>> +    src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>>>> +    src_fmt->width = crop->width;
> >>>>>> +    src_fmt->height = crop->height;
> >>>>>> +
> >>>>>> +    src_fmt = v4l2_subdev_state_get_format(state,
> >>>>>> +                           MALI_C55_ISP_PAD_SOURCE_BYPASS);
> >>>>>> +    src_fmt->code = fmt->code;
> >>>>>> +    src_fmt->width = crop->width;
> >>>>>> +    src_fmt->height = crop->height;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_isp_get_selection(struct v4l2_subdev *sd,
> >>>>>> +                      struct v4l2_subdev_state *state,
> >>>>>> +                      struct v4l2_subdev_selection *sel)
> >>>>>> +{
> >>>>>> +    if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> >>>>> 
> >>>>>      sel->pad != MALI_C55_ISP_PAD_SINK_VIDEO
> >>>>>
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    sel->r = *v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_isp_set_selection(struct v4l2_subdev *sd,
> >>>>>> +                      struct v4l2_subdev_state *state,
> >>>>>> +                      struct v4l2_subdev_selection *sel)
> >>>>>> +{
> >>>>>> +    struct v4l2_mbus_framefmt *src_fmt;
> >>>>>> +    struct v4l2_mbus_framefmt *fmt;
> >>>>> 
> >>>>> const
> >>>>>
> >>>>>> +    struct v4l2_rect *crop;
> >>>>>> +
> >>>>>> +    if (sel->pad || sel->target != V4L2_SEL_TGT_CROP)
> >>>>> 
> >>>>> Ditto.
> >>>>>
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +
> >>>>>> +    clamp_t(unsigned int, sel->r.left, 0, fmt->width);
> >>>>>> +    clamp_t(unsigned int, sel->r.top, 0, fmt->height);
> >>>>>> +    clamp_t(unsigned int, sel->r.width, MALI_C55_MIN_WIDTH,
> >>>>>> +        fmt->width - sel->r.left);
> >>>>>> +    clamp_t(unsigned int, sel->r.height, MALI_C55_MIN_HEIGHT,
> >>>>>> +        fmt->height - sel->r.top);
> >>>>>> +
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +    *crop = sel->r;
> >>>>>> +
> >>>>>> +    /* Propagate the crop rectangle sizes to the source pad format. */
> >>>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_ISP_PAD_SOURCE);
> >>>>>> +    src_fmt->width = crop->width;
> >>>>>> +    src_fmt->height = crop->height;
> >>>>> 
> >>>>> Can you confirm that cropping doesn't affect the bypass path ? And maybe
> >>>>> add a comment to mention it.
> >>>>>
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_pad_ops mali_c55_isp_pad_ops = {
> >>>>>> +    .enum_mbus_code        = mali_c55_isp_enum_mbus_code,
> >>>>>> +    .enum_frame_size    = mali_c55_isp_enum_frame_size,
> >>>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
> >>>>>> +    .set_fmt        = mali_c55_isp_set_fmt,
> >>>>>> +    .get_selection        = mali_c55_isp_get_selection,
> >>>>>> +    .set_selection        = mali_c55_isp_set_selection,
> >>>>>> +    .link_validate        = v4l2_subdev_link_validate_default,
> >>>>>> +};
> >>>>>> +
> >>>>>> +void mali_c55_isp_queue_event_sof(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct v4l2_event event = {
> >>>>>> +        .type = V4L2_EVENT_FRAME_SYNC,
> >>>>>> +    };
> >>>>>> +
> >>>>>> +    event.u.frame_sync.frame_sequence = mali_c55->isp.frame_sequence;
> >>>>>> +    v4l2_event_queue(mali_c55->isp.sd.devnode, &event);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int
> >>>>>> +mali_c55_isp_subscribe_event(struct v4l2_subdev *sd, struct v4l2_fh *fh,
> >>>>>> +                 struct v4l2_event_subscription *sub)
> >>>>>> +{
> >>>>>> +    if (sub->type != V4L2_EVENT_FRAME_SYNC)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    /* V4L2_EVENT_FRAME_SYNC doesn't require an id, so zero should be set */
> >>>>>> +    if (sub->id != 0)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    return v4l2_event_subscribe(fh, sub, 0, NULL);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_core_ops mali_c55_isp_core_ops = {
> >>>>>> +    .subscribe_event = mali_c55_isp_subscribe_event,
> >>>>>> +    .unsubscribe_event = v4l2_event_subdev_unsubscribe,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_ops mali_c55_isp_ops = {
> >>>>>> +    .pad    = &mali_c55_isp_pad_ops,
> >>>>>> +    .core    = &mali_c55_isp_core_ops,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_isp_init_state(struct v4l2_subdev *sd,
> >>>>>> +                   struct v4l2_subdev_state *sd_state)
> >>>>> 
> >>>>> You name this variable state in every other subdev operation handler.
> >>>>>
> >>>>>> +{
> >>>>>> +    struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >>>>>> +    struct v4l2_rect *in_crop;
> >>>>>> +
> >>>>>> +    sink_fmt = v4l2_subdev_state_get_format(sd_state,
> >>>>>> +                        MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +    src_fmt = v4l2_subdev_state_get_format(sd_state,
> >>>>>> +                           MALI_C55_ISP_PAD_SOURCE);
> >>>>>> +    in_crop = v4l2_subdev_state_get_crop(sd_state,
> >>>>>> +                         MALI_C55_ISP_PAD_SINK_VIDEO);
> >>>>>> +
> >>>>>> +    sink_fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>>>> +    sink_fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>>>> +    sink_fmt->field = V4L2_FIELD_NONE;
> >>>>>> +    sink_fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>>> 
> >>>>> You should initialize the colorspace fields too. Same below.
> >>>>>
> >>>>>> +
> >>>>>> +    *v4l2_subdev_state_get_format(sd_state,
> >>>>>> +                  MALI_C55_ISP_PAD_SOURCE_BYPASS) = *sink_fmt;
> >>>>>> +
> >>>>>> +    src_fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>>>> +    src_fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>>>> +    src_fmt->field = V4L2_FIELD_NONE;
> >>>>>> +    src_fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>>>> +
> >>>>>> +    in_crop->top = 0;
> >>>>>> +    in_crop->left = 0;
> >>>>>> +    in_crop->width = MALI_C55_DEFAULT_WIDTH;
> >>>>>> +    in_crop->height = MALI_C55_DEFAULT_HEIGHT;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_internal_ops mali_c55_isp_internal_ops = {
> >>>>>> +    .init_state = mali_c55_isp_init_state,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const struct media_entity_operations mali_c55_isp_media_ops = {
> >>>>>> +    .link_validate        = v4l2_subdev_link_validate,
> >>>>> 
> >>>>>      .link_validate = v4l2_subdev_link_validate,
> >>>>>
> >>>>> to match mali_c55_isp_internal_ops.
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_isp_notifier_bound(struct v4l2_async_notifier *notifier,
> >>>>>> +                       struct v4l2_subdev *subdev,
> >>>>>> +                       struct v4l2_async_connection *asc)
> >>>>>> +{
> >>>>>> +    struct mali_c55_isp *isp = container_of(notifier,
> >>>>>> +                        struct mali_c55_isp, notifier);
> >>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>>>> +    struct media_pad *pad = &isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO];
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * By default we'll flag this link enabled and the TPG disabled, but
> >>>>>> +     * no immutable flag because we need to be able to switch between the
> >>>>>> +     * two.
> >>>>>> +     */
> >>>>>> +    ret = v4l2_create_fwnode_links_to_pad(subdev, pad,
> >>>>>> +                          MEDIA_LNK_FL_ENABLED);
> >>>>>> +    if (ret)
> >>>>>> +        dev_err(mali_c55->dev, "failed to create link for %s\n",
> >>>>>> +            subdev->name);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_isp_notifier_complete(struct v4l2_async_notifier *notifier)
> >>>>>> +{
> >>>>>> +    struct mali_c55_isp *isp = container_of(notifier,
> >>>>>> +                        struct mali_c55_isp, notifier);
> >>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>>>> +
> >>>>>> +    return v4l2_device_register_subdev_nodes(&mali_c55->v4l2_dev);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_async_notifier_operations mali_c55_isp_notifier_ops = {
> >>>>>> +    .bound = mali_c55_isp_notifier_bound,
> >>>>>> +    .complete = mali_c55_isp_notifier_complete,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_isp_parse_endpoint(struct mali_c55_isp *isp)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = isp->mali_c55;
> >>>>>> +    struct v4l2_async_connection *asc;
> >>>>>> +    struct fwnode_handle *ep;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    v4l2_async_nf_init(&isp->notifier, &mali_c55->v4l2_dev);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * The ISP should have a single endpoint pointing to some flavour of
> >>>>>> +     * CSI-2 receiver...but for now at least we do want everything to work
> >>>>>> +     * normally even with no sensors connected, as we have the TPG. If we
> >>>>>> +     * don't find a sensor just warn and return success.
> >>>>>> +     */
> >>>>>> +    ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(mali_c55->dev),
> >>>>>> +                         0, 0, 0);
> >>>>>> +    if (!ep) {
> >>>>>> +        dev_warn(mali_c55->dev, "no local endpoint found\n");
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    asc = v4l2_async_nf_add_fwnode_remote(&isp->notifier, ep,
> >>>>>> +                          struct v4l2_async_connection);
> >>>>>> +    if (IS_ERR(asc)) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to add remote fwnode\n");
> >>>>>> +        ret = PTR_ERR(asc);
> >>>>>> +        goto err_put_ep;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    isp->notifier.ops = &mali_c55_isp_notifier_ops;
> >>>>>> +    ret = v4l2_async_nf_register(&isp->notifier);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "failed to register notifier\n");
> >>>>>> +        goto err_cleanup_nf;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    fwnode_handle_put(ep);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_cleanup_nf:
> >>>>>> +    v4l2_async_nf_cleanup(&isp->notifier);
> >>>>>> +err_put_ep:
> >>>>>> +    fwnode_handle_put(ep);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +int mali_c55_register_isp(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
> >>>>>> +    struct v4l2_subdev *sd = &isp->sd;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    isp->mali_c55 = mali_c55;
> >>>>>> +
> >>>>>> +    v4l2_subdev_init(sd, &mali_c55_isp_ops);
> >>>>>> +    sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> >>>>>> +    sd->entity.ops = &mali_c55_isp_media_ops;
> >>>>>> +    sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_ISP;
> >>>>>> +    sd->internal_ops = &mali_c55_isp_internal_ops;
> >>>>>> +    strscpy(sd->name, MALI_C55_DRIVER_NAME " isp", sizeof(sd->name));
> >>>>>> +
> >>>>>> +    isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
> >>>>> 
> >>>>> The MUST_CONNECT flag would make sense here.
> >>>>>
> >>>>>> + isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> >>>>>> +    isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
> >>>>>> +
> >>>>>> +    ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
> >>>>>> +                     isp->pads);
> >>>>>> +    if (ret)
> >>>>>> +        return ret;
> >>>>>> +
> >>>>>> +    ret = v4l2_subdev_init_finalize(sd);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_cleanup_media_entity;
> >>>>>> +
> >>>>>> +    ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_cleanup_subdev;
> >>>>>> +
> >>>>>> +    ret = mali_c55_isp_parse_endpoint(isp);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_cleanup_subdev;
> >>>>> 
> >>>>> As noted elsewhere, I think this belongs to mali-c55-core.c.
> >>>>>
> >>>>>> +
> >>>>>> +    mutex_init(&isp->lock);
> >>>>> 
> >>>>> This lock is used in mali-c55-capture.c only, that seems weird.
> >>>>>
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_cleanup_subdev:
> >>>>>> +    v4l2_subdev_cleanup(sd);
> >>>>>> +err_cleanup_media_entity:
> >>>>>> +    media_entity_cleanup(&sd->entity);
> >>>>>> +    isp->mali_c55 = NULL;
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_unregister_isp(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_isp *isp = &mali_c55->isp;
> >>>>>> +
> >>>>>> +    if (!isp->mali_c55)
> >>>>>> +        return;
> >>>>>> +
> >>>>>> +    mutex_destroy(&isp->lock);
> >>>>>> +    v4l2_async_nf_unregister(&isp->notifier);
> >>>>>> +    v4l2_async_nf_cleanup(&isp->notifier);
> >>>>>> +    v4l2_device_unregister_subdev(&isp->sd);
> >>>>>> +    v4l2_subdev_cleanup(&isp->sd);
> >>>>>> +    media_entity_cleanup(&isp->sd.entity);
> >>>>>> +}
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..cb27abde2aa5
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>>>> @@ -0,0 +1,258 @@
> >>>>>> +/* SPDX-License-Identifier: GPL-2.0 */
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Register definitions
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#ifndef _MALI_C55_REGISTERS_H
> >>>>>> +#define _MALI_C55_REGISTERS_H
> >>>>>> +
> >>>>>> +#include <linux/bits.h>
> >>>>>> +
> >>>>>> +/* ISP Common 0x00000 - 0x000ff */
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_API                0x00000
> >>>>>> +#define MALI_C55_REG_PRODUCT                0x00004
> >>>>>> +#define MALI_C55_REG_VERSION                0x00008
> >>>>>> +#define MALI_C55_REG_REVISION                0x0000c
> >>>>>> +#define MALI_C55_REG_PULSE_MODE                0x0003c
> >>>>>> +#define MALI_C55_REG_INPUT_MODE_REQUEST            0x0009c
> >>>>>> +#define MALI_C55_INPUT_SAFE_STOP            0x00
> >>>>>> +#define MALI_C55_INPUT_SAFE_START            0x01
> >>>>>> +#define MALI_C55_REG_MODE_STATUS            0x000a0
> >>>>>> +#define MALI_C55_REG_INTERRUPT_MASK_VECTOR        0x00030
> >>>>>> +#define MALI_C55_INTERRUPT_MASK_ALL            GENMASK(31, 0)
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_GLOBAL_MONITOR            0x00050
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_GEN_VIDEO                0x00080
> >>>>>> +#define MALI_C55_REG_GEN_VIDEO_ON_MASK            BIT(0)
> >>>>>> +#define MALI_C55_REG_GEN_VIDEO_MULTI_MASK        BIT(1)
> >>>>>> +#define MALI_C55_REG_GEN_PREFETCH_MASK GENMASK(31, 16)
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_MCU_CONFIG                0x00020
> >>>>>> +#define MALI_C55_REG_MCU_CONFIG_OVERRIDE_MASK        BIT(0)
> >>>>> 
> >>>>> #define MALI_C55_REG_MCU_CONFIG_OVERRIDE        BIT(0)
> >>>>>
> >>>>> Same in other places where applicable.
> >>>>>
> >>>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_MASK        BIT(1)
> >>>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PING        BIT(1)
> >>>>>> +#define MALI_C55_REG_MCU_CONFIG_WRITE_PONG        0x00
> >>>>>> +#define MALI_C55_REG_MULTI_CONTEXT_MODE_MASK        BIT(8)
> >>>>>> +#define MALI_C55_REG_PING_PONG_READ            0x00024
> >>>>>> +#define MALI_C55_REG_PING_PONG_READ_MASK        BIT(2)
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_INTERRUPT_CLEAR_VECTOR        0x00034
> >>>>>> +#define MALI_C55_REG_INTERRUPT_CLEAR            0x00040
> >>>>>> +#define MALI_C55_REG_INTERRUPT_STATUS_VECTOR        0x00044
> >>>>>> +#define MALI_C55_REG_GLOBAL_PARAMETER_STATUS        0x00068
> >>>>>> +#define MALI_C55_GPS_PONG_FITTED            BIT(0)
> >>>>>> +#define MALI_C55_GPS_WDR_FITTED                BIT(1)
> >>>>>> +#define MALI_C55_GPS_COMPRESSION_FITTED            BIT(2)
> >>>>>> +#define MALI_C55_GPS_TEMPER_FITTED            BIT(3)
> >>>>>> +#define MALI_C55_GPS_SINTER_LITE_FITTED            BIT(4)
> >>>>>> +#define MALI_C55_GPS_SINTER_FITTED            BIT(5)
> >>>>>> +#define MALI_C55_GPS_IRIDIX_LTM_FITTED            BIT(6)
> >>>>>> +#define MALI_C55_GPS_IRIDIX_GTM_FITTED            BIT(7)
> >>>>>> +#define MALI_C55_GPS_CNR_FITTED                BIT(8)
> >>>>>> +#define MALI_C55_GPS_FRSCALER_FITTED            BIT(9)
> >>>>>> +#define MALI_C55_GPS_DS_PIPE_FITTED            BIT(10)
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_BLANKING                0x00084
> >>>>>> +#define MALI_C55_REG_HBLANK_MASK            GENMASK(15, 0)
> >>>>>> +#define MALI_C55_REG_VBLANK_MASK            GENMASK(31, 16)
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_HC_START                0x00088
> >>>>>> +#define MALI_C55_HC_START(h)                (((h) & 0xffff) << 16)
> >>>>>> +#define MALI_C55_REG_HC_SIZE                0x0008c
> >>>>>> +#define MALI_C55_HC_SIZE(h)                ((h) & 0xffff)
> >>>>>> +#define MALI_C55_REG_VC_START_SIZE            0x00094
> >>>>>> +#define MALI_C55_VC_START(v)                ((v) & 0xffff)
> >>>>>> +#define MALI_C55_VC_SIZE(v)                (((v) & 0xffff) << 16)
> >>>>>> +
> >>>>>> +/* Ping/Pong Configuration Space */
> >>>>>> +#define MALI_C55_REG_BASE_ADDR                0x18e88
> >>>>>> +#define MALI_C55_REG_BYPASS_0                0x18eac
> >>>>>> +#define MALI_C55_REG_BYPASS_0_VIDEO_TEST        BIT(0)
> >>>>>> +#define MALI_C55_REG_BYPASS_0_INPUT_FMT            BIT(1)
> >>>>>> +#define MALI_C55_REG_BYPASS_0_DECOMPANDER        BIT(2)
> >>>>>> +#define MALI_C55_REG_BYPASS_0_SENSOR_OFFSET_WDR BIT(3)
> >>>>>> +#define MALI_C55_REG_BYPASS_0_GAIN_WDR            BIT(4)
> >>>>>> +#define MALI_C55_REG_BYPASS_0_FRAME_STITCH        BIT(5)
> >>>>>> +#define MALI_C55_REG_BYPASS_1                0x18eb0
> >>>>>> +#define MALI_C55_REG_BYPASS_1_DIGI_GAIN            BIT(0)
> >>>>>> +#define MALI_C55_REG_BYPASS_1_FE_SENSOR_OFFS        BIT(1)
> >>>>>> +#define MALI_C55_REG_BYPASS_1_FE_SQRT            BIT(2)
> >>>>>> +#define MALI_C55_REG_BYPASS_1_RAW_FE            BIT(3)
> >>>>>> +#define MALI_C55_REG_BYPASS_2                0x18eb8
> >>>>>> +#define MALI_C55_REG_BYPASS_2_SINTER            BIT(0)
> >>>>>> +#define MALI_C55_REG_BYPASS_2_TEMPER            BIT(1)
> >>>>>> +#define MALI_C55_REG_BYPASS_3                0x18ebc
> >>>>>> +#define MALI_C55_REG_BYPASS_3_SQUARE_BE            BIT(0)
> >>>>>> +#define MALI_C55_REG_BYPASS_3_SENSOR_OFFSET_PRE_SH BIT(1)
> >>>>>> +#define MALI_C55_REG_BYPASS_3_MESH_SHADING        BIT(3)
> >>>>>> +#define MALI_C55_REG_BYPASS_3_WHITE_BALANCE        BIT(4)
> >>>>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX            BIT(5)
> >>>>>> +#define MALI_C55_REG_BYPASS_3_IRIDIX_GAIN        BIT(6)
> >>>>>> +#define MALI_C55_REG_BYPASS_4                0x18ec0
> >>>>>> +#define MALI_C55_REG_BYPASS_4_DEMOSAIC_RGB        BIT(1)
> >>>>>> +#define MALI_C55_REG_BYPASS_4_PF_CORRECTION        BIT(3)
> >>>>>> +#define MALI_C55_REG_BYPASS_4_CCM            BIT(4)
> >>>>>> +#define MALI_C55_REG_BYPASS_4_CNR            BIT(5)
> >>>>>> +#define MALI_C55_REG_FR_BYPASS                0x18ec4
> >>>>>> +#define MALI_C55_REG_DS_BYPASS                0x18ec8
> >>>>>> +#define MALI_C55_BYPASS_CROP                BIT(0)
> >>>>>> +#define MALI_C55_BYPASS_SCALER                BIT(1)
> >>>>>> +#define MALI_C55_BYPASS_GAMMA_RGB            BIT(2)
> >>>>>> +#define MALI_C55_BYPASS_SHARPEN                BIT(3)
> >>>>>> +#define MALI_C55_BYPASS_CS_CONV                BIT(4)
> >>>>>> +#define MALI_C55_REG_ISP_RAW_BYPASS            0x18ecc
> >>>>>> +#define MALI_C55_ISP_RAW_BYPASS_BYPASS_MASK        BIT(0)
> >>>>>> +#define MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK GENMASK(9, 8)
> >>>>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS        2
> >>>>>> +#define MALI_C55_ISP_RAW_BYPASS_RGB_FR_BYPASS        1
> >>>>>> +#define MALI_C55_ISP_RAW_BYPASS_DS_PIPE_DISABLE BIT(1)
> >>>>>> +#define MALI_C55_ISP_RAW_BYPASS_RAW_BYPASS        BIT(0)
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_ACTIVE_WIDTH_MASK            0xffff
> >>>>>> +#define MALI_C55_REG_ACTIVE_HEIGHT_MASK 0xffff0000
> >>>>>> +#define MALI_C55_REG_BAYER_ORDER            0x18e8c
> >>>>>> +#define MALI_C55_BAYER_ORDER_MASK            GENMASK(1, 0)
> >>>>>> +#define MALI_C55_REG_TPG_CH0                0x18ed8
> >>>>>> +#define MALI_C55_TEST_PATTERN_ON_OFF            BIT(0)
> >>>>>> +#define MALI_C55_TEST_PATTERN_RGB_MASK            BIT(1)
> >>>>>> +#define MALI_C55_REG_TPG_R_BACKGROUND            0x18ee0
> >>>>>> +#define MALI_C55_REG_TPG_G_BACKGROUND            0x18ee4
> >>>>>> +#define MALI_C55_REG_TPG_B_BACKGROUND            0x18ee8
> >>>>>> +#define MALI_C55_TPG_BACKGROUND_MAX            0xfffff
> >>>>>> +#define MALI_C55_REG_INPUT_WIDTH            0x18f98
> >>>>>> +#define MALI_C55_INPUT_WIDTH_MASK            GENMASK(18, 16)
> >>>>>> +#define MALI_C55_INPUT_WIDTH_8BIT            0
> >>>>>> +#define MALI_C55_INPUT_WIDTH_10BIT            1
> >>>>>> +#define MALI_C55_INPUT_WIDTH_12BIT            2
> >>>>>> +#define MALI_C55_INPUT_WIDTH_14BIT            3
> >>>>>> +#define MALI_C55_INPUT_WIDTH_16BIT            4
> >>>>>> +#define MALI_C55_INPUT_WIDTH_20BIT            5
> >>>>>> +#define MALI_C55_REG_SPACE_SIZE                0x4000
> >>>>>> +#define MALI_C55_REG_CONFIG_SPACES_OFFSET        0x0ab6c
> >>>>>> +#define MALI_C55_CONFIG_SPACE_SIZE            0x1231c
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_SINTER_CONFIG            0x19348
> >>>>>> +#define MALI_C55_SINTER_VIEW_FILTER_MASK        GENMASK(1, 0)
> >>>>>> +#define MALI_C55_SINTER_SCALE_MODE_MASK GENMASK(3, 2)
> >>>>>> +#define MALI_C55_SINTER_ENABLE_MASK            BIT(4)
> >>>>>> +#define MALI_C55_SINTER_FILTER_SELECT_MASK        BIT(5)
> >>>>>> +#define MALI_C55_SINTER_INT_SELECT_MASK            BIT(6)
> >>>>>> +#define MALI_C55_SINTER_RM_ENABLE_MASK            BIT(7)
> >>>>>> +
> >>>>>> +/* Colour Correction Matrix Configuration */
> >>>>>> +#define MALI_C55_REG_CCM_ENABLE                0x1b07c
> >>>>>> +#define MALI_C55_CCM_ENABLE_MASK            BIT(0)
> >>>>>> +#define MALI_C55_REG_CCM_COEF_R_R            0x1b080
> >>>>>> +#define MALI_C55_REG_CCM_COEF_R_G            0x1b084
> >>>>>> +#define MALI_C55_REG_CCM_COEF_R_B            0x1b088
> >>>>>> +#define MALI_C55_REG_CCM_COEF_G_R            0x1b090
> >>>>>> +#define MALI_C55_REG_CCM_COEF_G_G            0x1b094
> >>>>>> +#define MALI_C55_REG_CCM_COEF_G_B            0x1b098
> >>>>>> +#define MALI_C55_REG_CCM_COEF_B_R            0x1b0a0
> >>>>>> +#define MALI_C55_REG_CCM_COEF_B_G            0x1b0a4
> >>>>>> +#define MALI_C55_REG_CCM_COEF_B_B            0x1b0a8
> >>>>>> +#define MALI_C55_CCM_COEF_MASK                GENMASK(12, 0)
> >>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_R            0x1b0b0
> >>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_G            0x1b0b4
> >>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_GAIN_B            0x1b0b8
> >>>>>> +#define MALI_C55_CCM_ANTIFOG_GAIN_MASK GENMASK(11, 0)
> >>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_R        0x1b0c0
> >>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_G        0x1b0c4
> >>>>>> +#define MALI_C55_REG_CCM_ANTIFOG_OFFSET_B        0x1b0c8
> >>>>>> +#define MALI_C55_CCM_ANTIFOG_OFFSET_MASK        GENMASK(11, 0)
> >>>>>> +
> >>>>>> +/*
> >>>>>> + * The Mali-C55 ISP has up to two output pipes; known as full resolution and
> >>>>>> + * down scaled. The register space for these is laid out identically, but offset
> >>>>>> + * by 372 bytes.
> >>>>>> + */
> >>>>>> +#define MALI_C55_CAP_DEV_FR_REG_OFFSET        0x0
> >>>>>> +#define MALI_C55_CAP_DEV_DS_REG_OFFSET        0x174
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_CS_CONV_CONFIG(offset)        (0x1c098 + (offset))
> >>>>>> +#define MALI_C55_CS_CONV_MATRIX_MASK            BIT(0)
> >>>>>> +#define MALI_C55_CS_CONV_FILTER_MASK            BIT(1)
> >>>>>> +#define MALI_C55_CS_CONV_HORZ_DOWNSAMPLE_MASK        BIT(2)
> >>>>>> +#define MALI_C55_CS_CONV_VERT_DOWNSAMPLE_MASK        BIT(3)
> >>>>>> +#define MALI_C55_REG_Y_WRITER_MODE(offset)        (0x1c0ec + (offset))
> >>>>>> +#define MALI_C55_REG_UV_WRITER_MODE(offset)        (0x1c144 + (offset))
> >>>>>> +#define MALI_C55_WRITER_MODE_MASK            GENMASK(4, 0)
> >>>>>> +#define MALI_C55_WRITER_SUBMODE_MASK            GENMASK(7, 6)
> >>>>>> +#define MALI_C55_WRITER_FRAME_WRITE_MASK        BIT(9)
> >>>>>> +#define MALI_C55_REG_ACTIVE_OUT_Y_SIZE(offset) (0x1c0f0 + (offset))
> >>>>>> +#define MALI_C55_REG_ACTIVE_OUT_UV_SIZE(offset) (0x1c148 + (offset))
> >>>>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_W(w)        ((w) << 0)
> >>>>>> +#define MALI_C55_REG_ACTIVE_OUT_SIZE_H(h)        ((h) << 16)
> >>>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_BASE(offset) (0x1c0f4 + (offset))
> >>>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_CONFIG(offset) (0x1c108 + (offset))
> >>>>>> +#define MALI_C55_REG_Y_WRITER_MAX_BANKS_MASK GENMASK(2, 0)
> >>>>>> +#define MALI_C55_REG_Y_WRITER_BANKS_RESTART        BIT(3)
> >>>>>> +#define MALI_C55_REG_Y_WRITER_OFFSET(offset) (0x1c10c + (offset))
> >>>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_BASE(offset) (0x1c14c + (offset))
> >>>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_CONFIG(offset) (0x1c160 + (offset))
> >>>>>> +#define MALI_C55_REG_UV_WRITER_MAX_BANKS_MASK GENMASK(2, 0)
> >>>>>> +#define MALI_C55_REG_UV_WRITER_BANKS_RESTART        BIT(3)
> >>>>>> +#define MALI_C55_REG_UV_WRITER_OFFSET(offset) (0x1c164 + (offset))
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_TEST_GEN_CH0_OFF_ON
> >>>>>> +#define MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE 0x18edc
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_CROP_EN(offset)            (0x1c028 + (offset))
> >>>>>> +#define MALI_C55_CROP_ENABLE                BIT(0)
> >>>>>> +#define MALI_C55_REG_CROP_X_START(offset)        (0x1c02c + (offset))
> >>>>>> +#define MALI_C55_REG_CROP_Y_START(offset)        (0x1c030 + (offset))
> >>>>>> +#define MALI_C55_REG_CROP_X_SIZE(offset)        (0x1c034 + (offset))
> >>>>>> +#define MALI_C55_REG_CROP_Y_SIZE(offset)        (0x1c038 + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_TIMEOUT_EN(offset) (0x1c040 + (offset))
> >>>>>> +#define MALI_C55_SCALER_TIMEOUT_EN            BIT(4)
> >>>>>> +#define MALI_C55_SCALER_TIMEOUT(t)            ((t) << 16)
> >>>>>> +#define MALI_C55_REG_SCALER_IN_WIDTH(offset) (0x1c044 + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_IN_HEIGHT(offset) (0x1c048 + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_OUT_WIDTH(offset) (0x1c04c + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_OUT_HEIGHT(offset) (0x1c050 + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_HFILT_TINC(offset) (0x1c054 + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_HFILT_COEF(offset) (0x1c058 + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_VFILT_TINC(offset) (0x1c05c + (offset))
> >>>>>> +#define MALI_C55_REG_SCALER_VFILT_COEF(offset) (0x1c060 + (offset))
> >>>>>> +
> >>>>>> +#define MALI_C55_REG_GAMMA_RGB_ENABLE(offset) (0x1c064 + (offset))
> >>>>>> +#define MALI_C55_GAMMA_ENABLE_MASK            BIT(0)
> >>>>>> +#define MALI_C55_REG_GAMMA_GAINS_1(offset)        (0x1c068 + (offset))
> >>>>>> +#define MALI_C55_GAMMA_GAIN_R_MASK            GENMASK(11, 0)
> >>>>>> +#define MALI_C55_GAMMA_GAIN_G_MASK            GENMASK(27, 16)
> >>>>>> +#define MALI_C55_REG_GAMMA_GAINS_2(offset)        (0x1c06c + (offset))
> >>>>>> +#define MALI_C55_GAMMA_GAIN_B_MASK            GENMASK(11, 0)
> >>>>>> +#define MALI_C55_REG_GAMMA_OFFSETS_1(offset) (0x1c070 + (offset))
> >>>>>> +#define MALI_C55_GAMMA_OFFSET_R_MASK            GENMASK(11, 0)
> >>>>>> +#define MALI_C55_GAMMA_OFFSET_G_MASK            GENMASK(27, 16)
> >>>>>> +#define MALI_C55_REG_GAMMA_OFFSETS_2(offset) (0x1c074 + (offset))
> >>>>>> +#define MALI_C55_GAMMA_OFFSET_B_MASK            GENMASK(11, 0)
> >>>>>> +
> >>>>>> +/* Output DMA Writer */
> >>>>>> +
> >>>>>> +#define MALI_C55_OUTPUT_DISABLED        0
> >>>>>> +#define MALI_C55_OUTPUT_RGB32            1
> >>>>>> +#define MALI_C55_OUTPUT_A2R10G10B10        2
> >>>>>> +#define MALI_C55_OUTPUT_RGB565            3
> >>>>>> +#define MALI_C55_OUTPUT_RGB24            4
> >>>>>> +#define MALI_C55_OUTPUT_GEN32            5
> >>>>>> +#define MALI_C55_OUTPUT_RAW16            6
> >>>>>> +#define MALI_C55_OUTPUT_AYUV            8
> >>>>>> +#define MALI_C55_OUTPUT_Y410            9
> >>>>>> +#define MALI_C55_OUTPUT_YUY2            10
> >>>>>> +#define MALI_C55_OUTPUT_UYVY            11
> >>>>>> +#define MALI_C55_OUTPUT_Y210            12
> >>>>>> +#define MALI_C55_OUTPUT_NV12_21            13
> >>>>>> +#define MALI_C55_OUTPUT_YUV_420_422        17
> >>>>>> +#define MALI_C55_OUTPUT_P210_P010        19
> >>>>>> +#define MALI_C55_OUTPUT_YUV422            20
> >>>>> 
> >>>>> I'd define this just below the MALI_C55_REG_UV_WRITER_MODE register
> >>>>> macro.
> >>>>>
> >>>>>> +
> >>>>>> +#define MALI_C55_OUTPUT_PLANE_ALT0        0
> >>>>>> +#define MALI_C55_OUTPUT_PLANE_ALT1        1
> >>>>>> +#define MALI_C55_OUTPUT_PLANE_ALT2        2
> >>>>> 
> >>>>> Same here ?
> >>>>>
> >>>>>> +
> >>>>>> +#endif /* _MALI_C55_REGISTERS_H */
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..8edae87f1e5f
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer-coefs.h
> >>>>>> @@ -0,0 +1,382 @@
> >>>>>> +/* SPDX-License-Identifier: GPL-2.0 */
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Resizer Coefficients
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#ifndef _MALI_C55_RESIZER_COEFS_H
> >>>>>> +#define _MALI_C55_RESIZER_COEFS_H
> >>>>>> +
> >>>>>> +#include "mali-c55-common.h"
> >>>>>> +
> >>>>>> +#define MALI_C55_RESIZER_COEFS_NUM_BANKS    8
> >>>>>> +#define MALI_C55_RESIZER_COEFS_NUM_ENTRIES    64
> >>>>> 
> >>>>> Do these belongs to mali-c55-registers.h ?
> >>>>>
> >>>>>> +
> >>>>>> +static const unsigned int mali_c55_scaler_h_filter_coefficients[8][64] = {
> >>>>>> +    {    /* Bank 0 */
> >>>>>> +        0x24fc0000, 0x0000fc24, 0x27fc0000, 0x0000fc21,
> >>>>>> +        0x28fc0000, 0x0000fd1f, 0x2cfb0000, 0x0000fd1c,
> >>>>>> +        0x2efb0000, 0x0000fd1a, 0x30fb0000, 0x0000fe17,
> >>>>>> +        0x32fb0000, 0x0000fe15, 0x35fb0000, 0x0000fe12,
> >>>>>> +        0x35fc0000, 0x0000ff10, 0x37fc0000, 0x0000ff0e,
> >>>>>> +        0x39fc0000, 0x0000ff0c, 0x3afd0000, 0x0000ff0a,
> >>>>>> +        0x3afe0000, 0x00000008, 0x3cfe0000, 0x00000006,
> >>>>>> +        0x3dff0000, 0x00000004, 0x3d000000, 0x00000003,
> >>>>>> +        0x3c020000, 0x00000002, 0x3d030000, 0x00000000,
> >>>>>> +        0x3d040000, 0x000000ff, 0x3c060000, 0x000000fe,
> >>>>>> +        0x3a080000, 0x000000fe, 0x3a0aff00, 0x000000fd,
> >>>>>> +        0x390cff00, 0x000000fc, 0x370eff00, 0x000000fc,
> >>>>>> +        0x3510ff00, 0x000000fc, 0x3512fe00, 0x000000fb,
> >>>>>> +        0x3215fe00, 0x000000fb, 0x3017fe00, 0x000000fb,
> >>>>>> +        0x2e1afd00, 0x000000fb, 0x2c1cfd00, 0x000000fb,
> >>>>>> +        0x281ffd00, 0x000000fc, 0x2721fc00, 0x000000fc,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 1 */
> >>>>>> +        0x25fb0000, 0x0000fb25, 0x27fb0000, 0x0000fb23,
> >>>>>> +        0x29fb0000, 0x0000fb21, 0x2afc0000, 0x0000fb1f,
> >>>>>> +        0x2cfc0000, 0x0000fb1d, 0x2efc0000, 0x0000fb1b,
> >>>>>> +        0x2ffd0000, 0x0000fb19, 0x2ffe0000, 0x0000fc17,
> >>>>>> +        0x31fe0000, 0x0000fc15, 0x32ff0000, 0x0000fc13,
> >>>>>> +        0x3400ff00, 0x0000fc11, 0x3301ff00, 0x0000fd10,
> >>>>>> +        0x3402ff00, 0x0000fd0e, 0x3503ff00, 0x0000fd0c,
> >>>>>> +        0x3505ff00, 0x0000fd0a, 0x3506fe00, 0x0000fe09,
> >>>>>> +        0x3607fe00, 0x0000fe07, 0x3509fe00, 0x0000fe06,
> >>>>>> +        0x350afd00, 0x0000ff05, 0x350cfd00, 0x0000ff03,
> >>>>>> +        0x340efd00, 0x0000ff02, 0x3310fd00, 0x0000ff01,
> >>>>>> +        0x3411fc00, 0x0000ff00, 0x3213fc00, 0x000000ff,
> >>>>>> +        0x3115fc00, 0x000000fe, 0x2f17fc00, 0x000000fe,
> >>>>>> +        0x2f19fb00, 0x000000fd, 0x2e1bfb00, 0x000000fc,
> >>>>>> +        0x2c1dfb00, 0x000000fc, 0x2a1ffb00, 0x000000fc,
> >>>>>> +        0x2921fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 2 */
> >>>>>> +        0x1f010000, 0x0000011f, 0x21010000, 0x0000001e,
> >>>>>> +        0x21020000, 0x0000001d, 0x22020000, 0x0000001c,
> >>>>>> +        0x23030000, 0x0000ff1b, 0x2404ff00, 0x0000ff1a,
> >>>>>> +        0x2504ff00, 0x0000ff19, 0x2505ff00, 0x0000ff18,
> >>>>>> +        0x2606ff00, 0x0000fe17, 0x2607ff00, 0x0000fe16,
> >>>>>> +        0x2708ff00, 0x0000fe14, 0x2709ff00, 0x0000fe13,
> >>>>>> +        0x270aff00, 0x0000fe12, 0x280bfe00, 0x0000fe11,
> >>>>>> +        0x280cfe00, 0x0000fe10, 0x280dfe00, 0x0000fe0f,
> >>>>>> +        0x280efe00, 0x0000fe0e, 0x280ffe00, 0x0000fe0d,
> >>>>>> +        0x2810fe00, 0x0000fe0c, 0x2811fe00, 0x0000fe0b,
> >>>>>> +        0x2712fe00, 0x0000ff0a, 0x2713fe00, 0x0000ff09,
> >>>>>> +        0x2714fe00, 0x0000ff08, 0x2616fe00, 0x0000ff07,
> >>>>>> +        0x2617fe00, 0x0000ff06, 0x2518ff00, 0x0000ff05,
> >>>>>> +        0x2519ff00, 0x0000ff04, 0x241aff00, 0x0000ff04,
> >>>>>> +        0x231bff00, 0x00000003, 0x221c0000, 0x00000002,
> >>>>>> +        0x211d0000, 0x00000002, 0x211e0000, 0x00000001,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 3 */
> >>>>>> +        0x1b06ff00, 0x00ff061b, 0x1b07ff00, 0x00ff061a,
> >>>>>> +        0x1c07ff00, 0x00ff051a, 0x1c08ff00, 0x00ff0519,
> >>>>>> +        0x1c09ff00, 0x00ff0419, 0x1d09ff00, 0x00ff0418,
> >>>>>> +        0x1e0aff00, 0x00ff0317, 0x1e0aff00, 0x00ff0317,
> >>>>>> +        0x1e0bff00, 0x00ff0316, 0x1f0cff00, 0x00ff0215,
> >>>>>> +        0x1e0cff00, 0x00000215, 0x1e0dff00, 0x00000214,
> >>>>>> +        0x1e0e0000, 0x00000113, 0x1e0e0000, 0x00000113,
> >>>>>> +        0x1e0f0000, 0x00000112, 0x1f100000, 0x00000011,
> >>>>>> +        0x20100000, 0x00000010, 0x1f110000, 0x00000010,
> >>>>>> +        0x1e120100, 0x0000000f, 0x1e130100, 0x0000000e,
> >>>>>> +        0x1e130100, 0x0000000e, 0x1e140200, 0x0000ff0d,
> >>>>>> +        0x1e150200, 0x0000ff0c, 0x1f1502ff, 0x0000ff0c,
> >>>>>> +        0x1e1603ff, 0x0000ff0b, 0x1e1703ff, 0x0000ff0a,
> >>>>>> +        0x1e1703ff, 0x0000ff0a, 0x1d1804ff, 0x0000ff09,
> >>>>>> +        0x1c1904ff, 0x0000ff09, 0x1c1905ff, 0x0000ff08,
> >>>>>> +        0x1c1a05ff, 0x0000ff07, 0x1b1a06ff, 0x0000ff07,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 4 */
> >>>>>> +        0x17090000, 0x00000917, 0x18090000, 0x00000916,
> >>>>>> +        0x170a0100, 0x00000816, 0x170a0100, 0x00000816,
> >>>>>> +        0x180b0100, 0x00000715, 0x180b0100, 0x00000715,
> >>>>>> +        0x170c0100, 0x00000715, 0x190c0100, 0x00000614,
> >>>>>> +        0x180d0100, 0x00000614, 0x190d0200, 0x00000513,
> >>>>>> +        0x180e0200, 0x00000513, 0x180e0200, 0x00000513,
> >>>>>> +        0x1a0e0200, 0x00000412, 0x190f0200, 0x00000412,
> >>>>>> +        0x190f0300, 0x00000411, 0x18100300, 0x00000411,
> >>>>>> +        0x1a100300, 0x00000310, 0x18110400, 0x00000310,
> >>>>>> +        0x19110400, 0x0000030f, 0x19120400, 0x0000020f,
> >>>>>> +        0x1a120400, 0x0000020e, 0x18130500, 0x0000020e,
> >>>>>> +        0x18130500, 0x0000020e, 0x19130500, 0x0000020d,
> >>>>>> +        0x18140600, 0x0000010d, 0x19140600, 0x0000010c,
> >>>>>> +        0x17150700, 0x0000010c, 0x18150700, 0x0000010b,
> >>>>>> +        0x18150700, 0x0000010b, 0x17160800, 0x0000010a,
> >>>>>> +        0x17160800, 0x0000010a, 0x18160900, 0x00000009,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 5 */
> >>>>>> +        0x120b0300, 0x00030b12, 0x120c0300, 0x00030b11,
> >>>>>> +        0x110c0400, 0x00030b11, 0x110c0400, 0x00030b11,
> >>>>>> +        0x130c0400, 0x00020a11, 0x120d0400, 0x00020a11,
> >>>>>> +        0x110d0500, 0x00020a11, 0x110d0500, 0x00020a11,
> >>>>>> +        0x130d0500, 0x00010911, 0x130e0500, 0x00010910,
> >>>>>> +        0x120e0600, 0x00010910, 0x120e0600, 0x00010910,
> >>>>>> +        0x130e0600, 0x00010810, 0x120f0600, 0x00010810,
> >>>>>> +        0x120f0700, 0x00000810, 0x130f0700, 0x0000080f,
> >>>>>> +        0x140f0700, 0x0000070f, 0x130f0800, 0x0000070f,
> >>>>>> +        0x12100800, 0x0000070f, 0x12100801, 0x0000060f,
> >>>>>> +        0x13100801, 0x0000060e, 0x12100901, 0x0000060e,
> >>>>>> +        0x12100901, 0x0000060e, 0x13100901, 0x0000050e,
> >>>>>> +        0x13110901, 0x0000050d, 0x11110a02, 0x0000050d,
> >>>>>> +        0x11110a02, 0x0000050d, 0x12110a02, 0x0000040d,
> >>>>>> +        0x13110a02, 0x0000040c, 0x11110b03, 0x0000040c,
> >>>>>> +        0x11110b03, 0x0000040c, 0x12110b03, 0x0000030c,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 6 */
> >>>>>> +        0x0b0a0805, 0x00080a0c, 0x0b0a0805, 0x00080a0c,
> >>>>>> +        0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
> >>>>>> +        0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
> >>>>>> +        0x0d0a0805, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
> >>>>>> +        0x0b0b0806, 0x00070a0b, 0x0c0b0806, 0x0007090b,
> >>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>>>> +        0x0b0b0906, 0x0007090b, 0x0c0b0906, 0x0006090b,
> >>>>>> +        0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>>>> +        0x0b0b0907, 0x0006090b, 0x0c0b0907, 0x0006080b,
> >>>>>> +        0x0b0b0a07, 0x0006080b, 0x0c0b0a07, 0x0006080a,
> >>>>>> +        0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
> >>>>>> +        0x0d0b0a07, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >>>>>> +        0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 7 */
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +        0x0909090a, 0x00090909, 0x0909090a, 0x00090909,
> >>>>>> +    }
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const unsigned int mali_c55_scaler_v_filter_coefficients[8][64] = {
> >>>>>> +    {    /* Bank 0 */
> >>>>>> +        0x2424fc00, 0x000000fc, 0x2721fc00, 0x000000fc,
> >>>>>> +        0x281ffd00, 0x000000fc, 0x2c1cfd00, 0x000000fb,
> >>>>>> +        0x2e1afd00, 0x000000fb, 0x3017fe00, 0x000000fb,
> >>>>>> +        0x3215fe00, 0x000000fb, 0x3512fe00, 0x000000fb,
> >>>>>> +        0x3510ff00, 0x000000fc, 0x370eff00, 0x000000fc,
> >>>>>> +        0x390cff00, 0x000000fc, 0x3a0aff00, 0x000000fd,
> >>>>>> +        0x3a080000, 0x000000fe, 0x3c060000, 0x000000fe,
> >>>>>> +        0x3d040000, 0x000000ff, 0x3d030000, 0x00000000,
> >>>>>> +        0x3c020000, 0x00000002, 0x3d000000, 0x00000003,
> >>>>>> +        0x3dff0000, 0x00000004, 0x3cfe0000, 0x00000006,
> >>>>>> +        0x3afe0000, 0x00000008, 0x3afd0000, 0x0000ff0a,
> >>>>>> +        0x39fc0000, 0x0000ff0c, 0x37fc0000, 0x0000ff0e,
> >>>>>> +        0x35fc0000, 0x0000ff10, 0x35fb0000, 0x0000fe12,
> >>>>>> +        0x32fb0000, 0x0000fe15, 0x30fb0000, 0x0000fe17,
> >>>>>> +        0x2efb0000, 0x0000fd1a, 0x2cfb0000, 0x0000fd1c,
> >>>>>> +        0x28fc0000, 0x0000fd1f, 0x27fc0000, 0x0000fc21,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 1 */
> >>>>>> +        0x2525fb00, 0x000000fb, 0x2723fb00, 0x000000fb,
> >>>>>> +        0x2921fb00, 0x000000fb, 0x2a1ffb00, 0x000000fc,
> >>>>>> +        0x2c1dfb00, 0x000000fc, 0x2e1bfb00, 0x000000fc,
> >>>>>> +        0x2f19fb00, 0x000000fd, 0x2f17fc00, 0x000000fe,
> >>>>>> +        0x3115fc00, 0x000000fe, 0x3213fc00, 0x000000ff,
> >>>>>> +        0x3411fc00, 0x0000ff00, 0x3310fd00, 0x0000ff01,
> >>>>>> +        0x340efd00, 0x0000ff02, 0x350cfd00, 0x0000ff03,
> >>>>>> +        0x350afd00, 0x0000ff05, 0x3509fe00, 0x0000fe06,
> >>>>>> +        0x3607fe00, 0x0000fe07, 0x3506fe00, 0x0000fe09,
> >>>>>> +        0x3505ff00, 0x0000fd0a, 0x3503ff00, 0x0000fd0c,
> >>>>>> +        0x3402ff00, 0x0000fd0e, 0x3301ff00, 0x0000fd10,
> >>>>>> +        0x3400ff00, 0x0000fc11, 0x32ff0000, 0x0000fc13,
> >>>>>> +        0x31fe0000, 0x0000fc15, 0x2ffe0000, 0x0000fc17,
> >>>>>> +        0x2ffd0000, 0x0000fb19, 0x2efc0000, 0x0000fb1b,
> >>>>>> +        0x2cfc0000, 0x0000fb1d, 0x2afc0000, 0x0000fb1f,
> >>>>>> +        0x29fb0000, 0x0000fb21, 0x27fb0000, 0x0000fb23,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 2 */
> >>>>>> +        0x1f1f0100, 0x00000001, 0x211e0000, 0x00000001,
> >>>>>> +        0x211d0000, 0x00000002, 0x221c0000, 0x00000002,
> >>>>>> +        0x231bff00, 0x00000003, 0x241aff00, 0x0000ff04,
> >>>>>> +        0x2519ff00, 0x0000ff04, 0x2518ff00, 0x0000ff05,
> >>>>>> +        0x2617fe00, 0x0000ff06, 0x2616fe00, 0x0000ff07,
> >>>>>> +        0x2714fe00, 0x0000ff08, 0x2713fe00, 0x0000ff09,
> >>>>>> +        0x2712fe00, 0x0000ff0a, 0x2811fe00, 0x0000fe0b,
> >>>>>> +        0x2810fe00, 0x0000fe0c, 0x280ffe00, 0x0000fe0d,
> >>>>>> +        0x280efe00, 0x0000fe0e, 0x280dfe00, 0x0000fe0f,
> >>>>>> +        0x280cfe00, 0x0000fe10, 0x280bfe00, 0x0000fe11,
> >>>>>> +        0x270aff00, 0x0000fe12, 0x2709ff00, 0x0000fe13,
> >>>>>> +        0x2708ff00, 0x0000fe14, 0x2607ff00, 0x0000fe16,
> >>>>>> +        0x2606ff00, 0x0000fe17, 0x2505ff00, 0x0000ff18,
> >>>>>> +        0x2504ff00, 0x0000ff19, 0x2404ff00, 0x0000ff1a,
> >>>>>> +        0x23030000, 0x0000ff1b, 0x22020000, 0x0000001c,
> >>>>>> +        0x21020000, 0x0000001d, 0x21010000, 0x0000001e,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 3 */
> >>>>>> +        0x1b1b06ff, 0x0000ff06, 0x1b1a06ff, 0x0000ff07,
> >>>>>> +        0x1c1a05ff, 0x0000ff07, 0x1c1905ff, 0x0000ff08,
> >>>>>> +        0x1c1904ff, 0x0000ff09, 0x1d1804ff, 0x0000ff09,
> >>>>>> +        0x1e1703ff, 0x0000ff0a, 0x1e1703ff, 0x0000ff0a,
> >>>>>> +        0x1e1603ff, 0x0000ff0b, 0x1f1502ff, 0x0000ff0c,
> >>>>>> +        0x1e150200, 0x0000ff0c, 0x1e140200, 0x0000ff0d,
> >>>>>> +        0x1e130100, 0x0000000e, 0x1e130100, 0x0000000e,
> >>>>>> +        0x1e120100, 0x0000000f, 0x1f110000, 0x00000010,
> >>>>>> +        0x20100000, 0x00000010, 0x1f100000, 0x00000011,
> >>>>>> +        0x1e0f0000, 0x00000112, 0x1e0e0000, 0x00000113,
> >>>>>> +        0x1e0e0000, 0x00000113, 0x1e0dff00, 0x00000214,
> >>>>>> +        0x1e0cff00, 0x00000215, 0x1f0cff00, 0x00ff0215,
> >>>>>> +        0x1e0bff00, 0x00ff0316, 0x1e0aff00, 0x00ff0317,
> >>>>>> +        0x1e0aff00, 0x00ff0317, 0x1d09ff00, 0x00ff0418,
> >>>>>> +        0x1c09ff00, 0x00ff0419, 0x1c08ff00, 0x00ff0519,
> >>>>>> +        0x1c07ff00, 0x00ff051a, 0x1b07ff00, 0x00ff061a,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 4 */
> >>>>>> +        0x17170900, 0x00000009, 0x18160900, 0x00000009,
> >>>>>> +        0x17160800, 0x0000010a, 0x17160800, 0x0000010a,
> >>>>>> +        0x18150700, 0x0000010b, 0x18150700, 0x0000010b,
> >>>>>> +        0x17150700, 0x0000010c, 0x19140600, 0x0000010c,
> >>>>>> +        0x18140600, 0x0000010d, 0x19130500, 0x0000020d,
> >>>>>> +        0x18130500, 0x0000020e, 0x18130500, 0x0000020e,
> >>>>>> +        0x1a120400, 0x0000020e, 0x19120400, 0x0000020f,
> >>>>>> +        0x19110400, 0x0000030f, 0x18110400, 0x00000310,
> >>>>>> +        0x1a100300, 0x00000310, 0x18100300, 0x00000411,
> >>>>>> +        0x190f0300, 0x00000411, 0x190f0200, 0x00000412,
> >>>>>> +        0x1a0e0200, 0x00000412, 0x180e0200, 0x00000513,
> >>>>>> +        0x180e0200, 0x00000513, 0x190d0200, 0x00000513,
> >>>>>> +        0x180d0100, 0x00000614, 0x190c0100, 0x00000614,
> >>>>>> +        0x170c0100, 0x00000715, 0x180b0100, 0x00000715,
> >>>>>> +        0x180b0100, 0x00000715, 0x170a0100, 0x00000816,
> >>>>>> +        0x170a0100, 0x00000816, 0x18090000, 0x00000916,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 5 */
> >>>>>> +        0x12120b03, 0x0000030b, 0x12110b03, 0x0000030c,
> >>>>>> +        0x11110b03, 0x0000040c, 0x11110b03, 0x0000040c,
> >>>>>> +        0x13110a02, 0x0000040c, 0x12110a02, 0x0000040d,
> >>>>>> +        0x11110a02, 0x0000050d, 0x11110a02, 0x0000050d,
> >>>>>> +        0x13110901, 0x0000050d, 0x13100901, 0x0000050e,
> >>>>>> +        0x12100901, 0x0000060e, 0x12100901, 0x0000060e,
> >>>>>> +        0x13100801, 0x0000060e, 0x12100801, 0x0000060f,
> >>>>>> +        0x12100800, 0x0000070f, 0x130f0800, 0x0000070f,
> >>>>>> +        0x140f0700, 0x0000070f, 0x130f0700, 0x0000080f,
> >>>>>> +        0x120f0700, 0x00000810, 0x120f0600, 0x00010810,
> >>>>>> +        0x130e0600, 0x00010810, 0x120e0600, 0x00010910,
> >>>>>> +        0x120e0600, 0x00010910, 0x130e0500, 0x00010910,
> >>>>>> +        0x130d0500, 0x00010911, 0x110d0500, 0x00020a11,
> >>>>>> +        0x110d0500, 0x00020a11, 0x120d0400, 0x00020a11,
> >>>>>> +        0x130c0400, 0x00020a11, 0x110c0400, 0x00030b11,
> >>>>>> +        0x110c0400, 0x00030b11, 0x120c0300, 0x00030b11,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 6 */
> >>>>>> +        0x0b0c0a08, 0x0005080a, 0x0b0c0a08, 0x0005080a,
> >>>>>> +        0x0c0b0a08, 0x0005080a, 0x0c0b0a08, 0x0005080a,
> >>>>>> +        0x0d0b0a07, 0x0005080a, 0x0d0b0a07, 0x0005080a,
> >>>>>> +        0x0d0b0a07, 0x0005080a, 0x0c0b0a07, 0x0006080a,
> >>>>>> +        0x0b0b0a07, 0x0006080b, 0x0c0b0907, 0x0006080b,
> >>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>>>> +        0x0b0b0907, 0x0006090b, 0x0b0b0907, 0x0006090b,
> >>>>>> +        0x0b0b0907, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >>>>>> +        0x0c0b0906, 0x0006090b, 0x0c0b0906, 0x0006090b,
> >>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>>>> +        0x0b0b0906, 0x0007090b, 0x0b0b0906, 0x0007090b,
> >>>>>> +        0x0b0b0906, 0x0007090b, 0x0c0b0806, 0x0007090b,
> >>>>>> +        0x0b0b0806, 0x00070a0b, 0x0c0a0806, 0x00070a0b,
> >>>>>> +        0x0d0a0805, 0x00070a0b, 0x0d0a0805, 0x00070a0b,
> >>>>>> +        0x0d0a0805, 0x00070a0b, 0x0c0a0805, 0x00080a0b,
> >>>>>> +        0x0c0a0805, 0x00080a0b, 0x0c0a0805, 0x00080a0b,
> >>>>>> +    },
> >>>>>> +    {    /* Bank 7 */
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +        0x09090909, 0x000a0909, 0x09090909, 0x000a0909,
> >>>>>> +    }
> >>>>>> +};
> >>>>>> +
> >>>>>> +struct mali_c55_resizer_coef_bank {
> >>>>>> +    unsigned int bank;
> >>>>> 
> >>>>> This is always equal to the index of the entry in the
> >>>>> mali_c55_coefficient_banks array, you can drop it.
> >>>>>
> >>>>>> +    unsigned int top;
> >>>>>> +    unsigned int bottom;
> >>>>> 
> >>>>> The bottom value of bank N is always equal to the top value of bank N+1.
> >>>>> You can simplify this by storing a single value.
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const struct mali_c55_resizer_coef_bank mali_c55_coefficient_banks[] = {
> >>>>>> +    {
> >>>>>> +        .bank = 0,
> >>>>>> +        .top = 1000,
> >>>>>> +        .bottom = 770,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .bank = 1,
> >>>>>> +        .top = 769,
> >>>>>> +        .bottom = 600,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .bank = 2,
> >>>>>> +        .top = 599,
> >>>>>> +        .bottom = 460,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .bank = 3,
> >>>>>> +        .top = 459,
> >>>>>> +        .bottom = 354,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .bank = 4,
> >>>>>> +        .top = 353,
> >>>>>> +        .bottom = 273,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .bank = 5,
> >>>>>> +        .top = 272,
> >>>>>> +        .bottom = 210,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .bank = 6,
> >>>>>> +        .top = 209,
> >>>>>> +        .bottom = 162,
> >>>>>> +    },
> >>>>>> +    {
> >>>>>> +        .bank = 7,
> >>>>>> +        .top = 161,
> >>>>>> +        .bottom = 125,
> >>>>>> +    },
> >>>>>> +};
> >>>>>> +
> >>>>> 
> >>>>> A small comment would be nice, such as
> >>>>>
> >>>>> /* Select a bank of resizer coefficients, based on the scaling ratio. */
> >>>>>
> >>>>>> +static unsigned int mali_c55_calculate_bank_num(struct mali_c55 *mali_c55,
> >>>>> 
> >>>>> This function is related to the resizers. Add "rsz" somewhere in the
> >>>>> function name, and pass a resizer pointer.
> >>>>>
> >>>>>> +                        unsigned int crop,
> >>>>>> +                        unsigned int scale)
> >>>>> 
> >>>>> I think those are the input and output sizes to the scaler. Rename them
> >>>>> to make it clearer.
> >>>>>
> >>>>>> +{
> >>>>>> +    unsigned int tmp;
> >>>>> 
> >>>>> tmp is almost always a bad variable name. Please use a more descriptive
> >>>>> name, size as rsz_ratio.
> >>>>>
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    tmp = (scale * 1000U) / crop;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_coefficient_banks); i++) {
> >>>>>> +        if (tmp >= mali_c55_coefficient_banks[i].bottom &&
> >>>>>> +            tmp <= mali_c55_coefficient_banks[i].top)
> >>>>>> +            return mali_c55_coefficient_banks[i].bank;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * We shouldn't ever get here, in theory. As we have no good choices
> >>>>>> +     * simply warn the user and use the first bank of coefficients.
> >>>>>> +     */
> >>>>>> +    dev_warn(mali_c55->dev, "scaling factor outside defined bounds\n");
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>> 
> >>>>> And everything else belongs to mali-c55-resizer.c. Drop this header
> >>>>> file.
> >>>>>
> >>>>>> +
> >>>>>> +#endif /* _MALI_C55_RESIZER_COEFS_H */
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..0a5a2969d3ce
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-resizer.c
> >>>>>> @@ -0,0 +1,779 @@
> >>>>>> +// SPDX-License-Identifier: GPL-2.0
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Image signal processor
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#include <linux/math.h>
> >>>>>> +#include <linux/minmax.h>
> >>>>>> +
> >>>>>> +#include <media/media-entity.h>
> >>>>>> +#include <media/v4l2-subdev.h>
> >>>>>> +
> >>>>>> +#include "mali-c55-common.h"
> >>>>>> +#include "mali-c55-registers.h"
> >>>>>> +#include "mali-c55-resizer-coefs.h"
> >>>>>> +
> >>>>>> +/* Scaling factor in Q4.20 format. */
> >>>>>> +#define MALI_C55_RZR_SCALER_FACTOR    (1U << 20)
> >>>>>> +
> >>>>>> +static const u32 rzr_non_bypass_src_fmts[] = {
> >>>>>> +    MEDIA_BUS_FMT_RGB121212_1X36,
> >>>>>> +    MEDIA_BUS_FMT_YUV10_1X30
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const char * const mali_c55_resizer_names[] = {
> >>>>>> +    [MALI_C55_RZR_FR] = "resizer fr",
> >>>>>> +    [MALI_C55_RZR_DS] = "resizer ds",
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_program_crop(struct mali_c55_resizer *rzr,
> >>>>>> +                     struct v4l2_subdev_state *state)
> >>>>>> +{
> >>>>>> +    unsigned int reg_offset = rzr->cap_dev->reg_offset;
> >>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>>>> +    struct v4l2_mbus_framefmt *fmt;
> >>>>>> +    struct v4l2_rect *crop;
> >>>> 
> >>>> const
> >>>>
> >>>>>> +
> >>>>>> +    /* Verify if crop should be enabled. */
> >>>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>>>> +
> >>>>>> +    if (fmt->width == crop->width && fmt->height == crop->height)
> >>>>>> +        return MALI_C55_BYPASS_CROP;
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_START(reg_offset),
> >>>>>> +               crop->left);
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_START(reg_offset),
> >>>>>> +               crop->top);
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_X_SIZE(reg_offset),
> >>>>>> +               crop->width);
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_Y_SIZE(reg_offset),
> >>>>>> +               crop->height);
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_CROP_EN(reg_offset),
> >>>>>> +               MALI_C55_CROP_ENABLE);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_program_resizer(struct mali_c55_resizer *rzr,
> >>>>>> +                    struct v4l2_subdev_state *state)
> >>>>>> +{
> >>>>>> +    unsigned int reg_offset = rzr->cap_dev->reg_offset;
> >>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>>>> +    struct v4l2_rect *crop, *scale;
> >>>> 
> >>>> const
> >>>>
> >>>> Once "[PATCH v4 0/3] media: v4l2-subdev: Support const-awareness in
> >>>> state accessors" gets merged, the state argument to this function can be
> >>>> made const too. Same for other functions, as applicable.
> >>>>
> >>>>>> +    unsigned int h_bank, v_bank;
> >>>>>> +    u64 h_scale, v_scale;
> >>>>>> +
> >>>>>> +    /* Verify if scaling should be enabled. */
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>>>> +    scale = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD, 0);
> >>>>>> +
> >>>>>> +    if (crop->width == scale->width && crop->height == scale->height)
> >>>>>> +        return MALI_C55_BYPASS_SCALER;
> >>>>>> +
> >>>>>> +    /* Program the V/H scaling factor in Q4.20 format. */
> >>>>>> +    h_scale = crop->width * MALI_C55_RZR_SCALER_FACTOR;
> >>>>>> +    v_scale = crop->height * MALI_C55_RZR_SCALER_FACTOR;
> >>>>>> +
> >>>>>> +    do_div(h_scale, scale->width);
> >>>>>> +    do_div(v_scale, scale->height);
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_IN_WIDTH(reg_offset),
> >>>>>> +               crop->width);
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_IN_HEIGHT(reg_offset),
> >>>>>> +               crop->height);
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_OUT_WIDTH(reg_offset),
> >>>>>> +               scale->width);
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_OUT_HEIGHT(reg_offset),
> >>>>>> +               scale->height);
> >>>>>> +
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_HFILT_TINC(reg_offset),
> >>>>>> +               h_scale);
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_VFILT_TINC(reg_offset),
> >>>>>> +               v_scale);
> >>>>>> +
> >>>>>> +    h_bank = mali_c55_calculate_bank_num(mali_c55, crop->width,
> >>>>>> +                         scale->width);
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_HFILT_COEF(reg_offset),
> >>>>>> +               h_bank);
> >>>>>> +
> >>>>>> +    v_bank = mali_c55_calculate_bank_num(mali_c55, crop->height,
> >>>>>> +                         scale->height);
> >>>>>> +    mali_c55_write(mali_c55,
> >>>>>> +               MALI_C55_REG_SCALER_VFILT_COEF(reg_offset),
> >>>>>> +               v_bank);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static void mali_c55_rzr_program(struct mali_c55_resizer *rzr,
> >>>>>> +                 struct v4l2_subdev_state *state)
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>>>> +    u32 bypass = 0;
> >>>>>> +
> >>>>>> +    /* Verify if cropping and scaling should be enabled. */
> >>>>>> +    bypass |= mali_c55_rzr_program_crop(rzr, state);
> >>>>>> +    bypass |= mali_c55_rzr_program_resizer(rzr, state);
> >>>>>> +
> >>>>>> +    mali_c55_update_bits(mali_c55, rzr->id == MALI_C55_RZR_FR ?
> >>>>>> +                 MALI_C55_REG_FR_BYPASS : MALI_C55_REG_DS_BYPASS,
> >>>>>> +                 MALI_C55_BYPASS_CROP | MALI_C55_BYPASS_SCALER,
> >>>>>> +                 bypass);
> >>>>>> +}
> >>>>>> +
> >>>>>> +/*
> >>>>>> + * Inspect the routing table to know which of the two (mutually exclusive)
> >>>>>> + * routes is enabled and return the sink pad id of the active route.
> >>>>>> + */
> >>>>>> +static unsigned int mali_c55_rzr_get_active_sink(struct v4l2_subdev_state *state)
> >>>>>> +{
> >>>>>> +    struct v4l2_subdev_krouting *routing = &state->routing;
> >>>>>> +    struct v4l2_subdev_route *route;
> >>>>>> +
> >>>>>> +    /* A single route is enabled at a time. */
> >>>>>> +    for_each_active_route(routing, route)
> >>>>>> +        return route->sink_pad;
> >>>>>> +
> >>>>>> +    return MALI_C55_RZR_SINK_PAD;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static u32 mali_c55_rzr_shift_mbus_code(u32 mbus_code)
> >>>>>> +{
> >>>>>> +    u32 corrected_code = 0;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * The ISP takes input in a 20-bit format, but can only output 16-bit
> >>>>>> +     * RAW bayer data (with the 4 least significant bits from the input
> >>>>>> +     * being lost). Return the 16-bit version of the 20-bit input formats.
> >>>>>> +     */
> >>>>>> +    switch (mbus_code) {
> >>>>>> +    case MEDIA_BUS_FMT_SBGGR20_1X20:
> >>>>>> +        corrected_code = MEDIA_BUS_FMT_SBGGR16_1X16;
> >>>>>> +        break;
> >>>>>> +    case MEDIA_BUS_FMT_SGBRG20_1X20:
> >>>>>> +        corrected_code = MEDIA_BUS_FMT_SGBRG16_1X16;
> >>>>>> +        break;
> >>>>>> +    case MEDIA_BUS_FMT_SGRBG20_1X20:
> >>>>>> +        corrected_code = MEDIA_BUS_FMT_SGRBG16_1X16;
> >>>>>> +        break;
> >>>>>> +    case MEDIA_BUS_FMT_SRGGB20_1X20:
> >>>>>> +        corrected_code = MEDIA_BUS_FMT_SRGGB16_1X16;
> >>>>>> +        break;
> >>>> 
> >>>> Would it make sense to add the shifted code to mali_c55_isp_fmt ?
> >>>>
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return corrected_code;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int __mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> >>>>>> +                      struct v4l2_subdev_state *state,
> >>>>>> +                      struct v4l2_subdev_krouting *routing)
> >>>> 
> >>>> I think the last argument can be const.
> >>>>
> >>>>>> +{
> >>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>>>> +                            sd);
> >>>> 
> >>>> A to_mali_c55_resizer() static inline function would be useful. Same for
> >>>> other components, where applicable.
> >>>>
> >>>>>> +    unsigned int active_sink = UINT_MAX;
> >>>>>> +    struct v4l2_mbus_framefmt *src_fmt;
> >>>>>> +    struct v4l2_rect *crop, *compose;
> >>>>>> +    struct v4l2_subdev_route *route;
> >>>>>> +    unsigned int active_routes = 0;
> >>>>>> +    struct v4l2_mbus_framefmt *fmt;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    ret = v4l2_subdev_routing_validate(sd, routing, 0);
> >>>>>> +    if (ret)
> >>>>>> +        return ret;
> >>>>>> +
> >>>>>> +    /* Only a single route can be enabled at a time. */
> >>>>>> +    for_each_active_route(routing, route) {
> >>>>>> +        if (++active_routes > 1) {
> >>>>>> +            dev_err(rzr->mali_c55->dev,
> >>>>>> +                "Only one route can be active");
> >>>> 
> >>>> No kernel log message with a level higher than dev_dbg() from
> >>>> user-controlled paths please, here and where applicable. This is to
> >>>> avoid giving applications an easy way to flood the kernel log.
> >>>>
> >>>>>> +            return -EINVAL;
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        active_sink = route->sink_pad;
> >>>>>> +    }
> >>>>>> +    if (active_sink == UINT_MAX) {
> >>>>>> +        dev_err(rzr->mali_c55->dev, "One route has to be active");
> >>>>>> +        return -EINVAL;
> >>>>>> +    }
> >>>> 
> >>>> The recommended handling of invalid routing is to adjust the routing
> >>>> table, not to return errors.
> >>>>
> >>>>>> +
> >>>>>> +    ret = v4l2_subdev_set_routing(sd, state, routing);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(rzr->mali_c55->dev, "Failed to set routing\n");
> >>>>>> +        return ret;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    fmt = v4l2_subdev_state_get_format(state, active_sink, 0);
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, active_sink, 0);
> >>>>>> +    compose = v4l2_subdev_state_get_compose(state, active_sink, 0);
> >>>>>> +
> >>>>>> +    fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>>>> +    fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>>>> +    fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>> 
> >>>> There are other colorspace-related fields.
> >>>>
> >>>>>> +    fmt->field = V4L2_FIELD_NONE;
> >>>> 
> >>>> I wonder if we should really update the sink pad format, or just
> >>>> propagate it. If we update it, I think it should be set to defaults on
> >>>> both sink pads, not just the active sink pad.
> >>>>
> >>>>>> +
> >>>>>> +    if (active_sink == MALI_C55_RZR_SINK_PAD) {
> >>>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>>>> +
> >>>>>> +        crop->left = crop->top = 0;
> >>>> 
> >>>>          crop->left = 0;
> >>>>          crop->top = 0;
> >>>>
> >>>>>> +        crop->width = MALI_C55_DEFAULT_WIDTH;
> >>>>>> +        crop->height = MALI_C55_DEFAULT_HEIGHT;
> >>>>>> +
> >>>>>> +        *compose = *crop;
> >>>>>> +    } else {
> >>>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* Propagate the format to the source pad */
> >>>>>> +    src_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD,
> >>>>>> +                           0);
> >>>>>> +    *src_fmt = *fmt;
> >>>>>> +
> >>>>>> +    /* In the event this is the bypass pad the mbus code needs correcting */
> >>>>>> +    if (active_sink == MALI_C55_RZR_SINK_BYPASS_PAD)
> >>>>>> +        src_fmt->code = mali_c55_rzr_shift_mbus_code(src_fmt->code);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>> +                       struct v4l2_subdev_state *state,
> >>>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
> >>>>>> +{
> >>>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
> >>>>>> +    const struct mali_c55_isp_fmt *fmt;
> >>>>>> +    unsigned int index = 0;
> >>>>>> +    u32 sink_pad;
> >>>>>> +
> >>>>>> +    switch (code->pad) {
> >>>>>> +    case MALI_C55_RZR_SINK_PAD:
> >>>>>> +        if (code->index)
> >>>>>> +            return -EINVAL;
> >>>>>> +
> >>>>>> +        code->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>>>> +
> >>>>>> +        return 0;
> >>>>>> +    case MALI_C55_RZR_SOURCE_PAD:
> >>>>>> +        sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>>>> +        sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> >>>>>> +
> >>>>>> +        /*
> >>>>>> +         * If the active route is from the Bypass sink pad, then the
> >>>>>> +         * source pad is a simple passthrough of the sink format,
> >>>>>> +         * downshifted to 16-bits.
> >>>>>> +         */
> >>>>>> +
> >>>>>> +        if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >>>>>> +            if (code->index)
> >>>>>> +                return -EINVAL;
> >>>>>> +
> >>>>>> +            code->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> >>>>>> +            if (!code->code)
> >>>>>> +                return -EINVAL;
> >>>>>> +
> >>>>>> +            return 0;
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        /*
> >>>>>> +         * If the active route is from the non-bypass sink then we can
> >>>>>> +         * select either RGB or conversion to YUV.
> >>>>>> +         */
> >>>>>> +
> >>>>>> +        if (code->index >= ARRAY_SIZE(rzr_non_bypass_src_fmts))
> >>>>>> +            return -EINVAL;
> >>>>>> +
> >>>>>> +        code->code = rzr_non_bypass_src_fmts[code->index];
> >>>>>> +
> >>>>>> +        return 0;
> >>>>>> +    case MALI_C55_RZR_SINK_BYPASS_PAD:
> >>>>>> +        for_each_mali_isp_fmt(fmt) {
> >>>>>> +            if (index++ == code->index) {
> >>>>>> +                code->code = fmt->code;
> >>>>>> +                return 0;
> >>>>>> +            }
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        break;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return -EINVAL;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_enum_frame_size(struct v4l2_subdev *sd,
> >>>>>> +                    struct v4l2_subdev_state *state,
> >>>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
> >>>>>> +{
> >>>>>> +    if (fse->index)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
> >>>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
> >>>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
> >>>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_set_sink_fmt(struct v4l2_subdev *sd,
> >>>>>> +                     struct v4l2_subdev_state *state,
> >>>>>> +                     struct v4l2_subdev_format *format)
> >>>>>> +{
> >>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>>>> +    struct v4l2_rect *rect;
> >>>>>> +    unsigned int sink_pad;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * Clamp to min/max and then reset crop and compose rectangles to the
> >>>>>> +     * newly applied size.
> >>>>>> +     */
> >>>>>> +    clamp_t(unsigned int, fmt->width,
> >>>>>> +        MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>>>>> +    clamp_t(unsigned int, fmt->height,
> >>>>>> +        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>>> 
> >>>> Please check comments for other components related to the colorspace
> >>>> fields, to decide how to handle them here.
> >>>>
> >>>>>> +
> >>>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>>>> +    if (sink_pad == MALI_C55_RZR_SINK_PAD) {
> >>>> 
> >>>> The selection here should depend on format->pad, not the active sink
> >>>> pad.
> >>>>
> >>>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>>>> +
> >>>>>> +        rect = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> >>>>>> +        rect->left = 0;
> >>>>>> +        rect->top = 0;
> >>>>>> +        rect->width = fmt->width;
> >>>>>> +        rect->height = fmt->height;
> >>>>>> +
> >>>>>> +        rect = v4l2_subdev_state_get_compose(state,
> >>>>>> +                             MALI_C55_RZR_SINK_PAD);
> >>>>>> +        rect->left = 0;
> >>>>>> +        rect->top = 0;
> >>>>>> +        rect->width = fmt->width;
> >>>>>> +        rect->height = fmt->height;
> >>>>>> +    } else {
> >>>>>> +        /*
> >>>>>> +         * Make sure the media bus code is one of the supported
> >>>>>> +         * ISP input media bus codes.
> >>>>>> +         */
> >>>>>> +        if (!mali_c55_isp_is_format_supported(fmt->code))
> >>>>>> +            fmt->code = MALI_C55_DEFAULT_MEDIA_BUS_FMT;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    *v4l2_subdev_state_get_format(state, sink_pad, 0) = *fmt;
> >>>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD, 0) = *fmt;
> >>>> 
> >>>> Propagation to the source pad, however, should depend on the active
> >>>> route. If format->pad is routed to the source pad, you should propagate,
> >>>> otherwise, you shouldn't.
> >>>>
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> >>>>>> +                       struct v4l2_subdev_state *state,
> >>>>>> +                       struct v4l2_subdev_format *format)
> >>>>>> +{
> >>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>>>> +                            sd);
> >>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
> >>>>>> +    struct v4l2_rect *crop, *compose;
> >>>>>> +    unsigned int sink_pad;
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>>>> +    sink_fmt = v4l2_subdev_state_get_format(state, sink_pad, 0);
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, sink_pad, 0);
> >>>>>> +    compose = v4l2_subdev_state_get_compose(state, sink_pad, 0);
> >>>>>> +
> >>>>>> +    /* FR Bypass pipe. */
> >>>>>> +
> >>>>>> +    if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >>>>>> +        /*
> >>>>>> +         * Format on the source pad is the same as the one on the
> >>>>>> +         * sink pad, downshifted to 16-bits.
> >>>>>> +         */
> >>>>>> +        fmt->code = mali_c55_rzr_shift_mbus_code(sink_fmt->code);
> >>>>>> +        if (!fmt->code)
> >>>>>> +            return -EINVAL;
> >>>>>> +
> >>>>>> +        /* RAW bypass disables scaling and cropping. */
> >>>>>> +        crop->top = compose->top = 0;
> >>>>>> +        crop->left = compose->left = 0;
> >>>>>> +        fmt->width = crop->width = compose->width = sink_fmt->width;
> >>>>>> +        fmt->height = crop->height = compose->height = sink_fmt->height;
> >>>> 
> >>>> I don't think this is right. This function sets the format on the source
> >>>> pad. Subdevs should propagate formats from the sink to the source, not
> >>>> the other way around.
> >>>>
> >>>> The only parameter that can be modified on the source pad (as far as I
> >>>> understand) is the media bus code. In the bypass path, I understand it's
> >>>> fixed, while in the other path, you can select between RGB and YUV. I
> >>>> think the following code is what you need to implement this function.
> >>>>
> >>>> static int mali_c55_rzr_set_source_fmt(struct v4l2_subdev *sd,
> >>>>                         struct v4l2_subdev_state *state,
> >>>>                         struct v4l2_subdev_format *format)
> >>>> {
> >>>>      struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>>                              sd);
> >>>>      struct v4l2_mbus_framefmt *fmt;
> >>>>
> >>>>      fmt = v4l2_subdev_state_get_format(state, format->pad);
> >>>>
> >>>>      /* In the non-bypass path the output format can be selected. */
> >>>>      if (mali_c55_rzr_get_active_sink(state) == MALI_C55_RZR_SINK_PAD) {
> >>>>          unsigned int i;
> >>>>
> >>>>          fmt->code = format->format.code;
> >>>>
> >>>>          for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> >>>>              if (fmt->code == rzr_non_bypass_src_fmts[i])
> >>>>                  break;
> >>>>          }
> >>>>
> >>>>          if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts))
> >>>>              fmt->code = rzr_non_bypass_src_fmts[0];
> >>>>      }
> >>>>
> >>>>      format->format = *fmt;
> >>>>
> >>>>      return 0;
> >>>> }
> >>>>
> >>>>>> +
> >>>>>> +        *v4l2_subdev_state_get_format(state,
> >>>>>> +                          MALI_C55_RZR_SOURCE_PAD) = *fmt;
> >>>>>> +
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* Regular processing pipe. */
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(rzr_non_bypass_src_fmts); i++) {
> >>>>>> +        if (fmt->code == rzr_non_bypass_src_fmts[i])
> >>>>>> +            break;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    if (i == ARRAY_SIZE(rzr_non_bypass_src_fmts)) {
> >>>>>> +        dev_dbg(rzr->mali_c55->dev,
> >>>>>> +            "Unsupported mbus code 0x%x: using default\n",
> >>>>>> +            fmt->code);
> >>>> 
> >>>> I think you can drop this message.
> >>>>
> >>>>>> +        fmt->code = MEDIA_BUS_FMT_RGB121212_1X36;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * The source pad format size comes directly from the sink pad
> >>>>>> +     * compose rectangle.
> >>>>>> +     */
> >>>>>> +    fmt->width = compose->width;
> >>>>>> +    fmt->height = compose->height;
> >>>>>> +
> >>>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_RZR_SOURCE_PAD) = *fmt;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_set_fmt(struct v4l2_subdev *sd,
> >>>>>> +                struct v4l2_subdev_state *state,
> >>>>>> +                struct v4l2_subdev_format *format)
> >>>>>> +{
> >>>>>> +    /*
> >>>>>> +     * On sink pads fmt is either fixed for the 'regular' processing
> >>>>>> +     * pad or a RAW format or 20-bit wide RGB/YUV format for the FR bypass
> >>>>>> +     * pad.
> >>>>>> +     *
> >>>>>> +     * On source pad sizes are the result of crop+compose on the sink
> >>>>>> +     * pad sizes, while the format depends on the active route.
> >>>>>> +     */
> >>>>>> +
> >>>>>> +    if (format->pad != MALI_C55_RZR_SOURCE_PAD)
> >>>>>> +        return mali_c55_rzr_set_sink_fmt(sd, state, format);
> >>>>>> +
> >>>>>> +    return mali_c55_rzr_set_source_fmt(sd, state, format);
> >>>> 
> >>>> Nitpicking,
> >>>>
> >>>>      if (format->pad == MALI_C55_RZR_SOURCE_PAD)
> >>>>          return mali_c55_rzr_set_source_fmt(sd, state, format);
> >>>>
> >>>>      return mali_c55_rzr_set_sink_fmt(sd, state, format);
> >>>>
> >>>> to match SOURCE_PAD and source_fmt.
> >>>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_get_selection(struct v4l2_subdev *sd,
> >>>>>> +                      struct v4l2_subdev_state *state,
> >>>>>> +                      struct v4l2_subdev_selection *sel)
> >>>>>> +{
> >>>>>> +    if (sel->pad != MALI_C55_RZR_SINK_PAD)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    if (sel->target != V4L2_SEL_TGT_CROP &&
> >>>>>> +        sel->target != V4L2_SEL_TGT_COMPOSE)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    sel->r = sel->target == V4L2_SEL_TGT_CROP
> >>>>>> +           ? *v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD)
> >>>>>> +           : *v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_set_selection(struct v4l2_subdev *sd,
> >>>>>> +                      struct v4l2_subdev_state *state,
> >>>>>> +                      struct v4l2_subdev_selection *sel)
> >>>>>> +{
> >>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>>>> +                            sd);
> >>>>>> +    struct v4l2_mbus_framefmt *source_fmt;
> >>>>>> +    struct v4l2_mbus_framefmt *sink_fmt;
> >>>>>> +    struct v4l2_rect *crop, *compose;
> >>>>>> +
> >>>>>> +    if (sel->pad != MALI_C55_RZR_SINK_PAD)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    if (sel->target != V4L2_SEL_TGT_CROP &&
> >>>>>> +        sel->target != V4L2_SEL_TGT_COMPOSE)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    source_fmt = v4l2_subdev_state_get_format(state,
> >>>>>> +                          MALI_C55_RZR_SOURCE_PAD);
> >>>>>> +    sink_fmt = v4l2_subdev_state_get_format(state, MALI_C55_RZR_SINK_PAD);
> >>>>>> +    crop = v4l2_subdev_state_get_crop(state, MALI_C55_RZR_SINK_PAD);
> >>>>>> +    compose = v4l2_subdev_state_get_compose(state, MALI_C55_RZR_SINK_PAD);
> >>>>>> +
> >>>>>> +    /* RAW bypass disables crop/scaling. */
> >>>>>> +    if (mali_c55_format_is_raw(source_fmt->code)) {
> >>>>>> +        crop->top = compose->top = 0;
> >>>>>> +        crop->left = compose->left = 0;
> >>>>>> +        crop->width = compose->width = sink_fmt->width;
> >>>>>> +        crop->height = compose->height = sink_fmt->height;
> >>>>>> +
> >>>>>> +        sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> >>>>>> +
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* During streaming, it is allowed to only change the crop rectangle. */
> >>>>>> +    if (rzr->streaming && sel->target != V4L2_SEL_TGT_CROP)
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +     /*
> >>>>>> +      * Update the desired target and then clamp the crop rectangle to the
> >>>>>> +      * sink format sizes and the compose size to the crop sizes.
> >>>>>> +      */
> >>>>>> +    if (sel->target == V4L2_SEL_TGT_CROP)
> >>>>>> +        *crop = sel->r;
> >>>>>> +    else
> >>>>>> +        *compose = sel->r;
> >>>>>> +
> >>>>>> +    clamp_t(unsigned int, crop->left, 0, sink_fmt->width);
> >>>>>> +    clamp_t(unsigned int, crop->top, 0, sink_fmt->height);
> >>>>>> +    clamp_t(unsigned int, crop->width, MALI_C55_MIN_WIDTH,
> >>>>>> +        sink_fmt->width - crop->left);
> >>>>>> +    clamp_t(unsigned int, crop->height, MALI_C55_MIN_HEIGHT,
> >>>>>> +        sink_fmt->height - crop->top);
> >>>>>> +
> >>>>>> +    if (rzr->streaming) {
> >>>>>> +        /*
> >>>>>> +         * Apply at runtime a crop rectangle on the resizer's sink only
> >>>>>> +         * if it doesn't require re-programming the scaler output sizes
> >>>>>> +         * as it would require changing the output buffer sizes as well.
> >>>>>> +         */
> >>>>>> +        if (sel->r.width < compose->width ||
> >>>>>> +            sel->r.height < compose->height)
> >>>>>> +            return -EINVAL;
> >>>>>> +
> >>>>>> +        *crop = sel->r;
> >>>>>> +        mali_c55_rzr_program(rzr, state);
> >>>>>> +
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    compose->left = 0;
> >>>>>> +    compose->top = 0;
> >>>>>> +    clamp_t(unsigned int, compose->left, 0, sink_fmt->width);
> >>>>>> +    clamp_t(unsigned int, compose->top, 0, sink_fmt->height);
> >>>>>> +    clamp_t(unsigned int, compose->width, crop->width / 8, crop->width);
> >>>>>> +    clamp_t(unsigned int, compose->height, crop->height / 8, crop->height);
> >>>>>> +
> >>>>>> +    sel->r = sel->target == V4L2_SEL_TGT_CROP ? *crop : *compose;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_set_routing(struct v4l2_subdev *sd,
> >>>>>> +                    struct v4l2_subdev_state *state,
> >>>>>> +                    enum v4l2_subdev_format_whence which,
> >>>>>> +                    struct v4l2_subdev_krouting *routing)
> >>>>>> +{
> >>>>>> +    if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
> >>>>>> +        media_entity_is_streaming(&sd->entity))
> >>>>>> +        return -EBUSY;
> >>>>>> +
> >>>>>> +    return __mali_c55_rzr_set_routing(sd, state, routing);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_pad_ops mali_c55_resizer_pad_ops = {
> >>>>>> +    .enum_mbus_code        = mali_c55_rzr_enum_mbus_code,
> >>>>>> +    .enum_frame_size    = mali_c55_rzr_enum_frame_size,
> >>>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
> >>>>>> +    .set_fmt        = mali_c55_rzr_set_fmt,
> >>>>>> +    .get_selection        = mali_c55_rzr_get_selection,
> >>>>>> +    .set_selection        = mali_c55_rzr_set_selection,
> >>>>>> +    .set_routing        = mali_c55_rzr_set_routing,
> >>>>>> +};
> >>>>>> +
> >>>>>> +void mali_c55_rzr_start_stream(struct mali_c55_resizer *rzr)
> >>>> 
> >>>> Could this be handled through the .enable_streams() and
> >>>> .disable_streams() operations ? They ensure that the stream state stored
> >>>> internal is correct. That may not matter much today, but I think it will
> >>>> become increasingly important in the future for the V4L2 core.
> >>>>
> >>>>>> +{
> >>>>>> +    struct mali_c55 *mali_c55 = rzr->mali_c55;
> >>>>>> +    struct v4l2_subdev *sd = &rzr->sd;
> >>>>>> +    struct v4l2_subdev_state *state;
> >>>>>> +    unsigned int sink_pad;
> >>>>>> +
> >>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>>>> +
> >>>>>> +    sink_pad = mali_c55_rzr_get_active_sink(state);
> >>>>>> +    if (sink_pad == MALI_C55_RZR_SINK_BYPASS_PAD) {
> >>>>>> +        /* Bypass FR pipe processing if the bypass route is active. */
> >>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >>>>>> + MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK,
> >>>>>> + MALI_C55_ISP_RAW_BYPASS_RAW_FR_BYPASS);
> >>>>>> +        goto unlock_state;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /* Disable bypass and use regular processing. */
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_ISP_RAW_BYPASS,
> >>>>>> +                 MALI_C55_ISP_RAW_BYPASS_FR_BYPASS_MASK, 0);
> >>>>>> +    mali_c55_rzr_program(rzr, state);
> >>>>>> +
> >>>>>> +unlock_state:
> >>>>>> +    rzr->streaming = true;
> >>>> 
> >>>> And hopefully you'll be able to replace this with
> >>>> v4l2_subdev_is_streaming(), introduced in "[PATCH v6 00/11] media:
> >>>> subdev: Improve stream enable/disable machinery" (Sakari has sent a pull
> >>>> request for v6.11 yesterday).
> >>>>
> >>>>>> +    v4l2_subdev_unlock_state(state);
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_rzr_stop_stream(struct mali_c55_resizer *rzr)
> >>>>>> +{
> >>>>>> +    struct v4l2_subdev *sd = &rzr->sd;
> >>>>>> +    struct v4l2_subdev_state *state;
> >>>>>> +
> >>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>>>> +    rzr->streaming = false;
> >>>>>> +    v4l2_subdev_unlock_state(state);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_ops mali_c55_resizer_ops = {
> >>>>>> +    .pad    = &mali_c55_resizer_pad_ops,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_rzr_init_state(struct v4l2_subdev *sd,
> >>>>>> +                   struct v4l2_subdev_state *state)
> >>>>>> +{
> >>>>>> +    struct mali_c55_resizer *rzr = container_of(sd, struct mali_c55_resizer,
> >>>>>> +                            sd);
> >>>>>> +    struct v4l2_subdev_krouting routing = { };
> >>>>>> +    struct v4l2_subdev_route *routes;
> >>>>>> +    unsigned int i;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    routes = kcalloc(rzr->num_routes, sizeof(*routes), GFP_KERNEL);
> >>>>>> +    if (!routes)
> >>>>>> +        return -ENOMEM;
> >>>>>> +
> >>>>>> +    for (i = 0; i < rzr->num_routes; ++i) {
> >>>>>> +        struct v4l2_subdev_route *route = &routes[i];
> >>>>>> +
> >>>>>> +        route->sink_pad = i
> >>>>>> +                ? MALI_C55_RZR_SINK_BYPASS_PAD
> >>>>>> +                : MALI_C55_RZR_SINK_PAD;
> >>>>>> +        route->source_pad = MALI_C55_RZR_SOURCE_PAD;
> >>>>>> +        if (route->sink_pad != MALI_C55_RZR_SINK_BYPASS_PAD)
> >>>>>> +            route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    routing.num_routes = rzr->num_routes;
> >>>>>> +    routing.routes = routes;
> >>>>>> +
> >>>>>> +    ret = __mali_c55_rzr_set_routing(sd, state, &routing);
> >>>>>> +    kfree(routes);
> >>>>>> +
> >>>>>> +    return ret;
> >>>> 
> >>>> I think this could be simplified.
> >>>>
> >>>>      struct v4l2_subdev_route routes[2] = {
> >>>>          {
> >>>>              .sink_pad = MALI_C55_RZR_SINK_PAD,
> >>>>              .source_pad = MALI_C55_RZR_SOURCE_PAD,
> >>>>              .flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> >>>>          }, {
> >>>>              .sink_pad = MALI_C55_RZR_SINK_BYPASS_PAD,
> >>>>              .source_pad = MALI_C55_RZR_SOURCE_PAD,
> >>>>          },
> >>>>      };
> >>>>      struct v4l2_subdev_krouting routing = {
> >>>>          .num_routes = rzr->num_routes,
> >>>>          .routes = routes,
> >>>>      };
> >>>>
> >>>>      return __mali_c55_rzr_set_routing(sd, state, &routing);
> >>>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_internal_ops mali_c55_resizer_internal_ops = {
> >>>>>> +    .init_state = mali_c55_rzr_init_state,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static void mali_c55_resizer_program_coefficients(struct mali_c55 *mali_c55,
> >>>>>> +                          unsigned int index)
> >>>>>> +{
> >>>>>> +    const unsigned int scaler_filt_coefmem_addrs[][2] = {
> >>>>>> +        [MALI_C55_RZR_FR] = {
> >>>>>> +            0x034A8, /* hfilt */
> >>>>>> +            0x044A8  /* vfilt */
> >>>>> 
> >>>>> Lowercase hex constants.
> >>>> 
> >>>> And addresses belong to the mali-c55-registers.h file.
> >>>>
> >>>>>> +        },
> >>>>>> +        [MALI_C55_RZR_DS] = {
> >>>>>> +            0x014A8, /* hfilt */
> >>>>>> +            0x024A8  /* vfilt */
> >>>>>> +        },
> >>>>>> +    };
> >>>>>> +    unsigned int haddr = scaler_filt_coefmem_addrs[index][0];
> >>>>>> +    unsigned int vaddr = scaler_filt_coefmem_addrs[index][1];
> >>>>>> +    unsigned int i, j;
> >>>>>> +
> >>>>>> +    for (i = 0; i < MALI_C55_RESIZER_COEFS_NUM_BANKS; i++) {
> >>>>>> +        for (j = 0; j < MALI_C55_RESIZER_COEFS_NUM_ENTRIES; j++) {
> >>>>>> +            mali_c55_write(mali_c55, haddr,
> >>>>>> + mali_c55_scaler_h_filter_coefficients[i][j]);
> >>>>>> +            mali_c55_write(mali_c55, vaddr,
> >>>>>> + mali_c55_scaler_v_filter_coefficients[i][j]);
> >>>>>> +
> >>>>>> +            haddr += sizeof(u32);
> >>>>>> +            vaddr += sizeof(u32);
> >>>>>> +        }
> >>>>>> +    }
> >>>> 
> >>>> How about memcpy_toio() ? I suppose this function isn't
> >>>> performance sensitive, so maybe usage of mali_c55_write() is better from
> >>>> a consistency point of view.
> >>>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +int mali_c55_register_resizers(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    unsigned int i;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; ++i) {
> >>>> 
> >>>> Moving the inner content to a separate mali_c55_register_resizer()
> >>>> function would increase readability I think, and remove usage of gotos.
> >>>> I would probably do the same for unregistration too, for consistency.
> >>>>
> >>>>>> +        struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> >>>>>> +        struct v4l2_subdev *sd = &rzr->sd;
> >>>>>> +        unsigned int num_pads = MALI_C55_RZR_NUM_PADS;
> >>>>>> +
> >>>>>> +        rzr->id = i;
> >>>>>> +        rzr->streaming = false;
> >>>>>> +
> >>>>>> +        if (rzr->id == MALI_C55_RZR_FR)
> >>>>>> +            rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_FR];
> >>>>>> +        else
> >>>>>> +            rzr->cap_dev = &mali_c55->cap_devs[MALI_C55_CAP_DEV_DS];
> >>>>>> +
> >>>>>> +        mali_c55_resizer_program_coefficients(mali_c55, i);
> >>>> 
> >>>> Should this be done at stream start, given that power may be cut off
> >>>> between streaming sessions ?
> >>>>
> >>>>>> +
> >>>>>> +        v4l2_subdev_init(sd, &mali_c55_resizer_ops);
> >>>>>> +        sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE
> >>>>>> +                 | V4L2_SUBDEV_FL_STREAMS;
> >>>>>> +        sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
> >>>>>> +        sd->internal_ops = &mali_c55_resizer_internal_ops;
> >>>>>> +        snprintf(sd->name, ARRAY_SIZE(sd->name), "%s %s",
> >>>> 
> >>>>          snprintf(sd->name, ARRAY_SIZE(sd->name), "%s resizer %s",
> >>>>
> >>>> and drop the "resizer " prefix from mali_c55_resizer_names. You can also
> >>>> make mali_c55_resizer_names a local static const variable.
> >>>>
> >>>>>> +             MALI_C55_DRIVER_NAME, mali_c55_resizer_names[i]);
> >>>>>> +
> >>>>>> +        rzr->pads[MALI_C55_RZR_SINK_PAD].flags = MEDIA_PAD_FL_SINK;
> >>>>>> +        rzr->pads[MALI_C55_RZR_SOURCE_PAD].flags = MEDIA_PAD_FL_SOURCE;
> >>>>>> +
> >>>>>> +        /* Only the FR pipe has a bypass pad. */
> >>>>>> +        if (rzr->id == MALI_C55_RZR_FR) {
> >>>>>> + rzr->pads[MALI_C55_RZR_SINK_BYPASS_PAD].flags =
> >>>>>> +                            MEDIA_PAD_FL_SINK;
> >>>>>> +            rzr->num_routes = 2;
> >>>>>> +        } else {
> >>>>>> +            num_pads -= 1;
> >>>>>> +            rzr->num_routes = 1;
> >>>>>> +        }
> >>>>>> +
> >>>>>> +        ret = media_entity_pads_init(&sd->entity, num_pads, rzr->pads);
> >>>>>> +        if (ret)
> >>>>>> +            return ret;
> >>>>>> +
> >>>>>> +        ret = v4l2_subdev_init_finalize(sd);
> >>>>>> +        if (ret)
> >>>>>> +            goto err_cleanup;
> >>>>>> +
> >>>>>> +        ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >>>>>> +        if (ret)
> >>>>>> +            goto err_cleanup;
> >>>>>> +
> >>>>>> +        rzr->mali_c55 = mali_c55;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_cleanup:
> >>>>>> +    for (; i >= 0; --i) {
> >>>>>> +        struct mali_c55_resizer *rzr = &mali_c55->resizers[i];
> >>>>>> +        struct v4l2_subdev *sd = &rzr->sd;
> >>>>>> +
> >>>>>> +        v4l2_subdev_cleanup(sd);
> >>>>>> +        media_entity_cleanup(&sd->entity);
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_unregister_resizers(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < MALI_C55_NUM_RZRS; i++) {
> >>>>>> +        struct mali_c55_resizer *resizer = &mali_c55->resizers[i];
> >>>>>> +
> >>>>>> +        if (!resizer->mali_c55)
> >>>>>> +            continue;
> >>>>>> +
> >>>>>> +        v4l2_device_unregister_subdev(&resizer->sd);
> >>>>>> +        v4l2_subdev_cleanup(&resizer->sd);
> >>>>>> +        media_entity_cleanup(&resizer->sd.entity);
> >>>>>> +    }
> >>>>>> +}
> >>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >>>>>> new file mode 100644
> >>>>>> index 000000000000..c7e699741c6d
> >>>>>> --- /dev/null
> >>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-tpg.c
> >>>>>> @@ -0,0 +1,402 @@
> >>>>>> +// SPDX-License-Identifier: GPL-2.0
> >>>>>> +/*
> >>>>>> + * ARM Mali-C55 ISP Driver - Test pattern generator
> >>>>>> + *
> >>>>>> + * Copyright (C) 2024 Ideas on Board Oy
> >>>>>> + */
> >>>>>> +
> >>>>>> +#include <linux/minmax.h>
> >>>>>> +#include <linux/string.h>
> >>>>>> +
> >>>>>> +#include <media/media-entity.h>
> >>>>>> +#include <media/v4l2-ctrls.h>
> >>>>>> +#include <media/v4l2-subdev.h>
> >>>>>> +
> >>>>>> +#include "mali-c55-common.h"
> >>>>>> +#include "mali-c55-registers.h"
> >>>>>> +
> >>>>>> +#define MALI_C55_TPG_SRC_PAD        0
> >>>>>> +#define MALI_C55_TPG_FIXED_HBLANK    0x20
> >>>>>> +#define MALI_C55_TPG_MAX_VBLANK        0xFFFF
> >>>>> 
> >>>>> Lowercase hex constants.
> >>>>>
> >>>>>> +#define MALI_C55_TPG_PIXEL_RATE        100000000
> >>>>> 
> >>>>> This should be exposed to applications using the V4L2_CID_PIXEL_RATE
> >>>>> control (read-only).
> >>>>>
> >>>>>> +
> >>>>>> +static const char * const mali_c55_tpg_test_pattern_menu[] = {
> >>>>>> +    "Flat field",
> >>>>>> +    "Horizontal gradient",
> >>>>>> +    "Vertical gradient",
> >>>>>> +    "Vertical bars",
> >>>>>> +    "Arbitrary rectangle",
> >>>>>> +    "White frame on black field"
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const u32 mali_c55_tpg_mbus_codes[] = {
> >>>>>> +    MEDIA_BUS_FMT_SRGGB20_1X20,
> >>>>>> +    MEDIA_BUS_FMT_RGB202020_1X60,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static void __mali_c55_tpg_calc_vblank(struct v4l2_mbus_framefmt *format,
> >>>>>> +                       int *def_vblank, int *min_vblank)
> >>>>> 
> >>>>> unsigned int ?
> >>>>>
> >>>>>> +{
> >>>>>> +    unsigned int hts;
> >>>>>> +    int tgt_fps;
> >>>>>> +    int vblank;
> >>>>>> +
> >>>>>> +    hts = format->width + MALI_C55_TPG_FIXED_HBLANK;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * The ISP has minimum vertical blanking requirements that must be
> >>>>>> +     * adhered to by the TPG. The minimum is a function of the Iridix blocks
> >>>>>> +     * clocking requirements and the width of the image and horizontal
> >>>>>> +     * blanking, but if we assume the worst case iVariance and sVariance
> >>>>>> +     * values then it boils down to the below.
> >>>>>> +     */
> >>>>>> +    *min_vblank = 15 + (120500 / hts);
> >>>>> 
> >>>>> I wonder if this should round up.
> >>>>>
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * We need to set a sensible default vblank for whatever format height
> >>>>>> +     * we happen to be given from set_fmt(). This function just targets
> >>>>>> +     * an even multiple of 15fps. If we can't get 15fps, let's target 5fps.
> >>>>>> +     * If we can't get 5fps we'll take whatever the minimum vblank gives us.
> >>>>>> +     */
> >>>>>> +    tgt_fps = MALI_C55_TPG_PIXEL_RATE / hts / (format->height + *min_vblank);
> >>>>>> +
> >>>>>> +    if (tgt_fps < 5)
> >>>>>> +        vblank = *min_vblank;
> >>>>>> +    else
> >>>>>> +        vblank = MALI_C55_TPG_PIXEL_RATE / hts
> >>>>>> +               / max(rounddown(tgt_fps, 15), 5);
> >>>>>> +
> >>>>>> +    *def_vblank = ALIGN_DOWN(vblank, 2) - format->height;
> >>>>> 
> >>>>> "vblank = vblank - height" doesn't seem right. The "else" branch stores
> >>>>> a vts in vblank, which doesn't seem right either. Maybe you meant
> >>>>> something like
> >>>>>
> >>>>>      if (tgt_fps < 5)
> >>>>>          def_vts = *min_vblank + format->height;
> >>>>>      else
> >>>>>          def_vts = MALI_C55_TPG_PIXEL_RATE / hts
> >>>>>              / max(rounddown(tgt_fps, 15), 5);
> >>>>>
> >>>>>      *def_vblank = ALIGN_DOWN(def_vts, 2) - format->height;
> >>>>>
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_tpg_s_ctrl(struct v4l2_ctrl *ctrl)
> >>>>>> +{
> >>>>>> +    struct mali_c55_tpg *tpg = container_of(ctrl->handler,
> >>>>>> +                        struct mali_c55_tpg,
> >>>>>> +                        ctrls.handler);
> >>>>>> +    struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
> >>>>>> +
> >>>>> 
> >>>>> Should you return here if the pipeline isn't streaming ?
> >>>>>
> >>>>>> +    switch (ctrl->id) {
> >>>>>> +    case V4L2_CID_TEST_PATTERN:
> >>>>>> +        mali_c55_write(mali_c55, MALI_C55_REG_TEST_GEN_CH0_PATTERN_TYPE,
> >>>>>> +                   ctrl->val);
> >>>>>> +        break;
> >>>>>> +    case V4L2_CID_VBLANK:
> >>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
> >>>>>> +                     MALI_C55_REG_VBLANK_MASK, ctrl->val);
> >>>>>> +        break;
> >>>>>> +    default:
> >>>>>> +        dev_err(mali_c55->dev, "invalid V4L2 control ID\n");
> >>>>>> +        return -EINVAL;
> >>>>> 
> >>>>> Can this happen ?
> >>>>>
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_ctrl_ops mali_c55_tpg_ctrl_ops = {
> >>>>>> +    .s_ctrl = &mali_c55_tpg_s_ctrl,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static void mali_c55_tpg_configure(struct mali_c55 *mali_c55,
> >>>>>> +                   struct v4l2_subdev *sd)
> >>>>>> +{
> >>>>>> +    struct v4l2_subdev_state *state;
> >>>>>> +    struct v4l2_mbus_framefmt *fmt;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * hblank needs setting, but is a read-only control and thus won't be
> >>>>>> +     * called during __v4l2_ctrl_handler_setup(). Do it here instead.
> >>>>>> +     */
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_BLANKING,
> >>>>>> +                 MALI_C55_REG_HBLANK_MASK,
> >>>>>> +                 MALI_C55_TPG_FIXED_HBLANK);
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >>>>>> +                 MALI_C55_REG_GEN_VIDEO_MULTI_MASK, 0x01);
> >>>>>> +
> >>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>>>> +    fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>>>> +
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >>>>>> +                 MALI_C55_TEST_PATTERN_RGB_MASK,
> >>>>>> +                 fmt->code == MEDIA_BUS_FMT_RGB202020_1X60 ?
> >>>>>> +                      0x01 : 0x0);
> >>>>>> +
> >>>>>> +    v4l2_subdev_unlock_state(state);
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_tpg_s_stream(struct v4l2_subdev *sd, int enable)
> >>>>>> +{
> >>>>>> +    struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
> >>>>>> +    struct mali_c55 *mali_c55 = container_of(tpg, struct mali_c55, tpg);
> >>>>>> +
> >>>>>> +    if (!enable) {
> >>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >>>>>> +                MALI_C55_TEST_PATTERN_ON_OFF, 0x00);
> >>>>>> +        mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >>>>>> +                MALI_C55_REG_GEN_VIDEO_ON_MASK, 0x00);
> >>>>>> +        return 0;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * One might reasonably expect the framesize to be set here
> >>>>>> +     * given it's configurable in .set_fmt(), but it's done in the
> >>>>>> +     * ISP subdevice's stream on func instead, as the same register
> >>>>> 
> >>>>> s/func/function/
> >>>>>
> >>>>>> +     * is also used to indicate the size of the data coming from the
> >>>>>> +     * sensor.
> >>>>>> +     */
> >>>>>> +    mali_c55_tpg_configure(mali_c55, sd);
> >>>>> 
> >>>>>      mali_c55_tpg_configure(tpg);
> >>>>>
> >>>>>> + __v4l2_ctrl_handler_setup(sd->ctrl_handler);
> >>>>>> +
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_TPG_CH0,
> >>>>>> +                 MALI_C55_TEST_PATTERN_ON_OFF,
> >>>>>> +                 MALI_C55_TEST_PATTERN_ON_OFF);
> >>>>>> +    mali_c55_update_bits(mali_c55, MALI_C55_REG_GEN_VIDEO,
> >>>>>> +                 MALI_C55_REG_GEN_VIDEO_ON_MASK,
> >>>>>> +                 MALI_C55_REG_GEN_VIDEO_ON_MASK);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_video_ops mali_c55_tpg_video_ops = {
> >>>>>> +    .s_stream = &mali_c55_tpg_s_stream,
> >>>>> 
> >>>>> Can we use .enable_streams() and .disable_streams() ?
> >>>>>
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_tpg_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>> +                       struct v4l2_subdev_state *state,
> >>>>>> +                       struct v4l2_subdev_mbus_code_enum *code)
> >>>>>> +{
> >>>>>> +    if (code->index >= ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    code->code = mali_c55_tpg_mbus_codes[code->index];
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_tpg_enum_frame_size(struct v4l2_subdev *sd,
> >>>>>> +                    struct v4l2_subdev_state *state,
> >>>>>> +                    struct v4l2_subdev_frame_size_enum *fse)
> >>>>>> +{
> >>>>> 
> >>>>> You sohuld verify here that fse->code is a supported value and return
> >>>>> -EINVAL otherwise.
> >>>>>
> >>>>>> +    if (fse->index > 0 || fse->pad > sd->entity.num_pads)
> >>>>> 
> >>>>> Drop the pad check, it's done in the subdev core already.
> >>>>>
> >>>>>> +        return -EINVAL;
> >>>>>> +
> >>>>>> +    fse->min_width = MALI_C55_MIN_WIDTH;
> >>>>>> +    fse->max_width = MALI_C55_MAX_WIDTH;
> >>>>>> +    fse->min_height = MALI_C55_MIN_HEIGHT;
> >>>>>> +    fse->max_height = MALI_C55_MAX_HEIGHT;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int mali_c55_tpg_set_fmt(struct v4l2_subdev *sd,
> >>>>>> +                struct v4l2_subdev_state *state,
> >>>>>> +                struct v4l2_subdev_format *format)
> >>>>>> +{
> >>>>>> +    struct mali_c55_tpg *tpg = container_of(sd, struct mali_c55_tpg, sd);
> >>>>>> +    struct v4l2_mbus_framefmt *fmt = &format->format;
> >>>>>> +    int vblank_def, vblank_min;
> >>>>>> +    unsigned int i;
> >>>>>> +
> >>>>>> +    for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> >>>>>> +        if (fmt->code == mali_c55_tpg_mbus_codes[i])
> >>>>>> +            break;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>>>>> +        fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * The TPG says that the test frame timing generation logic expects a
> >>>>>> +     * minimum framesize of 4x4 pixels, but given the rest of the ISP can't
> >>>>>> +     * handle anything smaller than 128x128 it seems pointless to allow a
> >>>>>> +     * smaller frame.
> >>>>>> +     */
> >>>>>> +    clamp_t(unsigned int, fmt->width, MALI_C55_MIN_WIDTH,
> >>>>>> +        MALI_C55_MAX_WIDTH);
> >>>>>> +    clamp_t(unsigned int, fmt->height, MALI_C55_MIN_HEIGHT,
> >>>>>> +        MALI_C55_MAX_HEIGHT);
> >>>>>> +
> >>>>>> +    *v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD) = *fmt;
> >>>>> 
> >>>>> You're allowing userspace to set fmt->field, as well as all the
> >>>>> colorspace parameters, to random values. I would instead do something
> >>>>> like
> >>>>>
> >>>>>      for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> >>>>>          if (format->format.code == mali_c55_tpg_mbus_codes[i])
> >>>>>              break;
> >>>>>      }
> >>>>>
> >>>>>      if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>>>>          format->format.code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>>>
> >>>>>      format->format.width = clamp(format->format.width,
> >>>>>                       MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>>>>      format->format.height = clamp(format->format.height,
> >>>>>                        MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>>>>
> >>>>>      fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>>>      fmt->code = format->format.code;
> >>>>>      fmt->width = format->format.width;
> >>>>>      fmt->height = format->format.height;
> >>>>>
> >>>>>      format->format = *fmt;
> >>>>>
> >>>>> Alternatively (which I think I like better),
> >>>>>
> >>>>>      fmt = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>>>
> >>>>>      fmt->code = format->format.code;
> >>>>>
> >>>>>      for (i = 0; i < ARRAY_SIZE(mali_c55_tpg_mbus_codes); i++) {
> >>>>>          if (fmt->code == mali_c55_tpg_mbus_codes[i])
> >>>>>              break;
> >>>>>      }
> >>>>>
> >>>>>      if (i == ARRAY_SIZE(mali_c55_tpg_mbus_codes))
> >>>>>          fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>>>
> >>>>>      fmt->width = clamp(format->format.width,
> >>>>>                 MALI_C55_MIN_WIDTH, MALI_C55_MAX_WIDTH);
> >>>>>      fmt->height = clamp(format->format.height,
> >>>>>                  MALI_C55_MIN_HEIGHT, MALI_C55_MAX_HEIGHT);
> >>>>>
> >>>>>      format->format = *fmt;
> >>>>>
> >>>>>> +
> >>>>>> +    if (format->which == V4L2_SUBDEV_FORMAT_TRY)
> >>>>>> +        return 0;
> >>>>>> +
> >>>>>> +    __mali_c55_tpg_calc_vblank(fmt, &vblank_def, &vblank_min);
> >>>>>> +    __v4l2_ctrl_modify_range(tpg->ctrls.vblank, vblank_min,
> >>>>>> +                 MALI_C55_TPG_MAX_VBLANK, 1, vblank_def);
> >>>>>> +    __v4l2_ctrl_s_ctrl(tpg->ctrls.vblank, vblank_def);
> >>>>> 
> >>>>> Move those three calls to a separate function, it will be reused below.
> >>>>> I'd name is mali_c55_tpg_update_vblank(). You can fold
> >>>>> __mali_c55_tpg_calc_vblank() in it.
> >>>>>
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_pad_ops mali_c55_tpg_pad_ops = {
> >>>>>> +    .enum_mbus_code        = mali_c55_tpg_enum_mbus_code,
> >>>>>> +    .enum_frame_size    = mali_c55_tpg_enum_frame_size,
> >>>>>> +    .get_fmt        = v4l2_subdev_get_fmt,
> >>>>>> +    .set_fmt        = mali_c55_tpg_set_fmt,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_ops mali_c55_tpg_ops = {
> >>>>>> +    .video    = &mali_c55_tpg_video_ops,
> >>>>>> +    .pad    = &mali_c55_tpg_pad_ops,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_tpg_init_state(struct v4l2_subdev *sd,
> >>>>>> +                   struct v4l2_subdev_state *sd_state)
> >>>>> 
> >>>>> You name this variable state in every other subdev operation handler.
> >>>>>
> >>>>>> +{
> >>>>>> +    struct v4l2_mbus_framefmt *fmt =
> >>>>>> +        v4l2_subdev_state_get_format(sd_state, MALI_C55_TPG_SRC_PAD);
> >>>>>> +
> >>>>>> +    fmt->width = MALI_C55_DEFAULT_WIDTH;
> >>>>>> +    fmt->height = MALI_C55_DEFAULT_HEIGHT;
> >>>>>> +    fmt->field = V4L2_FIELD_NONE;
> >>>>>> +    fmt->code = MEDIA_BUS_FMT_SRGGB20_1X20;
> >>>>> 
> >>>>> Initialize the colorspace fields too.
> >>>>>
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static const struct v4l2_subdev_internal_ops mali_c55_tpg_internal_ops = {
> >>>>>> +    .init_state = mali_c55_tpg_init_state,
> >>>>>> +};
> >>>>>> +
> >>>>>> +static int mali_c55_tpg_init_controls(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_tpg_ctrls *ctrls = &mali_c55->tpg.ctrls;
> >>>>>> +    struct v4l2_subdev *sd = &mali_c55->tpg.sd;
> >>>>>> +    struct v4l2_mbus_framefmt *format;
> >>>>>> +    struct v4l2_subdev_state *state;
> >>>>>> +    int vblank_def, vblank_min;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    state = v4l2_subdev_lock_and_get_active_state(sd);
> >>>>>> +    format = v4l2_subdev_state_get_format(state, MALI_C55_TPG_SRC_PAD);
> >>>>>> +
> >>>>>> +    ret = v4l2_ctrl_handler_init(&ctrls->handler, 1);
> >>>>> 
> >>>>> You have 3 controls.
> >>>>>
> >>>>>> +    if (ret)
> >>>>>> +        goto err_unlock;
> >>>>>> +
> >>>>>> +    ctrls->test_pattern = v4l2_ctrl_new_std_menu_items(&ctrls->handler,
> >>>>>> +                &mali_c55_tpg_ctrl_ops, V4L2_CID_TEST_PATTERN,
> >>>>>> +                ARRAY_SIZE(mali_c55_tpg_test_pattern_menu) - 1,
> >>>>>> +                0, 3, mali_c55_tpg_test_pattern_menu);
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * We fix hblank at the minimum allowed value and control framerate
> >>>>>> +     * solely through the vblank control.
> >>>>>> +     */
> >>>>>> +    ctrls->hblank = v4l2_ctrl_new_std(&ctrls->handler,
> >>>>>> +                &mali_c55_tpg_ctrl_ops,
> >>>>>> +                V4L2_CID_HBLANK, MALI_C55_TPG_FIXED_HBLANK,
> >>>>>> +                MALI_C55_TPG_FIXED_HBLANK, 1,
> >>>>>> +                MALI_C55_TPG_FIXED_HBLANK);
> >>>>>> +    if (ctrls->hblank)
> >>>>>> +        ctrls->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
> >>>>>> +
> >>>>>> +    __mali_c55_tpg_calc_vblank(format, &vblank_def, &vblank_min);
> >>>>> 
> >>>>> Drop this and initialize the control with default values. You can then
> >>>>> update the value by calling mali_c55_tpg_update_vblank() in
> >>>>> mali_c55_register_tpg().
> >>>>>
> >>>>> The reason is to share the same mutex between the control handler and
> >>>>> the subdev active state without having to add a separate mutex in the
> >>>>> mali_c55_tpg structure. The simplest way to do so is to initialize the
> >>>>> controls first, set sd->state_lock to point to the control handler lock,
> >>>>> and call v4l2_subdev_init_finalize() as the last step. As a consequence,
> >>>>> you can't access the active state when initializing controls.
> >>>>>
> >>>>> You can alternatively keep the lock in mali_c55_tpg and set
> >>>>> sd->state_lock to point to it, but I think that's more complex.
> >>>>>
> >>>>>> +    ctrls->vblank = v4l2_ctrl_new_std(&ctrls->handler,
> >>>>>> +                      &mali_c55_tpg_ctrl_ops,
> >>>>>> +                      V4L2_CID_VBLANK, vblank_min,
> >>>>>> +                      MALI_C55_TPG_MAX_VBLANK, 1,
> >>>>>> +                      vblank_def);
> >>>>>> +
> >>>>>> +    if (ctrls->handler.error) {
> >>>>>> +        dev_err(mali_c55->dev, "Error during v4l2 controls init\n");
> >>>>>> +        ret = ctrls->handler.error;
> >>>>>> +        goto err_free_handler;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    ctrls->handler.lock = &mali_c55->tpg.lock;
> >>>>> 
> >>>>> Drop this and drop the mutex. The control handler will use its internal
> >>>>> mutex.
> >>>>>
> >>>>>> +    mali_c55->tpg.sd.ctrl_handler = &ctrls->handler;
> >>>>>> +
> >>>>>> +    v4l2_subdev_unlock_state(state);
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_free_handler:
> >>>>>> +    v4l2_ctrl_handler_free(&ctrls->handler);
> >>>>>> +err_unlock:
> >>>>>> +    v4l2_subdev_unlock_state(state);
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +int mali_c55_register_tpg(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_tpg *tpg = &mali_c55->tpg;
> >>>>>> +    struct v4l2_subdev *sd = &tpg->sd;
> >>>>>> +    struct media_pad *pad = &tpg->pad;
> >>>>>> +    int ret;
> >>>>>> +
> >>>>>> +    mutex_init(&tpg->lock);
> >>>>>> +
> >>>>>> +    v4l2_subdev_init(sd, &mali_c55_tpg_ops);
> >>>>>> +    sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> >>>>>> +    sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
> >>>>> 
> >>>>> Should we introduce a TPG function ?
> >>>>>
> >>>>>> +    sd->internal_ops = &mali_c55_tpg_internal_ops;
> >>>>>> +    strscpy(sd->name, MALI_C55_DRIVER_NAME " tpg", sizeof(sd->name));
> >>>>>> +
> >>>>>> +    pad->flags = MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUST_CONNECT;
> >>>>> 
> >>>>> I don't think MEDIA_PAD_FL_MUST_CONNECT is right.
> >>>>>
> >>>>>> +    ret = media_entity_pads_init(&sd->entity, 1, pad);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev,
> >>>>>> +            "Failed to initialize media entity pads\n");
> >>>>>> +        goto err_destroy_mutex;
> >>>>>> +    }
> >>>>>> +
> >>>>> 
> >>>>>      sd->state_lock = sd->ctrl_handler->lock;
> >>>>>
> >>>>> to use the same lock for the controls and the active state. You need to
> >>>>> move this line and the v4l2_subdev_init_finalize() call after
> >>>>> mali_c55_tpg_init_controls() to get the control handler lock initialized
> >>>>> first.
> >>>>>
> >>>>>> +    ret = v4l2_subdev_init_finalize(sd);
> >>>>>> +    if (ret)
> >>>>>> +        goto err_cleanup_media_entity;
> >>>>>> +
> >>>>>> +    ret = mali_c55_tpg_init_controls(mali_c55);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev,
> >>>>>> +            "Error initialising controls\n");
> >>>>>> +        goto err_cleanup_subdev;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    ret = v4l2_device_register_subdev(&mali_c55->v4l2_dev, sd);
> >>>>>> +    if (ret) {
> >>>>>> +        dev_err(mali_c55->dev, "Failed to register tpg subdev\n");
> >>>>>> +        goto err_free_ctrl_handler;
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    /*
> >>>>>> +     * By default the colour settings lead to a very dim image that is
> >>>>>> +     * nearly indistinguishable from black on some monitor settings. Ramp
> >>>>>> +     * them up a bit so the image is brighter.
> >>>>>> +     */
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_R_BACKGROUND,
> >>>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_G_BACKGROUND,
> >>>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
> >>>>>> +    mali_c55_write(mali_c55, MALI_C55_REG_TPG_B_BACKGROUND,
> >>>>>> +               MALI_C55_TPG_BACKGROUND_MAX);
> >>>>>> +
> >>>>>> +    tpg->mali_c55 = mali_c55;
> >>>>>> +
> >>>>>> +    return 0;
> >>>>>> +
> >>>>>> +err_free_ctrl_handler:
> >>>>>> +    v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> >>>>>> +err_cleanup_subdev:
> >>>>>> +    v4l2_subdev_cleanup(sd);
> >>>>>> +err_cleanup_media_entity:
> >>>>>> +    media_entity_cleanup(&sd->entity);
> >>>>>> +err_destroy_mutex:
> >>>>>> +    mutex_destroy(&tpg->lock);
> >>>>>> +
> >>>>>> +    return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> +void mali_c55_unregister_tpg(struct mali_c55 *mali_c55)
> >>>>>> +{
> >>>>>> +    struct mali_c55_tpg *tpg = &mali_c55->tpg;
> >>>>>> +
> >>>>>> +    if (!tpg->mali_c55)
> >>>>>> +        return;
> >>>>>> +
> >>>>>> +    v4l2_device_unregister_subdev(&tpg->sd);
> >>>>>> +    v4l2_subdev_cleanup(&tpg->sd);
> >>>>>> +    media_entity_cleanup(&tpg->sd.entity);
> >>>>>> +    v4l2_ctrl_handler_free(&tpg->ctrls.handler);
> >>>>> 
> >>>>> Free the control handler just after v4l2_device_unregister_subdev() to
> >>>>> match the order in mali_c55_register_tpg().
> >>>>>
> >>>>>> +    mutex_destroy(&tpg->lock);
> >>>>>> +}

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 10/16] media: platform: Add mali-c55 3a stats devnode

[Dan Scally]

Hi Laurent

On 29/06/2024 16:04, Laurent Pinchart wrote:
> Hi Dan,
>
> On Thu, Jun 20, 2024 at 04:10:01PM +0100, Daniel Scally wrote:
>> On 16/06/2024 22:19, Laurent Pinchart wrote:
>>> On Wed, May 29, 2024 at 04:28:52PM +0100, Daniel Scally wrote:
>>>> Add a new code file to govern the 3a statistics capture node.
>>>>
>>>> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
>>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>>>> ---
>>>> Changes in v5:
>>>>
>>>> 	- New patch
>>>>
>>>>    drivers/media/platform/arm/mali-c55/Makefile  |   3 +-
>>>>    .../platform/arm/mali-c55/mali-c55-common.h   |  28 ++
>>>>    .../platform/arm/mali-c55/mali-c55-core.c     |  15 +
>>>>    .../platform/arm/mali-c55/mali-c55-isp.c      |   1 +
>>>>    .../arm/mali-c55/mali-c55-registers.h         |   3 +
>>>>    .../platform/arm/mali-c55/mali-c55-stats.c    | 350 ++++++++++++++++++
>>>>    6 files changed, 399 insertions(+), 1 deletion(-)
>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-stats.c
>>>>
>>>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile b/drivers/media/platform/arm/mali-c55/Makefile
>>>> index 77dcb2fbf0f4..cd5a64bf0c62 100644
>>>> --- a/drivers/media/platform/arm/mali-c55/Makefile
>>>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
>>>> @@ -4,6 +4,7 @@ mali-c55-y := mali-c55-capture.o \
>>>>    	      mali-c55-core.o \
>>>>    	      mali-c55-isp.o \
>>>>    	      mali-c55-tpg.o \
>>>> -	      mali-c55-resizer.o
>>>> +	      mali-c55-resizer.o \
>>>> +	      mali-c55-stats.o
>>>>    
>>>>    obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>> index 2d0c4d152beb..44119e04009b 100644
>>>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>> @@ -79,6 +79,7 @@ enum mali_c55_isp_pads {
>>>>    	MALI_C55_ISP_PAD_SINK_VIDEO,
>>>>    	MALI_C55_ISP_PAD_SOURCE,
>>>>    	MALI_C55_ISP_PAD_SOURCE_BYPASS,
>>>> +	MALI_C55_ISP_PAD_SOURCE_3A,
>>> Functions and structures are named with a "stats" suffix, let's call
>>> this MALI_C55_ISP_PAD_SOURCE_STATS.
>>>
>>>>    	MALI_C55_ISP_NUM_PADS,
>>>>    };
>>>>    
>>>> @@ -194,6 +195,28 @@ struct mali_c55_cap_dev {
>>>>    	bool streaming;
>>>>    };
>>>>    
>>>> +struct mali_c55_stats_buf {
>>>> +	struct vb2_v4l2_buffer vb;
>>>> +	spinlock_t lock;
>>> All locks require a comment to document what they protect. Same below.
>>>
>>>> +	unsigned int segments_remaining;
>>>> +	struct list_head queue;
>>>> +	bool failed;
>>>> +};
>>>> +
>>>> +struct mali_c55_stats {
>>>> +	struct mali_c55 *mali_c55;
>>>> +	struct video_device vdev;
>>>> +	struct dma_chan *channel;
>>>> +	struct vb2_queue queue;
>>>> +	struct media_pad pad;
>>>> +	struct mutex lock;
>>>> +
>>>> +	struct {
>>>> +		spinlock_t lock;
>>>> +		struct list_head queue;
>>>> +	} buffers;
>>>> +};
>>>> +
>>>>    enum mali_c55_config_spaces {
>>>>    	MALI_C55_CONFIG_PING,
>>>>    	MALI_C55_CONFIG_PONG,
>>>> @@ -224,6 +247,7 @@ struct mali_c55 {
>>>>    	struct mali_c55_isp isp;
>>>>    	struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
>>>>    	struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
>>>> +	struct mali_c55_stats stats;
>>>>    
>>>>    	struct list_head contexts;
>>>>    	enum mali_c55_config_spaces next_config;
>>>> @@ -245,6 +269,8 @@ int mali_c55_register_resizers(struct mali_c55 *mali_c55);
>>>>    void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
>>>>    int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
>>>>    void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
>>>> +int mali_c55_register_stats(struct mali_c55 *mali_c55);
>>>> +void mali_c55_unregister_stats(struct mali_c55 *mali_c55);
>>>>    struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
>>>>    void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>>>>    			     enum mali_c55_planes plane);
>>>> @@ -262,5 +288,7 @@ mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
>>>>    bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
>>>>    #define for_each_mali_isp_fmt(fmt)\
>>>>    	for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
>>>> +void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
>>>> +				enum mali_c55_config_spaces cfg_space);
>>>>    
>>>>    #endif /* _MALI_C55_COMMON_H */
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>> index 50caf5ee7474..9ea70010876c 100644
>>>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>> @@ -337,6 +337,16 @@ static int mali_c55_create_links(struct mali_c55 *mali_c55)
>>>>    		}
>>>>    	}
>>>>    
>>>> +	ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>>> +			MALI_C55_ISP_PAD_SOURCE_3A,
>>>> +			&mali_c55->stats.vdev.entity, 0,
>>>> +			MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev,
>>>> +			"failed to link ISP and 3a stats node\n");
>>> s/3a stats/stats/
>>>
>>>> +		goto err_remove_links;
>>>> +	}
>>>> +
>>>>    	return 0;
>>>>    
>>>>    err_remove_links:
>>>> @@ -350,6 +360,7 @@ static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
>>>>    	mali_c55_unregister_isp(mali_c55);
>>>>    	mali_c55_unregister_resizers(mali_c55);
>>>>    	mali_c55_unregister_capture_devs(mali_c55);
>>>> +	mali_c55_unregister_stats(mali_c55);
>>>>    }
>>>>    
>>>>    static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>>>> @@ -372,6 +383,10 @@ static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>>>>    	if (ret)
>>>>    		goto err_unregister_entities;
>>>>    
>>>> +	ret = mali_c55_register_stats(mali_c55);
>>>> +	if (ret)
>>>> +		goto err_unregister_entities;
>>>> +
>>>>    	ret = mali_c55_create_links(mali_c55);
>>>>    	if (ret)
>>>>    		goto err_unregister_entities;
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>> index ea8b7b866e7a..94876fba3353 100644
>>>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>> @@ -564,6 +564,7 @@ int mali_c55_register_isp(struct mali_c55 *mali_c55)
>>>>    	isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
>>>>    	isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>>>>    	isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
>>>> +	isp->pads[MALI_C55_ISP_PAD_SOURCE_3A].flags = MEDIA_PAD_FL_SOURCE;
>>>>    
>>>>    	ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
>>>>    				     isp->pads);
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>> index cb27abde2aa5..eb3719245ec3 100644
>>>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>> @@ -68,6 +68,9 @@
>>>>    #define MALI_C55_VC_START(v)				((v) & 0xffff)
>>>>    #define MALI_C55_VC_SIZE(v)				(((v) & 0xffff) << 16)
>>>>    
>>>> +#define MALI_C55_REG_1024BIN_HIST			0x054a8
>>>> +#define MALI_C55_1024BIN_HIST_SIZE			4096
>>>> +
>>>>    /* Ping/Pong Configuration Space */
>>>>    #define MALI_C55_REG_BASE_ADDR				0x18e88
>>>>    #define MALI_C55_REG_BYPASS_0				0x18eac
>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-stats.c b/drivers/media/platform/arm/mali-c55/mali-c55-stats.c
>>>> new file mode 100644
>>>> index 000000000000..aa40480ed814
>>>> --- /dev/null
>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-stats.c
>>>> @@ -0,0 +1,350 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * ARM Mali-C55 ISP Driver - 3A Statistics capture device
>>>> + *
>>>> + * Copyright (C) 2023 Ideas on Board Oy
>>>> + */
>>>> +
>>>> +#include <linux/dmaengine.h>
>>>> +#include <linux/media/arm/mali-c55-config.h>
>>>> +#include <linux/spinlock.h>
>>> You're missing some headers here, for
>>>
>>> container_of()
>>> dev_err()
>>> list_*()
>>> mutex_init()
>>> strscpy()
>>> strscpy()
>>>
>>>> +
>>>> +#include <media/media-entity.h>
>>>> +#include <media/v4l2-dev.h>
>>>> +#include <media/v4l2-event.h>
>>>> +#include <media/v4l2-fh.h>
>>>> +#include <media/v4l2-ioctl.h>
>>>> +#include <media/videobuf2-core.h>
>>>> +#include <media/videobuf2-dma-contig.h>
>>>> +
>>>> +#include "mali-c55-common.h"
>>>> +#include "mali-c55-registers.h"
>>>> +
>>>> +static unsigned int metering_space_addrs[] = {
>>> const
>>>
>>>> +	[MALI_C55_CONFIG_PING] = 0x095AC,
>>>> +	[MALI_C55_CONFIG_PONG] = 0x2156C,
>>> Lower-case hex constants.
>>>
>>>> +};
>>>> +
>>>> +static int mali_c55_stats_enum_fmt_meta_cap(struct file *file, void *fh,
>>>> +					    struct v4l2_fmtdesc *f)
>>>> +{
>>>> +	if (f->index || f->type != V4L2_BUF_TYPE_META_CAPTURE)
>>>> +		return -EINVAL;
>>>> +
>>>> +	f->pixelformat = V4L2_META_FMT_MALI_C55_3A_STATS;
>>> The format could be called V4L2_META_FMT_MALI_C55_STATS. While most
>>> statistics are related to one of the 3A algorithms, I think it would be
>>> better to name this generically. It's name bikeshedding only of course.
>>>
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_stats_g_fmt_meta_cap(struct file *file, void *fh,
>>>> +					 struct v4l2_format *f)
>>>> +{
>>>> +	static const struct v4l2_meta_format mfmt = {
>>>> +		.dataformat = V4L2_META_FMT_MALI_C55_3A_STATS,
>>>> +		.buffersize = sizeof(struct mali_c55_stats_buffer)
>>>> +	};
>>>> +
>>>> +	if (f->type != V4L2_BUF_TYPE_META_CAPTURE)
>>>> +		return -EINVAL;
>>>> +
>>>> +	f->fmt.meta = mfmt;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int mali_c55_stats_querycap(struct file *file,
>>>> +				   void *priv, struct v4l2_capability *cap)
>>>> +{
>>>> +	strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
>>>> +	strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static const struct v4l2_ioctl_ops mali_c55_stats_v4l2_ioctl_ops = {
>>>> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
>>>> +	.vidioc_querybuf = vb2_ioctl_querybuf,
>>>> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
>>>> +	.vidioc_qbuf = vb2_ioctl_qbuf,
>>>> +	.vidioc_expbuf = vb2_ioctl_expbuf,
>>>> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
>>>> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
>>>> +	.vidioc_streamon = vb2_ioctl_streamon,
>>>> +	.vidioc_streamoff = vb2_ioctl_streamoff,
>>>> +	.vidioc_enum_fmt_meta_cap = mali_c55_stats_enum_fmt_meta_cap,
>>>> +	.vidioc_g_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
>>>> +	.vidioc_s_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
>>>> +	.vidioc_try_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
>>>> +	.vidioc_querycap = mali_c55_stats_querycap,
>>>> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
>>>> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
>>>> +};
>>>> +
>>>> +static const struct v4l2_file_operations mali_c55_stats_v4l2_fops = {
>>>> +	.owner = THIS_MODULE,
>>>> +	.unlocked_ioctl = video_ioctl2,
>>>> +	.open = v4l2_fh_open,
>>>> +	.release = vb2_fop_release,
>>>> +	.poll = vb2_fop_poll,
>>>> +	.mmap = vb2_fop_mmap,
>>>> +};
>>>> +
>>>> +static int
>>>> +mali_c55_stats_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
>>>> +			   unsigned int *num_planes, unsigned int sizes[],
>>>> +			   struct device *alloc_devs[])
>>>> +{
>>>> +	struct mali_c55_stats *stats = vb2_get_drv_priv(q);
>>>> +
>>>> +	if (*num_planes && *num_planes > 1)
>>>> +		return -EINVAL;
>>>> +
>>>> +	if (sizes[0] && sizes[0] != sizeof(struct mali_c55_stats_buffer))
>>>> +		return -EINVAL;
>>>> +
>>>> +	*num_planes = 1;
>>>> +	sizes[0] = sizeof(struct mali_c55_stats_buffer);
>>>> +
>>>> +	if (stats->channel)
>>>> +		alloc_devs[0] = stats->channel->device->dev;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static void mali_c55_stats_buf_queue(struct vb2_buffer *vb)
>>>> +{
>>>> +	struct mali_c55_stats *stats = vb2_get_drv_priv(vb->vb2_queue);
>>>> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>>>> +	struct mali_c55_stats_buf *buf = container_of(vbuf,
>>>> +						struct mali_c55_stats_buf, vb);
>>>> +
>>>> +	vb2_set_plane_payload(vb, 0, sizeof(struct mali_c55_stats_buffer));
>>>> +	buf->segments_remaining = 2;
>>>> +	buf->failed = false;
>>>> +
>>>> +	spin_lock(&stats->buffers.lock);
>>> Isn't the DMA completion handler run from IRQ context ? If so you'll
>>> need to use spin_lock_irq() here and in the other function that are
>>> not called with interrupts disabled.
>> They're run in the bottom half of the interrupt handler; I'm under the
>> impression that that means the interrupts aren't disabled, and it's
>> safe to do...is that mistaken?
> I'm talking about the DMA completion handler, called by the DMA engine
> subsystem, not the IRQs of the C55 itself.


Ah! I follow you now sorry. Hm, that's an interesting thought...I think for the PL330 the answer is 
no; the interrupt handler wakes a thread that runs the callbacks for any DMA transfers that are 
completed...I don't know that that's a design rule we can rely on though so possibly it's better to 
be safe. Is there a disadvantage to using spin_lock_irq() outside of IRQ context?

>
>>>> +	list_add_tail(&buf->queue, &stats->buffers.queue);
>>>> +	spin_unlock(&stats->buffers.lock);
>>>> +}
>>>> +
>>>> +static void mali_c55_stats_stop_streaming(struct vb2_queue *q)
>>>> +{
>>>> +	struct mali_c55_stats *stats = vb2_get_drv_priv(q);
>>>> +	struct mali_c55_stats_buf *buf, *tmp;
>>>> +
>>>> +	spin_lock(&stats->buffers.lock);
>>>> +
>>>> +	list_for_each_entry_safe(buf, tmp, &stats->buffers.queue, queue) {
>>>> +		list_del(&buf->queue);
>>>> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
>>>> +	}
>>>> +
>>>> +	spin_unlock(&stats->buffers.lock);
>>>> +}
>>>> +
>>>> +static const struct vb2_ops mali_c55_stats_vb2_ops = {
>>>> +	.queue_setup = mali_c55_stats_queue_setup,
>>>> +	.buf_queue = mali_c55_stats_buf_queue,
>>>> +	.wait_prepare = vb2_ops_wait_prepare,
>>>> +	.wait_finish = vb2_ops_wait_finish,
>>>> +	.stop_streaming = mali_c55_stats_stop_streaming,
>>>> +};
>>>> +
>>>> +static void
>>>> +mali_c55_stats_metering_complete(void *param,
>>>> +				 const struct dmaengine_result *result)
>>>> +{
>>>> +	struct mali_c55_stats_buf *buf = param;
>>>> +
>>>> +	spin_lock(&buf->lock);
>>> I wonder if this is needed. Can the DMA engine call the completion
>>> handlers of two sequential DMA transfers in parallel ?
>> The DMA engine that's on the system we have can't...I wasn't sure
>> whether that was generically true.
> I think it is, but please double-check.
>
>>>> +
>>>> +	if (buf->failed)
>>>> +		goto out_unlock;
>>>> +
>>>> +	buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>>>> +
>>>> +	if (result->result != DMA_TRANS_NOERROR) {
>>>> +		buf->failed = true;
>>>> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
>>> This will possibly return the buffer to userspace after the first DMA
>>> transfer. Userspace could then requeue the buffer to the kernel before
>>> the completion of the second DMA transfer. That will cause trouble. I
>>> think you should instead do something like
>>>
>>> 	spin_lock(&buf->lock);
>>>
>>> 	if (result->result != DMA_TRANS_NOERROR)
>>> 		buf->failed = true;
>>>
>>> 	if (!--buf->segments_remaining) {
>>> 		buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>>> 		vb2_buffer_done(&buf->vb.vb2_buf, buf->failed ?
>>> 				VB2_BUF_STATE_ERROR : VB2_BUF_STATE_DONE);
>>> 	}
>>>
>>> 	spin_unlock(&buf->lock);
>>>
>>> The
>>>
>>> 	buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>>>
>>> line could also be moved to mali_c55_stats_fill_buffer(), which would
>>> make sure the timestamp is filled in case of DMA submission failures.
>> Okedokey
>>
>>>> +		goto out_unlock;
>>>> +	}
>>>> +
>>>> +	if (!--buf->segments_remaining)
>>>> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
>>>> +
>>>> +out_unlock:
>>>> +	spin_unlock(&buf->lock);
>>>> +}
>>>> +
>>>> +static int mali_c55_stats_dma_xfer(struct mali_c55_stats *stats, dma_addr_t src,
>>>> +				   dma_addr_t dst,
>>>> +				   struct mali_c55_stats_buf *buf,
>>>> +				   size_t length,
>>>> +				   void (*callback)(void *, const struct dmaengine_result *result))
>>> The same callback is used for both invocations of this function, you can
>>> drop the parameter and hardcode it below.
>> Yeah, not even sure now why I had a parameter.
>>
>>>> +{
>>>> +	struct dma_async_tx_descriptor *tx;
>>>> +	dma_cookie_t cookie;
>>>> +
>>>> +	tx = dmaengine_prep_dma_memcpy(stats->channel, dst, src, length, 0);
>>>> +	if (!tx) {
>>>> +		dev_err(stats->mali_c55->dev, "failed to prep stats DMA\n");
>>>> +		return -EIO;
>>>> +	}
>>>> +
>>>> +	tx->callback_result = callback;
>>>> +	tx->callback_param = buf;
>>>> +
>>>> +	cookie = dmaengine_submit(tx);
>>>> +	if (dma_submit_error(cookie)) {
>>>> +		dev_err(stats->mali_c55->dev, "failed to submit stats DMA\n");
>>>> +		return -EIO;
>>>> +	}
>>>> +
>>>> +	dma_async_issue_pending(stats->channel);
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
>>>> +				enum mali_c55_config_spaces cfg_space)
>>>> +{
>>>> +	struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>> +	struct mali_c55_stats *stats = &mali_c55->stats;
>>>> +	struct mali_c55_stats_buf *buf = NULL;
>>>> +	dma_addr_t src, dst;
>>>> +	int ret;
>>>> +
>>>> +	spin_lock(&stats->buffers.lock);
>>>> +	if (!list_empty(&stats->buffers.queue)) {
>>>> +		buf = list_first_entry(&stats->buffers.queue,
>>>> +				       struct mali_c55_stats_buf, queue);
>>>> +		list_del(&buf->queue);
>>>> +	}
>>>> +	spin_unlock(&stats->buffers.lock);
>>>> +
>>>> +	if (!buf)
>>>> +		return;
>>>> +
>>>> +	buf->vb.sequence = mali_c55->isp.frame_sequence;
>>>> +
>>>> +	/*
>>>> +	 * There are infact two noncontiguous sections of the ISP's
>>> s/infact/in fact/
>>>
>>>> +	 * memory space that hold statistics for 3a algorithms to use. A
>>> s/use. A/use: a/
>>>
>>>> +	 * section in each config space and a global section holding
>>>> +	 * histograms which is double buffered and so holds data for the
>>>> +	 * last frame. We need to read both.
>>>> +	 */
>>>> +	src = ctx->base + MALI_C55_REG_1024BIN_HIST;
>>>> +	dst = vb2_dma_contig_plane_dma_addr(&buf->vb.vb2_buf, 0);
>>>> +
>>>> +	ret = mali_c55_stats_dma_xfer(stats, src, dst, buf,
>>>> +				      MALI_C55_1024BIN_HIST_SIZE,
>>>> +				      mali_c55_stats_metering_complete);
>>>> +	if (ret)
>>>> +		goto err_fail_buffer;
>>>> +
>>>> +	src = ctx->base + metering_space_addrs[cfg_space];
>>>> +	dst += MALI_C55_1024BIN_HIST_SIZE;
>>>> +
>>>> +	ret = mali_c55_stats_dma_xfer(
>>>> +		stats, src, dst, buf,
>>>> +		sizeof(struct mali_c55_stats_buffer) - MALI_C55_1024BIN_HIST_SIZE,
>>>> +		mali_c55_stats_metering_complete);
>>>> +	if (ret) {
>>>> +		dmaengine_terminate_sync(stats->channel);
>>>> +		goto err_fail_buffer;
>>>> +	}
>>> I think you will need to terminate DMA transfers at stream off time.
>>>
>>>> +
>>>> +	return;
>>>> +
>>>> +err_fail_buffer:
>>>> +	vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
>>>> +}
>>>> +
>>>> +void mali_c55_unregister_stats(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	struct mali_c55_stats *stats = &mali_c55->stats;
>>>> +
>>>> +	if (!video_is_registered(&stats->vdev))
>>>> +		return;
>>>> +
>>>> +	vb2_video_unregister_device(&stats->vdev);
>>>> +	media_entity_cleanup(&stats->vdev.entity);
>>>> +	dma_release_channel(stats->channel);
>>>> +	mutex_destroy(&stats->lock);
>>>> +}
>>>> +
>>>> +int mali_c55_register_stats(struct mali_c55 *mali_c55)
>>>> +{
>>>> +	struct mali_c55_stats *stats = &mali_c55->stats;
>>>> +	struct video_device *vdev = &stats->vdev;
>>>> +	struct vb2_queue *vb2q = &stats->queue;
>>>> +	dma_cap_mask_t mask;
>>>> +	int ret;
>>>> +
>>>> +	mutex_init(&stats->lock);
>>>> +	INIT_LIST_HEAD(&stats->buffers.queue);
>>>> +
>>>> +	dma_cap_zero(mask);
>>>> +	dma_cap_set(DMA_MEMCPY, mask);
>>>> +
>>>> +	stats->channel = dma_request_channel(mask, 0, NULL);
>>> Do we need a CPU fallback in case no DMA is available ?
>> Yes, actually.
>>
>>> I'm still very curious to know how long it takes to perform the DMA
>>> transfer, compared to copying the data with the CPU, and especially
>>> compared to the frame duration.
>> On my list of things to test and report :)
> Looking forward to it :-)
>
>>>> +	if (!stats->channel) {
>>>> +		ret = -ENODEV;
>>>> +		goto err_destroy_mutex;
>>>> +	}
>>>> +
>>>> +	stats->pad.flags = MEDIA_PAD_FL_SINK;
>>>> +	ret = media_entity_pads_init(&stats->vdev.entity, 1, &stats->pad);
>>>> +	if (ret)
>>>> +		goto err_release_dma_channel;
>>>> +
>>>> +	vb2q->type = V4L2_BUF_TYPE_META_CAPTURE;
>>>> +	vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
>>>> +	vb2q->drv_priv = stats;
>>>> +	vb2q->mem_ops = &vb2_dma_contig_memops;
>>>> +	vb2q->ops = &mali_c55_stats_vb2_ops;
>>>> +	vb2q->buf_struct_size = sizeof(struct mali_c55_stats_buf);
>>>> +	vb2q->min_queued_buffers = 1;
>>>> +	vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
>>>> +	vb2q->lock = &stats->lock;
>>>> +	vb2q->dev = mali_c55->dev;
>>> That's not the right device. The device that performs the DMA operation
>>> is the DMA engine, and that's what you need to pass to vb2. Otherwise
>>> the DMA address returned by vb2_dma_contig_plane_dma_addr() will be
>>> mapped to the ISP device, not the DMA engine. In practice, if neither
>>> are behind an IOMMU, things will likely work, but when that's not the
>>> case, run into problems.
>>>
>>>> +
>>>> +	ret = vb2_queue_init(vb2q);
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev, "stats vb2 queue init failed\n");
>>>> +		goto err_cleanup_entity;
>>>> +	}
>>>> +
>>>> +	strscpy(stats->vdev.name, "mali-c55 3a stats", sizeof(stats->vdev.name));
>>> s/3a //
>>>
>>>> +	vdev->release = video_device_release_empty;
>>> That's never right. You should refcount the data structures to ensure
>>> proper lifetime management.
>>>
>>>> +	vdev->fops = &mali_c55_stats_v4l2_fops;
>>>> +	vdev->ioctl_ops = &mali_c55_stats_v4l2_ioctl_ops;
>>>> +	vdev->lock = &stats->lock;
>>>> +	vdev->v4l2_dev = &mali_c55->v4l2_dev;
>>>> +	vdev->queue = &stats->queue;
>>>> +	vdev->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING;
>>>> +	vdev->vfl_dir = VFL_DIR_RX;
>>>> +	video_set_drvdata(vdev, stats);
>>>> +
>>>> +	ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
>>>> +	if (ret) {
>>>> +		dev_err(mali_c55->dev,
>>>> +			"failed to register stats video device\n");
>>>> +		goto err_release_vb2q;
>>>> +	}
>>>> +
>>>> +	stats->mali_c55 = mali_c55;
>>>> +
>>>> +	return 0;
>>>> +
>>>> +err_release_vb2q:
>>>> +	vb2_queue_release(vb2q);
>>>> +err_cleanup_entity:
>>>> +	media_entity_cleanup(&stats->vdev.entity);
>>>> +err_release_dma_channel:
>>>> +	dma_release_channel(stats->channel);
>>>> +err_destroy_mutex:
>>>> +	mutex_destroy(&stats->lock);
>>>> +
>>>> +	return ret;
>>>> +}

Re: [PATCH v5 10/16] media: platform: Add mali-c55 3a stats devnode

[Dan Scally]

Morning Laurent

On 01/07/2024 16:12, Dan Scally wrote:
> Hi Laurent
>
> On 29/06/2024 16:04, Laurent Pinchart wrote:
>> Hi Dan,
>>
>> On Thu, Jun 20, 2024 at 04:10:01PM +0100, Daniel Scally wrote:
>>> On 16/06/2024 22:19, Laurent Pinchart wrote:
>>>> On Wed, May 29, 2024 at 04:28:52PM +0100, Daniel Scally wrote:
>>>>> Add a new code file to govern the 3a statistics capture node.
>>>>>
>>>>> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
>>>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
>>>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
>>>>> ---
>>>>> Changes in v5:
>>>>>
>>>>>     - New patch
>>>>>
>>>>>    drivers/media/platform/arm/mali-c55/Makefile  |   3 +-
>>>>>    .../platform/arm/mali-c55/mali-c55-common.h   |  28 ++
>>>>>    .../platform/arm/mali-c55/mali-c55-core.c     |  15 +
>>>>>    .../platform/arm/mali-c55/mali-c55-isp.c      |   1 +
>>>>>    .../arm/mali-c55/mali-c55-registers.h         |   3 +
>>>>>    .../platform/arm/mali-c55/mali-c55-stats.c    | 350 ++++++++++++++++++
>>>>>    6 files changed, 399 insertions(+), 1 deletion(-)
>>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-stats.c
>>>>>
>>>>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile 
>>>>> b/drivers/media/platform/arm/mali-c55/Makefile
>>>>> index 77dcb2fbf0f4..cd5a64bf0c62 100644
>>>>> --- a/drivers/media/platform/arm/mali-c55/Makefile
>>>>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
>>>>> @@ -4,6 +4,7 @@ mali-c55-y := mali-c55-capture.o \
>>>>>              mali-c55-core.o \
>>>>>              mali-c55-isp.o \
>>>>>              mali-c55-tpg.o \
>>>>> -          mali-c55-resizer.o
>>>>> +          mali-c55-resizer.o \
>>>>> +          mali-c55-stats.o
>>>>>       obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h 
>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>>> index 2d0c4d152beb..44119e04009b 100644
>>>>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
>>>>> @@ -79,6 +79,7 @@ enum mali_c55_isp_pads {
>>>>>        MALI_C55_ISP_PAD_SINK_VIDEO,
>>>>>        MALI_C55_ISP_PAD_SOURCE,
>>>>>        MALI_C55_ISP_PAD_SOURCE_BYPASS,
>>>>> +    MALI_C55_ISP_PAD_SOURCE_3A,
>>>> Functions and structures are named with a "stats" suffix, let's call
>>>> this MALI_C55_ISP_PAD_SOURCE_STATS.
>>>>
>>>>>        MALI_C55_ISP_NUM_PADS,
>>>>>    };
>>>>>    @@ -194,6 +195,28 @@ struct mali_c55_cap_dev {
>>>>>        bool streaming;
>>>>>    };
>>>>>    +struct mali_c55_stats_buf {
>>>>> +    struct vb2_v4l2_buffer vb;
>>>>> +    spinlock_t lock;
>>>> All locks require a comment to document what they protect. Same below.
>>>>
>>>>> +    unsigned int segments_remaining;
>>>>> +    struct list_head queue;
>>>>> +    bool failed;
>>>>> +};
>>>>> +
>>>>> +struct mali_c55_stats {
>>>>> +    struct mali_c55 *mali_c55;
>>>>> +    struct video_device vdev;
>>>>> +    struct dma_chan *channel;
>>>>> +    struct vb2_queue queue;
>>>>> +    struct media_pad pad;
>>>>> +    struct mutex lock;
>>>>> +
>>>>> +    struct {
>>>>> +        spinlock_t lock;
>>>>> +        struct list_head queue;
>>>>> +    } buffers;
>>>>> +};
>>>>> +
>>>>>    enum mali_c55_config_spaces {
>>>>>        MALI_C55_CONFIG_PING,
>>>>>        MALI_C55_CONFIG_PONG,
>>>>> @@ -224,6 +247,7 @@ struct mali_c55 {
>>>>>        struct mali_c55_isp isp;
>>>>>        struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
>>>>>        struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
>>>>> +    struct mali_c55_stats stats;
>>>>>           struct list_head contexts;
>>>>>        enum mali_c55_config_spaces next_config;
>>>>> @@ -245,6 +269,8 @@ int mali_c55_register_resizers(struct mali_c55 *mali_c55);
>>>>>    void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
>>>>>    int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
>>>>>    void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
>>>>> +int mali_c55_register_stats(struct mali_c55 *mali_c55);
>>>>> +void mali_c55_unregister_stats(struct mali_c55 *mali_c55);
>>>>>    struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
>>>>>    void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
>>>>>                     enum mali_c55_planes plane);
>>>>> @@ -262,5 +288,7 @@ mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
>>>>>    bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
>>>>>    #define for_each_mali_isp_fmt(fmt)\
>>>>>        for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
>>>>> +void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
>>>>> +                enum mali_c55_config_spaces cfg_space);
>>>>>       #endif /* _MALI_C55_COMMON_H */
>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c 
>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>>> index 50caf5ee7474..9ea70010876c 100644
>>>>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
>>>>> @@ -337,6 +337,16 @@ static int mali_c55_create_links(struct mali_c55 *mali_c55)
>>>>>            }
>>>>>        }
>>>>>    +    ret = media_create_pad_link(&mali_c55->isp.sd.entity,
>>>>> +            MALI_C55_ISP_PAD_SOURCE_3A,
>>>>> +            &mali_c55->stats.vdev.entity, 0,
>>>>> +            MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
>>>>> +    if (ret) {
>>>>> +        dev_err(mali_c55->dev,
>>>>> +            "failed to link ISP and 3a stats node\n");
>>>> s/3a stats/stats/
>>>>
>>>>> +        goto err_remove_links;
>>>>> +    }
>>>>> +
>>>>>        return 0;
>>>>>       err_remove_links:
>>>>> @@ -350,6 +360,7 @@ static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
>>>>>        mali_c55_unregister_isp(mali_c55);
>>>>>        mali_c55_unregister_resizers(mali_c55);
>>>>>        mali_c55_unregister_capture_devs(mali_c55);
>>>>> +    mali_c55_unregister_stats(mali_c55);
>>>>>    }
>>>>>       static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>>>>> @@ -372,6 +383,10 @@ static int mali_c55_register_entities(struct mali_c55 *mali_c55)
>>>>>        if (ret)
>>>>>            goto err_unregister_entities;
>>>>>    +    ret = mali_c55_register_stats(mali_c55);
>>>>> +    if (ret)
>>>>> +        goto err_unregister_entities;
>>>>> +
>>>>>        ret = mali_c55_create_links(mali_c55);
>>>>>        if (ret)
>>>>>            goto err_unregister_entities;
>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c 
>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>>> index ea8b7b866e7a..94876fba3353 100644
>>>>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
>>>>> @@ -564,6 +564,7 @@ int mali_c55_register_isp(struct mali_c55 *mali_c55)
>>>>>        isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
>>>>>        isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>>>>>        isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
>>>>> +    isp->pads[MALI_C55_ISP_PAD_SOURCE_3A].flags = MEDIA_PAD_FL_SOURCE;
>>>>>           ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
>>>>>                         isp->pads);
>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h 
>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>>> index cb27abde2aa5..eb3719245ec3 100644
>>>>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
>>>>> @@ -68,6 +68,9 @@
>>>>>    #define MALI_C55_VC_START(v)                ((v) & 0xffff)
>>>>>    #define MALI_C55_VC_SIZE(v)                (((v) & 0xffff) << 16)
>>>>>    +#define MALI_C55_REG_1024BIN_HIST            0x054a8
>>>>> +#define MALI_C55_1024BIN_HIST_SIZE            4096
>>>>> +
>>>>>    /* Ping/Pong Configuration Space */
>>>>>    #define MALI_C55_REG_BASE_ADDR                0x18e88
>>>>>    #define MALI_C55_REG_BYPASS_0                0x18eac
>>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-stats.c 
>>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-stats.c
>>>>> new file mode 100644
>>>>> index 000000000000..aa40480ed814
>>>>> --- /dev/null
>>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-stats.c
>>>>> @@ -0,0 +1,350 @@
>>>>> +// SPDX-License-Identifier: GPL-2.0
>>>>> +/*
>>>>> + * ARM Mali-C55 ISP Driver - 3A Statistics capture device
>>>>> + *
>>>>> + * Copyright (C) 2023 Ideas on Board Oy
>>>>> + */
>>>>> +
>>>>> +#include <linux/dmaengine.h>
>>>>> +#include <linux/media/arm/mali-c55-config.h>
>>>>> +#include <linux/spinlock.h>
>>>> You're missing some headers here, for
>>>>
>>>> container_of()
>>>> dev_err()
>>>> list_*()
>>>> mutex_init()
>>>> strscpy()
>>>> strscpy()
>>>>
>>>>> +
>>>>> +#include <media/media-entity.h>
>>>>> +#include <media/v4l2-dev.h>
>>>>> +#include <media/v4l2-event.h>
>>>>> +#include <media/v4l2-fh.h>
>>>>> +#include <media/v4l2-ioctl.h>
>>>>> +#include <media/videobuf2-core.h>
>>>>> +#include <media/videobuf2-dma-contig.h>
>>>>> +
>>>>> +#include "mali-c55-common.h"
>>>>> +#include "mali-c55-registers.h"
>>>>> +
>>>>> +static unsigned int metering_space_addrs[] = {
>>>> const
>>>>
>>>>> +    [MALI_C55_CONFIG_PING] = 0x095AC,
>>>>> +    [MALI_C55_CONFIG_PONG] = 0x2156C,
>>>> Lower-case hex constants.
>>>>
>>>>> +};
>>>>> +
>>>>> +static int mali_c55_stats_enum_fmt_meta_cap(struct file *file, void *fh,
>>>>> +                        struct v4l2_fmtdesc *f)
>>>>> +{
>>>>> +    if (f->index || f->type != V4L2_BUF_TYPE_META_CAPTURE)
>>>>> +        return -EINVAL;
>>>>> +
>>>>> +    f->pixelformat = V4L2_META_FMT_MALI_C55_3A_STATS;
>>>> The format could be called V4L2_META_FMT_MALI_C55_STATS. While most
>>>> statistics are related to one of the 3A algorithms, I think it would be
>>>> better to name this generically. It's name bikeshedding only of course.
>>>>
>>>>> +
>>>>> +    return 0;
>>>>> +}
>>>>> +
>>>>> +static int mali_c55_stats_g_fmt_meta_cap(struct file *file, void *fh,
>>>>> +                     struct v4l2_format *f)
>>>>> +{
>>>>> +    static const struct v4l2_meta_format mfmt = {
>>>>> +        .dataformat = V4L2_META_FMT_MALI_C55_3A_STATS,
>>>>> +        .buffersize = sizeof(struct mali_c55_stats_buffer)
>>>>> +    };
>>>>> +
>>>>> +    if (f->type != V4L2_BUF_TYPE_META_CAPTURE)
>>>>> +        return -EINVAL;
>>>>> +
>>>>> +    f->fmt.meta = mfmt;
>>>>> +
>>>>> +    return 0;
>>>>> +}
>>>>> +
>>>>> +static int mali_c55_stats_querycap(struct file *file,
>>>>> +                   void *priv, struct v4l2_capability *cap)
>>>>> +{
>>>>> +    strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
>>>>> +    strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
>>>>> +
>>>>> +    return 0;
>>>>> +}
>>>>> +
>>>>> +static const struct v4l2_ioctl_ops mali_c55_stats_v4l2_ioctl_ops = {
>>>>> +    .vidioc_reqbufs = vb2_ioctl_reqbufs,
>>>>> +    .vidioc_querybuf = vb2_ioctl_querybuf,
>>>>> +    .vidioc_create_bufs = vb2_ioctl_create_bufs,
>>>>> +    .vidioc_qbuf = vb2_ioctl_qbuf,
>>>>> +    .vidioc_expbuf = vb2_ioctl_expbuf,
>>>>> +    .vidioc_dqbuf = vb2_ioctl_dqbuf,
>>>>> +    .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
>>>>> +    .vidioc_streamon = vb2_ioctl_streamon,
>>>>> +    .vidioc_streamoff = vb2_ioctl_streamoff,
>>>>> +    .vidioc_enum_fmt_meta_cap = mali_c55_stats_enum_fmt_meta_cap,
>>>>> +    .vidioc_g_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
>>>>> +    .vidioc_s_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
>>>>> +    .vidioc_try_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
>>>>> +    .vidioc_querycap = mali_c55_stats_querycap,
>>>>> +    .vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
>>>>> +    .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
>>>>> +};
>>>>> +
>>>>> +static const struct v4l2_file_operations mali_c55_stats_v4l2_fops = {
>>>>> +    .owner = THIS_MODULE,
>>>>> +    .unlocked_ioctl = video_ioctl2,
>>>>> +    .open = v4l2_fh_open,
>>>>> +    .release = vb2_fop_release,
>>>>> +    .poll = vb2_fop_poll,
>>>>> +    .mmap = vb2_fop_mmap,
>>>>> +};
>>>>> +
>>>>> +static int
>>>>> +mali_c55_stats_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
>>>>> +               unsigned int *num_planes, unsigned int sizes[],
>>>>> +               struct device *alloc_devs[])
>>>>> +{
>>>>> +    struct mali_c55_stats *stats = vb2_get_drv_priv(q);
>>>>> +
>>>>> +    if (*num_planes && *num_planes > 1)
>>>>> +        return -EINVAL;
>>>>> +
>>>>> +    if (sizes[0] && sizes[0] != sizeof(struct mali_c55_stats_buffer))
>>>>> +        return -EINVAL;
>>>>> +
>>>>> +    *num_planes = 1;
>>>>> +    sizes[0] = sizeof(struct mali_c55_stats_buffer);
>>>>> +
>>>>> +    if (stats->channel)
>>>>> +        alloc_devs[0] = stats->channel->device->dev;
>>>>> +
>>>>> +    return 0;
>>>>> +}
>>>>> +
>>>>> +static void mali_c55_stats_buf_queue(struct vb2_buffer *vb)
>>>>> +{
>>>>> +    struct mali_c55_stats *stats = vb2_get_drv_priv(vb->vb2_queue);
>>>>> +    struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>>>>> +    struct mali_c55_stats_buf *buf = container_of(vbuf,
>>>>> +                        struct mali_c55_stats_buf, vb);
>>>>> +
>>>>> +    vb2_set_plane_payload(vb, 0, sizeof(struct mali_c55_stats_buffer));
>>>>> +    buf->segments_remaining = 2;
>>>>> +    buf->failed = false;
>>>>> +
>>>>> +    spin_lock(&stats->buffers.lock);
>>>> Isn't the DMA completion handler run from IRQ context ? If so you'll
>>>> need to use spin_lock_irq() here and in the other function that are
>>>> not called with interrupts disabled.
>>> They're run in the bottom half of the interrupt handler; I'm under the
>>> impression that that means the interrupts aren't disabled, and it's
>>> safe to do...is that mistaken?
>> I'm talking about the DMA completion handler, called by the DMA engine
>> subsystem, not the IRQs of the C55 itself.
>
>
> Ah! I follow you now sorry. Hm, that's an interesting thought...I think for the PL330 the answer 
> is no; the interrupt handler wakes a thread that runs the callbacks for any DMA transfers that are 
> completed...I don't know that that's a design rule we can rely on though so possibly it's better 
> to be safe. Is there a disadvantage to using spin_lock_irq() outside of IRQ context?


Actually I think from [1] that the answer is they're not run from interrupt context, the page says 
"Note that callbacks will always be invoked from the DMA engine's tasklet, never from interrupt 
context".


[1] Documentation/driver-api/dmaengine/client.rst

>
>>
>>>>> + list_add_tail(&buf->queue, &stats->buffers.queue);
>>>>> +    spin_unlock(&stats->buffers.lock);
>>>>> +}
>>>>> +
>>>>> +static void mali_c55_stats_stop_streaming(struct vb2_queue *q)
>>>>> +{
>>>>> +    struct mali_c55_stats *stats = vb2_get_drv_priv(q);
>>>>> +    struct mali_c55_stats_buf *buf, *tmp;
>>>>> +
>>>>> +    spin_lock(&stats->buffers.lock);
>>>>> +
>>>>> +    list_for_each_entry_safe(buf, tmp, &stats->buffers.queue, queue) {
>>>>> +        list_del(&buf->queue);
>>>>> +        vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
>>>>> +    }
>>>>> +
>>>>> +    spin_unlock(&stats->buffers.lock);
>>>>> +}
>>>>> +
>>>>> +static const struct vb2_ops mali_c55_stats_vb2_ops = {
>>>>> +    .queue_setup = mali_c55_stats_queue_setup,
>>>>> +    .buf_queue = mali_c55_stats_buf_queue,
>>>>> +    .wait_prepare = vb2_ops_wait_prepare,
>>>>> +    .wait_finish = vb2_ops_wait_finish,
>>>>> +    .stop_streaming = mali_c55_stats_stop_streaming,
>>>>> +};
>>>>> +
>>>>> +static void
>>>>> +mali_c55_stats_metering_complete(void *param,
>>>>> +                 const struct dmaengine_result *result)
>>>>> +{
>>>>> +    struct mali_c55_stats_buf *buf = param;
>>>>> +
>>>>> +    spin_lock(&buf->lock);
>>>> I wonder if this is needed. Can the DMA engine call the completion
>>>> handlers of two sequential DMA transfers in parallel ?
>>> The DMA engine that's on the system we have can't...I wasn't sure
>>> whether that was generically true.
>> I think it is, but please double-check.
>>
>>>>> +
>>>>> +    if (buf->failed)
>>>>> +        goto out_unlock;
>>>>> +
>>>>> +    buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>>>>> +
>>>>> +    if (result->result != DMA_TRANS_NOERROR) {
>>>>> +        buf->failed = true;
>>>>> +        vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
>>>> This will possibly return the buffer to userspace after the first DMA
>>>> transfer. Userspace could then requeue the buffer to the kernel before
>>>> the completion of the second DMA transfer. That will cause trouble. I
>>>> think you should instead do something like
>>>>
>>>>     spin_lock(&buf->lock);
>>>>
>>>>     if (result->result != DMA_TRANS_NOERROR)
>>>>         buf->failed = true;
>>>>
>>>>     if (!--buf->segments_remaining) {
>>>>         buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>>>>         vb2_buffer_done(&buf->vb.vb2_buf, buf->failed ?
>>>>                 VB2_BUF_STATE_ERROR : VB2_BUF_STATE_DONE);
>>>>     }
>>>>
>>>>     spin_unlock(&buf->lock);
>>>>
>>>> The
>>>>
>>>>     buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
>>>>
>>>> line could also be moved to mali_c55_stats_fill_buffer(), which would
>>>> make sure the timestamp is filled in case of DMA submission failures.
>>> Okedokey
>>>
>>>>> +        goto out_unlock;
>>>>> +    }
>>>>> +
>>>>> +    if (!--buf->segments_remaining)
>>>>> +        vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
>>>>> +
>>>>> +out_unlock:
>>>>> +    spin_unlock(&buf->lock);
>>>>> +}
>>>>> +
>>>>> +static int mali_c55_stats_dma_xfer(struct mali_c55_stats *stats, dma_addr_t src,
>>>>> +                   dma_addr_t dst,
>>>>> +                   struct mali_c55_stats_buf *buf,
>>>>> +                   size_t length,
>>>>> +                   void (*callback)(void *, const struct dmaengine_result *result))
>>>> The same callback is used for both invocations of this function, you can
>>>> drop the parameter and hardcode it below.
>>> Yeah, not even sure now why I had a parameter.
>>>
>>>>> +{
>>>>> +    struct dma_async_tx_descriptor *tx;
>>>>> +    dma_cookie_t cookie;
>>>>> +
>>>>> +    tx = dmaengine_prep_dma_memcpy(stats->channel, dst, src, length, 0);
>>>>> +    if (!tx) {
>>>>> +        dev_err(stats->mali_c55->dev, "failed to prep stats DMA\n");
>>>>> +        return -EIO;
>>>>> +    }
>>>>> +
>>>>> +    tx->callback_result = callback;
>>>>> +    tx->callback_param = buf;
>>>>> +
>>>>> +    cookie = dmaengine_submit(tx);
>>>>> +    if (dma_submit_error(cookie)) {
>>>>> +        dev_err(stats->mali_c55->dev, "failed to submit stats DMA\n");
>>>>> +        return -EIO;
>>>>> +    }
>>>>> +
>>>>> +    dma_async_issue_pending(stats->channel);
>>>>> +    return 0;
>>>>> +}
>>>>> +
>>>>> +void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
>>>>> +                enum mali_c55_config_spaces cfg_space)
>>>>> +{
>>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
>>>>> +    struct mali_c55_stats *stats = &mali_c55->stats;
>>>>> +    struct mali_c55_stats_buf *buf = NULL;
>>>>> +    dma_addr_t src, dst;
>>>>> +    int ret;
>>>>> +
>>>>> +    spin_lock(&stats->buffers.lock);
>>>>> +    if (!list_empty(&stats->buffers.queue)) {
>>>>> +        buf = list_first_entry(&stats->buffers.queue,
>>>>> +                       struct mali_c55_stats_buf, queue);
>>>>> +        list_del(&buf->queue);
>>>>> +    }
>>>>> +    spin_unlock(&stats->buffers.lock);
>>>>> +
>>>>> +    if (!buf)
>>>>> +        return;
>>>>> +
>>>>> +    buf->vb.sequence = mali_c55->isp.frame_sequence;
>>>>> +
>>>>> +    /*
>>>>> +     * There are infact two noncontiguous sections of the ISP's
>>>> s/infact/in fact/
>>>>
>>>>> +     * memory space that hold statistics for 3a algorithms to use. A
>>>> s/use. A/use: a/
>>>>
>>>>> +     * section in each config space and a global section holding
>>>>> +     * histograms which is double buffered and so holds data for the
>>>>> +     * last frame. We need to read both.
>>>>> +     */
>>>>> +    src = ctx->base + MALI_C55_REG_1024BIN_HIST;
>>>>> +    dst = vb2_dma_contig_plane_dma_addr(&buf->vb.vb2_buf, 0);
>>>>> +
>>>>> +    ret = mali_c55_stats_dma_xfer(stats, src, dst, buf,
>>>>> +                      MALI_C55_1024BIN_HIST_SIZE,
>>>>> +                      mali_c55_stats_metering_complete);
>>>>> +    if (ret)
>>>>> +        goto err_fail_buffer;
>>>>> +
>>>>> +    src = ctx->base + metering_space_addrs[cfg_space];
>>>>> +    dst += MALI_C55_1024BIN_HIST_SIZE;
>>>>> +
>>>>> +    ret = mali_c55_stats_dma_xfer(
>>>>> +        stats, src, dst, buf,
>>>>> +        sizeof(struct mali_c55_stats_buffer) - MALI_C55_1024BIN_HIST_SIZE,
>>>>> +        mali_c55_stats_metering_complete);
>>>>> +    if (ret) {
>>>>> +        dmaengine_terminate_sync(stats->channel);
>>>>> +        goto err_fail_buffer;
>>>>> +    }
>>>> I think you will need to terminate DMA transfers at stream off time.
>>>>
>>>>> +
>>>>> +    return;
>>>>> +
>>>>> +err_fail_buffer:
>>>>> +    vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
>>>>> +}
>>>>> +
>>>>> +void mali_c55_unregister_stats(struct mali_c55 *mali_c55)
>>>>> +{
>>>>> +    struct mali_c55_stats *stats = &mali_c55->stats;
>>>>> +
>>>>> +    if (!video_is_registered(&stats->vdev))
>>>>> +        return;
>>>>> +
>>>>> +    vb2_video_unregister_device(&stats->vdev);
>>>>> +    media_entity_cleanup(&stats->vdev.entity);
>>>>> +    dma_release_channel(stats->channel);
>>>>> +    mutex_destroy(&stats->lock);
>>>>> +}
>>>>> +
>>>>> +int mali_c55_register_stats(struct mali_c55 *mali_c55)
>>>>> +{
>>>>> +    struct mali_c55_stats *stats = &mali_c55->stats;
>>>>> +    struct video_device *vdev = &stats->vdev;
>>>>> +    struct vb2_queue *vb2q = &stats->queue;
>>>>> +    dma_cap_mask_t mask;
>>>>> +    int ret;
>>>>> +
>>>>> +    mutex_init(&stats->lock);
>>>>> +    INIT_LIST_HEAD(&stats->buffers.queue);
>>>>> +
>>>>> +    dma_cap_zero(mask);
>>>>> +    dma_cap_set(DMA_MEMCPY, mask);
>>>>> +
>>>>> +    stats->channel = dma_request_channel(mask, 0, NULL);
>>>> Do we need a CPU fallback in case no DMA is available ?
>>> Yes, actually.
>>>
>>>> I'm still very curious to know how long it takes to perform the DMA
>>>> transfer, compared to copying the data with the CPU, and especially
>>>> compared to the frame duration.
>>> On my list of things to test and report :)
>> Looking forward to it :-)
>>
>>>>> +    if (!stats->channel) {
>>>>> +        ret = -ENODEV;
>>>>> +        goto err_destroy_mutex;
>>>>> +    }
>>>>> +
>>>>> +    stats->pad.flags = MEDIA_PAD_FL_SINK;
>>>>> +    ret = media_entity_pads_init(&stats->vdev.entity, 1, &stats->pad);
>>>>> +    if (ret)
>>>>> +        goto err_release_dma_channel;
>>>>> +
>>>>> +    vb2q->type = V4L2_BUF_TYPE_META_CAPTURE;
>>>>> +    vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
>>>>> +    vb2q->drv_priv = stats;
>>>>> +    vb2q->mem_ops = &vb2_dma_contig_memops;
>>>>> +    vb2q->ops = &mali_c55_stats_vb2_ops;
>>>>> +    vb2q->buf_struct_size = sizeof(struct mali_c55_stats_buf);
>>>>> +    vb2q->min_queued_buffers = 1;
>>>>> +    vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
>>>>> +    vb2q->lock = &stats->lock;
>>>>> +    vb2q->dev = mali_c55->dev;
>>>> That's not the right device. The device that performs the DMA operation
>>>> is the DMA engine, and that's what you need to pass to vb2. Otherwise
>>>> the DMA address returned by vb2_dma_contig_plane_dma_addr() will be
>>>> mapped to the ISP device, not the DMA engine. In practice, if neither
>>>> are behind an IOMMU, things will likely work, but when that's not the
>>>> case, run into problems.
>>>>
>>>>> +
>>>>> +    ret = vb2_queue_init(vb2q);
>>>>> +    if (ret) {
>>>>> +        dev_err(mali_c55->dev, "stats vb2 queue init failed\n");
>>>>> +        goto err_cleanup_entity;
>>>>> +    }
>>>>> +
>>>>> +    strscpy(stats->vdev.name, "mali-c55 3a stats", sizeof(stats->vdev.name));
>>>> s/3a //
>>>>
>>>>> +    vdev->release = video_device_release_empty;
>>>> That's never right. You should refcount the data structures to ensure
>>>> proper lifetime management.
>>>>
>>>>> +    vdev->fops = &mali_c55_stats_v4l2_fops;
>>>>> +    vdev->ioctl_ops = &mali_c55_stats_v4l2_ioctl_ops;
>>>>> +    vdev->lock = &stats->lock;
>>>>> +    vdev->v4l2_dev = &mali_c55->v4l2_dev;
>>>>> +    vdev->queue = &stats->queue;
>>>>> +    vdev->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING;
>>>>> +    vdev->vfl_dir = VFL_DIR_RX;
>>>>> +    video_set_drvdata(vdev, stats);
>>>>> +
>>>>> +    ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
>>>>> +    if (ret) {
>>>>> +        dev_err(mali_c55->dev,
>>>>> +            "failed to register stats video device\n");
>>>>> +        goto err_release_vb2q;
>>>>> +    }
>>>>> +
>>>>> +    stats->mali_c55 = mali_c55;
>>>>> +
>>>>> +    return 0;
>>>>> +
>>>>> +err_release_vb2q:
>>>>> +    vb2_queue_release(vb2q);
>>>>> +err_cleanup_entity:
>>>>> +    media_entity_cleanup(&stats->vdev.entity);
>>>>> +err_release_dma_channel:
>>>>> +    dma_release_channel(stats->channel);
>>>>> +err_destroy_mutex:
>>>>> +    mutex_destroy(&stats->lock);
>>>>> +
>>>>> +    return ret;
>>>>> +}

Re: [PATCH v5 10/16] media: platform: Add mali-c55 3a stats devnode

[Laurent Pinchart]

Hi Dan,

On Tue, Jul 02, 2024 at 08:00:27AM +0100, Daniel Scally wrote:
> On 01/07/2024 16:12, Dan Scally wrote:
> > On 29/06/2024 16:04, Laurent Pinchart wrote:
> >> On Thu, Jun 20, 2024 at 04:10:01PM +0100, Daniel Scally wrote:
> >>> On 16/06/2024 22:19, Laurent Pinchart wrote:
> >>>> On Wed, May 29, 2024 at 04:28:52PM +0100, Daniel Scally wrote:
> >>>>> Add a new code file to govern the 3a statistics capture node.
> >>>>>
> >>>>> Acked-by: Nayden Kanchev  <nayden.kanchev@arm.com>
> >>>>> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >>>>> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> >>>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
> >>>>> ---
> >>>>> Changes in v5:
> >>>>>
> >>>>>     - New patch
> >>>>>
> >>>>>    drivers/media/platform/arm/mali-c55/Makefile  |   3 +-
> >>>>>    .../platform/arm/mali-c55/mali-c55-common.h   |  28 ++
> >>>>>    .../platform/arm/mali-c55/mali-c55-core.c     |  15 +
> >>>>>    .../platform/arm/mali-c55/mali-c55-isp.c      |   1 +
> >>>>>    .../arm/mali-c55/mali-c55-registers.h         |   3 +
> >>>>>    .../platform/arm/mali-c55/mali-c55-stats.c    | 350 ++++++++++++++++++
> >>>>>    6 files changed, 399 insertions(+), 1 deletion(-)
> >>>>>    create mode 100644 drivers/media/platform/arm/mali-c55/mali-c55-stats.c
> >>>>>
> >>>>> diff --git a/drivers/media/platform/arm/mali-c55/Makefile 
> >>>>> b/drivers/media/platform/arm/mali-c55/Makefile
> >>>>> index 77dcb2fbf0f4..cd5a64bf0c62 100644
> >>>>> --- a/drivers/media/platform/arm/mali-c55/Makefile
> >>>>> +++ b/drivers/media/platform/arm/mali-c55/Makefile
> >>>>> @@ -4,6 +4,7 @@ mali-c55-y := mali-c55-capture.o \
> >>>>>              mali-c55-core.o \
> >>>>>              mali-c55-isp.o \
> >>>>>              mali-c55-tpg.o \
> >>>>> -          mali-c55-resizer.o
> >>>>> +          mali-c55-resizer.o \
> >>>>> +          mali-c55-stats.o
> >>>>>       obj-$(CONFIG_VIDEO_MALI_C55) += mali-c55.o
> >>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-common.h 
> >>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>>> index 2d0c4d152beb..44119e04009b 100644
> >>>>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-common.h
> >>>>> @@ -79,6 +79,7 @@ enum mali_c55_isp_pads {
> >>>>>        MALI_C55_ISP_PAD_SINK_VIDEO,
> >>>>>        MALI_C55_ISP_PAD_SOURCE,
> >>>>>        MALI_C55_ISP_PAD_SOURCE_BYPASS,
> >>>>> +    MALI_C55_ISP_PAD_SOURCE_3A,
> >>>> Functions and structures are named with a "stats" suffix, let's call
> >>>> this MALI_C55_ISP_PAD_SOURCE_STATS.
> >>>>
> >>>>>        MALI_C55_ISP_NUM_PADS,
> >>>>>    };
> >>>>>    @@ -194,6 +195,28 @@ struct mali_c55_cap_dev {
> >>>>>        bool streaming;
> >>>>>    };
> >>>>>    +struct mali_c55_stats_buf {
> >>>>> +    struct vb2_v4l2_buffer vb;
> >>>>> +    spinlock_t lock;
> >>>> All locks require a comment to document what they protect. Same below.
> >>>>
> >>>>> +    unsigned int segments_remaining;
> >>>>> +    struct list_head queue;
> >>>>> +    bool failed;
> >>>>> +};
> >>>>> +
> >>>>> +struct mali_c55_stats {
> >>>>> +    struct mali_c55 *mali_c55;
> >>>>> +    struct video_device vdev;
> >>>>> +    struct dma_chan *channel;
> >>>>> +    struct vb2_queue queue;
> >>>>> +    struct media_pad pad;
> >>>>> +    struct mutex lock;
> >>>>> +
> >>>>> +    struct {
> >>>>> +        spinlock_t lock;
> >>>>> +        struct list_head queue;
> >>>>> +    } buffers;
> >>>>> +};
> >>>>> +
> >>>>>    enum mali_c55_config_spaces {
> >>>>>        MALI_C55_CONFIG_PING,
> >>>>>        MALI_C55_CONFIG_PONG,
> >>>>> @@ -224,6 +247,7 @@ struct mali_c55 {
> >>>>>        struct mali_c55_isp isp;
> >>>>>        struct mali_c55_resizer resizers[MALI_C55_NUM_RZRS];
> >>>>>        struct mali_c55_cap_dev cap_devs[MALI_C55_NUM_CAP_DEVS];
> >>>>> +    struct mali_c55_stats stats;
> >>>>>           struct list_head contexts;
> >>>>>        enum mali_c55_config_spaces next_config;
> >>>>> @@ -245,6 +269,8 @@ int mali_c55_register_resizers(struct mali_c55 *mali_c55);
> >>>>>    void mali_c55_unregister_resizers(struct mali_c55 *mali_c55);
> >>>>>    int mali_c55_register_capture_devs(struct mali_c55 *mali_c55);
> >>>>>    void mali_c55_unregister_capture_devs(struct mali_c55 *mali_c55);
> >>>>> +int mali_c55_register_stats(struct mali_c55 *mali_c55);
> >>>>> +void mali_c55_unregister_stats(struct mali_c55 *mali_c55);
> >>>>>    struct mali_c55_ctx *mali_c55_get_active_context(struct mali_c55 *mali_c55);
> >>>>>    void mali_c55_set_plane_done(struct mali_c55_cap_dev *cap_dev,
> >>>>>                     enum mali_c55_planes plane);
> >>>>> @@ -262,5 +288,7 @@ mali_c55_isp_fmt_next(const struct mali_c55_isp_fmt *fmt);
> >>>>>    bool mali_c55_isp_is_format_supported(unsigned int mbus_code);
> >>>>>    #define for_each_mali_isp_fmt(fmt)\
> >>>>>        for ((fmt) = NULL; ((fmt) = mali_c55_isp_fmt_next((fmt)));)
> >>>>> +void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
> >>>>> +                enum mali_c55_config_spaces cfg_space);
> >>>>>       #endif /* _MALI_C55_COMMON_H */
> >>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-core.c 
> >>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>>> index 50caf5ee7474..9ea70010876c 100644
> >>>>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-core.c
> >>>>> @@ -337,6 +337,16 @@ static int mali_c55_create_links(struct mali_c55 *mali_c55)
> >>>>>            }
> >>>>>        }
> >>>>>    +    ret = media_create_pad_link(&mali_c55->isp.sd.entity,
> >>>>> +            MALI_C55_ISP_PAD_SOURCE_3A,
> >>>>> +            &mali_c55->stats.vdev.entity, 0,
> >>>>> +            MEDIA_LNK_FL_ENABLED | MEDIA_LNK_FL_IMMUTABLE);
> >>>>> +    if (ret) {
> >>>>> +        dev_err(mali_c55->dev,
> >>>>> +            "failed to link ISP and 3a stats node\n");
> >>>> s/3a stats/stats/
> >>>>
> >>>>> +        goto err_remove_links;
> >>>>> +    }
> >>>>> +
> >>>>>        return 0;
> >>>>>       err_remove_links:
> >>>>> @@ -350,6 +360,7 @@ static void mali_c55_unregister_entities(struct mali_c55 *mali_c55)
> >>>>>        mali_c55_unregister_isp(mali_c55);
> >>>>>        mali_c55_unregister_resizers(mali_c55);
> >>>>>        mali_c55_unregister_capture_devs(mali_c55);
> >>>>> +    mali_c55_unregister_stats(mali_c55);
> >>>>>    }
> >>>>>       static int mali_c55_register_entities(struct mali_c55 *mali_c55)
> >>>>> @@ -372,6 +383,10 @@ static int mali_c55_register_entities(struct mali_c55 *mali_c55)
> >>>>>        if (ret)
> >>>>>            goto err_unregister_entities;
> >>>>>    +    ret = mali_c55_register_stats(mali_c55);
> >>>>> +    if (ret)
> >>>>> +        goto err_unregister_entities;
> >>>>> +
> >>>>>        ret = mali_c55_create_links(mali_c55);
> >>>>>        if (ret)
> >>>>>            goto err_unregister_entities;
> >>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c 
> >>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>>> index ea8b7b866e7a..94876fba3353 100644
> >>>>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-isp.c
> >>>>> @@ -564,6 +564,7 @@ int mali_c55_register_isp(struct mali_c55 *mali_c55)
> >>>>>        isp->pads[MALI_C55_ISP_PAD_SINK_VIDEO].flags = MEDIA_PAD_FL_SINK;
> >>>>>        isp->pads[MALI_C55_ISP_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> >>>>>        isp->pads[MALI_C55_ISP_PAD_SOURCE_BYPASS].flags = MEDIA_PAD_FL_SOURCE;
> >>>>> +    isp->pads[MALI_C55_ISP_PAD_SOURCE_3A].flags = MEDIA_PAD_FL_SOURCE;
> >>>>>           ret = media_entity_pads_init(&sd->entity, MALI_C55_ISP_NUM_PADS,
> >>>>>                         isp->pads);
> >>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h 
> >>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>>> index cb27abde2aa5..eb3719245ec3 100644
> >>>>> --- a/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-registers.h
> >>>>> @@ -68,6 +68,9 @@
> >>>>>    #define MALI_C55_VC_START(v)                ((v) & 0xffff)
> >>>>>    #define MALI_C55_VC_SIZE(v)                (((v) & 0xffff) << 16)
> >>>>>    +#define MALI_C55_REG_1024BIN_HIST            0x054a8
> >>>>> +#define MALI_C55_1024BIN_HIST_SIZE            4096
> >>>>> +
> >>>>>    /* Ping/Pong Configuration Space */
> >>>>>    #define MALI_C55_REG_BASE_ADDR                0x18e88
> >>>>>    #define MALI_C55_REG_BYPASS_0                0x18eac
> >>>>> diff --git a/drivers/media/platform/arm/mali-c55/mali-c55-stats.c 
> >>>>> b/drivers/media/platform/arm/mali-c55/mali-c55-stats.c
> >>>>> new file mode 100644
> >>>>> index 000000000000..aa40480ed814
> >>>>> --- /dev/null
> >>>>> +++ b/drivers/media/platform/arm/mali-c55/mali-c55-stats.c
> >>>>> @@ -0,0 +1,350 @@
> >>>>> +// SPDX-License-Identifier: GPL-2.0
> >>>>> +/*
> >>>>> + * ARM Mali-C55 ISP Driver - 3A Statistics capture device
> >>>>> + *
> >>>>> + * Copyright (C) 2023 Ideas on Board Oy
> >>>>> + */
> >>>>> +
> >>>>> +#include <linux/dmaengine.h>
> >>>>> +#include <linux/media/arm/mali-c55-config.h>
> >>>>> +#include <linux/spinlock.h>
> >>>> You're missing some headers here, for
> >>>>
> >>>> container_of()
> >>>> dev_err()
> >>>> list_*()
> >>>> mutex_init()
> >>>> strscpy()
> >>>> strscpy()
> >>>>
> >>>>> +
> >>>>> +#include <media/media-entity.h>
> >>>>> +#include <media/v4l2-dev.h>
> >>>>> +#include <media/v4l2-event.h>
> >>>>> +#include <media/v4l2-fh.h>
> >>>>> +#include <media/v4l2-ioctl.h>
> >>>>> +#include <media/videobuf2-core.h>
> >>>>> +#include <media/videobuf2-dma-contig.h>
> >>>>> +
> >>>>> +#include "mali-c55-common.h"
> >>>>> +#include "mali-c55-registers.h"
> >>>>> +
> >>>>> +static unsigned int metering_space_addrs[] = {
> >>>> const
> >>>>
> >>>>> +    [MALI_C55_CONFIG_PING] = 0x095AC,
> >>>>> +    [MALI_C55_CONFIG_PONG] = 0x2156C,
> >>>> Lower-case hex constants.
> >>>>
> >>>>> +};
> >>>>> +
> >>>>> +static int mali_c55_stats_enum_fmt_meta_cap(struct file *file, void *fh,
> >>>>> +                        struct v4l2_fmtdesc *f)
> >>>>> +{
> >>>>> +    if (f->index || f->type != V4L2_BUF_TYPE_META_CAPTURE)
> >>>>> +        return -EINVAL;
> >>>>> +
> >>>>> +    f->pixelformat = V4L2_META_FMT_MALI_C55_3A_STATS;
> >>>> The format could be called V4L2_META_FMT_MALI_C55_STATS. While most
> >>>> statistics are related to one of the 3A algorithms, I think it would be
> >>>> better to name this generically. It's name bikeshedding only of course.
> >>>>
> >>>>> +
> >>>>> +    return 0;
> >>>>> +}
> >>>>> +
> >>>>> +static int mali_c55_stats_g_fmt_meta_cap(struct file *file, void *fh,
> >>>>> +                     struct v4l2_format *f)
> >>>>> +{
> >>>>> +    static const struct v4l2_meta_format mfmt = {
> >>>>> +        .dataformat = V4L2_META_FMT_MALI_C55_3A_STATS,
> >>>>> +        .buffersize = sizeof(struct mali_c55_stats_buffer)
> >>>>> +    };
> >>>>> +
> >>>>> +    if (f->type != V4L2_BUF_TYPE_META_CAPTURE)
> >>>>> +        return -EINVAL;
> >>>>> +
> >>>>> +    f->fmt.meta = mfmt;
> >>>>> +
> >>>>> +    return 0;
> >>>>> +}
> >>>>> +
> >>>>> +static int mali_c55_stats_querycap(struct file *file,
> >>>>> +                   void *priv, struct v4l2_capability *cap)
> >>>>> +{
> >>>>> +    strscpy(cap->driver, MALI_C55_DRIVER_NAME, sizeof(cap->driver));
> >>>>> +    strscpy(cap->card, "ARM Mali-C55 ISP", sizeof(cap->card));
> >>>>> +
> >>>>> +    return 0;
> >>>>> +}
> >>>>> +
> >>>>> +static const struct v4l2_ioctl_ops mali_c55_stats_v4l2_ioctl_ops = {
> >>>>> +    .vidioc_reqbufs = vb2_ioctl_reqbufs,
> >>>>> +    .vidioc_querybuf = vb2_ioctl_querybuf,
> >>>>> +    .vidioc_create_bufs = vb2_ioctl_create_bufs,
> >>>>> +    .vidioc_qbuf = vb2_ioctl_qbuf,
> >>>>> +    .vidioc_expbuf = vb2_ioctl_expbuf,
> >>>>> +    .vidioc_dqbuf = vb2_ioctl_dqbuf,
> >>>>> +    .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> >>>>> +    .vidioc_streamon = vb2_ioctl_streamon,
> >>>>> +    .vidioc_streamoff = vb2_ioctl_streamoff,
> >>>>> +    .vidioc_enum_fmt_meta_cap = mali_c55_stats_enum_fmt_meta_cap,
> >>>>> +    .vidioc_g_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
> >>>>> +    .vidioc_s_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
> >>>>> +    .vidioc_try_fmt_meta_cap = mali_c55_stats_g_fmt_meta_cap,
> >>>>> +    .vidioc_querycap = mali_c55_stats_querycap,
> >>>>> +    .vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
> >>>>> +    .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
> >>>>> +};
> >>>>> +
> >>>>> +static const struct v4l2_file_operations mali_c55_stats_v4l2_fops = {
> >>>>> +    .owner = THIS_MODULE,
> >>>>> +    .unlocked_ioctl = video_ioctl2,
> >>>>> +    .open = v4l2_fh_open,
> >>>>> +    .release = vb2_fop_release,
> >>>>> +    .poll = vb2_fop_poll,
> >>>>> +    .mmap = vb2_fop_mmap,
> >>>>> +};
> >>>>> +
> >>>>> +static int
> >>>>> +mali_c55_stats_queue_setup(struct vb2_queue *q, unsigned int *num_buffers,
> >>>>> +               unsigned int *num_planes, unsigned int sizes[],
> >>>>> +               struct device *alloc_devs[])
> >>>>> +{
> >>>>> +    struct mali_c55_stats *stats = vb2_get_drv_priv(q);
> >>>>> +
> >>>>> +    if (*num_planes && *num_planes > 1)
> >>>>> +        return -EINVAL;
> >>>>> +
> >>>>> +    if (sizes[0] && sizes[0] != sizeof(struct mali_c55_stats_buffer))
> >>>>> +        return -EINVAL;
> >>>>> +
> >>>>> +    *num_planes = 1;
> >>>>> +    sizes[0] = sizeof(struct mali_c55_stats_buffer);
> >>>>> +
> >>>>> +    if (stats->channel)
> >>>>> +        alloc_devs[0] = stats->channel->device->dev;
> >>>>> +
> >>>>> +    return 0;
> >>>>> +}
> >>>>> +
> >>>>> +static void mali_c55_stats_buf_queue(struct vb2_buffer *vb)
> >>>>> +{
> >>>>> +    struct mali_c55_stats *stats = vb2_get_drv_priv(vb->vb2_queue);
> >>>>> +    struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> >>>>> +    struct mali_c55_stats_buf *buf = container_of(vbuf,
> >>>>> +                        struct mali_c55_stats_buf, vb);
> >>>>> +
> >>>>> +    vb2_set_plane_payload(vb, 0, sizeof(struct mali_c55_stats_buffer));
> >>>>> +    buf->segments_remaining = 2;
> >>>>> +    buf->failed = false;
> >>>>> +
> >>>>> +    spin_lock(&stats->buffers.lock);
> >>>>
> >>>> Isn't the DMA completion handler run from IRQ context ? If so you'll
> >>>> need to use spin_lock_irq() here and in the other function that are
> >>>> not called with interrupts disabled.
> >>>
> >>> They're run in the bottom half of the interrupt handler; I'm under the
> >>> impression that that means the interrupts aren't disabled, and it's
> >>> safe to do...is that mistaken?
> >>
> >> I'm talking about the DMA completion handler, called by the DMA engine
> >> subsystem, not the IRQs of the C55 itself.
> >
> > Ah! I follow you now sorry. Hm, that's an interesting thought...I think for the PL330 the answer 
> > is no; the interrupt handler wakes a thread that runs the callbacks for any DMA transfers that are 
> > completed...I don't know that that's a design rule we can rely on though so possibly it's better 
> > to be safe. Is there a disadvantage to using spin_lock_irq() outside of IRQ context?

spin_lock_irq() isn't meant for IRQ contexts. spin_lock_irq() means
"disable IRQs on this CPU and acquire the spinlock". spin_unlock_irq()
means "release the spinlock and enable IRQs on this CPU".

IRQs are not reentrant in Linux (RT support changes this a bit, but
API-wise the following explanation still holds), the kernel disables
interrupts on the local CPU before running an interrupt handler. This
means that, if the CPU has acquired a spinlock, and an interrupt occurs
on the same CPU, with the interrupt handler trying to acquire the same
spinlock, the kernel will deadlock as control will never be returned to
the context that holds the lock.

For this reason, when a spinlock can be acquired from an interrupt
handler, the *other* (non-IRQ) contexts need to call spin_lock_irq() to
disable interrupts on the local CPU before acquiring the lock.
spin_unlock_irq() will release the lock and reenable interrupts,
allowing the interrupt handler to proceed.

While spin_lock_irq() can technically be called from both IRQ and
non-IRQ context (as disabling interrupts if they're already disabled is
a no-op), calling spin_unlock_irq() from interrupt context means that
IRQs will be reenabled while the IRQ handler is still running. That will
lead to disaster sooner or later, so it's important that the interrupt
handler uses spin_lock() and spin_unlock().

In cases where the same code can be called from both IRQ context and
non-IRQ context, you should use spin_lock_irqsave() and
spin_unlock_irqrestore(). The former will save the IRQ state and disable
IRQs, while the latter will restore the IRQ state. Using
spin_lock_irqsave() and spin_lock_irqrestore() unconditionally is the
safest option as you don't have to think about contexts. The downside is
a small overhead due to the IRQ state save/restore.

The summary is that you can use the _irq{save,restore} variants
everywhere, but if you want to optimize the code, you need to

- Use spin_lock() and spin_unlock() in contexts where interrupts are
  disabled (IRQ handlers, or code sections that are covered by another
  spinlock already)

- Use spin_lock() and spin_unlock() in contexts where interrupts are
  enabled *if* the same lock can never be acquired *on the same CPU*
  from an IRQ-disabled context

- Use spin_lock_irq() and spin_unlock_irq() in contexts where interrupts
  are enabled *if* the same lock can be acquired *on the same CPU* from an
  IRQ-disabled context

- Use spin_lock_irqsave() and spin_unlock_irqrestore() in contexts where
  you can't guarantee the IRQ state (such as helper functions that can
  be called from different contexts)

> Actually I think from [1] that the answer is they're not run from interrupt context, the page says 
> "Note that callbacks will always be invoked from the DMA engine's tasklet, never from interrupt 
> context".

I'll leave as an exercise for the reader to determine if that context is
interruptible or not :-) You'll also need to check if the caller is
allowed to hold a spinlock while calling the callback.

> [1] Documentation/driver-api/dmaengine/client.rst
> 
> >>>>> + list_add_tail(&buf->queue, &stats->buffers.queue);
> >>>>> +    spin_unlock(&stats->buffers.lock);
> >>>>> +}
> >>>>> +
> >>>>> +static void mali_c55_stats_stop_streaming(struct vb2_queue *q)
> >>>>> +{
> >>>>> +    struct mali_c55_stats *stats = vb2_get_drv_priv(q);
> >>>>> +    struct mali_c55_stats_buf *buf, *tmp;
> >>>>> +
> >>>>> +    spin_lock(&stats->buffers.lock);
> >>>>> +
> >>>>> +    list_for_each_entry_safe(buf, tmp, &stats->buffers.queue, queue) {
> >>>>> +        list_del(&buf->queue);
> >>>>> +        vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
> >>>>> +    }
> >>>>> +
> >>>>> +    spin_unlock(&stats->buffers.lock);
> >>>>> +}
> >>>>> +
> >>>>> +static const struct vb2_ops mali_c55_stats_vb2_ops = {
> >>>>> +    .queue_setup = mali_c55_stats_queue_setup,
> >>>>> +    .buf_queue = mali_c55_stats_buf_queue,
> >>>>> +    .wait_prepare = vb2_ops_wait_prepare,
> >>>>> +    .wait_finish = vb2_ops_wait_finish,
> >>>>> +    .stop_streaming = mali_c55_stats_stop_streaming,
> >>>>> +};
> >>>>> +
> >>>>> +static void
> >>>>> +mali_c55_stats_metering_complete(void *param,
> >>>>> +                 const struct dmaengine_result *result)
> >>>>> +{
> >>>>> +    struct mali_c55_stats_buf *buf = param;
> >>>>> +
> >>>>> +    spin_lock(&buf->lock);
> >>>> I wonder if this is needed. Can the DMA engine call the completion
> >>>> handlers of two sequential DMA transfers in parallel ?
> >>> The DMA engine that's on the system we have can't...I wasn't sure
> >>> whether that was generically true.
> >> I think it is, but please double-check.
> >>
> >>>>> +
> >>>>> +    if (buf->failed)
> >>>>> +        goto out_unlock;
> >>>>> +
> >>>>> +    buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> >>>>> +
> >>>>> +    if (result->result != DMA_TRANS_NOERROR) {
> >>>>> +        buf->failed = true;
> >>>>> +        vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
> >>>> This will possibly return the buffer to userspace after the first DMA
> >>>> transfer. Userspace could then requeue the buffer to the kernel before
> >>>> the completion of the second DMA transfer. That will cause trouble. I
> >>>> think you should instead do something like
> >>>>
> >>>>     spin_lock(&buf->lock);
> >>>>
> >>>>     if (result->result != DMA_TRANS_NOERROR)
> >>>>         buf->failed = true;
> >>>>
> >>>>     if (!--buf->segments_remaining) {
> >>>>         buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> >>>>         vb2_buffer_done(&buf->vb.vb2_buf, buf->failed ?
> >>>>                 VB2_BUF_STATE_ERROR : VB2_BUF_STATE_DONE);
> >>>>     }
> >>>>
> >>>>     spin_unlock(&buf->lock);
> >>>>
> >>>> The
> >>>>
> >>>>     buf->vb.vb2_buf.timestamp = ktime_get_boottime_ns();
> >>>>
> >>>> line could also be moved to mali_c55_stats_fill_buffer(), which would
> >>>> make sure the timestamp is filled in case of DMA submission failures.
> >>> Okedokey
> >>>
> >>>>> +        goto out_unlock;
> >>>>> +    }
> >>>>> +
> >>>>> +    if (!--buf->segments_remaining)
> >>>>> +        vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
> >>>>> +
> >>>>> +out_unlock:
> >>>>> +    spin_unlock(&buf->lock);
> >>>>> +}
> >>>>> +
> >>>>> +static int mali_c55_stats_dma_xfer(struct mali_c55_stats *stats, dma_addr_t src,
> >>>>> +                   dma_addr_t dst,
> >>>>> +                   struct mali_c55_stats_buf *buf,
> >>>>> +                   size_t length,
> >>>>> +                   void (*callback)(void *, const struct dmaengine_result *result))
> >>>> The same callback is used for both invocations of this function, you can
> >>>> drop the parameter and hardcode it below.
> >>> Yeah, not even sure now why I had a parameter.
> >>>
> >>>>> +{
> >>>>> +    struct dma_async_tx_descriptor *tx;
> >>>>> +    dma_cookie_t cookie;
> >>>>> +
> >>>>> +    tx = dmaengine_prep_dma_memcpy(stats->channel, dst, src, length, 0);
> >>>>> +    if (!tx) {
> >>>>> +        dev_err(stats->mali_c55->dev, "failed to prep stats DMA\n");
> >>>>> +        return -EIO;
> >>>>> +    }
> >>>>> +
> >>>>> +    tx->callback_result = callback;
> >>>>> +    tx->callback_param = buf;
> >>>>> +
> >>>>> +    cookie = dmaengine_submit(tx);
> >>>>> +    if (dma_submit_error(cookie)) {
> >>>>> +        dev_err(stats->mali_c55->dev, "failed to submit stats DMA\n");
> >>>>> +        return -EIO;
> >>>>> +    }
> >>>>> +
> >>>>> +    dma_async_issue_pending(stats->channel);
> >>>>> +    return 0;
> >>>>> +}
> >>>>> +
> >>>>> +void mali_c55_stats_fill_buffer(struct mali_c55 *mali_c55,
> >>>>> +                enum mali_c55_config_spaces cfg_space)
> >>>>> +{
> >>>>> +    struct mali_c55_ctx *ctx = mali_c55_get_active_context(mali_c55);
> >>>>> +    struct mali_c55_stats *stats = &mali_c55->stats;
> >>>>> +    struct mali_c55_stats_buf *buf = NULL;
> >>>>> +    dma_addr_t src, dst;
> >>>>> +    int ret;
> >>>>> +
> >>>>> +    spin_lock(&stats->buffers.lock);
> >>>>> +    if (!list_empty(&stats->buffers.queue)) {
> >>>>> +        buf = list_first_entry(&stats->buffers.queue,
> >>>>> +                       struct mali_c55_stats_buf, queue);
> >>>>> +        list_del(&buf->queue);
> >>>>> +    }
> >>>>> +    spin_unlock(&stats->buffers.lock);
> >>>>> +
> >>>>> +    if (!buf)
> >>>>> +        return;
> >>>>> +
> >>>>> +    buf->vb.sequence = mali_c55->isp.frame_sequence;
> >>>>> +
> >>>>> +    /*
> >>>>> +     * There are infact two noncontiguous sections of the ISP's
> >>>> s/infact/in fact/
> >>>>
> >>>>> +     * memory space that hold statistics for 3a algorithms to use. A
> >>>> s/use. A/use: a/
> >>>>
> >>>>> +     * section in each config space and a global section holding
> >>>>> +     * histograms which is double buffered and so holds data for the
> >>>>> +     * last frame. We need to read both.
> >>>>> +     */
> >>>>> +    src = ctx->base + MALI_C55_REG_1024BIN_HIST;
> >>>>> +    dst = vb2_dma_contig_plane_dma_addr(&buf->vb.vb2_buf, 0);
> >>>>> +
> >>>>> +    ret = mali_c55_stats_dma_xfer(stats, src, dst, buf,
> >>>>> +                      MALI_C55_1024BIN_HIST_SIZE,
> >>>>> +                      mali_c55_stats_metering_complete);
> >>>>> +    if (ret)
> >>>>> +        goto err_fail_buffer;
> >>>>> +
> >>>>> +    src = ctx->base + metering_space_addrs[cfg_space];
> >>>>> +    dst += MALI_C55_1024BIN_HIST_SIZE;
> >>>>> +
> >>>>> +    ret = mali_c55_stats_dma_xfer(
> >>>>> +        stats, src, dst, buf,
> >>>>> +        sizeof(struct mali_c55_stats_buffer) - MALI_C55_1024BIN_HIST_SIZE,
> >>>>> +        mali_c55_stats_metering_complete);
> >>>>> +    if (ret) {
> >>>>> +        dmaengine_terminate_sync(stats->channel);
> >>>>> +        goto err_fail_buffer;
> >>>>> +    }
> >>>> I think you will need to terminate DMA transfers at stream off time.
> >>>>
> >>>>> +
> >>>>> +    return;
> >>>>> +
> >>>>> +err_fail_buffer:
> >>>>> +    vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
> >>>>> +}
> >>>>> +
> >>>>> +void mali_c55_unregister_stats(struct mali_c55 *mali_c55)
> >>>>> +{
> >>>>> +    struct mali_c55_stats *stats = &mali_c55->stats;
> >>>>> +
> >>>>> +    if (!video_is_registered(&stats->vdev))
> >>>>> +        return;
> >>>>> +
> >>>>> +    vb2_video_unregister_device(&stats->vdev);
> >>>>> +    media_entity_cleanup(&stats->vdev.entity);
> >>>>> +    dma_release_channel(stats->channel);
> >>>>> +    mutex_destroy(&stats->lock);
> >>>>> +}
> >>>>> +
> >>>>> +int mali_c55_register_stats(struct mali_c55 *mali_c55)
> >>>>> +{
> >>>>> +    struct mali_c55_stats *stats = &mali_c55->stats;
> >>>>> +    struct video_device *vdev = &stats->vdev;
> >>>>> +    struct vb2_queue *vb2q = &stats->queue;
> >>>>> +    dma_cap_mask_t mask;
> >>>>> +    int ret;
> >>>>> +
> >>>>> +    mutex_init(&stats->lock);
> >>>>> +    INIT_LIST_HEAD(&stats->buffers.queue);
> >>>>> +
> >>>>> +    dma_cap_zero(mask);
> >>>>> +    dma_cap_set(DMA_MEMCPY, mask);
> >>>>> +
> >>>>> +    stats->channel = dma_request_channel(mask, 0, NULL);
> >>>> Do we need a CPU fallback in case no DMA is available ?
> >>> Yes, actually.
> >>>
> >>>> I'm still very curious to know how long it takes to perform the DMA
> >>>> transfer, compared to copying the data with the CPU, and especially
> >>>> compared to the frame duration.
> >>> On my list of things to test and report :)
> >> Looking forward to it :-)
> >>
> >>>>> +    if (!stats->channel) {
> >>>>> +        ret = -ENODEV;
> >>>>> +        goto err_destroy_mutex;
> >>>>> +    }
> >>>>> +
> >>>>> +    stats->pad.flags = MEDIA_PAD_FL_SINK;
> >>>>> +    ret = media_entity_pads_init(&stats->vdev.entity, 1, &stats->pad);
> >>>>> +    if (ret)
> >>>>> +        goto err_release_dma_channel;
> >>>>> +
> >>>>> +    vb2q->type = V4L2_BUF_TYPE_META_CAPTURE;
> >>>>> +    vb2q->io_modes = VB2_MMAP | VB2_DMABUF;
> >>>>> +    vb2q->drv_priv = stats;
> >>>>> +    vb2q->mem_ops = &vb2_dma_contig_memops;
> >>>>> +    vb2q->ops = &mali_c55_stats_vb2_ops;
> >>>>> +    vb2q->buf_struct_size = sizeof(struct mali_c55_stats_buf);
> >>>>> +    vb2q->min_queued_buffers = 1;
> >>>>> +    vb2q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> >>>>> +    vb2q->lock = &stats->lock;
> >>>>> +    vb2q->dev = mali_c55->dev;
> >>>> That's not the right device. The device that performs the DMA operation
> >>>> is the DMA engine, and that's what you need to pass to vb2. Otherwise
> >>>> the DMA address returned by vb2_dma_contig_plane_dma_addr() will be
> >>>> mapped to the ISP device, not the DMA engine. In practice, if neither
> >>>> are behind an IOMMU, things will likely work, but when that's not the
> >>>> case, run into problems.
> >>>>
> >>>>> +
> >>>>> +    ret = vb2_queue_init(vb2q);
> >>>>> +    if (ret) {
> >>>>> +        dev_err(mali_c55->dev, "stats vb2 queue init failed\n");
> >>>>> +        goto err_cleanup_entity;
> >>>>> +    }
> >>>>> +
> >>>>> +    strscpy(stats->vdev.name, "mali-c55 3a stats", sizeof(stats->vdev.name));
> >>>> s/3a //
> >>>>
> >>>>> +    vdev->release = video_device_release_empty;
> >>>> That's never right. You should refcount the data structures to ensure
> >>>> proper lifetime management.
> >>>>
> >>>>> +    vdev->fops = &mali_c55_stats_v4l2_fops;
> >>>>> +    vdev->ioctl_ops = &mali_c55_stats_v4l2_ioctl_ops;
> >>>>> +    vdev->lock = &stats->lock;
> >>>>> +    vdev->v4l2_dev = &mali_c55->v4l2_dev;
> >>>>> +    vdev->queue = &stats->queue;
> >>>>> +    vdev->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING;
> >>>>> +    vdev->vfl_dir = VFL_DIR_RX;
> >>>>> +    video_set_drvdata(vdev, stats);
> >>>>> +
> >>>>> +    ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
> >>>>> +    if (ret) {
> >>>>> +        dev_err(mali_c55->dev,
> >>>>> +            "failed to register stats video device\n");
> >>>>> +        goto err_release_vb2q;
> >>>>> +    }
> >>>>> +
> >>>>> +    stats->mali_c55 = mali_c55;
> >>>>> +
> >>>>> +    return 0;
> >>>>> +
> >>>>> +err_release_vb2q:
> >>>>> +    vb2_queue_release(vb2q);
> >>>>> +err_cleanup_entity:
> >>>>> +    media_entity_cleanup(&stats->vdev.entity);
> >>>>> +err_release_dma_channel:
> >>>>> +    dma_release_channel(stats->channel);
> >>>>> +err_destroy_mutex:
> >>>>> +    mutex_destroy(&stats->lock);
> >>>>> +
> >>>>> +    return ret;
> >>>>> +}

-- 
Regards,

Laurent Pinchart