[PATCH v1 04/11] media: Documentation: Add NXP neoisp driver documentation

Antoine Bouyer
2026-04-13T16:04:42Z

Document the NXP neoisp driver both in the admin-guide for neoisp IP
description, and in the userpace-api for neoisp interface description.

Signed-off-by: Antoine Bouyer <antoine.bouyer@nxp.com>
---
.../admin-guide/media/nxp-neoisp-diagram.dot | 22 +++
.../admin-guide/media/nxp-neoisp.dot | 16 ++
.../admin-guide/media/nxp-neoisp.rst | 179 ++++++++++++++++++
.../admin-guide/media/v4l-drivers.rst | 1 +
.../userspace-api/media/v4l/meta-formats.rst | 1 +
.../media/v4l/metafmt-nxp-neoisp.rst | 70 +++++++
6 files changed, 289 insertions(+)
create mode 100644 Documentation/admin-guide/media/nxp-neoisp-diagram.dot
create mode 100644 Documentation/admin-guide/media/nxp-neoisp.dot
create mode 100644 Documentation/admin-guide/media/nxp-neoisp.rst
create mode 100644 Documentation/userspace-api/media/v4l/metafmt-nxp-neoisp.rst

diff --git a/Documentation/admin-guide/media/nxp-neoisp-diagram.dot b/Documentation/admin-guide/media/nxp-neoisp-diagram.dot
new file mode 100644
index 000000000000..ac0d950bc1dc
--- /dev/null
+++ b/Documentation/admin-guide/media/nxp-neoisp-diagram.dot
@@ -0,0 +1,22 @@
+digraph G {
+ rankdir = "LR";
+ node [shape=rect];
+ splines = ortho;
+ label = "Neoisp pipeline diagram";
+ {In0, In1 } -> HC -> {HDR_decomp0, HDR_decomp1};
+ HDR_decomp0 -> OBWB0 -> HDR_merge;
+ HDR_decomp1 -> OBWB1 -> HDR_merge;
+ HDR_merge -> RGBIR -> IR_compression;
+ RGBIR -> Statistics;
+ RGBIR -> OBWB2 ->BNR -> Vignetting -> ColorTemp;
+ Vignetting -> Demosaic -> RGB2YUV -> DRC -> AF;
+ DRC-> NR -> EE -> DF -> Gamma -> Packetizer;
+ DRC -> DMAP -> DF[weight=2];
+ DRC -> CCONV -> CAS -> Gamma;
+ {rank = "same"; RGBIR, DRC}
+ {rank = "same"; AF, NR, DMAP}
+ {rank = "same"; IR_compression, Vignetting}
+ IR_compression -> AXIOutDMA;
+ Packetizer -> AXIOutDMA [weight=3];
+ AXIOutDMA -> {"Out0", "Out1"}
+}
diff --git a/Documentation/admin-guide/media/nxp-neoisp.dot b/Documentation/admin-guide/media/nxp-neoisp.dot
new file mode 100644
index 000000000000..abcc2dc9bb55
--- /dev/null
+++ b/Documentation/admin-guide/media/nxp-neoisp.dot
@@ -0,0 +1,16 @@
+digraph board {
+ rankdir=TB
+ n00000001 [label="{{<port0> 0 | <port1> 1 | <port2> 2} | neoisp\n | {<port3> 3 | <port4> 4 | <port5> 5}}", shape=Mrecord, style=filled, fillcolor=green]
+ n00000001:port3 -> n00000020 [style=dashed]
+ n00000001:port4 -> n00000022 [style=dashed]
+ n00000001:port5 -> n00000024 [style=dashed]
+ n0000000a [label="neoisp-input0\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
+ n0000000a -> n00000001:port0 [style=bold]
+ n00000010 [label="neoisp-input1\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
+ n00000010 -> n00000001:port1 [style=dashed]
+ n00000016 [label="neoisp-params\n/dev/video2", shape=box, style=filled, fillcolor=yellow]
+ n00000016 -> n00000001:port2 [style=dashed]
+ n00000020 [label="neoisp-frame\n/dev/video3", shape=box, style=filled, fillcolor=yellow]
+ n00000022 [label="neoisp-ir\n/dev/video4", shape=box, style=filled, fillcolor=yellow]
+ n00000024 [label="neoisp-stats\n/dev/video5", shape=box, style=filled, fillcolor=yellow]
+}
diff --git a/Documentation/admin-guide/media/nxp-neoisp.rst b/Documentation/admin-guide/media/nxp-neoisp.rst
new file mode 100644
index 000000000000..64876612065d
--- /dev/null
+++ b/Documentation/admin-guide/media/nxp-neoisp.rst
@@ -0,0 +1,179 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=======================================
+NXP Neo Image Signal Processor (neoisp)
+=======================================
+
+Introduction
+============
+
+Neoisp performs a set of image processing tasks on the RAW camera stream, where
+the input camera stream and Neoisp processed output image are stored in DDR or
+any system memory fast enough to keep up with Neoisp processing.
+The overall Neoisp operation is frame based, that is 1 complete image frame is
+read and output pixel by pixel, line by line wise.
+
+Revisions
+=========
+
+NXP Neoisp driver supports the Neoisp V2 hw revision, used in i.MX95 Soc family.
+
+Neoisp hardware
+===============
+
+The Neoisp registers and pipeline processing are documented in the NXP Image
+Signal Processor Specification document (under NDA).
+
+The NEO pipeline block diagram is shown below:
+
+.. kernel-figure:: nxp-neoisp-diagram.dot
+ :alt: Diagram of neoisp pipeline
+ :align: center
+
+It is composed by the following HW blocks:
+
+- a Head Color (HC) selection block,
+- two HDR decompression blocks, one for each input,
+- three Optical Black correction and White Balance (OBWB) blocks at different
+ stages in the pipeline,
+- a HDR merge block for HDR image capture from the 2 input lines,
+- a RGB-IR to RGGB converter,
+- a Bayer Noise Reduction (BNR) block,
+- a Vignetting block, aka Lens Shading Correction (LSC),
+- a Demosaic block for RAW image conversion to RGB,
+- a RGB Color Correction Matrix (CCM) and Color Space Converter aka CSC or
+ RGB2YUV,
+- a Dynamic Range Compression (DRC) block,
+- the Denoising pipeline (composed by multiple blocks for Noise Reduction, Edge
+ Enhancement, Gamma Compensation, etc),
+- a Packetizer used for UV sub-sampling or RGB packing.
+
+All these blocks are controlled by SW through registers. Some of these
+registers are accessible by the uAPI, so that usespace application and Image
+Processing Algorithms (IPA) can configure them through the parameters buffers.
+
+Neoisp driver
+=============
+
+Neoisp driver is located under drivers/media/platform/nxp/neoisp.
+It uses the `V4L2 API` and the `V4L2 subdev API` to register capture and output
+video devices in addition to a subdevice for neoisp that connects the video
+devices into a media graph realized using the `Media Controller (MC) API`.
+
+Driver provides a core registration API that allows registering the neoisp core
+media devices into an external media graph. In this case, the neoisp is fully
+integrated into a main media graph which could include other subdevices such as
+camera sensors, crossbar routing, etc. It also provides a module parameter to
+allow running in standalone mode. In such case, the neoisp registers its media
+devices into its own media graph.
+
+The media topology registered by Neoisp driver is represented below:
+
+.. kernel-figure:: nxp-neoisp.dot
+ :alt: Diagram of neoisp media device topology
+ :align: center
+
+
+The neoisp registers the following video device nodes:
+
+- neoisp-input0: output device for RAW frames to be submitted to the ISP for processing.
+- neoisp-input1: output device for RAW frames short capture in HDR merge mode.
+- neoisp-params: output meta device for parameters provided by user space 3A algorithms.
+- neoisp-frame: capture device for RGB/YUV pixels of the processed images.
+- neoisp-ir: capture device for the infra-red pixels of the processed images.
+- neoisp-stats: capture meta device for generated image statistics for user space 3A algorithms.
+
+neoisp-input0, neoisp-input1
+----------------------------
+
+Images to be processed by Neoisp are queued to the neoisp-input0 (and
+neoisp-input1 when in HDR mode) output device nodes. Supported image formats
+as input to the ISP are:
+
+- Raw bayer formats:
+
+ - 8 bits raw (V4L2_PIX_FMT_SRGGB8, V4L2_PIX_FMT_SBGGR8, V4L2_PIX_FMT_SGBRG8,
+ V4L2_PIX_FMT_SGRBG8)
+ - 10 bits raw (V4L2_PIX_FMT_SRGGB10, V4L2_PIX_FMT_SBGGR10,
+ V4L2_PIX_FMT_SGBRG10, V4L2_PIX_FMT_SGRBG10)
+ - 12 bits raw (V4L2_PIX_FMT_SRGGB12, V4L2_PIX_FMT_SBGGR12,
+ V4L2_PIX_FMT_SGBRG12, V4L2_PIX_FMT_SGRBG12)
+ - 14 bits raw (V4L2_PIX_FMT_SRGGB14, V4L2_PIX_FMT_SBGGR14,
+ V4L2_PIX_FMT_SGBRG14, V4L2_PIX_FMT_SGRBG14)
+ - 16 bits raw (V4L2_PIX_FMT_SRGGB16, V4L2_PIX_FMT_SBGGR16,
+ V4L2_PIX_FMT_SGBRG16, V4L2_PIX_FMT_SGRBG16)
+
+- Monochrome formats:
+
+ - 8 bits Monochrome (V4L2_PIX_FMT_GREY)
+ - 10 bits Monochrome (V4L2_PIX_FMT_Y10)
+ - 12 bits Monochrome (V4L2_PIX_FMT_Y12)
+ - 14 bits Monochrome (V4L2_PIX_FMT_Y14)
+ - 16 bits Monochrome (V4L2_PIX_FMT_Y16)
+
+.. note::
+ RGBIr camera sensors are supported as well, and can be used through user
+ space activation of the IR block.
+
+.. note::
+ neoisp-input1 link is mutable and should be enabled in case a short capture
+ image buffer is provided to the ISP for HDR merge.
+
+.. _neoisp_params:
+
+neoisp-params
+-------------
+
+The neoisp-params output meta device receives configuration data to be written
+to Neoisp registers and internal memory for desired input image processing.
+This v4l2 device accepts the generic `extensible parameters` format.
+
+In this format, the parameters buffer is defined by the generic
+:c:type:`v4l2_isp_params_buffer`, and userspace should set
+:ref:`V4L2_META_FMT_NEO_ISP_EXT_PARAMS <v4l2-meta-fmt-neo-isp-ext-params>`
+as dataformat.
+
+When the related media link is disabled, the image decoding will be done based
+on the default parameters of the ISP.
+
+neoisp-frame
+------------
+
+The capture device writes to memory the RGB or YUV pixels of the image processed
+by Neoisp when the media link is enabled. If the related media link is disabled,
+the processed image will be written to dummy buffer and not delivered to the
+neoisp-frame video device node.
+
+neoisp-ir
+---------
+
+The capture device writes to memory the RGBIr pixels of the image processed by
+Neoisp when the media link is enabled. If the related media link is disabled,
+the processed image will not be delivered to the neoisp-ir video device node.
+
+.. _neoisp_stats:
+
+neoisp-stats
+------------
+
+The neoisp-stats capture meta device provides statistics data generated by
+Neoisp hardware while processing the input image. This v4l2 device accepts the
+`extensible statistics` format.
+
+In this format, the statistics buffer is defined by the generic
+:c:type:`v4l2_isp_stats_buffer`, and userspace should set
+:ref:`V4L2_META_FMT_NEO_ISP_EXT_STATS <v4l2-meta-fmt-neo-isp-ext-stats>` as
+dataformat.
+
+When the related media link is disabled, the decoding statistics will not be
+delivered to the neoisp-stats meta device node.
+
+Control
+=======
+
+To support additional neoisp hardware revisions, the read-only bitmask control
+`V4L2_CID_NEOISP_SUPPORTED_PARAMS_BLOCKS` can be used to query the list of
+supported blocks. Each bit represents the availability of the corresponding
+entry from the :c:type:`neoisp_param_block_type_e` enum. In current driver
+version, default and max values represent the blocks supported by the i.MX95
+SoC.
diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst
index d31da8e0a54f..445269f87b17 100644
--- a/Documentation/admin-guide/media/v4l-drivers.rst
+++ b/Documentation/admin-guide/media/v4l-drivers.rst
@@ -21,6 +21,7 @@ Video4Linux (V4L) driver-specific documentation
ivtv
mali-c55
mgb4
+ nxp-neoisp
omap3isp
philips
qcom_camss
diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index 3e0cab153f0a..46268d955d3a 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
@@ -18,6 +18,7 @@ These formats are used for the :ref:`metadata` interface only.
metafmt-d4xx
metafmt-generic
metafmt-intel-ipu3
+ metafmt-nxp-neoisp
metafmt-pisp-be
metafmt-pisp-fe
metafmt-rkisp1
diff --git a/Documentation/userspace-api/media/v4l/metafmt-nxp-neoisp.rst b/Documentation/userspace-api/media/v4l/metafmt-nxp-neoisp.rst
new file mode 100644
index 000000000000..c412b1595205
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/metafmt-nxp-neoisp.rst
@@ -0,0 +1,70 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+*****************************************
+V4L2_META_FMT_NEO_ISP_EXT_PARAMS ('nnep')
+*****************************************
+
+The Neoisp image signal processor is configured by userspace through a buffer
+of parameters to :ref:`neoisp-params <neoisp_params>` output device node using
+the :c:type:`v4l2_meta_format` interface.
+
+The parameters buffer uses the generic `extensible parameters` configuration
+format.
+
+.. _v4l2-meta-fmt-neo-isp-ext-params:
+
+Extensible parameters configuration format
+==========================================
+
+When using the `extensible parameters` configuration format, parameters are
+passed to the :ref:`neoisp-params <neoisp_params>` metadata output video node
+using the `V4L2_META_FMT_NEO_ISP_EXT_PARAMS` meta format.
+
+The buffer contains a single instance of the C structure
+:c:type:`v4l2_isp_params_buffer` defined in ``v4l2-isp.h``. The
+:c:type:`v4l2_isp_params_buffer` structure is designed to allow userspace to
+populate the data buffer with only the configuration data for the Neoisp blocks
+it intends to configure. The extensible parameters format design allows
+developers to define new block types to support new configuration parameters,
+and defines a versioning scheme so that it can be extended and versioned
+without breaking compatibility with existing applications.
+
+****************************************
+V4L2_META_FMT_NEO_ISP_EXT_STATS ('nnes')
+****************************************
+
+The Neoisp image signal processor generates statistics data while processing an
+input image. These statistics are captured in a buffer and provided to
+userspace through the :ref:`neoisp-stats <neoisp_stats>` capture video node
+using the :c:type:`v4l2_meta_format` interface. The statistics data are
+processed by userspace application to produce the next Neoisp parameters.
+
+The statistics buffer uses the generic `extensible statistics` format.
+
+.. _v4l2-meta-fmt-neo-isp-ext-stats:
+
+Extensible statistics format
+============================
+
+When using the `extensible statistics` format, the statistics buffer is passed
+from the :ref:`neoisp-stats <neoisp_stats>` metadata capture video node using
+the `V4L2_META_FMT_NEO_ISP_EXT_STATS` meta format.
+
+The buffer contains a single instance of the C structure
+:c:type:`v4l2_isp_stats_buffer` defined in ``v4l2-isp.h``. The
+:c:type:`v4l2_isp_stats_buffer` structure is designed to allow future Neoisp
+driver versions to populate the statistics buffer with future blocks
+statistics, and defines a versioning scheme so that it can be extended and
+versioned without breaking compatibility with existing applications.
+
+**********************
+Neoisp uAPI data types
+**********************
+
+This chapter describes the data types exposed to userspace by Neoisp driver.
+
+Some structure members are in a fixed-point format, in this case the related
+description will be ended by a fixed-point definition between parenthesis.
+
+.. kernel-doc:: include/uapi/linux/media/nxp/nxp_neoisp.h
+
--
2.51.0