[PATCH v6 0/3] add IPU3 CIO2 CSI2 driver

Yong Zhi
2017-10-26T02:24:54Z

The CIO2 driver exposes V4L2, V4L2 sub-device and Media controller interfaces
to the user space.

This series was tested on Kaby Lake based platform with 2 sensor configurations,
media topology was pasted at end for reference.

Link to user space implementation:

<URL:https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master>

===========
= history =
===========

version 6:
- Replaced NUM_FORMATS with ARRAY_SIZE(formats). (Sakari)
- cio2_fbpt_rearrange(): skip fbpt re-arrange when not needed.
- cio2_resume(): move cio2_fbpt_rearrange() to cio2_suspend() to avoid mem alloc
at resume time.(Sakari)
- cio2_pci_probe(): call cio2_fbpt_init_dummy() before v4l2_device_register().(Sakari)
- Fixed checkpatch.pl reported checks with --strict option.
- Added static keyword to cio2_queue_init().
- cio2_buffer_done(): assign ns in variable initialization.(Sakari)
- cio2_notifier_bound(): create hard link between cio2 queues and CSI-2 port.(Sakari)
- cio2_notifier_unbount(): remove obsolete comments.(Sakari)
- cio2_notifier_init(): return error when no subdevs found.(Sakari)
- cio2_find_queue_by_sensor_node(): remove the function (Tomasz Figa)
- ipu3-cio2.h:
Change CIO2_QUEUES 4 to match IPU3 capability.(Sakari)
Remove un-used "other_entries" in struct cio2_fbpt_entry.(Sakari)
- pixfmt-srggb10-ipu3.rst: Changed tabularcolumns to 4.(Sakari)
- Enable x86 32bit build (Sakari)

version 5:
- cio2_vb2_start_streaming():
- cio2_vb2_stop_streaming(): removed redundant call of csi2 sub-dev for s_stream.
- cio2_vb2_buf_queue(): disabled interrupts for the duration of the buf queue,
to prevent this code from being pre-empted, as suggested by Tomasz Figa,
to mitigate the effects of race conditions around vb2 buf queuing code.
Switched to a finite loop to check for the first free buffer and errored
out, when there are no buffers available. Removed calls to vb2_plane_vaddr()
Maintain correct buf queued count, in error cases.
- Implemented system sleep pm ops to support cio2 driver suspend/resume.
- Made the v4l2 buffer and SOF event use sequence from same source.
- cio2_vb2_queue_setup(): remove validating pixelformat suggested by Tomasz
Figa.
- cio2_v4l2_g_fmt()/cio2_v4l2_s_fmt(): seperated formats on sub-dev and video
device suggested by Sakari Ailus.
- cio2_v4l2_try_fmt(): seperated video node and subdev format in the get_fmt,
try_fmt and set_fmt callbacks.
- cio2_queue_event_sof(): added comments suggested by Hans Verkuil
- cio2_queue_init(): re-ordered q->subdev_pads settings. remove 4 lines for
quantization init.
- cio2_subdev_get_fmt(): get colorspace/xfer_func/ycbcr_enc/quantization
from sensor suggested by Hans Verkuil.
- cio2_fbpt_entry_init_buf(): stored offset of the first sg_list entry to
remove calls to vb2_plane_vaddr().
- cio2_subdev_open(): added new callback to intialize the try format.
- cio2_subdev_video_ops(): removed empty implementation suggested by Sakari Ailus.
- cio2_notifier_init(): added fwnode binding support for subdevices using
v4l2_async_notifier_parse_fwnode_endpoints()
Patch series v15 Unified fwnode endpoint parser, async sub-device notifier
support, N9 flash DTS is needed for the fwnode binding code to compile.
https://www.mail-archive.com/linux-media@vger.kernel.org/msg120239.html
This also requires the following patch (v1) for the fwnode binding to work
https://patchwork.kernel.org/patch/9986445/
- cio2_notifier_complete(): removed redundant call of
fwnode_graph_get_remote_endpoint() and fwnode_graph_parse_endpoint().
- Switched to Multi Plane APIs suggested by Tomasz Figa.
User space changes supporting multi plane APIs can be found here
https://chromium-review.googlesource.com/c/chromiumos/platform/arc-camera/+/683802
- ipu3-cio.h: moved macros out of struct cio2_fbpt_entry suggested by Hans Verkuil.
- cio2_hw_mbus_to_mipicode(): replaced with cio2_find_format().
- cio2_pci_probe(): cleaned up goto logic on error conditions suggested by Tomasz Figa.
- Fixed v4l2_compliance test failures
added 3 dummy function to pass v4l2_compliance test.
- Extended format example in pixfmt-srggb10-ipu3.rst to show DMA word boundary.

version 4:
- add cio2_video_link_validate() for video entity suggested by Sakari Ailus
- cio2_notifier_complete(): fix comments suggested by Sakari Ailus
- cio2_vb2_buf_queue(): fix the forever loop suggested by Tomasz Figa
- cio2_v4l2_querycap(): use vdev device_caps commented by Hans Verkuil
- cio2_vb2_buf_init(): allocate LOP table per page suggested by Tomasz Figa
- cio2_hw_init(): call cio2_csi2_calc_timing() earlier suggested by Tomasz Figa
- cio2_csi2_calc_timing(): add defalt settings for rx term/settle
- cio2_vb2_queue_setup(): remove num_planes checking suggested by Tomasz Figa
- cio2_buffer_done(): remove setting b->vbb.flags to V4L2_BUF_FLAG_DONE and
memset of vbb.timecode, also move vb2_set_plane_payload() to
cio2_vb2_buf_queue() suggested by Sakari Ailus
- cio2_queue_init(): export VB2_DMABUF io_modes suggested by Tomasz Figa
- cio2_vb2_return_all_buffers(): remove state from param list
suggested by Tomasz Figa
- cio2_vb2_buf_queue(): use vb2_is_streaming() instead of
vb2_start_streaming_called() suggested by Tomasz Figa
- cio2_pci_probe(): replace hard-coded linux driver version
suggested by Hans Verkuil
- ipu3-cio2.h: re-order the reg macros suggested by Sakari Ailus
- ipu3-cio2.h: add inline vb2q_to_cio2_queue() suggested by Sakari Ailus
- ipu3-cio2.h: add comments for CIO2_INT_IOC suggested by Tomasz Figa
- ipu3-cio2.h: adjust PBM watermark threshold from 53 to 48 (internal bugfix)
- run v4l2_compliance suggested by Hans Verkuil

Todo list:

- fix possible racy code in cio2_vb2_buf_queue()
- fix v4l2_compliance test failure
- switch to v4l2_pix_format_mplane API if future needs arise

version 3:
- remove cio2_set_power().
- replace dma_alloc_noncoherent() with dma_alloc_coherent().
- apply ffs tricks at possible places.
- change sensor_vc to local variable.
- move ktime_get_ns() a little earlier in the calling order.
- fix multiple assignments(I.e a = b =c)
- define CIO2_PAGE_SIZE for CIO2 PAGE_SIZE, SENSOR_VIR_CH_DFLT for default sensor virtual ch.
- rework cio2_csi2_calc_timing().
- update v4l2 async subdev field name from match.fwnode.fwn
to match.fwnode.fwnode.
- cherry-pick internal fix for triggering different irq on SOF and EOF.
- return -ENOMEM for vb2_dma_sg_plane_desc() in cio2_vb2_buf_init().
- add cio2_link_validate() placeholder for vdev.

version 2:
- remove all explicit DMA flush operations
- change dma_free_noncoherent() to dma_free_coherent()
- remove cio2_hw_mipi_lanes()
- replace v4l2_g_ext_ctrls() with v4l2_ctrl_g_ctrl()
in cio2_csi2_calc_timing().
- use ffs() to iterate the port_status in cio2_irq()
- add static inline file_to_cio2_queue() function
- comment dma_wmb(), cio2_rx_timing() and few other places
- use ktime_get_ns() for vb2_buf.timestamp in cio2_buffer_done()
- use of SET_RUNTIME_PM_OPS() macro for cio2_pm_ops
- use BIT() macro for bit difinitions
- remove un-used macros such as CIO2_QUEUE_WIDTH() in ipu3-cio2.h
- move the MODULE_AUTHOR() to the end of the file
- change file path to drivers/media/pci/intel/ipu3

version 1:
- Initial submission

Media device topology:

localhost bin # ./media-ctl -d /dev/media0 -p
Media controller API version 4.14.0

Media device information
------------------------
driver ipu3-cio2
model Intel IPU3 CIO2
serial
bus info PCI:0000:00:14.3
hw revision 0x0
driver version 4.14.0

Device topology
- entity 1: ipu3-csi2:0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev0
pad0: Sink
[fmt:SGRBG10_1X10/1936x1096 field:none]
<- "ov13858 8-0010":0 []
pad1: Source
[fmt:SGRBG10_1X10/1936x1096 field:none]
-> "ipu3-cio2:0":0 [ENABLED,IMMUTABLE]

- entity 4: ipu3-cio2:0 (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video0
pad0: Sink
<- "ipu3-csi2:0":1 [ENABLED,IMMUTABLE]

- entity 10: ipu3-csi2:1 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev1
pad0: Sink
[fmt:SGRBG10_1X10/1936x1096 field:none]
<- "ov5670 10-0036":0 []
pad1: Source
[fmt:SGRBG10_1X10/1936x1096 field:none]
-> "ipu3-cio2:1":0 [ENABLED,IMMUTABLE]

- entity 13: ipu3-cio2:1 (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video1
pad0: Sink
<- "ipu3-csi2:1":1 [ENABLED,IMMUTABLE]

- entity 19: ipu3-csi2:2 (2 pads, 1 link)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev2
pad0: Sink
pad1: Source
[fmt:SGRBG10_1X10/1936x1096 field:none]
-> "ipu3-cio2:2":0 [ENABLED,IMMUTABLE]

- entity 22: ipu3-cio2:2 (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video2
pad0: Sink
<- "ipu3-csi2:2":1 [ENABLED,IMMUTABLE]

- entity 28: ipu3-csi2:3 (2 pads, 1 link)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev3
pad0: Sink
pad1: Source
[fmt:SGRBG10_1X10/1936x1096 field:none]
-> "ipu3-cio2:3":0 [ENABLED,IMMUTABLE]

- entity 31: ipu3-cio2:3 (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video3
pad0: Sink
<- "ipu3-csi2:3":1 [ENABLED,IMMUTABLE]

- entity 37: ov13858 8-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev4
pad0: Source
[fmt:SGRBG10_1X10/4224x3136 field:none]
-> "ipu3-csi2:0":0 []

- entity 39: dw9714 8-000c (0 pad, 0 link)
type V4L2 subdev subtype Lens flags 0
device node name /dev/v4l-subdev5

- entity 40: ov5670 10-0036 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev6
pad0: Source
[fmt:SGRBG10_1X10/2592x1944 field:none]
-> "ipu3-csi2:1":0 []

localhost bin # ./v4l2-compliance -d /dev/video0
v4l2-compliance SHA : not available

Driver Info:
Driver name : ipu3-cio2
Card type : Intel IPU3 CIO2
Bus info : PCI:0000:00:14.3
Driver version: 4.14.0
Capabilities : 0x84201000
Video Capture Multiplanar
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04201000
Video Capture Multiplanar
Streaming
Extended Pix Format

Compliance test for device /dev/video0 (not using libv4l2):

Required ioctls:
test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
test second video open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK

Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_ENUMAUDOUT: OK (Not Supported)
test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
test VIDIOC_G/S_EDID: OK (Not Supported)

Test input 0:

Control ioctls:
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
Standard Controls: 0 Private Controls: 0

Format ioctls:
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
test VIDIOC_G/S_PARM: OK (Not Supported)
test VIDIOC_G_FBUF: OK (Not Supported)
test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK

Codec ioctls:
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
test VIDIOC_G_ENC_INDEX: OK (Not Supported)
test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
test VIDIOC_EXPBUF: OK

Test input 0:

Total: 43, Succeeded: 43, Failed: 0, Warnings: 0

Note:

Same results with other 3 video nodes.

Running v4l2_compliance test with -f option will fail, we can only run
stream test with the help of media controller to link, configure and enable the
sub-dev/pads first.

Yong Zhi (3):
videodev2.h, v4l2-ioctl: add IPU3 raw10 color format
doc-rst: add IPU3 raw10 bayer pixel format definitions
intel-ipu3: cio2: Add new MIPI-CSI2 driver

Documentation/media/uapi/v4l/pixfmt-rgb.rst | 1 +
.../media/uapi/v4l/pixfmt-srggb10-ipu3.rst | 335 ++++
drivers/media/pci/Kconfig | 2 +
drivers/media/pci/Makefile | 3 +-
drivers/media/pci/intel/Makefile | 5 +
drivers/media/pci/intel/ipu3/Kconfig | 19 +
drivers/media/pci/intel/ipu3/Makefile | 1 +
drivers/media/pci/intel/ipu3/ipu3-cio2.c | 2014 ++++++++++++++++++++
drivers/media/pci/intel/ipu3/ipu3-cio2.h | 449 +++++
drivers/media/v4l2-core/v4l2-ioctl.c | 4 +
include/uapi/linux/videodev2.h | 6 +
11 files changed, 2838 insertions(+), 1 deletion(-)
create mode 100644 Documentation/media/uapi/v4l/pixfmt-srggb10-ipu3.rst
create mode 100644 drivers/media/pci/intel/Makefile
create mode 100644 drivers/media/pci/intel/ipu3/Kconfig
create mode 100644 drivers/media/pci/intel/ipu3/Makefile
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-cio2.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-cio2.h

--
2.7.4