Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

[Laurent Pinchart <laurent.pinchart@ideasonboard.com>]

Hello Wentong,

On Fri, Feb 17, 2023 at 06:20:10AM +0000, Wu, Wentong wrote:
> On Wednesday, February 15, 2023 5:43 PM, Laurent Pinchart wrote:
> > On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> > > Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
> > > companion chip designed to provide secure and low power vision
> > > capability to IA platforms. IVSC is available in existing commercial
> > > platforms from multiple OEMs.
> > >
> > > The primary use case of IVSC is to bring in context awareness. IVSC
> > > interfaces directly with the platform main camera sensor via a CSI-2
> > > link and processes the image data with the embedded AI engine. The
> > > detected events are sent over I2C to ISH (Intel Sensor Hub) for
> > > additional data fusion from multiple sensors. The fusion results are
> > > used to implement advanced use cases like:
> > >  - Face detection to unlock screen
> > >  - Detect user presence to manage backlight setting or waking up
> > > system
> > 
> > Do you have plan to support these features in the ivsc driver in the future ?
> 
> Not sure, but the first step is to upstream this driver.

Sure, no problem.

> > > Since the Image Processing Unit(IPU) used on the host processor needs
> > > to configure the CSI-2 link in normal camera usages, the CSI-2 link
> > > and camera sensor can only be used in mutually-exclusive ways by host
> > > IPU and IVSC. By default the IVSC owns the CSI-2 link and camera
> > > sensor. The IPU driver can take ownership of the CSI-2 link and camera
> > > sensor using interfaces provided by this IVSC driver.
> > >
> > > Switching ownership requires an interface with two different hardware
> > > modules inside IVSC. The software interface to these modules is via
> > > Intel MEI (The Intel Management Engine) commands. These two hardware
> > > modules have two different MEI UUIDs to enumerate. These hardware
> > > modules are:
> > >  - ACE (Algorithm Context Engine): This module is for algorithm
> > > computing when IVSC owns camera sensor. Also ACE module controls
> > > camera sensor's ownership. This hardware module is used to set ownership of
> > > camera sensor.
> > >  - CSI (Camera Serial Interface): This module is used to route camera
> > > sensor data either to IVSC or to host for IPU driver and application.
> > >
> > > IVSC also provides a privacy mode. When privacy mode is turned on,
> > > camera sensor can't be used. This means that both ACE and host IPU
> > > can't get image data. And when this mode is turned on, host IPU driver
> > > is informed via a registered callback, so that user can be notified.
> > 
> > How does the privacy mode work, and how can the user trust that the closed-
> > source IVSC and IME firmwares will honour the privacy settings ?
> 
> No camera data will be allowed to go through IVSC, and then there will
> be no data on IVSC CSI transmitter side. 

But how can I be sure that the IVSC will not use the camera behind my
back, if it's all controlled through a closed-source firmware ?

> > > In summary, to acquire ownership of camera by IPU driver, first ACE
> > > module needs to be informed of ownership and then to setup MIPI CSI-2
> > > link for the camera sensor and IPU.
> > >
> > > Implementation:
> > > There are two different drivers to handle ACE and CSI hardware modules
> > > inside IVSC.
> > >  - mei_csi: MEI client driver to send commands and receive notifications from CSI module.
> > >  - mei_ace: MEI client driver to send commands and get status from ACE module.
> > > Interface is exposed via ivsc.h to acquire and release camera sensor and
> > > CSI-2 link.
> > 
> > Do I understand correctly, from your diagram below, that the communication
> > between the IME and IVSC goes through SPI ?
> > 
> > > Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
> > > -----------------------------------------------------------------------------
> > > | Host Processor                                                            |
> > > |                                                                           |
> > > |       -----------------       -----------------       ---------------     |
> > > |       |               |       |               |       |             | I2C |
> > > |       |      IPU      |       |      ISH      |       |camera driver|--|  |
> > > |       |               |       |               |       |             |  |  |
> > > |       -----------------       -----------------       ---------------  |  |
> > > |               |                       |                      |         |  |
> > > |               |                       |               ---------------  |  |
> > > |               |                       |               |             |  |  |
> > > |               |                       |               | IVSC driver |  |  |
> > > |               |                       |               |             |  |  |
> > > |               |                       |               ---------------  |  |
> > > |               |                       |                      |         |  |
> > > ----------------|-----------------------|----------------------|---------|---
> > >                 | CSI                   | I2C                  |SPI      |
> > >                 |                       |                      |         |
> > > ----------------|-----------------------|----------------      |         |
> > > | IVSC          |                                       |      |         |
> > > |               |                                       |      |         |
> > > |       -----------------       -----------------       |      |         |
> > > |       |               |       |               |       |      |         |
> > > |       |      CSI      |       |      ACE      |       |------|         |
> > > |       |               |       |               |       |                |
> > > |       -----------------       -----------------       |                |
> > > |               |                       | I2C           |                |
> > > ----------------|-----------------------|----------------                |
> > >                 | CSI                   |                                |
> > >                 |                       |                                |
> > >             --------------------------------                             |
> > >             |                              | I2C                         |
> > >             |         camera sensor        |-----------------------------|
> > >             |                              |
> > >             --------------------------------
> > >
> > > Wentong Wu (3):
> > >   media: pci: intel: ivsc: Add CSI submodule
> > >   media: pci: intel: ivsc: Add ACE submodule
> > >   media: pci: intel: ivsc: Add acquire/release API for ivsc
> > >
> > >  drivers/media/pci/Kconfig              |   1 +
> > >  drivers/media/pci/intel/Makefile       |   2 +
> > >  drivers/media/pci/intel/ivsc/Kconfig   |  12 +
> > >  drivers/media/pci/intel/ivsc/Makefile  |   7 +
> > >  drivers/media/pci/intel/ivsc/ivsc.c    |  84 +++++
> > >  drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
> > >  drivers/media/pci/intel/ivsc/mei_ace.h |> 36 ++
> > >  drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
> > >  drivers/media/pci/intel/ivsc/mei_csi.h |  60 ++++
> > >  include/linux/ivsc.h                   |  74 ++++
> > >  10 files changed, 1090 insertions(+)
> > >  create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
> > >  create mode 100644 drivers/media/pci/intel/ivsc/Makefile
> > >  create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
> > >  create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
> > >  create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
> > >  create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
> > >  create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
> > >  create mode 100644 include/linux/ivsc.h

-- 
Regards,

Laurent Pinchart