isp or soc-camera for image co-processors

isp or soc-camera for image co-processors

[Bhupesh SHARMA]

Hi Guennadi and Laurent,

We are now evaluating another ST platform that supports a image
co-processor between the camera sensor and the camera host (SoC).

The simple architecture diagram will be similar to one shown below
(for the sake of simplicity I show only a single sensor. At least 
two sensors can be supported by the co-processor):

----------------CSI-2 interface   	------------------------ CSI2 interface	------------------
|Camera sensor |<--------------->	|CSI2			CSI2	|<-------=------>	|SoC (ARM Based)	|
|    0         |				|serial		serial|			|			|
|		   |				|receiver	transmitter	|			|			|
|		   |CCI Interface		|				|			|			|
|	         |<--------------->	|CCI			CCI	|CCI/I2C Interface|			|
|              |				|master		slave	|<-------------->	|			|
----------------				|				|			|			|
						|				|SYNC signals	|			|
						|	         	ITU	|---------------> |			| 
						|              	CCIR	|Pixel CLK		|		   	|
						|		   Interface|--------------->	|          		|
						|	         		|ITU Data		|		  	|
						|              		|--------------->	|		   	|
						|Image	   video	|			|          		|
						|Processor    processing|			|		   	|
						|			logic	|			|			|
						-------------------------			-------------------

The co-processor supports a video progressing logic engine capable of
performing a variety of operations like image recovery, cropping, scaling,
gamma correction etc.

Now, evaluating the framework available for supporting for the camera
host, sensor and co-processor, I am wondering whether soc-camera(v4l2) can
support this complex design or something similar to the ISP driver written
for OMAP is the way forward.

Any pointers on the same and reference links to some documentation
will be highly appreciated.

Thanks for your help,
Regards,
Bhupesh
ST Microelectonics

Re: isp or soc-camera for image co-processors

[Laurent Pinchart]

Hi Bhupesh,

On Tuesday 01 March 2011 08:25:12 Bhupesh SHARMA wrote:
> Hi Guennadi and Laurent,
> 
> We are now evaluating another ST platform that supports a image
> co-processor between the camera sensor and the camera host (SoC).
> 
> The simple architecture diagram will be similar to one shown below
> (for the sake of simplicity I show only a single sensor. At least
> two sensors can be supported by the co-processor):

[snip] (as the ascii-art looks more like a Picasso painting with the quote 
characters)

> The co-processor supports a video progressing logic engine capable of
> performing a variety of operations like image recovery, cropping, scaling,
> gamma correction etc.
> 
> Now, evaluating the framework available for supporting for the camera
> host, sensor and co-processor, I am wondering whether soc-camera(v4l2) can
> support this complex design or something similar to the ISP driver written
> for OMAP is the way forward.

I think this can be a good candidate for the media controller API. It depends 
on how complex the co-processor is and what kind of processing it performs. I 
suppose there's no public datasheet.

You will probably need to enhance subdev registration, as I'm not aware of any 
existing use case such as yours where a chain of subdevs unknown to the host 
controller is connected to the host controller input.

> Any pointers on the same and reference links to some documentation
> will be highly appreciated.

-- 
Regards,

Laurent Pinchart

RE: isp or soc-camera for image co-processors

[Bhupesh SHARMA]

Hi Laurent,

> -----Original Message-----
> From: Laurent Pinchart [mailto:laurent.pinchart@ideasonboard.com]
> Sent: Tuesday, March 01, 2011 3:11 PM
> To: Bhupesh SHARMA
> Cc: Guennadi Liakhovetski; linux-media@vger.kernel.org
> Subject: Re: isp or soc-camera for image co-processors
> 
> Hi Bhupesh,
> 
> On Tuesday 01 March 2011 08:25:12 Bhupesh SHARMA wrote:
> > Hi Guennadi and Laurent,
> >
> > We are now evaluating another ST platform that supports a image
> > co-processor between the camera sensor and the camera host (SoC).
> >
> > The simple architecture diagram will be similar to one shown below
> > (for the sake of simplicity I show only a single sensor. At least
> > two sensors can be supported by the co-processor):
> 
> [snip] (as the ascii-art looks more like a Picasso painting with the
> quote
> characters)

:(
Despite my efforts to align it properly :)

> > The co-processor supports a video progressing logic engine capable of
> > performing a variety of operations like image recovery, cropping,
> scaling,
> > gamma correction etc.
> >
> > Now, evaluating the framework available for supporting for the camera
> > host, sensor and co-processor, I am wondering whether soc-
> camera(v4l2) can
> > support this complex design or something similar to the ISP driver
> written
> > for OMAP is the way forward.
> 
> I think this can be a good candidate for the media controller API. It
> depends
> on how complex the co-processor is and what kind of processing it
> performs. I
> suppose there's no public datasheet.
> 
> You will probably need to enhance subdev registration, as I'm not aware
> of any
> existing use case such as yours where a chain of subdevs unknown to the
> host
> controller is connected to the host controller input.

Could you please give me some documentation links for media controller API.
Are there are reference drivers that I can use for my study?
Unfortunately the data-sheet of the co-processor cannot be made public
as of yet.

Regards,
Bhupesh

Re: isp or soc-camera for image co-processors

[Laurent Pinchart]

Hi Bhupesh,

On Tuesday 01 March 2011 10:46:36 Bhupesh SHARMA wrote:
> On Tuesday, March 01, 2011 3:11 PM Laurent Pinchart wrote: 
> > On Tuesday 01 March 2011 08:25:12 Bhupesh SHARMA wrote:
> > > Hi Guennadi and Laurent,
> > > 
> > > We are now evaluating another ST platform that supports a image
> > > co-processor between the camera sensor and the camera host (SoC).
> > > 
> > > The simple architecture diagram will be similar to one shown below
> > > (for the sake of simplicity I show only a single sensor. At least
> > 
> > > two sensors can be supported by the co-processor):
> > [snip] (as the ascii-art looks more like a Picasso painting with the
> > quote
> > characters)
> :
> :(
> 
> Despite my efforts to align it properly :)

Try to configure your mailer to use spaces instead of tabs, or to make tabs 8 
spaces wide. It should then look good. Replies will usually mess the diagrams 
up though.

> > > The co-processor supports a video progressing logic engine capable of
> > > performing a variety of operations like image recovery, cropping,
> > > scaling, gamma correction etc.
> > > 
> > > Now, evaluating the framework available for supporting for the camera
> > > host, sensor and co-processor, I am wondering whether soc-camera(v4l2)
> > > can support this complex design or something similar to the ISP driver
> > > written for OMAP is the way forward.
> > 
> > I think this can be a good candidate for the media controller API. It
> > depends on how complex the co-processor is and what kind of processing it
> > performs. I suppose there's no public datasheet.
> > 
> > You will probably need to enhance subdev registration, as I'm not aware
> > of any existing use case such as yours where a chain of subdevs unknown to
> > the host controller is connected to the host controller input.
> 
> Could you please give me some documentation links for media controller API.

The media controller documentation is part of the V4L2 kernel documentation. 
You can find a compiled copy at http://www.ideasonboard.org/media/media/

The in-kernel API is documented in the kernel sources, in Documentation/media-
framework.txt

> Are there are reference drivers that I can use for my study?

The OMAP3 ISP driver.

> Unfortunately the data-sheet of the co-processor cannot be made public
> as of yet.

Can you publish a block diagram of the co-processor internals ?

-- 
Regards,

Laurent Pinchart

RE: isp or soc-camera for image co-processors

[Bhupesh SHARMA]

Hi Laurent,

> -----Original Message-----
> From: Laurent Pinchart [mailto:laurent.pinchart@ideasonboard.com]
> Sent: Tuesday, March 01, 2011 3:40 PM
> To: Bhupesh SHARMA
> Cc: Guennadi Liakhovetski; linux-media@vger.kernel.org
> Subject: Re: isp or soc-camera for image co-processors
> 
> Hi Bhupesh,
> 
> On Tuesday 01 March 2011 10:46:36 Bhupesh SHARMA wrote:
> > On Tuesday, March 01, 2011 3:11 PM Laurent Pinchart wrote:
> > > On Tuesday 01 March 2011 08:25:12 Bhupesh SHARMA wrote:
> > > > Hi Guennadi and Laurent,
> > > >
> > > > We are now evaluating another ST platform that supports a image
> > > > co-processor between the camera sensor and the camera host (SoC).
> > > >
> > > > The simple architecture diagram will be similar to one shown
> below
> > > > (for the sake of simplicity I show only a single sensor. At least
> > >
> > > > two sensors can be supported by the co-processor):
> > > [snip] (as the ascii-art looks more like a Picasso painting with
> the
> > > quote
> > > characters)
> > :
> > :(
> >
> > Despite my efforts to align it properly :)
> 
> Try to configure your mailer to use spaces instead of tabs, or to make
> tabs 8
> spaces wide. It should then look good. Replies will usually mess the
> diagrams
> up though.

Ok, I will try it :)

> > > > The co-processor supports a video progressing logic engine
> capable of
> > > > performing a variety of operations like image recovery, cropping,
> > > > scaling, gamma correction etc.
> > > >
> > > > Now, evaluating the framework available for supporting for the
> camera
> > > > host, sensor and co-processor, I am wondering whether soc-
> camera(v4l2)
> > > > can support this complex design or something similar to the ISP
> driver
> > > > written for OMAP is the way forward.
> > >
> > > I think this can be a good candidate for the media controller API.
> It
> > > depends on how complex the co-processor is and what kind of
> processing it
> > > performs. I suppose there's no public datasheet.
> > >
> > > You will probably need to enhance subdev registration, as I'm not
> aware
> > > of any existing use case such as yours where a chain of subdevs
> unknown to
> > > the host controller is connected to the host controller input.
> >
> > Could you please give me some documentation links for media
> controller API.
> 
> The media controller documentation is part of the V4L2 kernel
> documentation.
> You can find a compiled copy at
> http://www.ideasonboard.org/media/media/

Thanks, I will go through the same.

> The in-kernel API is documented in the kernel sources, in
> Documentation/media-
> framework.txt
> 
> > Are there are reference drivers that I can use for my study?
> 
> The OMAP3 ISP driver.

Thanks, I will go through the same.

> > Unfortunately the data-sheet of the co-processor cannot be made
> public
> > as of yet.
> 
> Can you publish a block diagram of the co-processor internals ?

I will check internally to see if I can send a block-diagram
of the co-processor internals to you. The internals seem similar to 
OMAP ISP (which I can see from the public TRM). However, our
co-processor doesn't have the circular buffer and MMU that ISP seem to
have (and some other features).

In the meantime I copy the features of the co-processor here for your review:

Input / Output interfaces of co-processor:
==========================================
- Sensor interfaces: 2 x MIPI CSI-2 receivers (1 x dual-lane up to 1.6 Gbit/s
 and 1 x single lane up to 800 Mbit/s)
- Host interface: MIPI CSI-2 dual lane transmitter (up to 1.6 Gbit/s) or ITU
 (8-bit CCIR interface, up to 100 MHz) - all with independent variable
 transmitter clock (PLL)
- Control interface: CCI (up to 400 kHz) or SPI

Sensor support:
===============
- Supports two MIPI compliant sensors of up to 8 Megapixel resolution
 (one sensor streaming at a time)
- Support for auto-focus (AF), extended depth of field (EDOF) and wide dynamic
 range (WDR)sensors 

Internal Features:
==================
- Versatile clock manager and internal buffer to accommodate a wide range of data rates
  between sensors and the coprocessor and between the coprocessor and the host.
- Synchronized flash gun control with red-eye reduction (pre-flash and main-flash strobes) for
  high-power LED or Xenon strobe light
- Low power standby mode
- Two video pipes (enables concurrent viewfinding and video/stills capture)
- Face detection and tracking algorithm
- Video stabilization
- Adaptive 4-channel lens shading and barrel distortion correction
- Statistics processor for advanced automatic exposure and white balance
- Automatic contrast stretch
- Nine-zone auto-focus with flexible actuator driver
- Digital zoom: smooth 16x down-scale capability and 4x up-scale capability
- Advanced noise and defect filtering
- Color reconstruction
- Adaptive color correction matrix
- Sharpness enhancement
- Programmable gamma correction
- Lighting frequency detection and automatic flicker reduction
- Image rotation/mirroring/flip for the viewfinder (up to 480 x 360)
- Special effects

Data Formats:
=============
- Output formats: JPEG, YUV4:2:2, YUV4:2:0, Planar YUV4:2:0 (up to 480 x 360), RGB888,
  RGB565, RGB444 
- JPEG compression with programmable quantization matrix and target file size

Regards,
Bhupesh

Re: isp or soc-camera for image co-processors

[Laurent Pinchart]

Hi Bhupesh,

On Tuesday 01 March 2011 12:04:34 Bhupesh SHARMA wrote:
> On Tuesday, March 01, 2011 3:40 PM Laurent Pinchart wrote: > > On Tuesday 01 
March 2011 10:46:36 Bhupesh SHARMA wrote:
> > > On Tuesday, March 01, 2011 3:11 PM Laurent Pinchart wrote:
> > > > On Tuesday 01 March 2011 08:25:12 Bhupesh SHARMA wrote:

[snip]

> > > Unfortunately the data-sheet of the co-processor cannot be made public
> > > as of yet.
> > 
> > Can you publish a block diagram of the co-processor internals ?
> 
> I will check internally to see if I can send a block-diagram of the co-
> processor internals to you. The internals seem similar to OMAP ISP (which I
> can see from the public TRM). However, our co-processor doesn't have the
> circular buffer and MMU that ISP seem to have (and some other features).

The co-processor doesn't write to the system main memory but outputs the data 
on a CSI2 or ITU interface, so that's not really surprising.

> In the meantime I copy the features of the co-processor here for your
> review:
> 
> Input / Output interfaces of co-processor:
> ==========================================
> - Sensor interfaces: 2 x MIPI CSI-2 receivers (1 x dual-lane up to 1.6
> Gbit/s and 1 x single lane up to 800 Mbit/s)
> - Host interface: MIPI CSI-2 dual lane transmitter (up to 1.6 Gbit/s) or
> ITU (8-bit CCIR interface, up to 100 MHz) - all with independent variable
> transmitter clock (PLL)
> - Control interface: CCI (up to 400 kHz) or SPI
> 
> Sensor support:
> ===============
> - Supports two MIPI compliant sensors of up to 8 Megapixel resolution
>  (one sensor streaming at a time)
> - Support for auto-focus (AF), extended depth of field (EDOF) and wide
> dynamic range (WDR)sensors
> 
> Internal Features:
> ==================
> - Versatile clock manager and internal buffer to accommodate a wide range
> of data rates between sensors and the coprocessor and between the
> coprocessor and the host.
> - Synchronized flash gun control with red-eye reduction (pre-flash and main-
> flash strobes) for high-power LED or Xenon strobe light
> - Low power standby mode
> - Two video pipes (enables concurrent viewfinding and video/stills capture)
> - Face detection and tracking algorithm
> - Video stabilization
> - Adaptive 4-channel lens shading and barrel distortion correction
> - Statistics processor for advanced automatic exposure and white balance
> - Automatic contrast stretch
> - Nine-zone auto-focus with flexible actuator driver
> - Digital zoom: smooth 16x down-scale capability and 4x up-scale capability
> - Advanced noise and defect filtering
> - Color reconstruction
> - Adaptive color correction matrix
> - Sharpness enhancement
> - Programmable gamma correction
> - Lighting frequency detection and automatic flicker reduction
> - Image rotation/mirroring/flip for the viewfinder (up to 480 x 360)
> - Special effects
> 
> Data Formats:
> =============
> - Output formats: JPEG, YUV4:2:2, YUV4:2:0, Planar YUV4:2:0 (up to 480 x
> 360), RGB888, RGB565, RGB444
> - JPEG compression with programmable quantization matrix and target file
> size

Given the complexity of the co-processor, I think it would make sense to break 
it in pieces and use the media controller API, especially if data routing is 
configurable inside the co-processor.

-- 
Regards,

Laurent Pinchart

Re: isp or soc-camera for image co-processors

[Sakari Ailus]

Hi Bhupesh and Laurent,

Bhupesh SHARMA wrote:
...
>> Try to configure your mailer to use spaces instead of tabs, or to make
>> tabs 8
>> spaces wide. It should then look good. Replies will usually mess the
>> diagrams
>> up though.
> 
> Ok, I will try it :)

Attachments are usually safe as well.

...

>>> Are there are reference drivers that I can use for my study?
>>
>> The OMAP3 ISP driver.
> 
> Thanks, I will go through the same.

The major difference in this to OMAP 3 is that the OMAP 3 does have
access to host side memory but the co-processor doesn't --- as it's a
CSI-2 link.

Additional CSI-2 receiver (and a driver for it) is required on the host
side. This receiver likely is not dependent on the co-processor so the
driver shouldn't be either.

For example, using this co-processor should well be possible with the
OMAP 3 ISP, in theory at least. What would be needed in this case is...
support for multiple complex Media device drivers under a single Media
device --- both drivers would be accessible through the same media device.

The co-processor would mostly look like a sensor for the OMAP 3 ISP
driver. Its internal topology would be more complex, though.

Just a few ideas; what do you think of this? :-)

>>> Unfortunately the data-sheet of the co-processor cannot be made
>> public
>>> as of yet.
>>
>> Can you publish a block diagram of the co-processor internals ?
> 
> I will check internally to see if I can send a block-diagram
> of the co-processor internals to you. The internals seem similar to 

I'd be very interested in this as well, thank you.

> OMAP ISP (which I can see from the public TRM). However, our
> co-processor doesn't have the circular buffer and MMU that ISP seem to
> have (and some other features).
> 
> In the meantime I copy the features of the co-processor here for your review:
> 
> Input / Output interfaces of co-processor:
> ==========================================
> - Sensor interfaces: 2 x MIPI CSI-2 receivers (1 x dual-lane up to 1.6 Gbit/s
>  and 1 x single lane up to 800 Mbit/s)
> - Host interface: MIPI CSI-2 dual lane transmitter (up to 1.6 Gbit/s) or ITU
>  (8-bit CCIR interface, up to 100 MHz) - all with independent variable
>  transmitter clock (PLL)
> - Control interface: CCI (up to 400 kHz) or SPI
> 
> Sensor support:
> ===============
> - Supports two MIPI compliant sensors of up to 8 Megapixel resolution
>  (one sensor streaming at a time)
> - Support for auto-focus (AF), extended depth of field (EDOF) and wide dynamic
>  range (WDR)sensors 
> 
> Internal Features:
> ==================
> - Versatile clock manager and internal buffer to accommodate a wide range of data rates
>   between sensors and the coprocessor and between the coprocessor and the host.
> - Synchronized flash gun control with red-eye reduction (pre-flash and main-flash strobes) for
>   high-power LED or Xenon strobe light

Does the co-processor have internal memory or can external memory be
attached to it for buffer storage?

Regards,

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: isp or soc-camera for image co-processors

[Laurent Pinchart]

Hi Sakari,

On Wednesday 02 March 2011 11:55:47 Sakari Ailus wrote:
> Bhupesh SHARMA wrote:

[snip]

> >>> Are there are reference drivers that I can use for my study?
> >> 
> >> The OMAP3 ISP driver.
> > 
> > Thanks, I will go through the same.
> 
> The major difference in this to OMAP 3 is that the OMAP 3 does have
> access to host side memory but the co-processor doesn't --- as it's a
> CSI-2 link.
> 
> Additional CSI-2 receiver (and a driver for it) is required on the host
> side. This receiver likely is not dependent on the co-processor so the
> driver shouldn't be either.
> 
> For example, using this co-processor should well be possible with the
> OMAP 3 ISP, in theory at least. What would be needed in this case is...
> support for multiple complex Media device drivers under a single Media
> device --- both drivers would be accessible through the same media device.
> 
> The co-processor would mostly look like a sensor for the OMAP 3 ISP
> driver. Its internal topology would be more complex, though.
> 
> Just a few ideas; what do you think of this? :-)

Hierachical subdevs is something that will be discussed during the next V4L2 
brainstorming meeting. We will need hierachical entities support in the Media 
Controller as well. This should help in this case, the co-processor entity 
will be made of several sub-entities.

-- 
Regards,

Laurent Pinchart

RE: isp or soc-camera for image co-processors

[Bhupesh SHARMA]

[-- Attachment #1: Type: text/plain, Size: 5196 bytes --]

Hi Sakari, Laurent and Guennadi,

> -----Original Message-----
> From: Sakari Ailus [mailto:sakari.ailus@maxwell.research.nokia.com]
> Sent: Wednesday, March 02, 2011 4:26 PM
> To: Bhupesh SHARMA
> Cc: Laurent Pinchart; Guennadi Liakhovetski; linux-
> media@vger.kernel.org
> Subject: Re: isp or soc-camera for image co-processors
> 
> Hi Bhupesh and Laurent,
> 
> Bhupesh SHARMA wrote:
> ...
> >> Try to configure your mailer to use spaces instead of tabs, or to
> make
> >> tabs 8
> >> spaces wide. It should then look good. Replies will usually mess the
> >> diagrams
> >> up though.
> >
> > Ok, I will try it :)
> 
> Attachments are usually safe as well.

Please find attached a top level diagram of the system.
One thing to note here is that the soc itself has a camera
interface IP that supports ITU interface. As the co-processor
supports both ITU and CSI-2 interface, it should not be a problem
to connect the two via ITU interface.

> ...
> 
> >>> Are there are reference drivers that I can use for my study?
> >>
> >> The OMAP3 ISP driver.
> >
> > Thanks, I will go through the same.
> 
> The major difference in this to OMAP 3 is that the OMAP 3 does have
> access to host side memory but the co-processor doesn't --- as it's a
> CSI-2 link.
> 
> Additional CSI-2 receiver (and a driver for it) is required on the host
> side. This receiver likely is not dependent on the co-processor so the
> driver shouldn't be either.
> 
> For example, using this co-processor should well be possible with the
> OMAP 3 ISP, in theory at least. What would be needed in this case is...
> support for multiple complex Media device drivers under a single Media
> device --- both drivers would be accessible through the same media
> device.
> 
> The co-processor would mostly look like a sensor for the OMAP 3 ISP
> driver. Its internal topology would be more complex, though.
> 
> Just a few ideas; what do you think of this? :-)

Yes, but I think I need to explain the system design better :
One, there is an camera interface IP within the SOC as well which 
has an internal buffer to store a line of image data and dma capabilities
to send this data to system ddr.

So, co-processor has no access to system MMU or buffers inside the main SoC,
but it has internal buffer to store data. It is connected via either a ITU or
CSI-2 interface to the SoC. This is the primary and major difference between our
architecture and OMAP 3 ISP.

As I read more the OMAP 3 TRM, I wonder whether SoC framework fits better
in our case, as we have three separate entities to consider here:
	- Camera Host inside the SoC
	- Camera Co-processor connected with host via CSI-2/ITU (data interface)
	  and I2C/CCI (control interface)
	- Camera sensor connected to the co-processor via CSI-2 (data interface)
	  and I2C/CCI (control interface)
What are your views?
Guennadi can you also pitch in with your thoughts..

I fear that neither soc-camera  nor media framework have support
for 3 entities listed above, as of now.

> >>> Unfortunately the data-sheet of the co-processor cannot be made
> >> public
> >>> as of yet.
> >>
> >> Can you publish a block diagram of the co-processor internals ?
> >
> > I will check internally to see if I can send a block-diagram
> > of the co-processor internals to you. The internals seem similar to
> 
> I'd be very interested in this as well, thank you.

I have raised a request internally to enquire about the same :)

> > OMAP ISP (which I can see from the public TRM). However, our
> > co-processor doesn't have the circular buffer and MMU that ISP seem
> to
> > have (and some other features).
> >
> > In the meantime I copy the features of the co-processor here for your
> review:
> >
> > Input / Output interfaces of co-processor:
> > ==========================================
> > - Sensor interfaces: 2 x MIPI CSI-2 receivers (1 x dual-lane up to
> 1.6 Gbit/s
> >  and 1 x single lane up to 800 Mbit/s)
> > - Host interface: MIPI CSI-2 dual lane transmitter (up to 1.6 Gbit/s)
> or ITU
> >  (8-bit CCIR interface, up to 100 MHz) - all with independent
> variable
> >  transmitter clock (PLL)
> > - Control interface: CCI (up to 400 kHz) or SPI
> >
> > Sensor support:
> > ===============
> > - Supports two MIPI compliant sensors of up to 8 Megapixel resolution
> >  (one sensor streaming at a time)
> > - Support for auto-focus (AF), extended depth of field (EDOF) and
> wide dynamic
> >  range (WDR)sensors
> >
> > Internal Features:
> > ==================
> > - Versatile clock manager and internal buffer to accommodate a wide
> range of data rates
> >   between sensors and the coprocessor and between the coprocessor and
> the host.
> > - Synchronized flash gun control with red-eye reduction (pre-flash
> and main-flash strobes) for
> >   high-power LED or Xenon strobe light
> 
> Does the co-processor have internal memory or can external memory be
> attached to it for buffer storage?
> 

The co-processor has no access to system MMU or buffers inside the main SoC,
but it has internal buffer to store data.

Regards,
Bhupesh

[-- Attachment #2: connection_diagram.JPG --]
[-- Type: image/jpeg, Size: 29878 bytes --]

RE: isp or soc-camera for image co-processors

[Guennadi Liakhovetski]

Hi Bhupesh

On Thu, 3 Mar 2011, Bhupesh SHARMA wrote:

> Hi Sakari, Laurent and Guennadi,
> 
> > -----Original Message-----
> > From: Sakari Ailus [mailto:sakari.ailus@maxwell.research.nokia.com]
> > Sent: Wednesday, March 02, 2011 4:26 PM
> > To: Bhupesh SHARMA
> > Cc: Laurent Pinchart; Guennadi Liakhovetski; linux-
> > media@vger.kernel.org
> > Subject: Re: isp or soc-camera for image co-processors
> > 
> > Hi Bhupesh and Laurent,
> > 
> > Bhupesh SHARMA wrote:
> > ...
> > >> Try to configure your mailer to use spaces instead of tabs, or to
> > make
> > >> tabs 8
> > >> spaces wide. It should then look good. Replies will usually mess the
> > >> diagrams
> > >> up though.
> > >
> > > Ok, I will try it :)
> > 
> > Attachments are usually safe as well.
> 
> Please find attached a top level diagram of the system.
> One thing to note here is that the soc itself has a camera
> interface IP that supports ITU interface. As the co-processor
> supports both ITU and CSI-2 interface, it should not be a problem
> to connect the two via ITU interface.
> 
> > ...
> > 
> > >>> Are there are reference drivers that I can use for my study?
> > >>
> > >> The OMAP3 ISP driver.
> > >
> > > Thanks, I will go through the same.
> > 
> > The major difference in this to OMAP 3 is that the OMAP 3 does have
> > access to host side memory but the co-processor doesn't --- as it's a
> > CSI-2 link.
> > 
> > Additional CSI-2 receiver (and a driver for it) is required on the host
> > side. This receiver likely is not dependent on the co-processor so the
> > driver shouldn't be either.
> > 
> > For example, using this co-processor should well be possible with the
> > OMAP 3 ISP, in theory at least. What would be needed in this case is...
> > support for multiple complex Media device drivers under a single Media
> > device --- both drivers would be accessible through the same media
> > device.
> > 
> > The co-processor would mostly look like a sensor for the OMAP 3 ISP
> > driver. Its internal topology would be more complex, though.
> > 
> > Just a few ideas; what do you think of this? :-)
> 
> Yes, but I think I need to explain the system design better :
> One, there is an camera interface IP within the SOC as well which 
> has an internal buffer to store a line of image data and dma capabilities
> to send this data to system ddr.
> 
> So, co-processor has no access to system MMU or buffers inside the main SoC,
> but it has internal buffer to store data. It is connected via either a ITU or
> CSI-2 interface to the SoC. This is the primary and major difference between our
> architecture and OMAP 3 ISP.
> 
> As I read more the OMAP 3 TRM, I wonder whether SoC framework fits better
> in our case, as we have three separate entities to consider here:
> 	- Camera Host inside the SoC
> 	- Camera Co-processor connected with host via CSI-2/ITU (data interface)
> 	  and I2C/CCI (control interface)
> 	- Camera sensor connected to the co-processor via CSI-2 (data interface)
> 	  and I2C/CCI (control interface)
> What are your views?
> Guennadi can you also pitch in with your thoughts..

The soc-camera interface used to provide two things: (1) a standard API 
between "camera hosts" (video interfaces on SoCs) amd "camera clients" 
(camera sensors, TV decoders,...), and (2) a number of helper functions to 
handle format enumeration, simplify control handling (this is being 
replaced by a more generic API), and provide a bus-configuration 
framework. It also takes over file operations and a couple other common 
functions. Currently, given the v4l2-subdev API, the new control framework 
and the approaching Media Controller API there is not much left, that the 
soc-camera framework is still _adding_ to the standard v4l2 functionality, 
maybe just format enumeration and bus-parameter negotiation.

Further, soc-camera is not very well suited for multi-component designs 
with more than 2 components. It is possible, like I've done with the CSI2 
interface on sh-mobile, but you inherit the limitation of the current v4l2 
API: all these components are controlled over just one device node in the 
user-space, so, it becomes difficult to let the user configure conponents 
separately. This problem is eliminated by the Media Controller API, which 
gives you access to each subdevice separately from the user-space.

At some point in the future soc-camera will also be converted to the MC 
API, but until that's done, using it for complex designs like yours will 
remain difficult, at least because you'll only get one device node for 
your user-space applications. I think, the MC API provides more important 
advantages to your situation, than the soc-camera framework. Of course, if 
you want to use any of its functionality, feel free to become creative and 
combine the two or even do the soc-camera to MC porting yourself;)

Thanks
Guennadi

> 
> I fear that neither soc-camera  nor media framework have support
> for 3 entities listed above, as of now.
> 
> > >>> Unfortunately the data-sheet of the co-processor cannot be made
> > >> public
> > >>> as of yet.
> > >>
> > >> Can you publish a block diagram of the co-processor internals ?
> > >
> > > I will check internally to see if I can send a block-diagram
> > > of the co-processor internals to you. The internals seem similar to
> > 
> > I'd be very interested in this as well, thank you.
> 
> I have raised a request internally to enquire about the same :)
> 
> > > OMAP ISP (which I can see from the public TRM). However, our
> > > co-processor doesn't have the circular buffer and MMU that ISP seem
> > to
> > > have (and some other features).
> > >
> > > In the meantime I copy the features of the co-processor here for your
> > review:
> > >
> > > Input / Output interfaces of co-processor:
> > > ==========================================
> > > - Sensor interfaces: 2 x MIPI CSI-2 receivers (1 x dual-lane up to
> > 1.6 Gbit/s
> > >  and 1 x single lane up to 800 Mbit/s)
> > > - Host interface: MIPI CSI-2 dual lane transmitter (up to 1.6 Gbit/s)
> > or ITU
> > >  (8-bit CCIR interface, up to 100 MHz) - all with independent
> > variable
> > >  transmitter clock (PLL)
> > > - Control interface: CCI (up to 400 kHz) or SPI
> > >
> > > Sensor support:
> > > ===============
> > > - Supports two MIPI compliant sensors of up to 8 Megapixel resolution
> > >  (one sensor streaming at a time)
> > > - Support for auto-focus (AF), extended depth of field (EDOF) and
> > wide dynamic
> > >  range (WDR)sensors
> > >
> > > Internal Features:
> > > ==================
> > > - Versatile clock manager and internal buffer to accommodate a wide
> > range of data rates
> > >   between sensors and the coprocessor and between the coprocessor and
> > the host.
> > > - Synchronized flash gun control with red-eye reduction (pre-flash
> > and main-flash strobes) for
> > >   high-power LED or Xenon strobe light
> > 
> > Does the co-processor have internal memory or can external memory be
> > attached to it for buffer storage?
> > 
> 
> The co-processor has no access to system MMU or buffers inside the main SoC,
> but it has internal buffer to store data.
> 
> Regards,
> Bhupesh
> 

---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: isp or soc-camera for image co-processors

[Sakari Ailus]

Bhupesh SHARMA wrote:
> Hi Sakari, Laurent and Guennadi,

Hi Bhupesh and others,

>> ...
>>
>>>>> Are there are reference drivers that I can use for my study?
>>>>
>>>> The OMAP3 ISP driver.
>>>
>>> Thanks, I will go through the same.
>>
>> The major difference in this to OMAP 3 is that the OMAP 3 does have
>> access to host side memory but the co-processor doesn't --- as it's a
>> CSI-2 link.
>>
>> Additional CSI-2 receiver (and a driver for it) is required on the host
>> side. This receiver likely is not dependent on the co-processor so the
>> driver shouldn't be either.
>>
>> For example, using this co-processor should well be possible with the
>> OMAP 3 ISP, in theory at least. What would be needed in this case is...
>> support for multiple complex Media device drivers under a single Media
>> device --- both drivers would be accessible through the same media
>> device.
>>
>> The co-processor would mostly look like a sensor for the OMAP 3 ISP
>> driver. Its internal topology would be more complex, though.
>>
>> Just a few ideas; what do you think of this? :-)
> 
> Yes, but I think I need to explain the system design better :
> One, there is an camera interface IP within the SOC as well which 
> has an internal buffer to store a line of image data and dma capabilities
> to send this data to system ddr.
> 
> So, co-processor has no access to system MMU or buffers inside the main SoC,
> but it has internal buffer to store data. It is connected via either a ITU or
> CSI-2 interface to the SoC. This is the primary and major difference between our
> architecture and OMAP 3 ISP.
> 
> As I read more the OMAP 3 TRM, I wonder whether SoC framework fits better
> in our case, as we have three separate entities to consider here:
> 	- Camera Host inside the SoC
> 	- Camera Co-processor connected with host via CSI-2/ITU (data interface)
> 	  and I2C/CCI (control interface)
> 	- Camera sensor connected to the co-processor via CSI-2 (data interface)
> 	  and I2C/CCI (control interface)
> What are your views?
> Guennadi can you also pitch in with your thoughts..
> 
> I fear that neither soc-camera  nor media framework have support
> for 3 entities listed above, as of now.

The Media controller interface allows enumerating entities and links in
the media device, activating the links between the entities. There is no
such thing as support for particular type of entity.

V4L2 subdevs on the other hand do give access to the V4L2 specifics of
the entities.

I think the Media controller & V4L2 subdevs are a good starting point
for supporting this kind of hardware.

Changes in the Media controller framework are likely needed to allow
more flexible graph definition than currently is possible. That should
make attaching this type of a device to any existing CSI-2 (or parallel)
receiver on the host CPU relatively easy.

>>>>> Unfortunately the data-sheet of the co-processor cannot be made
>>>> public
>>>>> as of yet.
>>>>
>>>> Can you publish a block diagram of the co-processor internals ?
>>>
>>> I will check internally to see if I can send a block-diagram
>>> of the co-processor internals to you. The internals seem similar to
>>
>> I'd be very interested in this as well, thank you.
> 
> I have raised a request internally to enquire about the same :)

Thanks!

>>> OMAP ISP (which I can see from the public TRM). However, our
>>> co-processor doesn't have the circular buffer and MMU that ISP seem
>> to
>>> have (and some other features).
>>>
>>> In the meantime I copy the features of the co-processor here for your
>> review:
>>>
>>> Input / Output interfaces of co-processor:
>>> ==========================================
>>> - Sensor interfaces: 2 x MIPI CSI-2 receivers (1 x dual-lane up to
>> 1.6 Gbit/s
>>>  and 1 x single lane up to 800 Mbit/s)
>>> - Host interface: MIPI CSI-2 dual lane transmitter (up to 1.6 Gbit/s)
>> or ITU
>>>  (8-bit CCIR interface, up to 100 MHz) - all with independent
>> variable
>>>  transmitter clock (PLL)
>>> - Control interface: CCI (up to 400 kHz) or SPI
>>>
>>> Sensor support:
>>> ===============
>>> - Supports two MIPI compliant sensors of up to 8 Megapixel resolution
>>>  (one sensor streaming at a time)
>>> - Support for auto-focus (AF), extended depth of field (EDOF) and
>> wide dynamic
>>>  range (WDR)sensors
>>>
>>> Internal Features:
>>> ==================
>>> - Versatile clock manager and internal buffer to accommodate a wide
>> range of data rates
>>>   between sensors and the coprocessor and between the coprocessor and
>> the host.
>>> - Synchronized flash gun control with red-eye reduction (pre-flash
>> and main-flash strobes) for
>>>   high-power LED or Xenon strobe light
>>
>> Does the co-processor have internal memory or can external memory be
>> attached to it for buffer storage?
>>
> 
> The co-processor has no access to system MMU or buffers inside the main SoC,
> but it has internal buffer to store data.

One more question... does the co-processor have enough memory to store
images themselves? The rotation functionality suggests that it has more
memory than is required to save just a few lines of image data.

What about the jpeg encoding; is the co-processor also able to provide
the same image as raw bayer or in YUV colour space to the host, for example?

Best regards,

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com