Re: Using MT9P031 digital sensor

[Gary Thomas <gary@mlbassoc.com>]

On 2011-11-28 04:07, Laurent Pinchart wrote:
> Hi Gary,
>
> On Friday 25 November 2011 12:50:25 Gary Thomas wrote:
>> On 2011-11-24 04:28, Laurent Pinchart wrote:
>>> On Wednesday 16 November 2011 13:03:11 Gary Thomas wrote:
>>>> On 2011-11-15 18:26, Laurent Pinchart wrote:
>>>>> On Monday 14 November 2011 12:42:54 Gary Thomas wrote:
>>>>>> On 2011-11-11 07:26, Laurent Pinchart wrote:
>>>>>>> On Wednesday 09 November 2011 17:24:26 Gary Thomas wrote:
>>>>>>>> On 2011-11-09 09:18, Laurent Pinchart wrote:
>>>>>>>>> On Wednesday 09 November 2011 12:01:34 Gary Thomas wrote:
>>>>>>>>>> On 2011-11-08 17:54, Laurent Pinchart wrote:
>>>>>>>>>>> On Tuesday 08 November 2011 14:38:55 Gary Thomas wrote:
>>>>>>>>>>>> On 2011-11-08 06:06, Laurent Pinchart wrote:
>>>>>>>>>>>>> On Tuesday 08 November 2011 13:52:25 Gary Thomas wrote:
>>>>>>>>>>>>>> On 2011-11-08 05:30, Javier Martinez Canillas wrote:
>>>>>>>>>>>>>>> On Tue, Nov 8, 2011 at 1:20 PM, Gary Thomas wrote:
>>>>>>>>>>>>>>>> On 2011-11-04 04:37, Laurent Pinchart wrote:
>>>>>>>>>>>>>>>>> On Tuesday 01 November 2011 19:52:49 Gary Thomas wrote:
>>>>>>>>>>>>>>>>>> I'm trying to use the MT9P031 digital sensor with the
>>>>>>>>>>>>>>>>>> Media Controller Framework.  media-ctl tells me that the
>>>>>>>>>>>>>>>>>> sensor is set to capture using SGRBG12  2592x1944
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Questions:
>>>>>>>>>>>>>>>>>> * What pixel format in ffmpeg does this correspond to?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I don't know if ffmpeg supports Bayer formats. The
>>>>>>>>>>>>>>>>> corresponding fourcc in V4L2 is BA12.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ffmpeg doesn't seem to support these formats
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If your sensor is hooked up to the OMAP3 ISP, you can then
>>>>>>>>>>>>>>>>> configure the pipeline to include the preview engine and
>>>>>>>>>>>>>>>>> the resizer, and capture YUV data
>>>>>>>>>>>>>>>>> at the resizer output.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am using the OMAP3 ISP, but it's a bit unclear to me how
>>>>>>>>>>>>>>>> to set up the pipeline
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Gary,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm also using another sensor mtv9034 with OMAP3 ISP, so
>>>>>>>>>>>>>>> maybe I can help you.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> using media-ctl (I looked for documentation on this tool,
>>>>>>>>>>>>>>>> but came up dry - is there any?)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Do you have an example of how to configure this using the
>>>>>>>>>>>>>>>> OMAP3 ISP?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is how I configure the pipeline to connect the CCDC with
>>>>>>>>>>>>>>> the Previewer and Resizer:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ./media-ctl -l '"mt9v032 3-005c":0->"OMAP3 ISP CCDC":0[1]'
>>>>>>>>>>>>>>> ./media-ctl -l '"OMAP3 ISP CCDC":2->"OMAP3 ISP preview":0[1]'
>>>>>>>>>>>>>>> ./media-ctl -l '"OMAP3 ISP preview":1->"OMAP3 ISP
>>>>>>>>>>>>>>> resizer":0[1]' ./media-ctl -l '"OMAP3 ISP resizer":1->"OMAP3
>>>>>>>>>>>>>>> ISP resizer output":0[1]' ./media-ctl -f '"mt9v032
>>>>>>>>>>>>>>> 3-005c":0[SGRBG10 752x480]' ./media-ctl -f  '"OMAP3 ISP
>>>>>>>>>>>>>>> CCDC":0 [SGRBG10 752x480]' ./media-ctl -f  '"OMAP3 ISP
>>>>>>>>>>>>>>> CCDC":1 [SGRBG10 752x480]' ./media-ctl -f  '"OMAP3 ISP
>>>>>>>>>>>>>>> preview":0 [SGRBG10 752x479]' ./media-ctl -f  '"OMAP3 ISP
>>>>>>>>>>>>>>> resizer":0 [YUYV 734x471]' ./media-ctl -f  '"OMAP3 ISP
>>>>>>>>>>>>>>> resizer":1 [YUYV 640x480]'
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hope it helps,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks, I'll give this a try.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I assume that your sensor is probably larger than 752x480 (the
>>>>>>>>>>>>>> mt9p031 is 2592x1944 raw) and that setting the smaller frame
>>>>>>>>>>>>>> size enables some scaling and/or cropping in the driver?
>>>>>>>>>>>>>
>>>>>>>>>>>>> The mt9v034 is a wide VGA 752x480 sensor if I'm not mistaken.
>>>>>>>>>>>>> You should modify the resolutions in the above commands
>>>>>>>>>>>>> according to your sensor. Note that the CCDC crops online line
>>>>>>>>>>>>> when outputting data to the preview engine, and that the
>>>>>>>>>>>>> preview engine crops 18 columsn and 8 lines. You can then
>>>>>>>>>>>>> scale the image by modifying the resizer output size.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.  After much trial and error (and some kernel printks to
>>>>>>>>>>>>
>>>>>>>>>>>> understand what parameters were failing), I came up with this
>>>>>
>>>>> sequence:
>>>>>>>>>>>>          media-ctl -r
>>>>>>>>>>>>          media-ctl -l '"mt9p031 3-005d":0->"OMAP3 ISP CCDC":0[1]'
>>>>>>>>>>>>          media-ctl -l '"OMAP3 ISP CCDC":2->"OMAP3 ISP
>>>>>>>>>>>>          preview":0[1]' media-ctl -l '"OMAP3 ISP
>>>>>>>>>>>>          preview":1->"OMAP3 ISP resizer":0[1]' media-ctl -l
>>>>>>>>>>>>          '"OMAP3 ISP resizer":1->"OMAP3 ISP resizer
>>>>>>>>>>>>          output":0[1]' media-ctl -f '"mt9p031 3-005d":0[SGRBG12
>>>>>>>>>>>>          2592x1944]' media-ctl -f  '"OMAP3 ISP CCDC":0 [SGRBG12
>>>>>>>>>>>>          2592x1944]'
>>>>>>>>>>>>          media-ctl -f  '"OMAP3 ISP CCDC":1 [SGRBG12 2592x1944]'
>>>>>>>>>>>>          media-ctl -f  '"OMAP3 ISP preview":0 [SGRBG12
>>>>>>>>>>>>          2592x1943]' media-ctl -f  '"OMAP3 ISP resizer":0 [YUYV
>>>>>>>>>>>>          2574x1935]' media-ctl -f  '"OMAP3 ISP resizer":1 [YUYV
>>>>>>>>>>>>          642x483]'
>>>>>>>>>>>>
>>>>>>>>>>>> When I tried to grab though, I got this:
>>>>>>>>>>>>
>>>>>>>>>>>> # yavta --capture=4 -f YUYV -s 642x483 -F /dev/video6
>>>>>>>>>>>> Device /dev/video6 opened.
>>>>>>>>>>>> Device `OMAP3 ISP resizer output' on `media' is a video capture
>>>>>>>>>>>> device. Video format set: YUYV (56595559) 642x483 buffer size
>>>>>>>>>>>> 633696 Video format: YUYV (56595559) 642x483 buffer size 633696
>>>>>>>>>>>> 8 buffers requested.
>>>>>>>>>>>> length: 633696 offset: 0
>>>>>>>>>>>> Buffer 0 mapped at address 0x4028c000.
>>>>>>>>>>>> length: 633696 offset: 634880
>>>>>>>>>>>> Buffer 1 mapped at address 0x403d0000.
>>>>>>>>>>>> length: 633696 offset: 1269760
>>>>>>>>>>>> Buffer 2 mapped at address 0x404b3000.
>>>>>>>>>>>> length: 633696 offset: 1904640
>>>>>>>>>>>> Buffer 3 mapped at address 0x4062b000.
>>>>>>>>>>>> length: 633696 offset: 2539520
>>>>>>>>>>>> Buffer 4 mapped at address 0x406d6000.
>>>>>>>>>>>> length: 633696 offset: 3174400
>>>>>>>>>>>> Buffer 5 mapped at address 0x40821000.
>>>>>>>>>>>> length: 633696 offset: 3809280
>>>>>>>>>>>> Buffer 6 mapped at address 0x4097c000.
>>>>>>>>>>>> length: 633696 offset: 4444160
>>>>>>>>>>>> Buffer 7 mapped at address 0x40adf000.
>>>>>>>>>>>>
>>>>>>>>>>>> Unable to handle kernel NULL pointer dereference at virtual
>>>>>>>>>>>> address 00000018
>>>>>>>>>>>
>>>>>>>>>>> Ouch :-(
>>>>>>>>>>>
>>>>>>>>>>> Could you please verify that arch/arm/mach-omap2/board-overo.c
>>>>>>>>>>> includes the following code, and that CONFIG_OMAP_MUX is enabled
>>>>>>>>>>> ?
>>>>>>>>>>
>>>>>>>>>> I'm not using an Overo board - rather one of our own internal
>>>>>>>>>> designs.
>>>>>>>>>
>>>>>>>>> My bad, sorry.
>>>>>>>>>
>>>>>>>>>> I have verified that the pull up/down on those pins is disabled.
>>>>>>>>>>
>>>>>>>>>> The failure is coming from this code in ispccdc.c
>>>>>>>>>>
>>>>>>>>>>         static void ccdc_hs_vs_isr(struct isp_ccdc_device *ccdc)
>>>>>>>>>>         {
>>>>>>>>>> 	
>>>>>>>>>> 	  struct isp_pipeline *pipe =
>>>>>>>>>> 		
>>>>>>>>>> 		to_isp_pipeline(&ccdc->video_out.video.entity);
>>>>>>>>>>
>>>>>>>>>> The value of pipe is NULL which leads to the failure.
>>>>>>>>>>
>>>>>>>>>> Questions:
>>>>>>>>>> * I assume that getting HS/VS interrupts is correct in this mode?
>>>>>>>>>> * Why is the statement not written (as all others are)
>>>>>>>>>>
>>>>>>>>>> 	struct isp_pipeline *pipe =
>>>>>>>>>> 	to_isp_pipeline(&ccdc->subdev.entity);
>>>>>>>>>> 	
>>>>>>>>>>         I tried this change and the kernel doesn't crash.
>>>>>>>>>>
>>>>>>>>>> I've found that I can get raw frames out of CCDC, but I don't get
>>>>>>>>>> anything at all when the output continues through the preview
>>>>>>>>>> and/or resize nodes.
>>>>>>>>>>
>>>>>>>>>> Ideas?
>>>>>>>>>
>>>>>>>>> I'm really puzzled, this should have been caught much earlier :-)
>>>>>>>>>
>>>>>>>>> Your analysis makes sense. Would you like to submit a patch
>>>>>>>>> yourself ? If not I can do it.
>>>>>>>>
>>>>>>>> Sure, I can submit a patch. I would like to figure out why it's not
>>>>>>>> working first.
>>>>>>>
>>>>>>> Oops, I've overlooked that, sorry.
>>>>>>>
>>>>>>>> Any ideas how I can debug this? I can't seem to get anything past
>>>>>>>> the CCDC, e.g. into the preview or resize units. Is there some way
>>>>>>>> to trace packets/data through the various stages? Any ideas what
>>>>>>>> might cause it to stall?
>>>>>>>
>>>>>>> How have you configured your pipeline ? Can you try tracing the
>>>>>>> preview engine and/or resizer interrupts ?
>>>>>>
>>>>>> Here's my pipeline:
>>>>>>       media-ctl -r
>>>>>>       media-ctl -l '"mt9p031 3-005d":0->"OMAP3 ISP CCDC":0[1]'
>>>>>>       media-ctl -l '"OMAP3 ISP CCDC":2->"OMAP3 ISP preview":0[1]'
>>>>>>       media-ctl -l '"OMAP3 ISP preview":1->"OMAP3 ISP resizer":0[1]'
>>>>>>       media-ctl -l '"OMAP3 ISP resizer":1->"OMAP3 ISP resizer
>>>>>>       output":0[1]' media-ctl -f '"mt9p031 3-005d":0[SGRBG12
>>>>>>       2592x1944]' media-ctl -f  '"OMAP3 ISP CCDC":0 [SGRBG10
>>>>>>       2592x1944]'
>>>>>>       media-ctl -f  '"OMAP3 ISP CCDC":1 [SGRBG10 2592x1944]'
>>>>>>       media-ctl -f  '"OMAP3 ISP preview":0 [SGRBG10 2592x1943]'
>>>>>>       media-ctl -f  '"OMAP3 ISP resizer":0 [YUYV 2574x1935]'
>>>>>>       media-ctl -f  '"OMAP3 ISP resizer":1 [YUYV 642x483]'
>>>>>>
>>>>>> The full media-ctl dump is at
>>>>>> http://www.mlbassoc.com/misc/pipeline.out
>>>>>>
>>>>>> When I try to grab from /dev/video6 (output node of resizer), I see
>>>>>> only previewer interrupts, no resizer interrrupts.  I added a simple
>>>>>> printk at each of the previewer/resizer *_isr functions, and I only
>>>>>>
>>>>>> ever see this one:
>>>>>>       omap3isp_preview_isr_frame_sync.1373
>>>>>>
>>>>>> Can you give me an overview of what events/interrupts should occur so
>>>>>> I can try to trace through the ISP to see where it is failing?
>>>>>
>>>>> The CCDC generates VD0, VD1 and HS/VS interrupts regardless of whether
>>>>> it processes video or not, as long as it receives a video stream at
>>>>> its input. The preview engine and resizer will only generate an
>>>>> interrupt after writing an image to memory. With your above
>>>>> configuration VD0, VD1, HS/VS and resizer interrupts should be
>>>>> generated.
>>>>>
>>>>> Your pipeline configuration looks correct, except that the downscaling
>>>>> factor is slightly larger than 4. Could you try to setup the resizer to
>>>>> output a 2574x1935 image instead of 642x483 ? If that works, try to
>>>>> downscale to 660x496. If that works as well, the driver should be fixed
>>>>> to disallow resolutions that won't work.
>>>>
>>>> No change.  I also tried using only the previewer like this:
>>>>      media-ctl -r
>>>>      media-ctl -l '"mt9p031 3-005d":0->"OMAP3 ISP CCDC":0[1]'
>>>>      media-ctl -l '"OMAP3 ISP CCDC":2->"OMAP3 ISP preview":0[1]'
>>>>      media-ctl -l '"OMAP3 ISP preview":1->"OMAP3 ISP preview
>>>>      output":0[1]' media-ctl -f '"mt9p031 3-005d":0[SGRBG12 2592x1944]'
>>>>      media-ctl -f  '"OMAP3 ISP CCDC":0 [SGRBG12 2592x1944]'
>>>>      media-ctl -f  '"OMAP3 ISP CCDC":1 [SGRBG10 2592x1944]'
>>>>      media-ctl -f  '"OMAP3 ISP preview":0 [SGRBG10 2592x1943]'
>>>>      media-ctl -f  '"OMAP3 ISP preview":1 [YUYV 2574x1935]'
>>>>
>>>>      yavta --capture=4 -f YUYV -s 2574x1935 -F /dev/video4
>>>>
>>>> I still only get the frame sync interrupts in the previewer, no buffer
>>>> interrupts, hence no data flowing to my application.  What else can I
>>>> look at?
>>>
>>> Do you get VD0 and VD1 interrupts ?
>>
>> Yes, the CCDC is working correctly, but nothing moves through the
>> previewer. Here's a trace of the interrupt sequence I get, repeated over
>> and over.  These are printed as __FUNCTION__.__LINE__
>> --- ccdc_vd0_isr.1615
>> --- ccdc_hs_vs_isr.1482
>> --- ccdc_vd1_isr.1664
>> --- omap3isp_preview_isr_frame_sync.1373
>>
>> What's the best tree to try this against?  3.2-rc2 doesn't have the BT656
>> stuff in it yet, so I've been still using my older tree (3.0.0 +
>> drivers/media from your tree)
>
> I thought you were using an MT9P031 ? That doesn't require BT656 support.
>

True, but I have one board that supports either sensor and I want to stay
with one source tree.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------