OMAP3 ISP previewer Y10 to UYVY conversion

OMAP3 ISP previewer Y10 to UYVY conversion

[Chris Whittenburg]

We're working on a DM3730 platform running a 3.5.7 kernel, using the
pipeline below to take a 12-bit monochrome sensor (Aptina AR0130) and
convert it to UYVY format for use with the TI codecs.

In general, this works, but the images end up looking washed out.
Running them thru a "normalize" function makes them look good again.
Looking at the levels histogram in gimp, I seem to be missing the high
end and low end values.

I've captured the 12-bit data from the CCDC, downconverted it to Y8,
and verified it looks ok, and is not washed out, so I'm suspecting the
isp previewer is doing something wrong in the simple Y10 to UYVY
conversion.

Does someone with experience on this topic have a recommendation on
what component might be causing the problem, or the best way to go
about isolating the issue?

media-ctl -r
media-ctl -v -l '"ar0130 3-0010":0->"OMAP3 ISP CCDC":0[1], "OMAP3 ISP
CCDC":2->"OMAP3 ISP preview":0[1], "OMAP3 ISP preview":1->"OMAP3 ISP
resizer":0[1], "OMAP3 ISP resizer":1->"OMAP3 ISP resizer output":0[1]'
media-ctl -v -V '"ar0130 3-0010":0['Y12' '640'x'480'], "OMAP3 ISP
CCDC":2['Y10' '640'x'480'], "OMAP3 ISP preview":1[UYVY '640'x'480'],
"OMAP3 ISP resizer":1[UYVY '640'x'480']'

Thanks,
Chris

Re: OMAP3 ISP previewer Y10 to UYVY conversion

[Sakari Ailus]

Hi Chris,

On Wed, Mar 18, 2015 at 02:56:36PM -0500, Chris Whittenburg wrote:
> We're working on a DM3730 platform running a 3.5.7 kernel, using the
> pipeline below to take a 12-bit monochrome sensor (Aptina AR0130) and
> convert it to UYVY format for use with the TI codecs.
> 
> In general, this works, but the images end up looking washed out.
> Running them thru a "normalize" function makes them look good again.
> Looking at the levels histogram in gimp, I seem to be missing the high
> end and low end values.

Do you know if the sensor has black level correction enabled? It appears to
have one, but I'm not completely sure what it does there. I'd check that it
is indeed enabled.

> I've captured the 12-bit data from the CCDC, downconverted it to Y8,
> and verified it looks ok, and is not washed out, so I'm suspecting the
> isp previewer is doing something wrong in the simple Y10 to UYVY
> conversion.

Not necessarily wrong, the black level correction might be enabled by
default, with the default configuration which works for most sensors (64 for
10-bit data, 16 for 8-bit etc.).

-- 
Regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: OMAP3 ISP previewer Y10 to UYVY conversion

[Chris Whittenburg]

Hi Sakari,

Thanks for the reply.

On Tue, Mar 24, 2015 at 6:51 PM, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> Do you know if the sensor has black level correction enabled? It appears to
> have one, but I'm not completely sure what it does there. I'd check that it
> is indeed enabled.

The ar0130cs does have black level correction enabled by default.

My thought is that since the 12-bit data from the CCDC looked ok, that
it was something outside the sensor itself.

>> I've captured the 12-bit data from the CCDC, downconverted it to Y8,
>> and verified it looks ok, and is not washed out, so I'm suspecting the
>> isp previewer is doing something wrong in the simple Y10 to UYVY
>> conversion.
>
> Not necessarily wrong, the black level correction might be enabled by
> default, with the default configuration which works for most sensors (64 for
> 10-bit data, 16 for 8-bit etc.).

Ok, I will check this.  You are referring to the "Camera ISP VPBE
Preview Black Adjustment" which is controlled by PRV_BLKADJOFF
register?

I also found that there are contrast and brightness settings in the
previewer which can be adjusted.  I'm not changing them from defaults,
so I thought the "Y" values would just get truncated to 8 bits and
mapped into the UYVY without being significantly altered.

Would your thought be the black level is more likely the issue rather
than brightness/contrast?

Is there anywhere else I should look?

Thanks,
Chris

Re: OMAP3 ISP previewer Y10 to UYVY conversion

[Sakari Ailus]

Hi Chris,

On Wed, Mar 25, 2015 at 09:12:56AM -0500, Chris Whittenburg wrote:
> Hi Sakari,
> 
> Thanks for the reply.
> 
> On Tue, Mar 24, 2015 at 6:51 PM, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> > Do you know if the sensor has black level correction enabled? It appears to
> > have one, but I'm not completely sure what it does there. I'd check that it
> > is indeed enabled.
> 
> The ar0130cs does have black level correction enabled by default.
> 
> My thought is that since the 12-bit data from the CCDC looked ok, that
> it was something outside the sensor itself.

I think I might have misunderstood your original mail. I thought the images
from CCDC were bad.

> >> I've captured the 12-bit data from the CCDC, downconverted it to Y8,
> >> and verified it looks ok, and is not washed out, so I'm suspecting the
> >> isp previewer is doing something wrong in the simple Y10 to UYVY
> >> conversion.
> >
> > Not necessarily wrong, the black level correction might be enabled by
> > default, with the default configuration which works for most sensors (64 for
> > 10-bit data, 16 for 8-bit etc.).
> 
> Ok, I will check this.  You are referring to the "Camera ISP VPBE
> Preview Black Adjustment" which is controlled by PRV_BLKADJOFF
> register?

I guess it wasn't. The value appears to be zero by default, which makes
sense.

> I also found that there are contrast and brightness settings in the
> previewer which can be adjusted.  I'm not changing them from defaults,
> so I thought the "Y" values would just get truncated to 8 bits and
> mapped into the UYVY without being significantly altered.
> 
> Would your thought be the black level is more likely the issue rather
> than brightness/contrast?

I haven't used this part of the ISP, perhaps Laurent has.

-- 
Regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: OMAP3 ISP previewer Y10 to UYVY conversion

[Laurent Pinchart]

Hello Chris,

On Thursday 26 March 2015 01:58:20 Sakari Ailus wrote:
> On Wed, Mar 25, 2015 at 09:12:56AM -0500, Chris Whittenburg wrote:
> > Hi Sakari,
> > 
> > Thanks for the reply.
> > 
> > On Tue, Mar 24, 2015 at 6:51 PM, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> > > Do you know if the sensor has black level correction enabled? It appears
> > > to have one, but I'm not completely sure what it does there. I'd check
> > > that it is indeed enabled.
> > 
> > The ar0130cs does have black level correction enabled by default.
> > 
> > My thought is that since the 12-bit data from the CCDC looked ok, that
> > it was something outside the sensor itself.
> 
> I think I might have misunderstood your original mail. I thought the images
> from CCDC were bad.
> 
> > >> I've captured the 12-bit data from the CCDC, downconverted it to Y8,
> > >> and verified it looks ok, and is not washed out, so I'm suspecting the
> > >> isp previewer is doing something wrong in the simple Y10 to UYVY
> > >> conversion.
> > > 
> > > Not necessarily wrong, the black level correction might be enabled by
> > > default, with the default configuration which works for most sensors (64
> > > for 10-bit data, 16 for 8-bit etc.).
> > 
> > Ok, I will check this.  You are referring to the "Camera ISP VPBE
> > Preview Black Adjustment" which is controlled by PRV_BLKADJOFF
> > register?
> 
> I guess it wasn't. The value appears to be zero by default, which makes
> sense.
> 
> > I also found that there are contrast and brightness settings in the
> > previewer which can be adjusted.  I'm not changing them from defaults,
> > so I thought the "Y" values would just get truncated to 8 bits and
> > mapped into the UYVY without being significantly altered.
> > 
> > Would your thought be the black level is more likely the issue rather
> > than brightness/contrast?
> 
> I haven't used this part of the ISP, perhaps Laurent has.

Black level compensation is applied by the CCDC before writing raw frames to 
memory. If your raw frames are correct BLC is probably not to blame.

The default contrast is x1.0 and the default brightness is +0.0, so I don't 
think those should be blame either.

I suspect the RGB2RGB conversion matrix to be wrong. The default setting is 
supposed to handle fluorescent lighting. You could try setting the RGB2RGB 
matrix to the identity matrix and see if this helps. See 
http://git.ideasonboard.org/omap3-isp-live.git/blob/HEAD:/isp/controls.c#l184 
for sample code.

Another matrix that could be worth being reprogrammed is the RGB2YUV matrix, 
which also defaults to fluorescent lighting. Sample code to reprogram it is 
available in the same location.

-- 
Regards,

Laurent Pinchart

Re: OMAP3 ISP previewer Y10 to UYVY conversion

[Chris Whittenburg]

On Tue, Apr 7, 2015 at 10:51 AM, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> I suspect the RGB2RGB conversion matrix to be wrong. The default setting is
> supposed to handle fluorescent lighting. You could try setting the RGB2RGB
> matrix to the identity matrix and see if this helps. See
> http://git.ideasonboard.org/omap3-isp-live.git/blob/HEAD:/isp/controls.c#l184
> for sample code.
>
> Another matrix that could be worth being reprogrammed is the RGB2YUV matrix,
> which also defaults to fluorescent lighting. Sample code to reprogram it is
> available in the same location.

Thanks Laurent...

Just to verify, the conversion matrix still plays a role even though
the input format is Y10?

Re: OMAP3 ISP previewer Y10 to UYVY conversion

[Laurent Pinchart]

Hi Chris,

On Tuesday 07 April 2015 11:45:10 Chris Whittenburg wrote:
> On Tue, Apr 7, 2015 at 10:51 AM, Laurent Pinchart wrote:
> > I suspect the RGB2RGB conversion matrix to be wrong. The default setting
> > is supposed to handle fluorescent lighting. You could try setting the
> > RGB2RGB matrix to the identity matrix and see if this helps. See
> > http://git.ideasonboard.org/omap3-isp-live.git/blob/HEAD:/isp/controls.c#l
> > 184 for sample code.
> > 
> > Another matrix that could be worth being reprogrammed is the RGB2YUV
> > matrix, which also defaults to fluorescent lighting. Sample code to
> > reprogram it is available in the same location.
> 
> Thanks Laurent...
> 
> Just to verify, the conversion matrix still plays a role even though
> the input format is Y10?

Yes it does. The preview engine first converts Y10 to RGB by duplicating the Y 
value for all three channels, then feeds it through the RGB2RGB matrix, the 
gamma table, and the RGB2YUV matrix.

-- 
Regards,

Laurent Pinchart

Re: OMAP3 ISP previewer Y10 to UYVY conversion

[Chris Whittenburg]

On Tue, Apr 7, 2015 at 10:51 AM, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> Black level compensation is applied by the CCDC before writing raw frames to
> memory. If your raw frames are correct BLC is probably not to blame.
>
> The default contrast is x1.0 and the default brightness is +0.0, so I don't
> think those should be blame either.
>
> I suspect the RGB2RGB conversion matrix to be wrong. The default setting is
> supposed to handle fluorescent lighting. You could try setting the RGB2RGB
> matrix to the identity matrix and see if this helps. See
> http://git.ideasonboard.org/omap3-isp-live.git/blob/HEAD:/isp/controls.c#l184
> for sample code.
>
> Another matrix that could be worth being reprogrammed is the RGB2YUV matrix,
> which also defaults to fluorescent lighting. Sample code to reprogram it is
> available in the same location.

I tried changing the rgb2rgb matrx to the identity matrix:

{0x0100, 0x0000, 0x0000},
{0x0000, 0x0100, 0x0000},
{0x0000, 0x0000, 0x0100}

And the csc (rgb2yuv) to this:
{256, 0, 0},
{0, 0, 0},
{0, 0, 0}

But I couldn't see much, if any, difference.

However, when I forced the gamma correction to be bypassed, it seemed to fix it.

Does that make sense?  I guess I don't understand it enough to
understand if gamma correction would have compressed all my luma
values.

Thanks,
Chris

Re: OMAP3 ISP previewer Y10 to UYVY conversion

[Laurent Pinchart]

Hi Chris,

On Thursday 16 April 2015 13:05:30 Chris Whittenburg wrote:
> On Tue, Apr 7, 2015 at 10:51 AM, Laurent Pinchart wrote:
> > Black level compensation is applied by the CCDC before writing raw frames
> > to memory. If your raw frames are correct BLC is probably not to blame.
> > 
> > The default contrast is x1.0 and the default brightness is +0.0, so I
> > don't think those should be blame either.
> > 
> > I suspect the RGB2RGB conversion matrix to be wrong. The default setting
> > is supposed to handle fluorescent lighting. You could try setting the
> > RGB2RGB matrix to the identity matrix and see if this helps. See
> > http://git.ideasonboard.org/omap3-isp-live.git/blob/HEAD:/isp/controls.c#l
> > 184 for sample code.
> > 
> > Another matrix that could be worth being reprogrammed is the RGB2YUV
> > matrix, which also defaults to fluorescent lighting. Sample code to
> > reprogram it is available in the same location.
> 
> I tried changing the rgb2rgb matrx to the identity matrix:
> 
> {0x0100, 0x0000, 0x0000},
> {0x0000, 0x0100, 0x0000},
> {0x0000, 0x0000, 0x0100}
> 
> And the csc (rgb2yuv) to this:
> {256, 0, 0},
> {0, 0, 0},
> {0, 0, 0}
> 
> But I couldn't see much, if any, difference.
> 
> However, when I forced the gamma correction to be bypassed, it seemed to fix
> it.
> 
> Does that make sense?  I guess I don't understand it enough to understand if
> gamma correction would have compressed all my luma values.

Yes, it makes sense. Gamma correction applies a non-linear transformation to 
the pixel values and can explain the problems you were seeing.

I've checked the default rgb2rgb matrix, and it should work fine for your case 
as all lines add up to 1.0. The default rgb2yuv matrix, however, limits Y 
values to 220, so you should modify it.

-- 
Regards,

Laurent Pinchart

Re: OMAP3 ISP previewer Y10 to UYVY conversion

[Chris Whittenburg]

On Fri, Apr 17, 2015 at 4:39 AM, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> Hi Chris,
>
> On Thursday 16 April 2015 13:05:30 Chris Whittenburg wrote:
>> On Tue, Apr 7, 2015 at 10:51 AM, Laurent Pinchart wrote:
>> > Black level compensation is applied by the CCDC before writing raw frames
>> > to memory. If your raw frames are correct BLC is probably not to blame.
>> >
>> > The default contrast is x1.0 and the default brightness is +0.0, so I
>> > don't think those should be blame either.
>> >
>> > I suspect the RGB2RGB conversion matrix to be wrong. The default setting
>> > is supposed to handle fluorescent lighting. You could try setting the
>> > RGB2RGB matrix to the identity matrix and see if this helps. See
>> > http://git.ideasonboard.org/omap3-isp-live.git/blob/HEAD:/isp/controls.c#l
>> > 184 for sample code.
>> >
>> > Another matrix that could be worth being reprogrammed is the RGB2YUV
>> > matrix, which also defaults to fluorescent lighting. Sample code to
>> > reprogram it is available in the same location.
>>
>> I tried changing the rgb2rgb matrx to the identity matrix:
>>
>> {0x0100, 0x0000, 0x0000},
>> {0x0000, 0x0100, 0x0000},
>> {0x0000, 0x0000, 0x0100}
>>
>> And the csc (rgb2yuv) to this:
>> {256, 0, 0},
>> {0, 0, 0},
>> {0, 0, 0}
>>
>> But I couldn't see much, if any, difference.
>>
>> However, when I forced the gamma correction to be bypassed, it seemed to fix
>> it.
>>
>> Does that make sense?  I guess I don't understand it enough to understand if
>> gamma correction would have compressed all my luma values.
>
> Yes, it makes sense. Gamma correction applies a non-linear transformation to
> the pixel values and can explain the problems you were seeing.
>
> I've checked the default rgb2rgb matrix, and it should work fine for your case
> as all lines add up to 1.0. The default rgb2yuv matrix, however, limits Y
> values to 220, so you should modify it.

I believe the formula used is:
Y = CSCRY*Rin + CSCGY*Gin + CSCBY*Bin +Yoffset

If CSCRY=1.0, and Rin is in the range 0 to 255, then the resulting Y
would be in the range [0, 255] as well, so why would the matrix limit
Y values to 220?  Is it because Rin, Gin, and Bin have already been
limited to the YCbCr range of [16, 240]?

Re: OMAP3 ISP previewer Y10 to UYVY conversion

[Laurent Pinchart]

Hi Chris,

On Tuesday 21 April 2015 11:04:14 Chris Whittenburg wrote:
> On Fri, Apr 17, 2015 at 4:39 AM, Laurent Pinchart wrote:
> > On Thursday 16 April 2015 13:05:30 Chris Whittenburg wrote:
> >> On Tue, Apr 7, 2015 at 10:51 AM, Laurent Pinchart wrote:
> >>> Black level compensation is applied by the CCDC before writing raw
> >>> frames to memory. If your raw frames are correct BLC is probably not to
> >>> blame.
> >>> 
> >>> The default contrast is x1.0 and the default brightness is +0.0, so I
> >>> don't think those should be blame either.
> >>> 
> >>> I suspect the RGB2RGB conversion matrix to be wrong. The default
> >>> setting is supposed to handle fluorescent lighting. You could try
> >>> setting the RGB2RGB matrix to the identity matrix and see if this
> >>> helps. See
> >>> http://git.ideasonboard.org/omap3-isp-live.git/blob/HEAD:/isp/controls.
> >>> c#l184 for sample code.
> >>> 
> >>> Another matrix that could be worth being reprogrammed is the RGB2YUV
> >>> matrix, which also defaults to fluorescent lighting. Sample code to
> >>> reprogram it is available in the same location.
> >> 
> >> I tried changing the rgb2rgb matrx to the identity matrix:
> >> 
> >> {0x0100, 0x0000, 0x0000},
> >> {0x0000, 0x0100, 0x0000},
> >> {0x0000, 0x0000, 0x0100}
> >> 
> >> And the csc (rgb2yuv) to this:
> >> {256, 0, 0},
> >> {0, 0, 0},
> >> {0, 0, 0}
> >> 
> >> But I couldn't see much, if any, difference.
> >> 
> >> However, when I forced the gamma correction to be bypassed, it seemed to
> >> fix it.
> >> 
> >> Does that make sense?  I guess I don't understand it enough to understand
> >> if gamma correction would have compressed all my luma values.
> > 
> > Yes, it makes sense. Gamma correction applies a non-linear transformation
> > to the pixel values and can explain the problems you were seeing.
> > 
> > I've checked the default rgb2rgb matrix, and it should work fine for your
> > case as all lines add up to 1.0. The default rgb2yuv matrix, however,
> > limits Y values to 220, so you should modify it.
> 
> I believe the formula used is:
> Y = CSCRY*Rin + CSCGY*Gin + CSCBY*Bin +Yoffset
> 
> If CSCRY=1.0, and Rin is in the range 0 to 255, then the resulting Y
> would be in the range [0, 255] as well, so why would the matrix limit
> Y values to 220?  Is it because Rin, Gin, and Bin have already been
> limited to the YCbCr range of [16, 240]?

I meant that the default matrix programmed by the omap3isp driver will limit 
the range to 220.

    {Y}   {  66, 129,  25} {R}
    {U} = { -38, -75, 112} {G}
    {V}   { 112, -94, -18} {B}

Y = 66*R + 129*G + 25*B

As R = G = B = y in your case, that's

Y = 66*y + 129*y + 25*y = 220*y

Coefficients are expressed in S10Q8 format (signed, 10 bits in total, 8 
decimal bits), so we end up with Y = 220/256 * y, limiting the Y range to 220 
(assuming the input value is limited to 8 bits).

-- 
Regards,

Laurent Pinchart