hacking MT9P031 (LI-5M03) driver in Ubuntu 12.04 on BeagleBoard xM?

hacking MT9P031 (LI-5M03) driver in Ubuntu 12.04 on BeagleBoard xM?

[Chris MacGregor]

Hi.  I was redirected to this list by a response to my post (below) on 
the BeagleBoard group.  I'm happy to help/cooperate/etc. in whatever way 
I reasonably can.

-------- Original Message --------

Hello, all.

I managed to get the MT9P031 driver (for the Leopard Imaging LI-5M03 
camera board) working using a slightly modified Ubuntu 12.04 kernel, 
including the mainline kernel version of the MT9P031 driver.  Once 
everything is clean and happy I will post info for those still trying to 
get there.  Meanwhile, though, there are some odd issues, and a few 
driver bugs I need to fix and features I need to add.  I wanted to reach 
out to others who are working with this hardware in current (not 
Angstrom 2.6.x) kernels so we can compare notes, and so we don't go off 
in the wrong (or a different) direction on solving some of the problems.

For our application, we are capturing video in raw format (raw Bayer), 
with MT9P031 -> CCDC -> CCDC output (no resizer etc.), reading from 
/dev/video2.  The new media controller framework is pretty cool once you 
get the hang of it - it addresses some significant deficiencies.  I just 
wish the new subdev selection was available, but it's not in 3.2.x...  
hopefully we can move to 3.5 soon and then I just need to implement it 
in the MT9P031 driver (if someone else doesn't do that before I get there).

Some of the issues:

1. To get it working, I had to patch in the Aptina driver mods for 
board-omap3beagle.c etc.  I'm not at all sure this is kosher since I'm 
using the mainline kernel driver, not the Aptina driver (nor the 
RidgeRun one, in which I had to fix a lot of bugs when we were doing 
this on a Leopardboard).  But without these changes, the camera was not 
recognized (likely because it wasn't being powered up).  I would think 
that someone out there must be using the driver in the mainline kernel, 
since it's in there, but how are they getting the camera to be recognized?

2. Max frame rate at full resolution seems to be 6.86 fps.  I think 
we're running at half clock speed.  We'd like to fix that.  I can track 
it down, but I don't want to duplicate work already done by someone 
else, and of course this likely relates to issue # 1, above.

3. When I start streaming, then stop streaming, then start streaming 
again without closing and reopening the device in between (and sometimes 
even if I do but reopen right after closing), the second time we start 
streaming, it appears that the green and non-green (red or blue as the 
case may be) pixels are swapped - as if it was offset by one column.  
But if I change the cropping (using VIDIOC_SUBDEV_S_FMT on 
/ev/v4l-subdev8, which is the MT9P031 directly) to include the black 
(inactive) pixels on the top and left, it is still true - but the black 
pixels don't change, only the active ones, even though they still start 
at the same offset (+10,+50 IIRC).  I don't even see how that should be 
possible.  The MT9P031 registers (all of them) are the same whether the 
swapping is occurring or not, and ditto for the CCDC registers per the 
dump in the kernel log.  Has anyone else seen this?  I have worked 
around it for now by closing the device (all of them), sleeping for 2 
ms, and then reopening and reconfiguring.  However, I'd really like to 
find a proper solution, or at least understand the root cause - it's 
kind of disturbing, especially since without the sleep it still didn't 
reliably work correctly.  This may also relate to issue # 1, above.

4. I need to add some additional controls (like a way to manipulate the 
vblank register setting so we can reduce the frame rate without just 
randomly dropping frames - we want to adjust the frame rate to what we 
can fairly reliably store without dropping frames - and access to the 
separate gain controls for R, Gr, Gb, and B, since we're using color 
sensors (cheaper) with IR illumination).  I'd like to get some feedback 
on the most appropriate way to do this.  Obviously I could just hack it 
in, but I'd rather do it right and hopefully get it into the mainline 
driver.  In 3.5-rc2, I see a definition for a VBLANK control, but it 
still isn't clear what ought to be used for separate gain controls.

5. The driver (and likewise the CCDC driver) needs a few small fixes, 
and I'd like to avoid duplication of effort, etc.

Thanks,
       Chris MacGregor

hacking MT9P031 (LI-5M03) driver in Ubuntu 12.04 on BeagleBoard xM?

[Chris MacGregor]

Hi.  I was redirected to this list by a response to my post (below) on 
the BeagleBoard group. I'm happy to help/cooperate/etc. in whatever way 
I reasonably can.

-------- Original Message --------

Hello, all.

I managed to get the MT9P031 driver (for the Leopard Imaging LI-5M03 
camera board) working using a slightly modified Ubuntu 12.04 kernel 
(3.2.x), including the mainline kernel version of the MT9P031 driver.  
Once everything is clean and happy I will post info for those still 
trying to get there.  Meanwhile, though, there are some odd issues, and 
a few driver bugs I need to fix and features I need to add.  I wanted to 
reach out to others who are working with this hardware in current (not 
Angstrom 2.6.x) kernels so we can compare notes, and so we don't go off 
in the wrong (or a different) direction on solving some of the problems.

For our application, we are capturing video in raw format (raw Bayer), 
with MT9P031 -> CCDC -> CCDC output (no resizer etc.), reading from 
/dev/video2.  The new media controller framework is pretty cool once you 
get the hang of it - it addresses some significant deficiencies.  I just 
wish the new subdev selection was available, but it's not in 3.2.x... 
hopefully we can move to 3.5 soon and then I just need to implement it 
in the MT9P031 driver (if someone else doesn't do that before I get there).

Some of the issues:

1. To get it working, I had to patch in the Aptina driver mods for 
board-omap3beagle.c etc.  I'm not at all sure this is kosher since I'm 
using the mainline kernel driver, not the Aptina driver (nor the 
RidgeRun one, in which I had to fix a lot of bugs when we were doing 
this on a Leopardboard).  But without these changes, the camera was not 
recognized (likely because it wasn't being powered up).  I would think 
that someone out there must be using the driver in the mainline kernel, 
since it's in there, but how are they getting the camera to be recognized?

2. Max frame rate at full resolution seems to be 6.86 fps.  I think 
we're running at half clock speed.  We'd like to fix that. I can track 
it down, but I don't want to duplicate work already done by someone 
else, and of course this likely relates to issue # 1, above.

3. When I start streaming, then stop streaming, then start streaming 
again without closing and reopening the device in between (and sometimes 
even if I do but reopen right after closing), the second time we start 
streaming, it appears that the green and non-green (red or blue as the 
case may be) pixels are swapped - as if it was offset by one column.  
But if I change the cropping (using VIDIOC_SUBDEV_S_FMT on 
/ev/v4l-subdev8, which is the MT9P031 directly) to include the black 
(inactive) pixels on the top and left, it is still true - but the black 
pixels don't change, only the active ones, even though they still start 
at the same offset (+10,+50 IIRC).  I don't even see how that should be 
possible.  The MT9P031 registers (all of them) are the same whether the 
swapping is occurring or not, and ditto for the CCDC registers per the 
dump in the kernel log.  Has anyone else seen this?  I have worked 
around it for now by closing the device (all of them), sleeping for 2 
ms, and then reopening and reconfiguring.  However, I'd really like to 
find a proper solution, or at least understand the root cause - it's 
kind of disturbing, especially since without the sleep it still didn't 
reliably work correctly.  This may also relate to issue # 1, above.

4. I need to add some additional controls (like a way to manipulate the 
vblank register setting so we can reduce the frame rate without just 
randomly dropping frames - we want to adjust the frame rate to what we 
can fairly reliably store without dropping frames - and access to the 
separate gain controls for R, Gr, Gb, and B, since we're using color 
sensors (cheaper) with IR illumination).  I'd like to get some feedback 
on the most appropriate way to do this.  Obviously I could just hack it 
in, but I'd rather do it right and hopefully get it into the mainline 
driver.  In 3.5-rc2, I see a definition for a VBLANK control, but it 
still isn't clear what ought to be used for separate gain controls.

5. The driver (and likewise the CCDC driver) needs a few small fixes, 
and I'd like to avoid duplication of effort, etc.

Thanks,
       Chris MacGregor

Re: hacking MT9P031 (LI-5M03) driver in Ubuntu 12.04 on BeagleBoard xM?

[Laurent Pinchart]

Hi Chris,

On Thursday 21 June 2012 09:38:11 Chris MacGregor wrote:
> Hi.  I was redirected to this list by a response to my post (below) on
> the BeagleBoard group.  I'm happy to help/cooperate/etc. in whatever way
> I reasonably can.
> 
> -------- Original Message --------
> 
> Hello, all.
> 
> I managed to get the MT9P031 driver (for the Leopard Imaging LI-5M03 camera
> board) working using a slightly modified Ubuntu 12.04 kernel, including the
> mainline kernel version of the MT9P031 driver. Once everything is clean and
> happy I will post info for those still trying to get there. Meanwhile,
> though, there are some odd issues, and a few driver bugs I need to fix and
> features I need to add. I wanted to reach out to others who are working with
> this hardware in current (not Angstrom 2.6.x) kernels so we can compare
> notes, and so we don't go off in the wrong (or a different) direction on
> solving some of the problems.
> 
> For our application, we are capturing video in raw format (raw Bayer), with
> MT9P031 -> CCDC -> CCDC output (no resizer etc.), reading from /dev/video2. 
> The new media controller framework is pretty cool once you get the hang of
> it - it addresses some significant deficiencies.

Thank you :-)

> I just wish the new subdev selection was available, but it's not in 3.2.x...
> hopefully we can move to 3.5 soon and then I just need to implement it in
> the MT9P031 driver (if someone else doesn't do that before I get there).
> 
> Some of the issues:
> 
> 1. To get it working, I had to patch in the Aptina driver mods for board-
> omap3beagle.c etc. I'm not at all sure this is kosher since I'm using the
> mainline kernel driver, not the Aptina driver (nor the RidgeRun one, in
> which I had to fix a lot of bugs when we were doing this on a Leopardboard). 
> But without these changes, the camera was not recognized (likely because it
> wasn't being powered up). I would think that someone out there must be using
> the driver in the mainline kernel, since it's in there, but how are they
> getting the camera to be recognized?

Unlike on PC hardware, operating systems on embedded hardware usually can't 
discover devices at runtime. The Linux kernel thus needs a list of the devices 
present on the system (both inside the SoC and on the board) to handle them 
properly.

That list is usually hardcoded in board code, and the Linux kernel on ARM 
recently started a migration to the Device Tree that provides such a hardware 
description from outside the kernel.

Devices present inside the SoC or directly on the board are not removable and 
can be hardcoded in board code or in the device tree. However, devices that 
come on add-on boards are problematic as they're not always present and can be 
replaced.

Several sensor modules exist for the Beagleboard-xM, we can't hardcode support 
for one of them in the mainline kernel. There's currently no way to properly 
support the different sensor modules with a single kernel, mostly because 
nobody developed a solution so far (although proposals have been posted to 
mailing lists).

For that reason I currently maintain board code with sensor support for 
several OMAP3 platforms in the omap3isp-sensors-board branch of my git tree at 
http://git.linuxtv.org/pinchartl/media.git. I'd be happy to push that to 
mainline if we had a good technical solution.

Where did you get the Aptina board code patch from ?

> 2. Max frame rate at full resolution seems to be 6.86 fps. I think we're
> running at half clock speed. We'd like to fix that. I can track it down, but
> I don't want to duplicate work already done by someone else, and of course
> this likely relates to issue # 1, above.

The clock speed is configurable, but the device is limited to 48 MHz when 
using 1.8V I/O. To reach 96 MHz we would have to power the I/O supply with 
2.8V and add a level shifter on the board, as the OMAP3 use 1/8V I/O.

> 3. When I start streaming, then stop streaming, then start streaming again
> without closing and reopening the device in between (and sometimes even if I
> do but reopen right after closing), the second time we start streaming, it
> appears that the green and non-green (red or blue as the case may be) pixels
> are swapped - as if it was offset by one column. But if I change the
> cropping (using VIDIOC_SUBDEV_S_FMT on /ev/v4l-subdev8, which is the MT9P031
> directly) to include the black (inactive) pixels on the top and left, it is
> still true - but the black pixels don't change, only the active ones, even
> though they still start at the same offset (+10,+50 IIRC). I don't even see
> how that should be possible.

Just to make sure I understand that properly, do you mean that the boundary 
between the black and non-black pixels doesn't move, when you expected it to 
be shifted by one column or one line if the color swap had been caused by an 
image shift ? Did you include both black lines and black columns in your test 
?

> The MT9P031 registers (all of them) are the same whether the swapping is
> occurring or not, and ditto for the CCDC registers per the dump in the
> kernel log. Has anyone else seen this? I have worked around it for now by
> closing the device (all of them), sleeping for 2 ms, and then reopening and
> reconfiguring. However, I'd really like to find a proper solution, or at
> least understand the root cause - it's kind of disturbing, especially since
> without the sleep it still didn't reliably work correctly. This may also
> relate to issue # 1, above.

I've never experienced that issue and I'm quite surprised. I'd like to check 
your board code first (although if all registers are identical it shouldn't 
play a big role).

What's your test procedure ?

> 4. I need to add some additional controls (like a way to manipulate the
> vblank register setting so we can reduce the frame rate without just
> randomly dropping frames - we want to adjust the frame rate to what we can
> fairly reliably store without dropping frames - and access to the separate
> gain controls for R, Gr, Gb, and B, since we're using color sensors
> (cheaper) with IR illumination). I'd like to get some feedback on the most
> appropriate way to do this. Obviously I could just hack it in, but I'd
> rather do it right and hopefully get it into the mainline driver. In
> 3.5-rc2, I see a definition for a VBLANK control,

That's the one you should use.

> but it still isn't clear what ought to be used for separate gain controls.

We need new per-component gain controls. The usual way to do this is to post a 
proposal to the linux-media mailing list, either in plain English if you want 
initial feedback on a complex idea before implementing it, or as a patch 
(don't forget to update Documentation/DocBook/media/v4l/controls.xml).

> 5. The driver (and likewise the CCDC driver) needs a few small fixes, and
> I'd like to avoid duplication of effort, etc.

The latest version of the driver with the patches that should be included in 
the next kernel version is available from my git tree (omap3isp-omap3isp-next 
branch, please note that the branch is regularly rebased). If the fixes are 
not included there, the best way to avoid effort duplication is to post a mail 
to the list with the description of the problem(s). If nobody answers with a 
patch, you will have to write one yourself. For very small fixes posting a 
patch up-front is usually easier and even less time-consuming.

-- 
Regards,

Laurent Pinchart

Re: hacking MT9P031 (LI-5M03) driver in Ubuntu 12.04 on BeagleBoard xM?

[Chris MacGregor]

Hi, Laurent.  Please see below.

On 06/27/2012 02:30 AM, Laurent Pinchart wrote:
> Hi Chris,
>
> On Thursday 21 June 2012 09:38:11 Chris MacGregor wrote:
>> ...
>>
>> Some of the issues:
>>
>> 1. To get it working, I had to patch in the Aptina driver mods for board-
>> omap3beagle.c etc. I'm not at all sure this is kosher since I'm using the
>> mainline kernel driver, not the Aptina driver (nor the RidgeRun one, in
>> which I had to fix a lot of bugs when we were doing this on a Leopardboard).
>> But without these changes, the camera was not recognized (likely because it
>> wasn't being powered up). I would think that someone out there must be using
>> the driver in the mainline kernel, since it's in there, but how are they
>> getting the camera to be recognized?
> Unlike on PC hardware, operating systems on embedded hardware usually can't
> discover devices at runtime. The Linux kernel thus needs a list of the devices
> present on the system (both inside the SoC and on the board) to handle them
> properly.
>
> That list is usually hardcoded in board code, and the Linux kernel on ARM
> recently started a migration to the Device Tree that provides such a hardware
> description from outside the kernel.
>
> Devices present inside the SoC or directly on the board are not removable and
> can be hardcoded in board code or in the device tree. However, devices that
> come on add-on boards are problematic as they're not always present and can be
> replaced.
>
> Several sensor modules exist for the Beagleboard-xM, we can't hardcode support
> for one of them in the mainline kernel. There's currently no way to properly
> support the different sensor modules with a single kernel, mostly because
> nobody developed a solution so far (although proposals have been posted to
> mailing lists).
>
> For that reason I currently maintain board code with sensor support for
> several OMAP3 platforms in the omap3isp-sensors-board branch of my git tree at
> http://git.linuxtv.org/pinchartl/media.git. I'd be happy to push that to
> mainline if we had a good technical solution.

I was expecting that the code would be in there somewhere, but (for the 
reason you describe) not enabled by default.  I was expecting that I 
might need to tweak the kernel config to enable the sensor support (and 
indeed I did), but I was very surprised that code necessary to make it 
actually work was nowhere in sight.

I would (naively, perhaps) think that it would be appropriate to have 
the code present, but enabled only if both MACH_OMAP3_BEAGLE and 
VIDEO_MT9P031 are enabled.  What am I missing?

I took a look at your branch (briefly).  The Ubuntu 12.04 kernel is 
currently based on a roughly 3.2.x (maybe 3.2.18?) kernel.  Do you 
happen to have a version of the board files that would be likely to work 
well on that vintage?

Or for that matter, do you have a recommendation as to what (whose) 
kernel would be my best bet for production use in a BeagleBoard xM (rev 
C, maybe some B's) using the Leopard Imaging LI-5M03 camera board?  I 
get the sense that the BB xM community is mostly on 3.2.x (or older) at 
the moment, and I hesitate to assume there isn't a good reason for 
that.  But if some kernel newer than Ubuntu's is reasonably stable and 
trustworthy on a BB xM, I'd be happy enough to use it.  (I do intend to 
continue with the Ubuntu 12.04 world for everything other than the kernel.)
> Where did you get the Aptina board code patch from ?

 From here: https://github.com/Aptina/BeagleBoard-xM
Which is linked to from here: 
http://blog.galemin.com/2011/04/li-5m03-camera-on-beagleboard-xm/

I started in trying to get the cameras working on our BeagleBoards (as 
the first step in switching from Leopardboards, with the same cameras), 
and quickly found that it was not at all obvious how to get the MT9P031 
to be recognized in Ubuntu's kernel, nor in Robert C. Nelson's.  I 
googled around and found Max Galemin's stuff, which got me a kernel that 
recognized the sensor but had some other quirks.  Further googling in 
search of an answer to why it worked in Max's kernel but not the others 
- despite no visibly relevant difference in the driver sources - led me 
to the above, or maybe I followed the link from Max's blog - I don't 
remember at this point. The patch file in there was the most significant 
clue, and so I crossed my fingers and patched in the files under 
arch/arm, and sure enough, it worked.  Mostly...

>> 2. Max frame rate at full resolution seems to be 6.86 fps. I think we're
>> running at half clock speed. We'd like to fix that. I can track it down, but
>> I don't want to duplicate work already done by someone else, and of course
>> this likely relates to issue # 1, above.
> The clock speed is configurable, but the device is limited to 48 MHz when
> using 1.8V I/O. To reach 96 MHz we would have to power the I/O supply with
> 2.8V and add a level shifter on the board, as the OMAP3 use 1/8V I/O.

This is definitely something we'll want to pursue.  Some of the 
organisms we're trying to watch (this is for University of Washington 
Seattle's Oceanography Dept.) move fast enough that we want the full 
frame rate (~14 fps) or closer to it.  We can use cropping to increase 
the frame rate, but sometimes we need the full field of view as well.

Do you have or can you point me to more details on the mods needed, both 
in the software and in the hardware?  (Has anyone out there already done 
this successfully?)

>> 3. When I start streaming, then stop streaming, then start streaming again
>> without closing and reopening the device in between (and sometimes even if I
>> do but reopen right after closing), the second time we start streaming, it
>> appears that the green and non-green (red or blue as the case may be) pixels
>> are swapped - as if it was offset by one column. But if I change the
>> cropping (using VIDIOC_SUBDEV_S_FMT on /ev/v4l-subdev8, which is the MT9P031
>> directly) to include the black (inactive) pixels on the top and left, it is
>> still true - but the black pixels don't change, only the active ones, even
>> though they still start at the same offset (+10,+50 IIRC). I don't even see
>> how that should be possible.
> Just to make sure I understand that properly, do you mean that the boundary
> between the black and non-black pixels doesn't move, when you expected it to
> be shifted by one column or one line if the color swap had been caused by an
> image shift ?

Yes, precisely that.

> Did you include both black lines and black columns in your test
> ?

Yes.  I set the crop region to start at (0,0), so I got 10 black columns 
and 50 black lines, IIRC.

>
>> The MT9P031 registers (all of them) are the same whether the swapping is
>> occurring or not, and ditto for the CCDC registers per the dump in the
>> kernel log. Has anyone else seen this? I have worked around it for now by
>> closing the device (all of them), sleeping for 2 ms, and then reopening and
>> reconfiguring. However, I'd really like to find a proper solution, or at
>> least understand the root cause - it's kind of disturbing, especially since
>> without the sleep it still didn't reliably work correctly. This may also
>> relate to issue # 1, above.
> I've never experienced that issue and I'm quite surprised. I'd like to check
> your board code first (although if all registers are identical it shouldn't
> play a big role).

Agreed, I'm highly suspicious of my current configuration, given how I 
arrived at it.  :-)

>
> What's your test procedure ?

I'm not sure I understand what you're asking for.  I have an app that 
configures everything (media controller, v4l-subdev8 for the mt9p031, 
etc.) and then starts streaming, capturing to a file or the network.  
The app can keep the device open, and under control of the UI, start and 
stop streaming repeatedly.  After a while I figured out that the 
swapping/shifting problem often happened about every other time I 
started streaming (though not 100% consistently). Eventually I was able 
to find that closing the device and reopening it would generally avoid 
the problem; when I did that from the UI it worked pretty well, but not 
when I did it directly within the code, until I added the 2 ms sleep.  
So by the end, my test procedure was to start it, see if it was screwed 
up or not, stop it, and repeat. Once I added the sleep, it seemed to be 
good enough as a temporary solution - I couldn't delay any longer.  The 
critters are in the water, and their schedule does not change no matter 
how good my excuses... :-/

>> 4. I need to add some additional controls (like a way to manipulate the
>> vblank register setting so we can reduce the frame rate without just
>> randomly dropping frames - we want to adjust the frame rate to what we can
>> fairly reliably store without dropping frames - and access to the separate
>> gain controls for R, Gr, Gb, and B, since we're using color sensors
>> (cheaper) with IR illumination). I'd like to get some feedback on the most
>> appropriate way to do this. Obviously I could just hack it in, but I'd
>> rather do it right and hopefully get it into the mainline driver. In
>> 3.5-rc2, I see a definition for a VBLANK control,
> That's the one you should use.
>
>> but it still isn't clear what ought to be used for separate gain controls.
> We need new per-component gain controls. The usual way to do this is to post a
> proposal to the linux-media mailing list, either in plain English if you want
> initial feedback on a complex idea before implementing it, or as a patch
> (don't forget to update Documentation/DocBook/media/v4l/controls.xml).

Thanks, I'll do that.  I wanted to find out what the consensus was on 
the general concept (e.g., do people agree that this is best addressed 
by exposing new per-component gain controls) before I took that step.  
I'll take your statement above as a sufficient consensus for now.  :-)

One sticky bit is that the driver currently implements a 
one-gain-to-fit-them-all approach.  Should I replace that with the new 
ones (simple and clean but breaks current users of it), or try to figure 
out a way for them to co-exist in a meaningful and coherent way (appears 
hairy and error-prone, but preserves compatibility)?

>> 5. The driver (and likewise the CCDC driver) needs a few small fixes, and
>> I'd like to avoid duplication of effort, etc.
> The latest version of the driver with the patches that should be included in
> the next kernel version is available from my git tree (omap3isp-omap3isp-next
> branch, please note that the branch is regularly rebased). If the fixes are
> not included there, the best way to avoid effort duplication is to post a mail
> to the list with the description of the problem(s). If nobody answers with a
> patch, you will have to write one yourself. For very small fixes posting a
> patch up-front is usually easier and even less time-consuming.

Okay, I'll post questions and/or patches here.  (Aiming for patches, but 
starting with questions where it seems appropriate.)

One other issue that it would be nice to address is why one or two 
frames (perhaps three, sometimes?) are corrupted at the start of 
streaming.  I don't think it's inherent to the sensor, since we never 
saw that with the very same sensor and camera board on a Leopardboard 
(DM365).  So it's either OMAP3-specific, or something in the drivers.  
I'm happy to help chase this down, but hoping someone else has already 
gotten a start on it.  Any clues out there?

Re: hacking MT9P031 (LI-5M03) driver in Ubuntu 12.04 on BeagleBoard xM?

[Laurent Pinchart]

Hi Chris,

On Thursday 28 June 2012 21:41:16 Chris MacGregor wrote:
> On 06/27/2012 02:30 AM, Laurent Pinchart wrote:
> > On Thursday 21 June 2012 09:38:11 Chris MacGregor wrote:
> >> ...
> >> 
> >> Some of the issues:
> >> 
> >> 1. To get it working, I had to patch in the Aptina driver mods for board-
> >> omap3beagle.c etc. I'm not at all sure this is kosher since I'm using the
> >> mainline kernel driver, not the Aptina driver (nor the RidgeRun one, in
> >> which I had to fix a lot of bugs when we were doing this on a
> >> Leopardboard). But without these changes, the camera was not recognized
> >> (likely because it wasn't being powered up). I would think that someone
> >> out there must be using the driver in the mainline kernel, since it's in
> >> there, but how are they getting the camera to be recognized?
> > 
> > Unlike on PC hardware, operating systems on embedded hardware usually
> > can't discover devices at runtime. The Linux kernel thus needs a list of
> > the devices present on the system (both inside the SoC and on the board)
> > to handle them properly.
> > 
> > That list is usually hardcoded in board code, and the Linux kernel on ARM
> > recently started a migration to the Device Tree that provides such a
> > hardware description from outside the kernel.
> > 
> > Devices present inside the SoC or directly on the board are not removable
> > and can be hardcoded in board code or in the device tree. However,
> > devices that come on add-on boards are problematic as they're not always
> > present and can be replaced.
> > 
> > Several sensor modules exist for the Beagleboard-xM, we can't hardcode
> > support for one of them in the mainline kernel. There's currently no way
> > to properly support the different sensor modules with a single kernel,
> > mostly because nobody developed a solution so far (although proposals
> > have been posted to mailing lists).
> > 
> > For that reason I currently maintain board code with sensor support for
> > several OMAP3 platforms in the omap3isp-sensors-board branch of my git
> > tree at http://git.linuxtv.org/pinchartl/media.git. I'd be happy to push
> > that to mainline if we had a good technical solution.
> 
> I was expecting that the code would be in there somewhere, but (for the
> reason you describe) not enabled by default. I was expecting that I might
> need to tweak the kernel config to enable the sensor support (and indeed I
> did), but I was very surprised that code necessary to make it actually work
> was nowhere in sight.
> 
> I would (naively, perhaps) think that it would be appropriate to have the
> code present, but enabled only if both MACH_OMAP3_BEAGLE and VIDEO_MT9P031
> are enabled.  What am I missing?

We're trying to move to a single kernel image for all ARMv7 platforms. A 
compile-time option wouldn't allow that., the option must be runtime-
configurable (possibly through a kernel command line argument, or through the 
device tree).

> I took a look at your branch (briefly). The Ubuntu 12.04 kernel is currently
> based on a roughly 3.2.x (maybe 3.2.18?) kernel. Do you happen to have a
> version of the board files that would be likely to work well on that
> vintage?

No, but it should be very easy to backport the code.

> Or for that matter, do you have a recommendation as to what (whose) kernel
> would be my best bet for production use in a BeagleBoard xM (rev C, maybe
> some B's) using the Leopard Imaging LI-5M03 camera board? I get the sense
> that the BB xM community is mostly on 3.2.x (or older) at the moment, and I
> hesitate to assume there isn't a good reason for that. But if some kernel
> newer than Ubuntu's is reasonably stable and trustworthy on a BB xM, I'd be
> happy enough to use it. (I do intend to continue with the Ubuntu 12.04 world
> for everything other than the kernel.)

I usually upgrade to the latest stable kernels when they are released, until 
there's a need to freeze the code base. Depending on your support 
requirements, picking up a long term stable version can be a good idea.

> > Where did you get the Aptina board code patch from ?
> 
>  From here: https://github.com/Aptina/BeagleBoard-xM

That's definitely outdated, the code is based on a very old OMAP3 ISP driver 
that was more or less broken by design. Nowadays anything other than the 
mainline version isn't supported by the community.

> Which is linked to from here:
> http://blog.galemin.com/2011/04/li-5m03-camera-on-beagleboard-xm/
> 
> I started in trying to get the cameras working on our BeagleBoards (as
> the first step in switching from Leopardboards, with the same cameras),
> and quickly found that it was not at all obvious how to get the MT9P031
> to be recognized in Ubuntu's kernel, nor in Robert C. Nelson's. I googled
> around and found Max Galemin's stuff, which got me a kernel that recognized
> the sensor but had some other quirks. Further googling in search of an
> answer to why it worked in Max's kernel but not the others - despite no
> visibly relevant difference in the driver sources - led me to the above, or
> maybe I followed the link from Max's blog - I don't remember at this point.
> The patch file in there was the most significant clue, and so I crossed my
> fingers and patched in the files under arch/arm, and sure enough, it worked. 
> Mostly...
> 
> >> 2. Max frame rate at full resolution seems to be 6.86 fps. I think we're
> >> running at half clock speed. We'd like to fix that. I can track it down,
> >> but I don't want to duplicate work already done by someone else, and of
> >> course this likely relates to issue # 1, above.
> > 
> > The clock speed is configurable, but the device is limited to 48 MHz when
> > using 1.8V I/O. To reach 96 MHz we would have to power the I/O supply with
> > 2.8V and add a level shifter on the board, as the OMAP3 use 1/8V I/O.
> 
> This is definitely something we'll want to pursue.  Some of the organisms
> we're trying to watch (this is for University of Washington Seattle's
> Oceanography Dept.) move fast enough that we want the full frame rate (~14
> fps) or closer to it. We can use cropping to increase the frame rate, but
> sometimes we need the full field of view as well.
> 
> Do you have or can you point me to more details on the mods needed, both in
> the software and in the hardware?  (Has anyone out there already done this
> successfully?)

I'm not aware of any sensor module with a 2.8V to 1.8V level shifter. If you 
want to ran the MT9P031 at 96 MHz with a Beagleboard-xM the best way would 
probably to design your own sensor board.

> >> 3. When I start streaming, then stop streaming, then start streaming
> >> again without closing and reopening the device in between (and sometimes
> >> even if I do but reopen right after closing), the second time we start
> >> streaming, it appears that the green and non-green (red or blue as the
> >> case may be) pixels are swapped - as if it was offset by one column. But
> >> if I change the cropping (using VIDIOC_SUBDEV_S_FMT on /ev/v4l-subdev8,
> >> which is the MT9P031 directly) to include the black (inactive) pixels on
> >> the top and left, it is still true - but the black pixels don't change,
> >> only the active ones, even though they still start at the same offset
> >> (+10,+50 IIRC). I don't even see how that should be possible.
> > 
> > Just to make sure I understand that properly, do you mean that the
> > boundary between the black and non-black pixels doesn't move, when you
> > expected it to be shifted by one column or one line if the color swap had
> > been caused by an image shift ?
> 
> Yes, precisely that.
> 
> > Did you include both black lines and black columns in your test ?
> 
> Yes.  I set the crop region to start at (0,0), so I got 10 black columns
> and 50 black lines, IIRC.
> 
> >> The MT9P031 registers (all of them) are the same whether the swapping is
> >> occurring or not, and ditto for the CCDC registers per the dump in the
> >> kernel log. Has anyone else seen this? I have worked around it for now by
> >> closing the device (all of them), sleeping for 2 ms, and then reopening
> >> and reconfiguring. However, I'd really like to find a proper solution, or
> >> at least understand the root cause - it's kind of disturbing, especially
> >> since without the sleep it still didn't reliably work correctly. This may
> >> also relate to issue # 1, above.
> > 
> > I've never experienced that issue and I'm quite surprised. I'd like to
> > check your board code first (although if all registers are identical it
> > shouldn't play a big role).
> 
> Agreed, I'm highly suspicious of my current configuration, given how I
> arrived at it.  :-)
> 
> > What's your test procedure ?
> 
> I'm not sure I understand what you're asking for. I have an app that
> configures everything (media controller, v4l-subdev8 for the mt9p031, etc.)
> and then starts streaming, capturing to a file or the network. The app can
> keep the device open, and under control of the UI, start and stop streaming
> repeatedly. After a while I figured out that the swapping/shifting problem
> often happened about every other time I started streaming (though not 100%
> consistently). Eventually I was able to find that closing the device and
> reopening it would generally avoid the problem; when I did that from the UI
> it worked pretty well, but not when I did it directly within the code, until
> I added the 2 ms sleep. So by the end, my test procedure was to start it,
> see if it was screwed up or not, stop it, and repeat. Once I added the
> sleep, it seemed to be good enough as a temporary solution - I couldn't
> delay any longer. The critters are in the water, and their schedule does not
> change no matter how good my excuses... :-/
> 
> >> 4. I need to add some additional controls (like a way to manipulate the
> >> vblank register setting so we can reduce the frame rate without just
> >> randomly dropping frames - we want to adjust the frame rate to what we
> >> can fairly reliably store without dropping frames - and access to the
> >> separate gain controls for R, Gr, Gb, and B, since we're using color
> >> sensors (cheaper) with IR illumination). I'd like to get some feedback on
> >> the most appropriate way to do this. Obviously I could just hack it in,
> >> but I'd rather do it right and hopefully get it into the mainline driver.
> >> In 3.5-rc2, I see a definition for a VBLANK control,
> > 
> > That's the one you should use.
> > 
> >> but it still isn't clear what ought to be used for separate gain
> >> controls.
> > 
> > We need new per-component gain controls. The usual way to do this is to
> > post a proposal to the linux-media mailing list, either in plain English
> > if you want initial feedback on a complex idea before implementing it, or
> > as a patch (don't forget to update
> > Documentation/DocBook/media/v4l/controls.xml).
>
> Thanks, I'll do that.  I wanted to find out what the consensus was on
> the general concept (e.g., do people agree that this is best addressed
> by exposing new per-component gain controls) before I took that step.
> I'll take your statement above as a sufficient consensus for now.  :-)
> 
> One sticky bit is that the driver currently implements a one-gain-to-fit-
> them-all approach. Should I replace that with the new ones (simple and clean
> but breaks current users of it), or try to figure out a way for them to co-
> exist in a meaningful and coherent way (appears hairy and error-prone, but
> preserves compatibility)?

We could remove the global gain control, and group the per-component controls 
in a cluster. That way, if the 4 gains are equal, the driver could perform a 
single I2C write instead of 4.

> >> 5. The driver (and likewise the CCDC driver) needs a few small fixes, and
> >> I'd like to avoid duplication of effort, etc.
> > 
> > The latest version of the driver with the patches that should be included
> > in the next kernel version is available from my git tree
> > (omap3isp-omap3isp-next branch, please note that the branch is regularly
> > rebased). If the fixes are not included there, the best way to avoid
> > effort duplication is to post a mail to the list with the description of
> > the problem(s). If nobody answers with a patch, you will have to write
> > one yourself. For very small fixes posting a patch up-front is usually
> > easier and even less time-consuming.
> 
> Okay, I'll post questions and/or patches here.  (Aiming for patches, but
> starting with questions where it seems appropriate.)
> 
> One other issue that it would be nice to address is why one or two frames
> (perhaps three, sometimes?) are corrupted at the start of streaming. I don't
> think it's inherent to the sensor, since we never saw that with the very
> same sensor and camera board on a Leopardboard (DM365). So it's either
> OMAP3-specific, or something in the drivers. I'm happy to help chase this
> down, but hoping someone else has already gotten a start on it.  Any clues
> out there?

How corrupt are they ?

-- 
Regards,

Laurent Pinchart

Re: hacking MT9P031 for i.mx

[Christoph Fritz]

Hi

On Mon, 2012-07-02 at 14:48 +0200, Laurent Pinchart wrote:
> On Thursday 28 June 2012 21:41:16 Chris MacGregor wrote:

> > > Where did you get the Aptina board code patch from ?
> > 
> >  From here: https://github.com/Aptina/BeagleBoard-xM
> 
> That's definitely outdated, the code is based on a very old OMAP3 ISP driver 
> that was more or less broken by design. Nowadays anything other than the 
> mainline version isn't supported by the community.

Is there a current (kernel ~3.6) git tree which shows how to add mt9p031
to platform code?

I'm also curious if it's possible to glue mt9p031 to a freescale i.mx35
platform. As far as I can see,
drivers/media/platform/soc_camera/mx3_camera.c would need v4l2_subdev
support?

Thanks
 -- Christoph

Re: hacking MT9P031 for i.mx

[Laurent Pinchart]

Hi Christoph,

On Friday 12 October 2012 14:10:43 Christoph Fritz wrote:
> On Mon, 2012-07-02 at 14:48 +0200, Laurent Pinchart wrote:
> > On Thursday 28 June 2012 21:41:16 Chris MacGregor wrote:
> > > > Where did you get the Aptina board code patch from ?
> > >  
> > >  From here: https://github.com/Aptina/BeagleBoard-xM
> > 
> > That's definitely outdated, the code is based on a very old OMAP3 ISP
> > driver that was more or less broken by design. Nowadays anything other
> > than the mainline version isn't supported by the community.
> 
> Is there a current (kernel ~3.6) git tree which shows how to add mt9p031
> to platform code?
> 
> I'm also curious if it's possible to glue mt9p031 to a freescale i.mx35
> platform. As far as I can see,
> drivers/media/platform/soc_camera/mx3_camera.c would need v4l2_subdev
> support?

soc-camera already uses v4l2_subdev, but requires soc-camera specific support 
in the sensor drivers. I've started working on a fix for that some time ago, 
some cleanup patches have reached mainline but I haven't been able to complete 
the work yet due to lack of time.

-- 
Regards,

Laurent Pinchart

Re: hacking MT9P031 for i.mx

[Laurent Pinchart]

On Friday 12 October 2012 14:10:43 Christoph Fritz wrote:
> On Mon, 2012-07-02 at 14:48 +0200, Laurent Pinchart wrote:
> > On Thursday 28 June 2012 21:41:16 Chris MacGregor wrote:
> > > > Where did you get the Aptina board code patch from ?
> > >  
> > >  From here: https://github.com/Aptina/BeagleBoard-xM
> > 
> > That's definitely outdated, the code is based on a very old OMAP3 ISP
> > driver that was more or less broken by design. Nowadays anything other
> > than the mainline version isn't supported by the community.
> 
> Is there a current (kernel ~3.6) git tree which shows how to add mt9p031
> to platform code?

Yes, at 
http://git.linuxtv.org/pinchartl/media.git/shortlog/refs/heads/omap3isp-
sensors-board

> I'm also curious if it's possible to glue mt9p031 to a freescale i.mx35
> platform. As far as I can see,
> drivers/media/platform/soc_camera/mx3_camera.c would need v4l2_subdev
> support?

-- 
Regards,

Laurent Pinchart

Re: hacking MT9P031 for i.mx

[Benoît Thébaudeau]

Hi all,

On Tuesday, October 16, 2012 10:04:57 PM, Laurent Pinchart wrote:
> On Friday 12 October 2012 14:10:43 Christoph Fritz wrote:
> > On Mon, 2012-07-02 at 14:48 +0200, Laurent Pinchart wrote:
> > > On Thursday 28 June 2012 21:41:16 Chris MacGregor wrote:
> > > > > Where did you get the Aptina board code patch from ?
> > > >  
> > > >  From here: https://github.com/Aptina/BeagleBoard-xM
> > > 
> > > That's definitely outdated, the code is based on a very old OMAP3
> > > ISP
> > > driver that was more or less broken by design. Nowadays anything
> > > other
> > > than the mainline version isn't supported by the community.
> > 
> > Is there a current (kernel ~3.6) git tree which shows how to add
> > mt9p031
> > to platform code?
> 
> Yes, at
> http://git.linuxtv.org/pinchartl/media.git/shortlog/refs/heads/omap3isp-
> sensors-board
> 
> > I'm also curious if it's possible to glue mt9p031 to a freescale
> > i.mx35
> > platform. As far as I can see,
> > drivers/media/platform/soc_camera/mx3_camera.c would need
> > v4l2_subdev
> > support?

I have not followed this thread, so I don't know exactly your issue, but FYI I
have an MT9M131 (of which the driver should hopefully be close to the MT9P031's)
working on i.MX35 with Linux 3.4.

I have local changes for that adding support for all possible formats to
mx3_camera and its IPU. I still have to upgrade to the latest Linux and to
prepare patches before posting them. I won't be able to do that before a few
weeks. However, if someone needs it, I can share my local changeset as a global
patch.

Best regards,
Benoît

Re: hacking MT9P031 for i.mx

[Christoph Fritz]

On Tue, Oct 16, 2012 at 11:04:36PM +0200, Benoît Thébaudeau wrote:
> On Tuesday, October 16, 2012 10:04:57 PM, Laurent Pinchart wrote:
> > > Is there a current (kernel ~3.6) git tree which shows how to add
> > > mt9p031
> > > to platform code?
> > 
> > Yes, at
> > http://git.linuxtv.org/pinchartl/media.git/shortlog/refs/heads/omap3isp-
> > sensors-board

Thanks!

> > > I'm also curious if it's possible to glue mt9p031 to a freescale
> > > i.mx35
> > > platform. As far as I can see,
> > > drivers/media/platform/soc_camera/mx3_camera.c would need
> > > v4l2_subdev
> > > support?
> 
> I have not followed this thread, so I don't know exactly your
> issue, but FYI I have an MT9M131 (of which the driver should
> hopefully be close to the MT9P031's) working on i.MX35 with Linux
> 3.4.

I have here a mt9p031-testing-board with an i.MX35 interface. So I'm
pretty interested in soc_camera support for mt9p031.

Laurent is already fixing this but haven't finished due to lack
of time.  When there is anything to test, I would be glad to do so.


Best regards
 -- Christoph

Re: hacking MT9P031 for i.mx

[Benoît Thébaudeau]

On Wednesday, October 17, 2012 11:14:06 AM, Christoph Fritz wrote:
> On Tue, Oct 16, 2012 at 11:04:36PM +0200, Benoît Thébaudeau wrote:
> > On Tuesday, October 16, 2012 10:04:57 PM, Laurent Pinchart wrote:
> > > > Is there a current (kernel ~3.6) git tree which shows how to
> > > > add
> > > > mt9p031
> > > > to platform code?
> > > 
> > > Yes, at
> > > http://git.linuxtv.org/pinchartl/media.git/shortlog/refs/heads/omap3isp-
> > > sensors-board
> 
> Thanks!
> 
> > > > I'm also curious if it's possible to glue mt9p031 to a
> > > > freescale
> > > > i.mx35
> > > > platform. As far as I can see,
> > > > drivers/media/platform/soc_camera/mx3_camera.c would need
> > > > v4l2_subdev
> > > > support?
> > 
> > I have not followed this thread, so I don't know exactly your
> > issue, but FYI I have an MT9M131 (of which the driver should
> > hopefully be close to the MT9P031's) working on i.MX35 with Linux
> > 3.4.
> 
> I have here a mt9p031-testing-board with an i.MX35 interface. So I'm
> pretty interested in soc_camera support for mt9p031.
> 
> Laurent is already fixing this but haven't finished due to lack
> of time.  When there is anything to test, I would be glad to do so.

OK. I meant that if my changes work for MT9M131, they might as well work for
MT9P031, except if there is another issue with the latter. So it's worth a try.

Anyway, I will post my changes as an RFC in a few minutes.

Best regards,
Benoît

Re: hacking MT9P031 for i.mx

[Laurent Pinchart]

Hi Christoph,

On Wednesday 17 October 2012 11:14:06 Christoph Fritz wrote:
> On Tue, Oct 16, 2012 at 11:04:36PM +0200, Benoît Thébaudeau wrote:
> > On Tuesday, October 16, 2012 10:04:57 PM, Laurent Pinchart wrote:
> > > > Is there a current (kernel ~3.6) git tree which shows how to add
> > > > mt9p031 to platform code?
> > > 
> > > Yes, at
> > > http://git.linuxtv.org/pinchartl/media.git/shortlog/refs/heads/omap3isp-
> > > sensors-board
> 
> Thanks!
> 
> > > > I'm also curious if it's possible to glue mt9p031 to a freescale
> > > > i.mx35 platform. As far as I can see,
> > > > drivers/media/platform/soc_camera/mx3_camera.c would need
> > > > v4l2_subdev support?
> > 
> > I have not followed this thread, so I don't know exactly your
> > issue, but FYI I have an MT9M131 (of which the driver should
> > hopefully be close to the MT9P031's) working on i.MX35 with Linux
> > 3.4.
> 
> I have here a mt9p031-testing-board with an i.MX35 interface. So I'm
> pretty interested in soc_camera support for mt9p031.
> 
> Laurent is already fixing this but haven't finished due to lack
> of time. When there is anything to test, I would be glad to do so.

Just to clarify the situation, I don't know when I'll have time to finish the 
patches, so if someone wants to take over please do (and CC me for review).

-- 
Regards,

Laurent Pinchart

Re: hacking MT9P031 for i.mx

[Richardson Leao]

 Laurent Pinchart <laurent.pinchart <at> ideasonboard.com> writes:

> 
> On Friday 12 October 2012 14:10:43 Christoph Fritz wrote:
> > On Mon, 2012-07-02 at 14:48 +0200, Laurent Pinchart wrote:
> > > On Thursday 28 June 2012 21:41:16 Chris MacGregor wrote:
> > > > > Where did you get the Aptina board code patch from ?
> > > >  
> > > >  From here: https://github.com/Aptina/BeagleBoard-xM
> > > 
> > > That's definitely outdated, the code is based on a very old OMAP3 ISP
> > > driver that was more or less broken by design. Nowadays anything other
> > > than the mainline version isn't supported by the community.
> > 
> > Is there a current (kernel ~3.6) git tree which shows how to add mt9p031
> > to platform code?
> 
> Yes, at 
> http://git.linuxtv.org/pinchartl/media.git/shortlog/refs/heads/omap3isp-
> sensors-board
> 
> > I'm also curious if it's possible to glue mt9p031 to a freescale i.mx35
> > platform. As far as I can see,
> > drivers/media/platform/soc_camera/mx3_camera.c would need v4l2_subdev
> > support?
> 


Dear Laurent,

sorry to bother you with this but I am stuck now for a couple of weeks trying to
make the LI-5M03 camera to work on the bb xm rev c. I am trying to build a
system to track neurons under a microscope. at the moment, I have the kernel
3.7.3 installed w the mt9p03 driver (I believe that it is a version that you
made) compiled as a module and ubuntu 12.10. The module loads fine, no complains
but I do not have /dev/mediaX or dev/videoX or any /dev/v4lXXX

I wonder if there are any tricks to have the device properly loaded. My linux
kernel and rootfs came from here:

http://www.eewiki.net/display/linuxonarm/BeagleBoard#BeagleBoard-LinuxBuildScript%3A

thanks immensely for your help!

richardson