Hi list.

I've recently been working on integrating autofocus functionality of
OV5640 camera into software stack on imx6-based board.

This mail contains patches to kernel, gstreamer-imx and Qt5 that make
things somehow working. In particular, declarative-camera example
focuses ok, custom Qt-based project also does.


Kernel patches are against imx_3.14.28_1.0.0_ga branch.

First patch is generic mxc_v4l2_capture cleanup. Unfortunately driver
was quite racy and did not survive even streaming stop+start from within
Qt. Had to do some cleanup to mitigate that.

Second patch contains actual implementation of focus-related v4l ioctls.

Autofocus firmware embedded into patch was taken from
  http://crewrktablets.arctablet.com:8080/opendev/kernel_rockchip.git,
  branch rockchip-3.10-rk3288
  file drivers/media/video/ov5640_af_firmware.c
and converted to format that existing driver uses. I've tried different
firmwares found over the net, this one shows the best behaviour.


For gstreamer, I had to add GstPhotography implementation stub to
imxv4l2videosrc. Implementation is far incomplete, only focus-related
operations are there.

gstreamer-imx patch is against version 0.11.1


For Qt, need several patches to qtmultimedia.  We use meta-qt5 layer,
frodo branch, which is Qt 5.4 based. We had to backport several patches
from 5.5 for better GStreamer 1.0 support. Also we had to create several
new patches. Attaching a tarball with all these.


With all these changes, single autofocus works ok with unmodified Qt's
declarative-camera example.

Easy way to check continuous autofocus is:

- apply this to declarative-camera.qml

--- a/declarative-camera.qml  2015-09-26 11:29:54.000000000 +0300
+++ b/declarative-camera.qml  2015-09-16 13:12:51.000000000 +0300
@@ -56,6 +56,7 @@
             StateChangeScript {
                 script: {
                     camera.captureMode = Camera.CaptureStillImage
+                    camera.focus.focusMode = Camera.FocusAuto
                     camera.start()
                 }
             }
@@ -68,6 +69,7 @@
             StateChangeScript {
                 script: {
                     camera.captureMode = Camera.CaptureVideo
+                    camera.focus.focusMode = Camera.FocusContinuous
                     camera.start()
                 }
             }

- start it via /usr/bin/qt5/qmlscene /path/to/declarative-camera.qml
[running declarative-camera binary without recompilation won't get these
changes since binary has qml embedded and does not re-read it from files
at execution time]

- from within demo, switch to "video" mode

- take a sheet of paper with printed text and try moving in near camera

- on playback screen, see camera's attempts to keep focused



Known issues/limitations:

*) Legal status of AF firmware is unclear.

*) AF firmware is loaded using byte operations, loading using group
writes is not implemented.

*) AF firmware is loaded unconditionally on device open; better to do it
on first focus-related operation.

*) Reloading AF firmware on suspend-resume is not implemented. With
current implementation, autofocus will stop working after suspend-resume
cycle. To restore, need to close and reopen camera device (which usually
means - restart camera-using program).

Proper fix for the two above should be: make "firmware not loaded" a
state of firmware-control automata and threat firmware loading as
required step for requested state changes; when suspending, just force
"firmware not loaded" state.

*) Driver does not have protection against AF firmware misfunction (such
as - operation not completes in reasonable time). Such misfuction
sometimes does happen. It is not fatal (newly issued operation succeeds)
but still incorrect.

*) Setting camera parameters is not implemented. Camera always captures
at 640x480 with all default settings.

*) Middleware (QT plugin + gstreamer) is incomplete and quite unfriendly
to debug and use. It basically works for trivial scenarios (streaming
video from camera to viewfinder, starting/stopping, capturing images)
but tends to fail on anything else. On failure, it just displays
"CameraBin error" message and stops working. This happens in different
cases, which includes inaccurate Qt application behavior (such as too
long frame processing, failing to release buffer in time, etc).
Extracting details on what happened is possible via GST_DEBUG but is
quite cumbersome and requires understanding of gstreamer internals.

*) 'imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_5 = 0x00000001'
message sometimes appears

*) 'ERROR: unrecognized std! 0 (PAL=ff, NTSC=b000)' message sometimes
appears

*) pressing Capture button in Qt C++ camera example
(qt5/examples/multimediawidgets/camera/camera) causes CameraBin error
(tried to investigate this, immediate reason is that after streaming
restart, QPainterVideoSurface receives first two frames ok but does not
become back ready after that and refuses 3rd frame; why this happens is
not clear yet)


Any comments on all this are welcome.

Nikita Yushchenko,
System Software Engineer at Software Development Center, RTsoft