Re: OV5640 autofocus kernel+gstreamer+Qt implementation

[Carlos Rafael Giani <dv@pseudoterminal.org>]

[-- Attachment #1: Type: text/plain, Size: 5816 bytes --]

Thanks, I integrated that patch into gstreamer-imx.
I am however thinking of copying the gstphotography files into 
gstreamer-imx until the GstPhotography API becomes stable, to avoid any 
future compilation problems due to API changes.

Carlos

Am 2015-09-29 um 22:10 schrieb Nikita Yushchenko:
> Hi list.
>
> I've recently been working on integrating autofocus functionality of
> OV5640 camera into software stack on imx6-based board.
>
> This mail contains patches to kernel, gstreamer-imx and Qt5 that make
> things somehow working. In particular, declarative-camera example
> focuses ok, custom Qt-based project also does.
>
>
> Kernel patches are against imx_3.14.28_1.0.0_ga branch.
>
> First patch is generic mxc_v4l2_capture cleanup. Unfortunately driver
> was quite racy and did not survive even streaming stop+start from within
> Qt. Had to do some cleanup to mitigate that.
>
> Second patch contains actual implementation of focus-related v4l ioctls.
>
> Autofocus firmware embedded into patch was taken from
>    http://crewrktablets.arctablet.com:8080/opendev/kernel_rockchip.git,
>    branch rockchip-3.10-rk3288
>    file drivers/media/video/ov5640_af_firmware.c
> and converted to format that existing driver uses. I've tried different
> firmwares found over the net, this one shows the best behaviour.
>
>
> For gstreamer, I had to add GstPhotography implementation stub to
> imxv4l2videosrc. Implementation is far incomplete, only focus-related
> operations are there.
>
> gstreamer-imx patch is against version 0.11.1
>
>
> For Qt, need several patches to qtmultimedia.  We use meta-qt5 layer,
> frodo branch, which is Qt 5.4 based. We had to backport several patches
> from 5.5 for better GStreamer 1.0 support. Also we had to create several
> new patches. Attaching a tarball with all these.
>
>
> With all these changes, single autofocus works ok with unmodified Qt's
> declarative-camera example.
>
> Easy way to check continuous autofocus is:
>
> - apply this to declarative-camera.qml
>
> --- a/declarative-camera.qml  2015-09-26 11:29:54.000000000 +0300
> +++ b/declarative-camera.qml  2015-09-16 13:12:51.000000000 +0300
> @@ -56,6 +56,7 @@
>               StateChangeScript {
>                   script: {
>                       camera.captureMode = Camera.CaptureStillImage
> +                    camera.focus.focusMode = Camera.FocusAuto
>                       camera.start()
>                   }
>               }
> @@ -68,6 +69,7 @@
>               StateChangeScript {
>                   script: {
>                       camera.captureMode = Camera.CaptureVideo
> +                    camera.focus.focusMode = Camera.FocusContinuous
>                       camera.start()
>                   }
>               }
>
> - start it via /usr/bin/qt5/qmlscene /path/to/declarative-camera.qml
> [running declarative-camera binary without recompilation won't get these
> changes since binary has qml embedded and does not re-read it from files
> at execution time]
>
> - from within demo, switch to "video" mode
>
> - take a sheet of paper with printed text and try moving in near camera
>
> - on playback screen, see camera's attempts to keep focused
>
>
>
> Known issues/limitations:
>
> *) Legal status of AF firmware is unclear.
>
> *) AF firmware is loaded using byte operations, loading using group
> writes is not implemented.
>
> *) AF firmware is loaded unconditionally on device open; better to do it
> on first focus-related operation.
>
> *) Reloading AF firmware on suspend-resume is not implemented. With
> current implementation, autofocus will stop working after suspend-resume
> cycle. To restore, need to close and reopen camera device (which usually
> means - restart camera-using program).
>
> Proper fix for the two above should be: make "firmware not loaded" a
> state of firmware-control automata and threat firmware loading as
> required step for requested state changes; when suspending, just force
> "firmware not loaded" state.
>
> *) Driver does not have protection against AF firmware misfunction (such
> as - operation not completes in reasonable time). Such misfuction
> sometimes does happen. It is not fatal (newly issued operation succeeds)
> but still incorrect.
>
> *) Setting camera parameters is not implemented. Camera always captures
> at 640x480 with all default settings.
>
> *) Middleware (QT plugin + gstreamer) is incomplete and quite unfriendly
> to debug and use. It basically works for trivial scenarios (streaming
> video from camera to viewfinder, starting/stopping, capturing images)
> but tends to fail on anything else. On failure, it just displays
> "CameraBin error" message and stops working. This happens in different
> cases, which includes inaccurate Qt application behavior (such as too
> long frame processing, failing to release buffer in time, etc).
> Extracting details on what happened is possible via GST_DEBUG but is
> quite cumbersome and requires understanding of gstreamer internals.
>
> *) 'imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_5 = 0x00000001'
> message sometimes appears
>
> *) 'ERROR: unrecognized std! 0 (PAL=ff, NTSC=b000)' message sometimes
> appears
>
> *) pressing Capture button in Qt C++ camera example
> (qt5/examples/multimediawidgets/camera/camera) causes CameraBin error
> (tried to investigate this, immediate reason is that after streaming
> restart, QPainterVideoSurface receives first two frames ok but does not
> become back ready after that and refuses 3rd frame; why this happens is
> not clear yet)
>
>
> Any comments on all this are welcome.
>
> Nikita Yushchenko,
> System Software Engineer at Software Development Center, RTsoft
>
>


[-- Attachment #2: Type: text/html, Size: 6166 bytes --]