Re: [RFC] Motion Detection API

[Sylwester Nawrocki <sylvester.nawrocki@gmail.com>]

Hi Sakari,

On 05/21/2013 07:30 PM, Sakari Ailus wrote:
> Hi Sylwester,
>
> My apologies for the late answer.

No problem at all, thank you for your follow up.

> Sylwester Nawrocki wrote:
>> On 05/08/2013 06:26 PM, Sakari Ailus wrote:
>>> On Tue, May 07, 2013 at 04:04:10PM +0200, Sylwester Nawrocki wrote:
>>>> On 05/07/2013 02:35 PM, Hans Verkuil wrote:
>>>>> A metadata plane works well if you have substantial amounts of data
>>>>> (e.g. histogram
>>>>> data) but it has the disadvantage of requiring you to use the MPLANE
>>>>> buffer types,
>>>>> something which standard apps do not support. I definitely think
>>>>> that is overkill for things like this.
>>>>
>>>> Standard application could use the MPLANE interface through the
>>>> libv4l-mplane
>>>> plugin [1]. And meta-data plane could be handled in libv4l, passed in
>>>> raw form from the kernel.
>>>>
>>>> There can be substantial amount of meta-data per frame and we were
>>>> considering
>>>> e.g. creating separate buffer queue for meta-data, to be able to use
>>>> mmaped
>>>> buffer at user space, rather than parsing and copying data multiple
>>>> times in
>>>> the kernel until it gets into user space and is further processed
>>>> there.
>>>
>>> What kind of metadata do you have?
>>
>> At least I can tell of three kinds of meta-data at the moment:
>>
>> a) face/smile/blink detection markers (rectangles), see struct
>> is_face_marker
>> in file [1] in the media tree for more details; these markers can be
>> available after an image frame is dequeued AFAIK, i.e. not immediately
>> together with the image data,
>>
>> b) EXIF tags (struct exif_attribute in file [1]), it's a preprocessed by
>> the ISP metadata appended to each buffer,
>
> This class includes other image file metadata types such as iptc and xmp.

Right, thanks for pointing it out.

It seems EXIF is most useful for device related image properties though.
At least this is my understanding from reading the "Guidelines For Handling
Image Metadata" [1].

XMP uses XML schemas and it seems more relevant to processing data in user
space applications. Although it defines various namespaces [2] and could be
a container of a set of EXIF-specific properties.

>> c) the object detection bitmap, and this one can have size comparable to
>> the actual image frame; I didn't see how it works in practice yet
>> though.
>>
>> For b) I have been re-considering using EXIF standard (chapter 4.6,
>> [2]) to
>> create some sane interface for the ISP driver.
>>
>> From performance POV only c) would need a meta-data specific buffer
>> queue, as
>> such data has similar characteristics to the actual image data and a DMA
>> engine
>> is used to capture those bitmaps.
[...]
>>> I'm in favour of using a separate video buffer queue for passing
>>> low-level
>>> metadata to user space.
>>
>> Sure. I certainly see a need for such an interface. I wouldn't like to
>> see it
>> as the only option, however. One of the main reasons of introducing
>> MPLANE
>> API was to allow capture of meta-data. We are going to finally prepare
>> some
>> RFC regarding usage of a separate plane for meta-data capture. I'm not
>> sure
>> yet how it would look exactly in detail, we've just discussed this topic
>> roughly with Andrzej.
>
> I'm fine that being not the only option; however it's unbeatable when it
> comes to latencies. So perhaps we should allow using multi-plane buffers
> for the same purpose as well.
>
> But how to choose between the two?

I think we need some example implementation for metadata capture over
multi-plane interface and with a separate video node. Without such
implementation/API draft it is a bit difficult to discuss this further.

> In complex media devices the metadata is written into the system memory in
> an entirely different place than the images themselves which typically
> require processing (especially if the sensor produces raw bayer images).
> This would likely mean that there will be multiple device nodes in this
> kind of situations.
>
> That said, in both cases extra infrastructure is required for configuring
> metadata format (possibly hardware-specific) and passing the control
> information between the subdev drivers. This requires new interfaces to the
> V4L2 subdev API. I'd think this part will still be common for both
> approaches.
>
>>> On some devices it seems the metadata consists of much higher level
>>> information.
>>
>> Indeed. It seems in case of devices like OMAP3 ISP we need to deal
>> mostly with
>> raw data from a Bayer sensor, while for the Exynos ISP I would need to
>> expose
>> something produced by the standalone ISP from such a raw metadata.
>
> Can the Exynos ISP also process raw bayer input, or is it YUV only?

Exynos4212 an later have a full camera ISP subsystem and can process raw
bayer data, as opposed to the Exynos4210 and earlier SoCs, where the camera
subsystem was supporting at most capturing raw bayer data transparently
to memory.

Unfortunately the Exynos Imaging Subsystem documentation is not yet
publicly available.

> I remember I have heard of and seen external ISPs but never have used
> those myself.

I think it might not matter that much if an ISP is external or local to
the main CPU/SoC. Presumably it may mean that different data busses/
mechanisms are used to communicate with the actual hardware, which should
be encapsulated by a driver anyway.

Thanks,
Sylwester

[1] http://www.metadataworkinggroup.org/specs
[2] 
http://www.adobe.com/content/dam/Adobe/en/devnet/xmp/pdfs/XMPSpecificationPart2.pdf