Re: per-frame camera metadata (again)

[Laurent Pinchart <laurent.pinchart@ideasonboard.com>]

Hello Karthik,

On Tuesday 22 December 2015 05:30:52 karthik poduval wrote:
> I have been wanting to share these thoughts for the group but was
> waiting for the right time which I think is now since Guennadi brought
> up this discussion.
> 
> For the Amazon Fire phone 4 corner camera, here is how we passed
> metadata from driver to application (which was a CV client requiring
> per frame metadata).
> 
> We took an unused field in struct v4l2_buffer (__u32 reserved in this
> case) and used it to pass in a pointer to a user space metadata object
> (i.e. struct app_metadata) to the driver via the VIDIOC_DQBUF ioctl
> call.
> 
> struct v4l2_buffer for reference.
> http://lxr.free-electrons.com/source/include/uapi/linux/videodev2.h#L836
> 
> The driver copied its local copy of the metadata object to the
> userspace metadata object using the copy_to_user primitive offered by
> the kernel.
> 
> Here is how we handled the metadata in the driver code.
> https://github.com/Fire-Phone/android_kernel_amazon_kodiak/blob/master/drive
> rs/media/platform/msm/camera_v2/camera/camera.c#L235
> 
> This was done before HAL V3 was available. With HAL V3, the metadata
> object can be the HAL v3 metadata buffer. Non Android devices can use
> custom metadata format (like the one we used).
> 
> With this approach, the metadata always accompanies the frame data as
> it's available along with the frame buffer inside struct v4l2_buffer
> from the VIDIOC_DQBUF ioctl call.
> 
> If the community likes this idea, the v4l2_buffer can now be
> officially modified to contain a pointer to user space metadata object
> v4l2_buffer.metadata and then metadata format and size can be agreed
> upon between application and driver.
> Thoughts ?

I see several issues with that approach. The first one is that the meta-data 
buffer is only passed at DQBUF time. Drivers thus need to copy data using the 
CPU instead of capturing meta-data directly to the buffer through DMA. If the 
meta-data size is small the performance impact can be negligible, but that 
might not be true in the general case.

A second issue is that the approach isn't generic enough in my opinion. If we 
want to attach additional data buffers to a v4l2_buffer I agree with Sakari 
that we should design a multi-part buffer API in order to not limit the 
implementation to meta-data, but support other kind of information such as 
statistics for instance.

Finally, it might be beneficial in some use cases to pass meta-data to 
userspace before the end of the frame (assuming meta-data is available earlier 
of course). That's certainly true for statistics, use cases for meta-data are 
less clear to me though.

-- 
Regards,

Laurent Pinchart