[RFC] More on subdev selections API: composition

[RFC] More on subdev selections API: composition

[Sakari Ailus]

Hi all,

I had a discussion with Tomasz a few days ago on the selection API and how
the composition fits to the proposal I made some time ago. I understand that
in V4L2 API the composition bounds rectangle, onto which the scaled images
are composed, is static in size. This makes no sense for subdevs as far as I
can see.

In composition use cases there are also more than one sink pad whereas
otherwise there is just a single pad in a subdev. In all use cases more than
one source pad likely isn't uncommon.

The problem with multiple sink pads is that which one you're referring to
when you're configuring the first processing step on the source. Without
composition there are no issues.

What I can think of is to create a special composition target which is not
bound to any pad, but reflects the size of the rectangle on which streams
may be composed from source pad. Cropping on source pad refers to the
coordinates of the composition rectangle. Composition target on sink pad in
the original proposal would be renamed as the scaling target. There would
also be no composition on source pads as it does not make that much sense.

To make configuration simple, accessing any unsupported rectangles should
return EINVAL. So devices not supporting composition would work as proposed
earlier: the compose rectangle would be omitted and the sink crop (if
supported) would refer to either scaling or crop targets or even the sink
format directly.

<URL:http://www.retiisi.org.uk/v4l2/tmp/format2.eps>

Alternatively I think we could as well drop composition support at this
point as we have no drivers using it. We still need to plan ahead how it
could be supported as the need likely arises at some point. As far as I see
the current interface proposal is compatible with composition.

Should we discuss this further on #v4l-meeting, I propose Tuesday 2012-01-31
15:00 Finnish time (13:00 GMT).

Kind regards,

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk

Re: [RFC] More on subdev selections API: composition

[Sylwester Nawrocki]

Hi Sakari,

On 01/29/2012 07:06 PM, Sakari Ailus wrote:
> The problem with multiple sink pads is that which one you're referring to
> when you're configuring the first processing step on the source. Without
> composition there are no issues.
> 
> What I can think of is to create a special composition target which is not
> bound to any pad, but reflects the size of the rectangle on which streams
> may be composed from source pad. Cropping on source pad refers to the
> coordinates of the composition rectangle. Composition target on sink pad in
> the original proposal would be renamed as the scaling target. There would

I would prefer to stick with CROP and COMPOSE target names on both video and
subdev nodes, not to add unnecessary confusion. 

> also be no composition on source pads as it does not make that much sense.

> To make configuration simple, accessing any unsupported rectangles should
> return EINVAL. So devices not supporting composition would work as proposed
> earlier: the compose rectangle would be omitted and the sink crop (if
> supported) would refer to either scaling or crop targets or even the sink
> format directly.
> 
> <URL:http://www.retiisi.org.uk/v4l2/tmp/format2.eps>
> 
> Alternatively I think we could as well drop composition support at this
> point as we have no drivers using it. We still need to plan ahead how it

No, that's not true there is no drivers supporting composition. On S5P FIMC
I currently use crop on source pad to configure composition rectangle on the
output buffers. The drivers forms following pipeline:

/dev/subdev?   /dev/subdev?    /dev/video? 
+--------+      +--------+      +-----+
| sensor |o----o| scaler |o----o| DMA |
+--------+      +--------+      +-----+
pads:     S0   SC0      SC1    D0

Physically the scaler and DMA are tightly coupled in one platform device, hence
SC1 <-> D0 link is immutable.
Format on D0 link is currently always same as configured with VIDIOC_S_FMT on
/dev/video?.

What is needed here seems just crop on sink pad (SC0) and compose on source pad
(SC1). But I'm not so sure about it, given your interpretation and after short
(well, not so) discussion with Tomasz.

TBH all functionality of the device could be exposed with the scaler subdev
removed and VIDIOC_S/G_SELECTION used on /dev/video?. It is getting overly
complicated with all the subdevs as above in place and your guys interpretation
of the subdev selection.

> could be supported as the need likely arises at some point. As far as I see
> the current interface proposal is compatible with composition.
> 
> Should we discuss this further on #v4l-meeting, I propose Tuesday 2012-01-31
> 15:00 Finnish time (13:00 GMT).


Regards,
-- 
Sylwester Nawrocki
Samsung Poland R&D Center

[ANN] Notes on subdev selections API on #v4l-meeting 2012-01-31

[Sakari Ailus]

Hi all,

We had a meeting on #v4l-meeting on the topic of subdev selections, and I
can summarise the conclusions as follows. The full meeting log is available
in [3].

Two main topics were discussed, the behaviour and functionality of the
selection API and the naming of different rectangles. A conclusion was
reached in both.

- The sink COMPOSE rectangle in my latest proposal [1] is a hardware limit
rather than something user configurable. This means the rectangle is
correctly considered as COMPOSE BOUNDS target in V4L2 selection API as
well.

  - This is actually correct in the previous proposal [2].

  - There are no dependencies between sink / source pad coordinates as such:
  both refer to the coordinates of the sink COMPOSE BOUNDS rectangle. Source
  CROP BOUNDS also refer to the same, and this rectangle might be different
  from sink COMPOSE BOUNDS rectangle.

- The COMPOSE rectangle will, as it was in the previous proposal, continue
to be called the COMPOSE rectangle (and not SCALING rectangle).

- The definition of the BOUNDS rectangle is "smallest rectangle which
contains all valid ACTIVE rectangles". This does not specify what is the
largest possible ACTIVE rectangle.

- It was agreed to drop DEFAULT targets for now. The same had been done
to PADDED targets already previously.

- More documentation is needed: a few sample use cases to be included into
documentation.

- No need for COMPOSE on source pads, so COMPOSE rectangle is will not be a
valid target on source pads. Sylwester's use case [4], composing into video
buffers, can be equally implemented using the compose rectangle on the sink
pad. It is to be discussed should this be rather implemented using the
selection API on the video node instead. The source pad image size will be
equal to the size of the source crop rectangle.

- Information to the diagram on what rectangles refer to which coordinates
to be added.

I will also make the necessary changes to my selection patchset and re-post
it.

[1] http://www.spinics.net/lists/linux-media/msg43723.html

[2] http://www.spinics.net/lists/linux-media/msg42991.html

[3] http://www.retiisi.org.uk/v4l2/notes/v4l2-selection-api-2012-01-31.txt

[4] http://www.spinics.net/lists/linux-media/msg43746.html

Cheers,

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk