Re: [RFC PATCH v2 06/17] drm/doc/rfc: Describe why prescriptive color pipeline is needed

[Pekka Paalanen <ppaalanen@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 9203 bytes --]

On Thu, 9 Nov 2023 10:17:11 +0000
"Shankar, Uma" <uma.shankar@intel.com> wrote:

> > -----Original Message-----
> > From: Joshua Ashton <joshua@froggi.es>
> > Sent: Wednesday, November 8, 2023 7:13 PM
> > To: Shankar, Uma <uma.shankar@intel.com>; Harry Wentland
> > <harry.wentland@amd.com>; dri-devel@lists.freedesktop.org

...

> > Subject: Re: [RFC PATCH v2 06/17] drm/doc/rfc: Describe why prescriptive color
> > pipeline is needed
> > 
> > 
> > 
> > On 11/8/23 12:18, Shankar, Uma wrote:  
> > >
> > >  
> > >> -----Original Message-----
> > >> From: Harry Wentland <harry.wentland@amd.com>
> > >> Sent: Friday, October 20, 2023 2:51 AM
> > >> To: dri-devel@lists.freedesktop.org

...

> > >> Subject: [RFC PATCH v2 06/17] drm/doc/rfc: Describe why prescriptive
> > >> color pipeline is needed

...

> > >> +An example of a drm_colorop object might look like one of these::
> > >> +
> > >> +    /* 1D enumerated curve */
> > >> +    Color operation 42
> > >> +    ├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3
> > >> + matrix, 3x4
> > >> matrix, 3D LUT, etc.} = 1D enumerated curve
> > >> +    ├─ "BYPASS": bool {true, false}
> > >> +    ├─ "CURVE_1D_TYPE": enum {sRGB EOTF, sRGB inverse EOTF, PQ EOTF,
> > >> + PQ
> > >> inverse EOTF, …}  
> > >
> > > Having the fixed function enum for some targeted input/output may not
> > > be scalable for all usecases. There are multiple colorspaces and
> > > transfer functions possible, so it will not be possible to cover all
> > > these by any enum definitions. Also, this will depend on the capabilities of  
> > respective hardware from various vendors.
> > 
> > The reason this exists is such that certain HW vendors such as AMD have transfer
> > functions implemented in HW. It is important to take advantage of these for both
> > precision and power reasons.  
> 
> Issue we see here is that, it will be too usecase and vendor specific.
> There will be BT601, BT709, BT2020, SRGB, HDR EOTF and many more. Not to forget
> we will need linearization and non-linearization enums for each of these.

I don't see that as a problem at all. It's not a combinatorial
explosion like input/output combinations in a single enum would be.
It's always a curve and its inverse at most.

It's KMS properties, not every driver needs to implement every
defined enum value but only those values it can and wants to support.
Userspace also sees the supported list, it does not need trial and
error.

This is the only way to actually use hard-wired curves. The
alternative would be for userspace to submit a LUT of some type, and
the driver needs to start guessing if it matches one of the hard-wired
curves the hardware supports, which is just not feasible.

Hard-wired curves are an addition, not a replacement, to custom
curves defined by parameters or various different LUT representations.
Many of these hard-wired curves will emerge as is from common use cases.

> Also 
> a CTM indication to convert colospace.

Did someone propose to enumerate matrices? I would not do that, unless
you literally have hard-wired matrices in hardware and cannot do custom
matrices.

> Also, if the underlying hardware block is 
> programmable, its not limited to be used only for the colorspace management but
> can be used for other color enhancements as well by a capable client.

Yes, that's why we have other types for curves, the programmable ones.

> Hence, we feel that it is bordering on being descriptive with too many possible
> combinations (not easy to generalize). So, if hardware is programmable, lets
> expose its capability through a blob and be generic.

It's not descriptive though. It's a prescription of a mathematical
function the hardware implements as fixed-function hardware. The
function is a curve. There is no implication that the curve must be
used with specific input or output color spaces.

> For any fixed function hardware where Lut etc is stored in ROM and just a control/enable
> bit is provided to driver, we can define a pipeline with a vendor specific color block. This
> can be identified with a flag (better ways can be discussed). 

No, there is no need for that. A curve type will do well.

A vendor specific colorop needs vendor specific userspace code to
program *at all*. A generic curve colorop might list some curve types
the userspace does not understand, but also curve types userspace does
understand. The understood curve types can still be used by userspace.

> For example, on some of the Intel platform, we had a fixed function to convert colorspaces
> directly with a bit setting. These kinds of things should be vendor specific and not be part
> of generic userspace implementation.

Why would you forbid generic userspace from making use of them?

> For reference:
> 001b	YUV601 to RGB601 YUV BT.601 to RGB BT.601 conversion.
> 010b	YUV709 to RGB709 YUV BT.709 to RGB BT.709 conversion.
> 011b	YUV2020 to RGB2020 YUV BT.2020 to RGB BT.2020 conversion.
> 100b	RGB709 to RGB2020 RGB BT.709 to RGB BT.2020 conversion.

This is nothing like the curves we talked about above.

Anyway, you can expose these fixed-function operations with a colorop
that has an enum choosing the conversion. There is no need to make it
vendor-specific at all. It's possible that only specific chips from
Intel support it, but nothing stops anyone else from implementing or
emulating the colorop if they can construct a hardware configuration
achieving the same result.

It seems there are already problems in exploding the number of
pipelines to expose, so it's best to try to avoid single-use colorops
and use enums in more generic colorops instead.

> 
> > Additionally, not every vendor implements bucketed/segemented LUTs the same
> > way, so it's not feasible to expose that in a way that's particularly useful or not
> > vendor-specific.  

Joshua, I see no problem here really. They are just another type of LUT
for a curve colorop, with a different configuration blob that can be
defined in the UAPI.

> If the underlying hardware is programmable, the structure which we propose to advertise
> the capability of the block to userspace will be sufficient to compute the LUT coefficients.
> The caps can be :
> 1. Number of segments in Lut
> 2. Precision of lut
> 3. Starting and ending point of the segment
> 4. Number of samples in the segment.
> 5. Any other flag which could be useful in this computation.
> 
> This way we can compute LUT's generically and send to driver. This will be scalable for all
> colorspaces, configurations and vendors.

Drop the mention of colorspaces, and I hope so. :-)

Color spaces don't quite exist in a prescriptive pipeline definition.

> > Thus we decided to have a regular 1D LUT modulated onto a known curve.
> > This is the only real cross-vendor solution here that allows HW curve
> > implementations to be taken advantage of and also works with
> > bucketing/segemented LUTs.
> > (Including vendors we are not aware of yet).
> > 
> > This also means that vendors that only support HW curves at some stages without
> > an actual LUT are also serviced.  
> 
> Any fixed function vendor implementation should be supported but with a vendor
> specific color block. Trying to come up with enums which aligns with some underlying
> hardware may not be scalable.

I disagree with both of you.

Who said there could be only one "degamma" block on a plane's pipeline?

If hardware is best modelled as a fixed-function selectable curve
followed by a custom curve, then expose exactly those two generic
colorops. Nothing stops a pipeline from having two curve colorops in
sequence with a disjoint set of supported types or features. If some
hardware does not have one of the curve colorops, then just don't add
the missing one in a pipeline.


Thanks,
pq

> > You are right that there *might* be some usecase not covered by this right now,
> > and that it would need kernel churn to implement new curves, but unfortunately
> > that's the compromise that we (so-far) have decided on in order to ensure
> > everyone can have good, precise, power-efficient support.  
> 
> Yes, we are aligned on this. But believe programmable hardware should be able to
> expose its caps. Fixed function hardware should be non-generic and vendor specific.
> 
> > It is always possible for us to extend the uAPI at a later date for other curves, or
> > other properties that might expose a generic segmented LUT interface (such as
> > what you have proposed for a while) for vendors that can support it.
> > (With the whole color pipeline thing, we can essentially do 'versioning'
> > with that, if we wanted a new 1D LUT type.)  
> 
> Most of the hardware vendors have programmable luts (including AMD), so it would be
> good to have this as a default generic compositor implementation. And yes, any new color
> block with a type can be added to the existing API's as the need arises without breaking
> compatibility.
> 
> Regards,
> Uma Shankar
> 
> > 
> > Thanks!
> > - Joshie 🐸✨

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]