Re: [RFC PATCH v2 00/18] Add DRM CRTC 3D LUT interface

[Pekka Paalanen <ppaalanen@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 11771 bytes --]

On Tue, 14 Feb 2023 11:19:47 +0200
Pekka Paalanen <ppaalanen@gmail.com> wrote:

> On Mon, 13 Feb 2023 18:26:55 -0100
> Melissa Wen <mwen@igalia.com> wrote:
> 
> > On 02/10, Pekka Paalanen wrote:  
> > > On Thu, 9 Feb 2023 13:27:02 -0100
> > > Melissa Wen <mwen@igalia.com> wrote:
> > >     
> > > > On 01/31, Pekka Paalanen wrote:    
> > > > > On Mon, 9 Jan 2023 14:38:09 -0100
> > > > > Melissa Wen <mwen@igalia.com> wrote:
> > > > >       
> > > > > > On 01/09, Melissa Wen wrote:      
> > > > > > > Hi,
> > > > > > > 
> > > > > > > After collecting comments in different places, here is a second version
> > > > > > > of the work on adding DRM CRTC 3D LUT support to the current DRM color
> > > > > > > mgmt interface. In comparison to previous proposals [1][2][3], here we
> > > > > > > add 3D LUT before gamma 1D LUT, but also a shaper 1D LUT before 3D LUT,
> > > > > > > that means the following DRM CRTC color correction pipeline:
> > > > > > > 
> > > > > > > Blend -> Degamma 1D LUT -> CTM -> Shaper 1D LUT -> 3D LUT -> Gamma 1D LUT      
> > > > > 
> > > > > Hi Melissa,
> > > > > 
> > > > > that makes sense to me, for CRTCs. It would be really good to have that
> > > > > as a diagram in the KMS UAPI documentation.
> > > > >       
> > > > 
> > > > Hi Pekka,
> > > > 
> > > > Thanks for your feedbacks and your time reviewing this proposal.    
> > > 
> > > No problem, and sorry it took so long!
> > > 
> > > I'm just finishing the catch-up with everything that happened during
> > > winter holidays.
> > >     
> > > > > If someone wants to add a 3D LUT to KMS planes as well, then I'm not
> > > > > sure if it should be this order or swapped. I will probably have an
> > > > > opinion about that once Weston is fully HDR capable and has been tried
> > > > > in the wild for a while with the HDR color operations fine-tuned based
> > > > > on community feedback. IOW, not for a long time. The YUV to RGB
> > > > > conversion factors in there as well.
> > > > >       
> > > > I see, this is also the reason I reuse here Alex Hung's proposal for
> > > > pre-blending API. I'll work on better documentation.
> > > >     
> > > > >       
> > > > > > > 
> > > > > > > and we also add a DRM CRTC LUT3D_MODE property, based on Alex Hung
> > > > > > > proposal for pre-blending 3D LUT [4] (Thanks!), instead of just a
> > > > > > > LUT3D_SIZE, that allows userspace to use different supported settings of
> > > > > > > 3D LUT, fitting VA-API and new color API better. In this sense, I
> > > > > > > adjusted the pre-blending proposal for post-blending usage.
> > > > > > > 
> > > > > > > Patches 1-6 targets the addition of shaper LUT and 3D LUT properties to
> > > > > > > the current DRM CRTC color mgmt pipeline. Patch 6 can be considered an
> > > > > > > extra/optional patch to define a default value for LUT3D_MODE, inspired
> > > > > > > by what we do for the plane blend mode property (pre-multiplied).
> > > > > > > 
> > > > > > > Patches 7-18 targets AMD display code to enable shaper and 3D LUT usage
> > > > > > > on DCN 301 (our HW case). Patches 7-9 performs code cleanups on current
> > > > > > > AMD DM colors code, patch 10 updates AMD stream in case of user 3D LUT
> > > > > > > changes, patch 11/12 rework AMD MPC 3D LUT resource handling by context
> > > > > > > for DCN 301 (easily extendible to other DCN families). Finally, from
> > > > > > > 13-18, we wire up SHAPER LUT, LUT3D and LUT3D MODE to AMD display
> > > > > > > driver, exposing modes supported by HW and programming user shaper and
> > > > > > > 3D LUT accordingly.
> > > > > > > 
> > > > > > > Our target userspace is Gamescope/SteamOS.
> > > > > > > 
> > > > > > > Basic IGT tests were based on [5][6] and are available here (in-progress):
> > > > > > > https://gitlab.freedesktop.org/mwen/igt-gpu-tools/-/commits/crtc-lut3d-api
> > > > > > > 
> > > > > > > [1] https://lore.kernel.org/all/20201221015730.28333-1-laurent.pinchart+renesas@ideasonboard.com/
> > > > > > > [2] https://github.com/vsyrjala/linux/commit/4d28e8ddf2a076f30f9e5bdc17cbb4656fe23e69
> > > > > > > [3] https://lore.kernel.org/amd-gfx/20220619223104.667413-1-mwen@igalia.com/
> > > > > > > [4] https://lore.kernel.org/dri-devel/20221004211451.1475215-1-alex.hung@amd.com/
> > > > > > > [5] https://patchwork.freedesktop.org/series/90165/
> > > > > > > [6] https://patchwork.freedesktop.org/series/109402/
> > > > > > > [VA_API] http://intel.github.io/libva/structVAProcFilterParameterBuffer3DLUT.html
> > > > > > > [KMS_pipe_API] https://gitlab.freedesktop.org/pq/color-and-hdr/-/issues/11
> > > > > > > 
> > > > > > > Let me know your thoughts.        
> > > > > > 
> > > > > > +Simon Ser, +Pekka Paalanen who might also be interested in this series.      
> > > > > 
> > > > > Unfortunately I don't have the patch emails to reply to, so here's a
> > > > > messy bunch of comments. I'll concentrate on the UAPI design as always.      
> > > > 
> > > > Sorry, the patchset is here: https://lore.kernel.org/dri-devel/20230109143846.1966301-1-mwen@igalia.com/
> > > > In the next version, I won't forget cc'ing you at first.    
> > > > > 
> > > > > +/*
> > > > > + * struct drm_mode_lut3d_mode - 3D LUT mode information.
> > > > > + * @lut_size: number of valid points on every dimension of 3D LUT.
> > > > > + * @lut_stride: number of points on every dimension of 3D LUT.
> > > > > + * @bit_depth: number of bits of RGB. If color_mode defines entries with higher
> > > > > + *             bit_depth the least significant bits will be truncated.
> > > > > + * @color_format: fourcc values, ex. DRM_FORMAT_XRGB16161616 or DRM_FORMAT_XBGR16161616.
> > > > > + * @flags: flags for hardware-sepcific features
> > > > > + */
> > > > > +struct drm_mode_lut3d_mode {
> > > > > +	__u16 lut_size;
> > > > > +	__u16 lut_stride[3];
> > > > > +	__u16 bit_depth;
> > > > > +	__u32 color_format;
> > > > > +	__u32 flags;
> > > > > +};

Btw. there is an odd number of u16 members before a u32 member. That
means there is invisible padding in this struct, I believe. I suppose
that's not different between 32-bit and 64-bit architectures, since
there are no 64-bit members, but it's still a hole. I assume the kernel
makes sure the hole cannot contain uninitialized values which would
leak kernel data to userspace...

For more things you might want to check, I think Daniel's botching up
ioctls article is appropriate. This is not an ioctl struct, but it's...
well, essentially it is. Just read-only.

https://www.kernel.org/doc/Documentation/ioctl/botching-up-ioctls.rst

...

> > > > > + * “LUT3D_MODE”:
> > > > > + *	Enum property to give the mode of the 3D lookup table to be set on the
> > > > > + *	LUT3D property. A mode specifies size, stride, bit depth and color
> > > > > + *	format and depends on the underlying hardware). If drivers support
> > > > > + *	multiple 3D LUT modes, they should be declared in a array of
> > > > > + *	drm_color_lut3d_mode and they will be advertised as an enum.
> > > > > 
> > > > > How does that work exactly? I didn't get it. I could guess, but having
> > > > > to guess on API is bad.      
> > > > 
> > > > The driver advertises all supported modes (each combination of values)
> > > > in a array as a enum, userspace can check all accepted modes and set the
> > > > one that fits the user 3D LUT settings. I think it's possible to get the
> > > > idea from this IGT test:
> > > > https://gitlab.freedesktop.org/mwen/igt-gpu-tools/-/commit/8771f444c3dcd126d7590d5a9b1b0db9706bbf6e#ed5dbc960ac210e3fbacd2361fe0270709767aaa_205_205    
> > > > >     
> > > 
> > > You lost me at "an array as an enum".
> > > 
> > > I understand there is a blob containing an array of struct
> > > drm_mode_lut3d_mode. What I don't understand is that you say LUT3D_MODE
> > > is an enum property. Where does the blob come from, then? What property
> > > provides the blob?
> > > 
> > > Am I correct in guessing that the values of LUT3D_MODE enum property
> > > are indices into the array in the blob, and that userspace will set it?
> > > That sounds good to me, if it's the integer value of the enum. But enum
> > > values also need string names, because that is how the values are
> > > usually recognized, so what name strings will be used?    
> > 
> > So, in this proposal, LUT3D_MODE is a list of indexes for a blob that describe a supported
> > 3D LUT mode  
> 
> Hi Melissa,
> 
> are you sure? I believe you are looking at and explaining some kernel
> internal APIs, and not the uAPI which is the important part. Internal
> APIs can always be changed later, uAPI cannot.
> 
> After I had sent that email, I might have understood how it was
> supposed to work instead: there is no array of struct
> drm_mode_lut3d_mode.
> 
> Instead, LUT3D_MODE is a KMS property of type enum. Each integer value
> of the enum is also a blob id. Each blob named by those ids is a single
> struct drm_mode_lut3d_mode that userspace needs to retrieve individually
> to understand what that specific enum value means.
> 
> > i.e. a `struct drm_mode_lut3d_mode` with size, stride, bit depth, etc. Strings here follow this pattern
> > `lut3d_{size}_{bit_depth}bit` [1]. When enabling 3D LUT support, the
> > driver should pass an array of `struct drm_mode_lut3d_mode` as supported
> > modes, with at least one element.  
> 
> To be clear, I do not care about kernel internal interfaces at all. I
> only care about the uAPI. Therefore talking about kernel internal API
> will only confuse me and every other userspace developer. Evidently it
> has even confused some IGT developers, as I point out some IGT code
> problems below.
> 
> If the struct drm_mode_lut3d_mode is the authoritative definition of
> what each enum value means, then I think the string names should not
> attempt to convey any meaning nor information. The strings should be
> just "blob id 77" etc. That makes it clear what the value is.

One more thing came to mind: extendability.

If it turns out that struct drm_mode_lut3d_mode is not good enough for
some future hardware, what could we do? Do you have a plan for that?

A) Thinking again, I would suggest to make the enum value string names
of the format "drm_mode_lut3d_mode %u". That way the name identifies the
blob binary structure and the blob id makes it unique. If we hope that
userspace checks the name, the kernel could later add a different
structure when necessary. Even if userspace fails in that, it would
still be an informative name.

B) Otherwise, or in addition, the structure needs to be designed to be
extendable like ioctl structs. A flags field maybe? But again we trust
that userspace checks the flags before trying to interpret any further.

C) Yet another option is a new DRM client CAP, which userspace can set
to tell the kernel that userspace does understand and expect that some
new kind of structure might be present. But this needs also A or B to
identify the struct.

D) Invent a whole new KMS property for the next generation 3D LUTs. In
that case the new and old properties probably need to be mutually
exclusive.


> Otherwise userspace will be tempted to use the string names only (as is
> the usual uAPI design) to find the right enum value, and not look into
> the struct drm_mode_lut3d_mode at all. Therefore, if you don't encode
> absolutely everything of struct drm_mode_lut3d_mode into the string
> name, you should encode nothing in the name. You might also have two
> different descriptions ending up with the same string name under the
> same enum property instance, and that must not happen.


Thanks,
pq

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]