[PATCH 0/2] drm/amd/display: fix BT.2020 YCbCr output CSC matrices

[Nathan Lucas <nlucasgit@gmail.com>]

This patch series fixes COLOR_SPACE_YCBCR2020_TYPE in dc_hw_sequencer.c
which is used for COLOR_SPACE_2020_YCBCR_LIMITED output but does not have
correct luma and chroma scaling for that output range, and fixes its
copies in dce_transform.c and dce110_opp_csc_v.c.

It looks like this matrix was originally added in commit 40df2f809e8f
("drm/amd/display: color space ycbcr709 support") as a raw BT.2020-NCL RGB
to YCbCr conversion matrix that had not yet been prepared for unsigned
limited or full-range quantization.  Commit 973a9c810c78
("drm/amd/display: Fix COLOR_SPACE_YCBCR2020_TYPE matrix") applied offsets
for unsigned limited-range quantization but did not fix the luma and chroma
scales which are still full range.  Because of this, output on a calibrated
display is too bright with incorrect color, although it still looks roughly
correct at low luminance in my testing.

Later, in commit 51e6668ab4ba
("drm/amd/display: add missing CSC entries for BT.2020 for DCE IPs"), the
incorrect COLOR_SPACE_YCBCR2020_TYPE matrix coefficients were duplicated to
DCE COLOR_SPACE_2020_YCBCR_LIMITED and COLOR_SPACE_2020_YCBCR_FULL, so
those matrices are also currently incorrect.

The first patch adds separate matrices for full and limited-range.  The
derivation is in the commit message.  The second patch replaces the copied
matrix coefficients in dce_transform.c and dce110_opp_csc_v.c with the new
ones.

For testing I have only been able to check the limited-range DCN path on a
9070 XT with a patched kernel.  amdgpu_dm_get_output_color_space does not
currently have a path to select COLOR_SPACE_2020_YCBCR_FULL although the
enum value exists.  Also, I cannot test the DCE paths as I do not have the
hardware to do so.

I ran into this issue measuring PQ test patterns through mpv with the
following command line:

mpv \
  --no-config \
  --vo=gpu-next \
  --gpu-context=displayvk \
  --gpu-api=vulkan \
  --vulkan-display-plane=3 \
  --vulkan-display-mode=22 \
  --target-trc=pq \
  --target-prim=bt.2020 \
  --target-colorspace-hint-mode=source \
  --target-colorspace-hint=yes \
  --tone-mapping=clip \
  --hdr-compute-peak=no \
  [pattern file]

Free test patterns:
https://diversifiedvideosolutions.com/hdr-10.html

Below is a table of values I measured on a Samsung S95H (US model) with a
Calibrite Display Pro HL meter.

Three sets of meter readings below:
Unpatched - Unpatched Fedora 44 kernel 7.1.4-200.fc44.x86_64
Patched   - The patches in these e-mails applied to Fedora 44 kernel
            7.1.4-200.fc44.x86_64
NVS/Kodi  - NVIDIA Shield Pro with Kodi

 5% HDR10/PQ
 Test Pattern        Meter Readings (nits)
==============   ==============================
PQ%       Nits   Unpatched   Patched   NVS/Kodi
15        1.01        1.63      1.01     1.02
25        5.24        9.68      5.46     5.12
35       18.78       40        20.5     21.2
45       56.55      128        62.0     62.1
55      148.03      369       155      155
65      384.71     1106       403      401
75      973.13     2681*     1030     1025

* This is at the peak capability for this TV.

PQ brightness overtracks a bit in Patched and NVS/Kodi, but that is likely
due to the TV's default calibration or my meter since the readings are so
similar.  Unpatched stands out as being way too bright.

Nathan Lucas (2):
  drm/amd/display: fix BT.2020 YCbCr limited output CSC matrix
  drm/amd/display: fix BT.2020 YCbCr output CSC matrices for DCE

 .../drm/amd/display/dc/core/dc_hw_sequencer.c | 31 ++++++++++++-------
 .../drm/amd/display/dc/dce/dce_transform.c    |  7 +++--
 .../amd/display/dc/dce110/dce110_opp_csc_v.c  |  7 +++--
 3 files changed, 28 insertions(+), 17 deletions(-)

-- 
2.55.0