Re: [PATCH RFC] dt-bindings: display: document display panel occlusions

[Rob Herring <robh@kernel.org>]

On Mon, Oct 09, 2023 at 06:32:50PM +0100, Caleb Connolly wrote:
> Some display panels found in modern phones and laptops feature
> non-standard display shapes with features like rounded corners, notches
> (sections of the display that are cut-out from the edge), and cutouts
> (such as circular "hole punch" camera cutouts, or wider pill-shaped
> "islands").
> 
> Some discussion has been underway previously on how best to describe
> these features [1][2], including a reference userspace implementation
> using SVG paths [3]. Using this previous discussion as a jumping off
> point, this RFC allows for describing the following display features:
> 
> * Corner radius (on a per-corner basis)
> * Circular or pill-shaped cutouts
> * Notches with arbitrary shapes
> 
> It's easy to make a case for only using rectangles to describe these
> missing parts of a display, however this severely limits their utility.
> Describing display occlusions as accurately as possible allows for a lot of
> useful UX features. For example, displaying a ring around a hole-punch
> camera when it's in use, or wrapping UI elements around a notch. These
> behaviours are only possible to implement when the dimensions are known
> with near pixel-perfect accuracy.
> 
> Cutouts are described as rectangles with a width, height, and corner
> radius.
> A radius of half the width longest edge will definitionally be an ellipse.
> This simple description is suitable for describing hole-punch cameras,
> as well
> as pill-shaped cutouts. I'm not aware of any devices that can't be
> described like this.
> 
> Notches are a little more complicated, they usually feature convex and
> concave corners as well as straight lines. Here they are described as a
> sequence of optionally disjoint arcs, where the space between one arc ending
> and another starting is inferred to be a straight line.
> 
> Each arc is described with an X and Y pixel coordinate, a radius, and a
> start and end point in degrees. These arcs can precisely describe the
> shape of a notch, and easily allow for a basic bounding box to be
> derived using the min/max coordinates specified in the path.
> 
> Some displays feature partially occluded edges ("waterfall edges") where
> the screen bends, it may be useful for user interfaces to avoid placing
> certain UI elements like buttons too close to these edges. These edges
> are described by a distance from the edge where it begins to be
> occluded, and the number of degrees that the display curves (this
> directly affects how usable this edge of the screen is).
> 
> [1]: https://lore.kernel.org/dri-devel/f8747f99-0695-5be0-841f-4f72ba5d5da3@connolly.tech/
> [2]: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/issues/87
> [3]: https://gitlab.gnome.org/World/Phosh/gmobile
> 
> Signed-off-by: Caleb Connolly <caleb.connolly@linaro.org>
> ---
> Some folks have previously suggested that this information belongs in
> userspace and not in devicetree. I would like to be clear that
> devicetree is for describing hardware, and parts of a display which can
> never actually be seen are very much properties of the underlying
> hardware.

Makes sense to me.

There's similar work happening for touchscreens/pads here[1]. Seems like 
there might be some overlap in terms of what the binding needs.

> ---
> base-commit: 268c4b354d66908697299063c44c0b553b01d935
> 
> // Caleb (they/them)
> ---
>  .../bindings/display/panel/panel-common.yaml       |   7 +
>  .../bindings/display/panel/panel-occlusions.yaml   | 182 +++++++++++++++++++++
>  2 files changed, 189 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/display/panel/panel-common.yaml b/Documentation/devicetree/bindings/display/panel/panel-common.yaml
> index 0a57a31f4f3d..6ea52a031872 100644
> --- a/Documentation/devicetree/bindings/display/panel/panel-common.yaml
> +++ b/Documentation/devicetree/bindings/display/panel/panel-common.yaml
> @@ -150,6 +150,13 @@ properties:
>        controller, this property contains a phandle that references the
>        controller.
>  
> +  occlusions:
> +    $ref: panel-occlusions.yaml#
> +    description:
> +      Some panels have non-standard shapes due to features like rounded corners,
> +      notches, cutouts, and "waterfall" edges. The panel-occlusions bindings
> +      can be used to describe these features.
> +
>  dependencies:
>    width-mm: [ height-mm ]
>    height-mm: [ width-mm ]
> diff --git a/Documentation/devicetree/bindings/display/panel/panel-occlusions.yaml b/Documentation/devicetree/bindings/display/panel/panel-occlusions.yaml
> new file mode 100644
> index 000000000000..0932026bbd8c
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/display/panel/panel-occlusions.yaml
> @@ -0,0 +1,182 @@
> +# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/display/panel/panel-occlusions.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: Common Properties for describing display notches, cutouts, and other occlusions
> +
> +maintainers:
> +  - Caleb Connolly <caleb.connolly@linaro.org>
> +
> +description: |

You don't need '|' unless there is formatting which is significant.

> +  This document defines devicetree nodes that can be used to describe
> +  different kind of display occlusions such as notches, camera cutouts, rounded
> +  corners, and other features that may require special treatment by the display
> +  subsystem. All pixel values should be given in the displays native resolution.
> +
> +properties:
> +  $nodename:
> +    const: occlusions
> +
> +  corners:
> +    type: object
> +    description: |
> +      An area of the display which is fully obscured by a notch.

notch?

> +    properties:
> +      radius-px:
> +        $ref: /schemas/types.yaml#/definitions/uint32
> +        description: Describes the radius when it's identical for all corners
> +
> +      radius-top-px:
> +        $ref: /schemas/types.yaml#/definitions/uint32
> +        description: Describes the radius when it's identical for both top corners
> +
> +      radius-bottom-px:
> +        $ref: /schemas/types.yaml#/definitions/uint32
> +        description: Describes the radius when it's identical for both top corners
> +
> +      radius-top-left-px:
> +        $ref: /schemas/types.yaml#/definitions/uint32
> +        description: The radius of the top left corner in pixels
> +
> +      radius-top-right-px:
> +        $ref: /schemas/types.yaml#/definitions/uint32
> +        description: The radius of the top right corner in pixels
> +
> +      radius-bottom-left-px:
> +        $ref: /schemas/types.yaml#/definitions/uint32
> +        description: The radius of the bottom left corner in pixels
> +
> +      radius-bottom-right-px:
> +        $ref: /schemas/types.yaml#/definitions/uint32
> +        description: The radius of the bottom right corner in pixels

I think I would define the corner positions the same way as cutouts 
rather than encoding the position into property names. 

I'm not sure I'd try to optimize describing all 4 corners other than the 
all 4 the same case.

> +
> +    dependencies:
> +      radius-top-left-px: [ radius-top-right-px ]
> +      radius-top-right-px: [ radius-top-left-px ]
> +      radius-bottom-left-px: [ radius-bottom-right-px ]
> +      radius-bottom-right-px: [ radius-bottom-left-px ]
> +
> +    anyOf:
> +      - required:
> +          - radius-px
> +      - required:
> +          - radius-top-px
> +          - radius-bottom-px
> +      - required:
> +          - radius-top-px
> +          - radius-bottom-left-px
> +          - radius-bottom-right-px
> +      - required:
> +          - radius-bottom-px
> +          - radius-top-left-px
> +          - radius-top-right-px
> +      - required:
> +          - radius-top-left-px
> +          - radius-top-right-px
> +          - radius-bottom-left-px
> +          - radius-bottom-right-px
> +
> +    additionalProperties: false
> +
> +patternProperties:
> +  "^cutout(-[0-9])?$":
> +    type: object
> +    description: |
> +      An area of the display which is not directly adjacent to an
> +      edge and is fully occluded (for example, a camera cutout).
> +    properties:
> +      x-position-px:
> +        $ref: /schemas/types.yaml#/definitions/uint32
> +        description: The horizontal coordinate of the center of the cutout.
> +
> +      y-position-px:
> +        $ref: /schemas/types.yaml#/definitions/uint32
> +        description: The horizontal coordinate of the center of the cutout.
> +
> +      width-px:
> +        $ref: /schemas/types.yaml#/definitions/uint32
> +        description: The width of the cutout in pixels.
> +
> +      height-px:
> +        $ref: /schemas/types.yaml#/definitions/uint32
> +        description: The width of the cutout in pixels.
> +
> +      corner-radius-px:
> +        $ref: /schemas/types.yaml#/definitions/uint32
> +        description: |
> +          The radius of the corners in pixels. For a circle this should be half of
> +          the width/height.
> +
> +    required:
> +      - x-position-px
> +      - y-position-px
> +      - width-px
> +      - height-px
> +      - corner-radius-px
> +
> +    additionalProperties: false
> +
> +  "^notch(-[0-9])?$":
> +    type: object
> +    description: |
> +      An area of the display which is fully occluded by a notch.
> +    properties:
> +      path:

Too vague a name. Ideally, a given property name has only 1 type 
(globally).

> +        $ref: /schemas/types.yaml#/definitions/uint32-matrix
> +        description: |
> +          Sequence of values defining the arcs which describe this path. Lines
> +          are inserted between arcs that don't directly meet.
> +        maxItems: 8 # Ought to cover most scenarios
> +        items:
> +          items:
> +            - description: X coordinate of center point in pixels
> +            - description: Y coordinate of center point in pixels
> +            - description: Circle radius
> +            - description: Starting angle in degrees
> +            - description: Ending angle in degrees
> +
> +    required:
> +      - path

Couldn't you just describe the whole edge of the screen this way? IOW, 
the corners and edges, too.

> +
> +    additionalProperties: false
> +
> +  "^(left|right|top|bottom)-edge$":
> +    type: object
> +    description: |
> +      An edge of the screen with reduced visibility due to a waterfall design
> +      or similar.
> +    properties:
> +      size-px:
> +        $ref: /schemas/types.yaml#/definitions/uint32
> +        description: The distance from the edge where it begins to obscure visbility
> +
> +      curve-degrees:
> +        $ref: /schemas/types.yaml#/definitions/uint32
> +        description: The number of degrees that the display curves.
> +        maximum: 100
> +
> +    required:
> +      - size-px
> +      - curve-degrees
> +
> +additionalProperties: false


[1] https://lore.kernel.org/all/20230510-feature-ts_virtobj_patch-v4-2-5c6c0fc1eed6@wolfvision.net/