[PATCH v4 3/3] Documentation/driver-api/cxl: ACPI PRM Address Translation Support and AMD Zen5 enablement

[Robert Richter <rrichter@amd.com>]

This adds a convention document for the following patch series:

 cxl: ACPI PRM Address Translation Support and AMD Zen5 enablement

Version 7 and later:

 https://lore.kernel.org/linux-cxl/20251114213931.30754-1-rrichter@amd.com/

Link: https://lore.kernel.org/linux-cxl/20251114213931.30754-1-rrichter@amd.com/
Reviewed-by: Gregory Price <gourry@gourry.net>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Alison Schofield <alison.schofield@intel.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Acked-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Robert Richter <rrichter@amd.com>
---
v4:
 * updated sob-chain (Dave, Alison, Jonathan, Dan),
 * added additional spec change for Normalized addressing and _DSM
   definition for address translation (Dan),
 * revised the entire doc and made small format and grammar fixups and
   wording improvements,
v3:
 * updated link in descriptions and add Link: tag (Jonathan Cameron),
 * clarified CFMWS and HPA range description (Jonathan Cameron),
 * use 80 chars line limit (Jonathan Cameron),
 * added block diagram to illustrate the example address mappings (Dave),
v2:
 * updated sob-chain,
 * spell fix in patch description (Randy),
 * made small changes as suggested by Randy,
 * Removed include:: <isonum.txt> line (Jon).
Signed-off-by: Robert Richter <rrichter@amd.com>
---
 Documentation/driver-api/cxl/conventions.rst  |   1 +
 .../driver-api/cxl/conventions/cxl-atl.rst    | 272 ++++++++++++++++++
 2 files changed, 273 insertions(+)
 create mode 100644 Documentation/driver-api/cxl/conventions/cxl-atl.rst

diff --git a/Documentation/driver-api/cxl/conventions.rst b/Documentation/driver-api/cxl/conventions.rst
index 9267a697b2fe..0d2e07279ad9 100644
--- a/Documentation/driver-api/cxl/conventions.rst
+++ b/Documentation/driver-api/cxl/conventions.rst
@@ -14,4 +14,5 @@ same convention.
    :caption: Contents
 
    conventions/cxl-lmh.rst
+   conventions/cxl-atl.rst
    conventions/template.rst
diff --git a/Documentation/driver-api/cxl/conventions/cxl-atl.rst b/Documentation/driver-api/cxl/conventions/cxl-atl.rst
new file mode 100644
index 000000000000..26429fe80ec9
--- /dev/null
+++ b/Documentation/driver-api/cxl/conventions/cxl-atl.rst
@@ -0,0 +1,272 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+ACPI PRM CXL Address Translation
+================================
+
+Document
+--------
+
+CXL Revision 3.2, Version 1.0
+
+License
+-------
+
+SPDX-License Identifier: CC-BY-4.0
+
+Creator/Contributors
+--------------------
+
+- Robert Richter, AMD et al.
+
+Summary of the Change
+---------------------
+
+The CXL Fixed Memory Window Structures (CFMWS) describe zero or more Host
+Physical Address (HPA) windows associated with one or more CXL Host Bridges.
+Each HPA range of a CXL Host Bridge is represented by a CFMWS entry. An HPA
+range may include addresses currently assigned to CXL.mem devices, or an OS may
+assign ranges from an address window to a device.
+
+Host-managed Device Memory is Device-attached memory that is mapped to system
+coherent address space and accessible to the Host using standard write-back
+semantics. The managed address range is configured in the CXL HDM Decoder
+registers of the device. An HDM Decoder in a device is responsible for
+converting HPA into DPA by stripping off specific address bits.
+
+CXL devices and CXL bridges use the same HPA space. It is common across all
+components that belong to the same host domain. The view of the address region
+must be consistent on the CXL.mem path between the Host and the Device.
+
+This is described in the *CXL 3.2 specification* (Table 1-1, 3.3.1,
+8.2.4.20, 9.13.1, 9.18.1.3). [#cxl-spec-3.2]_
+
+Depending on the interconnect architecture of the platform, components attached
+to a host may not share the same host physical address space. Those platforms
+need address translation to convert an HPA between the host and the attached
+component, such as a CXL device. The translation mechanism is host-specific and
+implementation dependent.
+
+For example, x86 AMD platforms use a Data Fabric that manages access to physical
+memory. Devices have their own memory space and can be configured to use
+'Normalized addresses' different from System Physical Addresses (SPA). Address
+translation is then needed. Details are described also under x86 AMD
+Documentation/admin-guide/RAS/address-translation.rst.
+
+Those AMD platforms provide PRM handlers in firmware to perform various types of
+address translation, including for CXL endpoints. AMD Zen5 systems implement the
+ACPI PRM CXL Address Translation firmware call. The ACPI PRM handler has a
+specific GUID to uniquely identify platforms with support for Normalized
+addressing. This is documented in the *ACPI v6.5 Porting Guide* (Address
+Translation - CXL DPA to System Physical Address). [#amd-ppr-58088]_
+
+When in Normalized address mode, HDM decoder address ranges must be configured
+and handled differently. Hardware addresses used in the HDM decoder
+configurations of an endpoint are not SPA and need to be translated from the
+endpoint's to its CXL host bridge's address range. This is especially important
+for finding an endpoint's associated CXL Host Bridge and HPA window described in
+the CFMWS. Additionally, the interleave decoding is done by the Data Fabric and
+the endpoint does not perform decoding when converting HPA to DPA. Instead,
+interleaving is switched off for the endpoint (1-way). Finally, address
+translation might also be needed to inspect the endpoint's hardware addresses,
+such as during profiling, tracing, or error handling.
+
+For example, with Normalized addressing the HDM decoders could look as follows::
+
+                          -------------------------------
+                          | Root Decoder (CFMWS)        |
+                          | SPA Range: 0x850000000      |
+                          | Size: 0x8000000000 (512 GB) |
+                          | Interleave Ways: 1          |
+                          -------------------------------
+                                        |
+                                        v
+                          -------------------------------
+                          | Host Bridge Decoder (HDM)   |
+                          | SPA Range: 0x850000000      |
+                          | Size: 0x8000000000 (512 GB) |
+                          | Interleave Ways: 4          |
+                          | Targets: endpoint5,8,11,13  |
+                          | Granularity: 256            |
+                          -------------------------------
+                                        |
+           -----------------------------+------------------------------
+           |                  |                   |                   |
+           v                  v                   v                   v
+ ------------------- ------------------- ------------------- -------------------
+ | endpoint5       | | endpoint8       | | endpoint11      | | endpoint13      |
+ | decoder5.0      | | decoder8.0      | | decoder11.0     | | decoder13.0     |
+ | PCIe:           | | PCIe:           | | PCIe:           | | PCIe:           |
+ |   0000:e2:00.0  | |   0000:e3:00.0  | |   0000:e4:00.0  | |   0000:e1:00.0  |
+ | DPA:            | | DPA:            | | DPA:            | | DPA:            |
+ |   Start: 0x0    | |   Start: 0x0    | |   Start: 0x0    | |   Start: 0x0    |
+ |   Size:         | |   Size:         | |   Size:         | |   Size:         |
+ |    0x2000000000 | |    0x2000000000 | |    0x2000000000 | |    0x2000000000 |
+ |    (128 GB)     | |    (128 GB)     | |    (128 GB)     | |    (128 GB)     |
+ | Interleaving:   | | Interleaving:   | | Interleaving:   | | Interleaving:   |
+ |   Ways: 1       | |   Ways: 1       | |   Ways: 1       | |   Ways: 1       |
+ |   Gran: 256     | |   Gran: 256     | |   Gran: 256     | |   Gran: 256     |
+ ------------------- ------------------- ------------------- -------------------
+          |                   |                   |                   |
+          v                   v                   v                   v
+         DPA                 DPA                 DPA                 DPA
+
+This shows the representation in sysfs:
+
+.. code-block:: none
+
+ /sys/bus/cxl/devices/endpoint5/decoder5.0/interleave_granularity:256
+ /sys/bus/cxl/devices/endpoint5/decoder5.0/interleave_ways:1
+ /sys/bus/cxl/devices/endpoint5/decoder5.0/size:0x2000000000
+ /sys/bus/cxl/devices/endpoint5/decoder5.0/start:0x0
+ /sys/bus/cxl/devices/endpoint8/decoder8.0/interleave_granularity:256
+ /sys/bus/cxl/devices/endpoint8/decoder8.0/interleave_ways:1
+ /sys/bus/cxl/devices/endpoint8/decoder8.0/size:0x2000000000
+ /sys/bus/cxl/devices/endpoint8/decoder8.0/start:0x0
+ /sys/bus/cxl/devices/endpoint11/decoder11.0/interleave_granularity:256
+ /sys/bus/cxl/devices/endpoint11/decoder11.0/interleave_ways:1
+ /sys/bus/cxl/devices/endpoint11/decoder11.0/size:0x2000000000
+ /sys/bus/cxl/devices/endpoint11/decoder11.0/start:0x0
+ /sys/bus/cxl/devices/endpoint13/decoder13.0/interleave_granularity:256
+ /sys/bus/cxl/devices/endpoint13/decoder13.0/interleave_ways:1
+ /sys/bus/cxl/devices/endpoint13/decoder13.0/size:0x2000000000
+ /sys/bus/cxl/devices/endpoint13/decoder13.0/start:0x0
+
+Note that the endpoint interleaving configurations use direct mapping (1-way).
+
+With PRM calls, the kernel can determine the following mappings:
+
+.. code-block:: none
+
+ cxl decoder5.0: address mapping found for 0000:e2:00.0 (hpa -> spa):
+   0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256
+ cxl decoder8.0: address mapping found for 0000:e3:00.0 (hpa -> spa):
+   0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256
+ cxl decoder11.0: address mapping found for 0000:e4:00.0 (hpa -> spa):
+   0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256
+ cxl decoder13.0: address mapping found for 0000:e1:00.0 (hpa -> spa):
+   0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256
+
+The corresponding CXL host bridge (HDM) decoders and root decoder (CFMWS) match
+the calculated endpoint mappings shown:
+
+.. code-block:: none
+
+ /sys/bus/cxl/devices/port1/decoder1.0/interleave_granularity:256
+ /sys/bus/cxl/devices/port1/decoder1.0/interleave_ways:4
+ /sys/bus/cxl/devices/port1/decoder1.0/size:0x8000000000
+ /sys/bus/cxl/devices/port1/decoder1.0/start:0x850000000
+ /sys/bus/cxl/devices/port1/decoder1.0/target_list:0,1,2,3
+ /sys/bus/cxl/devices/port1/decoder1.0/target_type:expander
+ /sys/bus/cxl/devices/root0/decoder0.0/interleave_granularity:256
+ /sys/bus/cxl/devices/root0/decoder0.0/interleave_ways:1
+ /sys/bus/cxl/devices/root0/decoder0.0/size:0x8000000000
+ /sys/bus/cxl/devices/root0/decoder0.0/start:0x850000000
+ /sys/bus/cxl/devices/root0/decoder0.0/target_list:7
+
+The following changes to the specification are needed:
+
+* Allow a CXL device to be in an HPA space other than the host's address space.
+
+* Allow the platform to use implementation-specific address translation when
+  crossing memory domains on the CXL.mem path between the host and the device.
+
+* Recommend that the platform provide a method for the OS to convert device
+  addresses to SPAs, such as a PRM handler or _DSM. Document _DSM as the
+  preferred method.
+
+* Specify that the kernel (Operating System) determines Endpoint SPA ranges and
+  interleaving configurations using platform-specific address translation
+  methods.
+
+Benefits of the Change
+----------------------
+
+Without the change, the Operating System may be unable to determine the memory
+region and Root Decoder for an Endpoint and its corresponding HDM decoder.
+Region creation would fail. Platforms with a different interconnect architecture
+would fail to set up and use CXL.
+
+References
+----------
+
+.. [#cxl-spec-3.2] Compute Express Link Specification, Revision 3.2, Version 1.0,
+   https://www.computeexpresslink.org/
+
+.. [#amd-ppr-58088] AMD Family 1Ah Models 00h–0Fh and Models 10h–1Fh,
+   ACPI v6.5 Porting Guide, Publication # 58088,
+   https://www.amd.com/en/search/documentation/hub.html
+
+Detailed Description of the Change
+----------------------------------
+
+The following describes the necessary changes to the *CXL 3.2 specification*
+[#cxl-spec-3.2]_:
+
+Add the following paragraph to the end of the section:
+
+**8.2.4.20 CXL HDM Decoder Capability Structure**
+
+"A device may use an HPA space that is not common to other components of the
+host domain. The platform is responsible for address translation when crossing
+HPA spaces. The Operating System must determine the interleaving configuration
+and perform address translation to the HPA ranges of the HDM decoders as
+needed. The translation mechanism is host-specific and implementation dependent.
+
+The platform may provide an interface that can be used by the Operating System
+to translate a DPA and determine its corresponding SPA, such as a Platform
+Runtime Mechanism (PRM) handler or a Device-Specific Method (_DSM). Section
+9.18.3.2 defines the *_DSM function for DPA to System Physical Address
+translation*, which is associated with the CXL Root Device (HID = “ACPI0017”).
+It is the preferred mechanism for the Operating System to translate addresses."
+
+Extend the following table with this _DSM function:
+
+**Table 9-30, _DSM Definitions for CXL Root Device**
+
++--------------------------------------+----------+-----------+---------------------------------------+
+| UUID                                 | Revision | Function  | Description                           |
++======================================+==========+===========+=======================================+
+| E105242A-CD04-F1A9-2D50-B3B870D67396 | 1        | 1         | CXL DPA to SPA (see Section 9.18.3.2) |
+|                                      +----------+-----------+---------------------------------------+
+|                                      | \-       | all other | Reserved                              |
++--------------------------------------+----------+-----------+---------------------------------------+
+
+Add the following section:
+
+**9.18.3.2 _DSM Function for DPA to System Physical Address Translation**
+
+"A platform may be configured to use 'Normalized addresses'. Host physical
+address (HPA) spaces are component-specific and differ from system physical
+addresses (SPAs). The endpoint has its own physical address space. All requests
+presented to the device already use Device Physical Addresses (DPAs). The CXL
+endpoint decoders have interleaving disabled (1-way interleaving) and the device
+does not perform HPA decoding to determine a DPA.
+
+This _DSM function translates a Device Physical Address (DPA) to a System
+Physical Address (SPA) for a specified CXL endpoint. In the host's address
+space, SPA and HPA are equivalent and the Operating System may use this function
+to determine the HPA that corresponds to a device address, for example when
+configuring HDM decoders on platforms with Normalized Addressing."
+
+**Table 9-32, _DSM for DPA to System Physical Address Translation, Inputs, and Outputs**
+
++--------------------------------------+-------+---------------------------------------------------+
+| Field                                | Size  | Description                                       |
++======================================+=======+===================================================+
+| **Input Package:**                                                                               |
++--------------------------------------+-------+---------------------------------------------------+
+| CXL Device Physical Address (DPA)    | QWORD | CXL DPA (e.g., from CXL Component                 |
+|                                      |       | Event Log)                                        |
++--------------------------------------+-------+---------------------------------------------------+
+| CXL Endpoint SBDF                    | DWORD | - Byte 3 - PCIe Segment                           |
+|                                      |       | - Byte 2 - Bus Number                             |
+|                                      |       | - Byte 1 - Device Number Bits[7:3]                |
+|                                      |       |            Function Number Bits[2:0]              |
+|                                      |       | - Byte 0 - RESERVED (MBZ)                         |
++--------------------------------------+-------+---------------------------------------------------+
+| Reserved                             | DWORD | Reserved (MBZ)                                    |
++--------------------------------------+-------+---------------------------------------------------+
+| **Return Package:**                                                                              |
++--------------------------------------+-------+---------------------------------------------------+
+| System Physical Address              | QWORD | System Physical Address                           |
++--------------------------------------+-------+---------------------------------------------------+
-- 
2.47.3