* [PATCH 0/4] cxl: Update CXL documentation for access coordinates calculation
@ 2025-05-14 0:31 Dave Jiang
2025-05-14 0:31 ` [PATCH 1/4] cxl: docs/platform/cdat reference documentation Dave Jiang
` (3 more replies)
0 siblings, 4 replies; 13+ messages in thread
From: Dave Jiang @ 2025-05-14 0:31 UTC (permalink / raw)
To: linux-cxl
Cc: dave, jonathan.cameron, alison.schofield, vishal.l.verma,
ira.weiny, dan.j.williams, gourry
Dave Jiang (4):
cxl: docs/platform/cdat reference documentation
cxl: docs/platform/acpi/srat fix memory table misalignment
cxl: docs/platform/acpi/srat Add generic target documentation
cxl: doc/linux/access-coordinates Update access coordinates
calculation methods
Documentation/driver-api/cxl/index.rst | 1 +
.../cxl/linux/access-coordinates.rst | 87 +++++++++++++++++++
.../driver-api/cxl/platform/acpi/srat.rst | 27 +++++-
.../driver-api/cxl/platform/cdat.rst | 24 +++++
.../driver-api/cxl/platform/cdat/dslbis.rst | 33 +++++++
.../driver-api/cxl/platform/cdat/dsmas.rst | 23 +++++
.../driver-api/cxl/platform/cdat/sslbis.rst | 45 ++++++++++
7 files changed, 237 insertions(+), 3 deletions(-)
create mode 100644 Documentation/driver-api/cxl/platform/cdat.rst
create mode 100644 Documentation/driver-api/cxl/platform/cdat/dslbis.rst
create mode 100644 Documentation/driver-api/cxl/platform/cdat/dsmas.rst
create mode 100644 Documentation/driver-api/cxl/platform/cdat/sslbis.rst
base-commit: 7855bc1362518673103bd9357827572207e6f6d9
--
2.49.0
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH 1/4] cxl: docs/platform/cdat reference documentation 2025-05-14 0:31 [PATCH 0/4] cxl: Update CXL documentation for access coordinates calculation Dave Jiang @ 2025-05-14 0:31 ` Dave Jiang 2025-05-14 16:12 ` Gregory Price 2025-05-14 0:31 ` [PATCH 2/4] cxl: docs/platform/acpi/srat fix memory table misalignment Dave Jiang ` (2 subsequent siblings) 3 siblings, 1 reply; 13+ messages in thread From: Dave Jiang @ 2025-05-14 0:31 UTC (permalink / raw) To: linux-cxl Cc: dave, jonathan.cameron, alison.schofield, vishal.l.verma, ira.weiny, dan.j.williams, gourry Add documentation for CDAT sub-tables for CXL usages. Signed-off-by: Dave Jiang <dave.jiang@intel.com> --- Documentation/driver-api/cxl/index.rst | 1 + .../driver-api/cxl/platform/cdat.rst | 24 ++++++++++ .../driver-api/cxl/platform/cdat/dslbis.rst | 33 ++++++++++++++ .../driver-api/cxl/platform/cdat/dsmas.rst | 23 ++++++++++ .../driver-api/cxl/platform/cdat/sslbis.rst | 45 +++++++++++++++++++ 5 files changed, 126 insertions(+) create mode 100644 Documentation/driver-api/cxl/platform/cdat.rst create mode 100644 Documentation/driver-api/cxl/platform/cdat/dslbis.rst create mode 100644 Documentation/driver-api/cxl/platform/cdat/dsmas.rst create mode 100644 Documentation/driver-api/cxl/platform/cdat/sslbis.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst index 366faf851fc7..9e1414ad3357 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -27,6 +27,7 @@ that have impacts on each other. The docs here break up configurations steps. platform/bios-and-efi platform/acpi + platform/cdat platform/example-configs .. toctree:: diff --git a/Documentation/driver-api/cxl/platform/cdat.rst b/Documentation/driver-api/cxl/platform/cdat.rst new file mode 100644 index 000000000000..3e0ce7099db3 --- /dev/null +++ b/Documentation/driver-api/cxl/platform/cdat.rst @@ -0,0 +1,24 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=========== +CDAT Tables +=========== + +The Coherent Device Attribute Table (CDAT) is created to provide a standard way to +expose the properties of a device such as an CXL accelerator or switch. The table +formatting is similar to ACPI tables. The tables are created to provide performance +calculation of a hot-plugged device where ACPI HMAT+SRAT tables are not able to +enumerate the performance by the BIOS ahead of time. + +The following CDAT tables contain *static* configuration and performance data about CXL devices. + +.. toctree:: + :maxdepth: 1 + + cdat/dsmas.rst + cdat/dslbis.rst + cdat/sslbis.rst + +The Linux CXL driver uses these tables in addition to attributes of the CXL links and +Generic Target performance data provided by HMAT+SRAT to create the whole path performance +for a CXL device. diff --git a/Documentation/driver-api/cxl/platform/cdat/dslbis.rst b/Documentation/driver-api/cxl/platform/cdat/dslbis.rst new file mode 100644 index 000000000000..91ae426b2e8b --- /dev/null +++ b/Documentation/driver-api/cxl/platform/cdat/dslbis.rst @@ -0,0 +1,33 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================================================================== +DSLBIS - Device Scoped Latency and Bandwidth Information Structure +================================================================== + +The Device Scoped Latency and Bandwidth Information Structure contains latency +and bandwidth information based on DSMADHandle matching. + +This table is used by Linux in conjunction with Device scoped Memory Affinity +Structure to determine the performance attributes of a CXL device. + +Example :: + + Structure Type : 01 [DSLBIS] + Reserved : 00 + Length : 18 <- 24d, size of structure + Handle : 0001 <- DSMAS handle + Flags : 00 <- Matches flag field for HMAT SLLBIS + Data Type : 00 <- Latency + Entry Basee Unit : 0000000000001000 <- Entry Base Unit field in HMAT SSLBIS + Entry : 010000000000 <- First byte used here, CXL LTC + Reserved : 0000 + + Structure Type : 01 [DSLBIS] + Reserved : 00 + Length : 18 <- 24d, size of structure + Handle : 0001 <- DSMAS handle + Flags : 00 <- Matches flag field for HMAT SLLBIS + Data Type : 03 <- Bandwidth + Entry Basee Unit : 0000000000001000 <- Entry Base Unit field in HMAT SSLBIS + Entry : 020000000000 <- First byte used here, CXL BW + Reserved : 0000 diff --git a/Documentation/driver-api/cxl/platform/cdat/dsmas.rst b/Documentation/driver-api/cxl/platform/cdat/dsmas.rst new file mode 100644 index 000000000000..8c32ddb3381c --- /dev/null +++ b/Documentation/driver-api/cxl/platform/cdat/dsmas.rst @@ -0,0 +1,23 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============================================== +DSMAS - Device Scoped Memory Affinity Structure +=============================================== + +The Device Scoped Memory Affinity Structure contains information such as +DSMADHandle, the DPA Base, and DPA Length. + +This table is used by Linux in conjunction with the Device Scoped Latency and +Bandwidth Information Structure (DSLBIS) to determine the performance +attributes of the CXL device itself. + +Example :: + + Structure Type : 00 [DSMAS] + Reserved : 00 + Length : 0018 <- 24d, size of structure + DSMADHandle : 01 + Flags : 00 + Reserved : 0000 + DPA Base : 0000000040000000 <- 1GiB base + DPA Length : 0000000080000000 <- 2GiB size diff --git a/Documentation/driver-api/cxl/platform/cdat/sslbis.rst b/Documentation/driver-api/cxl/platform/cdat/sslbis.rst new file mode 100644 index 000000000000..e299575493fa --- /dev/null +++ b/Documentation/driver-api/cxl/platform/cdat/sslbis.rst @@ -0,0 +1,45 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================================================================== +SSLBIS - Switch Scoped Latency and Bandwidth Information Structure +================================================================== + +The Switch Scoped Latency Bandwidth Information Structure contains information +about the latency and bandwidth of a switch. + +The table is used by Linux to compute the performance coordinates of a CXL path +from the device to the root port where a switch is part of the path. + +Example :: + + Structure Type : 05 [SSLBIS] + Reserved : 00 + Length : 20 <- 32d, length of record, including SSLB entries + Data Type : 00 <- Latency + Reserved : 000000 + Entry Base Unit : 00000000000000001000 <- Matches Entry Base Unit in HMAT SSLBIS + + <- SSLB Entry 0 + Port X ID : 0100 <- First port, 0100h represents an upstream port + Port Y ID : 0000 <- Second port, downstream port 0 + Latency : 0100 <- Port latency + Reserved : 0000 + <- SSLB Entry 1 + Port X ID : 0100 + Port Y ID : 0001 + Latency : 0100 + Reserved : 0000 + + + Structure Type : 05 [SSLBIS] + Reserved : 00 + Length : 18 <- 24d, length of record, including SSLB entry + Data Type : 03 <- Bandwidth + Reserved : 000000 + Entry Base Unit : 00000000000000001000 <- Matches Entry Base Unit in HMAT SSLBIS + + <- SSLB Entry 0 + Port X ID : 0100 <- First port, 0100h represents an upstream port + Port Y ID : FFFF <- Second port, FFFFh indicates any port + Bandwidth : 1200 <- Port bandwidth + Reserved : 0000 -- 2.49.0 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH 1/4] cxl: docs/platform/cdat reference documentation 2025-05-14 0:31 ` [PATCH 1/4] cxl: docs/platform/cdat reference documentation Dave Jiang @ 2025-05-14 16:12 ` Gregory Price 2025-05-14 19:02 ` Dave Jiang 0 siblings, 1 reply; 13+ messages in thread From: Gregory Price @ 2025-05-14 16:12 UTC (permalink / raw) To: Dave Jiang Cc: linux-cxl, dave, jonathan.cameron, alison.schofield, vishal.l.verma, ira.weiny, dan.j.williams On Tue, May 13, 2025 at 05:31:30PM -0700, Dave Jiang wrote: > Add documentation for CDAT sub-tables for CXL usages. > Mostly wording nits, but i recommend putting allt he subtables in cdat.rst directly and adding a terms list at the top so you can reference them cleanly on the same page without redefining them everywhere. See below. > Signed-off-by: Dave Jiang <dave.jiang@intel.com> > --- > Documentation/driver-api/cxl/index.rst | 1 + > .../driver-api/cxl/platform/cdat.rst | 24 ++++++++++ > .../driver-api/cxl/platform/cdat/dslbis.rst | 33 ++++++++++++++ > .../driver-api/cxl/platform/cdat/dsmas.rst | 23 ++++++++++ > .../driver-api/cxl/platform/cdat/sslbis.rst | 45 +++++++++++++++++++ > 5 files changed, 126 insertions(+) > create mode 100644 Documentation/driver-api/cxl/platform/cdat.rst > create mode 100644 Documentation/driver-api/cxl/platform/cdat/dslbis.rst > create mode 100644 Documentation/driver-api/cxl/platform/cdat/dsmas.rst > create mode 100644 Documentation/driver-api/cxl/platform/cdat/sslbis.rst > > diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst > index 366faf851fc7..9e1414ad3357 100644 > --- a/Documentation/driver-api/cxl/index.rst > +++ b/Documentation/driver-api/cxl/index.rst > @@ -27,6 +27,7 @@ that have impacts on each other. The docs here break up configurations steps. > > platform/bios-and-efi > platform/acpi > + platform/cdat > platform/example-configs > > .. toctree:: > diff --git a/Documentation/driver-api/cxl/platform/cdat.rst b/Documentation/driver-api/cxl/platform/cdat.rst > new file mode 100644 > index 000000000000..3e0ce7099db3 > --- /dev/null > +++ b/Documentation/driver-api/cxl/platform/cdat.rst > @@ -0,0 +1,24 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +=========== > +CDAT Tables > +=========== "Coherent Device Attribute Table Tables" :] I would name this ====================================== Coherent Device Attribute Table (CDAT) ====================================== > + > +The Coherent Device Attribute Table (CDAT) is created to provide a standard way to > +expose the properties of a device such as an CXL accelerator or switch. The table > +formatting is similar to ACPI tables. The tables are created to provide performance > +calculation of a hot-plugged device where ACPI HMAT+SRAT tables are not able to > +enumerate the performance by the BIOS ahead of time. I may suggest something like... The CDAT provides functional and performance attributes of devices such as CXL accelerators, switches, or endpoints. The table formatting is similar to ACPI tables. CDAT data may be parsed by BIOS at boot or may be enumerated at runtime (after device hotplug, for example). > + > +The following CDAT tables contain *static* configuration and performance data about CXL devices. > + > +.. toctree:: > + :maxdepth: 1 > + > + cdat/dsmas.rst > + cdat/dslbis.rst > + cdat/sslbis.rst > + > +The Linux CXL driver uses these tables in addition to attributes of the CXL links and ^^^^^^^^^^^^^ provided by what? > +Generic Target performance data provided by HMAT+SRAT to create the whole path performance ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The is provided by SRAT right? > +for a CXL device. Maybe simplified to: The CXL driver uses a combination of CDAT, HMAT, SRAT and other data to generate "whole path performance" data for a CXL device. > diff --git a/Documentation/driver-api/cxl/platform/cdat/dslbis.rst b/Documentation/driver-api/cxl/platform/cdat/dslbis.rst > new file mode 100644 > index 000000000000..91ae426b2e8b > --- /dev/null > +++ b/Documentation/driver-api/cxl/platform/cdat/dslbis.rst > @@ -0,0 +1,33 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +================================================================== > +DSLBIS - Device Scoped Latency and Bandwidth Information Structure > +================================================================== > + > +The Device Scoped Latency and Bandwidth Information Structure contains latency > +and bandwidth information based on DSMADHandle matching. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ What does a DSMADHandle represent? > + > +This table is used by Linux in conjunction with Device scoped Memory Affinity > +Structure to determine the performance attributes of a CXL device. swap the order of the last two paragraphs. Add (DSMAS) after "Device Scoped Memory Affinity Structure", introduce what a DSMADHandle is and what it represents before you reference it. > + > +Example :: > + > + Structure Type : 01 [DSLBIS] > + Reserved : 00 > + Length : 18 <- 24d, size of structure > + Handle : 0001 <- DSMAS handle > + Flags : 00 <- Matches flag field for HMAT SLLBIS > + Data Type : 00 <- Latency > + Entry Basee Unit : 0000000000001000 <- Entry Base Unit field in HMAT SSLBIS > + Entry : 010000000000 <- First byte used here, CXL LTC > + Reserved : 0000 > + > + Structure Type : 01 [DSLBIS] > + Reserved : 00 > + Length : 18 <- 24d, size of structure > + Handle : 0001 <- DSMAS handle > + Flags : 00 <- Matches flag field for HMAT SLLBIS > + Data Type : 03 <- Bandwidth > + Entry Basee Unit : 0000000000001000 <- Entry Base Unit field in HMAT SSLBIS > + Entry : 020000000000 <- First byte used here, CXL BW > + Reserved : 0000 > diff --git a/Documentation/driver-api/cxl/platform/cdat/dsmas.rst b/Documentation/driver-api/cxl/platform/cdat/dsmas.rst > new file mode 100644 > index 000000000000..8c32ddb3381c > --- /dev/null > +++ b/Documentation/driver-api/cxl/platform/cdat/dsmas.rst > @@ -0,0 +1,23 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +=============================================== > +DSMAS - Device Scoped Memory Affinity Structure > +=============================================== Device Scoped Memory Affinity Structure (DSMAS) > + > +The Device Scoped Memory Affinity Structure contains information such as ^^^^^^ DSMAS ^^^^^ Since you define it in the header, you can use the abbreviation > +DSMADHandle, the DPA Base, and DPA Length. > + It's probably better to put all these subtables in a single document (cdat.rst) and add a terms list at the top (such as DSMADHandle) > +This table is used by Linux in conjunction with the Device Scoped Latency and > +Bandwidth Information Structure (DSLBIS) to determine the performance > +attributes of the CXL device itself. > + > +Example :: > + > + Structure Type : 00 [DSMAS] > + Reserved : 00 > + Length : 0018 <- 24d, size of structure > + DSMADHandle : 01 > + Flags : 00 > + Reserved : 0000 > + DPA Base : 0000000040000000 <- 1GiB base > + DPA Length : 0000000080000000 <- 2GiB size > diff --git a/Documentation/driver-api/cxl/platform/cdat/sslbis.rst b/Documentation/driver-api/cxl/platform/cdat/sslbis.rst > new file mode 100644 > index 000000000000..e299575493fa > --- /dev/null > +++ b/Documentation/driver-api/cxl/platform/cdat/sslbis.rst > @@ -0,0 +1,45 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +================================================================== > +SSLBIS - Switch Scoped Latency and Bandwidth Information Structure > +================================================================== Switch Scoped Latency and Bandwidth Information Structure (SSLBIS) > + > +The Switch Scoped Latency Bandwidth Information Structure contains information ^^^^^^^^^^^^^^^^ SSLBIS ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > +about the latency and bandwidth of a switch. > + > +The table is used by Linux to compute the performance coordinates of a CXL path > +from the device to the root port where a switch is part of the path. > + > +Example :: > + > + Structure Type : 05 [SSLBIS] > + Reserved : 00 > + Length : 20 <- 32d, length of record, including SSLB entries > + Data Type : 00 <- Latency > + Reserved : 000000 > + Entry Base Unit : 00000000000000001000 <- Matches Entry Base Unit in HMAT SSLBIS > + > + <- SSLB Entry 0 > + Port X ID : 0100 <- First port, 0100h represents an upstream port > + Port Y ID : 0000 <- Second port, downstream port 0 > + Latency : 0100 <- Port latency > + Reserved : 0000 > + <- SSLB Entry 1 > + Port X ID : 0100 > + Port Y ID : 0001 > + Latency : 0100 > + Reserved : 0000 > + > + > + Structure Type : 05 [SSLBIS] > + Reserved : 00 > + Length : 18 <- 24d, length of record, including SSLB entry > + Data Type : 03 <- Bandwidth > + Reserved : 000000 > + Entry Base Unit : 00000000000000001000 <- Matches Entry Base Unit in HMAT SSLBIS > + > + <- SSLB Entry 0 > + Port X ID : 0100 <- First port, 0100h represents an upstream port > + Port Y ID : FFFF <- Second port, FFFFh indicates any port > + Bandwidth : 1200 <- Port bandwidth > + Reserved : 0000 > -- > 2.49.0 > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 1/4] cxl: docs/platform/cdat reference documentation 2025-05-14 16:12 ` Gregory Price @ 2025-05-14 19:02 ` Dave Jiang 2025-05-14 21:59 ` Gregory Price 0 siblings, 1 reply; 13+ messages in thread From: Dave Jiang @ 2025-05-14 19:02 UTC (permalink / raw) To: Gregory Price Cc: linux-cxl, dave, jonathan.cameron, alison.schofield, vishal.l.verma, ira.weiny, dan.j.williams On 5/14/25 9:12 AM, Gregory Price wrote: > On Tue, May 13, 2025 at 05:31:30PM -0700, Dave Jiang wrote: >> Add documentation for CDAT sub-tables for CXL usages. >> > > Mostly wording nits, but i recommend putting allt he subtables in > cdat.rst directly and adding a terms list at the top so you can > reference them cleanly on the same page without redefining them > everywhere. See below. Thanks for reviewing. I'll do that. > >> Signed-off-by: Dave Jiang <dave.jiang@intel.com> >> --- >> Documentation/driver-api/cxl/index.rst | 1 + >> .../driver-api/cxl/platform/cdat.rst | 24 ++++++++++ >> .../driver-api/cxl/platform/cdat/dslbis.rst | 33 ++++++++++++++ >> .../driver-api/cxl/platform/cdat/dsmas.rst | 23 ++++++++++ >> .../driver-api/cxl/platform/cdat/sslbis.rst | 45 +++++++++++++++++++ >> 5 files changed, 126 insertions(+) >> create mode 100644 Documentation/driver-api/cxl/platform/cdat.rst >> create mode 100644 Documentation/driver-api/cxl/platform/cdat/dslbis.rst >> create mode 100644 Documentation/driver-api/cxl/platform/cdat/dsmas.rst >> create mode 100644 Documentation/driver-api/cxl/platform/cdat/sslbis.rst >> >> diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst >> index 366faf851fc7..9e1414ad3357 100644 >> --- a/Documentation/driver-api/cxl/index.rst >> +++ b/Documentation/driver-api/cxl/index.rst >> @@ -27,6 +27,7 @@ that have impacts on each other. The docs here break up configurations steps. >> >> platform/bios-and-efi >> platform/acpi >> + platform/cdat >> platform/example-configs >> >> .. toctree:: >> diff --git a/Documentation/driver-api/cxl/platform/cdat.rst b/Documentation/driver-api/cxl/platform/cdat.rst >> new file mode 100644 >> index 000000000000..3e0ce7099db3 >> --- /dev/null >> +++ b/Documentation/driver-api/cxl/platform/cdat.rst >> @@ -0,0 +1,24 @@ >> +.. SPDX-License-Identifier: GPL-2.0 >> + >> +=========== >> +CDAT Tables >> +=========== > > "Coherent Device Attribute Table Tables" :] > > I would name this > > ====================================== > Coherent Device Attribute Table (CDAT) > ====================================== > >> + >> +The Coherent Device Attribute Table (CDAT) is created to provide a standard way to >> +expose the properties of a device such as an CXL accelerator or switch. The table >> +formatting is similar to ACPI tables. The tables are created to provide performance >> +calculation of a hot-plugged device where ACPI HMAT+SRAT tables are not able to >> +enumerate the performance by the BIOS ahead of time. > > I may suggest something like... > > The CDAT provides functional and performance attributes of devices such > as CXL accelerators, switches, or endpoints. The table formatting is > similar to ACPI tables. CDAT data may be parsed by BIOS at boot or may > be enumerated at runtime (after device hotplug, for example). > >> + >> +The following CDAT tables contain *static* configuration and performance data about CXL devices. >> + >> +.. toctree:: >> + :maxdepth: 1 >> + >> + cdat/dsmas.rst >> + cdat/dslbis.rst >> + cdat/sslbis.rst >> + >> +The Linux CXL driver uses these tables in addition to attributes of the CXL links and > ^^^^^^^^^^^^^ > provided by what? >> +Generic Target performance data provided by HMAT+SRAT to create the whole path performance > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > The is provided by SRAT right? SRAT only provide the association of a device handle and proximity domain for a generic target. The HMAT provides the performance numbers based on the PXM. DJ >> +for a CXL device. > > Maybe simplified to: > > The CXL driver uses a combination of CDAT, HMAT, SRAT and other data to > generate "whole path performance" data for a CXL device. > >> diff --git a/Documentation/driver-api/cxl/platform/cdat/dslbis.rst b/Documentation/driver-api/cxl/platform/cdat/dslbis.rst >> new file mode 100644 >> index 000000000000..91ae426b2e8b >> --- /dev/null >> +++ b/Documentation/driver-api/cxl/platform/cdat/dslbis.rst >> @@ -0,0 +1,33 @@ >> +.. SPDX-License-Identifier: GPL-2.0 >> + >> +================================================================== >> +DSLBIS - Device Scoped Latency and Bandwidth Information Structure >> +================================================================== >> + >> +The Device Scoped Latency and Bandwidth Information Structure contains latency >> +and bandwidth information based on DSMADHandle matching. > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > What does a DSMADHandle represent? >> + >> +This table is used by Linux in conjunction with Device scoped Memory Affinity >> +Structure to determine the performance attributes of a CXL device. > > swap the order of the last two paragraphs. Add (DSMAS) after "Device > Scoped Memory Affinity Structure", introduce what a DSMADHandle is and > what it represents before you reference it. > >> + >> +Example :: >> + >> + Structure Type : 01 [DSLBIS] >> + Reserved : 00 >> + Length : 18 <- 24d, size of structure >> + Handle : 0001 <- DSMAS handle >> + Flags : 00 <- Matches flag field for HMAT SLLBIS >> + Data Type : 00 <- Latency >> + Entry Basee Unit : 0000000000001000 <- Entry Base Unit field in HMAT SSLBIS >> + Entry : 010000000000 <- First byte used here, CXL LTC >> + Reserved : 0000 >> + >> + Structure Type : 01 [DSLBIS] >> + Reserved : 00 >> + Length : 18 <- 24d, size of structure >> + Handle : 0001 <- DSMAS handle >> + Flags : 00 <- Matches flag field for HMAT SLLBIS >> + Data Type : 03 <- Bandwidth >> + Entry Basee Unit : 0000000000001000 <- Entry Base Unit field in HMAT SSLBIS >> + Entry : 020000000000 <- First byte used here, CXL BW >> + Reserved : 0000 >> diff --git a/Documentation/driver-api/cxl/platform/cdat/dsmas.rst b/Documentation/driver-api/cxl/platform/cdat/dsmas.rst >> new file mode 100644 >> index 000000000000..8c32ddb3381c >> --- /dev/null >> +++ b/Documentation/driver-api/cxl/platform/cdat/dsmas.rst >> @@ -0,0 +1,23 @@ >> +.. SPDX-License-Identifier: GPL-2.0 >> + >> +=============================================== >> +DSMAS - Device Scoped Memory Affinity Structure >> +=============================================== > > Device Scoped Memory Affinity Structure (DSMAS) > >> + >> +The Device Scoped Memory Affinity Structure contains information such as > ^^^^^^ DSMAS ^^^^^ > Since you define it in the header, you can use the abbreviation > >> +DSMADHandle, the DPA Base, and DPA Length. >> + > > It's probably better to put all these subtables in a single document > (cdat.rst) and add a terms list at the top (such as DSMADHandle) > >> +This table is used by Linux in conjunction with the Device Scoped Latency and >> +Bandwidth Information Structure (DSLBIS) to determine the performance >> +attributes of the CXL device itself. >> + >> +Example :: >> + >> + Structure Type : 00 [DSMAS] >> + Reserved : 00 >> + Length : 0018 <- 24d, size of structure >> + DSMADHandle : 01 >> + Flags : 00 >> + Reserved : 0000 >> + DPA Base : 0000000040000000 <- 1GiB base >> + DPA Length : 0000000080000000 <- 2GiB size >> diff --git a/Documentation/driver-api/cxl/platform/cdat/sslbis.rst b/Documentation/driver-api/cxl/platform/cdat/sslbis.rst >> new file mode 100644 >> index 000000000000..e299575493fa >> --- /dev/null >> +++ b/Documentation/driver-api/cxl/platform/cdat/sslbis.rst >> @@ -0,0 +1,45 @@ >> +.. SPDX-License-Identifier: GPL-2.0 >> + >> +================================================================== >> +SSLBIS - Switch Scoped Latency and Bandwidth Information Structure >> +================================================================== > > Switch Scoped Latency and Bandwidth Information Structure (SSLBIS) > >> + >> +The Switch Scoped Latency Bandwidth Information Structure contains information > ^^^^^^^^^^^^^^^^ SSLBIS ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> +about the latency and bandwidth of a switch. >> + >> +The table is used by Linux to compute the performance coordinates of a CXL path >> +from the device to the root port where a switch is part of the path. >> + >> +Example :: >> + >> + Structure Type : 05 [SSLBIS] >> + Reserved : 00 >> + Length : 20 <- 32d, length of record, including SSLB entries >> + Data Type : 00 <- Latency >> + Reserved : 000000 >> + Entry Base Unit : 00000000000000001000 <- Matches Entry Base Unit in HMAT SSLBIS >> + >> + <- SSLB Entry 0 >> + Port X ID : 0100 <- First port, 0100h represents an upstream port >> + Port Y ID : 0000 <- Second port, downstream port 0 >> + Latency : 0100 <- Port latency >> + Reserved : 0000 >> + <- SSLB Entry 1 >> + Port X ID : 0100 >> + Port Y ID : 0001 >> + Latency : 0100 >> + Reserved : 0000 >> + >> + >> + Structure Type : 05 [SSLBIS] >> + Reserved : 00 >> + Length : 18 <- 24d, length of record, including SSLB entry >> + Data Type : 03 <- Bandwidth >> + Reserved : 000000 >> + Entry Base Unit : 00000000000000001000 <- Matches Entry Base Unit in HMAT SSLBIS >> + >> + <- SSLB Entry 0 >> + Port X ID : 0100 <- First port, 0100h represents an upstream port >> + Port Y ID : FFFF <- Second port, FFFFh indicates any port >> + Bandwidth : 1200 <- Port bandwidth >> + Reserved : 0000 >> -- >> 2.49.0 >> > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 1/4] cxl: docs/platform/cdat reference documentation 2025-05-14 19:02 ` Dave Jiang @ 2025-05-14 21:59 ` Gregory Price 2025-05-14 22:15 ` Dave Jiang 0 siblings, 1 reply; 13+ messages in thread From: Gregory Price @ 2025-05-14 21:59 UTC (permalink / raw) To: Dave Jiang Cc: linux-cxl, dave, jonathan.cameron, alison.schofield, vishal.l.verma, ira.weiny, dan.j.williams On Wed, May 14, 2025 at 12:02:01PM -0700, Dave Jiang wrote: > > > ^^^^^^^^^^^^^ > > provided by what? > >> +Generic Target performance data provided by HMAT+SRAT to create the whole path performance > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > The is provided by SRAT right? > > SRAT only provide the association of a device handle and proximity domain for a generic target. The HMAT provides the performance numbers based on the PXM. > > DJ > Hm, maybe this is best expressed in an example configuration? I find it hard to keep the various table contents in my head while reading document pieces like this, so having examples all in one doc that show the relationships would be nice. ~Gregory ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 1/4] cxl: docs/platform/cdat reference documentation 2025-05-14 21:59 ` Gregory Price @ 2025-05-14 22:15 ` Dave Jiang 0 siblings, 0 replies; 13+ messages in thread From: Dave Jiang @ 2025-05-14 22:15 UTC (permalink / raw) To: Gregory Price Cc: linux-cxl, dave, jonathan.cameron, alison.schofield, vishal.l.verma, ira.weiny, dan.j.williams On 5/14/25 2:59 PM, Gregory Price wrote: > On Wed, May 14, 2025 at 12:02:01PM -0700, Dave Jiang wrote: >> >>> ^^^^^^^^^^^^^ >>> provided by what? >>>> +Generic Target performance data provided by HMAT+SRAT to create the whole path performance >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>> The is provided by SRAT right? >> >> SRAT only provide the association of a device handle and proximity domain for a generic target. The HMAT provides the performance numbers based on the PXM. >> >> DJ >> > > Hm, maybe this is best expressed in an example configuration? > > I find it hard to keep the various table contents in my head while > reading document pieces like this, so having examples all in one doc > that show the relationships would be nice. I wonder if I should add the explanation in the SRAT document when talking about Generic Port as that's probably the relevant location. > > ~Gregory ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH 2/4] cxl: docs/platform/acpi/srat fix memory table misalignment 2025-05-14 0:31 [PATCH 0/4] cxl: Update CXL documentation for access coordinates calculation Dave Jiang 2025-05-14 0:31 ` [PATCH 1/4] cxl: docs/platform/cdat reference documentation Dave Jiang @ 2025-05-14 0:31 ` Dave Jiang 2025-05-14 1:40 ` Gregory Price 2025-05-14 0:31 ` [PATCH 3/4] cxl: docs/platform/acpi/srat Add generic target documentation Dave Jiang 2025-05-14 0:31 ` [PATCH 4/4] cxl: doc/linux/access-coordinates Update access coordinates calculation methods Dave Jiang 3 siblings, 1 reply; 13+ messages in thread From: Dave Jiang @ 2025-05-14 0:31 UTC (permalink / raw) To: linux-cxl Cc: dave, jonathan.cameron, alison.schofield, vishal.l.verma, ira.weiny, dan.j.williams, gourry Fix the misalignment in the SRAT table example. Signed-off-by: Dave Jiang <dave.jiang@intel.com> --- Documentation/driver-api/cxl/platform/acpi/srat.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Documentation/driver-api/cxl/platform/acpi/srat.rst b/Documentation/driver-api/cxl/platform/acpi/srat.rst index 56d7bbb18c3b..cc7687645de7 100644 --- a/Documentation/driver-api/cxl/platform/acpi/srat.rst +++ b/Documentation/driver-api/cxl/platform/acpi/srat.rst @@ -39,6 +39,6 @@ Example :: Address Length : 0000003CA0000000 Reserved2 : 00000000 Flags (decoded below) : 0000000B - Enabled : 1 - Hot Pluggable : 1 - Non-Volatile : 0 + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 -- 2.49.0 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH 2/4] cxl: docs/platform/acpi/srat fix memory table misalignment 2025-05-14 0:31 ` [PATCH 2/4] cxl: docs/platform/acpi/srat fix memory table misalignment Dave Jiang @ 2025-05-14 1:40 ` Gregory Price 2025-05-14 15:22 ` Dave Jiang 0 siblings, 1 reply; 13+ messages in thread From: Gregory Price @ 2025-05-14 1:40 UTC (permalink / raw) To: Dave Jiang Cc: linux-cxl, dave, jonathan.cameron, alison.schofield, vishal.l.verma, ira.weiny, dan.j.williams On Tue, May 13, 2025 at 05:31:31PM -0700, Dave Jiang wrote: > Fix the misalignment in the SRAT table example. > This misalignment was intentional, but i don't have a strong feeling either way. (This is how it comes out of iasl -d) > Signed-off-by: Dave Jiang <dave.jiang@intel.com> > --- > Documentation/driver-api/cxl/platform/acpi/srat.rst | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/Documentation/driver-api/cxl/platform/acpi/srat.rst b/Documentation/driver-api/cxl/platform/acpi/srat.rst > index 56d7bbb18c3b..cc7687645de7 100644 > --- a/Documentation/driver-api/cxl/platform/acpi/srat.rst > +++ b/Documentation/driver-api/cxl/platform/acpi/srat.rst > @@ -39,6 +39,6 @@ Example :: > Address Length : 0000003CA0000000 > Reserved2 : 00000000 > Flags (decoded below) : 0000000B > - Enabled : 1 > - Hot Pluggable : 1 > - Non-Volatile : 0 > + Enabled : 1 > + Hot Pluggable : 1 > + Non-Volatile : 0 > -- > 2.49.0 > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 2/4] cxl: docs/platform/acpi/srat fix memory table misalignment 2025-05-14 1:40 ` Gregory Price @ 2025-05-14 15:22 ` Dave Jiang 0 siblings, 0 replies; 13+ messages in thread From: Dave Jiang @ 2025-05-14 15:22 UTC (permalink / raw) To: Gregory Price Cc: linux-cxl, dave, jonathan.cameron, alison.schofield, vishal.l.verma, ira.weiny, dan.j.williams On 5/13/25 6:40 PM, Gregory Price wrote: > On Tue, May 13, 2025 at 05:31:31PM -0700, Dave Jiang wrote: >> Fix the misalignment in the SRAT table example. >> > > This misalignment was intentional, but i don't have a strong feeling > either way. > > (This is how it comes out of iasl -d) I can drop it. It just looks weird in documentation. :) > >> Signed-off-by: Dave Jiang <dave.jiang@intel.com> >> --- >> Documentation/driver-api/cxl/platform/acpi/srat.rst | 6 +++--- >> 1 file changed, 3 insertions(+), 3 deletions(-) >> >> diff --git a/Documentation/driver-api/cxl/platform/acpi/srat.rst b/Documentation/driver-api/cxl/platform/acpi/srat.rst >> index 56d7bbb18c3b..cc7687645de7 100644 >> --- a/Documentation/driver-api/cxl/platform/acpi/srat.rst >> +++ b/Documentation/driver-api/cxl/platform/acpi/srat.rst >> @@ -39,6 +39,6 @@ Example :: >> Address Length : 0000003CA0000000 >> Reserved2 : 00000000 >> Flags (decoded below) : 0000000B >> - Enabled : 1 >> - Hot Pluggable : 1 >> - Non-Volatile : 0 >> + Enabled : 1 >> + Hot Pluggable : 1 >> + Non-Volatile : 0 >> -- >> 2.49.0 >> ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH 3/4] cxl: docs/platform/acpi/srat Add generic target documentation 2025-05-14 0:31 [PATCH 0/4] cxl: Update CXL documentation for access coordinates calculation Dave Jiang 2025-05-14 0:31 ` [PATCH 1/4] cxl: docs/platform/cdat reference documentation Dave Jiang 2025-05-14 0:31 ` [PATCH 2/4] cxl: docs/platform/acpi/srat fix memory table misalignment Dave Jiang @ 2025-05-14 0:31 ` Dave Jiang 2025-05-14 16:15 ` Gregory Price 2025-05-14 0:31 ` [PATCH 4/4] cxl: doc/linux/access-coordinates Update access coordinates calculation methods Dave Jiang 3 siblings, 1 reply; 13+ messages in thread From: Dave Jiang @ 2025-05-14 0:31 UTC (permalink / raw) To: linux-cxl Cc: dave, jonathan.cameron, alison.schofield, vishal.l.verma, ira.weiny, dan.j.williams, gourry Add description in the SRAT document to descript the Generic Port Affinity sub-table. Signed-off-by: Dave Jiang <dave.jiang@intel.com> --- .../driver-api/cxl/platform/acpi/srat.rst | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/Documentation/driver-api/cxl/platform/acpi/srat.rst b/Documentation/driver-api/cxl/platform/acpi/srat.rst index cc7687645de7..500413b96e70 100644 --- a/Documentation/driver-api/cxl/platform/acpi/srat.rst +++ b/Documentation/driver-api/cxl/platform/acpi/srat.rst @@ -42,3 +42,24 @@ Example :: Enabled : 1 Hot Pluggable : 1 Non-Volatile : 0 + + +Generic Port Affinity +===================== +The Generic Port Affinity subtable provides an association between a proximity +domain and a device handle representing a Generic Port such as a CXL host +bridge. With the association, latency and bandwidth numbers can be retrieved +from the SRAT for the path between CPU(s) (initiator) and the Generic Port. +This helps with constructing the performance coordinates for CXL devices that +are hot-plugged and not able to be enumerated by the platform firmware. + +Example :: + + Subtable Type : 06 [Generic Port Affinity] + Length : 20 <- 32d, length of table + Reserved : 00 + Device Handle Type : 00 <- 0 - ACPI, 1 - PCI + Proximity Domain : 00000001 + Device Handle : ACPI0016:01 + Flags : 00000001 <- Bit 0 (Enabled) + Reserved : 00000000 -- 2.49.0 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH 3/4] cxl: docs/platform/acpi/srat Add generic target documentation 2025-05-14 0:31 ` [PATCH 3/4] cxl: docs/platform/acpi/srat Add generic target documentation Dave Jiang @ 2025-05-14 16:15 ` Gregory Price 0 siblings, 0 replies; 13+ messages in thread From: Gregory Price @ 2025-05-14 16:15 UTC (permalink / raw) To: Dave Jiang Cc: linux-cxl, dave, jonathan.cameron, alison.schofield, vishal.l.verma, ira.weiny, dan.j.williams On Tue, May 13, 2025 at 05:31:32PM -0700, Dave Jiang wrote: > Add description in the SRAT document to descript the Generic Port > Affinity sub-table. > > Signed-off-by: Dave Jiang <dave.jiang@intel.com> one wording nit Reviewed-by: Gregory Price <gourry@gourry.net> > --- > .../driver-api/cxl/platform/acpi/srat.rst | 21 +++++++++++++++++++ > 1 file changed, 21 insertions(+) > > diff --git a/Documentation/driver-api/cxl/platform/acpi/srat.rst b/Documentation/driver-api/cxl/platform/acpi/srat.rst > index cc7687645de7..500413b96e70 100644 > --- a/Documentation/driver-api/cxl/platform/acpi/srat.rst > +++ b/Documentation/driver-api/cxl/platform/acpi/srat.rst > @@ -42,3 +42,24 @@ Example :: > Enabled : 1 > Hot Pluggable : 1 > Non-Volatile : 0 > + > + > +Generic Port Affinity > +===================== > +The Generic Port Affinity subtable provides an association between a proximity > +domain and a device handle representing a Generic Port such as a CXL host > +bridge. With the association, latency and bandwidth numbers can be retrieved > +from the SRAT for the path between CPU(s) (initiator) and the Generic Port. > +This helps with constructing the performance coordinates for CXL devices that > +are hot-plugged and not able to be enumerated by the platform firmware. This is used to construct performance coordinates for hotplugged CXL devices, which cannot be enumerated at boot by platform firmware. > + > +Example :: > + > + Subtable Type : 06 [Generic Port Affinity] > + Length : 20 <- 32d, length of table > + Reserved : 00 > + Device Handle Type : 00 <- 0 - ACPI, 1 - PCI > + Proximity Domain : 00000001 > + Device Handle : ACPI0016:01 > + Flags : 00000001 <- Bit 0 (Enabled) > + Reserved : 00000000 > -- > 2.49.0 > ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH 4/4] cxl: doc/linux/access-coordinates Update access coordinates calculation methods 2025-05-14 0:31 [PATCH 0/4] cxl: Update CXL documentation for access coordinates calculation Dave Jiang ` (2 preceding siblings ...) 2025-05-14 0:31 ` [PATCH 3/4] cxl: docs/platform/acpi/srat Add generic target documentation Dave Jiang @ 2025-05-14 0:31 ` Dave Jiang 2025-05-14 16:30 ` Gregory Price 3 siblings, 1 reply; 13+ messages in thread From: Dave Jiang @ 2025-05-14 0:31 UTC (permalink / raw) To: linux-cxl Cc: dave, jonathan.cameron, alison.schofield, vishal.l.verma, ira.weiny, dan.j.williams, gourry Add documentation on how to calculate the access coordinates for a given CXL region in detail. Signed-off-by: Dave Jiang <dave.jiang@intel.com> --- .../cxl/linux/access-coordinates.rst | 87 +++++++++++++++++++ 1 file changed, 87 insertions(+) diff --git a/Documentation/driver-api/cxl/linux/access-coordinates.rst b/Documentation/driver-api/cxl/linux/access-coordinates.rst index 71024fa0f561..cf86920f083a 100644 --- a/Documentation/driver-api/cxl/linux/access-coordinates.rst +++ b/Documentation/driver-api/cxl/linux/access-coordinates.rst @@ -5,6 +5,84 @@ CXL Access Coordinates Computation ================================== +Latency and Bandwidth Calculation +================================= +A memory region performance coordinates (latency and bandwidth) are typically +provided via ACPI tables :doc:`SRAT <../platform/acpi/srat>` and +:doc:`HMAT <../platform/acpi/hmat>`. However, the platform firmware (BIOS) is +not able to annotate those for CXL devices that are hot-plugged since they do +not exist during platform firmware initialization. The CXL driver can compute +the performance coordinates by retrieving data from several components. + +The :doc:`SRAT <../platform/acpi/srat>` provides a Generic Port Affinity +subtable that ties a proximity domain to a device handle, which in this case +would be the CXL hostbridge. Using this association, the performance +coordinates for the Generic Port can be retrieved from the +:doc:`HMAT <../platform/acpi/hmat>` subtable. This piece represents the +performance coordinates between a CPU to the Generic Port (CXL hostbridge). + +The :doc:`CDAT <../platform/cdat>` provides the performance coordinates for +the CXL device itself. That is the bandwidth and latency to access that device's +memory region. The :doc:`DSMAS <../platform/cdat/dsmas>` subtable provides a +DSMAD handle that is tied to a Device Physical Address (DPA) range. The +:doc:`DSLBIS <../platform/cdat/dslbis>` subtable provides the performance +coordinates that's tied to a DSMAD handle and this ties the two table +entries together to provide the performance coordinates for each DPA +region. For example, if a device exports a DRAM region and a PMEM region, +then there would be different performance characteristsics for each of those +regions. + +If there's a CXL switch in the topology, then the performance coordinates for the +switch is provided by :doc:`SSLBIS <../platform/cdat/sslbis>` subtable. This +provides the bandwidth and latency for traversing the switch between the switch +upstream port and the switch downstream port that points to the endpoint device. + +Simple topology example:: + + GP0/HB0/ACPI0016-0 + RP0 + | + | L0 + | + SW 0 / USP0 + SW 0 / DSP0 + | + | L1 + | + EP0 + +In this example, there is a CXL switch between an endpoint and a root port. +Latency in this example is calculated as such: +L(EP0) - Latency from EP0 CDAT DSMAS+DSLBIS +L(L1) - Link latency between EP0 and SW0DSP0 +L(SW0) - Latency for the switch from SW0 CDAT SSLBIS. +L(L0) - Link latency between SW0 and RP0 +L(RP0) - Latency from root port to CPU via SRAT and HMAT (Generic Port). +Total read and write latencies are the sum of all these parts. + +Bandwidth in this example is calculated as such: +B(EP0) - Bandwidth from EP0 CDAT DSMAS+DSLBIS +B(L1) - Link bandwidth between EP0 and SW0DSP0 +B(SW0) - Bandwidth for the switch from SW0 CDAT SSLBIS. +B(L0) - Link bandwidth between SW0 and RP0 +B(RP0) - Bandwidth from root port to CPU via SRAT and HMAT (Generic Port). +The total read and write bandwidth is the min() of all these parts. + +To calculate the link bandwidth: +LinkOperatingFrequency (GT/s) is the current negotiated link speed. +DataRatePerLink (MB/s) = LinkOperatingFrequency / 8 +Bandwidth (MB/s) = PCIeCurrentLinkWidth * DataRatePerLink +Where PCIeCurrentLinkWidth is the number of lanes in the link. + +To calculate the link latency: +LinkLatency (picoseconds) = FlitSize / LinkBandwidth (MB/s) + +See `CXL Memory Device SW Guide r1.0 <https://www.intel.com/content/www/us/en/content-details/643805/cxl-memory-device-software-guide.html>`_, +section 2.11.3 and 2.11.4 for details. + +In the end, the access coordinates for a constructed memory region is calculated from one +or more memory partitions from each of the CXL device(s). + Shared Upstream Link Calculation ================================ For certain CXL region construction with endpoints behind CXL switches (SW) or @@ -90,3 +168,12 @@ under the same ACPI0017 device to form a new xarray. Finally, the cxl_region_update_bandwidth() is called and the aggregated bandwidth from all the members of the last xarray is updated for the access coordinates residing in the cxl region (cxlr) context. + +QTG ID +====== +Each :doc:`CEDT <../platform/acpi/cedt>` has a QTG ID field. This field provides +the ID that associates with a QoS Throttling Group (QTG) for the CFMWS window. +Once the access coordinates are calculated, an ACPI Device Specific Method can +be issued to the ACPI0016 device to retrieve the QTG ID depends on the access +coordinates provided. The QTG ID for the device can be used as guidance to match +to the CFMWS to setup the best Linux root decoder for the device performance. -- 2.49.0 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH 4/4] cxl: doc/linux/access-coordinates Update access coordinates calculation methods 2025-05-14 0:31 ` [PATCH 4/4] cxl: doc/linux/access-coordinates Update access coordinates calculation methods Dave Jiang @ 2025-05-14 16:30 ` Gregory Price 0 siblings, 0 replies; 13+ messages in thread From: Gregory Price @ 2025-05-14 16:30 UTC (permalink / raw) To: Dave Jiang Cc: linux-cxl, dave, jonathan.cameron, alison.schofield, vishal.l.verma, ira.weiny, dan.j.williams On Tue, May 13, 2025 at 05:31:33PM -0700, Dave Jiang wrote: > Add documentation on how to calculate the access coordinates for a given > CXL region in detail. > > Signed-off-by: Dave Jiang <dave.jiang@intel.com> This is awesome, thanks Dave! Minor nits inline Reviewed-by: Gregory Price <gourry@gourry.net> > --- > .../cxl/linux/access-coordinates.rst | 87 +++++++++++++++++++ > 1 file changed, 87 insertions(+) > > diff --git a/Documentation/driver-api/cxl/linux/access-coordinates.rst b/Documentation/driver-api/cxl/linux/access-coordinates.rst > index 71024fa0f561..cf86920f083a 100644 > --- a/Documentation/driver-api/cxl/linux/access-coordinates.rst > +++ b/Documentation/driver-api/cxl/linux/access-coordinates.rst > @@ -5,6 +5,84 @@ > CXL Access Coordinates Computation > ================================== > > +Latency and Bandwidth Calculation > +================================= > +A memory region performance coordinates (latency and bandwidth) are typically > +provided via ACPI tables :doc:`SRAT <../platform/acpi/srat>` and > +:doc:`HMAT <../platform/acpi/hmat>`. However, the platform firmware (BIOS) is > +not able to annotate those for CXL devices that are hot-plugged since they do > +not exist during platform firmware initialization. The CXL driver can compute > +the performance coordinates by retrieving data from several components. > + > +The :doc:`SRAT <../platform/acpi/srat>` provides a Generic Port Affinity > +subtable that ties a proximity domain to a device handle, which in this case > +would be the CXL hostbridge. Using this association, the performance > +coordinates for the Generic Port can be retrieved from the > +:doc:`HMAT <../platform/acpi/hmat>` subtable. This piece represents the > +performance coordinates between a CPU to the Generic Port (CXL hostbridge). "from a CPU to the Generic Port", or "between a CPU and a Generic Port" either works > + > +The :doc:`CDAT <../platform/cdat>` provides the performance coordinates for > +the CXL device itself. That is the bandwidth and latency to access that device's > +memory region. The :doc:`DSMAS <../platform/cdat/dsmas>` subtable provides a > +DSMAD handle that is tied to a Device Physical Address (DPA) range. The Similar to other doc - need to define DSMAD. > +:doc:`DSLBIS <../platform/cdat/dslbis>` subtable provides the performance > +coordinates that's tied to a DSMAD handle and this ties the two table > +entries together to provide the performance coordinates for each DPA > +region. For example, if a device exports a DRAM region and a PMEM region, > +then there would be different performance characteristsics for each of those > +regions. > + > +If there's a CXL switch in the topology, then the performance coordinates for the > +switch is provided by :doc:`SSLBIS <../platform/cdat/sslbis>` subtable. This > +provides the bandwidth and latency for traversing the switch between the switch > +upstream port and the switch downstream port that points to the endpoint device. > + > +Simple topology example:: > + > + GP0/HB0/ACPI0016-0 > + RP0 > + | > + | L0 > + | > + SW 0 / USP0 > + SW 0 / DSP0 > + | > + | L1 > + | > + EP0 > + > +In this example, there is a CXL switch between an endpoint and a root port. > +Latency in this example is calculated as such: > +L(EP0) - Latency from EP0 CDAT DSMAS+DSLBIS > +L(L1) - Link latency between EP0 and SW0DSP0 > +L(SW0) - Latency for the switch from SW0 CDAT SSLBIS. > +L(L0) - Link latency between SW0 and RP0 > +L(RP0) - Latency from root port to CPU via SRAT and HMAT (Generic Port). > +Total read and write latencies are the sum of all these parts. > + > +Bandwidth in this example is calculated as such: > +B(EP0) - Bandwidth from EP0 CDAT DSMAS+DSLBIS > +B(L1) - Link bandwidth between EP0 and SW0DSP0 > +B(SW0) - Bandwidth for the switch from SW0 CDAT SSLBIS. > +B(L0) - Link bandwidth between SW0 and RP0 > +B(RP0) - Bandwidth from root port to CPU via SRAT and HMAT (Generic Port). > +The total read and write bandwidth is the min() of all these parts. > + > +To calculate the link bandwidth: > +LinkOperatingFrequency (GT/s) is the current negotiated link speed. > +DataRatePerLink (MB/s) = LinkOperatingFrequency / 8 > +Bandwidth (MB/s) = PCIeCurrentLinkWidth * DataRatePerLink > +Where PCIeCurrentLinkWidth is the number of lanes in the link. > + > +To calculate the link latency: > +LinkLatency (picoseconds) = FlitSize / LinkBandwidth (MB/s) > + > +See `CXL Memory Device SW Guide r1.0 <https://www.intel.com/content/www/us/en/content-details/643805/cxl-memory-device-software-guide.html>`_, > +section 2.11.3 and 2.11.4 for details. > + > +In the end, the access coordinates for a constructed memory region is calculated from one > +or more memory partitions from each of the CXL device(s). > + > Shared Upstream Link Calculation > ================================ > For certain CXL region construction with endpoints behind CXL switches (SW) or > @@ -90,3 +168,12 @@ under the same ACPI0017 device to form a new xarray. > Finally, the cxl_region_update_bandwidth() is called and the aggregated > bandwidth from all the members of the last xarray is updated for the > access coordinates residing in the cxl region (cxlr) context. > + > +QTG ID > +====== > +Each :doc:`CEDT <../platform/acpi/cedt>` has a QTG ID field. This field provides > +the ID that associates with a QoS Throttling Group (QTG) for the CFMWS window. > +Once the access coordinates are calculated, an ACPI Device Specific Method can > +be issued to the ACPI0016 device to retrieve the QTG ID depends on the access > +coordinates provided. The QTG ID for the device can be used as guidance to match > +to the CFMWS to setup the best Linux root decoder for the device performance. > -- > 2.49.0 > ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2025-05-14 22:15 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-05-14 0:31 [PATCH 0/4] cxl: Update CXL documentation for access coordinates calculation Dave Jiang 2025-05-14 0:31 ` [PATCH 1/4] cxl: docs/platform/cdat reference documentation Dave Jiang 2025-05-14 16:12 ` Gregory Price 2025-05-14 19:02 ` Dave Jiang 2025-05-14 21:59 ` Gregory Price 2025-05-14 22:15 ` Dave Jiang 2025-05-14 0:31 ` [PATCH 2/4] cxl: docs/platform/acpi/srat fix memory table misalignment Dave Jiang 2025-05-14 1:40 ` Gregory Price 2025-05-14 15:22 ` Dave Jiang 2025-05-14 0:31 ` [PATCH 3/4] cxl: docs/platform/acpi/srat Add generic target documentation Dave Jiang 2025-05-14 16:15 ` Gregory Price 2025-05-14 0:31 ` [PATCH 4/4] cxl: doc/linux/access-coordinates Update access coordinates calculation methods Dave Jiang 2025-05-14 16:30 ` Gregory Price
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox