Re: [PATCH v10 00/22] cxl: Add support for QTG ID retrieval for CXL subsystem

[Jonathan Cameron <Jonathan.Cameron@Huawei.com>]

On Tue, 10 Oct 2023 18:04:56 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

Awkward to rename a patch series, but feels like the QTG ID part may be the last thing
added but there is a whole bunch of stuff in between to enable that which
isn't covered by the one line description.

J

> v10:
> - Remove memory allocation in DSM handler. Add input parameters to request number of ids
>   to return. (Dan)
> - Only export a single qos_class sysfs attribute, qos_class0 for devices. (Dan)
> - Add sanity check of device qos_class against root decoders (Dan)
> - Recombine all relevant patches for cxl upstream merge
> - Rebased against v6.6-rc4
> 
> v9:
> - Correct DSM input package to use integers. (Dan)
> - Remove all endien conversions. (Dan)
> 
> v8:
> - Correct DSM return package parsing to use integers
> 
> v7:
> - Minor changes. Please see specific patches for log entries addressing comments
>   from v6.
> 
> v6:
> - Please see specific patches for log entries addressing comments from v5.
> - Use CDAT sub-table structs as is, adjust w/o common header. Changing causes
>   significant ACPICA code changes.
> - Deal with table header now that is a union. (Jonathan)
> - Move DSMAS list destroy to core/cdat.c. (Jonathan)
> - Retain and display entire QTG ID list from _DSM. (Jonathan)
> 
> v5:
> - Please see specific patches for log entries addressing comments from v4.
> - Split out ACPI and generic node code to send separately to respective maintainers
> - Reworked to use ACPI tables code for CDAT parsing (Dan)
> - Cache relevant perf data under dports (Dan)
> - Add cxl root callback for QTG ID _DSM (Dan)
> - Rename 'node_hmem_attr' to 'access_coordinate' (Dan)
> - Change qtg_id sysfs attrib to qos_class (Dan)
> 
> v4:
> - Reworked PCIe link path latency calculation
> - 0-day fixes
> - Removed unused qos_list from cxl_memdev and its stray usages
> 
> v3:
> - Please see specific patches for log entries addressing comments from v2.
> - Refactor cxl_port_probe() additions. (Alison)
> - Convert to use 'struct node_hmem_attrs'
> - Refactor to use common code for genport target allocation.
> - Add third array entry for target hmem_attrs to store genport locality data.
> - Go back to per partition QTG ID. (Dan)
> 
> v2:
> - Please see specific patches for log entries addressing comments from v1.
> - Removed ACPICA code usages.
> - Removed PCI subsystem helpers for latency and bandwidth.
> - Add CXL switch CDAT parsing support (SSLBIS)
> - Add generic port SRAT+HMAT support (ACPI)
> - Export a single QTG ID via sysfs per memory device (Dan)
> - Provide rest of DSMAS range info in debugfs (Dan)
> 
> Hi Dan,
> Please consider taking the entire series including the CXL bits for the next
> convenient merge window. Thanks!
> 
> This series adds the retrieval of QoS Throttling Group (QTG) IDs for the CXL Fixed
> Memory Window Structure (CFMWS) and the CXL memory device. It provides the QTG IDs
> to user space to provide some guidance with putting the proper DPA range under the
> appropriate CFMWS window for a hot-plugged CXL memory device.
> 
> The CFMWS structure contains a QTG ID that is associated with the memory window that the
> structure exports. On Linux, the CFMWS is represented as a CXL root decoder. The QTG
> ID will be attached to the CXL root decoder and exported as a sysfs attribute (qos_class).
> 
> The QTG IDs for a device is retrieved via sending a _DSM method to the ACPI0017 device.
> The _DSM expects an input package of 4 DWORDS that contains the read latency, write
> latency, read bandwidth, and write banwidth. These are the caluclated numbers for the
> path between the CXL device and the CPU. The list of QTG IDs are also exported as a sysfs
> attribute under the mem device memory partition type:
> /sys/bus/cxl/devices/memX/ram/qos_class
> /sys/bus/cxl/devices/memX/pmem/qos_class
> A mapping of DPA ranges and it's correlated QTG IDs are found under
> /sys/kernel/debug/cxl/memX/qtgmap. Each DSMAS from the device CDAT will provide a DPA
> range.
> 
> The latency numbers are the aggregated latencies for the path between the CXL device and
> the CPU. If a CXL device is directly attached to the CXL HB, the latency
> would be the aggregated latencies from the device Coherent Device Attribute Table (CDAT),
> the caluclated PCIe link latency between the device and the HB, and the generic port data
> from ACPI SRAT+HMAT. The bandwidth in this configuration would be the minimum between the
> CDAT bandwidth number, link bandwidth between the device and the HB, and the bandwidth data
> from the generic port data via ACPI SRAT+HMAT.
> 
> If a configuration has a switch in between then the latency would be the aggregated
> latencies from the device CDAT, the link latency between device and switch, the
> latency from the switch CDAT, the link latency between switch and the HB, and the
> generic port latency between the CPU and the CXL HB. The bandwidth calculation would be the
> min of device CDAT bandwidth, link bandwith between device and switch, switch CDAT
> bandwidth, the link bandwidth between switch and HB, and the generic port bandwidth
> 
> There can be 0 or more switches between the CXL device and the CXL HB. There are detailed
> examples on calculating bandwidth and latency in the CXL Memory Device Software Guide [4].
> 
> The CDAT provides Device Scoped Memory Affinity Structures (DSMAS) that contains the
> Device Physical Address (DPA) range and the related Device Scoped Latency and Bandwidth
> Informat Stuctures (DSLBIS). Each DSLBIS provides a latency or bandwidth entry that is
> tied to a DSMAS entry via a per DSMAS unique DSMAD handle.
> 
> Test setup is done with runqemu genport support branch [6]. The setup provides 2 CXL HBs
> with one HB having a CXL switch underneath. It also provides generic port support detailed
> below.
> 
> A hacked up qemu branch is used to support generic port SRAT and HMAT [7].
> 
> To create the appropriate HMAT entries for generic port, the following qemu paramters must
> be added:
> 
> -object genport,id=$X -numa node,genport=genport$X,nodeid=$Y,initiator=$Z
> -numa hmat-lb,initiator=$Z,target=$X,hierarchy=memory,data-type=access-latency,latency=$latency
> -numa hmat-lb,initiator=$Z,target=$X,hierarchy=memory,data-type=access-bandwidth,bandwidth=$bandwidthM
> for ((i = 0; i < total_nodes; i++)); do
> 	for ((j = 0; j < cxl_hbs; j++ )); do	# 2 CXL HBs
> 		-numa dist,src=$i,dst=$X,val=$dist
> 	done
> done
> 
> See the genport run_qemu branch for full details.
> 
> [1]: https://www.computeexpresslink.org/download-the-specification
> [2]: https://uefi.org/sites/default/files/resources/Coherent%20Device%20Attribute%20Table_1.01.pdf
> [3]: https://uefi.org/sites/default/files/resources/ACPI_Spec_6_5_Aug29.pdf
> [4]: https://cdrdv2-public.intel.com/643805/643805_CXL%20Memory%20Device%20SW%20Guide_Rev1p0.pdf
> [5]: https://lore.kernel.org/linux-cxl/20230313195530.GA1532686@bhelgaas/T/#t
> [6]: https://git.kernel.org/pub/scm/linux/kernel/git/djiang/linux.git/log/?h=cxl-qtg
> [7]: https://github.com/pmem/run_qemu/tree/djiang/genport
> [8]: https://github.com/davejiang/qemu/tree/genport
> 
> ---
> 
> base-commit: 178e1ea6a68f12967ee0e9afc4d79a2939acd43c
> 
> ---
> Dave Jiang (22):
>       cxl: Export QTG ids from CFMWS to sysfs as qos_class attribute
>       cxl: Add checksum verification to CDAT from CXL
>       cxl: Add support for reading CXL switch CDAT table
>       acpi: Move common tables helper functions to common lib
>       lib/firmware_table: tables: Add CDAT table parsing support
>       base/node / acpi: Change 'node_hmem_attrs' to 'access_coordinates'
>       acpi: numa: Create enum for memory_target access coordinates indexing
>       acpi: numa: Add genport target allocation to the HMAT parsing
>       acpi: Break out nesting for hmat_parse_locality()
>       acpi: numa: Add setting of generic port system locality attributes
>       acpi: numa: Add helper function to retrieve the performance attributes
>       cxl: Add callback to parse the DSMAS subtables from CDAT
>       cxl: Add callback to parse the DSLBIS subtable from CDAT
>       cxl: Add callback to parse the SSLBIS subtable from CDAT
>       cxl: Add support for _DSM Function for retrieving QTG ID
>       cxl: Calculate and store PCI link latency for the downstream ports
>       cxl: Store the access coordinates for the generic ports
>       cxl: Add helper function that calculate performance data for downstream ports
>       cxl: Compute the entire CXL path latency and bandwidth data
>       cxl: Store QTG IDs and related info to the CXL memory device context
>       cxl: Export sysfs attributes for memory device QoS class
>       cxl: Check qos_class validity on memdev probe
> 
> 
>  Documentation/ABI/testing/sysfs-bus-cxl |  49 +++++
>  MAINTAINERS                             |   2 +
>  drivers/acpi/Kconfig                    |   1 +
>  drivers/acpi/numa/hmat.c                | 172 +++++++++++++---
>  drivers/acpi/tables.c                   | 178 +----------------
>  drivers/base/node.c                     |  12 +-
>  drivers/cxl/Kconfig                     |   1 +
>  drivers/cxl/acpi.c                      | 151 +++++++++++++-
>  drivers/cxl/core/Makefile               |   1 +
>  drivers/cxl/core/cdat.c                 | 249 ++++++++++++++++++++++++
>  drivers/cxl/core/mbox.c                 |   1 +
>  drivers/cxl/core/memdev.c               |  34 ++++
>  drivers/cxl/core/pci.c                  | 125 ++++++++++--
>  drivers/cxl/core/port.c                 | 124 +++++++++++-
>  drivers/cxl/cxl.h                       |  74 +++++++
>  drivers/cxl/cxlmem.h                    |  23 +++
>  drivers/cxl/cxlpci.h                    |  15 ++
>  drivers/cxl/mem.c                       |  68 +++++++
>  drivers/cxl/port.c                      | 109 +++++++++++
>  include/linux/acpi.h                    |  54 +++--
>  include/linux/fw_table.h                |  52 +++++
>  include/linux/node.h                    |   8 +-
>  lib/Kconfig                             |   3 +
>  lib/Makefile                            |   2 +
>  lib/fw_table.c                          | 237 ++++++++++++++++++++++
>  25 files changed, 1478 insertions(+), 267 deletions(-)
>  create mode 100644 drivers/cxl/core/cdat.c
>  create mode 100644 include/linux/fw_table.h
>  create mode 100644 lib/fw_table.c
> 
> --
> 
>