Re: [PATCH v3 07/17] cxl: docs/linux - overview

Dave Jiang
2025-05-13T00:09:03Z

On 5/12/25 9:21 AM, Gregory Price wrote:
> Add type-3 device configuration overview that explains the probe
> process for a type-3 device from early-boot through memory-hotplug.
>
> Signed-off-by: Gregory Price <gourry@gourry.net>

Reviewed-by: Dave Jiang <dave.jiang@intel.com>
> ---
> Documentation/driver-api/cxl/index.rst | 3 +-
> .../driver-api/cxl/linux/overview.rst | 103 ++++++++++++++++++
> 2 files changed, 105 insertions(+), 1 deletion(-)
> create mode 100644 Documentation/driver-api/cxl/linux/overview.rst
>
> diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst
> index 6a5fb7e00c52..bc2228c77c32 100644
> --- a/Documentation/driver-api/cxl/index.rst
> +++ b/Documentation/driver-api/cxl/index.rst
> @@ -30,9 +30,10 @@ that have impacts on each other. The docs here break up configurations steps.
> platform/example-configs
>
> .. toctree::
> - :maxdepth: 1
> + :maxdepth: 2
> :caption: Linux Kernel Configuration
>
> + linux/overview
> linux/access-coordinates
>
>
> diff --git a/Documentation/driver-api/cxl/linux/overview.rst b/Documentation/driver-api/cxl/linux/overview.rst
> new file mode 100644
> index 000000000000..648beb2c8c83
> --- /dev/null
> +++ b/Documentation/driver-api/cxl/linux/overview.rst
> @@ -0,0 +1,103 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +========
> +Overview
> +========
> +
> +This section presents the configuration process of a CXL Type-3 memory device,
> +and how it is ultimately exposed to users as either a :code:`DAX` device or
> +normal memory pages via the kernel's page allocator.
> +
> +Portions marked with a bullet are points at which certain kernel objects
> +are generated.
> +
> +1) Early Boot
> +
> + a) BIOS, Build, and Boot Parameters
> +
> + i) EFI_MEMORY_SP
> + ii) CONFIG_EFI_SOFT_RESERVE
> + iii) CONFIG_MHP_DEFAULT_ONLINE_TYPE
> + iv) nosoftreserve
> +
> + b) Memory Map Creation
> +
> + i) EFI Memory Map / E820 Consulted for Soft-Reserved
> +
> + * CXL Memory is set aside to be handled by the CXL driver
> +
> + * Soft-Reserved IO Resource created for CFMWS entry
> +
> + c) NUMA Node Creation
> +
> + * Nodes created from ACPI CEDT CFMWS and SRAT Proximity domains (PXM)
> +
> + d) Memory Tier Creation
> +
> + * A default memory_tier is created with all nodes.
> +
> + e) Contiguous Memory Allocation
> +
> + * Any requested CMA is allocated from Online nodes
> +
> + f) Init Finishes, Drivers start probing
> +
> +2) ACPI and PCI Drivers
> +
> + a) Detects PCI device is CXL, marking it for probe by CXL driver
> +
> +3) CXL Driver Operation
> +
> + a) Base device creation
> +
> + * root, port, and memdev devices created
> + * CEDT CFMWS IO Resource creation
> +
> + b) Decoder creation
> +
> + * root, switch, and endpoint decoders created
> +
> + c) Logical device creation
> +
> + * memory_region and endpoint devices created
> +
> + d) Devices are associated with each other
> +
> + * If auto-decoder (BIOS-programmed decoders), driver validates
> + configurations, builds associations, and locks configs at probe time.
> +
> + * If user-configured, validation and associations are built at
> + decoder-commit time.
> +
> + e) Regions surfaced as DAX region
> +
> + * dax_region created
> +
> + * DAX device created via DAX driver
> +
> +4) DAX Driver Operation
> +
> + a) DAX driver surfaces DAX region as one of two dax device modes
> +
> + * kmem - dax device is converted to hotplug memory blocks
> +
> + * DAX kmem IO Resource creation
> +
> + * hmem - dax device is left as daxdev to be accessed as a file.
> +
> + * If hmem, journey ends here.
> +
> + b) DAX kmem surfaces memory region to Memory Hotplug to add to page
> + allocator as "driver managed memory"
> +
> +5) Memory Hotplug
> +
> + a) mhp component surfaces a dax device memory region as multiple memory
> + blocks to the page allocator
> +
> + * blocks appear in :code:`/sys/bus/memory/devices` and linked to a NUMA node
> +
> + b) blocks are onlined into the requested zone (NORMAL or MOVABLE)
> +
> + * Memory is marked "Driver Managed" to avoid kexec from using it as region
> + for kernel updates