* Re: [PATCH v3 2/2] Documentation/driver-api/cxl: device hotplug section
2025-12-19 17:05 ` [PATCH v3 2/2] Documentation/driver-api/cxl: device hotplug section Gregory Price
@ 2025-12-20 6:56 ` Alejandro Lucero Palau
2026-01-05 21:10 ` Dave Jiang
2026-01-07 2:19 ` Alison Schofield
2 siblings, 0 replies; 13+ messages in thread
From: Alejandro Lucero Palau @ 2025-12-20 6:56 UTC (permalink / raw)
To: Gregory Price, linux-cxl
Cc: linux-doc, linux-kernel, kernel-team, dave, jonathan.cameron,
dave.jiang, alison.schofield, vishal.l.verma, ira.weiny,
dan.j.williams, corbet, rakuram.e96
On 12/19/25 17:05, Gregory Price wrote:
> Describe cxl memory device hotplug implications, in particular how the
> platform CEDT CFMWS must be described to support successful hot-add of
> memory devices.
>
> Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
> Signed-off-by: Gregory Price <gourry@gourry.net>
Reviewed-by: Alejandro Lucero Palau <alucerop@amd.com>
> ---
> v3: Jonathan updates, change some italics to bold, add some bits about
> Linux's expectations for BIOS/EFI behavior at runtime.
>
> Documentation/driver-api/cxl/index.rst | 1 +
> .../driver-api/cxl/platform/bios-and-efi.rst | 3 +
> .../cxl/platform/device-hotplug.rst | 130 ++++++++++++++++++
> 3 files changed, 134 insertions(+)
> create mode 100644 Documentation/driver-api/cxl/platform/device-hotplug.rst
>
> diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst
> index c1106a68b67c..5a734988a5af 100644
> --- a/Documentation/driver-api/cxl/index.rst
> +++ b/Documentation/driver-api/cxl/index.rst
> @@ -30,6 +30,7 @@ that have impacts on each other. The docs here break up configurations steps.
> platform/acpi
> platform/cdat
> platform/example-configs
> + platform/device-hotplug
>
> .. toctree::
> :maxdepth: 2
> diff --git a/Documentation/driver-api/cxl/platform/bios-and-efi.rst b/Documentation/driver-api/cxl/platform/bios-and-efi.rst
> index 9034c206cf8e..a4b44c018f09 100644
> --- a/Documentation/driver-api/cxl/platform/bios-and-efi.rst
> +++ b/Documentation/driver-api/cxl/platform/bios-and-efi.rst
> @@ -49,6 +49,9 @@ up without requiring CXL driver support. These platform vendors should
> test their configurations with the existing CXL driver and provide driver
> support for their auto-configurations if features like RAS are required.
>
> +Platforms requiring boot-time programming and/or locking of CXL fabric
> +components may prevent features, such as device hot-plug, from working.
> +
> UEFI Settings
> =============
> If your platform supports it, the :code:`uefisettings` command can be used to
> diff --git a/Documentation/driver-api/cxl/platform/device-hotplug.rst b/Documentation/driver-api/cxl/platform/device-hotplug.rst
> new file mode 100644
> index 000000000000..e4a065fdd3ec
> --- /dev/null
> +++ b/Documentation/driver-api/cxl/platform/device-hotplug.rst
> @@ -0,0 +1,130 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==================
> +CXL Device Hotplug
> +==================
> +
> +Device hotplug refers to *physical* hotplug of a device (addition or removal
> +of a physical device from the machine).
> +
> +BIOS/EFI software is expected to configure sufficient resources **at boot
> +time** to allow hotplugged devices to be configured by software (such as
> +proximity domains, HPA regions, and host-bridge configurations).
> +
> +BIOS/EFI is not expected (**nor suggested**) to configure hotplugged
> +devices at hotplug time (i.e. HDM decoders should be left unprogrammed).
> +
> +This document covers some examples of those resources, but should not
> +be considered exhaustive.
> +
> +Hot-Remove
> +==========
> +Hot removal of a device typically requires careful removal of software
> +constructs (memory regions, associated drivers) which manage these devices.
> +
> +Hard-removing a CXL.mem device without carefully tearing down driver stacks
> +is likely to cause the system to machine-check (or at least SIGBUS if memory
> +access is limited to user space).
> +
> +Memory Device Hot-Add
> +=====================
> +A device present at boot may be associated with a CXL Fixed Memory Window
> +reported in :doc:`CEDT<acpi/cedt>`. That CFMWS may match the size of the
> +device, but the construction of the CEDT CFMWS is platform-defined.
> +
> +Hot-adding a memory device requires this pre-defined, **static** CFMWS to
> +have sufficient HPA space to describe that device.
> +
> +There are a few common scenarios to consider.
> +
> +Single-Endpoint Memory Device Present at Boot
> +---------------------------------------------
> +A device present at boot likely had its capacity reported in the
> +:doc:`CEDT<acpi/cedt>`. If a device is removed and a new device hotplugged,
> +the capacity of the new device will be limited to the original CFMWS capacity.
> +
> +Adding capacity larger than the original device will cause memory region
> +creation to fail if the region size is greater than the CFMWS size.
> +
> +The CFMWS is **static** and cannot be adjusted. Platforms which may expect
> +different sized devices to be hotplugged must allocate sufficient CFMWS space
> +**at boot time** to cover all future expected devices.
> +
> +Multi-Endpoint Memory Device Present at Boot
> +--------------------------------------------
> +Non-switch-based Multi-Endpoint devices are outside the scope of what the
> +CXL specification describes, but they are technically possible. We describe
> +them here for instructive reasons only - this does not imply Linux support.
> +
> +A hot-plug capable CXL memory device, such as one which presents multiple
> +expanders as a single large-capacity device, should report the **maximum
> +possible capacity** for the device at boot. ::
> +
> + HB0
> + RP0
> + |
> + [Multi-Endpoint Memory Device]
> + _____|_____
> + | |
> + [Endpoint0] [Empty]
> +
> +
> +Limiting the size to the capacity preset at boot will limit hot-add support
> +to replacing capacity that was present at boot.
> +
> +No CXL Device Present at Boot
> +-----------------------------
> +When no CXL memory device is present on boot, some platforms omit the CFMWS
> +in the :doc:`CEDT<acpi/cedt>`. When this occurs, hot-add is not possible.
> +
> +This describes the base case for any given device not being present at boot.
> +If a future possible device is not described in the CEDT at boot, hot-add
> +of that device is either limited or not possible.
> +
> +For a platform to support hot-add of a full memory device, it must allocate
> +a CEDT CFMWS region with sufficient memory capacity to cover all future
> +potentially added capacity (along with any relevant CEDT CHBS entry).
> +
> +To support memory hotplug directly on the host bridge/root port, or on a switch
> +downstream of the host bridge, a platform must construct a CEDT CFMWS at boot
> +with sufficient resources to support the max possible (or expected) hotplug
> +memory capacity. ::
> +
> + HB0 HB1
> + RP0 RP1 RP2
> + | | |
> + Empty Empty USP
> + ________|________
> + | | | |
> + DSP DSP DSP DSP
> + | | | |
> + All Empty
> +
> +For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with
> +a pre-configured amount of memory capacity (per host bridge, or host bridge
> +interleave set), even if no device is attached to Root Ports or Downstream
> +Ports at boot (as depicted in the figure above).
> +
> +
> +Interleave Sets
> +===============
> +
> +Host Bridge Interleave
> +----------------------
> +Host-bridge interleaved memory regions are defined **statically** in the
> +:doc:`CEDT<acpi/cedt>`. To apply cross-host-bridge interleave, a CFMWS entry
> +describing that interleave must have been provided **at boot**. Hotplugged
> +devices cannot add host-bridge interleave capabilities at hotplug time.
> +
> +See the :doc:`Flexible CEDT Configuration<example-configurations/flexible>`
> +example to see how a platform can provide this kind of flexibility regarding
> +hotplugged memory devices. BIOS/EFI software should consider options to
> +present flexible CEDT configurations with hotplug support.
> +
> +HDM Interleave
> +--------------
> +Decoder-applied interleave can flexibly handle hotplugged devices, as decoders
> +can be re-programmed after hotplug.
> +
> +To add or remove a device to/from an existing HDM-applied interleaved region,
> +that region must be torn down an re-created.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v3 2/2] Documentation/driver-api/cxl: device hotplug section
2025-12-19 17:05 ` [PATCH v3 2/2] Documentation/driver-api/cxl: device hotplug section Gregory Price
2025-12-20 6:56 ` Alejandro Lucero Palau
@ 2026-01-05 21:10 ` Dave Jiang
2026-01-07 2:19 ` Alison Schofield
2 siblings, 0 replies; 13+ messages in thread
From: Dave Jiang @ 2026-01-05 21:10 UTC (permalink / raw)
To: Gregory Price, linux-cxl
Cc: linux-doc, linux-kernel, kernel-team, dave, jonathan.cameron,
alison.schofield, vishal.l.verma, ira.weiny, dan.j.williams,
corbet, rakuram.e96, alucerop
On 12/19/25 10:05 AM, Gregory Price wrote:
> Describe cxl memory device hotplug implications, in particular how the
> platform CEDT CFMWS must be described to support successful hot-add of
> memory devices.
>
> Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
> Signed-off-by: Gregory Price <gourry@gourry.net>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
> ---
> v3: Jonathan updates, change some italics to bold, add some bits about
> Linux's expectations for BIOS/EFI behavior at runtime.
>
> Documentation/driver-api/cxl/index.rst | 1 +
> .../driver-api/cxl/platform/bios-and-efi.rst | 3 +
> .../cxl/platform/device-hotplug.rst | 130 ++++++++++++++++++
> 3 files changed, 134 insertions(+)
> create mode 100644 Documentation/driver-api/cxl/platform/device-hotplug.rst
>
> diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst
> index c1106a68b67c..5a734988a5af 100644
> --- a/Documentation/driver-api/cxl/index.rst
> +++ b/Documentation/driver-api/cxl/index.rst
> @@ -30,6 +30,7 @@ that have impacts on each other. The docs here break up configurations steps.
> platform/acpi
> platform/cdat
> platform/example-configs
> + platform/device-hotplug
>
> .. toctree::
> :maxdepth: 2
> diff --git a/Documentation/driver-api/cxl/platform/bios-and-efi.rst b/Documentation/driver-api/cxl/platform/bios-and-efi.rst
> index 9034c206cf8e..a4b44c018f09 100644
> --- a/Documentation/driver-api/cxl/platform/bios-and-efi.rst
> +++ b/Documentation/driver-api/cxl/platform/bios-and-efi.rst
> @@ -49,6 +49,9 @@ up without requiring CXL driver support. These platform vendors should
> test their configurations with the existing CXL driver and provide driver
> support for their auto-configurations if features like RAS are required.
>
> +Platforms requiring boot-time programming and/or locking of CXL fabric
> +components may prevent features, such as device hot-plug, from working.
> +
> UEFI Settings
> =============
> If your platform supports it, the :code:`uefisettings` command can be used to
> diff --git a/Documentation/driver-api/cxl/platform/device-hotplug.rst b/Documentation/driver-api/cxl/platform/device-hotplug.rst
> new file mode 100644
> index 000000000000..e4a065fdd3ec
> --- /dev/null
> +++ b/Documentation/driver-api/cxl/platform/device-hotplug.rst
> @@ -0,0 +1,130 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==================
> +CXL Device Hotplug
> +==================
> +
> +Device hotplug refers to *physical* hotplug of a device (addition or removal
> +of a physical device from the machine).
> +
> +BIOS/EFI software is expected to configure sufficient resources **at boot
> +time** to allow hotplugged devices to be configured by software (such as
> +proximity domains, HPA regions, and host-bridge configurations).
> +
> +BIOS/EFI is not expected (**nor suggested**) to configure hotplugged
> +devices at hotplug time (i.e. HDM decoders should be left unprogrammed).
> +
> +This document covers some examples of those resources, but should not
> +be considered exhaustive.
> +
> +Hot-Remove
> +==========
> +Hot removal of a device typically requires careful removal of software
> +constructs (memory regions, associated drivers) which manage these devices.
> +
> +Hard-removing a CXL.mem device without carefully tearing down driver stacks
> +is likely to cause the system to machine-check (or at least SIGBUS if memory
> +access is limited to user space).
> +
> +Memory Device Hot-Add
> +=====================
> +A device present at boot may be associated with a CXL Fixed Memory Window
> +reported in :doc:`CEDT<acpi/cedt>`. That CFMWS may match the size of the
> +device, but the construction of the CEDT CFMWS is platform-defined.
> +
> +Hot-adding a memory device requires this pre-defined, **static** CFMWS to
> +have sufficient HPA space to describe that device.
> +
> +There are a few common scenarios to consider.
> +
> +Single-Endpoint Memory Device Present at Boot
> +---------------------------------------------
> +A device present at boot likely had its capacity reported in the
> +:doc:`CEDT<acpi/cedt>`. If a device is removed and a new device hotplugged,
> +the capacity of the new device will be limited to the original CFMWS capacity.
> +
> +Adding capacity larger than the original device will cause memory region
> +creation to fail if the region size is greater than the CFMWS size.
> +
> +The CFMWS is **static** and cannot be adjusted. Platforms which may expect
> +different sized devices to be hotplugged must allocate sufficient CFMWS space
> +**at boot time** to cover all future expected devices.
> +
> +Multi-Endpoint Memory Device Present at Boot
> +--------------------------------------------
> +Non-switch-based Multi-Endpoint devices are outside the scope of what the
> +CXL specification describes, but they are technically possible. We describe
> +them here for instructive reasons only - this does not imply Linux support.
> +
> +A hot-plug capable CXL memory device, such as one which presents multiple
> +expanders as a single large-capacity device, should report the **maximum
> +possible capacity** for the device at boot. ::
> +
> + HB0
> + RP0
> + |
> + [Multi-Endpoint Memory Device]
> + _____|_____
> + | |
> + [Endpoint0] [Empty]
> +
> +
> +Limiting the size to the capacity preset at boot will limit hot-add support
> +to replacing capacity that was present at boot.
> +
> +No CXL Device Present at Boot
> +-----------------------------
> +When no CXL memory device is present on boot, some platforms omit the CFMWS
> +in the :doc:`CEDT<acpi/cedt>`. When this occurs, hot-add is not possible.
> +
> +This describes the base case for any given device not being present at boot.
> +If a future possible device is not described in the CEDT at boot, hot-add
> +of that device is either limited or not possible.
> +
> +For a platform to support hot-add of a full memory device, it must allocate
> +a CEDT CFMWS region with sufficient memory capacity to cover all future
> +potentially added capacity (along with any relevant CEDT CHBS entry).
> +
> +To support memory hotplug directly on the host bridge/root port, or on a switch
> +downstream of the host bridge, a platform must construct a CEDT CFMWS at boot
> +with sufficient resources to support the max possible (or expected) hotplug
> +memory capacity. ::
> +
> + HB0 HB1
> + RP0 RP1 RP2
> + | | |
> + Empty Empty USP
> + ________|________
> + | | | |
> + DSP DSP DSP DSP
> + | | | |
> + All Empty
> +
> +For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with
> +a pre-configured amount of memory capacity (per host bridge, or host bridge
> +interleave set), even if no device is attached to Root Ports or Downstream
> +Ports at boot (as depicted in the figure above).
> +
> +
> +Interleave Sets
> +===============
> +
> +Host Bridge Interleave
> +----------------------
> +Host-bridge interleaved memory regions are defined **statically** in the
> +:doc:`CEDT<acpi/cedt>`. To apply cross-host-bridge interleave, a CFMWS entry
> +describing that interleave must have been provided **at boot**. Hotplugged
> +devices cannot add host-bridge interleave capabilities at hotplug time.
> +
> +See the :doc:`Flexible CEDT Configuration<example-configurations/flexible>`
> +example to see how a platform can provide this kind of flexibility regarding
> +hotplugged memory devices. BIOS/EFI software should consider options to
> +present flexible CEDT configurations with hotplug support.
> +
> +HDM Interleave
> +--------------
> +Decoder-applied interleave can flexibly handle hotplugged devices, as decoders
> +can be re-programmed after hotplug.
> +
> +To add or remove a device to/from an existing HDM-applied interleaved region,
> +that region must be torn down an re-created.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v3 2/2] Documentation/driver-api/cxl: device hotplug section
2025-12-19 17:05 ` [PATCH v3 2/2] Documentation/driver-api/cxl: device hotplug section Gregory Price
2025-12-20 6:56 ` Alejandro Lucero Palau
2026-01-05 21:10 ` Dave Jiang
@ 2026-01-07 2:19 ` Alison Schofield
2026-01-07 15:54 ` Gregory Price
2 siblings, 1 reply; 13+ messages in thread
From: Alison Schofield @ 2026-01-07 2:19 UTC (permalink / raw)
To: Gregory Price
Cc: linux-cxl, linux-doc, linux-kernel, kernel-team, dave,
jonathan.cameron, dave.jiang, vishal.l.verma, ira.weiny,
dan.j.williams, corbet, rakuram.e96, alucerop
On Fri, Dec 19, 2025 at 12:05:38PM -0500, Gregory Price wrote:
> Describe cxl memory device hotplug implications, in particular how the
> platform CEDT CFMWS must be described to support successful hot-add of
> memory devices.
Who is the intended audience for this doc?
Maybe it's already in another section not being edited, but I'd
expect to see CXL spec references.
>
> Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
> v3: Jonathan updates, change some italics to bold, add some bits about
> Linux's expectations for BIOS/EFI behavior at runtime.
>
> Documentation/driver-api/cxl/index.rst | 1 +
> .../driver-api/cxl/platform/bios-and-efi.rst | 3 +
> .../cxl/platform/device-hotplug.rst | 130 ++++++++++++++++++
> 3 files changed, 134 insertions(+)
> create mode 100644 Documentation/driver-api/cxl/platform/device-hotplug.rst
>
> diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst
> index c1106a68b67c..5a734988a5af 100644
> --- a/Documentation/driver-api/cxl/index.rst
> +++ b/Documentation/driver-api/cxl/index.rst
> @@ -30,6 +30,7 @@ that have impacts on each other. The docs here break up configurations steps.
> platform/acpi
> platform/cdat
> platform/example-configs
> + platform/device-hotplug
>
> .. toctree::
> :maxdepth: 2
> diff --git a/Documentation/driver-api/cxl/platform/bios-and-efi.rst b/Documentation/driver-api/cxl/platform/bios-and-efi.rst
> index 9034c206cf8e..a4b44c018f09 100644
> --- a/Documentation/driver-api/cxl/platform/bios-and-efi.rst
> +++ b/Documentation/driver-api/cxl/platform/bios-and-efi.rst
> @@ -49,6 +49,9 @@ up without requiring CXL driver support. These platform vendors should
> test their configurations with the existing CXL driver and provide driver
> support for their auto-configurations if features like RAS are required.
>
> +Platforms requiring boot-time programming and/or locking of CXL fabric
> +components may prevent features, such as device hot-plug, from working.
> +
> UEFI Settings
> =============
> If your platform supports it, the :code:`uefisettings` command can be used to
> diff --git a/Documentation/driver-api/cxl/platform/device-hotplug.rst b/Documentation/driver-api/cxl/platform/device-hotplug.rst
> new file mode 100644
> index 000000000000..e4a065fdd3ec
> --- /dev/null
> +++ b/Documentation/driver-api/cxl/platform/device-hotplug.rst
> @@ -0,0 +1,130 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==================
> +CXL Device Hotplug
> +==================
> +
> +Device hotplug refers to *physical* hotplug of a device (addition or removal
> +of a physical device from the machine).
> +
> +BIOS/EFI software is expected to configure sufficient resources **at boot
> +time** to allow hotplugged devices to be configured by software (such as
> +proximity domains, HPA regions, and host-bridge configurations).
> +
> +BIOS/EFI is not expected (**nor suggested**) to configure hotplugged
> +devices at hotplug time (i.e. HDM decoders should be left unprogrammed).
> +
> +This document covers some examples of those resources, but should not
> +be considered exhaustive.
Disclaimer noted, but as soon as you start writing about it, questions
arise.
> +
> +Hot-Remove
> +==========
> +Hot removal of a device typically requires careful removal of software
> +constructs (memory regions, associated drivers) which manage these devices.
> +
> +Hard-removing a CXL.mem device without carefully tearing down driver stacks
> +is likely to cause the system to machine-check (or at least SIGBUS if memory
> +access is limited to user space).
> +
> +Memory Device Hot-Add
> +=====================
> +A device present at boot may be associated with a CXL Fixed Memory Window
> +reported in :doc:`CEDT<acpi/cedt>`. That CFMWS may match the size of the
> +device, but the construction of the CEDT CFMWS is platform-defined.
> +
> +Hot-adding a memory device requires this pre-defined, **static** CFMWS to
> +have sufficient HPA space to describe that device.
> +
Isn't it more like the usable capacity of the hot added device will be
limited to the HPA space described in the CFMWS?
a bit similar comment below -
> +There are a few common scenarios to consider.
> +
> +Single-Endpoint Memory Device Present at Boot
> +---------------------------------------------
> +A device present at boot likely had its capacity reported in the
> +:doc:`CEDT<acpi/cedt>`. If a device is removed and a new device hotplugged,
> +the capacity of the new device will be limited to the original CFMWS capacity.
> +
> +Adding capacity larger than the original device will cause memory region
> +creation to fail if the region size is greater than the CFMWS size.
While it’s true that the CFMWS bounds the maximum HPA space available,
hotplugging a device with larger physical capacity doesn’t necessarily
cause region creation to fail outright. It only fails if the requested
region size exceeds the CFMWS size. Users can still create smaller
regions that fit within the existing CFMWS window, even if the device
itself has additional unused capacity
> +
> +The CFMWS is **static** and cannot be adjusted. Platforms which may expect
> +different sized devices to be hotplugged must allocate sufficient CFMWS space
> +**at boot time** to cover all future expected devices.
Yes. Above is crisp. Might be crisper by suggesting that the 'sufficient'
CFMWS space could be achieved with one monster CFMWS or multiple CFMWS of
all their imaginable capacities.
Like my first comment, I'm not clear on why we are dipping our toe in
here, when understanding requires CXL spec, platform provider, and more
guidance. Are we describing anything Linux specific here.
-- Alison
> +
> +Multi-Endpoint Memory Device Present at Boot
> +--------------------------------------------
> +Non-switch-based Multi-Endpoint devices are outside the scope of what the
> +CXL specification describes, but they are technically possible. We describe
> +them here for instructive reasons only - this does not imply Linux support.
> +
> +A hot-plug capable CXL memory device, such as one which presents multiple
> +expanders as a single large-capacity device, should report the **maximum
> +possible capacity** for the device at boot. ::
> +
> + HB0
> + RP0
> + |
> + [Multi-Endpoint Memory Device]
> + _____|_____
> + | |
> + [Endpoint0] [Empty]
> +
> +
> +Limiting the size to the capacity preset at boot will limit hot-add support
> +to replacing capacity that was present at boot.
> +
> +No CXL Device Present at Boot
> +-----------------------------
> +When no CXL memory device is present on boot, some platforms omit the CFMWS
> +in the :doc:`CEDT<acpi/cedt>`. When this occurs, hot-add is not possible.
> +
> +This describes the base case for any given device not being present at boot.
> +If a future possible device is not described in the CEDT at boot, hot-add
> +of that device is either limited or not possible.
> +
> +For a platform to support hot-add of a full memory device, it must allocate
> +a CEDT CFMWS region with sufficient memory capacity to cover all future
> +potentially added capacity (along with any relevant CEDT CHBS entry).
> +
> +To support memory hotplug directly on the host bridge/root port, or on a switch
> +downstream of the host bridge, a platform must construct a CEDT CFMWS at boot
> +with sufficient resources to support the max possible (or expected) hotplug
> +memory capacity. ::
> +
> + HB0 HB1
> + RP0 RP1 RP2
> + | | |
> + Empty Empty USP
> + ________|________
> + | | | |
> + DSP DSP DSP DSP
> + | | | |
> + All Empty
> +
> +For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with
> +a pre-configured amount of memory capacity (per host bridge, or host bridge
> +interleave set), even if no device is attached to Root Ports or Downstream
> +Ports at boot (as depicted in the figure above).
> +
> +
> +Interleave Sets
> +===============
> +
> +Host Bridge Interleave
> +----------------------
> +Host-bridge interleaved memory regions are defined **statically** in the
> +:doc:`CEDT<acpi/cedt>`. To apply cross-host-bridge interleave, a CFMWS entry
> +describing that interleave must have been provided **at boot**. Hotplugged
> +devices cannot add host-bridge interleave capabilities at hotplug time.
> +
> +See the :doc:`Flexible CEDT Configuration<example-configurations/flexible>`
> +example to see how a platform can provide this kind of flexibility regarding
> +hotplugged memory devices. BIOS/EFI software should consider options to
> +present flexible CEDT configurations with hotplug support.
> +
> +HDM Interleave
> +--------------
> +Decoder-applied interleave can flexibly handle hotplugged devices, as decoders
> +can be re-programmed after hotplug.
> +
> +To add or remove a device to/from an existing HDM-applied interleaved region,
> +that region must be torn down an re-created.
> --
> 2.52.0
>
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v3 2/2] Documentation/driver-api/cxl: device hotplug section
2026-01-07 2:19 ` Alison Schofield
@ 2026-01-07 15:54 ` Gregory Price
2026-01-08 5:42 ` Alison Schofield
0 siblings, 1 reply; 13+ messages in thread
From: Gregory Price @ 2026-01-07 15:54 UTC (permalink / raw)
To: Alison Schofield
Cc: linux-cxl, linux-doc, linux-kernel, kernel-team, dave,
jonathan.cameron, dave.jiang, vishal.l.verma, ira.weiny,
dan.j.williams, corbet, rakuram.e96, alucerop
On Tue, Jan 06, 2026 at 06:19:28PM -0800, Alison Schofield wrote:
> On Fri, Dec 19, 2025 at 12:05:38PM -0500, Gregory Price wrote:
> > Describe cxl memory device hotplug implications, in particular how the
> > platform CEDT CFMWS must be described to support successful hot-add of
> > memory devices.
>
> Who is the intended audience for this doc?
>
> Maybe it's already in another section not being edited, but I'd
> expect to see CXL spec references.
>
The audience for this particular doc is platform folks who want to know
what limitations there are on cxl-device hotplug in linux.
This originated from a discussion at LPC where some device folks
understood the hotplug process, but did not understand the linux
limitations (in particular around the CFMWS and node creation).
Tl;dr: Linux consumes ACPI tables to generate NUMA nodes and constructs
memory regions. This is linux "having an opinion" on "how ACPI tables
should reasonably be constructed" - and this tries to document that.
This is all obviously mutable, but it at least helps drive understanding
between device, bios, platform, and linux if we at least have this
document that says "this is what Linux wants/expects".
I'm not sure what if anything I would reference in the spec, since this
largely describes what linux wants, not what the spec says is possible.
Happy to improve.
> > +Memory Device Hot-Add
> > +=====================
> > +A device present at boot may be associated with a CXL Fixed Memory Window
> > +reported in :doc:`CEDT<acpi/cedt>`. That CFMWS may match the size of the
> > +device, but the construction of the CEDT CFMWS is platform-defined.
> > +
> > +Hot-adding a memory device requires this pre-defined, **static** CFMWS to
> > +have sufficient HPA space to describe that device.
> > +
>
> Isn't it more like the usable capacity of the hot added device will be
> limited to the HPA space described in the CFMWS?
>
> a bit similar comment below -
>
>
This is more precise and concise. I can change that.
> > +Single-Endpoint Memory Device Present at Boot
> > +---------------------------------------------
> > +A device present at boot likely had its capacity reported in the
> > +:doc:`CEDT<acpi/cedt>`. If a device is removed and a new device hotplugged,
> > +the capacity of the new device will be limited to the original CFMWS capacity.
> > +
> > +Adding capacity larger than the original device will cause memory region
> > +creation to fail if the region size is greater than the CFMWS size.
>
> While it’s true that the CFMWS bounds the maximum HPA space available,
> hotplugging a device with larger physical capacity doesn’t necessarily
> cause region creation to fail outright. It only fails if the requested
> region size exceeds the CFMWS size. Users can still create smaller
> regions that fit within the existing CFMWS window, even if the device
> itself has additional unused capacity
This does say: "if the region size is greater than the CFMWS size"
But maybe it's more accurate to say "if the unused CFMWS HPA space is
smaller than a user-defined memory region capacity".
This is really all saying the same thing though - if your device
capacity is smaller than your CFMWS, you can't map the entire device.
>
> > +
> > +The CFMWS is **static** and cannot be adjusted. Platforms which may expect
> > +different sized devices to be hotplugged must allocate sufficient CFMWS space
> > +**at boot time** to cover all future expected devices.
>
> Yes. Above is crisp. Might be crisper by suggesting that the 'sufficient'
> CFMWS space could be achieved with one monster CFMWS or multiple CFMWS of
> all their imaginable capacities.
>
> Like my first comment, I'm not clear on why we are dipping our toe in
> here, when understanding requires CXL spec, platform provider, and more
> guidance. Are we describing anything Linux specific here.
>
Linux specific piece here is the fact that once Linux proceeds past
__init, you cannot achieve hotplug for CXL memory not described (or
insufficiently described) in the ACPI tables.
In this case - memory is special. We're (linux) basically telling
platform vendors "You can't add memory you haven't described in some way
since the beginning of time".
It's unclear that the same would be true on all operating systems, but
it is true on linux.
---
general note - i'm happy to drop this if folks think it's not the right
place to put it, but based on LPC discussions and feedback from Jonathan
and Dan, it seemed useful.
Maybe I should just be writing a book instead, lol. (Please no)
~Gregory
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v3 2/2] Documentation/driver-api/cxl: device hotplug section
2026-01-07 15:54 ` Gregory Price
@ 2026-01-08 5:42 ` Alison Schofield
0 siblings, 0 replies; 13+ messages in thread
From: Alison Schofield @ 2026-01-08 5:42 UTC (permalink / raw)
To: Gregory Price
Cc: linux-cxl, linux-doc, linux-kernel, kernel-team, dave,
jonathan.cameron, dave.jiang, vishal.l.verma, ira.weiny,
dan.j.williams, corbet, rakuram.e96, alucerop
On Wed, Jan 07, 2026 at 10:54:16AM -0500, Gregory Price wrote:
> On Tue, Jan 06, 2026 at 06:19:28PM -0800, Alison Schofield wrote:
> > On Fri, Dec 19, 2025 at 12:05:38PM -0500, Gregory Price wrote:
> > > Describe cxl memory device hotplug implications, in particular how the
> > > platform CEDT CFMWS must be described to support successful hot-add of
> > > memory devices.
> >
> > Who is the intended audience for this doc?
> >
> > Maybe it's already in another section not being edited, but I'd
> > expect to see CXL spec references.
> >
I'm fine with this merged. I starting to read it as more prescriptive
than it is wanting to be. It drives home the point that "You can't add
memory you haven't described in some way since the beginning of time".
Reviewed-by: Alison Schofield <alison.schofield@intel.com>
snip
^ permalink raw reply [flat|nested] 13+ messages in thread