[PATCH v2] Documentation/driver-api/cxl: device hotplug section

[PATCH v2] Documentation/driver-api/cxl: device hotplug section

[Gregory Price]

Describe cxl memory device hotplug implications, in particular how the
platform CEDT CFMWS must be described to support successful hot-add of
memory devices.

Signed-off-by: Gregory Price <gourry@gourry.net>
---
v2: Jonathan's clarifications and diagrams.

 Documentation/driver-api/cxl/index.rst        |   1 +
 .../cxl/platform/device-hotplug.rst           | 112 ++++++++++++++++++
 2 files changed, 113 insertions(+)
 create mode 100644 Documentation/driver-api/cxl/platform/device-hotplug.rst

diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst
index c1106a68b67c..5a734988a5af 100644
--- a/Documentation/driver-api/cxl/index.rst
+++ b/Documentation/driver-api/cxl/index.rst
@@ -30,6 +30,7 @@ that have impacts on each other.  The docs here break up configurations steps.
    platform/acpi
    platform/cdat
    platform/example-configs
+   platform/device-hotplug
 
 .. toctree::
    :maxdepth: 2
diff --git a/Documentation/driver-api/cxl/platform/device-hotplug.rst b/Documentation/driver-api/cxl/platform/device-hotplug.rst
new file mode 100644
index 000000000000..617e340bd556
--- /dev/null
+++ b/Documentation/driver-api/cxl/platform/device-hotplug.rst
@@ -0,0 +1,112 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==================
+CXL Device Hotplug
+==================
+
+Device hotplug refers to *physical* hotplug of a device (addition or removal
+of a physical device from the machine).
+
+Hot-Remove
+==========
+Hot removal of a device typically requires careful removal of software
+constructs (memory regions, associated drivers) which manage these devices.
+
+Hard-removing a CXL.mem device without carefully tearing down driver stacks
+is likely to cause the system to machine-check (or at least SIGBUS if memory
+access is limited to user space).
+
+Memory Device Hot-Add
+=====================
+A device present at boot will be associated with a CXL Fixed Memory Window
+reported in :doc:`CEDT<acpi/cedt>`.  That CFMWS may match the size of the
+device, but the construction of the CEDT CFMWS is platform-defined.
+
+Hot-adding a memory device requires this pre-defined (*static*) CFMWS has
+sufficient space to describe that device.
+
+There are a few common scenarios to consider.
+
+Single-Endpoint Memory Device Present at Boot
+---------------------------------------------
+A device present at boot likely had its capacity reported in the
+:doc:`CEDT<acpi/cedt>`.  If a device is removed and a new device hotplugged,
+the capacity of the new device will be limited to the original CFMWS capacity.
+
+Adding capacity larger than the original device will cause memory region
+creation to fail if the region size is greater than the CFMWS size.
+
+The CFMWS is *static* and cannot be adjusted.  Platforms which may expect
+different sized devices to be hotplugged must allocate sufficient CFMWS space
+*at boot time* to cover all future expected devices.
+
+Multi-Endpoint Memory Device Present at Boot
+--------------------------------------------
+A hot-plug capable CXL memory device, such as one which presents multiple
+expanders as a single large-capacity device, should report the maximum
+*possible* capacity for the device at boot. ::
+
+                  HB0
+                  RP0
+                   |
+     [Multi-Endpoint Memory Device]
+              _____|_____
+             |          |
+        [Endpoint0]   [Empty]
+
+
+Limiting the size to the capacity preset at boot will limit hot-add support
+to replacing capacity that was present at boot.
+
+No CXL Device Present at Boot
+-----------------------------
+When no CXL memory device is present on boot, some platforms omit the CFMWS
+in the :doc:`CEDT<acpi/cedt>`.  When this occurs, hot-add is not possible.
+
+For a platform to support hot-add of a full memory device, it must allocate
+a CEDT CFMWS region with sufficient memory capacity to cover all future
+potentially added capacity.
+
+To support memory hotplug directly on the host bridge, or on a switch
+downstream of the host bridge (but not contained within a CXL memory device),
+a platform must construct a CEDT CFMWS at boot with sufficient resources to
+support the max possible (or expected) hotplug memory capacity. ::
+
+         HB0                 HB1
+      RP0    RP1             RP2
+       |      |               |
+     Empty  Empty            USP
+                      ________|________
+                      |    |    |     |
+                     DSP  DSP  DSP   DSP
+                      |    |    |    |
+                         All  Empty
+
+For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with
+a pre-configured amount of memory capacity (per host bridge, or host bridge
+interleave set), even if no device is attached to Root Ports or Downstream
+Ports at boot (as depicted in the figure above).
+
+
+Interleave Sets
+===============
+
+Host Bridge Interleave
+----------------------
+Host-bridge interleaved memory regions are defined *statically* in the
+:doc:`CEDT<acpi/cedt>`.  To apply cross-host-bridge interleave, a CFMWS entry
+describing that interleave must have been provided *at boot*.  Hotplugged
+devices cannot add host-bridge interleave capabilities at hotplug time.
+
+See the :doc:`Flexible CEDT Configuration<example-configurations/flexible>`
+example to see how a platform can provide this kind of flexibility regarding
+hotplugged memory devices.  BIOS/EFI software should consider options to
+present flexible CEDT configurations with hotplug support.
+
+HDM Interleave
+--------------
+Decoder-applied interleave can flexibly handle hotplugged devices, as decoders
+can be re-programmed after hotplug.
+
+To add or remove a device to/from an existing HDM-applied interleaved region,
+that region must be torn down an re-created.
-- 
2.52.0

Re: [PATCH v2] Documentation/driver-api/cxl: device hotplug section

[Randy Dunlap]

Hi,

On 12/18/25 9:07 AM, Gregory Price wrote:
> Describe cxl memory device hotplug implications, in particular how the
> platform CEDT CFMWS must be described to support successful hot-add of
> memory devices.
> 
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
> v2: Jonathan's clarifications and diagrams.
> 
>  Documentation/driver-api/cxl/index.rst        |   1 +
>  .../cxl/platform/device-hotplug.rst           | 112 ++++++++++++++++++
>  2 files changed, 113 insertions(+)
>  create mode 100644 Documentation/driver-api/cxl/platform/device-hotplug.rst
> 

> diff --git a/Documentation/driver-api/cxl/platform/device-hotplug.rst b/Documentation/driver-api/cxl/platform/device-hotplug.rst
> new file mode 100644
> index 000000000000..617e340bd556
> --- /dev/null
> +++ b/Documentation/driver-api/cxl/platform/device-hotplug.rst
> @@ -0,0 +1,112 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==================
> +CXL Device Hotplug
> +==================
> +
> +Device hotplug refers to *physical* hotplug of a device (addition or removal
> +of a physical device from the machine).
> +
> +Hot-Remove
> +==========
> +Hot removal of a device typically requires careful removal of software
> +constructs (memory regions, associated drivers) which manage these devices.
> +
> +Hard-removing a CXL.mem device without carefully tearing down driver stacks
> +is likely to cause the system to machine-check (or at least SIGBUS if memory
> +access is limited to user space).
> +
> +Memory Device Hot-Add
> +=====================
> +A device present at boot will be associated with a CXL Fixed Memory Window
> +reported in :doc:`CEDT<acpi/cedt>`.  That CFMWS may match the size of the
> +device, but the construction of the CEDT CFMWS is platform-defined.
> +
> +Hot-adding a memory device requires this pre-defined (*static*) CFMWS has
> +sufficient space to describe that device.
> +
> +There are a few common scenarios to consider.
> +
> +Single-Endpoint Memory Device Present at Boot
> +---------------------------------------------
> +A device present at boot likely had its capacity reported in the
> +:doc:`CEDT<acpi/cedt>`.  If a device is removed and a new device hotplugged,
> +the capacity of the new device will be limited to the original CFMWS capacity.
> +
> +Adding capacity larger than the original device will cause memory region
> +creation to fail if the region size is greater than the CFMWS size.
> +
> +The CFMWS is *static* and cannot be adjusted.  Platforms which may expect
> +different sized devices to be hotplugged must allocate sufficient CFMWS space
> +*at boot time* to cover all future expected devices.
> +
> +Multi-Endpoint Memory Device Present at Boot
> +--------------------------------------------
> +A hot-plug capable CXL memory device, such as one which presents multiple
> +expanders as a single large-capacity device, should report the maximum
> +*possible* capacity for the device at boot. ::
> +
> +                  HB0
> +                  RP0
> +                   |
> +     [Multi-Endpoint Memory Device]
> +              _____|_____
> +             |          |
> +        [Endpoint0]   [Empty]
> +
> +
> +Limiting the size to the capacity preset at boot will limit hot-add support
> +to replacing capacity that was present at boot.
> +
> +No CXL Device Present at Boot
> +-----------------------------
> +When no CXL memory device is present on boot, some platforms omit the CFMWS
> +in the :doc:`CEDT<acpi/cedt>`.  When this occurs, hot-add is not possible.
> +
> +For a platform to support hot-add of a full memory device, it must allocate
> +a CEDT CFMWS region with sufficient memory capacity to cover all future
> +potentially added capacity.
> +
> +To support memory hotplug directly on the host bridge, or on a switch
> +downstream of the host bridge (but not contained within a CXL memory device),
> +a platform must construct a CEDT CFMWS at boot with sufficient resources to
> +support the max possible (or expected) hotplug memory capacity. ::
> +
> +         HB0                 HB1
> +      RP0    RP1             RP2
> +       |      |               |
> +     Empty  Empty            USP
> +                      ________|________
> +                      |    |    |     |
> +                     DSP  DSP  DSP   DSP
> +                      |    |    |    |
> +                         All  Empty
> +
> +For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with
> +a pre-configured amount of memory capacity (per host bridge, or host bridge
> +interleave set), even if no device is attached to Root Ports or Downstream
> +Ports at boot (as depicted in the figure above).
> +
> +
> +Interleave Sets
> +===============
> +
> +Host Bridge Interleave
> +----------------------
> +Host-bridge interleaved memory regions are defined *statically* in the
> +:doc:`CEDT<acpi/cedt>`.  To apply cross-host-bridge interleave, a CFMWS entry
> +describing that interleave must have been provided *at boot*.  Hotplugged
> +devices cannot add host-bridge interleave capabilities at hotplug time.
> +
> +See the :doc:`Flexible CEDT Configuration<example-configurations/flexible>`
> +example to see how a platform can provide this kind of flexibility regarding
> +hotplugged memory devices.  BIOS/EFI software should consider options to
> +present flexible CEDT configurations with hotplug support.
> +
> +HDM Interleave
> +--------------
> +Decoder-applied interleave can flexibly handle hotplugged devices, as decoders
> +can be re-programmed after hotplug.
> +
> +To add or remove a device to/from an existing HDM-applied interleaved region,
> +that region must be torn down an re-created.

                                 ^s/an/and/

and one note: There are several places where you use one asterisk for emphasis,
e.g., *statically*. This means <italic> in Sphinx/ReST. If you happen
to mean/want bold, then use **bold text** (double asterisks).

Otherwise it LGTM. Thanks.

Reviewed-by: Randy Dunlap <rdunlap@infradead.org>
Tested-by: Randy Dunlap <rdunlap@infradead.org>

-- 
~Randy

Re: [PATCH v2] Documentation/driver-api/cxl: device hotplug section

[Gregory Price]

On Thu, Dec 18, 2025 at 11:36:41AM -0800, Randy Dunlap wrote:
> > +To add or remove a device to/from an existing HDM-applied interleaved region,
> > +that region must be torn down an re-created.
> 
>                                  ^s/an/and/
                                     doh.
> 
> and one note: There are several places where you use one asterisk for emphasis,
> e.g., *statically*. This means <italic> in Sphinx/ReST. If you happen
> to mean/want bold, then use **bold text** (double asterisks).
> 

I did intend italics, but if you think it should be bold instead given
the contents, happy to change it.

> Otherwise it LGTM. Thanks.
> 
> Reviewed-by: Randy Dunlap <rdunlap@infradead.org>
> Tested-by: Randy Dunlap <rdunlap@infradead.org>
> 
> -- 
> ~Randy
> 

Re: [PATCH v2] Documentation/driver-api/cxl: device hotplug section

[Randy Dunlap]

On 12/18/25 12:17 PM, Gregory Price wrote:
> On Thu, Dec 18, 2025 at 11:36:41AM -0800, Randy Dunlap wrote:
>>> +To add or remove a device to/from an existing HDM-applied interleaved region,
>>> +that region must be torn down an re-created.
>>
>>                                  ^s/an/and/
>                                      doh.
>>
>> and one note: There are several places where you use one asterisk for emphasis,
>> e.g., *statically*. This means <italic> in Sphinx/ReST. If you happen
>> to mean/want bold, then use **bold text** (double asterisks).
>>
> 
> I did intend italics, but if you think it should be bold instead given
> the contents, happy to change it.

No, it's fine as is. Just checking.

-- 
~Randy

Re: [PATCH v2] Documentation/driver-api/cxl: device hotplug section

[Jonathan Cameron]

On Thu, 18 Dec 2025 12:07:47 -0500
Gregory Price <gourry@gourry.net> wrote:

> Describe cxl memory device hotplug implications, in particular how the
> platform CEDT CFMWS must be described to support successful hot-add of
> memory devices.
> 
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
> v2: Jonathan's clarifications and diagrams.

One request for a reference given I'm not sure the whole multi-endpoint memory
device thing is going to hang around so that might confuse people reading
this in future - particularly as it isn't something the spec mentions.

With something added there (or a statement that there isn't anything public that
will do the job)

Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>

> diff --git a/Documentation/driver-api/cxl/platform/device-hotplug.rst b/Documentation/driver-api/cxl/platform/device-hotplug.rst
> new file mode 100644
> index 000000000000..617e340bd556
> --- /dev/null
> +++ b/Documentation/driver-api/cxl/platform/device-hotplug.rst

> +
> +Multi-Endpoint Memory Device Present at Boot
> +--------------------------------------------
> +A hot-plug capable CXL memory device, such as one which presents multiple
> +expanders as a single large-capacity device, should report the maximum
> +*possible* capacity for the device at boot. ::
> +
> +                  HB0
> +                  RP0
> +                   |
> +     [Multi-Endpoint Memory Device]

So this is the weird switch as end point thing?   Maybe a reference.
My guess is these will go away as switch and memory device vendors catch
up with the spec, but maybe I'm wrong.

> +              _____|_____
> +             |          |
> +        [Endpoint0]   [Empty]
> +
> +
> +Limiting the size to the capacity preset at boot will limit hot-add support
> +to replacing capacity that was present at boot.

Re: [PATCH v2] Documentation/driver-api/cxl: device hotplug section

[Gregory Price]

On Fri, Dec 19, 2025 at 10:55:18AM +0000, Jonathan Cameron wrote:
> On Thu, 18 Dec 2025 12:07:47 -0500
> Gregory Price <gourry@gourry.net> wrote:
> 
> > +
> > +Multi-Endpoint Memory Device Present at Boot
> > +--------------------------------------------
> > +A hot-plug capable CXL memory device, such as one which presents multiple
> > +expanders as a single large-capacity device, should report the maximum
> > +*possible* capacity for the device at boot. ::
> > +
> > +                  HB0
> > +                  RP0
> > +                   |
> > +     [Multi-Endpoint Memory Device]
> 
> So this is the weird switch as end point thing?   Maybe a reference.
> My guess is these will go away as switch and memory device vendors catch
> up with the spec, but maybe I'm wrong.
> 

I guess I just don't want to dictate the innards of a multi-endpoint
memory device.  It *really really* implies there must be some kind of
switch inside - but that switch might not even be runtime programmable
or discoverable (basically all the settings get locked on boot and it
becomes passthrough).

If you'd rather just not have this section at all, I'm ok with that.
The switch case below this covers the base case for a switch-based
device where everything is programmable.

> > +              _____|_____
> > +             |          |
> > +        [Endpoint0]   [Empty]
> > +
> > +
> > +Limiting the size to the capacity preset at boot will limit hot-add support
> > +to replacing capacity that was present at boot.
> 
> 

Re: [PATCH v2] Documentation/driver-api/cxl: device hotplug section

[Jonathan Cameron]

On Fri, 19 Dec 2025 09:31:32 -0500
Gregory Price <gourry@gourry.net> wrote:

> On Fri, Dec 19, 2025 at 10:55:18AM +0000, Jonathan Cameron wrote:
> > On Thu, 18 Dec 2025 12:07:47 -0500
> > Gregory Price <gourry@gourry.net> wrote:
> >   
> > > +
> > > +Multi-Endpoint Memory Device Present at Boot
> > > +--------------------------------------------
> > > +A hot-plug capable CXL memory device, such as one which presents multiple
> > > +expanders as a single large-capacity device, should report the maximum
> > > +*possible* capacity for the device at boot. ::
> > > +
> > > +                  HB0
> > > +                  RP0
> > > +                   |
> > > +     [Multi-Endpoint Memory Device]  
> > 
> > So this is the weird switch as end point thing?   Maybe a reference.
> > My guess is these will go away as switch and memory device vendors catch
> > up with the spec, but maybe I'm wrong.
> >   
> 
> I guess I just don't want to dictate the innards of a multi-endpoint
> memory device.  It *really really* implies there must be some kind of
> switch inside - but that switch might not even be runtime programmable
> or discoverable (basically all the settings get locked on boot and it
> becomes passthrough).
> 
> If you'd rather just not have this section at all, I'm ok with that.
> The switch case below this covers the base case for a switch-based
> device where everything is programmable.

Maybe just state at the top of this section that these things are
outside of the types of device that the CXL specification explicitly
talks about. That should be enough of a future breadcrumb for people
to decide if they can ignore these.


> 
> > > +              _____|_____
> > > +             |          |
> > > +        [Endpoint0]   [Empty]
> > > +
> > > +
> > > +Limiting the size to the capacity preset at boot will limit hot-add support
> > > +to replacing capacity that was present at boot.  
> > 
> >   

Re: [PATCH v2] Documentation/driver-api/cxl: device hotplug section

[Alejandro Lucero Palau]

Hi Gregory,


Thank you for this addition to documentation.

Just a comment below about something we talked about at LPC.


On 12/18/25 17:07, Gregory Price wrote:
> Describe cxl memory device hotplug implications, in particular how the
> platform CEDT CFMWS must be described to support successful hot-add of
> memory devices.
>
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
> v2: Jonathan's clarifications and diagrams.
>
>   Documentation/driver-api/cxl/index.rst        |   1 +
>   .../cxl/platform/device-hotplug.rst           | 112 ++++++++++++++++++
>   2 files changed, 113 insertions(+)
>   create mode 100644 Documentation/driver-api/cxl/platform/device-hotplug.rst
>
> diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst
> index c1106a68b67c..5a734988a5af 100644
> --- a/Documentation/driver-api/cxl/index.rst
> +++ b/Documentation/driver-api/cxl/index.rst
> @@ -30,6 +30,7 @@ that have impacts on each other.  The docs here break up configurations steps.
>      platform/acpi
>      platform/cdat
>      platform/example-configs
> +   platform/device-hotplug
>   
>   .. toctree::
>      :maxdepth: 2
> diff --git a/Documentation/driver-api/cxl/platform/device-hotplug.rst b/Documentation/driver-api/cxl/platform/device-hotplug.rst
> new file mode 100644
> index 000000000000..617e340bd556
> --- /dev/null
> +++ b/Documentation/driver-api/cxl/platform/device-hotplug.rst
> @@ -0,0 +1,112 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==================
> +CXL Device Hotplug
> +==================
> +
> +Device hotplug refers to *physical* hotplug of a device (addition or removal
> +of a physical device from the machine).
> +
> +Hot-Remove
> +==========
> +Hot removal of a device typically requires careful removal of software
> +constructs (memory regions, associated drivers) which manage these devices.
> +
> +Hard-removing a CXL.mem device without carefully tearing down driver stacks
> +is likely to cause the system to machine-check (or at least SIGBUS if memory
> +access is limited to user space).
> +
> +Memory Device Hot-Add
> +=====================
> +A device present at boot will be associated with a CXL Fixed Memory Window
> +reported in :doc:`CEDT<acpi/cedt>`.  That CFMWS may match the size of the
> +device, but the construction of the CEDT CFMWS is platform-defined.
> +
> +Hot-adding a memory device requires this pre-defined (*static*) CFMWS has
> +sufficient space to describe that device.
> +
> +There are a few common scenarios to consider.
> +
> +Single-Endpoint Memory Device Present at Boot
> +---------------------------------------------
> +A device present at boot likely had its capacity reported in the
> +:doc:`CEDT<acpi/cedt>`.  If a device is removed and a new device hotplugged,
> +the capacity of the new device will be limited to the original CFMWS capacity.
> +
> +Adding capacity larger than the original device will cause memory region
> +creation to fail if the region size is greater than the CFMWS size.
> +
> +The CFMWS is *static* and cannot be adjusted.  Platforms which may expect
> +different sized devices to be hotplugged must allocate sufficient CFMWS space
> +*at boot time* to cover all future expected devices.
> +
> +Multi-Endpoint Memory Device Present at Boot
> +--------------------------------------------
> +A hot-plug capable CXL memory device, such as one which presents multiple
> +expanders as a single large-capacity device, should report the maximum
> +*possible* capacity for the device at boot. ::
> +
> +                  HB0
> +                  RP0
> +                   |
> +     [Multi-Endpoint Memory Device]
> +              _____|_____
> +             |          |
> +        [Endpoint0]   [Empty]
> +
> +
> +Limiting the size to the capacity preset at boot will limit hot-add support
> +to replacing capacity that was present at boot.
> +
> +No CXL Device Present at Boot
> +-----------------------------
> +When no CXL memory device is present on boot, some platforms omit the CFMWS
> +in the :doc:`CEDT<acpi/cedt>`.  When this occurs, hot-add is not possible.
> +
> +For a platform to support hot-add of a full memory device, it must allocate
> +a CEDT CFMWS region with sufficient memory capacity to cover all future
> +potentially added capacity.
> +
> +To support memory hotplug directly on the host bridge, or on a switch
> +downstream of the host bridge (but not contained within a CXL memory device),
> +a platform must construct a CEDT CFMWS at boot with sufficient resources to
> +support the max possible (or expected) hotplug memory capacity. ::
> +
> +         HB0                 HB1
> +      RP0    RP1             RP2
> +       |      |               |
> +     Empty  Empty            USP
> +                      ________|________
> +                      |    |    |     |
> +                     DSP  DSP  DSP   DSP
> +                      |    |    |    |
> +                         All  Empty
> +
> +For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with
> +a pre-configured amount of memory capacity (per host bridge, or host bridge
> +interleave set), even if no device is attached to Root Ports or Downstream
> +Ports at boot (as depicted in the figure above).


All this is fine, but my concern is what the BIOS will do when a device 
ends up being hotplugged. Assuming that programmability from the 
Host/user space for creating regions on demand based on requirements 
like bandwidth, and therefore relying on interleaving and granularity (I 
know this is not trivial but I think this is possible in the near 
future, or at least possible to be demanded) then the BIOS should not do 
any HDM programming at all ...


I think it is worth to document this somehow and maybe to discuss this 
with BIOSes vendors if we consider this convenient.


Thank you,

Alejandro


> +
> +
> +Interleave Sets
> +===============
> +
> +Host Bridge Interleave
> +----------------------
> +Host-bridge interleaved memory regions are defined *statically* in the
> +:doc:`CEDT<acpi/cedt>`.  To apply cross-host-bridge interleave, a CFMWS entry
> +describing that interleave must have been provided *at boot*.  Hotplugged
> +devices cannot add host-bridge interleave capabilities at hotplug time.
> +
> +See the :doc:`Flexible CEDT Configuration<example-configurations/flexible>`
> +example to see how a platform can provide this kind of flexibility regarding
> +hotplugged memory devices.  BIOS/EFI software should consider options to
> +present flexible CEDT configurations with hotplug support.
> +
> +HDM Interleave
> +--------------
> +Decoder-applied interleave can flexibly handle hotplugged devices, as decoders
> +can be re-programmed after hotplug.
> +
> +To add or remove a device to/from an existing HDM-applied interleaved region,
> +that region must be torn down an re-created.

Re: [PATCH v2] Documentation/driver-api/cxl: device hotplug section

[Gregory Price]

On Fri, Dec 19, 2025 at 03:06:21PM +0000, Alejandro Lucero Palau wrote:
> > +To support memory hotplug directly on the host bridge, or on a switch
> > +downstream of the host bridge (but not contained within a CXL memory device),
> > +a platform must construct a CEDT CFMWS at boot with sufficient resources to
> > +support the max possible (or expected) hotplug memory capacity. ::
> > +
> > +         HB0                 HB1
> > +      RP0    RP1             RP2
> > +       |      |               |
> > +     Empty  Empty            USP
> > +                      ________|________
> > +                      |    |    |     |
> > +                     DSP  DSP  DSP   DSP
> > +                      |    |    |    |
> > +                         All  Empty
> > +
> > +For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with
> > +a pre-configured amount of memory capacity (per host bridge, or host bridge
> > +interleave set), even if no device is attached to Root Ports or Downstream
> > +Ports at boot (as depicted in the figure above).
> 
> All this is fine, but my concern is what the BIOS will do when a device ends
> up being hotplugged. 

It should do nothing more than what is required to bring the device up
and make it visible on the PCIe/CXL link.  BIOS should not be
programmign the device at runtime.

I realize this is likely infeasible for certain setups, like the current
setup of Zen5 and it's address translation setup, but then I think most
platforms right now don't really handle any of this.

> Assuming that programmability from the Host/user space
> for creating regions on demand based on requirements like bandwidth, and
> therefore relying on interleaving and granularity (I know this is not
> trivial but I think this is possible in the near future, or at least
> possible to be demanded) then the BIOS should not do any HDM programming at
> all ...
> 

Correct.

That said, I can only speak from a BIOS -> Linux transition perspective.
I know enough about the ACPI spec, CXL spec, Linux MM, and the Linux
driver to say "what linux expects" - but I have absolutely no idea what
BIOS *actually does* at boot or hotplug time.

If you have some of that perspective, please propose changes to the
documentation based on what is feasible/possible.  All I can do is write
from the base of what I know.

> I think it is worth to document this somehow and maybe to discuss this with
> BIOSes vendors if we consider this convenient.
>

The entire point of this document is really to spell out that the BIOS
is only responsible for setting up the environment *at boot*, and then
at hotplug time the BIOS should do nothing wrt to programming stuff :]

I can add this explicitly at the top of this document in v3 (and
probably should be spelled out in the BIOS/EFI config doc)
https://docs.kernel.org/driver-api/cxl/platform/bios-and-efi.html


Would something like this suffice?

"""
Linux expects BIOS/EFI software to construct sufficient ACPI tables
(such as CEDT, SRAT, HMAT, etc) and platform-specific configurations
(such as HPA spaces and host-bridge interleave configurations) to allow
the Linux driver to subsequently configure the devices in the CXL fabric
at runtime.

Programming of HDM decoders and switch ports is not required, and may be
deferred to the CXL driver based on admin policy (e.g. udev rules).

Some platforms may require pre-programming HDM decoders and locking them
due to quirks (see: Zen5 address translation), but this is not the normal,
"expected" configuration path.  This should be avoided if possible.

For hotplugged devices, BIOS/EFI software is expected to configure
sufficient resources **at boot time** to allow hotplugged devices
to be configured by software (such as proximity domains, HPA regions,
and host-bridge configurations).

BIOS/EFI is not expected (**nor suggested**) to configure hotplugged
devices at hotplug time (i.e. HDM decoders should be leftunprogrammed).
"""

~Gregory