[PATCH v4] cxl: docs/driver-api/conventions resolve conflicts between CFMWS, LMH, Decoders

[PATCH v4] cxl: docs/driver-api/conventions resolve conflicts between CFMWS, LMH, Decoders

[Fabio M. De Francesco]

Add documentation on how to resolve conflicts between CXL Fixed Memory
Windows, Platform Low Memory Holes, intermediate Switch and Endpoint
Decoders.

Cc: Ira Weiny <ira.weiny@intel.com>
Signed-off-by: Fabio M. De Francesco <fabio.m.de.francesco@linux.intel.com>
---

v3 -> v4: Show and explain how CFMWS, Root Decoders, Intermediate
	  Switch and Endpoint Decoders match and attach Regions in
	  x86 platforms with Low Memory Holes (Dave, Gregory, Ira)
	  Remove a wrong argument about large interleaves (Jonathan)

v2 -> v3: Rework a few phrases for better clarity.
	  Fix grammar and syntactic errors (Randy, Alok).
	  Fix semantic errors ("size does not comply", Alok).
	  Fix technical errors ("decoder's total memory?", Alok).
	  
v1 -> v2: Rewrite "Summary of the Change" section, 3r paragraph.

 Documentation/driver-api/cxl/conventions.rst | 111 +++++++++++++++++++
 1 file changed, 111 insertions(+)

diff --git a/Documentation/driver-api/cxl/conventions.rst b/Documentation/driver-api/cxl/conventions.rst
index da347a81a237..714240ed2e04 100644
--- a/Documentation/driver-api/cxl/conventions.rst
+++ b/Documentation/driver-api/cxl/conventions.rst
@@ -45,3 +45,114 @@ Detailed Description of the Change
 ----------------------------------
 
 <Propose spec language that corrects the conflict.>
+
+
+Resolve conflict between CFMWS, Platform Memory Holes, and Endpoint Decoders
+============================================================================
+
+Document
+--------
+
+CXL Revision 3.2, Version 1.0
+
+License
+-------
+
+SPDX-License Identifier: CC-BY-4.0
+
+Creator/Contributors
+--------------------
+
+Fabio M. De Francesco, Intel
+Dan J. Williams, Intel
+Mahesh Natu, Intel
+
+Summary of the Change
+---------------------
+
+According to the current CXL Specifications (Revision 3.2, Version 1.0),
+the CXL Fixed Memory Window Structure (CFMWS) describes zero or more Host
+Physical Address (HPA) windows associated with each CXL Host Bridge. Each
+window represents a contiguous HPA range that may be interleaved across
+one or more targets, including CXL Host Bridges. Each window has a set of
+restrictions that govern its usage. It is the OSPM’s responsibility to
+utilize each window for the specified use.
+
+Table 9-22 states the Window Size field contains the total number of
+consecutive bytes of HPA this window represents. This value must be a
+multiple of the Number of Interleave Ways * 256 MB.
+
+Platform Firmware (BIOS) might reserve physical addresses below 4 GB,
+such as the Low Memory Hole for PCIe MMIO. In such cases, the CFMWS Range
+Size may not adhere to the NIW * 256 MB rule.
+
+On these systems, BIOS publishes CFMWS to communicate the active System
+Physical Address (SPA) ranges that map to a subset of the Host Physical
+Address (HPA) ranges. The SPA range trims out the hole, resulting in lost
+capacity in the endpoint with no SPA to map to the CXL HPA range that
+exceeds the matching CFMWS range.
+
+E.g, a real x86 platform with two CFMWS, 384 GB total memory, and LMH
+starting at 2 GB:
+
+Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways | Granularity
+  0    |   0 GB     |     2 GB   |      0 GB        |       3 GB       |  12  |    256
+  1    |   4 GB     |   380 GB   |      0 GB        |     380 GB       |  12  |    256
+
+HDM decoder base and HDM decoder size represent all the 12 Endpoint
+Decoders of a 12 way region and all the intermediate Switch Decoders.
+They are configured by the BIOS according to the NIW * 256MB rule,
+resulting in a HPA range size of 3GB.
+
+The CFMWS Base and CFMWS Size are used to configure the Root Decoder HPA
+range base and size. CFMWS cannot intersect Memory Holes, then the CFMWS[0]
+size is smaller (2GB) than that of the Switch and Endpoint Decoders that
+make the hierarchy (3GB).
+
+On that platform, only the first 2GB will be potentially usable but,
+because of the current specs, Linux fails to make them available to the
+users. The driver expects that Root Decoder HPA size, which is equal to
+the CFMWS from which it is configured, to be greater or equal to the
+matching Switch and Endpoint HDM Decoders.
+
+The CXL driver fails to construct Regions and to attach Endpoint and
+intermediate Switch Decoders to those Regions after their construction.
+
+In order to succeed with Region construction and Decoders attachment,
+Linux must construct Regions with Root Decoders size, and then attach to
+them all the intermediate Switch and Endpoint Decoders that are part of the
+hierarchy, even though the Decoders HPA range sizes may be larger than
+those Regions whose sizes are trimmed by Low Memory Holes.
+
+Benefits of the Change
+----------------------
+
+Without this change, the OSPM wouldn't match Intermediate and Endpoint
+Decoders with Root Decoders configured with CFMWS HPA sizes that don't
+align with the NIW * 256MB constraint, leading to lost memdev capacity.
+This change allows the OSPM to construct Regions and attach Intermediate
+Switch and Endpoint Decoders to them, so that the addressable part of the
+memory devices total capacity is not lost.
+
+References
+----------
+
+Compute Express Link Specification Revision 3.2, Version 1.0
+<https://www.computeexpresslink.org/>
+
+Detailed Description of the Change
+----------------------------------
+
+The description of the Window Size field in table 9-22 needs to account
+for platforms with Low Memory Holes, where SPA ranges might be subsets of
+the endpoints' HPA. Therefore, it has to be changed to the following:
+
+"The total number of consecutive bytes of HPA this window represents.
+This value shall be a multiple of NIW * 256 MB. On platforms that reserve
+physical addresses below 4 GB, such as the Low Memory Hole for PCIe MMIO
+on x86 or a requirement for greater than 8-way interleave CXL Regions
+starting at address 0, an instance of CFMWS whose Base HPA is 0 might have
+a window size that doesn't align with the NIW * 256 MB constraint. Note
+that the matching intermediate Switch and Endpoint Decoders' HPA range
+sizes must still align to the above-mentioned rule, but the memory capacity
+that exceeds the CFMWS window size will not be accessible."
-- 
2.50.1

Re: [PATCH v4] cxl: docs/driver-api/conventions resolve conflicts between CFMWS, LMH, Decoders

[Dave Jiang]

On 8/20/25 8:06 AM, Fabio M. De Francesco wrote:
> Add documentation on how to resolve conflicts between CXL Fixed Memory
> Windows, Platform Low Memory Holes, intermediate Switch and Endpoint
> Decoders.
> 
> Cc: Ira Weiny <ira.weiny@intel.com>
> Signed-off-by: Fabio M. De Francesco <fabio.m.de.francesco@linux.intel.com>
> ---
> 
> v3 -> v4: Show and explain how CFMWS, Root Decoders, Intermediate
> 	  Switch and Endpoint Decoders match and attach Regions in
> 	  x86 platforms with Low Memory Holes (Dave, Gregory, Ira)
> 	  Remove a wrong argument about large interleaves (Jonathan)
> 
> v2 -> v3: Rework a few phrases for better clarity.
> 	  Fix grammar and syntactic errors (Randy, Alok).
> 	  Fix semantic errors ("size does not comply", Alok).
> 	  Fix technical errors ("decoder's total memory?", Alok).
> 	  
> v1 -> v2: Rewrite "Summary of the Change" section, 3r paragraph.
> 
>  Documentation/driver-api/cxl/conventions.rst | 111 +++++++++++++++++++
>  1 file changed, 111 insertions(+)
> 
> diff --git a/Documentation/driver-api/cxl/conventions.rst b/Documentation/driver-api/cxl/conventions.rst
> index da347a81a237..714240ed2e04 100644
> --- a/Documentation/driver-api/cxl/conventions.rst
> +++ b/Documentation/driver-api/cxl/conventions.rst
> @@ -45,3 +45,114 @@ Detailed Description of the Change
>  ----------------------------------
>  
>  <Propose spec language that corrects the conflict.>
> +
> +
> +Resolve conflict between CFMWS, Platform Memory Holes, and Endpoint Decoders
> +============================================================================
> +
> +Document
> +--------
> +
> +CXL Revision 3.2, Version 1.0
> +
> +License
> +-------
> +
> +SPDX-License Identifier: CC-BY-4.0
> +
> +Creator/Contributors
> +--------------------
> +
> +Fabio M. De Francesco, Intel
> +Dan J. Williams, Intel
> +Mahesh Natu, Intel
> +
> +Summary of the Change
> +---------------------
> +
> +According to the current CXL Specifications (Revision 3.2, Version 1.0),

spell out CXL on first use

> +the CXL Fixed Memory Window Structure (CFMWS) describes zero or more Host
> +Physical Address (HPA) windows associated with each CXL Host Bridge. Each
> +window represents a contiguous HPA range that may be interleaved across
> +one or more targets, including CXL Host Bridges. Each window has a set of
> +restrictions that govern its usage. It is the OSPM’s responsibility to

spell out OSPM on first use.

> +utilize each window for the specified use.
> +
> +Table 9-22 states the Window Size field contains the total number of
> +consecutive bytes of HPA this window represents. This value must be a
> +multiple of the Number of Interleave Ways * 256 MB.
> +
> +Platform Firmware (BIOS) might reserve physical addresses below 4 GB,
> +such as the Low Memory Hole for PCIe MMIO. In such cases, the CFMWS Range

Platform Firmware (BIOS) might reserve physical addresses below 4 GB where a
memory gap such as the Low Memory Hole for PCIe MMIO may exist. 

> +Size may not adhere to the NIW * 256 MB rule.
> +
> +On these systems, BIOS publishes CFMWS to communicate the active System
> +Physical Address (SPA) ranges that map to a subset of the Host Physical
> +Address (HPA) ranges. The SPA range trims out the hole, resulting in lost

So in the first paragraph, HPA is said to be described by CFMWS. But here a
brand new term SPA is introduced. I think you may need a paragraph above this
to talk about SPA vs HPA and SPA's relationship to CFMWS. Otherwise I think unless
the reader is knowledgeable all this, it is very confusing.

> +capacity in the endpoint with no SPA to map to the CXL HPA range that
> +exceeds the matching CFMWS range.
> +
> +E.g, a real x86 platform with two CFMWS, 384 GB total memory, and LMH
> +starting at 2 GB:
> +
> +Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways | Granularity
> +  0    |   0 GB     |     2 GB   |      0 GB        |       3 GB       |  12  |    256
> +  1    |   4 GB     |   380 GB   |      0 GB        |     380 GB       |  12  |    256
> +
> +HDM decoder base and HDM decoder size represent all the 12 Endpoint
> +Decoders of a 12 way region and all the intermediate Switch Decoders.
> +They are configured by the BIOS according to the NIW * 256MB rule,
> +resulting in a HPA range size of 3GB.
> +
> +The CFMWS Base and CFMWS Size are used to configure the Root Decoder HPA
> +range base and size. CFMWS cannot intersect Memory Holes, then the CFMWS[0]
> +size is smaller (2GB) than that of the Switch and Endpoint Decoders that
> +make the hierarchy (3GB).
> +
> +On that platform, only the first 2GB will be potentially usable but,
> +because of the current specs, Linux fails to make them available to the
> +users. The driver expects that Root Decoder HPA size, which is equal to
> +the CFMWS from which it is configured, to be greater or equal to the
> +matching Switch and Endpoint HDM Decoders.
> +
> +The CXL driver fails to construct Regions and to attach Endpoint and
> +intermediate Switch Decoders to those Regions after their construction.
> +
> +In order to succeed with Region construction and Decoders attachment,
> +Linux must construct Regions with Root Decoders size, and then attach to

'a Region' and 'Root Decoder'?

> +them all the intermediate Switch and Endpoint Decoders that are part of the
> +hierarchy, even though the Decoders HPA range sizes may be larger than
> +those Regions whose sizes are trimmed by Low Memory Holes.
> +
> +Benefits of the Change
> +----------------------
> +
> +Without this change, the OSPM wouldn't match Intermediate and Endpoint

s/Without this change,/Without the change/

> +Decoders with Root Decoders configured with CFMWS HPA sizes that don't
> +align with the NIW * 256MB constraint, leading to lost memdev capacity.

s/, leading/and leads to/

DJ

> +This change allows the OSPM to construct Regions and attach Intermediate
> +Switch and Endpoint Decoders to them, so that the addressable part of the
> +memory devices total capacity is not lost.
> +
> +References
> +----------
> +
> +Compute Express Link Specification Revision 3.2, Version 1.0
> +<https://www.computeexpresslink.org/>
> +
> +Detailed Description of the Change
> +----------------------------------
> +
> +The description of the Window Size field in table 9-22 needs to account
> +for platforms with Low Memory Holes, where SPA ranges might be subsets of
> +the endpoints' HPA. Therefore, it has to be changed to the following:
> +
> +"The total number of consecutive bytes of HPA this window represents.
> +This value shall be a multiple of NIW * 256 MB. On platforms that reserve
> +physical addresses below 4 GB, such as the Low Memory Hole for PCIe MMIO
> +on x86 or a requirement for greater than 8-way interleave CXL Regions
> +starting at address 0, an instance of CFMWS whose Base HPA is 0 might have
> +a window size that doesn't align with the NIW * 256 MB constraint. Note
> +that the matching intermediate Switch and Endpoint Decoders' HPA range
> +sizes must still align to the above-mentioned rule, but the memory capacity
> +that exceeds the CFMWS window size will not be accessible."

Re: [PATCH v4] cxl: docs/driver-api/conventions resolve conflicts between CFMWS, LMH, Decoders

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]

On Wed, Aug 20, 2025 at 05:06:39PM +0200, Fabio M. De Francesco wrote:
> +Creator/Contributors
> +--------------------
> +
> +Fabio M. De Francesco, Intel
> +Dan J. Williams, Intel
> +Mahesh Natu, Intel

Maybe bullet list?

> +E.g, a real x86 platform with two CFMWS, 384 GB total memory, and LMH
> +starting at 2 GB:
> +
> +Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways | Granularity
> +  0    |   0 GB     |     2 GB   |      0 GB        |       3 GB       |  12  |    256
> +  1    |   4 GB     |   380 GB   |      0 GB        |     380 GB       |  12  |    256
> +

I see CFMWS table above as normal paragraph instead in htmldocs output.

Thanks.

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v4] cxl: docs/driver-api/conventions resolve conflicts between CFMWS, LMH, Decoders

[Robert Richter]

Hi Fabio,

questions inline.

On 20.08.25 17:06:39, Fabio M. De Francesco wrote:
> Add documentation on how to resolve conflicts between CXL Fixed Memory
> Windows, Platform Low Memory Holes, intermediate Switch and Endpoint
> Decoders.
> 
> Cc: Ira Weiny <ira.weiny@intel.com>
> Signed-off-by: Fabio M. De Francesco <fabio.m.de.francesco@linux.intel.com>
> ---
> 
> v3 -> v4: Show and explain how CFMWS, Root Decoders, Intermediate
> 	  Switch and Endpoint Decoders match and attach Regions in
> 	  x86 platforms with Low Memory Holes (Dave, Gregory, Ira)
> 	  Remove a wrong argument about large interleaves (Jonathan)
> 
> v2 -> v3: Rework a few phrases for better clarity.
> 	  Fix grammar and syntactic errors (Randy, Alok).
> 	  Fix semantic errors ("size does not comply", Alok).
> 	  Fix technical errors ("decoder's total memory?", Alok).
> 	  
> v1 -> v2: Rewrite "Summary of the Change" section, 3r paragraph.
> 
>  Documentation/driver-api/cxl/conventions.rst | 111 +++++++++++++++++++
>  1 file changed, 111 insertions(+)
> 
> diff --git a/Documentation/driver-api/cxl/conventions.rst b/Documentation/driver-api/cxl/conventions.rst
> index da347a81a237..714240ed2e04 100644
> --- a/Documentation/driver-api/cxl/conventions.rst
> +++ b/Documentation/driver-api/cxl/conventions.rst
> @@ -45,3 +45,114 @@ Detailed Description of the Change
>  ----------------------------------
>  
>  <Propose spec language that corrects the conflict.>
> +
> +
> +Resolve conflict between CFMWS, Platform Memory Holes, and Endpoint Decoders
> +============================================================================
> +
> +Document
> +--------
> +
> +CXL Revision 3.2, Version 1.0
> +
> +License
> +-------
> +
> +SPDX-License Identifier: CC-BY-4.0
> +
> +Creator/Contributors
> +--------------------
> +
> +Fabio M. De Francesco, Intel
> +Dan J. Williams, Intel
> +Mahesh Natu, Intel
> +
> +Summary of the Change
> +---------------------
> +
> +According to the current CXL Specifications (Revision 3.2, Version 1.0),
> +the CXL Fixed Memory Window Structure (CFMWS) describes zero or more Host
> +Physical Address (HPA) windows associated with each CXL Host Bridge. Each
> +window represents a contiguous HPA range that may be interleaved across
> +one or more targets, including CXL Host Bridges. Each window has a set of
> +restrictions that govern its usage. It is the OSPM’s responsibility to
> +utilize each window for the specified use.
> +
> +Table 9-22 states the Window Size field contains the total number of
> +consecutive bytes of HPA this window represents. This value must be a
> +multiple of the Number of Interleave Ways * 256 MB.
> +
> +Platform Firmware (BIOS) might reserve physical addresses below 4 GB,
> +such as the Low Memory Hole for PCIe MMIO. In such cases, the CFMWS Range
> +Size may not adhere to the NIW * 256 MB rule.
> +
> +On these systems, BIOS publishes CFMWS to communicate the active System
> +Physical Address (SPA) ranges that map to a subset of the Host Physical
> +Address (HPA) ranges. The SPA range trims out the hole, resulting in lost
> +capacity in the endpoint with no SPA to map to the CXL HPA range that
> +exceeds the matching CFMWS range.
> +
> +E.g, a real x86 platform with two CFMWS, 384 GB total memory, and LMH
> +starting at 2 GB:
> +
> +Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways | Granularity
> +  0    |   0 GB     |     2 GB   |      0 GB        |       3 GB       |  12  |    256

Could you explain the zero-base limit and how this is special to LMH
or multiple of 3-way configs? What if the HPA range is non-cxl
already? E.g. my system shows this:

[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000075b5ffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000075b60000-0x0000000075baafff] ACPI NVS
...

> +  1    |   4 GB     |   380 GB   |      0 GB        |     380 GB       |  12  |    256

The EP's HDM decoder's HPA ranges overlap now as both start at 0.
Isn't that a spec violation: "Decoder m must cover an HPA range that
is below decoder m+1."?

For the second decoder, shouln't the upper limit be cut at 378 GB
(multiple of 3, or 372, multiple of 12)? But since the CFMWS Base is
non-zero that range is not detected to cut it?

> +
> +HDM decoder base and HDM decoder size represent all the 12 Endpoint
> +Decoders of a 12 way region and all the intermediate Switch Decoders.
> +They are configured by the BIOS according to the NIW * 256MB rule,
> +resulting in a HPA range size of 3GB.
> +
> +The CFMWS Base and CFMWS Size are used to configure the Root Decoder HPA
> +range base and size. CFMWS cannot intersect Memory Holes, then the CFMWS[0]
> +size is smaller (2GB) than that of the Switch and Endpoint Decoders that
> +make the hierarchy (3GB).
> +
> +On that platform, only the first 2GB will be potentially usable but,
> +because of the current specs, Linux fails to make them available to the
> +users. The driver expects that Root Decoder HPA size, which is equal to
> +the CFMWS from which it is configured, to be greater or equal to the
> +matching Switch and Endpoint HDM Decoders.
> +
> +The CXL driver fails to construct Regions and to attach Endpoint and
> +intermediate Switch Decoders to those Regions after their construction.
> +
> +In order to succeed with Region construction and Decoders attachment,
> +Linux must construct Regions with Root Decoders size, and then attach to
> +them all the intermediate Switch and Endpoint Decoders that are part of the
> +hierarchy, even though the Decoders HPA range sizes may be larger than
> +those Regions whose sizes are trimmed by Low Memory Holes.
> +
> +Benefits of the Change
> +----------------------
> +
> +Without this change, the OSPM wouldn't match Intermediate and Endpoint
> +Decoders with Root Decoders configured with CFMWS HPA sizes that don't
> +align with the NIW * 256MB constraint, leading to lost memdev capacity.
> +This change allows the OSPM to construct Regions and attach Intermediate
> +Switch and Endpoint Decoders to them, so that the addressable part of the
> +memory devices total capacity is not lost.
> +
> +References
> +----------
> +
> +Compute Express Link Specification Revision 3.2, Version 1.0
> +<https://www.computeexpresslink.org/>
> +
> +Detailed Description of the Change
> +----------------------------------
> +
> +The description of the Window Size field in table 9-22 needs to account
> +for platforms with Low Memory Holes, where SPA ranges might be subsets of
> +the endpoints' HPA. Therefore, it has to be changed to the following:
> +
> +"The total number of consecutive bytes of HPA this window represents.
> +This value shall be a multiple of NIW * 256 MB. On platforms that reserve
> +physical addresses below 4 GB, such as the Low Memory Hole for PCIe MMIO
> +on x86 or a requirement for greater than 8-way interleave CXL Regions
> +starting at address 0, an instance of CFMWS whose Base HPA is 0 might have
> +a window size that doesn't align with the NIW * 256 MB constraint. Note
> +that the matching intermediate Switch and Endpoint Decoders' HPA range
> +sizes must still align to the above-mentioned rule, but the memory capacity
> +that exceeds the CFMWS window size will not be accessible."

Have you considered to just allow smaller CFMWS ranges that just cut
the boundaries accordingly? That is, just search for a CFMWS range
within the EP's HPA ranges (or even multiple CFMWS ranges) and only
enable that HPA range? That would be more general and removes some
limitations, such as zero-base and below 4 GB only.

Thanks,

-Robert

> -- 
> 2.50.1
> 

Re: [PATCH v4] cxl: docs/driver-api/conventions resolve conflicts between CFMWS, LMH, Decoders

[Fabio M. De Francesco]

Hi Robert,

On Tuesday, August 26, 2025 3:49:58 PM Central European Summer Time Robert Richter wrote:
> Hi Fabio,
> 
> questions inline.
> 
[snip]
>
> > +
> > +On these systems, BIOS publishes CFMWS to communicate the active System
> > +Physical Address (SPA) ranges that map to a subset of the Host Physical
> > +Address (HPA) ranges. The SPA range trims out the hole, resulting in lost
> > +capacity in the endpoint with no SPA to map to the CXL HPA range that
> > +exceeds the matching CFMWS range.
> > +
> > +E.g, a real x86 platform with two CFMWS, 384 GB total memory, and LMH
> > +starting at 2 GB:
> > +
> > +Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways | Granularity
> > +  0    |   0 GB     |     2 GB   |      0 GB        |       3 GB       |  12  |    256
> 
> Could you explain the zero-base limit and how this is special to LMH
>
Linux follows the CXL specs and so it allows the construction of CXL Regions
and the attachment of HDM Decoders to them only if the Specs are not violated.

This document addresses only one of many possible violations. The proposed 
solution is not general to every possible Memory Hole on purpose.[1]

The proposed strategy wants to allow exceptions only if the CFMWS HPA range
starts at 0 and ends before 4GB. It only deals with Holes in Low Memory. The
many other combination of circumstances that lead to failures are out of the
scope of this document.
> 
> or multiple of 3-way configs?
>
It applies to all possible NIW configs.
>
> What if the HPA range is non-cxl already?
>
This solution applies to all CFMWS HPA range that start at zero and end 
before 4GB, regardless of the motivation behind memory reserve.
>
> E.g. my system shows this:
> 
> [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
> [    0.000000] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
> [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000075b5ffff] usable
> [    0.000000] BIOS-e820: [mem 0x0000000075b60000-0x0000000075baafff] ACPI NVS
> ...
> 
> > +  1    |   4 GB     |   380 GB   |      0 GB        |     380 GB       |  12  |    256
> 
> The EP's HDM decoder's HPA ranges overlap now as both start at 0.
> Isn't that a spec violation: "Decoder m must cover an HPA range that
> is below decoder m+1."?
> 
The HDM Decoder's HPA range in the second line starts at the fourth GB.
I made a copy/paste mistake and I'll fix it with the next version of this
patch. Thanks for spotting it.
> 
> For the second decoder, shouldn't the upper limit be cut at 378 GB
> (multiple of 3, or 372, multiple of 12)? But since the CFMWS Base is
> non-zero that range is not detected to cut it?
>
Another mistake. Anyway, please notice that all ranges above 4GB are 
out of the scope of this document. On purpose. 
> >
[snip]
>
> > +Detailed Description of the Change
> > +----------------------------------
> > +
> > +The description of the Window Size field in table 9-22 needs to account
> > +for platforms with Low Memory Holes, where SPA ranges might be subsets of
> > +the endpoints' HPA. Therefore, it has to be changed to the following:
> > +
> > +"The total number of consecutive bytes of HPA this window represents.
> > +This value shall be a multiple of NIW * 256 MB. On platforms that reserve
> > +physical addresses below 4 GB, such as the Low Memory Hole for PCIe MMIO
> > +on x86 or a requirement for greater than 8-way interleave CXL Regions
> > +starting at address 0, an instance of CFMWS whose Base HPA is 0 might have
> > +a window size that doesn't align with the NIW * 256 MB constraint. Note
> > +that the matching intermediate Switch and Endpoint Decoders' HPA range
> > +sizes must still align to the above-mentioned rule, but the memory capacity
> > +that exceeds the CFMWS window size will not be accessible."
> 
> Have you considered to just allow smaller CFMWS ranges that just cut
> the boundaries accordingly? That is, just search for a CFMWS range
> within the EP's HPA ranges (or even multiple CFMWS ranges) and only
> enable that HPA range? That would be more general and removes some
> limitations, such as zero-base and below 4 GB only.
> 
This solution doesn't want to be a general solution for all kinds of Memory 
Holes. Dan has been very clear about cutting out solutions to general cases.
This solution is limited on purpose.[1]

Thanks,

Fabio

[1] https://lore.kernel.org/linux-cxl/67ec4d61c3fd6_288d2947b@dwillia2-xfh.jf.intel.com.notmuch/

Re: [PATCH v4] cxl: docs/driver-api/conventions resolve conflicts between CFMWS, LMH, Decoders

[Robert Richter]

On 01.09.25 14:22:00, Fabio M. De Francesco wrote:
> Hi Robert,
> 
> On Tuesday, August 26, 2025 3:49:58 PM Central European Summer Time Robert Richter wrote:
> > Hi Fabio,
> > 
> > questions inline.
> > 
> [snip]
> >
> > > +
> > > +On these systems, BIOS publishes CFMWS to communicate the active System
> > > +Physical Address (SPA) ranges that map to a subset of the Host Physical
> > > +Address (HPA) ranges. The SPA range trims out the hole, resulting in lost
> > > +capacity in the endpoint with no SPA to map to the CXL HPA range that
> > > +exceeds the matching CFMWS range.
> > > +
> > > +E.g, a real x86 platform with two CFMWS, 384 GB total memory, and LMH
> > > +starting at 2 GB:
> > > +
> > > +Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways | Granularity
> > > +  0    |   0 GB     |     2 GB   |      0 GB        |       3 GB       |  12  |    256
> > 
> > Could you explain the zero-base limit and how this is special to LMH
> >
> Linux follows the CXL specs and so it allows the construction of CXL Regions
> and the attachment of HDM Decoders to them only if the Specs are not violated.
> 
> This document addresses only one of many possible violations. The proposed 
> solution is not general to every possible Memory Hole on purpose.[1]
> 
> The proposed strategy wants to allow exceptions only if the CFMWS HPA range
> starts at 0 and ends before 4GB. It only deals with Holes in Low Memory. The
> many other combination of circumstances that lead to failures are out of the
> scope of this document.
> > 
> > or multiple of 3-way configs?
> >
> It applies to all possible NIW configs.
> >
> > What if the HPA range is non-cxl already?
> >
> This solution applies to all CFMWS HPA range that start at zero and end 
> before 4GB, regardless of the motivation behind memory reserve.
> >
> > E.g. my system shows this:
> > 
> > [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
> > [    0.000000] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
> > [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000075b5ffff] usable
> > [    0.000000] BIOS-e820: [mem 0x0000000075b60000-0x0000000075baafff] ACPI NVS
> > ...

See below, it is still unclear you you are handling this.

> > 
> > > +  1    |   4 GB     |   380 GB   |      0 GB        |     380 GB       |  12  |    256
> > 
> > The EP's HDM decoder's HPA ranges overlap now as both start at 0.
> > Isn't that a spec violation: "Decoder m must cover an HPA range that
> > is below decoder m+1."?
> > 
> The HDM Decoder's HPA range in the second line starts at the fourth GB.
> I made a copy/paste mistake and I'll fix it with the next version of this
> patch. Thanks for spotting it.
> > 
> > For the second decoder, shouldn't the upper limit be cut at 378 GB
> > (multiple of 3, or 372, multiple of 12)? But since the CFMWS Base is
> > non-zero that range is not detected to cut it?
> >
> Another mistake. Anyway, please notice that all ranges above 4GB are 
> out of the scope of this document. On purpose. 

Let's see your fixes.

> > >
> [snip]
> >
> > > +Detailed Description of the Change
> > > +----------------------------------
> > > +
> > > +The description of the Window Size field in table 9-22 needs to account
> > > +for platforms with Low Memory Holes, where SPA ranges might be subsets of
> > > +the endpoints' HPA. Therefore, it has to be changed to the following:
> > > +
> > > +"The total number of consecutive bytes of HPA this window represents.
> > > +This value shall be a multiple of NIW * 256 MB. On platforms that reserve

Add a line break to mark the text as a special case.

> > > +physical addresses below 4 GB, such as the Low Memory Hole for PCIe MMIO
> > > +on x86 or a requirement for greater than 8-way interleave CXL Regions
> > > +starting at address 0, an instance of CFMWS whose Base HPA is 0 might have
> > > +a window size that doesn't align with the NIW * 256 MB constraint. Note
> > > +that the matching intermediate Switch and Endpoint Decoders' HPA range
> > > +sizes must still align to the above-mentioned rule, but the memory capacity
> > > +that exceeds the CFMWS window size will not be accessible."
> > 
> > Have you considered to just allow smaller CFMWS ranges that just cut
> > the boundaries accordingly? That is, just search for a CFMWS range
> > within the EP's HPA ranges (or even multiple CFMWS ranges) and only
> > enable that HPA range? That would be more general and removes some
> > limitations, such as zero-base and below 4 GB only.
> > 
> This solution doesn't want to be a general solution for all kinds of Memory 
> Holes. Dan has been very clear about cutting out solutions to general cases.
> This solution is limited on purpose.[1]

I still don't get how you set the actual base address if it must be
zero to trigger the quirk. How does this work with SRAT and what about
other memory ranges that are in the same range?

It should be clearly marked as a quirk that only under the specific
conditions. The implemenation should be separate from the main path to
better isolate the change. That was unclear to me. Still, there are
the questions above.

Thanks,

-Robert

> 
> Thanks,
> 
> Fabio
> 
> [1] https://lore.kernel.org/linux-cxl/67ec4d61c3fd6_288d2947b@dwillia2-xfh.jf.intel.com.notmuch/
> 
> 
> 

Re: [PATCH v4] cxl: docs/driver-api/conventions resolve conflicts between CFMWS, LMH, Decoders

[Gregory Price]

On Wed, Aug 20, 2025 at 05:06:39PM +0200, Fabio M. De Francesco wrote:
> +
> +E.g, a real x86 platform with two CFMWS, 384 GB total memory, and LMH
> +starting at 2 GB:
> +
> +Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways | Granularity
> +  0    |   0 GB     |     2 GB   |      0 GB        |       3 GB       |  12  |    256
> +  1    |   4 GB     |   380 GB   |      0 GB        |     380 GB       |  12  |    256
> +

This may be a dumb question, but... how is validation supposed to work?

Like in theory according to the above something like the following would
also be valid:

Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size
  0    |   4 GB     |   380 GB   |      2 GB        |     382 GB      

(ignoring ways/granularity, i didn't adjust those).

The entirety of the CFMWS would be contained within the HDM decoder, but
with carve-outs on either end.  This would be "allowed" according to the
logic here.

This would effectively allow all HDM decoder base/size values to be valid
as long as one CFMWS is contained entirely within it.

As a result, wouldn't it then also be valid to have an HDM Decoder cover
more than one CFMWS range (two full CFMWS described by a single HDM
decoder).

That seems like it could cause issues.

~Gregory

Re: [PATCH v4] cxl: docs/driver-api/conventions resolve conflicts between CFMWS, LMH, Decoders

[Fabio M. De Francesco]

On Wednesday, August 27, 2025 10:23:46 PM Central European Summer Time Gregory Price wrote:
> On Wed, Aug 20, 2025 at 05:06:39PM +0200, Fabio M. De Francesco wrote:
> > +
> > +E.g, a real x86 platform with two CFMWS, 384 GB total memory, and LMH
> > +starting at 2 GB:
> > +
> > +Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways | Granularity
> > +  0    |   0 GB     |     2 GB   |      0 GB        |       3 GB       |  12  |    256
> > +  1    |   4 GB     |   380 GB   |      0 GB        |     380 GB       |  12  |    256
> > +
> 
> This may be a dumb question, but... how is validation supposed to work?
> 
> Like in theory according to the above something like the following would
> also be valid:
> 
> Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size
>   0    |   4 GB     |   380 GB   |      2 GB        |     382 GB      
> 
I'm sorry, it seems that a mistake with copy/pasting I made has led you to 
hypothesize a case that is out of scope of this document.

A case like the one you described will still lead the CXL driver to fail.

Please refer to my reply to Robert and to an old email from Dan.[1][2]

Thanks,

Fabio

[1] https://lore.kernel.org/linux-cxl/4179950.vuYhMxLoTh@fdefranc-mobl3/
[2] https://lore.kernel.org/linux-cxl/67ec4d61c3fd6_288d2947b@dwillia2-xfh.jf.intel.com.notmuch/
>
> (ignoring ways/granularity, i didn't adjust those).
> 
> The entirety of the CFMWS would be contained within the HDM decoder, but
> with carve-outs on either end.  This would be "allowed" according to the
> logic here.
> 
> This would effectively allow all HDM decoder base/size values to be valid
> as long as one CFMWS is contained entirely within it.
> 
> As a result, wouldn't it then also be valid to have an HDM Decoder cover
> more than one CFMWS range (two full CFMWS described by a single HDM
> decoder).
> 
> That seems like it could cause issues.
> 
> ~Gregory
> 
> 

[PATCH v4] cxl: docs/driver-api/conventions resolve conflicts between CFMWS, LMH, Decoders

[Fabio M. De Francesco]

Add documentation on how to resolve conflicts between CXL Fixed Memory
Windows, Platform Low Memory Holes, intermediate Switch and Endpoint
Decoders.

Cc: Ira Weiny <ira.weiny@intel.com>
Signed-off-by: Fabio M. De Francesco <fabio.m.de.francesco@linux.intel.com>
---

v3 -> v4: Show and explain how CFMWS, Root Decoders, Intermediate
	  Switch and Endpoint Decoders match and attach Regions in
	  x86 platforms with Low Memory Holes (Dave, Gregory, Ira)
	  Remove a wrong argument about large interleaves (Jonathan)

v2 -> v3: Rework a few phrases for better clarity.
	  Fix grammar and syntactic errors (Randy, Alok).
	  Fix semantic errors ("size does not comply", Alok).
	  Fix technical errors ("decoder's total memory?", Alok).
	  
v1 -> v2: Rewrite "Summary of the Change" section, 3r paragraph.

 Documentation/driver-api/cxl/conventions.rst | 111 +++++++++++++++++++
 1 file changed, 111 insertions(+)

diff --git a/Documentation/driver-api/cxl/conventions.rst b/Documentation/driver-api/cxl/conventions.rst
index da347a81a237..714240ed2e04 100644
--- a/Documentation/driver-api/cxl/conventions.rst
+++ b/Documentation/driver-api/cxl/conventions.rst
@@ -45,3 +45,114 @@ Detailed Description of the Change
 ----------------------------------
 
 <Propose spec language that corrects the conflict.>
+
+
+Resolve conflict between CFMWS, Platform Memory Holes, and Endpoint Decoders
+============================================================================
+
+Document
+--------
+
+CXL Revision 3.2, Version 1.0
+
+License
+-------
+
+SPDX-License Identifier: CC-BY-4.0
+
+Creator/Contributors
+--------------------
+
+Fabio M. De Francesco, Intel
+Dan J. Williams, Intel
+Mahesh Natu, Intel
+
+Summary of the Change
+---------------------
+
+According to the current CXL Specifications (Revision 3.2, Version 1.0),
+the CXL Fixed Memory Window Structure (CFMWS) describes zero or more Host
+Physical Address (HPA) windows associated with each CXL Host Bridge. Each
+window represents a contiguous HPA range that may be interleaved across
+one or more targets, including CXL Host Bridges. Each window has a set of
+restrictions that govern its usage. It is the OSPM’s responsibility to
+utilize each window for the specified use.
+
+Table 9-22 states the Window Size field contains the total number of
+consecutive bytes of HPA this window represents. This value must be a
+multiple of the Number of Interleave Ways * 256 MB.
+
+Platform Firmware (BIOS) might reserve physical addresses below 4 GB,
+such as the Low Memory Hole for PCIe MMIO. In such cases, the CFMWS Range
+Size may not adhere to the NIW * 256 MB rule.
+
+On these systems, BIOS publishes CFMWS to communicate the active System
+Physical Address (SPA) ranges that map to a subset of the Host Physical
+Address (HPA) ranges. The SPA range trims out the hole, resulting in lost
+capacity in the endpoint with no SPA to map to the CXL HPA range that
+exceeds the matching CFMWS range.
+
+E.g, a real x86 platform with two CFMWS, 384 GB total memory, and LMH
+starting at 2 GB:
+
+Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways | Granularity
+  0    |   0 GB     |     2 GB   |      0 GB        |       3 GB       |  12  |    256
+  1    |   4 GB     |   380 GB   |      0 GB        |     380 GB       |  12  |    256
+
+HDM decoder base and HDM decoder size represent all the 12 Endpoint
+Decoders of a 12 way region and all the intermediate Switch Decoders.
+They are configured by the BIOS according to the NIW * 256MB rule,
+resulting in a HPA range size of 3GB.
+
+The CFMWS Base and CFMWS Size are used to configure the Root Decoder HPA
+range base and size. CFMWS cannot intersect Memory Holes, then the CFMWS[0]
+size is smaller (2GB) than that of the Switch and Endpoint Decoders that
+make the hierarchy (3GB).
+
+On that platform, only the first 2GB will be potentially usable but,
+because of the current specs, Linux fails to make them available to the
+users. The driver expects that Root Decoder HPA size, which is equal to
+the CFMWS from which it is configured, to be greater or equal to the
+matching Switch and Endpoint HDM Decoders.
+
+The CXL driver fails to construct Regions and to attach Endpoint and
+intermediate Switch Decoders to those Regions after their construction.
+
+In order to succeed with Region construction and Decoders attachment,
+Linux must construct Regions with Root Decoders size, and then attach to
+them all the intermediate Switch and Endpoint Decoders that are part of the
+hierarchy, even though the Decoders HPA range sizes may be larger than
+those Regions whose sizes are trimmed by Low Memory Holes.
+
+Benefits of the Change
+----------------------
+
+Without this change, the OSPM wouldn't match Intermediate and Endpoint
+Decoders with Root Decoders configured with CFMWS HPA sizes that don't
+align with the NIW * 256MB constraint, leading to lost memdev capacity.
+This change allows the OSPM to construct Regions and attach Intermediate
+Switch and Endpoint Decoders to them, so that the addressable part of the
+memory devices total capacity is not lost.
+
+References
+----------
+
+Compute Express Link Specification Revision 3.2, Version 1.0
+<https://www.computeexpresslink.org/>
+
+Detailed Description of the Change
+----------------------------------
+
+The description of the Window Size field in table 9-22 needs to account
+for platforms with Low Memory Holes, where SPA ranges might be subsets of
+the endpoints' HPA. Therefore, it has to be changed to the following:
+
+"The total number of consecutive bytes of HPA this window represents.
+This value shall be a multiple of NIW * 256 MB. On platforms that reserve
+physical addresses below 4 GB, such as the Low Memory Hole for PCIe MMIO
+on x86 or a requirement for greater than 8-way interleave CXL Regions
+starting at address 0, an instance of CFMWS whose Base HPA is 0 might have
+a window size that doesn't align with the NIW * 256 MB constraint. Note
+that the matching intermediate Switch and Endpoint Decoders' HPA range
+sizes must still align to the above-mentioned rule, but the memory capacity
+that exceeds the CFMWS window size will not be accessible."
-- 
2.50.1