[PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS, LMH, ED

[PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS, LMH, ED

[Fabio M. De Francesco]

Add documentation on how to resolve conflicts between CXL Fixed Memory
Windows, Platform Memory Holes, and Endpoint Decoders.

Signed-off-by: Fabio M. De Francesco <fabio.m.de.francesco@linux.intel.com>
---

v2 -> v3: Rework a few phrases for better clarity.
	  Fix grammar and syntactic errors (Randy, Alok).
	  Fix semantic errors ("size does not comply", Alok).
	  Fix technical errors ("decoder's total memory?", Alok).
	  
v1 -> v2: Rewrite "Summary of the Change" section, 3r paragraph.

 Documentation/driver-api/cxl/conventions.rst | 85 ++++++++++++++++++++
 1 file changed, 85 insertions(+)

diff --git a/Documentation/driver-api/cxl/conventions.rst b/Documentation/driver-api/cxl/conventions.rst
index da347a81a237..d6c8f4cf2f5b 100644
--- a/Documentation/driver-api/cxl/conventions.rst
+++ b/Documentation/driver-api/cxl/conventions.rst
@@ -45,3 +45,88 @@ Detailed Description of the Change
 ----------------------------------
 
 <Propose spec language that corrects the conflict.>
+
+
+Resolve conflict between CFMWS, Platform Memory Holes, and Endpoint Decoders
+============================================================================
+
+Document
+--------
+
+CXL Revision 3.2, Version 1.0
+
+License
+-------
+
+SPDX-License Identifier: CC-BY-4.0
+
+Creator/Contributors
+--------------------
+
+Fabio M. De Francesco, Intel
+Dan J. Williams, Intel
+Mahesh Natu, Intel
+
+Summary of the Change
+---------------------
+
+According to the current CXL Specifications (Revision 3.2, Version 1.0)
+the CXL Fixed Memory Window Structure (CFMWS) describes zero or more Host
+Physical Address (HPA) windows that are associated with each CXL Host
+Bridge. Each window represents a contiguous HPA range that may be
+interleaved across one or more targets, some of which are CXL Host Bridges.
+Associated with each window is a set of restrictions that govern its usage.
+It is the OSPM’s responsibility to utilize each window for the specified
+use.
+
+Table 9-22 states the Window Size field contains the total number of
+consecutive bytes of HPA this window represents and this value shall be a
+multiple of Number of Interleave Ways * 256 MB.
+
+Platform Firmware (BIOS) might reserve part of physical addresses below
+4 GB (e.g., the Low Memory Hole that describes PCIe memory space for MMIO
+or a requirement for the greater than 8 way interleave CXL regions starting
+at address 0). In that case the Window Size value cannot be anymore
+constrained to the NIW * 256 MB above-mentioned rule.
+
+On those systems, BIOS publishes CFMWS which communicate the active System
+Physical Address (SPA) ranges that map to a subset of the Host Physical
+Address (HPA) ranges. The SPA range trims out the hole, and capacity in the
+endpoint is lost with no SPA to map to CXL HPA in that hole.
+
+The description of the Window Size field in table 9-22 needs to take that
+special case into account.
+
+Note that the Endpoint Decoders HPA range sizes have to comply with the
+alignment constraints and so a part of their memory capacity might not be
+accessible if their size exceeds the matching CFMWS range's.
+
+Benefits of the Change
+----------------------
+
+Without this change, the OSPM wouldn't match Endpoint Decoders with CFMWS
+whose Window Size don't fit the alignment constraints and so the memdev
+capacity would be lost. This change allows the OSPM to match Endpoint
+Decoders whose HPA range size exceeds the matching CFMWS and create
+regions that at least utilize a part of the memory devices total capacity.
+
+References
+----------
+
+Compute Express Link Specification Revision 3.2, Version 1.0
+<https://www.computeexpresslink.org/>
+
+Detailed Description of the Change
+----------------------------------
+
+The current description of a CFMWS Window Size (Table 9-22) is replaced
+with:
+
+"The total number of consecutive bytes of HPA this window represents. This
+value shall be a multiple of NIW*256 MB. On platforms that reserve physical
+addresses below 4 GB for special use (e.g., the Low Memory Hole for PCIe
+MMIO on x86), an instance of CFMWS whose Base HPA is 0 might have a window
+size that doesn't align with the NIW*256 MB constraint; note that the
+matching Endpoint Decoders HPA range size must still align to the
+above-mentioned rule and so the memory capacity that might exceed the CFMWS
+window size will not be accessible.".

base-commit: a021802c18c4c30dff3db9bd355cacb68521f1aa
-- 
2.49.0

Re: [PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS, LMH, ED

[Gregory Price]

On Mon, Jun 23, 2025 at 05:29:02PM +0200, Fabio M. De Francesco wrote:
> Add documentation on how to resolve conflicts between CXL Fixed Memory
> Windows, Platform Memory Holes, and Endpoint Decoders.
> 
> Signed-off-by: Fabio M. De Francesco <fabio.m.de.francesco@linux.intel.com>

I won't block a doc update on a suggestion so

Reviewed-by: Gregory Price <gourry@gourry.net>

> +Platform Firmware (BIOS) might reserve part of physical addresses below
> +4 GB (e.g., the Low Memory Hole that describes PCIe memory space for MMIO
> +or a requirement for the greater than 8 way interleave CXL regions starting
> +at address 0). In that case the Window Size value cannot be anymore
> +constrained to the NIW * 256 MB above-mentioned rule.

It might be nice to have a diagram that explains this visually, as it's
difficult for me to understand the implications through words alone...

which is likely why the conflict exists in the first place :]

~Gregory

Re: [PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS, LMH, ED

[Dave Jiang]

On 6/23/25 12:19 PM, Gregory Price wrote:
> On Mon, Jun 23, 2025 at 05:29:02PM +0200, Fabio M. De Francesco wrote:
>> Add documentation on how to resolve conflicts between CXL Fixed Memory
>> Windows, Platform Memory Holes, and Endpoint Decoders.
>>
>> Signed-off-by: Fabio M. De Francesco <fabio.m.de.francesco@linux.intel.com>
> 
> I won't block a doc update on a suggestion so
> 
> Reviewed-by: Gregory Price <gourry@gourry.net>
> 
>> +Platform Firmware (BIOS) might reserve part of physical addresses below
>> +4 GB (e.g., the Low Memory Hole that describes PCIe memory space for MMIO
>> +or a requirement for the greater than 8 way interleave CXL regions starting
>> +at address 0). In that case the Window Size value cannot be anymore
>> +constrained to the NIW * 256 MB above-mentioned rule.
> 
> It might be nice to have a diagram that explains this visually, as it's
> difficult for me to understand the implications through words alone...

+1 on request for diagram to explain. We should try to document this issue as clearly as possible. Thank you.

> 
> which is likely why the conflict exists in the first place :]
> 
> ~Gregory

Re: [PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS, LMH, ED

[Gregory Price]

On Tue, Jul 01, 2025 at 08:23:57AM -0700, Dave Jiang wrote:
> 
> 
> On 6/23/25 12:19 PM, Gregory Price wrote:
> > On Mon, Jun 23, 2025 at 05:29:02PM +0200, Fabio M. De Francesco wrote:
> >> Add documentation on how to resolve conflicts between CXL Fixed Memory
> >> Windows, Platform Memory Holes, and Endpoint Decoders.
> >>
> >> Signed-off-by: Fabio M. De Francesco <fabio.m.de.francesco@linux.intel.com>
> > 
> > I won't block a doc update on a suggestion so
> > 
> > Reviewed-by: Gregory Price <gourry@gourry.net>
> > 
> >> +Platform Firmware (BIOS) might reserve part of physical addresses below
> >> +4 GB (e.g., the Low Memory Hole that describes PCIe memory space for MMIO
> >> +or a requirement for the greater than 8 way interleave CXL regions starting
> >> +at address 0). In that case the Window Size value cannot be anymore
> >> +constrained to the NIW * 256 MB above-mentioned rule.
> > 
> > It might be nice to have a diagram that explains this visually, as it's
> > difficult for me to understand the implications through words alone...
> 
> +1 on request for diagram to explain. We should try to document this issue as clearly as possible. Thank you.
>

At the very least, it would be nice to have an explicitly example that
explains the expected cfmws/decoder configurations that are valid but
"technically" violate the spec

I *think* this basically boils down to "CFMWS size is not aligned, but
all the decoders it targets are"?  If I understand this correctly?

> > 
> > which is likely why the conflict exists in the first place :]
> > 
> > ~Gregory
> 

Re: [PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS, LMH, ED

[Fabio M. De Francesco]

On Thursday, July 3, 2025 9:40:21 PM Central European Summer Time Gregory Price wrote:
> On Tue, Jul 01, 2025 at 08:23:57AM -0700, Dave Jiang wrote:
> > 
> > 
> > On 6/23/25 12:19 PM, Gregory Price wrote:
> > > On Mon, Jun 23, 2025 at 05:29:02PM +0200, Fabio M. De Francesco wrote:
> > >> Add documentation on how to resolve conflicts between CXL Fixed Memory
> > >> Windows, Platform Memory Holes, and Endpoint Decoders.
> > >>
> > >> Signed-off-by: Fabio M. De Francesco <fabio.m.de.francesco@linux.intel.com>
> > > 
> > > I won't block a doc update on a suggestion so
> > > 
> > > Reviewed-by: Gregory Price <gourry@gourry.net>
> > > 
> > >> +Platform Firmware (BIOS) might reserve part of physical addresses below
> > >> +4 GB (e.g., the Low Memory Hole that describes PCIe memory space for MMIO
> > >> +or a requirement for the greater than 8 way interleave CXL regions starting
> > >> +at address 0). In that case the Window Size value cannot be anymore
> > >> +constrained to the NIW * 256 MB above-mentioned rule.
> > > 
> > > It might be nice to have a diagram that explains this visually, as it's
> > > difficult for me to understand the implications through words alone...
> > 
> > +1 on request for diagram to explain. We should try to document this issue as clearly as possible. Thank you.
> >
> 
> At the very least, it would be nice to have an explicitly example that
> explains the expected cfmws/decoder configurations that are valid but
> "technically" violate the spec
> 
> I *think* this basically boils down to "CFMWS size is not aligned, but
> all the decoders it targets are"?  If I understand this correctly?
> 
Yes, this is the intended meaning. 

E.g, two windows, 384 GB total memory, LMH at 2 GB:

Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways | Granularity
  0    |   0 GB     |     2 GB   |      0 GB        |       3 GB       |  12  |    256
  1    |   4 GB     |   380 GB   |      0 GB        |     380 GB       |  12  |    256

12 *256MB = 3GB, the Top of Low memory below 4GB is set as 2GB in the platform for MMIO Low Mem Hole (2-4GB). Only 2GB will be potentially available for user, but currently the CXL driver expects a CFMWS range's size to be greater or equal to the HDM Decoder's, so it  doesn't match them and we lose those 2GB described by CFMWS0. If we allow the matching, we make those 2GB available to users.

Is a table like the one above good enough to make this subject clearer?

Thanks,

Fabio
>
> > > 
> > > which is likely why the conflict exists in the first place :]
> > > 
> > > ~Gregory
> > 
> 

Re: [PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS, LMH, ED

[Gregory Price]

On Fri, Jul 04, 2025 at 12:05:33PM +0200, Fabio M. De Francesco wrote:
> On Thursday, July 3, 2025 9:40:21 PM Central European Summer Time Gregory Price wrote:
> > On Tue, Jul 01, 2025 at 08:23:57AM -0700, Dave Jiang wrote:
> > > 
> > > 
> > > On 6/23/25 12:19 PM, Gregory Price wrote:
> > > > On Mon, Jun 23, 2025 at 05:29:02PM +0200, Fabio M. De Francesco wrote:
> > > >> Add documentation on how to resolve conflicts between CXL Fixed Memory
> > > >> Windows, Platform Memory Holes, and Endpoint Decoders.
> > > >>
> > > >> Signed-off-by: Fabio M. De Francesco <fabio.m.de.francesco@linux.intel.com>
> > > > 
> > > > I won't block a doc update on a suggestion so
> > > > 
> > > > Reviewed-by: Gregory Price <gourry@gourry.net>
> > > > 
> > > >> +Platform Firmware (BIOS) might reserve part of physical addresses below
> > > >> +4 GB (e.g., the Low Memory Hole that describes PCIe memory space for MMIO
> > > >> +or a requirement for the greater than 8 way interleave CXL regions starting
> > > >> +at address 0). In that case the Window Size value cannot be anymore
> > > >> +constrained to the NIW * 256 MB above-mentioned rule.
> > > > 
> > > > It might be nice to have a diagram that explains this visually, as it's
> > > > difficult for me to understand the implications through words alone...
> > > 
> > > +1 on request for diagram to explain. We should try to document this issue as clearly as possible. Thank you.
> > >
> > 
> > At the very least, it would be nice to have an explicitly example that
> > explains the expected cfmws/decoder configurations that are valid but
> > "technically" violate the spec
> > 
> > I *think* this basically boils down to "CFMWS size is not aligned, but
> > all the decoders it targets are"?  If I understand this correctly?
> > 
> Yes, this is the intended meaning. 
> 
> E.g, two windows, 384 GB total memory, LMH at 2 GB:
> 
> Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways | Granularity
>   0    |   0 GB     |     2 GB   |      0 GB        |       3 GB       |  12  |    256
>   1    |   4 GB     |   380 GB   |      0 GB        |     380 GB       |  12  |    256


In this example, does the root decoder have 0/2 and the host bridge
decoder have 0/3?  Because root decoders objects are created by reading
the CFMWS Base/Size - while host bridge decoder objects are created by
reading the host bridge DVSECS.

Or does the linux code read CFMWS Base/Size and create a root decoder
with 0/3 because "that's what it should have"?  etc

I think you need to describe what the expected behavior is for what linux
will produce in terms of the decoder objects given the above.

> 
> 12 *256MB = 3GB, the Top of Low memory below 4GB is set as 2GB in the platform for MMIO Low Mem Hole (2-4GB). Only 2GB will be potentially available for user, but currently the CXL driver expects a CFMWS range's size to be greater or equal to the HDM Decoder's, so it  doesn't match them and we lose those 2GB described by CFMWS0. If we allow the matching, we make those 2GB available to users.
> 
> Is a table like the one above good enough to make this subject clearer?
> 
> Thanks,
> 
> Fabio
> >
> > > > 
> > > > which is likely why the conflict exists in the first place :]
> > > > 
> > > > ~Gregory
> > > 
> > 
> 
> 
> 
> 

Re: [PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS, LMH, ED

[Fabio M. De Francesco]

On Monday, July 7, 2025 9:55:03 PM Central European Summer Time Gregory Price wrote:
> On Fri, Jul 04, 2025 at 12:05:33PM +0200, Fabio M. De Francesco wrote:
> > On Thursday, July 3, 2025 9:40:21 PM Central European Summer Time Gregory Price wrote:
> > > On Tue, Jul 01, 2025 at 08:23:57AM -0700, Dave Jiang wrote:
> > > > 
> > > > 
> > > > On 6/23/25 12:19 PM, Gregory Price wrote:
> > > > > On Mon, Jun 23, 2025 at 05:29:02PM +0200, Fabio M. De Francesco wrote:
> > > > >> Add documentation on how to resolve conflicts between CXL Fixed Memory
> > > > >> Windows, Platform Memory Holes, and Endpoint Decoders.
> > > > >>
> > > > >> Signed-off-by: Fabio M. De Francesco <fabio.m.de.francesco@linux.intel.com>
> > > > > 
> > > > > I won't block a doc update on a suggestion so
> > > > > 
> > > > > Reviewed-by: Gregory Price <gourry@gourry.net>
> > > > > 
> > > > >> +Platform Firmware (BIOS) might reserve part of physical addresses below
> > > > >> +4 GB (e.g., the Low Memory Hole that describes PCIe memory space for MMIO
> > > > >> +or a requirement for the greater than 8 way interleave CXL regions starting
> > > > >> +at address 0). In that case the Window Size value cannot be anymore
> > > > >> +constrained to the NIW * 256 MB above-mentioned rule.
> > > > > 
> > > > > It might be nice to have a diagram that explains this visually, as it's
> > > > > difficult for me to understand the implications through words alone...
> > > > 
> > > > +1 on request for diagram to explain. We should try to document this issue as clearly as possible. Thank you.
> > > >
> > > 
> > > At the very least, it would be nice to have an explicitly example that
> > > explains the expected cfmws/decoder configurations that are valid but
> > > "technically" violate the spec
> > > 
> > > I *think* this basically boils down to "CFMWS size is not aligned, but
> > > all the decoders it targets are"?  If I understand this correctly?
> > > 
> > Yes, this is the intended meaning. 
> > 
> > E.g, two windows, 384 GB total memory, LMH at 2 GB:
> > 
> > Window | CFMWS Base | CFMWS Size | HDM Decoder Base | HDM Decoder Size | Ways | Granularity
> >   0    |   0 GB     |     2 GB   |      0 GB        |       3 GB       |  12  |    256
> >   1    |   4 GB     |   380 GB   |      0 GB        |     380 GB       |  12  |    256
> 
> 
> In this example, does the root decoder have 0/2 and the host bridge
> decoder have 0/3?  Because root decoders objects are created by reading
> the CFMWS Base/Size - while host bridge decoder objects are created by
> reading the host bridge DVSECS.
>
> Or does the linux code read CFMWS Base/Size and create a root decoder
> with 0/3 because "that's what it should have"?  etc
>
No, sorry for any confusion. I also noticed that some unwanted additional
lines slipped in my last reply, probably for a copy/paste mistake.

Let me explain...

The table above shows a real configuration copied from an x86 platform 
where the Low Memory Hole (LMH) starts at 2GB. 

The"HDM Decoder Base/Size" refers specifically to the CXL Endpoint 
Decoders HPA range Base/Size. The first row of the table describes the 
first window (CFMWS[0]), whose HPA rage base/size is 0/2GB, and the 
Endpoint Decoder that the CXL driver should match with that CFMWS, 
whose HPA range base/size is 0/3GB.

The Endpoint Decoder follows the 256MB x 12 interleave ways rule 
mandated by the CXL specifications. Instead the CFMWS is truncated 
immediately before the beginning of the LMH, because it needs to publish
only those memory locations that the system can address.   

Currently the CXL driver fails during the Region construction and 
subsequent Endpoint Decoders attachment because it cannot match Root 
Decoders (that are created with HPA ranges base/size equal to CFMWS)
and Switch Decoders with Endpoint decoders due to this size discrepancy. 

The driver expects that the Endpoint Decoders HPA ranges to be contained 
into their corresponding Root Decoders. Furthermore, Linux fails to 
attach Endpoint decoders to already constructed CXL Regions because of 
the same size discrepancy issue. 
> 
> I think you need to describe what the expected behavior is for what linux
> will produce in terms of the decoder objects given the above.
>
The expected behavior is that Linux should be able to match the Endpoint 
Decoder with the Root Decoder range even if the CFMWS size is smaller 
than the Decoder's, as long as the latter adheres to the 256MB * interleave 
ways rule. Furthermore, Linux should be able to match the Endpoint decoders 
with already constructed CXL Regions and allow the attachment process to 
succeed. 

This would enable the utilization of the 2GB memory described by CFMWS[0].

I hope this time I explained the LMH interference with CXL Regions 
construction and Endpoint Decoders attachment with better clarity.

If this explanation suffices, I will incorporate it into the next version
of this patch and also explain that "HDM Decoder" stands for Endpoint Decoder 
and that the CFMWS HPA base/size describes the System Physical Address (SPA) 
which the CXL driver uses to make Root Decoders HPA range base/size. 

Thanks,

Fabio

Re: [PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS, LMH, ED

[Gregory Price]

On Thu, Jul 17, 2025 at 04:14:13PM +0200, Fabio M. De Francesco wrote:
> The table above shows a real configuration copied from an x86 platform 
> where the Low Memory Hole (LMH) starts at 2GB. 
> 
> The"HDM Decoder Base/Size" refers specifically to the CXL Endpoint 
> Decoders HPA range Base/Size. The first row of the table describes the 
> first window (CFMWS[0]), whose HPA rage base/size is 0/2GB, and the 
> Endpoint Decoder that the CXL driver should match with that CFMWS, 
> whose HPA range base/size is 0/3GB.

The only thing i ask is being more precise with decoder references.

HDM Decoder can refer to any of: root, switch, hb, or endpoint decoders.

Below you make this distinct in the explanation, but in the table it's
simply general "HDM Decoder".  All I ask is for a bit more clarity on
what decoder will contain what values to avoid further ambiguity.

> The driver expects that the Endpoint Decoders HPA ranges to be contained 
> into their corresponding Root Decoders. Furthermore, Linux fails to 
> attach Endpoint decoders to already constructed CXL Regions because of 
> the same size discrepancy issue. 
> > 
> > I think you need to describe what the expected behavior is for what linux
> > will produce in terms of the decoder objects given the above.
> >
> The expected behavior is that Linux should be able to match the Endpoint 
> Decoder with the Root Decoder range even if the CFMWS size is smaller 
> than the Decoder's, as long as the latter adheres to the 256MB * interleave 
> ways rule. Furthermore, Linux should be able to match the Endpoint decoders 
> with already constructed CXL Regions and allow the attachment process to 
> succeed. 
> 

You may also need to describe more than just the contents of the
endpoint decoder.  What would the content of any intermediate decoders
be (matching the root or matching the endpoint?).

> If this explanation suffices, I will incorporate it into the next version
> of this patch and also explain that "HDM Decoder" stands for Endpoint Decoder 
> and that the CFMWS HPA base/size describes the System Physical Address (SPA) 
> which the CXL driver uses to make Root Decoders HPA range base/size. 
> 

This explanation is better, just need a few more bits of data and I
think you're good to go.

~Gregory

Re: [PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS, LMH, ED

[Ira Weiny]

Gregory Price wrote:
> On Thu, Jul 17, 2025 at 04:14:13PM +0200, Fabio M. De Francesco wrote:
> > The table above shows a real configuration copied from an x86 platform 
> > where the Low Memory Hole (LMH) starts at 2GB. 
> > 
> > The"HDM Decoder Base/Size" refers specifically to the CXL Endpoint 
> > Decoders HPA range Base/Size. The first row of the table describes the 
> > first window (CFMWS[0]), whose HPA rage base/size is 0/2GB, and the 
> > Endpoint Decoder that the CXL driver should match with that CFMWS, 
> > whose HPA range base/size is 0/3GB.
> 
> The only thing i ask is being more precise with decoder references.
> 
> HDM Decoder can refer to any of: root, switch, hb, or endpoint decoders.
> 
> Below you make this distinct in the explanation, but in the table it's
> simply general "HDM Decoder".  All I ask is for a bit more clarity on
> what decoder will contain what values to avoid further ambiguity.
> 
> > The driver expects that the Endpoint Decoders HPA ranges to be contained 
> > into their corresponding Root Decoders. Furthermore, Linux fails to 
> > attach Endpoint decoders to already constructed CXL Regions because of 
> > the same size discrepancy issue. 
> > > 
> > > I think you need to describe what the expected behavior is for what linux
> > > will produce in terms of the decoder objects given the above.
> > >
> > The expected behavior is that Linux should be able to match the Endpoint 
> > Decoder with the Root Decoder range even if the CFMWS size is smaller 
> > than the Decoder's, as long as the latter adheres to the 256MB * interleave 
> > ways rule. Furthermore, Linux should be able to match the Endpoint decoders 
> > with already constructed CXL Regions and allow the attachment process to 
> > succeed. 
> > 
> 
> You may also need to describe more than just the contents of the
> endpoint decoder.  What would the content of any intermediate decoders
> be (matching the root or matching the endpoint?).

I wonder if the explanation is lacking intermediate decoders because the
test system does not have them?

Is this true Fabio?

Regardless I think Gregory is correct here.  We should try and document
the general case.

Ira

> 
> > If this explanation suffices, I will incorporate it into the next version
> > of this patch and also explain that "HDM Decoder" stands for Endpoint Decoder 
> > and that the CFMWS HPA base/size describes the System Physical Address (SPA) 
> > which the CXL driver uses to make Root Decoders HPA range base/size. 
> > 
> 
> This explanation is better, just need a few more bits of data and I
> think you're good to go.
> 
> ~Gregory

Re: [PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS, LMH, ED

[Fabio M. De Francesco]

On Monday, July 21, 2025 2:51:14 AM Central European Summer Time Gregory Price wrote:
> On Thu, Jul 17, 2025 at 04:14:13PM +0200, Fabio M. De Francesco wrote:
> > The table above shows a real configuration copied from an x86 platform 
> > where the Low Memory Hole (LMH) starts at 2GB. 
> > 
> > The"HDM Decoder Base/Size" refers specifically to the CXL Endpoint 
> > Decoders HPA range Base/Size. The first row of the table describes the 
> > first window (CFMWS[0]), whose HPA rage base/size is 0/2GB, and the 
> > Endpoint Decoder that the CXL driver should match with that CFMWS, 
> > whose HPA range base/size is 0/3GB.
> 
> The only thing i ask is being more precise with decoder references.
> 
> HDM Decoder can refer to any of: root, switch, hb, or endpoint decoders.
> 
> Below you make this distinct in the explanation, but in the table it's
> simply general "HDM Decoder".  All I ask is for a bit more clarity on
> what decoder will contain what values to avoid further ambiguity.
> 
I agree, "HDM Decoder" was ambiguous. I will relabel it to "(intermediate) 
Switch and Endpoint Decoders".
>
> > The driver expects that the Endpoint Decoders HPA ranges to be contained 
> > into their corresponding Root Decoders. Furthermore, Linux fails to 
> > attach Endpoint decoders to already constructed CXL Regions because of 
> > the same size discrepancy issue. 
> > > 
> > > I think you need to describe what the expected behavior is for what linux
> > > will produce in terms of the decoder objects given the above.
> > >
> > The expected behavior is that Linux should be able to match the Endpoint 
> > Decoder with the Root Decoder range even if the CFMWS size is smaller 
> > than the Decoder's, as long as the latter adheres to the 256MB * interleave 
> > ways rule. Furthermore, Linux should be able to match the Endpoint decoders 
> > with already constructed CXL Regions and allow the attachment process to 
> > succeed. 
> > 
> 
> You may also need to describe more than just the contents of the
> endpoint decoder.  What would the content of any intermediate decoders
> be (matching the root or matching the endpoint?).
>
I think that the output of the simulation of a hole in the cxl_test 
mocked CXL topology, with a CFMWS HPA size of 3 * 256MiB, and Switch and 
Endpoint Decoders HPA sizes of 4 * 256MiB can help to clarify. The Region
Interleave ways is 2. 

In this example the CXL driver is patched and correctly deals with a 
simulated LMH that trimmed the CFMWS[0] range size and made it not aligned
to 256MiB * IW.

[root@fedora ndctl]# cxl list -RDu -r5
[
  {
    "root decoders":[
      {
        "decoder":"decoder9.0",
        "resource":"0x3ff010000000",
        "size":"768.00 MiB (805.31 MB)",
        "interleave_ways":1,
        "max_available_extent":0,
        "volatile_capable":true,
        "qos_class":42,
        "nr_targets":1,
        "regions:decoder9.0":[
          {
            "region":"region5",
            "resource":"0x3ff010000000",
            "size":"768.00 MiB (805.31 MB)",
            "type":"ram",
            "interleave_ways":2,
            "interleave_granularity":4096,
            "decode_state":"commit"
          }
        ]
      }
    ]
  },
  {
    "port decoders":[
      {
        "decoder":"decoder10.0",
        "resource":"0x3ff010000000",
        "size":"1024.00 MiB (1073.74 MB)",
        "interleave_ways":1,
        "region":"region5",
        "nr_targets":1
      },
      {
        "decoder":"decoder14.0",
        "resource":"0x3ff010000000",
        "size":"1024.00 MiB (1073.74 MB)",
        "interleave_ways":2,
        "interleave_granularity":4096,
        "region":"region5",
        "nr_targets":2
      }
    ]
  },
  {
    "endpoint decoders":[
      {
        "decoder":"decoder28.0",
        "resource":"0x3ff010000000",
        "size":"1024.00 MiB (1073.74 MB)",
        "interleave_ways":2,
        "interleave_granularity":4096,
        "region":"region5",
        "dpa_resource":"0",
        "dpa_size":"384.00 MiB (402.65 MB)",
        "mode":"ram"
      },
      {
        "decoder":"decoder23.0",
        "resource":"0x3ff010000000",
        "size":"1024.00 MiB (1073.74 MB)",
        "interleave_ways":2,
        "interleave_granularity":4096,
        "region":"region5",
        "dpa_resource":"0",
        "dpa_size":"384.00 MiB (402.65 MB)",
        "mode":"ram"
      }
    ]
  }
]

The construction of Regions and subsequent attachment of Switch and 
Endpoint Decoders is based on matching Root Decoders and existing Regions
with Switch and Endpoint Decoders. If these objects can't be matched 
between them, the Regions can't be constructed and/or attached with 
the Switch and Endpoint Decoders.     

The CXL driver can always match Endpoint Decoders with Switch Decoders 
because their HPA range sizes are the same regardless of LMH's.  

But with LMH's that trim the CFMWS HPA range to smaller sizes, a 
non-patched CXL driver can't match the Root Decoders and Regions with the 
Switch and Endpoint decoders. Therefore, the CXL Region construction and 
Decoders attachment can't succeed.
      }
    ]
  }
]

The construction of Regions and subsequent attachment of Switch and 
Endpoint Decoders is based on matching Root Decoders and existing Regions
with Switch and Endpoint Decoders. If these objects can't be matched 
between them, the Regions can't be constructed and/or attached with 
the Switch and Endpoint Decoders.     

Anyway, the CXL driver can always match Endpoint Decoders with Switch
Decoders because the HPA range sizes are the same also when LMH's are  

But with an LMH that trim the CFMWS HPA range size to a smaller size, a 
non-patched CXL driver can't match the Root Decoders and Regions with the 
Switch and Endpoint decoders. Therefore, the CXL Region construction and 
Decoders attachment can't succeed.
>
> > If this explanation suffices, I will incorporate it into the next version
> > of this patch and also explain that "HDM Decoder" stands for Endpoint Decoder 
> > and that the CFMWS HPA base/size describes the System Physical Address (SPA) 
> > which the CXL driver uses to make Root Decoders HPA range base/size. 
> > 
> 
> This explanation is better, just need a few more bits of data and I
> think you're good to go.
> 
Thanks,

Fabio

Fabio
>
> ~Gregory
> 

Re: [PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS, LMH, ED

[Jonathan Cameron]

On Mon, 23 Jun 2025 17:29:02 +0200
"Fabio M. De Francesco" <fabio.m.de.francesco@linux.intel.com> wrote:

> Add documentation on how to resolve conflicts between CXL Fixed Memory
> Windows, Platform Memory Holes, and Endpoint Decoders.
> 
> Signed-off-by: Fabio M. De Francesco <fabio.m.de.francesco@linux.intel.com>
> ---
> 
> v2 -> v3: Rework a few phrases for better clarity.
> 	  Fix grammar and syntactic errors (Randy, Alok).
> 	  Fix semantic errors ("size does not comply", Alok).
> 	  Fix technical errors ("decoder's total memory?", Alok).
> 	  
> v1 -> v2: Rewrite "Summary of the Change" section, 3r paragraph.
> 
>  Documentation/driver-api/cxl/conventions.rst | 85 ++++++++++++++++++++
>  1 file changed, 85 insertions(+)
> 
> diff --git a/Documentation/driver-api/cxl/conventions.rst b/Documentation/driver-api/cxl/conventions.rst
> index da347a81a237..d6c8f4cf2f5b 100644
> --- a/Documentation/driver-api/cxl/conventions.rst
> +++ b/Documentation/driver-api/cxl/conventions.rst
> @@ -45,3 +45,88 @@ Detailed Description of the Change
>  ----------------------------------
>  
>  <Propose spec language that corrects the conflict.>
> +
> +
> +Resolve conflict between CFMWS, Platform Memory Holes, and Endpoint Decoders
> +============================================================================
> +
> +Document
> +--------
> +
> +CXL Revision 3.2, Version 1.0
> +
> +License
> +-------
> +
> +SPDX-License Identifier: CC-BY-4.0
> +
> +Creator/Contributors
> +--------------------
> +
> +Fabio M. De Francesco, Intel
> +Dan J. Williams, Intel
> +Mahesh Natu, Intel
> +
> +Summary of the Change
> +---------------------
> +
> +According to the current CXL Specifications (Revision 3.2, Version 1.0)
> +the CXL Fixed Memory Window Structure (CFMWS) describes zero or more Host
> +Physical Address (HPA) windows that are associated with each CXL Host
> +Bridge. Each window represents a contiguous HPA range that may be
> +interleaved across one or more targets, some of which are CXL Host Bridges.
> +Associated with each window is a set of restrictions that govern its usage.
> +It is the OSPM’s responsibility to utilize each window for the specified
> +use.
> +
> +Table 9-22 states the Window Size field contains the total number of
> +consecutive bytes of HPA this window represents and this value shall be a
> +multiple of Number of Interleave Ways * 256 MB.
> +
> +Platform Firmware (BIOS) might reserve part of physical addresses below
> +4 GB (e.g., the Low Memory Hole that describes PCIe memory space for MMIO
> +or a requirement for the greater than 8 way interleave CXL regions starting
> +at address 0). In that case the Window Size value cannot be anymore
> +constrained to the NIW * 256 MB above-mentioned rule.

I'm not following argument for large interleave at address 0 being a problem
(if we ignore the low memory hole and similar as a separate issue).  Even
if it is the interaction with the low memory hole, is 12 way interleave
of 256MiB devices a problem?  Fills up to 3GiB.


> +
> +On those systems, BIOS publishes CFMWS which communicate the active System
> +Physical Address (SPA) ranges that map to a subset of the Host Physical
> +Address (HPA) ranges. The SPA range trims out the hole, and capacity in the
> +endpoint is lost with no SPA to map to CXL HPA in that hole.
> +
> +The description of the Window Size field in table 9-22 needs to take that
> +special case into account.
> +
> +Note that the Endpoint Decoders HPA range sizes have to comply with the
> +alignment constraints and so a part of their memory capacity might not be
> +accessible if their size exceeds the matching CFMWS range's.
> +
> +Benefits of the Change
> +----------------------
> +
> +Without this change, the OSPM wouldn't match Endpoint Decoders with CFMWS
> +whose Window Size don't fit the alignment constraints and so the memdev
> +capacity would be lost. This change allows the OSPM to match Endpoint
> +Decoders whose HPA range size exceeds the matching CFMWS and create
> +regions that at least utilize a part of the memory devices total capacity.
> +
> +References
> +----------
> +
> +Compute Express Link Specification Revision 3.2, Version 1.0
> +<https://www.computeexpresslink.org/>
> +
> +Detailed Description of the Change
> +----------------------------------
> +
> +The current description of a CFMWS Window Size (Table 9-22) is replaced
> +with:
> +
> +"The total number of consecutive bytes of HPA this window represents. This
> +value shall be a multiple of NIW*256 MB. On platforms that reserve physical
> +addresses below 4 GB for special use (e.g., the Low Memory Hole for PCIe
> +MMIO on x86), an instance of CFMWS whose Base HPA is 0 might have a window
> +size that doesn't align with the NIW*256 MB constraint; note that the
> +matching Endpoint Decoders HPA range size must still align to the
> +above-mentioned rule and so the memory capacity that might exceed the CFMWS
> +window size will not be accessible.".
> 
> base-commit: a021802c18c4c30dff3db9bd355cacb68521f1aa

Re: [PATCH v3] cxl: docs/driver-api/conventions resolve conflicts btw CFMWS, LMH, ED

[Fabio M. De Francesco]

On Tuesday, July 1, 2025 3:17:47 PM Central European Summer Time Jonathan Cameron wrote:
> On Mon, 23 Jun 2025 17:29:02 +0200
> "Fabio M. De Francesco" <fabio.m.de.francesco@linux.intel.com> wrote:
> 
> > Add documentation on how to resolve conflicts between CXL Fixed Memory
> > Windows, Platform Memory Holes, and Endpoint Decoders.
> > 
> > Signed-off-by: Fabio M. De Francesco <fabio.m.de.francesco@linux.intel.com>
> > ---
> > 
> > v2 -> v3: Rework a few phrases for better clarity.
> > 	  Fix grammar and syntactic errors (Randy, Alok).
> > 	  Fix semantic errors ("size does not comply", Alok).
> > 	  Fix technical errors ("decoder's total memory?", Alok).
> > 	  
> > v1 -> v2: Rewrite "Summary of the Change" section, 3r paragraph.
> > 
> >  Documentation/driver-api/cxl/conventions.rst | 85 ++++++++++++++++++++
> >  1 file changed, 85 insertions(+)
> > 
> > diff --git a/Documentation/driver-api/cxl/conventions.rst b/Documentation/driver-api/cxl/conventions.rst
> > index da347a81a237..d6c8f4cf2f5b 100644
> > --- a/Documentation/driver-api/cxl/conventions.rst
> > +++ b/Documentation/driver-api/cxl/conventions.rst
> > @@ -45,3 +45,88 @@ Detailed Description of the Change
> >  ----------------------------------
> >  
> >  <Propose spec language that corrects the conflict.>
> > +
> > +
> > +Resolve conflict between CFMWS, Platform Memory Holes, and Endpoint Decoders
> > +============================================================================
> > +
> > +Document
> > +--------
> > +
> > +CXL Revision 3.2, Version 1.0
> > +
> > +License
> > +-------
> > +
> > +SPDX-License Identifier: CC-BY-4.0
> > +
> > +Creator/Contributors
> > +--------------------
> > +
> > +Fabio M. De Francesco, Intel
> > +Dan J. Williams, Intel
> > +Mahesh Natu, Intel
> > +
> > +Summary of the Change
> > +---------------------
> > +
> > +According to the current CXL Specifications (Revision 3.2, Version 1.0)
> > +the CXL Fixed Memory Window Structure (CFMWS) describes zero or more Host
> > +Physical Address (HPA) windows that are associated with each CXL Host
> > +Bridge. Each window represents a contiguous HPA range that may be
> > +interleaved across one or more targets, some of which are CXL Host Bridges.
> > +Associated with each window is a set of restrictions that govern its usage.
> > +It is the OSPM’s responsibility to utilize each window for the specified
> > +use.
> > +
> > +Table 9-22 states the Window Size field contains the total number of
> > +consecutive bytes of HPA this window represents and this value shall be a
> > +multiple of Number of Interleave Ways * 256 MB.
> > +
> > +Platform Firmware (BIOS) might reserve part of physical addresses below
> > +4 GB (e.g., the Low Memory Hole that describes PCIe memory space for MMIO
> > +or a requirement for the greater than 8 way interleave CXL regions starting
> > +at address 0). In that case the Window Size value cannot be anymore
> > +constrained to the NIW * 256 MB above-mentioned rule.
> 
> I'm not following argument for large interleave at address 0 being a problem
> (if we ignore the low memory hole and similar as a separate issue). 
>
> Even
> if it is the interaction with the low memory hole, is 12 way interleave
> of 256MiB devices a problem?  Fills up to 3GiB. 
> 
Right, I'll drop that argument for large interleaves. The problem is still the 
MMIO hole that might intersect the 3 GiB requirement for 12 way interleave.

Thanks,

Fabio