Re: [PATCH v4 6/6] Documentation/cxl: Describe mixed-granularity regions

[Alison Schofield <alison.schofield@intel.com>]

On Fri, Aug 21, 2026 at 09:35:27PM +0100, Jonathan Cameron wrote:
> On Thu, 20 Aug 2026 16:31:24 -0700
> Alison Schofield <alison.schofield@intel.com> wrote:
> 
> Hi Alison,
> 
> Can we pull this up to be at top of patch set?  People really need
> to read this first, so make that easy!

In v5 it's Patch 1. 

> 
> > Mixed-granularity region support introduces interleave relationships
> > that are not obvious from the existing CXL documentation. This is
> > particularly true for the 3-, 6-, and 12-way configurations and for
> > understanding which configurations permitted by the CXL Specification
> > are supported by Linux.
> > 
> > Document the mixed-granularity model and Linux's coarse-to-fine
> > restriction. Include the relevant configurations from CXL 4.0 Section
> > 9.13.1.1 and annotate their Linux support status.
> > 
> > This intentionally repeats information from the CXL Specification.
> > The specification remains authoritative, but showing the Linux support
> > policy alongside the legal configurations ensures readers do not
> > mistake an unsupported Linux setup for a configuration unsupported by
> > the CXL Specification.
> > 
> > Assisted-by: Claude:Opus-5
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> 
> My biggest queries are:
> - Naming.  bike shed time :)  I think this evolved to point where 'mixed'
>   no longer describes what is being built. I'm not sure what is mixed.
> 
> - Are we sure people are doing coarse to fine?
>   They might be - I remember a discussion with Dan way back where I was
>   arguing that was the natural way round, but he convinced me that fine
>   as fast as possible made more sense, as about spreading larger hotspots
>   and linear accesses onto as many paths as possible as quickly as
>   possible.
> 
>   Given the reason to do this is either a hardware restriction, or non
>   power of 2, going fine as fast as possible may still make sense.
> 
> Honestly I don't (I think) have any skin in the game here so if this
> works for you I am fine with restricting things - as long as we make
> it even clearer what is going on!


I tried to make both of these clearer in v5. In particular, the
mixed-granularity section now defines what is "mixed" up front, and
clarifies that refining toward the endpoints is not a preference over
Cross-Link First. More on both below where you called them out in the
document.

> 
> > ---
> >  .../driver-api/cxl/linux/cxl-driver.rst       | 135 ++++++++++++++++++
> >  1 file changed, 135 insertions(+)
> > 
> > diff --git a/Documentation/driver-api/cxl/linux/cxl-driver.rst b/Documentation/driver-api/cxl/linux/cxl-driver.rst
> > index dd6dd17dc536..4e56c18294ef 100644
> > --- a/Documentation/driver-api/cxl/linux/cxl-driver.rst
> > +++ b/Documentation/driver-api/cxl/linux/cxl-driver.rst
> > @@ -602,6 +602,11 @@ derived from their upstream port connections. In `Cross-Link First` interleave
> >  configurations, the :code:`interleave_granularity` of a decoder is equal to
> >  :code:`parent_interleave_granularity * parent_interleave_ways`.
> >  
> > +When the region granularity is finer than the granularity of an interleaving
> > +root decoder, the relation inverts: the :code:`interleave_granularity` of a
> > +decoder is equal to :code:`parent_interleave_granularity / interleave_ways`.
> > +See `Mixed Granularity`_.
> > +
> >  At Endpoint
> >  ~~~~~~~~~~~
> >  `Endpoint Decoders` are programmed similar to Host Bridge and Switch decoders,
> > @@ -619,6 +624,136 @@ from HPA to DPA.  This is why they must be aware of the entire interleave set.
> >  Linux does not support unbalanced interleave configurations.  As a result, all
> >  endpoints in an interleave set must have the same ways and granularity.
> >  
> > +Mixed Granularity
> > +~~~~~~~~~~~~~~~~~
> 
> My main question here is why are we calling them mixed?
> From that name I was assuming we were doing
> 
> Root 4K
> HB   1K
> Sw   2K
> 
> Where the granularity isn't monotonic.


I kept "mixed-granularity", but reworked the opening to define it before
using it. In this series, the mix is between the root and region
granularities, not arbitrary changes in granularity down the hierarchy.

The new text starts with:

Linux has required that a region's interleave_granularity equal the
interleave_granularity of its interleaving root decoder. A
mixed-granularity region lifts that restriction. The region granularity
may be finer than the root's, so the interleave refines from the root
toward the endpoints. The mix is between the root and the region, not
granularity varying arbitrarily down the hierarchy.

Please take a look and see if that makes the terminology clearer.

> 
> > +Every decoder advances one target every multiple of its own granularity, and
> > +the decoders below it subdivide the span their parent assigns to a single
> > +target.  Linux supports two orderings of granularity down the hierarchy.
> > +
> > +The `Cross-Link First` example above shows the first ordering, where the region
> > +granularity equals the granularity of the root decoder and granularity coarsens
> > +toward the endpoints.  In the second ordering the region granularity is finer
> > +than the root decoder's and granularity refines toward the endpoints, reaching
> > +the region granularity at the innermost interleaving decoder.  A region using
> > +that ordering is a *mixed-granularity* region.  A mixed-granularity region
> > +requires an interleaving root decoder.

I also reworked this part to address the coarse-to-fine question above.

Cross-Link First remains the fine-as-early-as-possible case and is
unchanged by this series. Refining toward the endpoints applies when the
region granularity is finer than the root granularity; it isn't intended
as a preferred alternative to Cross-Link First.

The new text makes that explicit:

Refining is not a preference. Once the region granularity is finer than
the root's, Linux gives each level below the root a single granularity,
its parent's divided by its own ways, so the ordering follows from the
region and root settings rather than being chosen. [...] Reaching a
fine granularity as early as possible remains available through
Cross-Link First, which this does not change.

The restriction is that Linux does not support a hierarchy that refines
and then coarsens again.


> > +
> > +Linux supports only monotonic granularity hierarchies, either coarsening or
> > +refining from the root toward the endpoints.  The CXL Specification does not
> > +require a monotonic ordering, see `Mod3 Interleave Configurations`_.
> > +
> > +For an 8-way mixed-granularity region below a 2-way interleaving root decoder
> > +at 4096, where each host bridge routes through two levels of switch, Linux
> > +programs::
> > +
> > +  Level             Ways    Granularity
> > +  -----             ----    -----------
> > +  Root                 2         4096
> > +  Host bridge          1         4096
> > +  Upper switch         2         2048
> > +  Lower switch         2         1024
> > +  Endpoint             8         1024
> 
> Given multi switch restrictions, why not just do one level and make the
> host bridge do 2 way interleave (to two RPs each of which has a switch below)
> 
> I don't think that changes the logic, but it reflects more standard
> CXL topology (if no PBR fun involved)

Agree. In v5 the example is changed to:

Level		Ways	Granularity
-----		----	-----------
Root		2	4096
Host Bridge	2	2048
Switch 		2	1024
Endpoint	8	1024

The cxl-test topology is changed the same way. So now the documented
and tested example are cross-HB, host bridge, then switch.

> 
> 
> > +
> > +Each decoder contributes to an endpoint's region position in proportion to its
> > +granularity::
> > +
> > +  position += target_position *
> > +              decoder_granularity / region_granularity
> > +
> > +The root above selects a host bridge every 4096 bytes, so it advances one
> > +target every four region positions, while the lower switch advances one target
> > +every position.  When the region granularity equals the root granularity, the
> > +root advances one target per region position and each level's weight is the
> > +number of ways below it.
> > +
> > +The ways and granularity of a mixed-granularity region must describe the same
> > +interleave span as the root decoder::
> > +
> > +  root_ways * root_granularity == region_ways * region_granularity
> > +
> > +A region that does not describe that span either leaves part of the range
> > +unclaimed or reaches beyond it, and Linux rejects it.  A same-granularity
> > +region below a power-of-two root decoder spans a multiple of the root's range
> > +rather than one target's share of it, and is not subject to this relationship.
> > +
> > +Mod3 Interleave Configurations
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +A 3-way, 6-way, or 12-way interleave, known as a Mod3 interleave, selects its
> > +target with a factor-of-three selection rather than from binary interleave
> > +selector bits alone.  CXL 4.0 Section 9.13.1.1 defines the 3-way selection as
> > +the address above the decoder granularity taken modulo 3.  A 6-way selection
> > +claims one binary HPA bit at the decoder granularity and takes the modulo 3 of
> > +the address above that bit, and a 12-way selection claims two.
> > +
> > +The factor-of-three selection is not an additional binary selector bit, so a
> > +Mod3 interleave distributes across the hierarchy as::
> > +
> > +  3  = 3
> > +  6  = 3 * 2
> > +  12 = 3 * 4
> > +
> > +The cross-host bridge selection carries the factor of three, and the remaining
> > +x2 or x4 is binary interleave selection below it.  A 6-way region at IGB across
> > +three host bridges is therefore::
> > +
> > +  Device-level region:   6-way @ IGB
> > +  Cross-host bridge:     3-way @ 2*IGB
> > +  Below the root:        2-way @ IGB
> > +
> > +where both levels describe the same interleave span::
> > +
> > +  3 * (2 * IGB) == 6 * IGB
> > +
> > +The CXL Specification defines the legal Mod3 compositions and is normative.
> > +CXL 4.0 Section 9.13.1.1, "Legal Interleaving Configurations: 12-way, 6-way,
> > +and 3-way", Tables 9-6, 9-7, and 9-8 list them for a 12-way, 6-way, and 3-way
> > +device-level interleave at IGB.  Those tables are summarized below, annotated
> > +with the subset Linux supports.
> > +
> > +CXL 4.0 Table 9-8, 3-way device-level interleave at IGB::
> > +
> > +  Row  Cross-host bridge  Host bridge    Switch         Linux
> > +  ---  -----------------  -----------    ------         -----
> > +   1   3-way @ IGB        none           none           supported
> > +
> > +CXL 4.0 Table 9-7, 6-way device-level interleave at IGB::
> > +
> > +  Row  Cross-host bridge  Host bridge    Switch         Linux
> > +  ---  -----------------  -----------    ------         -----
> > +   1   6-way @ IGB        none           none           supported
> > +   2   3-way @ 2*IGB      2-way @ IGB    none           supported
> > +   3   3-way @ 2*IGB      none           2-way @ IGB    supported
> > +
> > +CXL 4.0 Table 9-6, 12-way device-level interleave at IGB::
> > +
> > +  Row  Cross-host bridge  Host bridge    Switch         Linux
> > +  ---  -----------------  -----------    ------         -----
> > +   1   12-way @ IGB       none           none           supported
> > +   2   6-way @ 2*IGB      2-way @ IGB    none           supported
> > +   3   6-way @ 2*IGB      none           2-way @ IGB    supported
> > +   4   3-way @ 4*IGB      4-way @ IGB    none           supported
> > +   5   3-way @ 4*IGB      none           4-way @ IGB    supported
> > +   6   3-way @ 4*IGB      2-way @ IGB    2-way @ 2*IGB  unsupported
> > +   7   3-way @ 4*IGB      2-way @ 2*IGB  2-way @ IGB    supported
> > +
> > +Table 9-6 row 6 is legal per the CXL Specification and unsupported by Linux.
> 
> but is unsupported by Linux.
> (perhaps clearer?)

Yes, changed.

> 
> > +Walking it from the root toward the endpoints, granularity goes::
> > +
> > +  4*IGB -> IGB -> 2*IGB
> > +
> > +which refines and then coarsens.  Row 7 interleaves the same 12 endpoints at
> > +the same granularity with those two levels exchanged::
> > +
> > +  4*IGB -> 2*IGB -> IGB
> > +
> > +which is monotonic.  Linux programs row 7 for a user region and assembles an
> > +auto region whose decoders are programmed that way.  An auto region matching
> > +row 6 is not assembled.
> > +
> > +Leaving row 6 unsupported does not prevent a 12-way device-level interleave.
> > +The specification defines six other legal compositions, all monotonic and
> > +supported by Linux.
> 
> Silly question - does anyone actually care about 12 way? :)  This would all
> be much easier without it.

Yes, someone cares/cared, that's why I started this 6/12 way support.

I don't think dropping 12-way simplifies the implementation much. The
factor-of-three handling is already needed for 3-way and 6-way, and 12-way
uses the same handling with a wider power-of-two interleave below it.

I also think the 12-way example is useful here. Table 9-6 has the
non-monotonic layout Linux does not support (row 6) alongside the
monotonic arrangement it does support (row 7). That makes the Linux
restriction fairly concrete.

I added a third CFMWS to the cxl_test topology so the supported 12-way
case is covered as well.

-- Alison

> 
> > +
> >  Example Configurations
> >  ======================
> >  .. toctree::
> 
>