[PATCH v5 0/7] cxl: Support mixed-granularity region interleaves

[Alison Schofield <alison.schofield@intel.com>]

JonathanC, I kept your tags on Patches 2,5.

RobertR, Same comment as in v4. I've dropped the Originally-by tags since the
implementation no longer carries the selector-bit approach from your original
patch. Let me know what authorship tags you'd like.


Changes in v5:
Series wide:
- Rebase onto 7.3-rc1
- Move the documentation patch to the front of the series (Jonathan)
- Reword commit messages to use new terms of documentation (Jonathan)
- Wrap code and code comments closer to 80 columns (Jonathan)
- Rewrap all commit messages to 75 columns (Jonathan)
- Split combined declarations (Jonathan)
Patch 1: Documentation/cxl: Describe mixed-granularity regions
- Define a mixed-granularity region and monotonic requirement (Jonathan)
- Define span, region position, and weight (Jonathan)
- Explain refining granularity and cross-link first (Jonathan)
- Use an 8-way example interleaved at the host bridge w one switch (Jonathan)
- Clarify config 'legal per the CXL Spec but unsupported by Linux' (Jonathan)
Patch 2: cxl/region: Warn on user region position mismatch
- Log the position mismatch against the region device and name the decoder
Patch 3: cxl/region: Generalize endpoint position mapping
- Pass the region interleave granularity into cxl_calc_interleave_pos() instead
  of using the endpoint decoder granularity (RichardC)
- Rewrite the example comment to define a region position and each decoder
  level's weight, rather than what it spans (Jonathan)
New Patch 4: cxl/region: Name the interleave locals in cxl_port_setup_targets()
- Rename iw and ig to child_iw and child_ig (Jonathan)
Patch 5: cxl/region: Support mixed-granularity auto regions
- Restore the topology-derived peer distance, using 1 only for mixed-granularity
  regions, avoiding a NULL deref on the auto path
- Reject a layout whose derived granularity comes out finer than the region
  granularity, which the topology-derived distance used to catch
- Define what a span is where the root and region spans are compared (Jonathan)
- Add comment that the power-of-two exemption is for same-granularity regions
  that span multiple root interleaves, and that Mod3 roots are width-matched
  instead (Jonathan, Sashiko)
- Report both spans in the span-check dev_dbg
Patch 6: cxl/region: Support mixed-granularity user created regions
- Add a dev_dbg naming both granularities to the coarser-than-root rejection
Patch 7: cxl/test: Add a topology to test mixed-granularity regions
- Rebuild the topology as cross-HB, then HB, then switch (Jonathan)
- Add a third window to support a 12-way region at 256
- Drop the cascade of two switch levels (Jonathan)
- Replace bare '2' with NR_CXL_MIX_GRAN_RPS_PER_HB
- Create and use a chbs_range() helper
Link to v4:
https://lore.kernel.org/linux-cxl/cover.1787255388.git.alison.schofield@intel.com/

Changes in v4:
- Rework port decoder setup around parent granularity and target count,
  dropping the selector-walk and selector-derived granularity machinery
  from v3 (RobertR)
- Replace the mixed-granularity-specific endpoint position handling with
  one weighted position calculation for existing and mixed-gran regions
- Derive mixed-gran port decoder settings directly from the parent interleave
  geometry
- Promote the user-region position self-test from dev_dbg() to dev_warn()
- Update the documentation patch
- Update the cover letter summary and series structure
Link to v3:
https://lore.kernel.org/linux-cxl/cover.1785444498.git.alison.schofield@intel.com/

Changes in v3:
- Reduce scope to coarse-to-fine ordering (RichardC, Sashiko)
- Reject out-of-order layouts (RichardC, Sashiko)
- P1: Factor the selector walk and per-port checks into helpers (RobertR)
  That pre-work is the new Patch 1/8 and the Series Structure section of
  this cover letter describe same below.
- P3: Derive granularity from the highest available selector span (RobertR)
- P3: Validate auto-programmed granularity against the derived value
- P4: Rename root_pos_stride() to root_positions_per_target() (RobertR)
- P4: Drop the "stride" terminology throughout (RobertR)
- The complex Patch 4 of v2 is split into patches 4-6 in v3.
  Also noted in the Series Structure section of cover letter below.
- P9: Move the peer-distance walk and per-function walkthroughs from the doc
  into in-code comments and kernel-doc (RobertR)
- Drop Reviewed-by tags
Link to v2:
https://lore.kernel.org/linux-cxl/cover.1781199122.git.alison.schofield@intel.com/

Changes in v2:
- Patch 1,2: Defer the unused selector var store to keep P1 bisectable (Sashiko)
- Patch 1: Make divide by 3 in get_selctor() work on 32-bit builds) (lkp)
- Patch 4: Use local vars in cxl_region_attach() for readability (DaveJ)
- Patch 5: Add NULL checks on unused mock arrays (Sashiko)
- Resolved errant err_rch unwind with rc7 merge (DaveJ)
- Rebase onto 7.1-rc7
- Update commit logs in 1,2,5 to align w changes in v2
Link to v1:
https://lore.kernel.org/all/cover.1780095671.git.alison.schofield@intel.com/


Begin Cover Letter:

Linux has required that a region's interleave granularity equal the interleave
granularity of its interleaving root decoder. A mixed-granularity region lifts
that restriction: the region granularity may be finer than the root's, so the
interleave refines from the root toward the endpoints. The mix is between the
root and the region, not granularity varying arbitrarily down the hierarchy.

A CXL region may interleave across multiple decoder levels: root, optional
switches, and endpoint. The equal-granularity requirement blocks legal
mixed-granularity arrangements permitted by CXL 4.0 Section 9.13.1 and makes
some 6-way and 12-way configurations defined in Section 9.13.1.1 (Tables 9-6,
9-7) impossible to create.

Two prior proposals addressed parts of this gap:

AlisonS introduced position arithmetic and sysfs gating for auto and
user-created regions to support the 6- and 12-way interleave configurations
without a same-granularity alternative:
https://lore.kernel.org/all/20250306232239.2609017-1-alison.schofield@intel.com/

RobertR introduced an HPA selector-bit model for auto regions that allows
multi-level power-of-two interleaves regardless of granularity ordering:
https://lore.kernel.org/all/20251028094754.72816-1-rrichter@amd.com/

This series adds support for CXL regions where interleave granularity differs
between levels of the decoder hierarchy. The support applies to both firmware
programmed auto regions and user created regions.

Linux supports mixed-granularity configurations that are monotonic from the
interleaving root toward the endpoints. Granularity may remain the same or
become finer at each interleaving level. Configurations that refine and then
become coarser are permitted by the CXL Specification but are not supported
by Linux.

This support covers power-of-two and Mod3 interleaves at any depth. Of the Mod3
configs listed in CXL 4.0 Section 9.13.1.1, all are supported except Table 9-6
row 6, the refine-then-coarsen 12-way composition. See the Patch 1 documentation
update for additional info.


Series structure in v5:
-----------------------
Patch 1 documents the mixed-granularity model and Linux support policy.
It comes first so that the terms the rest of the series uses - span,
region position, weight - are defined before they are used.

Patch 2 promotes the existing user-region endpoint position self-test to
a visible diagnostic.

Patch 3 generalizes endpoint position calculation to account for
granularity changes between decoder levels.

Patch 4 names the interleave locals in cxl_port_setup_targets(). It is a
mechanical rename, carried separately so it does not muddy Patch 5.

Patch 5 enables mixed-granularity auto regions and validates their
coarse-to-fine decoder geometry.

Patch 6 enables mixed-granularity user created regions by relaxing the
existing root/region granularity restriction.

Patch 7 adds a cxl_test topology and coverage for mixed-granularity
regions.


The companion NDCTL patchset that allows mixed-granularity 'cxl create-region'
and adds the unit test is posted here:
https://lore.kernel.org/nvdimm/cover.1788469004.git.alison.schofield@intel.com/


Alison Schofield (7):
  Documentation/cxl: Describe mixed-granularity regions
  cxl/region: Warn on user region position mismatch
  cxl/region: Generalize endpoint position mapping
  cxl/region: Name the interleave locals in cxl_port_setup_targets()
  cxl/region: Support mixed-granularity auto regions
  cxl/region: Support mixed-granularity user created regions
  cxl/test: Add a topology to test mixed-granularity regions

 .../driver-api/cxl/linux/cxl-driver.rst       | 153 +++++++
 drivers/cxl/core/region.c                     | 247 +++++++----
 tools/testing/cxl/test/cxl.c                  | 410 +++++++++++++++++-
 3 files changed, 698 insertions(+), 112 deletions(-)


base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
-- 
2.37.3