Re: [PATCH v5 0/7] cxl: Support mixed-granularity region interleaves

[Dave Jiang <dave.jiang@intel.com>]

On 9/3/26 4:23 PM, Alison Schofield wrote:
> 
> JonathanC, I kept your tags on Patches 2,5.
> 
> RobertR, Same comment as in v4. I've dropped the Originally-by tags since the
> implementation no longer carries the selector-bit approach from your original
> patch. Let me know what authorship tags you'd like.
> 
> 
> Changes in v5:
> Series wide:
> - Rebase onto 7.3-rc1
> - Move the documentation patch to the front of the series (Jonathan)
> - Reword commit messages to use new terms of documentation (Jonathan)
> - Wrap code and code comments closer to 80 columns (Jonathan)
> - Rewrap all commit messages to 75 columns (Jonathan)
> - Split combined declarations (Jonathan)
> Patch 1: Documentation/cxl: Describe mixed-granularity regions
> - Define a mixed-granularity region and monotonic requirement (Jonathan)
> - Define span, region position, and weight (Jonathan)
> - Explain refining granularity and cross-link first (Jonathan)
> - Use an 8-way example interleaved at the host bridge w one switch (Jonathan)
> - Clarify config 'legal per the CXL Spec but unsupported by Linux' (Jonathan)
> Patch 2: cxl/region: Warn on user region position mismatch
> - Log the position mismatch against the region device and name the decoder
> Patch 3: cxl/region: Generalize endpoint position mapping
> - Pass the region interleave granularity into cxl_calc_interleave_pos() instead
>   of using the endpoint decoder granularity (RichardC)
> - Rewrite the example comment to define a region position and each decoder
>   level's weight, rather than what it spans (Jonathan)
> New Patch 4: cxl/region: Name the interleave locals in cxl_port_setup_targets()
> - Rename iw and ig to child_iw and child_ig (Jonathan)
> Patch 5: cxl/region: Support mixed-granularity auto regions
> - Restore the topology-derived peer distance, using 1 only for mixed-granularity
>   regions, avoiding a NULL deref on the auto path
> - Reject a layout whose derived granularity comes out finer than the region
>   granularity, which the topology-derived distance used to catch
> - Define what a span is where the root and region spans are compared (Jonathan)
> - Add comment that the power-of-two exemption is for same-granularity regions
>   that span multiple root interleaves, and that Mod3 roots are width-matched
>   instead (Jonathan, Sashiko)
> - Report both spans in the span-check dev_dbg
> Patch 6: cxl/region: Support mixed-granularity user created regions
> - Add a dev_dbg naming both granularities to the coarser-than-root rejection
> Patch 7: cxl/test: Add a topology to test mixed-granularity regions
> - Rebuild the topology as cross-HB, then HB, then switch (Jonathan)
> - Add a third window to support a 12-way region at 256
> - Drop the cascade of two switch levels (Jonathan)
> - Replace bare '2' with NR_CXL_MIX_GRAN_RPS_PER_HB
> - Create and use a chbs_range() helper
> Link to v4:
> https://lore.kernel.org/linux-cxl/cover.1787255388.git.alison.schofield@intel.com/
> 
> Changes in v4:
> - Rework port decoder setup around parent granularity and target count,
>   dropping the selector-walk and selector-derived granularity machinery
>   from v3 (RobertR)
> - Replace the mixed-granularity-specific endpoint position handling with
>   one weighted position calculation for existing and mixed-gran regions
> - Derive mixed-gran port decoder settings directly from the parent interleave
>   geometry
> - Promote the user-region position self-test from dev_dbg() to dev_warn()
> - Update the documentation patch
> - Update the cover letter summary and series structure
> Link to v3:
> https://lore.kernel.org/linux-cxl/cover.1785444498.git.alison.schofield@intel.com/
> 
> Changes in v3:
> - Reduce scope to coarse-to-fine ordering (RichardC, Sashiko)
> - Reject out-of-order layouts (RichardC, Sashiko)
> - P1: Factor the selector walk and per-port checks into helpers (RobertR)
>   That pre-work is the new Patch 1/8 and the Series Structure section of
>   this cover letter describe same below.
> - P3: Derive granularity from the highest available selector span (RobertR)
> - P3: Validate auto-programmed granularity against the derived value
> - P4: Rename root_pos_stride() to root_positions_per_target() (RobertR)
> - P4: Drop the "stride" terminology throughout (RobertR)
> - The complex Patch 4 of v2 is split into patches 4-6 in v3.
>   Also noted in the Series Structure section of cover letter below.
> - P9: Move the peer-distance walk and per-function walkthroughs from the doc
>   into in-code comments and kernel-doc (RobertR)
> - Drop Reviewed-by tags
> Link to v2:
> https://lore.kernel.org/linux-cxl/cover.1781199122.git.alison.schofield@intel.com/
> 
> Changes in v2:
> - Patch 1,2: Defer the unused selector var store to keep P1 bisectable (Sashiko)
> - Patch 1: Make divide by 3 in get_selctor() work on 32-bit builds) (lkp)
> - Patch 4: Use local vars in cxl_region_attach() for readability (DaveJ)
> - Patch 5: Add NULL checks on unused mock arrays (Sashiko)
> - Resolved errant err_rch unwind with rc7 merge (DaveJ)
> - Rebase onto 7.1-rc7
> - Update commit logs in 1,2,5 to align w changes in v2
> Link to v1:
> https://lore.kernel.org/all/cover.1780095671.git.alison.schofield@intel.com/
> 
> 
> Begin Cover Letter:
> 
> Linux has required that a region's interleave granularity equal the interleave
> granularity of its interleaving root decoder. A mixed-granularity region lifts
> that restriction: the region granularity may be finer than the root's, so the
> interleave refines from the root toward the endpoints. The mix is between the
> root and the region, not granularity varying arbitrarily down the hierarchy.
> 
> A CXL region may interleave across multiple decoder levels: root, optional
> switches, and endpoint. The equal-granularity requirement blocks legal
> mixed-granularity arrangements permitted by CXL 4.0 Section 9.13.1 and makes
> some 6-way and 12-way configurations defined in Section 9.13.1.1 (Tables 9-6,
> 9-7) impossible to create.
> 
> Two prior proposals addressed parts of this gap:
> 
> AlisonS introduced position arithmetic and sysfs gating for auto and
> user-created regions to support the 6- and 12-way interleave configurations
> without a same-granularity alternative:
> https://lore.kernel.org/all/20250306232239.2609017-1-alison.schofield@intel.com/
> 
> RobertR introduced an HPA selector-bit model for auto regions that allows
> multi-level power-of-two interleaves regardless of granularity ordering:
> https://lore.kernel.org/all/20251028094754.72816-1-rrichter@amd.com/
> 
> This series adds support for CXL regions where interleave granularity differs
> between levels of the decoder hierarchy. The support applies to both firmware
> programmed auto regions and user created regions.
> 
> Linux supports mixed-granularity configurations that are monotonic from the
> interleaving root toward the endpoints. Granularity may remain the same or
> become finer at each interleaving level. Configurations that refine and then
> become coarser are permitted by the CXL Specification but are not supported
> by Linux.
> 
> This support covers power-of-two and Mod3 interleaves at any depth. Of the Mod3
> configs listed in CXL 4.0 Section 9.13.1.1, all are supported except Table 9-6
> row 6, the refine-then-coarsen 12-way composition. See the Patch 1 documentation
> update for additional info.
> 
> 
> Series structure in v5:
> -----------------------
> Patch 1 documents the mixed-granularity model and Linux support policy.
> It comes first so that the terms the rest of the series uses - span,
> region position, weight - are defined before they are used.
> 
> Patch 2 promotes the existing user-region endpoint position self-test to
> a visible diagnostic.
> 
> Patch 3 generalizes endpoint position calculation to account for
> granularity changes between decoder levels.
> 
> Patch 4 names the interleave locals in cxl_port_setup_targets(). It is a
> mechanical rename, carried separately so it does not muddy Patch 5.
> 
> Patch 5 enables mixed-granularity auto regions and validates their
> coarse-to-fine decoder geometry.
> 
> Patch 6 enables mixed-granularity user created regions by relaxing the
> existing root/region granularity restriction.
> 
> Patch 7 adds a cxl_test topology and coverage for mixed-granularity
> regions.
> 
> 
> The companion NDCTL patchset that allows mixed-granularity 'cxl create-region'
> and adds the unit test is posted here:
> https://lore.kernel.org/nvdimm/cover.1788469004.git.alison.schofield@intel.com/
> 
> 
> Alison Schofield (7):
>   Documentation/cxl: Describe mixed-granularity regions
>   cxl/region: Warn on user region position mismatch
>   cxl/region: Generalize endpoint position mapping
>   cxl/region: Name the interleave locals in cxl_port_setup_targets()
>   cxl/region: Support mixed-granularity auto regions
>   cxl/region: Support mixed-granularity user created regions
>   cxl/test: Add a topology to test mixed-granularity regions
> 
>  .../driver-api/cxl/linux/cxl-driver.rst       | 153 +++++++
>  drivers/cxl/core/region.c                     | 247 +++++++----
>  tools/testing/cxl/test/cxl.c                  | 410 +++++++++++++++++-
>  3 files changed, 698 insertions(+), 112 deletions(-)
> 
> 
> base-commit: cee9395acd8043be0644b25c34bfa86623f2b935

2/7 merged to cxl/next:
682d3772be80