Re: [PATCH] Documentation: CXL Maturity Map

[Dan Williams <dan.j.williams@intel.com>]

Jonathan Cameron wrote:
> On Fri, 24 May 2024 17:53:49 -0700
> Dan Williams <dan.j.williams@intel.com> wrote:
> 
> > Provide a survey of the work-in-progress maturity (implementation
> > status) of various aspects of the CXL subsystem.
> > 
> > Clarify that in addition to ongoing upkeep relative to specification
> > updates, there are some long running themes in the driver that respond
> > to the discovery of new corner cases (bugs) and new use cases (feature
> > extensions).
> > 
> > The primary audience is distribution maintainers, but it also serves as
> > a guide for kernel developers to understand what aspects of the CXL
> > subsystem need more help. It is a landing page to document ongoing
> > progress, and a guide to discern exposure to work-in-progress features.
> > 
> > Help wanted / welcome to expand on the "Details" section.
> > 
> > Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> This feels like a really bit opportunity for bikeshedding..
> 
> Ah well, a few comments inline. I've tried restrain my natural tendency
> to argue about number assignments.
> 
> Adam's email address was garbled in my email client - fixed up.
> 
> > ---
> >  Documentation/driver-api/cxl/index.rst        |    2 
> >  Documentation/driver-api/cxl/maturity-map.rst |  173 +++++++++++++++++++++++++
> >  MAINTAINERS                                   |    1 
> >  3 files changed, 176 insertions(+)
> >  create mode 100644 Documentation/driver-api/cxl/maturity-map.rst
> > 
> > diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst
> > index 036e49553542..12b82725d322 100644
> > --- a/Documentation/driver-api/cxl/index.rst
> > +++ b/Documentation/driver-api/cxl/index.rst
> > @@ -9,4 +9,6 @@ Compute Express Link
> >  
> >     memory-devices
> >  
> > +   maturity-map
> > +
> >  .. only::  subproject and html
> > diff --git a/Documentation/driver-api/cxl/maturity-map.rst b/Documentation/driver-api/cxl/maturity-map.rst
> > new file mode 100644
> > index 000000000000..9c5bff6484dd
> > --- /dev/null
> > +++ b/Documentation/driver-api/cxl/maturity-map.rst
> > @@ -0,0 +1,173 @@
> > +.. SPDX-License-Identifier: GPL-2.0
> > +.. include:: <isonum.txt>
> > +
> > +===========================================
> > +Compute Express Link Subsystem Maturity Map
> > +===========================================
> > +
> > +The Linux CXL subsystem tracks the dynamic `CXL specification
> > +<https://computeexpresslink.org/cxl-specification-landing-page>`_ that
> > +continues to respond to new use cases with new features, capability
> > +updates and fixes. At any given point some aspects of the subsystem are
> > +more mature than others. While the periodic pull requests summarize the
> > +`work being incorporated each merge window
> > +<https://lore.kernel.org/linux-cxl/?q=s%3APULL+s%3ACXL+tc%3Atorvalds+NOT+s%3ARe>`_,
> > +those do not always convey progress relative to a starting point and a
> > +future end goal.
> > +
> > +What follows is a coarse breakdown of the subsystem's major
> > +responsibilities along with a maturity score. The expectation is that
> > +the change-history of this document provides an overview summary of the
> > +subsystem maturation over time.
> > +
> > +The maturity scores are:
> > +
> > +- [3] Mature: Work in this area is complete and no changes on the horizon.
> > +  Note that this score can regress from one kernel release to the next
> > +  based on new test results or end user reports.
> > +
> > +- [2] Stabilizing: Major functionality operational, common cases are
> > +  mature, but known corner cases are still a work in progress.
> > +
> > +- [1] Initial: Capability that has exited the Proof of Concept phase, but
> > +  may still have significant gaps to close and fixes to apply as real
> > +  world testing occurs.
> > +
> > +- [0] Known gap: Feature is on a medium to long term horizon to
> > +  implement.  If the specification has a feature that does even have a '0'
> > +  score in this document, there is a good chance that no one in the
> > +  linux-cxl@vger.kernel.org community has started to look at it.
> > +
> > +- X: Out of scope for kernel enabling, or kernel enabling not required
> > +
> > +Feature and Capabilities
> > +========================
> > +
> > +Enumeration / Provisioning
> > +--------------------------
> > +All of the fundamental enumeration an object model of the subsystem is
> > +in place, but there are several corner cases that are pending closure.
> > +
> 
> Feels like we should in passing mention that we support none of the
> bits of CXL fabrics that are host visible (so mainly G-FAM I think).
> 
> [0] Fabrics / G-FAM (chapter 7)
> [0] Global Access Enpoint

Are those [0] or [X], not even sure what the kernel is expected to do with those?

> > +
> > +* [2] CXL Window Enumeration
> > +
> > +  * [0] :ref:`Extended-linear memory-side cache <extended-linear>`
> > +  * [0] Low Memory-hole
> > +  * [0] Hetero-interleave
> > +
> > +* [2] Switch Enumeration
> > +
> > +  * [0] CXL register enumeration link-up dependency
> > +
> > +* [2] HDM Decoder Configuration
> > +
> > +  * [0] Decoder target and granularity constraints
> 
> Feels like you added this one just so we can tick it off ;)

True, the list was a bit of brain dump. I also expect that "git log -p
Documentation/driver-api/cxl/maturity-map.rst" to show small edits like
this. No need to overthink this informal list.

> > +
> > +2 Performance enumeration
> > +
> > +  * [3] Endpoint CDAT
> > +  * [3] Switch CDAT
> > +  * [1] CDAT to Core-mm integration
> 
> [1] x86
> [0] Arm64
> [0] All other arch.
> 
> I guess could argue that's a core-mm problem, but the CEDT code
> just all functions that are only implemented on x86.

Oh, good point, will add.

> 
> > +  * [1] Shared link
> 
> [0] Shared link. We've not merged that one yet though getting closer.

Ok.

> 
> > +
> > +* [2] Hotplug
> > +  (see CXL Window Enumeration)
> > +
> > +  * [0] Handle Soft Reserved conflicts
> > +
> > +* [0] :ref:`RCH link status <rch-link-status>`
> > +
> > +RAS
> > +---
> > +In many ways CXL can be seen as a standardization of what would normally
> > +be handled by custom EDAC drivers. The open development here is
> > +mainly caused by the enumeration corner cases above.
> > +
> > +* [3] Component events (OS)
> > +* [2] Component events (FFM)
> > +* [1] Endpoint protocol errors (OS)
> > +* [1] Endpoint protocol errors (FFM)
> > +* [0] Switch protocol errors (OS)
> > +* [1] Switch protocol errors (FFM)
> > +* [2] DPA->HPA Address translation
> > +
> > +    * [1] XOR Interleave translation
> > +      (see CXL Window Enumeration)
> > +
> > +* [1] Memory Failure coordination
> > +* [0] Scrub control
> 
> [0] PPR
> [0] Sparing 
> [0] Device built in test

Added.

> 
> > +* [2] ACPI error injection EINJ
> > +
> > +  * [0] EINJ v2
> > +  * [X] Compliance DOE
> > +
> > +* [2] Native error injection
> > +* [3] RCH error handling
> > +* [1] VH error handling
> > +
> > +Mailbox commands
> > +----------------
> > +
> > +* [3] Firmware update
> > +* [3] Health / Alerts
> > +* [1] :ref:`Background commands <background-commands>`
> > +* [3] Santization
> > +* [3] Security commands
> > +* [3] RAW Command Debug Passthrough
> > +* [0] CEL-only-validtion Passthrough
> > +* [0] Switch CCI
> > +* [3] Timestamp
> > +* [1] PMU
> 
> Not mailbox, plus breakdown into switch vs type3 - see below.

Done.

> 
> > +* [1] PMEM labels
> > +* [0] PMEM GPF / Dirty Shutdown
> > +
> 
> PMU
> ---
> 
> [1] Type 3 PMU
> [0] Switch USP/ DSP, Root Port

...got it.

> 
> > +Security
> > +--------
> > +
> > +* [X] CXL Trusted Execution Environment Security Protocol (TSP)
> > +* [X] CXL IDE (subsumed by TSP)
> > +
> > +Multi-host memory
> > +-----------------
> > +
> > +* [0] Dynamic Capacity Device Support
> > +* [0] Sharing
> 
> Break this one up
> 
> Memory-pooling
> --------------
> 
> [1] Hotplug of LDs (via PCI hotplug)
> [0] Dynamic Capacity Device (DCD) Support
> 
> Multi-host sharing
> ------------------
> 
> Requires DCD support.
> [0] Hardware coherent shared memory
> [0] Software managed coherency shared memory

Looks good.