Re: CXL volatile memory: How to restore the previous region/Interleave set

[Gregory Price <gregory.price@memverge.com>]

On Wed, May 29, 2024 at 10:19:21AM +0000, Zhijian Li (Fujitsu) wrote:
> Thanks Dan,
> 
> 
> On 29/05/2024 09:08, Dan Williams wrote:
> > Hi Zhijian,
> > 
> > I dropped members@computeexpresslink.org from this thread. If those
> > folks are interested they can follow this discussion here:
> > 
> > https://lore.kernel.org/r/36106fcf-1062-4961-8918-4471fd313a74@fujitsu.com
> > 
> > Otherwise, the way the wider Linux community learns about consortium
> > deliberations is through new published spec revisions.
> 
> Agreed. thank you.
> 
> 
> > 
> > Zhijian Li (Fujitsu) wrote:
> >> Hey CXL and Linux-CXL communities,
> >>
> >> I am trying to understand how the current hardware and software can work
> >> together to restore the previous region/Interleave Set configuration for CXL
> >> volatile memory upon the next boot, but I don't have the answer yet.
> >> Therefore, I have several questions and hope you can provide some suggestions
> >> and thoughts. Thank you.
> >>
> >> Q1, First, I would like to ask about the scope of LSA. According to CXL r3.0
> >> section 9.13.2, it seems that LSA applies to CXL memory (including volatile
> >> memory and persistent memory), but it does not explicitly state whether LSA
> >> is mandatory. My understanding is:
> >> - LSA is mandatory for persistent memory
> >> - LSA is optional for volatile memory
> >> Is this understanding correct?
> > 
> > I would say it differently. LSA is mandatory for persistent memory, and
> > irrelevant for volatile memory.
> 
> 
> Another reason for my above understanding is that the current QEMU
> documentation[1] also states this, and the actual code behaves accordingly
> (mandatory for persistent memory and optional for volatile memory).
> 
> [1] https://github.com/qemu/qemu/blob/79d7475f39f1b0f05fcb159f5cdcbf162340dc7e/docs/system/devices/cxl.rst?plain=1#L324
> 
> Anyone have other opinion on this?
> 
> Hope the CXL consortium can help on confirming on this point to
> members@computeexpresslink.org privately.
> (Currently the original mail cannot reach members@computeexpresslink.org
> until getting its approval)
>

Just to be concrete:

CXL Spec 3.1: 8.2.9.9.2.3

"The Label Storage Area (LSA) *shall be* supported by a memory device
that provides persistent memory capacity and *may be* supported by a
device that provides only volatile memory capacity"

For persistent: Required
For volatile: Optional

What Dan is saying is that an voltile-only device with an LSA is
possible, but the LSA isn't particularly novel or useful since the
data in the voltile region is destroyed when the device is reset.

Recording things like interleave set configurations in LSA doesn't
really make sense, given that devices probably shouldn't know about each
other, and an interleave set kind of implies knowledge of other devices.

So as Dan said: An LSA is irrelevant for a voltile device.  Anything
you'd use it for is probably better accomplished some other way.

> > 
> > The expectation is that BIOS, or the OS for hotplug devices, deploys a
> > default region configuration policy. That policy in the common is likely
> > one of either maximizing performance (maximize interleave across
> > host-bridges), or maximizing error isolation (create an x1-interleave
> > region per endpoint).
> >> What is currently missing on the Linux OS side is a default policy for
> > unmapped volatile capacity after all initial device probing has
> > completed.
> 
> I would say I can imagine the policy you mentioned. *policy* would work
> on some cases, but
> For a customized region, a default/predefined *policy* is not sufficient.
> In my mind, a customized region could be constructed with part of all
> the available memdevs(SLD/MLD/DCD), and customized interleave set.
> 
> So the the previous used configurations including memdevs and interleave set
> should be stored in a persistent storage that the region can be restored
> correctly.
>

There's any number of reasons this is a bad idea, the most obvious of
which is that your recording information about a set of devices on each
device.  What if some of those devices go away (or are upgraded, change,
whatever) and you need to do something different? What if you move that
device to another machine?  The state machine you create with this setup
is pretty awful.  I suspect in answering these questions, you end up
just resolving to save the configuration to disk instead of the LSAs -
which (as Dan said) makes the LSA irrelevant.

Really what you want is a smarter daemon that detects the topology and
implements the best policy given the current environment.

~Gregory