Re: CXL volatile memory: How to restore the previous region/Interleave set

["Zhijian Li (Fujitsu)" <lizhijian@fujitsu.com>]

On 29/05/2024 23:44, Gregory Price wrote:
> On Wed, May 29, 2024 at 10:19:21AM +0000, Zhijian Li (Fujitsu) wrote:
>> Thanks Dan,
>>
>>
>> On 29/05/2024 09:08, Dan Williams wrote:
>>> Hi Zhijian,
>>>
>>> I dropped members@computeexpresslink.org from this thread. If those
>>> folks are interested they can follow this discussion here:
>>>
>>> https://lore.kernel.org/r/36106fcf-1062-4961-8918-4471fd313a74@fujitsu.com
>>>
>>> Otherwise, the way the wider Linux community learns about consortium
>>> deliberations is through new published spec revisions.
>>
>> Agreed. thank you.
>>
>>
>>>
>>> Zhijian Li (Fujitsu) wrote:
>>>> Hey CXL and Linux-CXL communities,
>>>>
>>>> I am trying to understand how the current hardware and software can work
>>>> together to restore the previous region/Interleave Set configuration for CXL
>>>> volatile memory upon the next boot, but I don't have the answer yet.
>>>> Therefore, I have several questions and hope you can provide some suggestions
>>>> and thoughts. Thank you.
>>>>
>>>> Q1, First, I would like to ask about the scope of LSA. According to CXL r3.0
>>>> section 9.13.2, it seems that LSA applies to CXL memory (including volatile
>>>> memory and persistent memory), but it does not explicitly state whether LSA
>>>> is mandatory. My understanding is:
>>>> - LSA is mandatory for persistent memory
>>>> - LSA is optional for volatile memory
>>>> Is this understanding correct?
>>>
>>> I would say it differently. LSA is mandatory for persistent memory, and
>>> irrelevant for volatile memory.
>>
>>
>> Another reason for my above understanding is that the current QEMU
>> documentation[1] also states this, and the actual code behaves accordingly
>> (mandatory for persistent memory and optional for volatile memory).
>>
>> [1] https://github.com/qemu/qemu/blob/79d7475f39f1b0f05fcb159f5cdcbf162340dc7e/docs/system/devices/cxl.rst?plain=1#L324
>>
>> Anyone have other opinion on this?
>>
>> Hope the CXL consortium can help on confirming on this point to
>> members@computeexpresslink.org privately.
>> (Currently the original mail cannot reach members@computeexpresslink.org
>> until getting its approval)
>>
> 
> Just to be concrete:
> 
> CXL Spec 3.1: 8.2.9.9.2.3
> 
> "The Label Storage Area (LSA) *shall be* supported by a memory device
> that provides persistent memory capacity and *may be* supported by a
> device that provides only volatile memory capacity"

Thanks for the reference, I missed it before.


> 
> For persistent: Required
> For volatile: Optional
> 
> What Dan is saying is that an voltile-only device with an LSA is
> possible, but the LSA isn't particularly novel or useful since the
> data in the voltile region is destroyed when the device is reset.
> 

Well, I misunderstood this in the previous reply.
Many thanks for your explanation on this point.


> Recording things like interleave set configurations in LSA doesn't
> really make sense, given that devices probably shouldn't know about each
> other, and an interleave set kind of implies knowledge of other devices.
> > So as Dan said: An LSA is irrelevant for a voltile device.  Anything
> you'd use it for is probably better accomplished some other way.> 
>>>
>>> The expectation is that BIOS, or the OS for hotplug devices, deploys a
>>> default region configuration policy. That policy in the common is likely
>>> one of either maximizing performance (maximize interleave across
>>> host-bridges), or maximizing error isolation (create an x1-interleave
>>> region per endpoint).
>>>> What is currently missing on the Linux OS side is a default policy for
>>> unmapped volatile capacity after all initial device probing has
>>> completed.
>>
>> I would say I can imagine the policy you mentioned. *policy* would work
>> on some cases, but
>> For a customized region, a default/predefined *policy* is not sufficient.
>> In my mind, a customized region could be constructed with part of all
>> the available memdevs(SLD/MLD/DCD), and customized interleave set.
>>
>> So the the previous used configurations including memdevs and interleave set
>> should be stored in a persistent storage that the region can be restored
>> correctly.
>>
> 
> There's any number of reasons this is a bad idea, the most obvious of
> which is that your recording information about a set of devices on each
> device.  What if some of those devices go away (or are upgraded, change,
> whatever) and you need to do something different? What if you move that
> device to another machine?  The state machine you create with this setup
> is pretty awful.  

I suspect in answering these questions, you end up
> just resolving to save the configuration to disk instead of the LSAs -
> which (as Dan said) makes the LSA irrelevant.
> 

In my understanding, LSA Region Label should include the needed information
to restore the previous interleave set(across memdevs). Of course, these
information on LSA are different from the one saved in the disk.


> Really what you want is a smarter daemon that detects the topology and
> implements the best policy given the current environment.


Anyway, I think I have gotten the answer from your and Dan's reply.

Thanks again,

Zhijian


> 
> ~Gregory