Re: FW: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL

[David Hildenbrand <david@redhat.com>]

On 23.03.23 11:51, Kyungsan Kim wrote:
> I appreciate dan for the careful advice.
> 
>> Kyungsan Kim wrote:
>> [..]
>>>> In addition to CXL memory, we may have other kind of memory in the
>>>> system, for example, HBM (High Bandwidth Memory), memory in FPGA card,
>>>> memory in GPU card, etc.  I guess that we need to consider them
>>>> together.  Do we need to add one zone type for each kind of memory?
>>>
>>> We also don't think a new zone is needed for every single memory
>>> device.  Our viewpoint is the sole ZONE_NORMAL becomes not enough to
>>> manage multiple volatile memory devices due to the increased device
>>> types.  Including CXL DRAM, we think the ZONE_EXMEM can be used to
>>> represent extended volatile memories that have different HW
>>> characteristics.
>>
>> Some advice for the LSF/MM discussion, the rationale will need to be
>> more than "we think the ZONE_EXMEM can be used to represent extended
>> volatile memories that have different HW characteristics". It needs to
>> be along the lines of "yes, to date Linux has been able to describe DDR
>> with NUMA effects, PMEM with high write overhead, and HBM with improved
>> bandwidth not necessarily latency, all without adding a new ZONE, but a
>> new ZONE is absolutely required now to enable use case FOO, or address
>> unfixable NUMA problem BAR." Without FOO and BAR to discuss the code
>> maintainability concern of "fewer degress of freedom in the ZONE
>> dimension" starts to dominate.
> 
> One problem we experienced was occured in the combination of hot-remove and kerelspace allocation usecases.
> ZONE_NORMAL allows kernel context allocation, but it does not allow hot-remove because kernel resides all the time.
> ZONE_MOVABLE allows hot-remove due to the page migration, but it only allows userspace allocation.
> Alternatively, we allocated a kernel context out of ZONE_MOVABLE by adding GFP_MOVABLE flag.

That sounds like a bad hack :) .

> In case, oops and system hang has occasionally occured because ZONE_MOVABLE can be swapped.
> We resolved the issue using ZONE_EXMEM by allowing seletively choice of the two usecases.

I once raised the idea of a ZONE_PREFER_MOVABLE [1], maybe that's 
similar to what you have in mind here. In general, adding new zones is 
frowned upon.

> As you well know, among heterogeneous DRAM devices, CXL DRAM is the first PCIe basis device, which allows hot-pluggability, different RAS, and extended connectivity.
> So, we thought it could be a graceful approach adding a new zone and separately manage the new features.
> 
> Kindly let me know any advice or comment on our thoughts.

[1] https://www.lkml.org/lkml/2020/9/9/667

-- 
Thanks,

David / dhildenb