Re: FW: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL

[David Hildenbrand <david@redhat.com>]

On 24.03.23 10:09, Kyungsan Kim wrote:
> Thank you David Hinderbrand for your interest on this topic.
> 
>>>
>>>> Kyungsan Kim wrote:
>>>> [..]
>>>>>> In addition to CXL memory, we may have other kind of memory in the
>>>>>> system, for example, HBM (High Bandwidth Memory), memory in FPGA card,
>>>>>> memory in GPU card, etc.  I guess that we need to consider them
>>>>>> together.  Do we need to add one zone type for each kind of memory?
>>>>>
>>>>> We also don't think a new zone is needed for every single memory
>>>>> device.  Our viewpoint is the sole ZONE_NORMAL becomes not enough to
>>>>> manage multiple volatile memory devices due to the increased device
>>>>> types.  Including CXL DRAM, we think the ZONE_EXMEM can be used to
>>>>> represent extended volatile memories that have different HW
>>>>> characteristics.
>>>>
>>>> Some advice for the LSF/MM discussion, the rationale will need to be
>>>> more than "we think the ZONE_EXMEM can be used to represent extended
>>>> volatile memories that have different HW characteristics". It needs to
>>>> be along the lines of "yes, to date Linux has been able to describe DDR
>>>> with NUMA effects, PMEM with high write overhead, and HBM with improved
>>>> bandwidth not necessarily latency, all without adding a new ZONE, but a
>>>> new ZONE is absolutely required now to enable use case FOO, or address
>>>> unfixable NUMA problem BAR." Without FOO and BAR to discuss the code
>>>> maintainability concern of "fewer degress of freedom in the ZONE
>>>> dimension" starts to dominate.
>>>
>>> One problem we experienced was occured in the combination of hot-remove and kerelspace allocation usecases.
>>> ZONE_NORMAL allows kernel context allocation, but it does not allow hot-remove because kernel resides all the time.
>>> ZONE_MOVABLE allows hot-remove due to the page migration, but it only allows userspace allocation.
>>> Alternatively, we allocated a kernel context out of ZONE_MOVABLE by adding GFP_MOVABLE flag.
> 
>> That sounds like a bad hack :) .
> I consent you.
> 
>>> In case, oops and system hang has occasionally occured because ZONE_MOVABLE can be swapped.
>>> We resolved the issue using ZONE_EXMEM by allowing seletively choice of the two usecases.
> 
>> I once raised the idea of a ZONE_PREFER_MOVABLE [1], maybe that's
>> similar to what you have in mind here. In general, adding new zones is
>> frowned upon.
> 
> Actually, we have already studied your idea and thought it is similar with us in 2 aspects.
> 1. ZONE_PREFER_MOVABLE allows a kernelspace allocation using a new zone
> 2. ZONE_PREFER_MOVABLE helps less fragmentation by splitting zones, and ordering allocation requests from the zones.
> 
> We think ZONE_EXMEM also helps less fragmentation.
> Because it is a separated zone and handles a page allocation as movable by default.

So how is it different that it would justify a different (more confusing 
IMHO) name? :) Of course, names don't matter that much, but I'd be 
interested in which other aspect that zone would be "special".

-- 
Thanks,

David / dhildenb