Re: [LSF/MM/BPF TOPIC] Restricting or migrating unmovable kernel allocations from slow tier

David Hildenbrand
2025-02-04T10:00:05Z

On 01.02.25 14:29, Hyeonggon Yoo wrote:
> Hi,
>
> Byungchul and I would like to suggest a topic about the performance impact of
> kernel allocations on CXL memory.
>
> As CXL-enabled servers and memory devices are being developed, CXL-supported
> hardware is expected to continue emerging in the coming years.
>
> The Linux kernel supports hot-plugging CXL memory via dax/kmem functionality.
> The hot-plugged memory allows either unmovable kernel allocations
> (ZONE_NORMAL), or restricts them to movable allocations (ZONE_MOVABLE)
> depending on the hot-plug policy.
>
> Recently, Byungchul and I observed a measurable performance degradation with
> memhp_default_state=online compared to memhp_default_state=online_movable
> on a server where the ratio of memory capacity between DRAM and CXL is 1:2
> when running the llama.cpp workload with the default mempolicy.
> The workload performs LLM inference and pressures the memory subsystem
> due to its large working set size.
>
> Obviously, allowing kernel allocations from CXL memory degrades performance
> because kernel memory like page tables, kernel stacks, and slab allocations,
> is accessed frequently and may reside in physical memory with significantly
> higher access latency.
>
> However, as far as I can tell there are at least two reasons why we need to
> support ZONE_NORMAL for CXL memory (please add if there are more):
> 1. When hot-plugging a huge amount of CXL memory, the size of
> the struct page array might not fit into DRAM
> -> This could be relaxed with memmap_on_memory

There are some others, although most are less significant, and I tried
documenting them here:

https://www.kernel.org/doc/html/latest/admin-guide/mm/memory-hotplug.html#zone-movable-sizing-considerations

E.g., a 4 KiB page requires a single PTE (8 bytes) to be mapped into
user space, corresponding to 0.2 %. At least for anonymous memory,
PMD-sized THPs don't help, because we still have to allocate the page
table to be prepared for a PMD->PTE remapping. In the worst case, the
directmap requires another 0.2 % (but usually, we rely on PMD mappings).
So that usage depends on how you are intending to use the CXL memory
(e.g., pagecache vs. anonymous memory).

> 2. To hot-unplug CXL memory, pages in CXL memory should be
migrated to DRAM,
> which means sometimes some portion of CXL memory should be
ZONE_NORMAL.

I don't quite understand that argument for ZONE_NORMAL.

--
Cheers,

David / dhildenb