Re: [PATCH v11 0/8] mm/demotion: Memory tiers and demotion

["Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>]

"Huang, Ying" <ying.huang@intel.com> writes:

> "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com> writes:
>
>> The current kernel has the basic memory tiering support: Inactive pages on a
>> higher tier NUMA node can be migrated (demoted) to a lower tier NUMA node to
>> make room for new allocations on the higher tier NUMA node. Frequently accessed
>> pages on a lower tier NUMA node can be migrated (promoted) to a higher tier NUMA
>> node to improve the performance.
>>
>> In the current kernel, memory tiers are defined implicitly via a demotion path
>> relationship between NUMA nodes, which is created during the kernel
>> initialization and updated when a NUMA node is hot-added or hot-removed. The
>> current implementation puts all nodes with CPU into the top tier, and builds the
>> tier hierarchy tier-by-tier by establishing the per-node demotion targets based
>> on the distances between nodes.
>>
>> This current memory tier kernel interface needs to be improved for several
>> important use cases:
>>
>> * The current tier initialization code always initializes each memory-only NUMA
>>   node into a lower tier. But a memory-only NUMA node may have a high
>>   performance memory device (e.g. a DRAM device attached via CXL.mem or a
>>   DRAM-backed memory-only node on a virtual machine) and should be put into a
>>   higher tier.
>>
>> * The current tier hierarchy always puts CPU nodes into the top tier. But on a
>>   system with HBM (e.g. GPU memory) devices, these memory-only HBM NUMA nodes
>>   should be in the top tier, and DRAM nodes with CPUs are better to be placed
>>   into the next lower tier.
>>
>> * Also because the current tier hierarchy always puts CPU nodes into the top
>>   tier, when a CPU is hot-added (or hot-removed) and triggers a memory node from
>>   CPU-less into a CPU node (or vice versa), the memory tier hierarchy gets
>>   changed, even though no memory node is added or removed. This can make the
>>   tier hierarchy unstable and make it difficult to support tier-based memory
>>   accounting.
>>
>> * A higher tier node can only be demoted to selected nodes on the next lower
>>   tier as defined by the demotion path, not any other node from any lower tier.
>>   This strict, hard-coded demotion order does not work in all use cases (e.g.
>>   some use cases may want to allow cross-socket demotion to another node in the
>>   same demotion tier as a fallback when the preferred demotion node is out of
>>   space), and has resulted in the feature request for an interface to override
>>   the system-wide, per-node demotion order from the userspace. This demotion
>>   order is also inconsistent with the page allocation fallback order when all
>>   the nodes in a higher tier are out of space: The page allocation can fall back
>>   to any node from any lower tier, whereas the demotion order doesn't allow
>>   that.
>>
>> This patch series make the creation of memory tiers explicit under
>> the control of device driver.
>>
>> Memory Tier Initialization
>> ==========================
>>
>> Linux kernel presents memory devices as NUMA nodes and each memory device is of
>> a specific type. The memory type of a device is represented by its abstract 
>> distance. A memory tier corresponds to a range of abstract distance. This allows
>> for classifying memory devices with a specific performance range into a memory
>> tier.
>>
>> By default, all memory nodes are assigned to the default tier with
>> abstract distance 512.
>>
>> A device driver can move its memory nodes from the default tier. For example,
>> PMEM can move its memory nodes below the default tier, whereas GPU can move its
>> memory nodes above the default tier.
>>
>> The kernel initialization code makes the decision on which exact tier a memory
>> node should be assigned to based on the requests from the device drivers as well
>> as the memory device hardware information provided by the firmware.
>>
>> Hot-adding/removing CPUs doesn't affect memory tier hierarchy.
>
> Some patch description of [0/8] is same as that of [1/8] originally.  It
> appears that you revised [1/8], but forget to revise [0/8] too.  Please
> do that.

I just sent v12 making sure smaller value of abstract distance imply
faster(higher) memory tier. I missed in that in v11. 

-aneesh