Re: [RFC PATCH 0/9 v2] mm/memcontrol: Make memory cgroup limits tier-aware

["David Hildenbrand (Arm)" <david@kernel.org>]

On 5/11/26 22:03, Joshua Hahn wrote:
> On Mon, May 11, 2026 at 5:56 PM David Hildenbrand (Arm)
> <david@kernel.org> wrote:
>>
>> On 4/23/26 22:34, Joshua Hahn wrote:
>>> INTRODUCTION
>>> ============
>>> Memory cgroups provide an interface that allow multiple works on a host to
>>> co-exist via weak and strong memory isolation guarantees. This works, because
>>> for the most part, all memory has equal utility. Isolating a cgroup’s memory
>>> footprint restricts how much it can hurt other workloads competing for memory,
>>> or protects it from other cgroups looking for more memory.
>>>
>>> However, on systems with tiered memory (e.g. CXL), memory utility is no longer
>>> homogeneous; toptier and lowtier memory provide different performance
>>> characteristics and have different scarcity, meaning memory footprint no longer
>>> serves as an accurate representation of a cgroup’s consumption of the system’s
>>> limited resources. As an extreme example, a cgroup with 10G of toptier
>>> (e.g. DRAM) memory and a cgroup with 10G of lowtier (e.g. CXL) memory both
>>> appear to be consuming the same amount of system resources from memcg’s
>>> perspective, despite the performance asymmetry between the two workloads.
>>>
>>> Therefore on tiered systems, memory isolation cannot currently happen, as
>>> workloads that are well-behaved within their memcg limits may still hurt the
>>> performance of other well-behaving workloads by hogging more than its
>>> “fair share” of toptier memory.
>>>
>>> Introduce tier-aware memcg limits, which establish independent toptier limits
>>> that scale with the memory limits and the ratio of toptier:total memory
>>> available on the system.
>>>
>>> INTERFACE
>>> =========
>>> This series introduces only one adjustable knob to userspace; a new cgroup mount
>>> option “memory_tiered_limits” which toggles whether the cgroup mount will scale
>>> toptier limits. It also introduces 4 new read-only sysfs entries per-cgroup:
>>> memory.toptier_{min, low, high, max}.
>>>
>>> The new toptier memory limits are scaled according to the amount of toptier
>>> memory and total memory available on the system as such:
>>>
>>> memory.toptier_high = (toptier_mem / total_mem) * memory.high
>>>
>>> For instance, on a host with 100GB memory, with 75G toptier and 25G CXL, the
>>> “toptier ratio” would be 75 / 100 = 0.75. A cgroup with the following memcg
>>> limits {min: 8G, low: 12G, high: 20G, max: 24G} might see toptier limits scaled
>>> at {min: 6G, low: 9G, high: 15G, max: 18G}.
> 
> Hi David!!
> 
> It was great seeing you at LSFMMBPF. I didn't get a chance to have a
> conversation with you at Zagreb but hopefully I will be less shy and say
> hello next conference : -)

Sorry for the late reply :) Yes, hopefully talk to you in person next time!

> 
>> Assume you have a bigger hierarchy (HBP, DRAM, CXL), or assume you have multiple
>> NUMA nodes with a hierarchy each.
>>
>> Your proposal doesn't really seem to be very versatile, or am I wrong?
> 
> Let me address these comments separately!
> 
> First, for the multi-numa-per-tier case, I think this is already pretty well
> handled by my series. Once we realize that a memcg is consuming too much memory
> from a tier, we trigger reclaim from that memcg via
> try_to_free_mem_cgroup_pages,
> which as far as I can tell already handles the multi-numa per memcg case.
> Other than restricting the scan_control's nodemask to target the nodes
> from that tier, I don't think there's anything else to be done.

I am not really sure about the "toptier" notion, in particular if your memory
hierarchy is not static.

Imagine that you have only DRAM in your system. Then you hotplug some slow CXL
memory. The you hotplug some HBM CXL memory.

The meanting of "toptier" changes, which is rather confusing to me.

> 
> Next for the 3+ tier case, I think this is a lot more scalable than it seems
> at first. This series depends on another RFC that I sent out [1],
> which pushes the concept of "stock" from memcg to page_counter, which means
> that it is more scalable to just add more page counters to each memcg.
> This means that each tier would just need another page_counter to track its
> memory usage, and we trigger selective reclaim on the tier that is being
> targeted via the scan control nodemask we introduce in this series.
> 
> At my talk in LSFMMBPF, Usama noted that the user-visible API should probably
> remain the same, no matter what. The way I have currently established the
> memcg files aren't really scalable, so Usama suggested turning the
> "memory.toptier_XXX" sysfs files to "memory.tiered_XXX", which would include
> a newline-separated / space-separated list of per-tier limits. Something like:
> 
> $ cat memory.tiered_max
> tier_0 20971520
> tier_1 31457280
> ...

memcgs are currently not NUMA aware, so this could be one possible direction.

But is "tier" the right abstraction? I really don't know.

Are tier numbers really what we want? I don't know.

We'd really need some feedback from memcg folks.

> 
> So we have a way to make both the user-facing side stable, and the internals
> also more scalable.
> 
> With that said, I've opted to leave the internals to 2 tiers for now -- I think
> it is not too late to add the generalization series when we start seeing
> 3+ tier systems out there in the wild. My goal was to introduce tieredness,
> and we can work towards generalization in a future work.
> 
> On that note, it seems like in general mm is aware of 3+ tiers, but most of the
> existing work revolves around distinguishing between toptier/everything else.
> I got this impression from reading mm/memory-tiers.c -- but please feel free
> to correct me if you feel like I have the wrong idea here : -)

I assume you're right, but it's something different if we end up providing a
proper memcg API towards the users.

> 
> So perhaps the generalization work would benefit from first introducing more
> general tier awareness (not just toptier vs. rest) in memory-tiers.c.
> 
> What do you think? Does this approach of introducing toptier restriction for
> now, and then generalizing in future work make sense to you?

I'd suspect that we'd want to support multiple tiers from day one -- especially
get the interfaces right.

-- 
Cheers,

David