Re: [RFC v2 0/1] mm: add source selector to memory.reclaim

["Li Zhe" <lizhe.67@bytedance.com>]

On 11/13/26 7:48 AM, Joshua Hahn wrote:
> On Mon, Nov 02, 2026 at 03:04:39PM +0800, Li Zhe wrote:
>
> Hi Zhe,
>
> Sorry that I got to this so late. I must have missed it in my inbox
> somehow!


Hi Joshua,

Thanks for the review and suggestions!


>
>> memory.reclaim is a useful proactive reclaim interface for memcgs, but
>> tiered memory makes "reclaim N bytes from this memcg" underspecified for
>> userspace controllers that manage memory placement across tiers. The
>> kernel reclaim path may demote some folios and evict others. Without a
>> way to declare where reclaim pressure should be applied, userspace can
>> only reclaim from the memcg as a whole and observe one aggregate progress
>> value.
> Does this series do anything to distinguish demotions from evicting?
> If not, I'm not sure why it is mentioned above.


No, it does not. Thanks for pointing this out. It is indeed unnecessary
to mention it here.

>
>> There is prior history here [1]. The old nodes= proposal tried to let
>> memory.reclaim scan selected NUMA nodes, but the discussion left several
>> ABI questions unresolved:
>>
>> - nodes= did not clearly say whether the mask was a source selector, a
>>    reclaim domain, or a demotion-target constraint.
>> - The implementation did not apply the mask to the whole reclaim path,
>>    so demotion could allocate outside the mask. If userspace interpreted
>>    the mask as a domain or target constraint, the ABI did not provide
>>    that.
>> - Selecting top-tier nodes could become an implicit way to request
>>    demotion without uncharge. That would make userspace depend on a
>>    particular reclaim policy and could make future policy changes hard.
>> - The separate completion-accounting question remained: demotion can
>>    count as reclaim progress even though memory.current is not reduced.
>>
>> This series takes a narrower approach. It adds a typed source= key which
>> declares only the resident source to which the existing memory.reclaim
>> path should apply pressure. It is not a demotion target selector, not a
>> demotion-only goal, not an eviction-only goal, and not a change to the
>> existing completion accounting. Leaving source unspecified preserves the
>> existing memory.reclaim behavior.
> Right, I was a bit confused why you presented not distinguishing between
> demotion and eviction as a problem, since this patch just preserves
> the existing behavior for reclaim.


Agreed. The patch only selects the reclaim source and otherwise preserves
the existing memory.reclaim behavior. I will make that clearer.

>
>> In that sense, this series extends node-scoped reclaim pressure to the
>> memcg memory.reclaim interface. The existing node memory_reclaim
>> interface can apply pressure to a node, but it is not scoped to a
>> particular memcg. This series lets userspace apply the same kind of
>> source-scoped pressure to a cgroup's resident memory on selected nodes,
>> while still using the normal memcg reclaim path.
>>
>> The typed source= form also follows a direction that has recently been
>> discussed for memory.reclaim. In the zswap proactive writeback series,
>> the interface was changed from a one-off zswap_writeback_only key to
>> source=zswap so that other reclaim sources could be selected in the
>> future [2][3]. This series uses the same extensible key shape, but
>> defines a different source class: resident NUMA nodes. source=zswap
>> bypasses ordinary reclaim and performs zswap writeback; source=nodes:
>> <nodelist> only constrains where the normal memcg reclaim path applies
>> pressure.
>>
>> The main benefit of this series is to give userspace a steering knob for
>> per-node memory placement control. A tiering controller can still build
>> per-tier policy on top of it by translating a tier to its node list, for
>> example from /sys/devices/virtual/memory_tiering/memory_tierN/nodelist.
>> It can then apply reclaim pressure to the nodes that should be relieved.
>> For example, it can apply pressure to top-tier nodes so that, under the
>> existing tiering policy, cold folios can usually be demoted to lower
>> tiers. Another controller may first reclaim from lower-tier nodes so that
>> later top-tier reclaim has room for demotion. A controller can also
>> reclaim from terminal-tier nodes when it needs actual charge reduction.
>>
>> The existing memory.reclaim interface cannot express this. It applies
>> pressure to the memcg as a whole, so on a multi-tier system the kernel
>> may make progress in a tier different from the one userspace is trying
>> to control. That makes a userspace feedback loop for tier ratios
>> ineffective: the controller can observe that a specific node or tier is
>> over its desired resident footprint, but it cannot direct the next
>> reclaim request to the corresponding source nodes.
>>
>> This version only adds source=nodes:<nodelist>, which is the basic
>> source selector. More convenience selectors can be considered later, but
>> the tier use case is already expressible by resolving a tier's current
>> nodelist in userspace and passing that nodelist to
>> source=nodes:<nodelist>. Starting with the node-based interface keeps the
>> ABI smaller while preserving the source= namespace for future selectors.
>>
>> The remaining issues from the old discussion are intentionally left out
>> of this series. If userspace needs to control demotion targets, that
>> should be a separate target/demotion-policy ABI rather than folded into
>> source=. If userspace needs a strict "bytes actually uncharged"
>> completion mode, that should be discussed as a separate accounting or
>> goal extension because it would change the observable meaning of
>> memory.reclaim progress. This series only adds source selection for the
>> existing reclaim policy.
>>
>> Testing:
>>
>> I ran a simple userspace feedback controller on a CXL test machine. The
>> machine had nodes 0-2 as the DRAM tier and node 3 as the CXL tier. The
>> basic matrix used node 0 as the DRAM source and node 3 as the CXL
>> source for the target memcg, with fio and stress-ng as the load
>> generators. The controller:
>>
>> - read anon/file bytes from memory.numa_stat by node;
>> - aggregated them into DRAM and CXL resident bytes;
>> - targeted a 1:1 DRAM:CXL resident ratio for the target memcg;
>> - selected the side above 55%;
>> - wrote memory.reclaim with source=nodes:<selected nodes>; and
>> - slept for the configured control period before the next sample.
>>
>> The controller loop was approximately:
>>
>> while running:
>>          stat = read_memory_numa_stat()
>>          dram = resident_bytes(stat, dram_nodes)
>>          cxl = resident_bytes(stat, cxl_nodes)
>>          total = dram + cxl
>>
>>          if dram > total * 0.55:
>>                  source_nodes = dram_nodes
>>                  excess = dram - total / 2
>>          else if cxl > total * 0.55:
>>                  source_nodes = cxl_nodes
>>                  excess = cxl - total / 2
>>          else:
>>                  sleep(period)
>>                  continue
>>
>>          bytes = clamp(excess * gain, min_step, max_step)
>>          write(memory.reclaim,
>>                "$bytes source=nodes:<source_nodes>")
>>          sleep(period)
>>
>> The requested reclaim size was proportional to the excess bytes, with
>> per-workload min/max bounds. I tested 1G, 5G, 10G, 20G and 50G
>> workloads with 1s, 2s, 5s and 10s control periods. A run was considered
>> within target when max(abs(dram_ratio - 0.5), abs(cxl_ratio - 0.5)) was
>> not greater than 5 percentage points.
>>
>> Basic feedback-controller matrix, final deviation from the 1:1 target:
>>
>> The 1s, 2s, 5s and 10s columns are controller periods.
>>
>> The file-cache-CXL-high case used fio under numactl to create about 25%
>> DRAM and 75% CXL page cache. It reached within 5% by at most the
>> eighth control cycle, and all 20 runs ended within 5%.
> These results are really confusing. What does the % mean here?
> % deviation from a 50-50 split? I want to review the code but I want
> to understand what the test is actually trying to do first.


Yes, the percentage is the final deviation from the 50:50 target:

   max(abs(dram_ratio - 0.5), abs(cxl_ratio - 0.5))

I should have stated that directly. I will make the test metric clearer
in v3.

>
> In any case, I think the important tests to demonstrate the code are:
> - How does this perform when the entire system is under pressure?
> - How are multiple cgroups simultaneously present on the host dealt
>    with? Are they also able to manage an effective split?


That makes sense. I will try to extend the tests to cover system-wide
pressure and multiple active cgroups in v3.

>
>> size  1s     2s     5s     10s
>>   1G   3.96%  3.96%  4.45%  3.96%
>>   5G   2.65%  2.75%  2.65%  2.65%
>> 10G   2.65%  2.68%  2.65%  2.69%
>> 20G   2.65%  2.66%  2.66%  2.64%
>> 50G   4.31%  4.31%  4.31%  4.31%
>>
>> The anon-DRAM-high case used stress-ng under numactl to create
>> anonymous memory on DRAM and used swappiness=max so that the normal
>> reclaim path could demote or reclaim anonymous memory. It reached
>> within 5% by at most the seventh control cycle, and all 20 runs ended
>> within 5%.
>>
>> size  1s     2s     5s     10s
>>   1G   0.15%  0.02%  0.16%  0.02%
>>   5G   3.16%  3.13%  3.14%  3.13%
>> 10G   3.13%  3.13%  3.15%  3.12%
>> 20G   3.12%  3.13%  3.12%  3.12%
>> 50G   1.88%  1.88%  1.88%  1.88%
>>
>> For comparison, I also ran the same controller and workloads without
>> the source=nodes selector. The controller still chose the side above the
>> target and computed the same request size, but wrote plain
>> memory.reclaim without a source constraint.
>>
>> case                  final <=5%  reached <=5%  worst deviation
>> file-cache CXL high        1/20          1/20     50.00%
>> anon DRAM high            20/20         20/20      5.00%
>>
>> This shows why the source selector matters for placement control. In
>> the file-cache-CXL-high case, plain memory.reclaim could not reliably
>> control the target ratio without source=nodes. In the anon-DRAM-high
>> case, plain memory.reclaim worked well. This is likely because, in this
>> setup, the whole-memcg reclaim walk reached DRAM nodes before CXL nodes,
>> so it already applied pressure to the over-target DRAM side first for
>> this workload.
>>
>> I also ran a disturbed case with external reclaim and demotion activity.
>> The target memcg used stress-ng under numactl to create about 25% DRAM
>> and 75% CXL anonymous memory. NUMA demotion and zswap were enabled, and
>> watermark_scale_factor was raised to 3000 so background reclaim became
>> visible in a short run. A ballast cgroup kept DRAM free memory low, and
>> a separate CXL-heavy cgroup kept reclaim activity present on the CXL
>> tier. The result below was measured on the target memcg.
>>
>> During the matrix, pgdemote_kswapd increased in 16/20 runs. The
>> CXL-free-down/DRAM-free-up/pgdemote pattern appeared in 13/20 runs.
>> Even with that external activity, the target memcg controller still
>> drove its own DRAM:CXL resident ratio toward 1:1.
>>
>> Disturbed case, final deviation from the 1:1 target. The 1s, 2s, 5s and
>> 10s columns are controller periods:
>>
>> size  1s     2s     5s     10s
>>   1G   1.18%  1.11%  1.11%  1.42%
>>   5G   2.14%  2.06%  2.14%  2.14%
>> 10G   2.13%  2.10%  2.15%  2.11%
>> 20G   2.15%  2.23%  2.15%  2.43%
>> 50G   4.87%  4.87%  4.89%  4.87%
>>
>> All 20 disturbed runs reached within 5% by the third control cycle and
>> ended within 5%. This does not make memory.reclaim a one-shot footprint
>> guarantee, but it shows that source=nodes can act as a useful
>> feedback-loop actuator.
>>
> Zhe, it would be great if the commit message could be a lot, lot
> shorter. I feel that this is just really too hard to read. And I'm not
> sure if there's a need to have a separate cover letter for a single
> patch.
>
> Please, I feel like this is really difficult to read. I think the
> commit message could be as short as 3 paragraphs and I think it would
> still capture everything that you're trying to say.


Agreed, and sorry for making this too long. I will make v3 much shorter
and drop the separate cover letter if it remains a single patch.

>
> Sorry again for the super late reply, Zhe. I'll be more on top of review
> next time! Have a great day,


Thanks again for the review and suggestions!

Thanks,
Zhe

> Joshua
>
>> References:
>> [1] https://lore.kernel.org/all/Y5bsmpCyeryu3Zz1@dhcp22.suse.cz/
>> [2] https://lore.kernel.org/all/20260629112032.20423-1-jiahao.kernel@gmail.com/
>> [3] https://lore.kernel.org/all/20260629112032.20423-5-jiahao.kernel@gmail.com/
>>
>> Li Zhe (1):
>>    mm: add source selector to memory.reclaim
>>
>>   Documentation/admin-guide/cgroup-v2.rst |  9 +++++
>>   mm/bpf_memcontrol.c                     |  3 +-
>>   mm/internal.h                           |  3 +-
>>   mm/memcontrol-v1.c                      |  6 ++-
>>   mm/memcontrol.c                         | 11 ++++--
>>   mm/vmscan.c                             | 49 +++++++++++++++++++++++--
>>   6 files changed, 70 insertions(+), 11 deletions(-)
>>
>> ---
>> v1: https://lore.kernel.org/all/20261019141401.11927-1-lizhe.67@bytedance.com/
>>
>> Changes since v1:
>> - Reworded the cover letter and documentation to describe the interface
>>    as applying reclaim pressure to a selected resident source.
>> - Added the lower-tier use case where userspace reclaims from a lower
>>    tier first so that later top-tier reclaim has room for demotion.
>> - Dropped patch 2, the tier convenience selector, for now. Tier-aware
>>    userspace can resolve a tier to a nodelist and use source=nodes.
>> - Added references to the zswap proactive writeback discussion, which
>>    also uses the typed source= form to keep memory.reclaim extensible.
>> - Added CXL feedback-controller test results using fio and stress-ng
>>    load generators, including a no-source baseline.
>>
>> -- 
>> 2.20.1