Re: [LSF/MM/BPF TOPIC] DAMON Requirements for Access-aware MM of Future

Gregory Price
2025-01-14T03:06:12Z

On Wed, Jan 01, 2025 at 02:20:39PM -0800, SeongJae Park wrote:
> Hi all,
>
>
> I find a few interesting and promising projects that aim to do efficient access
> pattern-aware memory management of near future, including below (alphabetically
> sorted).
>
> - Promotion of unmapped page cache folios
> (https://lore.kernel.org/20241210213744.2968-1-gourry@gourry.net)

I'll break down a few observations I made while hacking on unmapped
page cache promotion - and my concerns for a leveraging DAMON here.

Additionally some other concerns I've seen raised about duplicating
promotion logic across various kernel components.

Latest RFC:
https://lore.kernel.org/linux-mm/20250107000346.1338481-1-gourry@gourry.net/

Basic Premise:
Use folio_mark_accessed() as a measure of hotness for promotion.
Defer promotion to task_work due to locking complexities.

My major concerns / lessons learned from this exercise include:

1) The cost of checking promotion candidacy can be problematic

In my microbenchmark in the last RFC version, I showed that while
the performance upside (~22-25%) is substantial, there was a
non-trivial cost associated with injecting even a single global
boolean check in the file_read() path. This was unexpected.

I can probably optimize the disabled case with a likely() clause,
but I did not expect such sensitivity. This tells me injecting
an unconditional call into DAMON may be too much overhead.

I would need to explore this further - including whether it is
feasible to inject such a large dependency into swap.c

This may not affect all cases, but it does affect at least this one.

2) The complexity of "when it is safe" to promote a folio is subtle
at best, and "actively hostile" at worst.

I learned in v1 of the RFC that promotion inline with fma() is not
feasible due to a few contexts (task dying in particular) in which
migration is not safe. I deferred to task work because I noticed
prior attempts (in development notes) had seen similar issues.

Adding a folio reference and/or page flag to defer that migration to
another context (i.g. async kthread) solves this at the expensive of
implementation complexity. (leaked folios if done wrong)

I'd have to look at whether it's worth the increased complexity to
aggregate this (particular) identification mechanism - but I think
there is clear value to aggregating promotion.

I could see some value in pumping tracking bits into DAMON - but I
also see value is making tasks handle promotion as a form of fairness.

3) There were expressed opinions on runtime fairness WRT to promotion.

There's two competing thoughts:
A) Making accessing tasks eat inline promotion cost captures that
cost in their runtime slice, promoting fairness in scheduling.

B) Aggregating promotion to an external thread can reduce inline
faults and tail latencies, but may hides per-task cost. This
is a concern if one task drives all the promotions, effectingly
stealing an entire core by nature of the async design.

I don't have a good answer to this, just an observation that charging
promotion time to the identifying task was a concern that was raised.

4) TPP and Unmapped Page Promotion may affect each other.

There is a rate-limiting mechanism in the migration path that was
intended to prevent over-pressuring bandwidth with aggressive
migrations - prevent major memory stalls.

By adding more pressure on this limit from an additional source,
we're obviously increasing the time it takes to converge.

This is probably the greatest argument for creating a new, aggregated
promotion mechanism to serve all of these identification mechanism.

This would make it easier for us to determine whether/what
identification mechanisms can be aggregated while enabling forward
progress on each of them separately.

5) Scarce resources

We need to be careful not to consume excessive amounts of resources
in an attempt to track all these identifying mechanisms. Even 1 byte
per folio is 256MB on a 1TB machine. This gets out of hand quick.

With task-work, I was able to add no additional resource consumption,
but deferring to a fully async scenario and needing to track things
like last-accessing CPU, timestamps, and etc.

We'll need to examine this closely if we decide to aggregate either
of these mechanisms.

~Gregory