Re: [RFC PATCH v0 0/2] Batch migration for NUMA balancing

[SeongJae Park <sj@kernel.org>]

On Mon, 26 May 2025 10:50:02 +0530 Bharata B Rao <bharata@amd.com> wrote:

> Hi SJ,
> 
> On 22-May-25 12:15 AM, SeongJae Park wrote:
> > Hi Bharata,
> > 
> > On Wed, 21 May 2025 13:32:36 +0530 Bharata B Rao <bharata@amd.com> wrote:
> > 
> >> Hi,
> >>
> >> This is an attempt to convert the NUMA balancing to do batched
> >> migration instead of migrating one folio at a time. The basic
> >> idea is to collect (from hint fault handler) the folios to be
> >> migrated in a list and batch-migrate them from task_work context.
> >> More details about the specifics are present in patch 2/2.
> >>
> >> During LSFMM[1] and subsequent discussions in MM alignment calls[2],
> >> it was suggested that separate migration threads to handle migration
> >> or promotion request may be desirable. Existing NUMA balancing, hot
> >> page promotion and other future promotion techniques could off-load
> >> migration part to these threads. Or if we manage to have a single
> >> source of hotness truth like kpromoted[3], then that too can hand
> >> over migration requests to the migration threads. I am envisaging
> >> that different hotness sources like kmmscand[4], MGLRU[5], IBS[6]
> >> and CXL HMU would push hot page info to kpromoted, which would
> >> then isolate and push the folios to be promoted to the migrator
> >> thread.
> > 
> > I think (or, hope) it would also be not very worthless or rude to mention other
> > existing and ongoing works that have potentials to serve for similar purpose or
> > collaborate in future, here.
> > 
> > DAMON is designed for a sort of multi-source access information handling.  In
> > LSFMM, I proposed[1] damon_report_access() interface for making it easier to be
> > extended for more types of access information.  Currenlty damon_report_access()
> > is under early development.  I think this has a potential to serve something
> > similar to your single source goal.
> > 
> > Also in LSFMM, I proposed damos_add_folio() for a case that callers want to
> > utilize DAMON worker thread (kdamond) as an asynchronous memory
> > management operations execution thread while using its other features such as
> > [auto-tuned] quotas.  I think this has a potential to serve something similar
> > to your migration threads.  I haven't started damos_add_folio() development
> > yet, though.
> > 
> > I remember we discussed about DAMON on mailing list and in LSFMM a bit, on your
> > session.  IIRC, you were also looking for a time to see if there is a chance to
> > use DAMON in some way.  Due to the technical issue, we were unable to discuss
> > on the two new proposals on my LSFMM session, and it has been a bit while since
> > our last discussion.  So if you don't mind, I'd like to ask if you have some
> > opinions or comments about these.
> > 
> > [1] https://lwn.net/Articles/1016525/
> 
> Since this patchset was just about making the migration batched and 
> async for NUMAB, I didn't mention DAMON as an alternative here.

I was thinking a clarification like this could be useful for readers though,
since you were mentioning the future work together.  Thank you for clarifying.

> 
> One of the concerns I always had about DAMON when it is considered as 
> replacement for existing hot page migration is its current inability to 
> gather and maintain hot page info at per-folio granularity.

I think this is a very valid concern.  But I don't think DAMON should be a
_replacement_.  Rather, I'm looking for a chance to make existing approaches
help each other.  For example, I recommend running DAMON-based memory
tiering[1] together with the LRU-based demotion.  I think there is no reason to
discourage using it together with NUMAB-2 based promotion, if the folio
granularity is a real issue.  That is, still NUMAB-2 will do synchronous
promotion, but DAMON will do it asynchronously, so the amount of synchronous
promotions and its overhead will reduce.

I didn't encourage using NUMB-2 based promotion together with DAMON-based
memory tiering[1] not because I show a problem at such co-usage, but just
because I found no clear benefit of that from my test setup.  In theory, I
think running those together makes sense.

That said, we're also making efforts for overcoming the folio-granularity issue
on DAMON side, too.  We implemented page-level filters that motivated by SK
hynix' test results, and developed monitoring intervals auto-tuning for overall
monitoring results accuracy.  We proposed damon_report_access() and
damos_add_folios() as yet another opportunitis to better deal with the issue.
I was curious about your opinion to damon_report_access() and
damos_add_folios() for the reason.  I understand that could be out of the scope
of this patch series, though.

> How much 
> that eventually matters to the workloads has to be really seen.

Cannot agree more.  Nonetheless, as mentioned abovely, my test setup[1] didn't
show the problem.  That said, I'm not really convinced with my test setup, and
I don't think the test setup is good for verifying the problem.  Hence I'm
trying to make a better test setup for this.  I'll share more of the new setup
if I make some progress.  I will also be more than happy to learn about other's
test setup if they have a good one or suggestions.

[1] https://lore.kernel.org/20250420194030.75838-1-sj@kernel.org


Thanks,
SJ

[...]